OAR Cloud Project

This project aims at creating a light cloud computing architecture on top of the batch scheduler OAR. The main goal is to bring the best of the cloud computing in the Grid world.

You can access to the Git repository on github.

Project Members

This project is proposed by:

Olivier Richard - Teacher and researcher in RICM's Polytech Grenoble training

Three students from RICM are working on it:

Jordan Calvi (RICM4)
Alexandre Maurice (RICM4)
Michael Mercier (RICM5)

State of the art

This project begins on January 2013. Here is the current state of the art concerning the cloud computing technologies and the underlying virtualization technologies.

Cloud computing

Definition

The cloud computing is the use computing and storing resources through a network. It is providing a stack of services, from the hardware to the applications, named "Something as a Service". You can read a more exhaustive definition on Wikipedia.

IaaS

The layer we are interested in, is the Infrastructure as a Service layer (IaaS). It provided access to computation resources with an hardware abstraction made by virtualization. The leader in this domain is Amazon EC2. There IaaS proprietary platform is considered as the industrial standard and almost every other cloud computing solution are compatible with it's API. They provide, through a Command Line Interface (CLI) and a Web interface, a simple way to deploy scaling web applications and services. This is an on-demand self-service and you pay only when you are using it. The main features of Amazon EC2 are (from the Amazon EC2 web site):

Select a pre-configured, templated Amazon Machine Image (AMI) to get up and running immediately. Or create an AMI containing your applications, libraries, data, and associated configuration settings.
Configure security and network access on your Amazon EC2 instance.
Choose which instance type(s) you want, then start, terminate, and monitor as many instances of your AMI as needed, using the web service APIs or the variety of management tools provided.
Determine whether you want to run in multiple locations, utilize static IP endpoints, or attach persistent block storage to your instances.
Pay only for the resources that you actually consume, like instance-hours or data transfer.

But there is lots of alternatives to Amazon. Several company provide the same kind of services using proprietary or open source solutions. You can see a comparison of some of these solutions here.

Virtualization technologies

Hardware virtualization

There is a lot of system virtualization technologies. For system virtualization Amazon is using the XEN virtualization technology like most of their competitors. As we are looking for open source technologies, there is two serious alternatives to consider: KVM and LXC.

Xen: Well known and used in big mainframe cloud solution, Xen is consider as reliable. It is a bare-metal (type1) hypervisor that pivot directly on hardware. This means that every operating systems that runs on top of this hypervisor needs to be ported to fit with the paravirtualization API. It recently provide an hardware-assisted virtualization like KVM.

KVM: Included to the Linux Kernel, KVM (Kernel-based Virtual Machine) is using hardware-assisted virtualization for x86 processors with the virtualization extensions (Intel VT or AMD-V). It seems to be quite equivalent to Xen, but some say that it's quite easier to use.

LXC: The most recent one, LXC for LinuX Container use an other kind of virtualization technique based on isolation. The major drawback is that it works only for Linux OS or program. Also, the actual stable version (0.7.5) is still a work-in-progress and the API is not stabilize yet. There is lots of security issues not really solve for now[1] [2]. The major advantage is speed and the fact that LXC is directly accessible into the kernel.

We want to know if LXC can be a good alternative to Xen or KVM.

Libvirt is a useful tool for virtualization. It allows to use different kind of virtualization tools (above-mentioned) within a single interface. It is using XML configuration file to describe the VMs and a dedicated driver for each kind of virtualization tool.

Network virtualization

In cloud computing infrastructure we need to connect the Virtual Machines (VMs). To achieve this, the network virtualization is used. In the Linux world the most common virtual network used the virtual interface named bridge. The bridge is forwarding packet at Layer 2, so it is independent of upper layer protocols. It can be manage via the brctl command. Advanced features, like filtering or firewalling, are done using the ebtable project.

A more complete tools named OpenVSwitch is open source and commonly used in lots of cloud computing solutions. It's basically a virtual switch that can handle VMs inside a node or between multiple node over real switch.

To fit our demand regarding the network we will try to use OpenVSwitch to build our solution.

Conception

Here is the big picture of what could be the entire solution. It aims at giving us an overview of the project and helping us to design a roadmap.

Context

There is two kind of actors that are dealing with OAR cloud, users and administrators. The F.u* and the F.a* are the user and Administrator features describe below.

Features

User

Main features:

F.u.0 Connect to an account

F.u.1 Launch and configure one or more instances

F.u.2 Deploy an image on one or more instances

F.u.3 Modify and save images

F.u.4 Setup alarms based on rules using metrics

F.u.5 Being inform by e-mail and/or notification for interesting events

Advanced features:

F.u.6 Automated resize of an instance (adapt the resources) using predefined rules and schedule

F.u.7 Load balancing between several instances

F.u.8 Advanced Network configuration for user: ACL, subnets, VPN...

Administrator

F.a.0 Create/delete user account

F.a.1 Add/remove and manage resources

F.a.2 Visualize resources and instances states

F.a.3 Install and update nodes operating systems

F.a.4 Handle users access rights

F.a.5 Setup alarms based on rules using metrics

F.a.6 Being inform by e-mail and/or notification for interesting events

Logical Views

Here is the logical view of the OAR Cloud system. Every component on this diagram represents a software component type. The links between these components represent the communication between them.

Version 0.1

This is the first version of what could be the entire system architecture. It is design to make all the above features works.

Description of the main components:

AccountManager: Handle users and admins access rights
AccessPoint: The system access point reached by the different access tools
InstanceManager: Manage the creation, configuration and deletion of instances all over the severals nodes. It also handles the appliances persistence and deployment
UserCLI & AdminCLI: Command line access tools for users and admins

Version 0.2

Not relevant : skip it!

Version 0.3

This logical view was made after the M1 milestone and based on the knowledge gained this step. It is a much more minimal approach. However, it only complete the F.u.0 to F.u.3 and F.a.0 to F.a.4 features.

Description of the main component:

OARVirtCLI: It is a command line tool (CLI) that is install on the user's computer. It allows to :

create an instance
display the available images
run one or more images on an instance
connect to a running image
stop a running image
delete an instance

How it works

The user create an instance

He gets the available images from his instance

He launch the the chosen image

Then he connect himself to the image using ssh

 ssh root@IP

Milestones

This table presents the milestones of the project. Each Milestones are described below.

Project milestones
	OAR Resource reservation	LXC	libvirt	OpenVSwitch	VM SSH Access	Access Security	Network isolation	Resource cleanup	Client CLI	Storage
M1	X	X						X
M2		X	X	X
M3	X	X	X	X	X			X

M1

In an Ubuntu 12.04 LTS environement

install and configure OAR
install and configure LXC
make OAR reservation
launch one or more VM using LXC
Connect to the VM
check if killing the job do kill the VM
script this!

The cigri devel appliance was used as an configuration example for this. The Ubuntu 12.04 LTS distribution has been chosen because it seems to be one of the few distributions where LXC works out-of-the-box.

Encountered problems

OAR settings

the job manager "job_resource_manager_cgroups.pl" generate cpuset errors

   
   [job_resource_manager_cgroups][41][DEBUG] init
   mount: special device none does not exist
   rm: cannot remove `/dev/cpuset': Is a directory
   ln: failed to create symbolic link `/dev/cpuset/oar_cgroups': Operation not permitted
   [job_resource_manager_cgroups][41][ERROR] Failed to mount cgroup pseudo filesystem

the job manager "job_resource_manager.pl" generate cpuset errors too

   
   [job_resource_manager][40][DEBUG] init
   [debug] [2013-02-21 20:03:15.153] [MetaSched] Start of meta scheduler
   sh: 1: cannot create /dev/cpuset//oar/cpu_exclusive: Permission denied
   [job_resource_manager][40][ERROR] Failed to create cpuset /oar

I thought the problem come from a database conflict so I tried to use

   
   % sudo  oar-database --reset
   Are you sure you want to reset your database ? (The database content will be lost) [y/N]: y
   resetting the database 'oar'...
   ERROR 1064 (42000) at line 2: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'schema' at line 1
   Fail to execute /usr/lib/oar/database/mysql_reset_structure.sql
   . at /usr/sbin/oar-database line 188, <FIN> line 1.

I tried to run the `update_cpuset_id.sh` script but it shows an error message either:

   
   % sudo /etc/oar/update_cpuset_id.sh 127.0.0.1
   The authenticity of host '[127.0.0.1]:6667 ([127.0.0.1]:6667)' can't be established.
   RSA key fingerprint is 72:91:a6:40:29:60:b2:c2:18:ba:b7:66:4a:c5:d7:2f.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '[127.0.0.1]:6667' (RSA) to the list of known hosts.
   Permission denied (publickey,keyboard-interactive).
   DBD::mysql::st execute failed: Unknown column 'ip' in 'where clause' at /usr/share/perl5/OAR/IO.pm line 4774.
   DBD::mysql::st fetchrow_hashref failed: fetch() without execute() at /usr/share/perl5/OAR/IO.pm line 4776.

The problem comes from the cgroup-lite service that run by default in an Ubuntu 12.04. Stop this service using

 service cgroup-lite stop

solve the problem for OAR but puts LXC down.

I find a trick to make OAR and LXC working together: I disable the cpuset feature of OAR. In the /etc/oar/oar.conf (there is a copy in the M1 folder) I have comment CPUSET_PATH and set to yes OARSUB_FORCE_JOB_KEY as it is provided in the CPUSET_PATH comment.

Thus, I could run an LXC container inside a job. The container was vanished when the job has been killed.

Questions

Is the OAR cpuset mandatory, even if the LXC manage it?
Is LXC mature enough to use it in production?

M1 Conclusion

During this milestone we learned several things:

libvirt uses a separate implementation of LXC and not the LXC package itself. Also, it can be harder to use and cuts some LXC functionalities like reboot and shutdown a container.
by default OAR and LXC (like it works on Ubuntu 12.04) are in conflict because of the use of cgroups by OAR. The current default mount point of cgroups with LXC use /sys/fs/cgroup/. Each subsystems (cpuset, cpuacct, memory,...) is mount separately on /sys/fs/cgroup/<cgroup-subsystem> where <cgroup-subsystem> is one of the subsystem name. It seems that rewriting the cgroups management part of the job_resource_manager_cgroups.pl file, to fit with the cgroups mount points, should eventually makes them works together. After some try, we found out that oarsh and oarsh_shell should be modify too as it is said in the documentation but the functions to modify does not exists. However it might be possible to configure OAR and to works with LXC if we dig deeper.
The LXC security is not efficient by default. Some distribution are using SELinux to secure the container but OAR is not working with SELinux. Even so, there is several ways to make a container secure using AppArmor or systemd with a sandbox[3].
LXC is quite a young project and it's not stabilized for now. The documentation is a bit poor and makes LXC works depends on a lots of parameters. Moreover, depending on the Linux distribution the LXC configuration is totally different. It makes this tool really hard to understand and configure.

We can notice that systemd-nspawn, a kind of container launcher based on systemd. It is currently just a hack tool but it might become an LXC alternative for systemd based distributions.

M2

In Ubuntu 12.04 LTS

install and configure LXC, libvirt and OpenVswitch
launch at least 2 VMs
make the VMs to ping each others
script this!

Problems encountered during M2

We found this page explaining how to use OpenVswitch (OVS) to connect two EC2 instance with a GRE tunnel. The script attached to this page was used to set up an OVS bridge to replace the default linux bridge. Two problems appeared:

the containers with OVS bridge makes more then 2 minutes to be launched while it takes less then 10 seconds with Linux bridges
when you are using an OVS bridge between two containers. It's is possible to reach one by an other but sometimes one of the container loose his IP address and breaks the connection.

M2 Conclusion

The use of OVS as the main bridge for an LXC container is unstable. It seems that LXC and OpenVswitch are not really compatible for now.

M3

TODO

Tools

Internal links

UML

External links

a getting started page for using LXC with libvirt
the cgroups documentation
a slide show about LXC security.

Journal

08/04

add a script to configure LXC with Open vSwitch and connect 2 VMs

19/03

add content to M2 part

15/03

begin presentation (for Michael who's leaving the project the 21/03)

12/03

add 0.3 version in conception part

11/03

begin of M2

09/03

add State of the art part

08/03

M1 is closed. OAR cpuset and LXC does not work in competition in the same node (see why)

22/02

M1 works in simple mode (OAR does not use cpuset)

19/02

milestone definitions

07/02

add Conception part

04/02

We have specified the subject
Distribute the work between us.
- Jordan: LXC and Libvirt
- Alexandre: OpenVSwitch and Libvirt
- Michael: OAR and global architecture

Proj-2012-2013-OAR-Cloud