Oz version 0.1.0

I've just sent the following mail to aeolus-devel@lists.fedorahosted.org and cloud@lists.fedoraproject.org, announcing the first release of the Oz tool:

All,
I'm pleased to announce the first release of Oz, version 0.1.0. Oz is
a tool for doing automated installation of operating systems using the native
installation tools. While Oz is part of the Aeolus umbrella, and thus intended
to be used in a cloud management environment, it is also useful as a standalone
tool to do guest installation.

This first release focusses on providing the basic infrastructure to do guest
installs. There are 3 possible phases to a guest install (and all 3 are
optional):

1) Installation of a JEOS. Oz creates an automated installation file (a
kickstart, winnt.sif, preseed, etc), and then launches a KVM virtual machine
to do the installation. By default, a minimal JEOS is always installed to
reduce the likelihood of installation failure (though this behavior can be
overriden)

2) Customization. Once a guest is installed from step 1) (or otherwise
provided), the virtual machine is launched and a native tool is used to do
installation of additional packages and files. For instance, in the case of
modern linux, ssh is used to upload files and run commands to install
additional packages.

3) Manifest generation. After the guest installation and customization are
done (if any), a package manifest (and some other metadata) can be extracted
from the image.

As mentioned, all 3 steps above are optional; you can start at step 1 and run
all the way to step 3 using oz, or you can install a guest using another
method and skip directly to steps 2 or 3. Documentation of how to do these
sorts of things are on the Oz webpage at http://aeolusproject.org/oz.html.

At present, Oz can install the following types of guests:
Fedora: 7, 8, 9, 10, 11, 12, 13, 14
Fedora Core: 1, 2, 3, 4, 5, 6
RHEL 2.1: GOLD, U2, U3, U4, U5, U6
Fedora Core: 1, 2, 3, 4, 5, 6
RHEL 2.1: GOLD, U2, U3, U4, U5, U6
RHEL 3: GOLD, U1, U2, U3, U4, U5, U6, U7, U8, U9
RHEL 4: GOLD, U1, U2, U3, U4, U5, U6, U7, U8
RHEL 5: GOLD, U1, U2, U3, U4, U5, U6
RHEL 6: 0
Ubuntu: 6.10, 7.04, 7.10, 8.04.[1,2,3,4], 8.10, 9.04, 9.10
Windows: 2000, XP, 2003
RHL: 7.0, 7.1, 7.2, 7.3, 8, 9
OpenSUSE: 11.1, 11.2, 11.3

Note that not all guest types can do all 3 of the steps above. The Oz webpage
lists which backends can do which phases.

Instructions on how to download Oz are at
http://aeolusproject.org/oz-download.html

As usual, comments, questions, bug reports, and patches are welcome for Oz.

Headed off to FUDCon

In a couple of hours I'll be boarding a plane to Phoenix to attend my first FUDCon. Assuming my talk gets voted on, I'll be talking about Deltacloud and Aeolus, two of the cloud projects currently going on at RedHat. I don't go to very many conferences, and I've never been to FUDCon before, so this should be an interesting experience.

If you are going, see you there!

Exiting a Python program

You'd think this would be a straightforward topic, wouldn't you? In general it is simple, but there is a gotcha.

When you want to exit a program written in python, the typical way to do it is to call sys.exit(status) like so:

import sys

sys.exit(0)

For simple programs, this works great; as far as the python code is concerned, control leaves the interpreter right at the sys.exit method call. If you look under the hood, though, sys.exit is a little more interesting. It does not immediately call the libc function exit(), but instead raises a SystemExit[1] exception. If nothing catches the exception, then the python interpreter catches it at the end, does not print a stack trace, and then calls exit.

There are situations in which you do not want to raise an exception to exit the program. The example I recently came across was debugging some of my python code in Oz[2]. Oz is careful to clean up after itself, which means that on many exceptions it catches the exception, does some cleanup steps, and then re-raises the exception. In general this is a good thing to do, but sometimes when debugging I want to totally skip the cleanup steps so I can examine what went wrong.

Enter os._exit(). This function is a thin wrapper around the libc function exit(), so it does not raise an exception and leaves the program immediately. You would use it like:

import os

os._exit(1)

[1] http://docs.python.org/library/exceptions.html
[2] http://www.aeolusproject.org/oz.html

Deltacloud support in upstream Condor

One of the main tasks that the Aeolus Conductor has to do is to schedule large numbers of cloud instances, possibly across several different cloud providers. In previous projects we've rolled our own sort of scheduler, but instead of re-inventing something for the Conductor, we decided to re-use a known component.

In this case we re-used Condor, a well-known Grid (and other) scheduler. There is good precedent for putting cloud-like functionality into condor; not too long ago, direct EC2 support was added, so using condor in this way was not unprecedented.

We didn't want to use the existing condor EC2 infrastructure, however, since that ties you into a single cloud. Instead we decided to write new code to utilize the Deltacloud API, which will give us nice cross-cloud functionality.

I'm pleased to say that as of January 7, the code to manage cloud instances via deltacloud is in upstream condor. That means that this code will be part of the condor 7.6 release. It is a bit complicated to use, which is why we hide the whole thing behind the conductor. However, it is possible to use the deltacloud support by hand, which I will describe below.

In order to take full advantage of condor's matching and scheduling capabilities, our current solution relies on having provider classads and instance classads. The provider classads describe each of the cloud provider combinations, along with their values. For instance, if you had two AMIs on EC2 that you wanted to be able to use, you would generate two separate classads with all of the data the same except for the AMI id. A provider classad looks something like:

Name="provider_combination_1"
MyType="Machine"
Requirements=true
# Stuff needed to match:
hardwareprofile="large"
image="fedora13"
realm="1"
# Backend info to complete this job:
image_key="ami-ac3fc9c5"
hardwareprofile_key="m1.large"
realm_key="us-east-1a"
provider_url="http://localhost:3002/api"
username="ec2_username"
password="ec2_secret_access_key"
cloud_account_id="1"
keypair="mykeypair"

(pruned for brevity and to protect the innocent). There are a few things to note here. First, in the "stuff needed to match" category are generic things the job classad will match against. That is, the instance classad should not need to know any specifics of the cloud provider backend, just that it wants to run Fedora 13 on a large type instance. Once a provider (or providers) have been found that matches those requirements, the values in the "backend info the complete this job" section are substituted into the actual instance classad, and then the job is submitted.

The jobs themselves then look something like:

universe = grid
executable = job_name
notification = never
requirements = hardwareprofile == "large" && image == "fedora13"
grid_resource = deltacloud $$(provider_url)
DeltacloudUsername = $$(username)
DeltacloudPassword = $$(password)
DeltacloudImageId = $$(image_key)
DeltacloudHardwareProfile = $$(hardwareprofile_key)
DeltacloudKeyname = $$(keypair)
queue

There is a lot going on here, so let's break it down. The first three fields are necessary, but not that interesting; they just give the job a name, tell condor to use the grid universe, and tell condor not to send notifications. The interesting things start happening on the "requirements" line. This is the line that specifies what has to match in order for this job to run. In this case, we are saying that condor gridmanager should look through the provider ads looking for one that has a hardwareprofile that matches "large" and an image that matches "fedora13". These values match what was put in our provider ad above, so condor selects that provider ad. Once it selects that provider ad, it can fill in the rest of the job ad $$ substitutions and submit the job. For instance, in our job ad DeltacloudUsername = $$(username). Once our provider ad has been selected above, that $$(username) gets replaced with "ec2_username". The same happens for the rest of the $$ substitutions.

Once condor has made the match, filled in the values and submitted the job, it will then contact the deltacloud core (from the grid_resource line) to submit the job. If all has gone well to this point, then the instance will actually be launched in the cloud. The instance can also be controlled via condor; if you "condor_rm" the job, then the instance in the cloud will be killed off.

ruby-libvirt in EPEL

Ohad Levy has recently implemented support for launching VMs via libvirt in The Foreman. As part of that, he needed ruby-libvirt bindings available in RHEL 5 and 6, so I've now built EPEL packages for both of these. They are in the "updates-candidates" tag for the time being; to get direct access, the packages are here:

EPEL 5
EPEL 6

Writing Ruby Extensions in C - Part 12, Allocating memory

This is the twelfth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talked about dealing with numbers. The eighth post talked about strings. The ninth post focused on arrays. The tenth post looked at hashes. The eleventh post explored blocks and callbacks. This post will look at allocating and freeing memory.

Allocating memory

When creating a new ruby object, memory will be automatically allocated from the garbage collector as needed.

If the ruby extension needs to allocate C-style memory, the basic malloc/realloc/calloc calls can be used. However, there are ruby counterparts that do the work of malloc/realloc/calloc in a slightly better way. The advantage of the following calls is that they first try to allocate memory, and if they fail, they will invoke the garbage collector to free up a bit of memory and try again. That way if the program is low on memory, or the address space is fragmented because of the ruby memory allocator, these functions will succeed where basic malloc/realloc/calloc would fail:

• ALLOC(type) - allocate a structure of the pointer type
• ALLOC_N(type, num) - allocate num structures of pointer type
• REALLOC_N(var, type, num) - realloc var to num structure of pointer type

It is important to use xfree() to free the memory allocated by these calls. In the nominal case there isn't much difference between regular free() and xfree(), but if ruby is built a certain way, xfree() does some additional internal accounting. In any case, there is no reason not to use xfree(), so it is recommended to always use xfree(). Thanks to SodaBrew for pointing this out in the comments.

A simple example to demonstrate the use of these functions:

 1) struct mystruct {
 2)     int a;
 3)     char *b;
 4) };
 5)
 6) static VALUE implementation(VALUE a) {
 7)     struct mystruct *single;
 8)     struct mystruct *multiple;
 9)
10)     single = ALLOC(struct mystruct);
11)     xfree(single);
12)
13)     multiple = ALLOC_N(struct mystruct, 5);
14)
15)     REALLOC_N(multiple, struct mystruct, 10);
16)
17)     xfree(multiple);
18)
19)     return Qnil;
20) }

Lines 1 through 4 just define a simple structure containing a char * and an int. The implementation of a ruby method on lines 6 through 20 show the use of the allocation functions. Line 10 shows the allocation of a single structure of type struct mystruct, which is freed on line 11. Line 13 shows the allocation of an array of 5 elements of struct mystructs into the multiple pointer. Line 15 shows the reallocation of the multiple array to 10 elements. Notice that since REALLOC_N is a macro, it operates slightly differently than realloc(); in particular, there is no need (and no way) to re-assign the pointer. Finally, line 17 frees up the multiple pointer and line 19 returns successfully from the function.

Error handling and not leaking memory

Ruby is a garbage collected language, meaning that applications don't generally have to worry about freeing memory after it is used. This garbage collection extends into C extension modules, but only to a certain point. If you are writing a C extension to ruby, there are some places that you have to worry about keeping track of your pointers and freeing them up. To understand why, we need to dig a little into the memory allocation functions of ruby.

When you are writing pure ruby, and execute a line of code like:

x = ['a']

the ruby virtual machine causes some memory to come into existence to hold that list for you. The way that this memory is allocated is with rb_ary_new() (or one of its derivatives). The call chain looks like: rb_ary_new() -> rb_ary_new2() -> ary_new() -> NEWOBJ() -> rb_newobj(). Inside of rb_newobj(), no memory is actually allocated; instead, the new object that we need to come into existence is just taken off of the list of free objects, and the free list head is moved to the next object. If it turns out that no memory is available in this freelist, the garbage collector is run to try to reap some memory, and then the memory is given to this new object. Because this memory is coming from the freelist, it is all involved with (and can later be reaped by) the garbage collection.

When you allocate memory in C code using malloc (or one of its derivatives), no such thing happens. The memory is properly allocated, but it is not involved in any of the garbage collection schemes. This leads to 2 problems:

1. Since malloc isn't involved in the garbage collection, the malloc can fail earlier than it normally would due to address space fragmentation. This isn't generally a problem on 64-bit architectures, but it could crop up as a problem on 32-bit ones.
2. If a ruby call in your extension module fails, it will throw an exception. In ruby, exceptions are done via a longjmp out of the extension code and into the ruby exception handling code. If you have allocated any memory with malloc and friends, you have now lost the pointers to that memory, so you now have a memory leak (apparently this problem is much worse when dealing with C++; see [1]).

Problem 1) is partially solved by using the built-in ruby ALLOC, ALLOC_N, and ruby_xmalloc functions. Problem 2) is much more insidious, and more difficult to handle. Luckily, it is not impossible to handle.

Assume you have the following code snippet:

 1) int *ids;
 2) VALUE result;
 3) int i;
 4)
 5) ids = ALLOC_N(int, 5);
 6) for (i = 0; i < 5; i++)
 7)     ids[i] = i;
 8)
 9) result = rb_ary_new2();
10)
11) for (i = 0; i < 5; i++)
12)     rb_ary_push(result, INT2NUM(ids[i]));
13)
14) xfree(ids);

(while this is a bit of a contrived example, it actually bears a lot of resemblance to this[2] code in ruby-libvirt)

What this code is trying to do is to create an array full of the values in the "ids" array. If there are no errors, then this code works absolutely fine and doesn't leak any memory (ids gets freed at line 14, and the ruby array will get reaped by the garbage collector eventually). However, if either rb_ary_new2() or rb_ary_push() fails in lines 9 or 12, then they will automatically longjmp to the ruby exception handler, completely skipping the xfree at line 14. This code has now leaked memory.

The way to fix this is to interrupt ruby's normal longjmp on exception mechanism so that you can insert code of your own before throwing the exception. The rb_protect() ruby call can be used to do exactly this. Unfortunately the interface is a bit clunky, but we have to do what we have to do.

rb_protect() takes 3 arguments: a name of a callback function that takes 1 (and exactly 1 argument), the argument to pass to that callback function, and a pointer to an integer to store the exception address (if any). Because the callback function can only take one argument, typical usage is to create a callback "wrapper" that takes the one and only argument. The data that you pass in can be anything, so if you want to pass in multiple arguments, you can do so by passing in a pointer to a structure containing all of the data that you need. An example should help clarify some of this:

 1) struct rb_ary_push_arg {
 2)     VALUE arr;
 3)     VALUE value;
 4) };
 5)
 6) static VALUE rb_ary_push_wrap(VALUE arg) {
 7)     struct rb_ary_push_arg *e = (struct rb_ary_push_arg *)arg;
 8)
 9)     return rb_ary_push(e->arr, e->value);
10) }
11)
12) int *ids;
13) VALUE result;
14) int i;
15) int exception = 0;
16) struct rb_ary_push_arg args;
17)
18) ids = ALLOC_N(int, 5);
19) for (i = 0; i < 5; i++)
20)     ids[i] = i;
21)
22) result = rb_ary_new2();
23)
24) for (i = 0; i < 5; i++) {
25)     args.arr = result;
26)     args.value = INT2NUM(ids[i]);
27)     rb_protect(rb_ary_push_wrap, (VALUE)&args, &exception);
28)     if (exception) {
29)         xfree(ids);
30)         rb_jump_tag(exception);
31)     }
32) }
33)
34) xfree(ids);

Now when we add entries to the ruby array, we are doing so through the rb_ary_push_wrap() function, called by rb_protect(). This means that if rb_ary_push() fails for any reason and throws an exception, control will be returned back to the code above at line 28, but with exception set to a non-zero number. We have a chance to clean up after ourselves, and then continue propagating the exception with rb_jump_tag(). Note that with the use of a proper structure, we can pass any number of arguments through to the wrapper function, so we can use this for all internal ruby functions. Notice that I did not wrap rb_ary_new2(), even though that can cause the same problem; I leave this as an exercise to the reader.

[1] http://www.thoughtsincomputation.com/posts/ruby-c-extensions-c-and-weird-crashing-on-rb_raise
[2] http://libvirt.org/git/?p=ruby-libvirt.git;a=blob;f=ext/libvirt/domain.c;h=eb4426252af635311e14e234a62780fbd4048f0b;hb=HEAD#l80

Aeolusproject.org

One of the projects I work on at Red Hat is cloud management. Our cloud projects rely on Deltacloud, a cross-cloud API, to do the real work on the cloud.

Due to some historical baggage, some of the other cloud projects were also under the Deltacloud name although they didn't directly deal with the API. In order to reduce confusion, we have renamed these projects under a new umbrella project, Aeolus. The website for this is http://www.aeolusproject.org. The main project under the Aeolus umbrella is the Conductor, a web-based UI for managing instances in clouds. There are also several other subprojects that help the Conductor do its job, including:

• Oz (an automated guest installation system)
• Image Factory (a project to do installation of guests along with transformation of them to particular cloud backends)
• Image Warehouse (a project to move data from place to place depending on user-specified rules)
• Audrey (a system for configuring cloud instances on-the-fly)

If you have interest in what Red Hat is doing in the cloud, please stop by the website and see what we have to offer. As always, we welcome comments, questions, constructive criticism, and contributions; please send such to the Aeolus mailing list.