Friday, January 28, 2011

Headed off to FUDCon

In a couple of hours I'll be boarding a plane to Phoenix to attend my first FUDCon. Assuming my talk gets voted on, I'll be talking about Deltacloud and Aeolus, two of the cloud projects currently going on at RedHat. I don't go to very many conferences, and I've never been to FUDCon before, so this should be an interesting experience.

If you are going, see you there!

Monday, January 24, 2011

Exiting a Python program

You'd think this would be a straightforward topic, wouldn't you? In general it is simple, but there is a gotcha.

When you want to exit a program written in python, the typical way to do it is to call sys.exit(status) like so:

import sys

sys.exit(0)
For simple programs, this works great; as far as the python code is concerned, control leaves the interpreter right at the sys.exit method call. If you look under the hood, though, sys.exit is a little more interesting. It does not immediately call the libc function exit(), but instead raises a SystemExit[1] exception. If nothing catches the exception, then the python interpreter catches it at the end, does not print a stack trace, and then calls exit.

There are situations in which you do not want to raise an exception to exit the program. The example I recently came across was debugging some of my python code in Oz[2]. Oz is careful to clean up after itself, which means that on many exceptions it catches the exception, does some cleanup steps, and then re-raises the exception. In general this is a good thing to do, but sometimes when debugging I want to totally skip the cleanup steps so I can examine what went wrong.

Enter os._exit(). This function is a thin wrapper around the libc function exit(), so it does not raise an exception and leaves the program immediately. You would use it like:

import os

os._exit(1)

[1] http://docs.python.org/library/exceptions.html
[2] http://www.aeolusproject.org/oz.html

Friday, January 21, 2011

Deltacloud support in upstream Condor

One of the main tasks that the Aeolus Conductor has to do is to schedule large numbers of cloud instances, possibly across several different cloud providers. In previous projects we've rolled our own sort of scheduler, but instead of re-inventing something for the Conductor, we decided to re-use a known component.

In this case we re-used Condor, a well-known Grid (and other) scheduler. There is good precedent for putting cloud-like functionality into condor; not too long ago, direct EC2 support was added, so using condor in this way was not unprecedented.

We didn't want to use the existing condor EC2 infrastructure, however, since that ties you into a single cloud. Instead we decided to write new code to utilize the Deltacloud API, which will give us nice cross-cloud functionality.

I'm pleased to say that as of January 7, the code to manage cloud instances via deltacloud is in upstream condor. That means that this code will be part of the condor 7.6 release. It is a bit complicated to use, which is why we hide the whole thing behind the conductor. However, it is possible to use the deltacloud support by hand, which I will describe below.

In order to take full advantage of condor's matching and scheduling capabilities, our current solution relies on having provider classads and instance classads. The provider classads describe each of the cloud provider combinations, along with their values. For instance, if you had two AMIs on EC2 that you wanted to be able to use, you would generate two separate classads with all of the data the same except for the AMI id. A provider classad looks something like:

Name="provider_combination_1"
MyType="Machine"
Requirements=true
# Stuff needed to match:
hardwareprofile="large"
image="fedora13"
realm="1"
# Backend info to complete this job:
image_key="ami-ac3fc9c5"
hardwareprofile_key="m1.large"
realm_key="us-east-1a"
provider_url="http://localhost:3002/api"
username="ec2_username"
password="ec2_secret_access_key"
cloud_account_id="1"
keypair="mykeypair"

(pruned for brevity and to protect the innocent). There are a few things to note here. First, in the "stuff needed to match" category are generic things the job classad will match against. That is, the instance classad should not need to know any specifics of the cloud provider backend, just that it wants to run Fedora 13 on a large type instance. Once a provider (or providers) have been found that matches those requirements, the values in the "backend info the complete this job" section are substituted into the actual instance classad, and then the job is submitted.

The jobs themselves then look something like:
universe = grid
executable = job_name
notification = never
requirements = hardwareprofile == "large" && image == "fedora13"
grid_resource = deltacloud $$(provider_url)
DeltacloudUsername = $$(username)
DeltacloudPassword = $$(password)
DeltacloudImageId = $$(image_key)
DeltacloudHardwareProfile = $$(hardwareprofile_key)
DeltacloudKeyname = $$(keypair)
queue

There is a lot going on here, so let's break it down. The first three fields are necessary, but not that interesting; they just give the job a name, tell condor to use the grid universe, and tell condor not to send notifications. The interesting things start happening on the "requirements" line. This is the line that specifies what has to match in order for this job to run. In this case, we are saying that condor gridmanager should look through the provider ads looking for one that has a hardwareprofile that matches "large" and an image that matches "fedora13". These values match what was put in our provider ad above, so condor selects that provider ad. Once it selects that provider ad, it can fill in the rest of the job ad $$ substitutions and submit the job. For instance, in our job ad DeltacloudUsername = $$(username). Once our provider ad has been selected above, that $$(username) gets replaced with "ec2_username". The same happens for the rest of the $$ substitutions.

Once condor has made the match, filled in the values and submitted the job, it will then contact the deltacloud core (from the grid_resource line) to submit the job. If all has gone well to this point, then the instance will actually be launched in the cloud. The instance can also be controlled via condor; if you "condor_rm" the job, then the instance in the cloud will be killed off.

Thursday, January 20, 2011

ruby-libvirt in EPEL

Ohad Levy has recently implemented support for launching VMs via libvirt in The Foreman. As part of that, he needed ruby-libvirt bindings available in RHEL 5 and 6, so I've now built EPEL packages for both of these. They are in the "updates-candidates" tag for the time being; to get direct access, the packages are here:

EPEL 5
EPEL 6

Tuesday, January 18, 2011

Writing Ruby Extensions in C - Part 12, Allocating memory

This is the twelfth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talked about dealing with numbers. The eighth post talked about strings. The ninth post focused on arrays. The tenth post looked at hashes. The eleventh post explored blocks and callbacks. This post will look at allocating and freeing memory.

Allocating memory


When creating a new ruby object, memory will be automatically allocated from the garbage collector as needed.

If the ruby extension needs to allocate C-style memory, the basic malloc/realloc/calloc calls can be used. However, there are ruby counterparts that do the work of malloc/realloc/calloc in a slightly better way. The advantage of the following calls is that they first try to allocate memory, and if they fail, they will invoke the garbage collector to free up a bit of memory and try again. That way if the program is low on memory, or the address space is fragmented because of the ruby memory allocator, these functions will succeed where basic malloc/realloc/calloc would fail:
  • ALLOC(type) - allocate a structure of the pointer type
  • ALLOC_N(type, num) - allocate num structures of pointer type
  • REALLOC_N(var, type, num) - realloc var to num structure of pointer type

It is important to use xfree() to free the memory allocated by these calls. In the nominal case there isn't much difference between regular free() and xfree(), but if ruby is built a certain way, xfree() does some additional internal accounting. In any case, there is no reason not to use xfree(), so it is recommended to always use xfree(). Thanks to SodaBrew for pointing this out in the comments.

A simple example to demonstrate the use of these functions:

 1) struct mystruct {
 2)     int a;
 3)     char *b;
 4) };
 5)
 6) static VALUE implementation(VALUE a) {
 7)     struct mystruct *single;
 8)     struct mystruct *multiple;
 9)
10)     single = ALLOC(struct mystruct);
11)     xfree(single);
12)
13)     multiple = ALLOC_N(struct mystruct, 5);
14)
15)     REALLOC_N(multiple, struct mystruct, 10);
16)
17)     xfree(multiple);
18)
19)     return Qnil;
20) }

Lines 1 through 4 just define a simple structure containing a char * and an int. The implementation of a ruby method on lines 6 through 20 show the use of the allocation functions. Line 10 shows the allocation of a single structure of type struct mystruct, which is freed on line 11. Line 13 shows the allocation of an array of 5 elements of struct mystructs into the multiple pointer. Line 15 shows the reallocation of the multiple array to 10 elements. Notice that since REALLOC_N is a macro, it operates slightly differently than realloc(); in particular, there is no need (and no way) to re-assign the pointer. Finally, line 17 frees up the multiple pointer and line 19 returns successfully from the function.

Error handling and not leaking memory


Ruby is a garbage collected language, meaning that applications don't generally have to worry about freeing memory after it is used. This garbage collection extends into C extension modules, but only to a certain point. If you are writing a C extension to ruby, there are some places that you have to worry about keeping track of your pointers and freeing them up. To understand why, we need to dig a little into the memory allocation functions of ruby.

When you are writing pure ruby, and execute a line of code like:

x = ['a']

the ruby virtual machine causes some memory to come into existence to hold that list for you. The way that this memory is allocated is with rb_ary_new() (or one of its derivatives). The call chain looks like: rb_ary_new() -> rb_ary_new2() -> ary_new() -> NEWOBJ() -> rb_newobj(). Inside of rb_newobj(), no memory is actually allocated; instead, the new object that we need to come into existence is just taken off of the list of free objects, and the free list head is moved to the next object. If it turns out that no memory is available in this freelist, the garbage collector is run to try to reap some memory, and then the memory is given to this new object. Because this memory is coming from the freelist, it is all involved with (and can later be reaped by) the garbage collection.

When you allocate memory in C code using malloc (or one of its derivatives), no such thing happens. The memory is properly allocated, but it is not involved in any of the garbage collection schemes. This leads to 2 problems:
  1. Since malloc isn't involved in the garbage collection, the malloc can fail earlier than it normally would due to address space fragmentation. This isn't generally a problem on 64-bit architectures, but it could crop up as a problem on 32-bit ones.
  2. If a ruby call in your extension module fails, it will throw an exception. In ruby, exceptions are done via a longjmp out of the extension code and into the ruby exception handling code. If you have allocated any memory with malloc and friends, you have now lost the pointers to that memory, so you now have a memory leak (apparently this problem is much worse when dealing with C++; see [1]).

Problem 1) is partially solved by using the built-in ruby ALLOC, ALLOC_N, and ruby_xmalloc functions. Problem 2) is much more insidious, and more difficult to handle. Luckily, it is not impossible to handle.

Assume you have the following code snippet:


 1) int *ids;
 2) VALUE result;
 3) int i;
 4)
 5) ids = ALLOC_N(int, 5);
 6) for (i = 0; i < 5; i++)
 7)     ids[i] = i;
 8)
 9) result = rb_ary_new2();
10)
11) for (i = 0; i < 5; i++)
12)     rb_ary_push(result, INT2NUM(ids[i]));
13)
14) xfree(ids);

(while this is a bit of a contrived example, it actually bears a lot of resemblance to this[2] code in ruby-libvirt)

What this code is trying to do is to create an array full of the values in the "ids" array. If there are no errors, then this code works absolutely fine and doesn't leak any memory (ids gets freed at line 14, and the ruby array will get reaped by the garbage collector eventually). However, if either rb_ary_new2() or rb_ary_push() fails in lines 9 or 12, then they will automatically longjmp to the ruby exception handler, completely skipping the xfree at line 14. This code has now leaked memory.

The way to fix this is to interrupt ruby's normal longjmp on exception mechanism so that you can insert code of your own before throwing the exception. The rb_protect() ruby call can be used to do exactly this. Unfortunately the interface is a bit clunky, but we have to do what we have to do.

rb_protect() takes 3 arguments: a name of a callback function that takes 1 (and exactly 1 argument), the argument to pass to that callback function, and a pointer to an integer to store the exception address (if any). Because the callback function can only take one argument, typical usage is to create a callback "wrapper" that takes the one and only argument. The data that you pass in can be anything, so if you want to pass in multiple arguments, you can do so by passing in a pointer to a structure containing all of the data that you need. An example should help clarify some of this:


 1) struct rb_ary_push_arg {
 2)     VALUE arr;
 3)     VALUE value;
 4) };
 5)
 6) static VALUE rb_ary_push_wrap(VALUE arg) {
 7)     struct rb_ary_push_arg *e = (struct rb_ary_push_arg *)arg;
 8)
 9)     return rb_ary_push(e->arr, e->value);
10) }
11)
12) int *ids;
13) VALUE result;
14) int i;
15) int exception = 0;
16) struct rb_ary_push_arg args;
17)
18) ids = ALLOC_N(int, 5);
19) for (i = 0; i < 5; i++)
20)     ids[i] = i;
21)
22) result = rb_ary_new2();
23)
24) for (i = 0; i < 5; i++) {
25)     args.arr = result;
26)     args.value = INT2NUM(ids[i]);
27)     rb_protect(rb_ary_push_wrap, (VALUE)&args, &exception);
28)     if (exception) {
29)         xfree(ids);
30)         rb_jump_tag(exception);
31)     }
32) }
33)
34) xfree(ids);

Now when we add entries to the ruby array, we are doing so through the rb_ary_push_wrap() function, called by rb_protect(). This means that if rb_ary_push() fails for any reason and throws an exception, control will be returned back to the code above at line 28, but with exception set to a non-zero number. We have a chance to clean up after ourselves, and then continue propagating the exception with rb_jump_tag(). Note that with the use of a proper structure, we can pass any number of arguments through to the wrapper function, so we can use this for all internal ruby functions. Notice that I did not wrap rb_ary_new2(), even though that can cause the same problem; I leave this as an exercise to the reader.

[1] http://www.thoughtsincomputation.com/posts/ruby-c-extensions-c-and-weird-crashing-on-rb_raise
[2] http://libvirt.org/git/?p=ruby-libvirt.git;a=blob;f=ext/libvirt/domain.c;h=eb4426252af635311e14e234a62780fbd4048f0b;hb=HEAD#l80

Monday, January 17, 2011

Aeolusproject.org

One of the projects I work on at Red Hat is cloud management. Our cloud projects rely on Deltacloud, a cross-cloud API, to do the real work on the cloud.

Due to some historical baggage, some of the other cloud projects were also under the Deltacloud name although they didn't directly deal with the API. In order to reduce confusion, we have renamed these projects under a new umbrella project, Aeolus. The website for this is http://www.aeolusproject.org. The main project under the Aeolus umbrella is the Conductor, a web-based UI for managing instances in clouds. There are also several other subprojects that help the Conductor do its job, including:

  • Oz (an automated guest installation system)
  • Image Factory (a project to do installation of guests along with transformation of them to particular cloud backends)
  • Image Warehouse (a project to move data from place to place depending on user-specified rules)
  • Audrey (a system for configuring cloud instances on-the-fly)

If you have interest in what Red Hat is doing in the cloud, please stop by the website and see what we have to offer. As always, we welcome comments, questions, constructive criticism, and contributions; please send such to the Aeolus mailing list.

Writing Ruby Extensions in C - Part 11, Blocks and Callbacks

This is the eleventh in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talked about dealing with numbers. The eighth post talked about strings. The ninth post focused on arrays. The tenth post looked at hashes. This post will talk about blocks and callbacks.

Blocks


Blocks [1] are a great idiom in ruby, equivalent to anonymous functions attached to a line of code. As with many other things in ruby C extensions, they are fairly easy to deal with. There are a few functions to know about:
  • rb_block_given_p() - returns 1 if a block was given to this ruby function, 0 otherwise
  • rb_yield(value) - yield a single value to the given block
  • rb_yield_values() - yield multiple values to the given block

In ruby terms "yield"ing sends a value from a statement into a block. If you want to yield multiple values to a ruby block, you have two options: rb_yield() with an array or hash, and rb_yield_values(). They both work equally well, though rb_yield_values() with multiple values is a bit more idiomatic to ruby. It is also possible for the ruby block to return a result from the block; the return value of the last statement of the block will be returned from the rb_yield() or rb_yield_values() call. However, note that the last line of the block cannot be a return; in that case, the value will be lost forever. Unfortunately this puts a bit of a burden on the consumers of the APIs, but it is coded into the ruby runtime[2]. The following example will demonstrate all of these calls.

First let's look at the ruby code:


 1) obj.rb_yield_example {|single|
 2)     puts "Single element is #{single}"
 3)     "done"
 4) }
 5)
 6) obj.rb_yield_values_example {|first, second, third|
 7)     puts "1st is #{first}, 2nd is #{second}, 3rd is #{third}"
 8)     "done"
 9) }

Now let's look at the C code to implement the above:


 1) static VALUE example_rb_yield(VALUE c) {
 2)     VALUE result;
 3)
 4)     if (!rb_block_given_p())
 5)         rb_raise(rb_eArgError, "Expected block");
 6)
 7)     result = rb_yield(rb_str_new2("hello"));
 8)
 9)     fprintf(stderr, "Return value from block is %s\n",
10)             StringValueCStr(result));
11)
12)     return Qnil;
13) }
14)
15) static VALUE example_rb_yield_values(VALUE c){
16)     VALUE result;
17)
18)     if (!rb_block_given_p())
19)         rb_raise(rb_eArgError, "Expected block");
20)
21)     result = rb_yield_values(3, rb_str_new2("first"),
22)                              rb_str_new2("second"),
23)                              rb_str_new2("third"));
24)
25)     fprintf(stderr, "Return value from block is %s\n",
26)             StringValueCStr(result));
27)
28)     return Qnil;
29) }
30)
31) rb_define_method(c_obj, "rb_yield_example",
32)                  example_rb_yield, 0);
33) rb_define_method(c_obj, "rb_yield_values_example",
34)                  example_rb_yield_values, 0);


Callbacks


Although blocks are idiomatic to ruby and should be used wherever possible, there are situations in which they do not work. For instance, if a ruby method needs to be used as a callback for an asynchronous event, blocks do not work; they are only active for the duration of the method call the block is attached to. If it is necessary to call a particular ruby method from a C library asynchronous callback, there are 2 options:

  1. Procs (lambdas)
  2. Named Methods

Procs are more idiomatic to ruby, but as far as I can tell there isn't a whole lot of advantages to Procs over named methods. I'll go through both of them after setting up the example.

Let's assume that the C library being wrapped requires callbacks for asynchronous events. In this case, the library is expecting a function pointer with a signature looking like:

int (*asynccallback)(int event, void *userdata);

(that is, the function must take an event and a void pointer in, and return an int result). Also assume that we have to register the callback with the library:

void register_async_callback(int (*cb)(int, void *), void *userdata);

How would we go about calling a ruby method that the user writes when the library does the asynchronous callback?

Procs


With Procs, we would have the user of our ruby library create a Proc and pass it to the extension. An example ruby client:

 1)  cb = Proc.new {|event, userdata|
 2)      puts "event is #{event}, userdata is #{userdata}"
 3)  }
 4)
 5)  ruby_extension.register_async_proc(cb, "my user data")

Note that the body of the Proc can be any valid ruby; here we simple print out the arguments that were passed into the Proc.

In the extension, we would define a method called "register_async_proc" that takes 2 arguments: the Proc and the user data that we want passed through to the Proc. The extension C code would look something like:


 1) int internal_callback(int event, void *userdata) {
 2)     VALUE passthrough = (VALUE)userdata;
 3)     VALUE cb;
 4)     VALUE cbdata;
 5)
 6)     cb = rb_ary_entry(passthrough, 0);
 7)     cbdata = rb_ary_entry(passthrough, 1);
 8)
 9)     rb_funcall(cb, rb_intern("call"), 2, INT2NUM(event),
10)                cbdata);
11)
12)     return 0;
13) }
14)
15) VALUE ext_register(VALUE obj, VALUE cb, VALUE userdata) {
16)     VALUE passthrough;
17)
18)     if (rb_class_of(cb) != rb_cProc)
19)         rb_raise(rb_eTypeError, "Expected Proc callback");
20)
21)     passthrough = rb_ary_new();
22)     rb_ary_store(passthrough, 0, cb);
23)     rb_ary_store(passthrough, 1, userdata);
24)
25)     register_async_callback(internal_callback,
26)                             (void *)passthrough);
27) }
28)
29) rb_define_method(c_extension, "register_async_proc",
30)                  ext_register, 2);

The above is not a lot of code, but there is a lot going on, so let's step through it one line at a time starting from the end. Line 29 defines our new method called register_async_proc, that will call the internal extension function ext_register (lines 15 to 27) with 2 arguments. Lines 18 and 19 inside of ext_register check to make sure that what the user actually passed us was a Proc. Lines 21 through 23 set up a new ruby array that contains both the callback that the user gave to us and any additional user data that they want passed into the Proc. Line 25 calls the C library function register_async_callback with our *internal* callback, and the ruby array that we set up in lines 21 through 23. There are a couple of things to note with this. First, we cannot use the ruby Proc as the callback directly; the Proc will have the wrong signature, and the C library doesn't have any idea of how to marshal data so that ruby can understand it. Instead, we have the C library call an internal callback inside the extension; this internal callback will marshal the data for the ruby callback, and then invoke the ruby callback. The second thing to note about line 25 is that we pass the array that we created in lines 21 through 23 to the C library in the "opaque" callback data. It is imperative that the C library function provide a void * pointer for user data, otherwise this technique cannot work.

After line 25, the asynchronous callback is set up. When an event happens in the C library, it will callback to the function given to it by register_async_callback. In our case, this callback is internal_callback, lines 1 through 13. The first thing that internal_callback does on line 2 is to cast the void * back to a VALUE so we can operate on it. In lines 6 and 7, the array that was created and registered earlier is pulled apart into separate pieces. Finally, line 9 calls out to the Proc that was originally registered by the user, passing the event that happened and the original user data to be passed into the Proc.

Named methods


Named method callbacks work very similarly to Proc callbacks, so I won't go into great lengths to describe them. I'll show the (very similar) example code, and explain the differences to the Proc callback method.

First the ruby client code:

 1)  def cb(event, userdata)
 2)      puts "event is #{event}, userdata is #{userdata}"
 3)  end
 4)
 5)  ruby_extension.register_async_symbol(:cb, "my user data")

There are two important differences to the Proc code; the fact that the callback is a real method (defined with def), and how we pass it into the extension call. We cannot just use "cb", because otherwise ruby attempts to execute the function cb before calling register_async_symbol. Instead we have to pass the Symbol that represents the callback method.

Now we look at the extension code:

 1) int internal_callback(int event, void *userdata) {
 2)     VALUE passthrough = (VALUE)userdata;
 3)     VALUE cb;
 4)     VALUE cbdata;
 5)
 6)     cb = rb_ary_entry(passthrough, 0);
 7)     cbdata = rb_ary_entry(passthrough, 1);
 8)
 9)     rb_funcall(rb_class_of(cb), rb_to_id(cb), 2, INT2NUM(event),
10)                cbdata);
11)
12)     return 0;
13) }
14)
15) VALUE ext_register(VALUE obj, VALUE cb, VALUE userdata) {
16)     VALUE passthrough;
17)
18)     if (rb_class_of(cb) != rb_cSymbol)
19)         rb_raise(rb_eTypeError, "Expected Symbol callback");
20)
21)     passthrough = rb_ary_new();
22)     rb_ary_store(passthrough, 0, cb);
23)     rb_ary_store(passthrough, 1, userdata);
24)
25)     register_async_callback(internal_callback,
26)                             (void *)passthrough);
27) }
28)
29) rb_define_method(c_extension, "register_async_symbol",
30)                  ext_register, 2);

The differences are minor. Line 29 defines this as "register_async_symbol" instead of "register_async_proc". Line 18 checks to make sure that this is of type rb_cSymbol instead of rb_cProc. Line 9 is where the biggest difference is. Instead of using the "call" method to invoke the Proc, we instead use the class and the ID of the method that the user originally gave to us.

[1] http://ruby-doc.org/docs/ProgrammingRuby/html/tut_containers.html
[2] http://stackoverflow.com/questions/1435743/why-does-explicit-return-make-a-difference-in-a-proc

Sunday, January 16, 2011

Writing Ruby Extensions in C - Part 10, Hashes

This is the tenth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talked about dealing with numbers. The eighth post talked about strings. The ninth post focused on arrays. This post will look at hashes.

Hashes


The nice thing about hashes in ruby C extensions is that they act very much like the ruby hashes they represent. There are a few functions to know about:
  • rb_hash_new() - create a new ruby Hash
  • rb_hash_aset(hash, key, value) - set the hash key to value
  • rb_hash_aref(hash, key) - get the value for hash key
  • rb_hash_foreach(hash, callback, args) - call callback for each key,value pair in the hash. Callback must have a prototype of int (*cb)(VALUE key, VALUE val, VALUE in)

An example will demonstrate this:

 1) int do_print(VALUE key, VALUE val, VALUE in) {
 2)      fprintf(stderr, "Input data is %s\n", StringValueCStr(in));
 3)
 4)      fprintf(stderr, "Key %s=>Value %s\n", StringValueCStr(key),
 5)              StringValueCStr(val));
 6)
 7)      return ST_CONTINUE;
 8) }
 9)
10) VALUE result;
11) VALUE val;
12)
13) result = rb_hash_new();
14) // result is now {}
15) rb_hash_aset(result, rb_str_new2("mykey"),
16)              rb_str_new2("myvalue"));
17) // result is now {"mykey"=>"myvalue"}
18) rb_hash_aset(result, rb_str_new2("anotherkey"),
19)              rb_str_new2("anotherval"));
20) // result is now {"mykey"=>"myvalue",
21) //                "anotherkey"=>"anotherval"}
22) rb_hash_aset(result, rb_str_new2("mykey"),
23)              rb_str_new2("differentval"));
24) // result is now {"mykey"=>"differentval",
25) //                "anotherkey"=>"anotherval"}
26) val = rb_hash_aref(result, rb_str_new2("mykey"));
27) // result is now {"mykey"=>"differentval",
28) //                "anotherkey"=>"anotherval"},
29) // val is "differentval"
30) rb_hash_delete(result, rb_str_new2("mykey"));
31) // result is now {"anotherkey"=>"anotherval"}
32)
33) rb_hash_foreach(result, do_print, rb_str_new2("passthrough"));

Most of this is pretty straightforward. The most interesting part of this is line 33, where we perform an operation on all elements in the hash by utilizing a callback. This callback is defined on lines 1 through 8, and takes in the key, value, and the user data provided to the original rb_hash_foreach() call. The return code from the callback defines what happens to the processing of the rest of the hash. If the return value is ST_CONTINUE, then the rest of the hash is processed as normal. If the return value is ST_STOP, then no further processing of the hash is done. If the return value is ST_DELETE, then the current hash key is deleted from the hash and the rest of the hash is processed. If the return value is ST_CHECK, then the hash is checked to see if it has been modified during this operation. If so, processing of the hash stops.

Update: Fixed up the example code to show on the screen.

Saturday, January 15, 2011

Writing Ruby Extensions in C - Part 9, Arrays

This is the ninth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talked about dealing with numbers. The eighth post talked about strings. This post will focus on arrays.

Arrays


The nice thing about arrays in ruby C extensions is that they act very much like the ruby arrays they represent. There are a few functions to know about:
  • rb_ary_new() - create a new array with 0 elements. Elements can be added later using rb_ary_push(), rb_ary_store(), or rb_ary_unshift().
  • rb_ary_new2(size) - create a new array with size elements
  • rb_ary_store(array, index, value) - put the ruby value into array at index. This can be used to create sparse arrays; intervening elements that have not yet had values assigned will be set to nil
  • rb_ary_push(array, value) - put value at the end of the array
  • rb_ary_unshift(array, value) - put value at the start of the array
  • rb_ary_pop(array) - pop the last element of array off and return it
  • rb_ary_shift(array) - remove the first element of array and return it
  • rb_ary_entry(array, index) - examine array element located at index without changing array
  • rb_ary_dup(array) - copy array and return the copy
  • rb_ary_to_s(array) - invoke the "to_s" method on the array. Note that this concatenates the array elements together without spacing, so is not generally useful
  • rb_ary_join(array, string_object) - create a string by converting each element of the array to a string separated by string_object. If string_object is Qnil, then no separator is used
  • rb_ary_reverse(array) - reverse the order of all of the elements in array
  • rb_ary_to_ary(ruby_object) - create an array out of any ruby object. If the object is already an array, a reference to the same object is returned. If the object supports the "to_ary" method, then "to_ary" is invoked on the object and the result is returned. If neither of the previous are true, then a new array with 1 element containing the object is returned

An example should make most of this clear:

 1) VALUE result, elem, arr2, mystr;
 2)
 3) result = rb_ary_new();
 4) // result is now []
 5) rb_ary_push(result, INT2FIX(1));
 6) // result is now [1]
 7) rb_ary_push(result, INT2FIX(2));
 8) // result is now [1, 2]
 9) rb_ary_unshift(result, INT2FIX(0));
10) // result is now [0, 1, 2]
11) rb_ary_store(result, 3, INT2FIX(3));
12) // result is now [0, 1, 2, 3]
13) rb_ary_store(result, 5, INT2FIX(5));
14) // result is now [0, 1, 2, 3, nil, 5]
15) elem = rb_ary_pop(result);
16) // result is now [0, 1, 2, 3, nil] and elem is 5
17) elem = rb_ary_shift(result);
18) // result is now [1, 2, 3, nil] and elem is 0
19) elem = rb_ary_entry(result, 0);
20) // result is now [1, 2, 3, nil] and elem is 1
21) arr2 = rb_ary_dup(result);
22) // result is now [1, 2, 3, nil] and arr2 is [1, 2, 3, nil]
23) mystr = rb_ary_to_s(result);
24) // result is now [1, 2, 3, nil] and mystr is 123
25) mystr = rb_ary_join(result, rb_str_new2("-"));
26) // result is now [1, 2, 3, nil] and mystr is 1-2-3-
27) rb_ary_reverse(result);
28) // result is now [nil, 3, 2, 1]
29) rb_ary_shift(result);
30) // result is now [3, 2, 1]
31) result = rb_ary_to_ary(rb_str_new2("hello"));
32) // result is now ["hello"]

Friday, January 14, 2011

Writing Ruby Extensions in C - Part 8, Strings

This is the eighth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. The seventh post talk about dealing with numbers. This post will talk about strings.

Dealing with Strings


It is fairly easy to convert C-style strings to ruby string objects, and vice-versa. There are a few functions to know about:
  • rb_str_new(c_str, length) - take the char * c_str pointer and a length in, and return a ruby string object. Note that c_str does *not* have to be NULL terminated; this is one way to deal with binary data
  • rb_str_new2(c_str) - take the NULL terminated char * c_str pointer in, and return a ruby string object
  • rb_str_dup(ruby_string_object) - take ruby_string_object in and return a copy
  • rb_str_plus(string_object_1, string_object_2) - concatenate string_object_1 and string_object_2 and return the result without modifying either object
  • rb_str_times(string_object_1, fixnum_object) - concatenate string_object_1 with itself fixnum_object number of times and return the result
  • rb_str_substr(string_object, begin, length) - return the substring of string_object starting at position begin and going for length characters. If length is less than 0, then "nil" is returned. If begin is passed the end of the array or before the beginning of the array, then "nil" is returned. Otherwise, this function returns the substring of string_object that matches begin..length, though it may be cut short if there are not enough characters in the array
  • rb_str_cat(string_object, c_str, length) - take the char * c_str pointer and length in, and concatenate onto the end of string_object
  • rb_str_cat2(string_object, c_str) - take the NULL-terminated char *c_str pointer in, and concatenate onto the end of string_object
  • rb_str_append(string_object_1, string_object_2) - concatenate string_object_2 onto string_object_1
  • rb_str_concat(string_object, ruby_object) - concatenate ruby_object onto string_object_1. If ruby_object is a FIXNUM between 0 and 255, then it is first converted to a character before concatenation. Otherwise it behaves exactly the same as rb_str_append
  • StringValueCStr(ruby_object) - take ruby_object in, attempt to convert it to a String, and return the NULL terminated C-style char *
  • StringValue(ruby_object) - take ruby_object in and attempt to convert it to a String. Assuming this is successful, the C char * pointer for the string is available via the macro RSTRING_PTR(return_value) and the length of the string is available via the macro RSTRING_LEN(return_value). This is useful to retrieve binary data out of a String object

An example should make most of this clear:

 1) VALUE result, str2, substr;
 2)
 3) result = rb_str_new2("hello");
 4) // result is now "hello"
 5) str2 = rb_str_dup(result);
 6) // result is now "hello", str2 is now "hello"
 7) result = rb_str_plus(result, rb_str_new2(" there"));
 8) // result is now "hello there"
 9) result = rb_str_times(result, INT2FIX(2));
10) // result is now "hello therehello there"
11) substr = rb_str_substr(result, 0, 2);
12) // result is now "hello therehello there", substr is "he"
13) substr = rb_str_substr(result, -2, 2);
14) // result is now "hello therehello there", substr is "re"
15) substr = rb_str_substr(result, -2, 5);
16) // result is now "hello therehello there", substr is "re"
17) // (substring was cut short because the length goes past the end of the string)
18) substr = rb_str_substr(result, 0, -1);
19) // result is now "hello therehello there", substr is Qnil
20) // (length is negative)
21) substr = rb_str_substr(result, 23, 1);
22) // result is now "hello therehello there", substr is Qnil
23) // (requested start point after end of string)
24) substr = rb_str_substr(result, -23, 1);
25) // result is now "hello therehello there", substr is Qnil
26) // (requested start point before beginning of string)
27) rb_str_cat(result, "wow", 3);
28) // result is now "hello therehello therewow"
29) rb_str_cat2(result, "bob");
30) // result is now "hello therehello therewowbob"
31) rb_str_append(result, rb_str_new2("again"));
32) // result is now "hello therehello therewowbobagain"
33) rb_str_concat(result, INT2FIX(33));
34) // result is now "hello therehello therewowbobagain!"
35) fprintf(stderr, "Result is %s\n", StringValueCStr(result));
36) // "hello therehello there wowbobagain!" is printed to stderr

Update: modified the code to fit in the pre box.

Thursday, January 13, 2011

Writing Ruby Extensions in C - Part 7, Numbers

This is the seventh in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. The sixth post talked about ruby catch and throw blocks. This post will talk about numbers.

Dealing with numbers


Numbers are pretty easy to deal with in a ruby C extension. There are two possible types of Ruby numbers; FIXNUMs and Bignums. FIXNUMs are very fast since they just use the native long type of the architecture. However, due to some implementation details, the range of a FIXNUM is limited to one-half of the native long type. If larger (or smaller) numbers need to be manipulated, Bignums are full-blown ruby objects that can represent any number of any size, at a performance cost. The ruby C extension API has support for converting native integer types to ruby FIXNUMs and Bignums and vice-versa. Some of the functions are:
  • INT2FIX(int) - take an int and convert it to a FIXNUM object (but see INT2NUM below)
  • LONG2FIX(long) - synonym for INT2FIX
  • CHR2FIX(char) - take an ASCII character (0x00-0xff) and convert it to a FIXNUM object
  • INT2NUM(int) - take an int and convert it to a FIXNUM object if it will fit; otherwise, convert to a Bignum object. Since this does the right thing in all circumstances, this should always be used in place of INT2FIX
  • LONG2NUM(long) - synonym for INT2NUM
  • UINT2NUM(unsigned int) - take an unsigned int and convert it to a FIXNUM object if it will fit; otherwise, convert to a Bignum object
  • ULONG2NUM(unsigned long int) - synonym for UINT2NUM
  • LL2NUM(long long) - take a long long int and convert it to a FIXNUM object if it will fit; otherwise, convert to a Bignum object
  • ULL2NUM(unsigned long long) - take an unsigned long long int and convert it to a FIXNUM object if it will fit; otherwise, convert to a Bignum object
  • OFFT2NUM(off_t) - take an off_t and convert it to a FIXNUM object if it will fit; otherwise, convert to a Bignum object
  • FIX2LONG(fixnum_object) - take a FIXNUM object and return the long representation (but see NUM2LONG below)
  • FIX2ULONG(fixnum_object) - take a FIXNUM object and return the unsigned long representation (but see NUM2ULONG below)
  • FIX2INT(fixnum_object) - take a FIXNUM object and return the int representation (but see NUM2INT below)
  • FIX2UINT(fixnum_object) - take a FIXNUM object and return the unsigned int representation (but see NUM2UINT below)
  • NUM2LONG(numeric_object) - take a FIXNUM or Bignum object in and return the long representation. Since this does the right thing in all circumstances, this should be used in favor of FIX2LONG
  • NUM2ULONG(numeric_object) - take a FIXNUM or Bignum object in and return the unsigned long representation. Since this does the right thing in all circumstances, this should be used in favor of FIX2ULONG
  • NUM2INT(numeric_object) - take a FIXNUM or Bignum object in and return the int representation. Since this does the right thing in all circumstances, this should be used in favor of FIX2INT
  • NUM2UINT(numeric_object) - take a FIXNUM or Bignum object in and return the unsigned int representation. Since this does the right thing in all circumstances, this should be used in favor of FIX2UINT
  • NUM2LL(numeric_object) - take a FIXNUM or Bignum object in and return the long long representation
  • NUM2ULL(numeric_object) - take a FIXNUM or Bignum object in and return the unsigned long long representation
  • NUM2OFFT(numeric_object) - take a FIXNUM or Bignum object in and return the off_t representation
  • NUM2DBL(numeric_object) - take a FIXNUM or Bignum object in and return the double representation
  • NUM2CHR(ruby_object) - take ruby_object in and return the char representation of the object. If ruby_object is a string, then the char of the first character in the string is returned. Otherwise, NUM2INT is run on the object and the result is returned
For this particular topic I'll omit the example. There aren't really a lot of interesting things to show or odd corner cases that you need to deal with when working with numbers.

Wednesday, January 12, 2011

Writing Ruby Extensions in C - Part 6, Catch/Throw

This is the sixth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. The fifth post focused on creating and handling exceptions. This post will talk about ruby catch and throw blocks.

Catch/Throw

In ruby, raising exceptions is used to transfer control out of a block of code when something goes wrong. Ruby has a second mechanism for transferring control to blocks called catch/throw. Any ruby block can be labelled via catch(), and then any line of code within that block can throw() to terminate the rest of the block. This also works with nested catch/throw blocks so an inner nested throw could throw all the way back out to the outer block. Essentially, they are a fancy goto mechanim; see [1] for some examples. How can we catch and throw from within our C extension module? Like exceptions, we accomplish this through callbacks.

To set up a catch in a C extension, the rb_catch() function is used. rb_catch() takes 3 parameters: the first parameter is the name of the catch block, the second parameter is the name of the callback to invoke in block context, and the third parameter is data to be passed to the callback. As may be expected, the callback function must take a single VALUE parameter in and return a VALUE.

To return to a catch point in a C extension, the rb_throw() function is used. rb_throw() takes two parameters: the name of the catch block to return to, and the return value (which can be any valid ruby object, including Qnil). If rb_throw() is executed, control is returned from the point of the rb_throw() to the end of the rb_catch() block, and execution continues from there.

An example can demonstrate much of this. First let's look at the C code to implement an example catch/throw:

 1) static VALUE m_example;
 2)
 3) static VALUE catch_cb(VALUE val, VALUE args, VALUE self) {
 4)     rb_yield(args);
 5)     return Qnil;
 6) }
 7)
 8) static VALUE example_method(VALUE klass) {
 9)     VALUE res;
10)
11)     if (!rb_block_given_p())
12)         rb_raise(rb_eStandardError, "Expected a block");
13)
14)     res = rb_catch("catchpoint", catch_cb, rb_str_new2("val"));
15)     if (TYPE(res) != T_FIXNUM)
16)         rb_throw("catchpoint", Qnil);
17)
18)     return res;
19) }
20)
21) void Init_example() {
22)     m_example = rb_define_module("Example");
23)
24)     rb_define_module_function(m_example, "method",
25)                               example_method, 0);
26) }
Lines 21 through 26 set up the extension module, as described elsewhere.

Lines 8 through 19 implement the module function "method". Line 11 checks if a block is given; if not, an exception is raised on line 12. Line 14 sets up an rb_catch() named "catchpoint". The callback catch_cb() will be executed, and a new string of "val" will be passed into the callback. Lines 3 through 6 implement the callback; the value is yielded to the block initially passed into "method", and a nil is returned (which is ignored). Line 15 checks the return value from the block; if it is not a number, then line 16 does an rb_throw() to abort the entire block (with control passing to the line of ruby code after the Example::method call). If the value from the block is a number, then it is returned at line 18. Note that this particular sequence of calls is contrived, since the value returned from the block is just returned to the caller. Still, I think it is a good example of what can be done with rb_catch() and rb_throw().

Now let's look at some example ruby code that might utilize the above code:

require 'example'

# if the method were to be called like this, an exception would be
# raised since no block is given
# retval = Example::method

# if the method were to be called like this, an exception would be
# raised since the return value from the block is not a number
# retval = Example::method {|input|
#     "hello"
# }

# this works properly, since the return value is a number
retval = Example::method {|input|
    puts "Input is #{input}"
    6
}

[1] http://ruby-doc.org/docs/ProgrammingRuby/html/tut_exceptions.html

Tuesday, January 11, 2011

Writing Ruby Extensions in C - Part 5, Exceptions

This is the fifth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. The fourth post talked about types and return values. This post will focus on creating and handling exceptions.

Exceptions

When a method implementation in a ruby C extension encounters an error, the typical response is to throw an exception (a value indicating error can also be returned, but that is not idiomatic). The exception to be thrown can either be one of the built-in exception classes, or a custom defined exception class. The built-in exception classes are:
  • rb_eException
  • rb_eStandardError
  • rb_eSystemExit
  • rb_eInterrupt
  • rb_eSignal
  • rb_eFatal
  • rb_eArgError
  • rb_eEOFError
  • rb_eIndexError
  • rb_eStopIteration
  • rb_eRangeError
  • rb_eIOError
  • rb_eRuntimeError
  • rb_eSecurityError
  • rb_eSystemCallError
  • rb_eThreadError
  • rb_eTypeError
  • rb_eZeroDivError
  • rb_eNotImpError
  • rb_eNoMemError
  • rb_eNoMethodError
  • rb_eFloatDomainError
  • rb_eLocalJumpError
  • rb_eSysStackError
  • rb_eRegexpError
  • rb_eScriptError
  • rb_eNameError
  • rb_eSyntaxError
  • rb_eLoadError

Extension modules should usually define a custom exception class for errors related directly to the extension, and use one of the built-in exception classes for standard errors. The custom exception class should generally be a subclass of rb_eException or rb_eStandardError, though if the module has special needs any of the built-in exception classes can be used. Example:

 1) static VALUE m_example;
 2) static VALUE e_ExampleError;
 3)
 4) static VALUE exception_impl(VALUE klass, VALUE input) {
 5)     if (TYPE(input) != T_FIXNUM)
 6)         rb_raise(rb_eTypeError, "invalid type for input");
 7)
 8)     if (NUM2INT(input) == -1)
 9)         rb_raise(e_ExampleError, "input was < 0");
10)         return Qnil;
11) }
12)
13) void Init_example() {
14)     m_example = rb_define_module("Example");
15)
16)     e_ExampleError = rb_define_class_under(m_example, "Error",
17)                                            rb_eStandardError);
18)
19)     rb_define_module_function(m_example, "exception_example",
20)                               exception_impl, 1);
21) }
Line 14 sets up the extension module. Line 16 creates the custom exception class as a subclass of rb_eStandardError. Now if the extension module runs into a situation that it can't accept, it can raise e_ExampleError and throw an exception of type Example::Error. Line 19 defines a module function that demonstrates the use of standard and custom exceptions. If Example::exception_example is called with an argument that is not a number, it raises the ArgumentError exception on line 6 (side-note: Check_Type should really be used to do this type of checking, but for example purposes we omit that). If Example::exception_example is called with a number argument that is -1, then the custom exception Example::Error is raised on line 9. Otherwise, the method succeeds and Qnil is returned.

Raising exceptions

There are a few different ways to raise exceptions:
  • rb_raise(error_class, error_string, ...) - the main interface for raising exceptions. A new exception object of class type error_class is created and then raised, with the error message set to error_string (plus any printf-style arguments)
  • rb_fatal(error_string, ...) - a function for raising an exception of type rb_eFatal with the error message set to error_string (plus any printf-style arguments). After this call the entire ruby interpreter will exit, so extension modules typically should not use it
  • rb_bug(error_string, ...) - prints out the error string (plus any printf-style arguments) and then calls abort(). Since this call doesn't allocate an error object or do any of the other typical exception handling steps, it isn't technically a function to raise exceptions. This function should only be used when a bug in the interpreter is found, and as such, should not be used by extension modules
  • rb_sys_fail(error_string) - raises an exception based on errno. Ruby defines a separate class for each of the errno values (such as Errno::EAGAIN, Errno::EACCESS, etc), and this function will raise an exception of the type that corresponds to the current errno
  • rb_notimplement() - raises an exception of rb_eNotImpError. This is used when a particular function is implemented on one platform, but possibly not on other platforms that ruby supports
  • rb_exc_new2(error_class, error_string) - allocate a new exception object of type error_class, and set the error message to error_string. Note that rb_exc_new2() does not accept printf-style options, so the string will have to be fully-formed before passing it to rb_exc_new2()
  • rb_exc_raise(error_object) - a low-level interface to raise exceptions that have been allocated by rb_exc_new2()
  • rb_exc_fatal(error_object) - a low-level interface to raise a fatal exception that has been allocated by rb_exc_new2(). After this call the entire ruby interpreter will exit, so extension modules typically should not use it
The example below shows the use of rb_raise() and rb_exc_raise(), which are the only two calls that extension modules should really use.

 1) static VALUE m_example;
 2) static VALUE e_ExampleError;
 3)
 4) static VALUE example_method(VALUE klass, VALUE input) {
 5)     VALUE exception;
 6)
 7)     if (TYPE(input) != T_FIXNUM)
 8)         rb_raise(rb_eTypeError, "invalid type for input");
 9)
10)     if (NUM2INT(input) < 0) {
11)         exception=rb_exc_new2(e_ExampleError, "input was < 0");
12)         rb_iv_set(exception, "@additional_info",
13)                   rb_str_new2("additional information"));
14)         rb_exc_raise(exception);
15)     }
16)
17)     return Qnil;
18) }
19)
20) void Init_example() {
21)     m_example = rb_define_module("Example");
22)
23)     e_ExampleError = rb_define_class_under(m_example, "Error",
24)                                            rb_eStandardError);
25)     rb_define_attr(e_ExampleError, "additional_info", 1, 0);
26)
27)     rb_define_module_function(m_example, "method",
28)                               example_method, 1);
29) }
Lines 20 through 29 show the module initialization. Since this is described in more detail elsewhere, I'll only point out line 25, where a custom attribute for the error class e_ExampleError is defined. When an error occurs in the extension module, additional error information can be placed into that attribute, and any caller can look inside of the error object to retrieve that additional information.

Lines 4 through 18 implement an example method that takes one and only one input parameter. Line 7 checks to see if the input value is a number, and if not an exception is raised with rb_raise() on line 8. Line 10 checks to see if the number is less than 0. If it is, then a new exception object of type e_ExampleError is allocated on line 11 with rb_exc_new2(), and the additional_info attribute of the object is set to "additional information" on line 12. As with most other things, the value that additional_info is set to can be any valid ruby object. Line 14 then raises the exception. This example shows very clearly the power of rb_exc_new2() and rb_exc_raise(), in that additional error information can be passed through to callers.

Handling exceptions

The other half of dealing with exceptions in an extension module is handling exceptions in C code when they are thrown from ruby functions. How is that done since C has no raise/rescue type mechanism? Through callbacks.

There are a few functions that can be used for handling exceptions:
  • rb_ensure(cb, cb_args, ensure, ensure_args) - Call function cb with cb_args. The callback must take in a single VALUE parameter and return VALUE. When cb() finishes, regardless of whether it completes successfully or raises an exception, call ensure with ensure_args. The ensure function must take in a single VALUE parameter and return VALUE
  • rb_protect(cb, cb_args, line_pointer) - Call cb with cb_args. The callback must take in a single VALUE parameter and return VALUE. If an exception is raised by cb(), store the exception handler point in line_pointer and return control. It is then the responsibility of the caller to call rb_jump_tag() to return to the exception point
  • rb_jump_tag(line) - do a longjmp to the line saved by rb_protect(). No code after this statement will be executed
  • rb_rescue(cb, cb_args, rescue, rescue_args) - Call function cb with cb_args. The callback must take in a single VALUE parameter and return VALUE. If cb() raises any exception, rescue is called with rescue_args. The rescue callback should take in two VALUE parameters and return VALUE

Another example should make some of this clear:

 1) static VALUE cb(VALUE args) {
 2)     if (TYPE(args) != T_FIXNUM)
 3)         rb_raise(rb_eTypeError, "expected a number");
 4)     return Qnil;
 5) }
 6)
 7) static VALUE ensure(VALUE args) {
 8)     fprintf(stderr, "Ensure value is %s\n",
 9)               StringValueCStr(args));
10)     return Qnil;
11) }
12)
13) static VALUE rescue(VALUE args, VALUE exception_object) {
14)     fprintf(stderr, "Rescue args %s, object classname %s\n",
15)             StringValueCStr(args),
16)             rb_obj_classname(exception_object));
17)     return Qnil;
18) }
19)
20) VALUE res;
21) int exception;
22)
23) res = rb_ensure(cb, INT2NUM(0), ensure, rb_str_new2("data"));
24) res = rb_ensure(cb, rb_str_new2("bad"), ensure,
25)                 rb_str_new2("data"));
26)
27) res = rb_protect(cb, INT2NUM(0), &exception);
28) res = rb_protect(cb, rb_str_new2("bad"), &exception);
29) if (exception) {
30)     fprintf(stderr, "Failed cb\n");
31)     rb_jump_tag(exception);
32) }
33)
34) res = rb_rescue(cb, INT2NUM(0), rescue, rb_str_new2("data"));
35) res = rb_rescue(cb, rb_str_new2("bad"), rescue,
36                    rb_str_new2("data"));
Line 23 kicks off the action with a call to rb_ensure(). In this first rb_ensure, we pass a FIXNUM object to cb(), which means that no exception is raised. Because of the rb_ensure(), however, the ensure() callback on lines 7 through 11 is called anyway and does some printing.

Line 24 passes a String object to cb(), which causes cb() to raise an exception. Because of the rb_ensure, the ensure() callback on lines 7 through 11 is called and does some printing. Importantly, after ensure() is called the exception is propagated, so in reality none of the code after line 21 will be executed (we'll ignore this fact for the sake of this example).

Line 27 uses rb_protect() to call the callback; since a FIXNUM object is passed, no exception is raised. Note that if the call that is being wrapped by rb_protect() does not raise an exception, exception is always initialized to 0.

Line 28 uses rb_protect() to call cb() with a String object, which causes an exception to be raised. Because rb_protect() is being used, control will be returned to the calling code at line 29, and that code can then check for the exception. Since an exception was raised, the "exception" integer will have a non-0 number and the code can do whatever we need to clean up and then propagate the exception further with rb_jump_tag() on line 31.

Line 34 uses the rb_rescue() wrapper to call cb(). Since a FIXNUM object is passed to cb(), no exception is raised and no callbacks other than cb() are called.

Line 35 uses rb_rescue() to call cb() with a String object, which causes an exception to be raised and the rescue() callback to be executed. The rescue() callback on lines 13 through 18 takes two arguments: the VALUE initially passed into the rb_rescue() rescue_args, and the exception_object that caused the exception. Based on the exception_object, the rescue() callback can choose to handle this exception or not.

Example

Before finishing this post, I'll leave you with another example. When writing ruby code, the full begin..rescue block goes something like:

begin
  ...
rescue FooException => e
  ...
rescue
  ...
else
  ...
ensure
  ...
How would we implement this in C?

 1) static VALUE foo_exception_rescue(VALUE args) {
 2)     fprintf(stderr, "foo_exception_rescue value is %s\n",
 3)             StringValueCStr(args));
 4)     return Qnil;
 5) }
 6)
 7) static VALUE other_exception_rescue(VALUE args) {
 8)     fprintf(stderr, "other_exception_rescue value is %s\n",
 9)             StringValueCStr(args));
10)     return Qnil;
11) }
12)
13) static VALUE rescue(VALUE args, VALUE exception_object) {
14)     if (strcmp(rb_obj_classname(exception_object),
15)                "FooException") == 0)
16)         return foo_exception_rescue(args);
17)     else
18)         return other_exception_rescue(args);
19) }
20)
21) static VALUE cb(VALUE args) {
22)     return rb_rescue(cb, args, rescue, rb_str_new2("data"));
23) }
24)
25) static VALUE ensure(VALUE args) {
26)     fprintf(stderr, "Ensure args %s\n", StringValueCStr(args));
27)     return Qnil;
28) }
29)
30) VALUE res;
31)
32) res = rb_ensure(cb, INT2NUM(0), ensure, rb_str_new2("data"));
This example implements almost the entire ability of the ruby begin..rescue blocks. What it does not implement is the "else" clause; I have not yet come up with a good way to do that. If you think of something to make this example work for the "else" clause, please leave a comment.

Monday, January 10, 2011

Writing Ruby Extensions in C - Part 4, Types and Return Values

This is the fourth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. This short post will focus on some details of method implementation, including how to check the types that are being passed to extension methods and the legal return values from the extension methods.

Ruby types


When implementing a ruby method in C, the method may expect certain arguments to be of a certain type. For instance, it is possible that the ruby method expects a number, and only a number, as the input parameter. The ruby C extension API provides several functions to check if an input parameter is a certain type:
  • TYPE(ruby_object) - return the builtin type of ruby_object. The builtin types distinguish between things like TrueClass, FalseClass, FIXNUM, etc. It explicitly does not distinguish between complicated object types; use either CLASS_OF() or rb_obj_classname() for that. The builtin types that may be returned are:
    • T_NONE
    • T_NIL
    • T_OBJECT
    • T_CLASS
    • T_ICLASS
    • T_MODULE
    • T_FLOAT
    • T_STRING
    • T_REGEXP
    • T_ARRAY
    • T_FIXNUM
    • T_HASH
    • T_STRUCT
    • T_BIGNUM
    • T_FILE
    • T_TRUE
    • T_FALSE
    • T_DATA
    • T_MATCH
    • T_SYMBOL
    • T_BLKTAG
    • T_UNDEF
    • T_VARMAP
    • T_SCOPE
    • T_NODE
  • NIL_P(ruby_object) - test if ruby_object is the nil object
  • CheckType(ruby_object, builtin_type) - check to make sure that ruby_object is of type builtin_type (one of the T_* types listed above). If it is not, an exception is raised
  • CLASS_OF(ruby_object) - return the ruby class VALUE that corresponds to ruby_object. Note that this can distinguish between built-in class types (such as rb_cSymbol) as well as more complicated class types (such as those defined by the API user)
  • rb_obj_classname(ruby_object) - return the char * string representation of the class corresponding to ruby_object

Return values


Every ruby method implemented in C has to return a VALUE. This VALUE can either be a ruby object (such as that returned by INT2NUM), or one of the special values:
  • Qnil - ruby "nil"
  • Qtrue - ruby "true"
  • Qfalse - ruby "false"

Methods that are expected to either succeed or raise an exception typically return Qnil to indicate success.

Thursday, January 6, 2011

Writing Ruby Extensions in C - Part 3, Extension Initialization

This is the third in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The posts from here on out will focus on the C code. This post talks about initializing the module and setting up classes.

Initializing the module


There is a bit of magic involved with initially loading the extension module into ruby. Assuming the extension module is called "example", then the C code that implements the extension must have an initialization function that looks like:

 1) static VALUE m_example;
 2)
 3) void Init_example() {
 4)     m_example = rb_define_module("Example");
 5)     example_library_initialize();
 6) }

Line 1 sets up the variable that holds the reference to the module. Line 3 is a function that must be called "Init_<extension_name>", take no parameters, and return nothing. When the ruby interpreter encounters a line of code such as "require 'example'", it will call this initialization function to set things up. Line 4 actually defines the module for us and calls it "Example". Finally, line 5 does whatever initialization is necessary for the library that is being wrapped. In this case, it just calls the example_library_initialize() function.

Defining classes, constants, and methods


Once the module itself has been initialized, functions, classes, methods, and attributes can be added to it. These are pretty easy to use:
  • rb_define_module_function(module, "function_name", implementation, number_of_args) - define function_name for module. Assuming the module is called "Example", functions like this can be invoked from ruby code like:

    out = Example::function_name

    The implementation should be a C function that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.
  • rb_define_class_under(module, "class_name", super_class) - define a new class named "class_name" under the module. super_class can be one of the pre-defined types (rb_cObject, rb_cArray, etc) or a class that has been defined in this module.
  • rb_define_method(class, "method_name", implementation, number_of_args) - define a new method for class. The implementation should be a C function that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.
  • rb_define_const(class, "CONST", value) - define a new constant for class with value. Assuming the module is called "Example" and the class is called "Class", these can be accessed in ruby code like:

    puts Example::Class::CONST

    The value can be any legal ruby type.
  • rb_define_attr(class, "attr_name", read, write) - define a new attribute for class called attr_name. The read and write parameters should each be 0 or 1, depending on whether you want a read implementation and/or a write_implementation for this attribute, respectively.
  • rb_define_singleton_method(class, "method_name", implementation, number_of_args) - define a new singleton method for class. The implementation should be a C functions that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.

Implementing methods


Using the above methods, it is pretty straightforward to define module functions, class methods, and singleton methods. There is a bit of work necessary to understand the C implementation of these methods. The first thing to realize is that the "number_of_args" as the last parameter of the rb_define_* call defines how many parameters the method will take. For no parameters, you would pass 0, for one parameter you would pass 1, etc. When you go to implement the method in C, your C function must take the number of parameters, plus one for the class (this will be shown in the example below).

You can also pass -1, which tells ruby that you want to take optional arguments. When you go to implement the method in C, the C function must take exactly 3 arguments: int argc, VALUE *argv, VALUE klass. The argc parameter defines how many arguments were passed, the argv parameter is all of the arguments in an array, and the last parameter is the klass itself. To properly parse the arguments, the rb_scan_args(argc, argv, "format", ...) should be called. A brief explanation of rb_scan_args is below; for more information, see the document at [1].

The first two arguments to rb_scan_args() are the argc and argv passed into the function. The third argument is a string that defines how many required and how many optional parameters the method requires. The last parameters are pointers to VALUEs to place the value of the arguments in. For instance, to have 1 required and 2 optional parameters to the method, format should be "12" and 3 additional VALUE parameters should be passed to rb_scan_args(). To have no required and 1 optional parameters to the method, format should be "01" and 1 additional VALUE parameter should be passed to rb_scan_args(). Note that if less than the number of required parameters is passed to the method, an ArgumentError exception will be raised. All optional arguments are set to the value that was passed, if any, or "nil".

Let's take a look at an example to show all of this off:

 1) static VALUE m_example;
 2) static VALUE c_example;
 3)
 4) static VALUE mymethod(VALUE c, VALUE arg) {
 5)      fprintf(stderr, "Called mymethod with one arguments\n");
 6)      return Qnil;
 7) }
 8)
 9) static VALUE myvariablemethod(int argc, VALUE *argv, VALUE c) {
10)      VALUE optional;
11)
12)      fprintf(stderr, "Called myvariablemethod with variable
                          arguments\n");
13)
14)      rb_scan_args(argc, argv, "01", &optional);
15)
16)      return Qnil;
17) }
18)
19) void Init_example() {
20)     m_example = rb_define_module("Example");
21)     c_example = rb_define_class_under(m_example, "Class",
                                          rb_cObject);
22)
23)     rb_define_attr(c_example, "my_readonly_attr", 1, 0);
24)     rb_define_attr(c_example, "my_readwrite_attr", 1, 1);
25)
26)     rb_define_const(c_example, "MYCONST", INT2NUM(5));
27)
28)     rb_define_method(c_example, "mymethod", example_mymethod, 1);
29)     rb_define_method(c_example, "myvariablemethod",
                         example_variable_method, -1);
30) }

Lines 19 through 30 are the entry point for the extension. Line 20 defines and stores the module called "Example". Line 21 defines and stores the class "Class" under the module "Example". Line 23 defines a new read-only attribute for the class; this is equivalent to attr_reader in ruby code. This is read-only because the 3rd parameter is 1 and the 4th parameter is 0, meaning to generate a read method but no write method for this attribute. Line 24 defines a new read-write attribute for the class; this is equivalent to attr_accessor in ruby code. This is read-write because the 3rd parameter is 1 and the 4th parameter is 1, meaning to generate both read and write methods. Line 26 defines a new constant for the class called "MYCONST" with a value of 5; this can be accessed in ruby code via Example::Class::MYCONST. Line 28 defines a new method for "Example::Class" called "mymethod" that takes exactly one parameter. Line 29 defines a new method for "Example::Class" called "myvariablemethod" that takes a variable number of parameters.

Now that we have looked at the extension initialization, we can examine the implementation of the methods. Lines 4 through 7 implement the "mymethod" method; the first parameter is the class itself, and the second parameter is the required argument. Lines 9 through 17 implement the "myvariablemethod" method. As described earlier, this takes the number of arguments in argc, the argument array in argv, and the class in c. Line 14 uses rb_scan_args to define zero required arguments and one optional argument. We pass the address of the VALUE "optional" to rb_scan_args(); if an argument is given, this will be filled in with the argument, otherwise it will be set to "nil".

[1] http://www.oreillynet.com/ruby/blog/2007/04/c_extension_authors_use_rb_sca_1.html

Update: edited to make the examples readable

Wednesday, January 5, 2011

Writing Ruby Extensions in C - Part 2, RDoc

This is the second in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. This post focuses on documentation generation.

RDoc and ri


RDoc is the documentation generation system for ruby. The general idea is that the source code is marked up with specially-formatted comments, and then the rdoc tool is run against the source to generate the documentation. The output from this is either HTML documentation, or ri documentation, or both. Generating rdoc documentation is a simple matter of:
  1. Annotating the source code with the appropriate tags. The basic form of an RDoc tag is:
    
    /*
     * call-seq:
     *   obj.method(required, optional=0) -> retval
     *
     * Call +wrappedLibraryFunction
     * +[http://www.example.org/docs.html#wrappedLibraryFunction]
     * to execute wrappedLibraryFunction.  This method takes a
     * single required argument, and one optional argument that
     * defaults to 0 if not specified.  It returns retval, which
     * can be any valid ruby object
     */
    

    Most of my own knowledge about RDoc syntax comes from [1]; it is highly suggested reading. For more real-world examples of markup, please look at the ruby-libvirt bindings[3]; all of the methods are properly marked-up for RDoc.
  2. Adding appropriate task(s) to the Rakefile. This is very easy as rake has pre-defined tasks for generating RDoc documentation:
    
    1) require 'rake/rdoctask'
    2)
    3) RDOC_FILES = FileList["README.rdoc", "ext/example.c"]
    4)
    5) Rake::RDocTask.new do |rd|
    6)     rd.main = "README.rdoc"
    7)     rd.rdoc_dir = "doc/site/api"
    8)     rd.rdoc_files.include(RDOC_FILES)
    9) end
    10)
    11) Rake::RDocTask.new(:ri) do |rd|
    12)     rd.main = "README.rdoc"
    13)     rd.rdoc_dir = "doc/ri"
    14)     rd.options << "--ri-system"
    15)     rd.rdoc_files.include(RDOC_FILES)
    16) end
    

    Line 1 pulls in the rake rdoctask that does most of the work for us. Line 3 defines the files that will be looked at for generating the rdoc. Note that the order of files is important; if there are dependencies between C files, the earlier dependencies must be listed first. Lines 5 through 9 define the main rdoc task. By default Rake::RDocTask creates a task called "rdoc", so nothing needs to be supplied for that. The "main" attribute of the rd specifies where the top-level documentation comes from. The "rdoc_dir" attribute specifies where the output will go. The "rdoc_files" attributes specifies which files to look at; here we point it at the list defined at line 3. With this task in place, we can now execute:
    
    $ rake rdoc
    

    at the command-line and the rdoc files will be generated from the C files and placed in doc/site/api. Lines 11 through 16 look very similar to the previous rdoc command, with a couple of differences. First, since we supply a symbol to the Rake::RDocTask.new method, we get a task named "ri" instead of rdoc. Second, we specify an option in line 14 that tells rdoc to generate the ri documentation instead of the HTML rdoc documentation. Execution is again easy:
    
    $ rake ri
    

    This will generate the ri documentation from the C files and place the output in doc/ri.
While the idea behind RDoc is very cool, the actual implementation is a little bit weak for C extensions. RDoc just cannot handle several common C idioms:
  • Using a macro to define constants - I used to have code like:
    
    #define DEF_DOMSTATE(name) rb_define_const(c_domain, #name, INT2NUM(VIR_DOMAIN_##name))
    DEF_DOMSTATE(NOSTATE);
    DEF_DOMSTATE(RUNNING);
    

    in ruby-libvirt. This was nice because I didn't have to repeat myself twice on every definition line. Since RDoc couldn't handle the macro, I had to remove all of these to get proper RDoc documentation.
  • Classes and methods split across multiple files - this one is an absolute deal-breaker for me. ruby-libvirt consists of around 7500 lines of C code, and having all of that in one file is just not feasible. Instead I have the code split along functional lines, which makes maintenance much easier. However, RDoc as of ruby 1.8.7 cannot follow the dependencies across different files, and hence almost none of my documentation was being generated. Luckily I found a patch[2] that makes RDoc smart enough to work across different files, but it sucks because I have to continually patch my local Ruby version. Maybe 1.9 fixes this in a better way; the RDoc parser seems to have been completely re-written, so there is hope on that front.
  • Having methods for a class defined in a different file - this one isn't a C idiom as such, but it seems like a simple thing. Given the nature of the ruby-libvirt bindings, I used to have all of the methods concerning a particular class (say, Libvirt::Network) in the same file. That included the lookup and definition methods, which are technically methods of class Libvirt::Connect (e.g. network = conn.lookup_network_by_name('netname')). However, RDoc also cannot handle this, so I was missing the RDoc documentation for all of the lookup and definition methods. I've now changed this to have all of the lookup and definition methods in the connect.c file, but it clutters that file unnecessarily. Again, maybe the Ruby 1.9 rewrite of RDoc fixes this.
That being said, RDoc is the canonical Ruby way to generate documentation, so whatever limitations it has must be worked around. The above is just a list of problems that I have come across that need workarounds in order to properly generate RDoc documentation.
[1] http://www.rubyfleebie.com/an-introduction-to-rdoc/
[2] http://marc.info/?l=ruby-core&m=110691458204738&w=2
[3] http://libvirt.org/git/?p=ruby-libvirt.git;a=tree

Update: edited to make the example RDoc tagging readable
Update: edited to make the references readable
Update: edited to fix up minor formatting problem

Tuesday, January 4, 2011

Writing Ruby Extensions in C - Part 1, Project Setup

Earlier this year, I took over maintainership of the ruby-libvirt bindings[1]. While I had been contributing to the bindings on and off for the last couple of years, taking over maintainership has led me to learning about a whole range of issues deep inside ruby. Subsequently, I've found that while there is information scattered around the internet about writing these bindings, comprehensive guides (with examples) seem to be lacking. This series of blog posts aim to be a guide for anyone interested in some of the finer details of writing ruby extensions in C. All of these notes apply to Ruby 1.8. In theory, most of this also applies to Ruby 1.9, but I have not personally tested them or done much with Ruby 1.9, so your mileage may vary.

This information is culled from various places around the internet, along with reading the ruby source code and banging my head against a wall until things worked. The most useful resources I have found, besides the ruby sources, are at [2] and [3].

This first post will talk about the general structure of a ruby extension project, including documentation and building. Further posts will talk about programming considerations, including defining classes and methods, memory management, etc.

(NOTE: actually writing ruby extensions by hand seems to be kind of passe nowadays. Apparently FFI[4] is all the rage. That being said, I still find this a useful exercise, if to nobody but myself)

Directory structure

The directory structure of a ruby project is flexible, though most of the ruby extensions that I have seen follow a very similar pattern. Usually the top-level of the project contains a directory listing that looks like:
COPYING
NEWS
Rakefile
README.rdoc
doc/
ext/
The COPYING file contains the license for the project. The NEWS file typically contains information about releases. The Rakefile defines rake targets for the project (see the section about Rakefiles for more information). The README.rdoc file contains the header information that will be used when generating the RDoc documentation; see the post about RDoc for more details. The doc subdirectory contains any additional documentation about the project, including the code for the website, example usage of the code, etc. The ext/ directory typically contains the C source code for the extension module, which can be in any number of files (though note the caveat in the RDoc post about automatically generating RDoc documentation from multiple C files). The ext/ directory also contains the extconf.rb file (see the extconf and mkmf section), which controls how to build the extension.

mkmf and extconf

extconf and mkmf are the parts of the ruby extension build system that generate the header files and Makefile(s) needed to compile the C part of the program. Like the Rakefile, it is run through ruby so has all of the power of ruby at its disposal. A file named extconf.rb is generally placed in the ext/ subdirectory of the project, and extconf requires mkmf to do all of the heavy lifting for it. An example extconf.rb looks like:

 1) require 'mkmf'
 2)
 3) RbConfig::MAKEFILE_CONFIG['CC'] = ENV['CC'] if ENV['CC']
 4)
 5) extension_name = 'example'
 6)
 7) unless pkg_config('library_to_link_to')
 8)     raise "library_to_link_to not found"
 9) end
10)
11) have_func('useful_function', 'library_to_link_to/lib.h')
12) have_type('useful_type', 'library_to_link_to/lib.h')
13)
14) create_header
15) create_makefile(extension_name)
Line 1 just pulls in the mkmf module, which is what does all of the hard work here.

Line 3 isn't strictly necessary, but gives the ability to easily use alternate compilers to build the extension. Since mkmf detects the compiler at Makefile creation time, this isn't very interesting until you consider static analysis tools, which tend to substitute the standard compiler with their own enhanced version. By having this line of code at the top, the Rakefile is prepared to let these static analysis tools do their thing (and help improve your code).

Line 5 defines the extension name, which is used later.

Lines 7 through 9 do a pkgconfig check to see if the library necessary to build this extension exists. Typically you will need to have the development package of the library you want to use installed, including the header files. If the library cannot be found, an exception will be raised and no Makefiles will be generated. Note that this is a required first step; all of the have_*() functions later on work by trying to compile and link a program with the function, type, or constant that you are looking for, so they need to know where to find the library to link against.

Line 11 uses the mkmf function have_func() to determine if the library installed on the build system has the function 'useful_function' defined in the header file 'library_to_link_to/lib.h'. If the function is found, then a macro called HAVE_ will be defined in extconf.h (which all of the C files in the project should #include).

Line 12 uses the mkmf function have_type() to determine if the library installed on the build system has the structure 'useful_type' defined in the header file 'library_to_link_to/lib.h'. If the structure is found, then a macro called HAVE_TYPE_ will be defined in extconf.h.

Line 14 actually creates the header file extconf.h, based on the results from all of the previous have_*() functions. The extconf.h file should be #include'd by all of the C files in the project to gain access to the HAVE_* macros that extconf defines.

Line 15 creates the Makefile based on all of the previous information.
While the recommended way to invoke the extconf.rb is through the Rakefile (see the next section), you can also run it by hand to test it out. If the extconf.rb file is located in the recommended ext/ subdirectory, you can run:
$ cd ext
$ ruby extconf.rb
The mkmf commands should run, and if everything goes smoothly, the extconf.h and Makefile will be generated inside of the ext/ subdirectory. If things do not succeed, the output to stdout, or to mkmf.log should help to debug the problem.

Rakefile

Once the extconf is in place, the next step is to create a Rakefile. As the name suggests, Rakefiles are the ruby analog to Makefiles; they allow automation of arbitrary tasks with possible dependencies between them. They also only re-build pieces of the code that have changed since the last invocation. The main difference between Rakefiles and Makefiles is that Rakefiles are written in ruby, so you have the full power of ruby at your disposal.
With that said, let's take a look at a Rakefile. I'll preface this discussion by saying that I don't know all that much about Rakefiles, other than the bare minimum to get them working. There are additional resources out on the web to describe them in depth[5], so if you want to know more, please look there.

 1) require 'rake/clean'
 2)
 3) EXT_CONF = 'ext/extconf.rb'
 4) MAKEFILE = 'ext/Makefile'
 5) MODULE = 'ext/example.so'
 6) SRC = Dir.glob('ext/*.c')
 7) SRC << MAKEFILE
 8)
 9) CLEAN.include [ 'ext/*.o', 'ext/depend', MODULE ]
10) CLOBBER.include [ 'config.save', 'ext/mkmf.log', 'ext/extconf.h',
                      MAKEFILE ]
11)
12) file MAKEFILE => EXT_CONF do |t|
13)     Dir::chdir(File::dirname(EXT_CONF)) do
14)         unless sh "ruby #{File::basename(EXT_CONF)}"
15)             $stderr.puts "Failed to run extconf"
16)             break
17)         end
18)     end
19) end
20) file MODULE => SRC do |t|
21)     Dir::chdir(File::dirname(EXT_CONF)) do
22)         unless sh "make"
23)             $stderr.puts "make failed"
24)             break
25)         end
26)     end
27) end
28) desc "Build the native library"
29) task :build => MODULE
Line 1 brings in the rake task that we care about. There are many more pre-defined rake tasks available; some of them will be described in further posts.

Lines 3 through 7 set up some global ruby variables that we will use later on. The important point to note here is that we have the full power of ruby available to us, including doing directory globs, array concatenation, etc.
Lines 9 and 10 set up the list of files that will get removed during the CLEAN and CLOBBER steps, respectively. 'rake clean' will clean out the development files listed in the CLEAN variable, and 'rake clobber' will clean out the development files in the CLEAN and CLOBBER variables.

Lines 12 through 29 are the meat of the build task. Lines 28 and 29 set up the start of the dependency chain; any time the rake target of "build" is entered, it depends on everything in MODULE (which is 'ext/example.so'). When rake encounters that, it goes looking for any other dependencies that MODULE may have. In this case, we've defined that MODULE depends on SRC, which is a list of all C files in ext/, plus the Makefile. Since the Makefile is going to be auto-generated by mkmf, we have another dependency between the Makefile and EXT_CONF (which is responsible for generating the makefile). At this point we've reached the end of our dependency chain, so the block at lines 13 through 18 is executed, which produces the Makefile. Once that is done rake goes back up the dependency chain and executes the block at lines 21 to 26, which actually does the build using make. At the end of all of this, the extension module should be properly built (assuming no compile errors, of course).

Gem

The ruby gem system aims to be a package manager for pieces of ruby code. While my personal opinion is that this system re-invents operating system package managers (poorly), they are an integral part of the ruby experience. Gems can be easily built using a few rakefile commands, and they are generally registered at http://rubygems.org. A few minor additions to the Rakefile are used to setup the task:

 1) require 'rake/gempackagetask'
 2)
 3) PKG_FILES = FileList[
 4)     "Rakefile", "COPYING", "README", "NEWS", "README.rdoc",
 5)     "ext/*.[ch]", "ext/MANIFEST", "ext/extconf.rb",
 6) ]
 7)
 8) SPEC = Gem::Specification.new do |s|
 9)     s.name = "example"
10)     s.version = "1.0"
11)     s.email = "list@example.com"
12)     s.homepage = "http://example.org/"
13)     s.summary = "C bindings"
14)     s.files = PKG_FILES
15)     s.required_ruby_version = '>= 1.8.1'
16)     s.extensions = "ext/extconf.rb"
17)     s.author = "List of Authors"
18)     s.rubyforge_project = "None"
19)     s.description = "C Bindings"
20) end
21)
22) Rake::GemPackageTask.new(SPEC) do |pkg|
23)     pkg.need_tar = true
24)     pkg.need_zip = true
25) end
Line 1 brings in the rake gempackagetask. Lines 3 through 6 define the files that we want included in the package; ruby globs can be used here. Lines 8 through 20 are the meat of the gem specification, and are pretty straightforward; just replace the fields with ones appropriate for your project. Finally, lines 22 through 25 define the task itself. To actually build the gem, you would now run:

$ rake gem

[1] http://libvirt.org/ruby
[2] http://ruby-doc.org/core
[3] http://ruby-doc.org/docs/ProgrammingRuby/html/ext_ruby.html
[4] https://github.com/ffi/ffi
[5] http://jasonseifer.com/2010/04/06/rake-tutorial

Update: modified some of the examples to make sure the code wasn't cut-off