Monday, January 10, 2011

Writing Ruby Extensions in C - Part 4, Types and Return Values

This is the fourth in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The third post talked about initializing the module and setting up classes. This short post will focus on some details of method implementation, including how to check the types that are being passed to extension methods and the legal return values from the extension methods.

Ruby types


When implementing a ruby method in C, the method may expect certain arguments to be of a certain type. For instance, it is possible that the ruby method expects a number, and only a number, as the input parameter. The ruby C extension API provides several functions to check if an input parameter is a certain type:
  • TYPE(ruby_object) - return the builtin type of ruby_object. The builtin types distinguish between things like TrueClass, FalseClass, FIXNUM, etc. It explicitly does not distinguish between complicated object types; use either CLASS_OF() or rb_obj_classname() for that. The builtin types that may be returned are:
    • T_NONE
    • T_NIL
    • T_OBJECT
    • T_CLASS
    • T_ICLASS
    • T_MODULE
    • T_FLOAT
    • T_STRING
    • T_REGEXP
    • T_ARRAY
    • T_FIXNUM
    • T_HASH
    • T_STRUCT
    • T_BIGNUM
    • T_FILE
    • T_TRUE
    • T_FALSE
    • T_DATA
    • T_MATCH
    • T_SYMBOL
    • T_BLKTAG
    • T_UNDEF
    • T_VARMAP
    • T_SCOPE
    • T_NODE
  • NIL_P(ruby_object) - test if ruby_object is the nil object
  • CheckType(ruby_object, builtin_type) - check to make sure that ruby_object is of type builtin_type (one of the T_* types listed above). If it is not, an exception is raised
  • CLASS_OF(ruby_object) - return the ruby class VALUE that corresponds to ruby_object. Note that this can distinguish between built-in class types (such as rb_cSymbol) as well as more complicated class types (such as those defined by the API user)
  • rb_obj_classname(ruby_object) - return the char * string representation of the class corresponding to ruby_object

Return values


Every ruby method implemented in C has to return a VALUE. This VALUE can either be a ruby object (such as that returned by INT2NUM), or one of the special values:
  • Qnil - ruby "nil"
  • Qtrue - ruby "true"
  • Qfalse - ruby "false"

Methods that are expected to either succeed or raise an exception typically return Qnil to indicate success.

Thursday, January 6, 2011

Writing Ruby Extensions in C - Part 3, Extension Initialization

This is the third in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. The second post talked about generating documentation. The posts from here on out will focus on the C code. This post talks about initializing the module and setting up classes.

Initializing the module


There is a bit of magic involved with initially loading the extension module into ruby. Assuming the extension module is called "example", then the C code that implements the extension must have an initialization function that looks like:

 1) static VALUE m_example;
 2)
 3) void Init_example() {
 4)     m_example = rb_define_module("Example");
 5)     example_library_initialize();
 6) }

Line 1 sets up the variable that holds the reference to the module. Line 3 is a function that must be called "Init_<extension_name>", take no parameters, and return nothing. When the ruby interpreter encounters a line of code such as "require 'example'", it will call this initialization function to set things up. Line 4 actually defines the module for us and calls it "Example". Finally, line 5 does whatever initialization is necessary for the library that is being wrapped. In this case, it just calls the example_library_initialize() function.

Defining classes, constants, and methods


Once the module itself has been initialized, functions, classes, methods, and attributes can be added to it. These are pretty easy to use:
  • rb_define_module_function(module, "function_name", implementation, number_of_args) - define function_name for module. Assuming the module is called "Example", functions like this can be invoked from ruby code like:

    out = Example::function_name

    The implementation should be a C function that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.
  • rb_define_class_under(module, "class_name", super_class) - define a new class named "class_name" under the module. super_class can be one of the pre-defined types (rb_cObject, rb_cArray, etc) or a class that has been defined in this module.
  • rb_define_method(class, "method_name", implementation, number_of_args) - define a new method for class. The implementation should be a C function that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.
  • rb_define_const(class, "CONST", value) - define a new constant for class with value. Assuming the module is called "Example" and the class is called "Class", these can be accessed in ruby code like:

    puts Example::Class::CONST

    The value can be any legal ruby type.
  • rb_define_attr(class, "attr_name", read, write) - define a new attribute for class called attr_name. The read and write parameters should each be 0 or 1, depending on whether you want a read implementation and/or a write_implementation for this attribute, respectively.
  • rb_define_singleton_method(class, "method_name", implementation, number_of_args) - define a new singleton method for class. The implementation should be a C functions that takes number_of_args and returns a VALUE. See "Implementing methods" below for more explanation of implementation of methods in C.

Implementing methods


Using the above methods, it is pretty straightforward to define module functions, class methods, and singleton methods. There is a bit of work necessary to understand the C implementation of these methods. The first thing to realize is that the "number_of_args" as the last parameter of the rb_define_* call defines how many parameters the method will take. For no parameters, you would pass 0, for one parameter you would pass 1, etc. When you go to implement the method in C, your C function must take the number of parameters, plus one for the class (this will be shown in the example below).

You can also pass -1, which tells ruby that you want to take optional arguments. When you go to implement the method in C, the C function must take exactly 3 arguments: int argc, VALUE *argv, VALUE klass. The argc parameter defines how many arguments were passed, the argv parameter is all of the arguments in an array, and the last parameter is the klass itself. To properly parse the arguments, the rb_scan_args(argc, argv, "format", ...) should be called. A brief explanation of rb_scan_args is below; for more information, see the document at [1].

The first two arguments to rb_scan_args() are the argc and argv passed into the function. The third argument is a string that defines how many required and how many optional parameters the method requires. The last parameters are pointers to VALUEs to place the value of the arguments in. For instance, to have 1 required and 2 optional parameters to the method, format should be "12" and 3 additional VALUE parameters should be passed to rb_scan_args(). To have no required and 1 optional parameters to the method, format should be "01" and 1 additional VALUE parameter should be passed to rb_scan_args(). Note that if less than the number of required parameters is passed to the method, an ArgumentError exception will be raised. All optional arguments are set to the value that was passed, if any, or "nil".

Let's take a look at an example to show all of this off:

 1) static VALUE m_example;
 2) static VALUE c_example;
 3)
 4) static VALUE mymethod(VALUE c, VALUE arg) {
 5)      fprintf(stderr, "Called mymethod with one arguments\n");
 6)      return Qnil;
 7) }
 8)
 9) static VALUE myvariablemethod(int argc, VALUE *argv, VALUE c) {
10)      VALUE optional;
11)
12)      fprintf(stderr, "Called myvariablemethod with variable
                          arguments\n");
13)
14)      rb_scan_args(argc, argv, "01", &optional);
15)
16)      return Qnil;
17) }
18)
19) void Init_example() {
20)     m_example = rb_define_module("Example");
21)     c_example = rb_define_class_under(m_example, "Class",
                                          rb_cObject);
22)
23)     rb_define_attr(c_example, "my_readonly_attr", 1, 0);
24)     rb_define_attr(c_example, "my_readwrite_attr", 1, 1);
25)
26)     rb_define_const(c_example, "MYCONST", INT2NUM(5));
27)
28)     rb_define_method(c_example, "mymethod", example_mymethod, 1);
29)     rb_define_method(c_example, "myvariablemethod",
                         example_variable_method, -1);
30) }

Lines 19 through 30 are the entry point for the extension. Line 20 defines and stores the module called "Example". Line 21 defines and stores the class "Class" under the module "Example". Line 23 defines a new read-only attribute for the class; this is equivalent to attr_reader in ruby code. This is read-only because the 3rd parameter is 1 and the 4th parameter is 0, meaning to generate a read method but no write method for this attribute. Line 24 defines a new read-write attribute for the class; this is equivalent to attr_accessor in ruby code. This is read-write because the 3rd parameter is 1 and the 4th parameter is 1, meaning to generate both read and write methods. Line 26 defines a new constant for the class called "MYCONST" with a value of 5; this can be accessed in ruby code via Example::Class::MYCONST. Line 28 defines a new method for "Example::Class" called "mymethod" that takes exactly one parameter. Line 29 defines a new method for "Example::Class" called "myvariablemethod" that takes a variable number of parameters.

Now that we have looked at the extension initialization, we can examine the implementation of the methods. Lines 4 through 7 implement the "mymethod" method; the first parameter is the class itself, and the second parameter is the required argument. Lines 9 through 17 implement the "myvariablemethod" method. As described earlier, this takes the number of arguments in argc, the argument array in argv, and the class in c. Line 14 uses rb_scan_args to define zero required arguments and one optional argument. We pass the address of the VALUE "optional" to rb_scan_args(); if an argument is given, this will be filled in with the argument, otherwise it will be set to "nil".

[1] http://www.oreillynet.com/ruby/blog/2007/04/c_extension_authors_use_rb_sca_1.html

Update: edited to make the examples readable

Wednesday, January 5, 2011

Writing Ruby Extensions in C - Part 2, RDoc

This is the second in my series of posts about writing ruby extensions in C. The first post talked about the basic structure of a project, including how to set up building. This post focuses on documentation generation.

RDoc and ri


RDoc is the documentation generation system for ruby. The general idea is that the source code is marked up with specially-formatted comments, and then the rdoc tool is run against the source to generate the documentation. The output from this is either HTML documentation, or ri documentation, or both. Generating rdoc documentation is a simple matter of:
  1. Annotating the source code with the appropriate tags. The basic form of an RDoc tag is:
    
    /*
     * call-seq:
     *   obj.method(required, optional=0) -> retval
     *
     * Call +wrappedLibraryFunction
     * +[http://www.example.org/docs.html#wrappedLibraryFunction]
     * to execute wrappedLibraryFunction.  This method takes a
     * single required argument, and one optional argument that
     * defaults to 0 if not specified.  It returns retval, which
     * can be any valid ruby object
     */
    

    Most of my own knowledge about RDoc syntax comes from [1]; it is highly suggested reading. For more real-world examples of markup, please look at the ruby-libvirt bindings[3]; all of the methods are properly marked-up for RDoc.
  2. Adding appropriate task(s) to the Rakefile. This is very easy as rake has pre-defined tasks for generating RDoc documentation:
    
    1) require 'rake/rdoctask'
    2)
    3) RDOC_FILES = FileList["README.rdoc", "ext/example.c"]
    4)
    5) Rake::RDocTask.new do |rd|
    6)     rd.main = "README.rdoc"
    7)     rd.rdoc_dir = "doc/site/api"
    8)     rd.rdoc_files.include(RDOC_FILES)
    9) end
    10)
    11) Rake::RDocTask.new(:ri) do |rd|
    12)     rd.main = "README.rdoc"
    13)     rd.rdoc_dir = "doc/ri"
    14)     rd.options << "--ri-system"
    15)     rd.rdoc_files.include(RDOC_FILES)
    16) end
    

    Line 1 pulls in the rake rdoctask that does most of the work for us. Line 3 defines the files that will be looked at for generating the rdoc. Note that the order of files is important; if there are dependencies between C files, the earlier dependencies must be listed first. Lines 5 through 9 define the main rdoc task. By default Rake::RDocTask creates a task called "rdoc", so nothing needs to be supplied for that. The "main" attribute of the rd specifies where the top-level documentation comes from. The "rdoc_dir" attribute specifies where the output will go. The "rdoc_files" attributes specifies which files to look at; here we point it at the list defined at line 3. With this task in place, we can now execute:
    
    $ rake rdoc
    

    at the command-line and the rdoc files will be generated from the C files and placed in doc/site/api. Lines 11 through 16 look very similar to the previous rdoc command, with a couple of differences. First, since we supply a symbol to the Rake::RDocTask.new method, we get a task named "ri" instead of rdoc. Second, we specify an option in line 14 that tells rdoc to generate the ri documentation instead of the HTML rdoc documentation. Execution is again easy:
    
    $ rake ri
    

    This will generate the ri documentation from the C files and place the output in doc/ri.
While the idea behind RDoc is very cool, the actual implementation is a little bit weak for C extensions. RDoc just cannot handle several common C idioms:
  • Using a macro to define constants - I used to have code like:
    
    #define DEF_DOMSTATE(name) rb_define_const(c_domain, #name, INT2NUM(VIR_DOMAIN_##name))
    DEF_DOMSTATE(NOSTATE);
    DEF_DOMSTATE(RUNNING);
    

    in ruby-libvirt. This was nice because I didn't have to repeat myself twice on every definition line. Since RDoc couldn't handle the macro, I had to remove all of these to get proper RDoc documentation.
  • Classes and methods split across multiple files - this one is an absolute deal-breaker for me. ruby-libvirt consists of around 7500 lines of C code, and having all of that in one file is just not feasible. Instead I have the code split along functional lines, which makes maintenance much easier. However, RDoc as of ruby 1.8.7 cannot follow the dependencies across different files, and hence almost none of my documentation was being generated. Luckily I found a patch[2] that makes RDoc smart enough to work across different files, but it sucks because I have to continually patch my local Ruby version. Maybe 1.9 fixes this in a better way; the RDoc parser seems to have been completely re-written, so there is hope on that front.
  • Having methods for a class defined in a different file - this one isn't a C idiom as such, but it seems like a simple thing. Given the nature of the ruby-libvirt bindings, I used to have all of the methods concerning a particular class (say, Libvirt::Network) in the same file. That included the lookup and definition methods, which are technically methods of class Libvirt::Connect (e.g. network = conn.lookup_network_by_name('netname')). However, RDoc also cannot handle this, so I was missing the RDoc documentation for all of the lookup and definition methods. I've now changed this to have all of the lookup and definition methods in the connect.c file, but it clutters that file unnecessarily. Again, maybe the Ruby 1.9 rewrite of RDoc fixes this.
That being said, RDoc is the canonical Ruby way to generate documentation, so whatever limitations it has must be worked around. The above is just a list of problems that I have come across that need workarounds in order to properly generate RDoc documentation.
[1] http://www.rubyfleebie.com/an-introduction-to-rdoc/
[2] http://marc.info/?l=ruby-core&m=110691458204738&w=2
[3] http://libvirt.org/git/?p=ruby-libvirt.git;a=tree

Update: edited to make the example RDoc tagging readable
Update: edited to make the references readable
Update: edited to fix up minor formatting problem

Tuesday, January 4, 2011

Writing Ruby Extensions in C - Part 1, Project Setup

Earlier this year, I took over maintainership of the ruby-libvirt bindings[1]. While I had been contributing to the bindings on and off for the last couple of years, taking over maintainership has led me to learning about a whole range of issues deep inside ruby. Subsequently, I've found that while there is information scattered around the internet about writing these bindings, comprehensive guides (with examples) seem to be lacking. This series of blog posts aim to be a guide for anyone interested in some of the finer details of writing ruby extensions in C. All of these notes apply to Ruby 1.8. In theory, most of this also applies to Ruby 1.9, but I have not personally tested them or done much with Ruby 1.9, so your mileage may vary.

This information is culled from various places around the internet, along with reading the ruby source code and banging my head against a wall until things worked. The most useful resources I have found, besides the ruby sources, are at [2] and [3].

This first post will talk about the general structure of a ruby extension project, including documentation and building. Further posts will talk about programming considerations, including defining classes and methods, memory management, etc.

(NOTE: actually writing ruby extensions by hand seems to be kind of passe nowadays. Apparently FFI[4] is all the rage. That being said, I still find this a useful exercise, if to nobody but myself)

Directory structure

The directory structure of a ruby project is flexible, though most of the ruby extensions that I have seen follow a very similar pattern. Usually the top-level of the project contains a directory listing that looks like:
COPYING
NEWS
Rakefile
README.rdoc
doc/
ext/
The COPYING file contains the license for the project. The NEWS file typically contains information about releases. The Rakefile defines rake targets for the project (see the section about Rakefiles for more information). The README.rdoc file contains the header information that will be used when generating the RDoc documentation; see the post about RDoc for more details. The doc subdirectory contains any additional documentation about the project, including the code for the website, example usage of the code, etc. The ext/ directory typically contains the C source code for the extension module, which can be in any number of files (though note the caveat in the RDoc post about automatically generating RDoc documentation from multiple C files). The ext/ directory also contains the extconf.rb file (see the extconf and mkmf section), which controls how to build the extension.

mkmf and extconf

extconf and mkmf are the parts of the ruby extension build system that generate the header files and Makefile(s) needed to compile the C part of the program. Like the Rakefile, it is run through ruby so has all of the power of ruby at its disposal. A file named extconf.rb is generally placed in the ext/ subdirectory of the project, and extconf requires mkmf to do all of the heavy lifting for it. An example extconf.rb looks like:

 1) require 'mkmf'
 2)
 3) RbConfig::MAKEFILE_CONFIG['CC'] = ENV['CC'] if ENV['CC']
 4)
 5) extension_name = 'example'
 6)
 7) unless pkg_config('library_to_link_to')
 8)     raise "library_to_link_to not found"
 9) end
10)
11) have_func('useful_function', 'library_to_link_to/lib.h')
12) have_type('useful_type', 'library_to_link_to/lib.h')
13)
14) create_header
15) create_makefile(extension_name)
Line 1 just pulls in the mkmf module, which is what does all of the hard work here.

Line 3 isn't strictly necessary, but gives the ability to easily use alternate compilers to build the extension. Since mkmf detects the compiler at Makefile creation time, this isn't very interesting until you consider static analysis tools, which tend to substitute the standard compiler with their own enhanced version. By having this line of code at the top, the Rakefile is prepared to let these static analysis tools do their thing (and help improve your code).

Line 5 defines the extension name, which is used later.

Lines 7 through 9 do a pkgconfig check to see if the library necessary to build this extension exists. Typically you will need to have the development package of the library you want to use installed, including the header files. If the library cannot be found, an exception will be raised and no Makefiles will be generated. Note that this is a required first step; all of the have_*() functions later on work by trying to compile and link a program with the function, type, or constant that you are looking for, so they need to know where to find the library to link against.

Line 11 uses the mkmf function have_func() to determine if the library installed on the build system has the function 'useful_function' defined in the header file 'library_to_link_to/lib.h'. If the function is found, then a macro called HAVE_ will be defined in extconf.h (which all of the C files in the project should #include).

Line 12 uses the mkmf function have_type() to determine if the library installed on the build system has the structure 'useful_type' defined in the header file 'library_to_link_to/lib.h'. If the structure is found, then a macro called HAVE_TYPE_ will be defined in extconf.h.

Line 14 actually creates the header file extconf.h, based on the results from all of the previous have_*() functions. The extconf.h file should be #include'd by all of the C files in the project to gain access to the HAVE_* macros that extconf defines.

Line 15 creates the Makefile based on all of the previous information.
While the recommended way to invoke the extconf.rb is through the Rakefile (see the next section), you can also run it by hand to test it out. If the extconf.rb file is located in the recommended ext/ subdirectory, you can run:
$ cd ext
$ ruby extconf.rb
The mkmf commands should run, and if everything goes smoothly, the extconf.h and Makefile will be generated inside of the ext/ subdirectory. If things do not succeed, the output to stdout, or to mkmf.log should help to debug the problem.

Rakefile

Once the extconf is in place, the next step is to create a Rakefile. As the name suggests, Rakefiles are the ruby analog to Makefiles; they allow automation of arbitrary tasks with possible dependencies between them. They also only re-build pieces of the code that have changed since the last invocation. The main difference between Rakefiles and Makefiles is that Rakefiles are written in ruby, so you have the full power of ruby at your disposal.
With that said, let's take a look at a Rakefile. I'll preface this discussion by saying that I don't know all that much about Rakefiles, other than the bare minimum to get them working. There are additional resources out on the web to describe them in depth[5], so if you want to know more, please look there.

 1) require 'rake/clean'
 2)
 3) EXT_CONF = 'ext/extconf.rb'
 4) MAKEFILE = 'ext/Makefile'
 5) MODULE = 'ext/example.so'
 6) SRC = Dir.glob('ext/*.c')
 7) SRC << MAKEFILE
 8)
 9) CLEAN.include [ 'ext/*.o', 'ext/depend', MODULE ]
10) CLOBBER.include [ 'config.save', 'ext/mkmf.log', 'ext/extconf.h',
                      MAKEFILE ]
11)
12) file MAKEFILE => EXT_CONF do |t|
13)     Dir::chdir(File::dirname(EXT_CONF)) do
14)         unless sh "ruby #{File::basename(EXT_CONF)}"
15)             $stderr.puts "Failed to run extconf"
16)             break
17)         end
18)     end
19) end
20) file MODULE => SRC do |t|
21)     Dir::chdir(File::dirname(EXT_CONF)) do
22)         unless sh "make"
23)             $stderr.puts "make failed"
24)             break
25)         end
26)     end
27) end
28) desc "Build the native library"
29) task :build => MODULE
Line 1 brings in the rake task that we care about. There are many more pre-defined rake tasks available; some of them will be described in further posts.

Lines 3 through 7 set up some global ruby variables that we will use later on. The important point to note here is that we have the full power of ruby available to us, including doing directory globs, array concatenation, etc.
Lines 9 and 10 set up the list of files that will get removed during the CLEAN and CLOBBER steps, respectively. 'rake clean' will clean out the development files listed in the CLEAN variable, and 'rake clobber' will clean out the development files in the CLEAN and CLOBBER variables.

Lines 12 through 29 are the meat of the build task. Lines 28 and 29 set up the start of the dependency chain; any time the rake target of "build" is entered, it depends on everything in MODULE (which is 'ext/example.so'). When rake encounters that, it goes looking for any other dependencies that MODULE may have. In this case, we've defined that MODULE depends on SRC, which is a list of all C files in ext/, plus the Makefile. Since the Makefile is going to be auto-generated by mkmf, we have another dependency between the Makefile and EXT_CONF (which is responsible for generating the makefile). At this point we've reached the end of our dependency chain, so the block at lines 13 through 18 is executed, which produces the Makefile. Once that is done rake goes back up the dependency chain and executes the block at lines 21 to 26, which actually does the build using make. At the end of all of this, the extension module should be properly built (assuming no compile errors, of course).

Gem

The ruby gem system aims to be a package manager for pieces of ruby code. While my personal opinion is that this system re-invents operating system package managers (poorly), they are an integral part of the ruby experience. Gems can be easily built using a few rakefile commands, and they are generally registered at http://rubygems.org. A few minor additions to the Rakefile are used to setup the task:

 1) require 'rake/gempackagetask'
 2)
 3) PKG_FILES = FileList[
 4)     "Rakefile", "COPYING", "README", "NEWS", "README.rdoc",
 5)     "ext/*.[ch]", "ext/MANIFEST", "ext/extconf.rb",
 6) ]
 7)
 8) SPEC = Gem::Specification.new do |s|
 9)     s.name = "example"
10)     s.version = "1.0"
11)     s.email = "list@example.com"
12)     s.homepage = "http://example.org/"
13)     s.summary = "C bindings"
14)     s.files = PKG_FILES
15)     s.required_ruby_version = '>= 1.8.1'
16)     s.extensions = "ext/extconf.rb"
17)     s.author = "List of Authors"
18)     s.rubyforge_project = "None"
19)     s.description = "C Bindings"
20) end
21)
22) Rake::GemPackageTask.new(SPEC) do |pkg|
23)     pkg.need_tar = true
24)     pkg.need_zip = true
25) end
Line 1 brings in the rake gempackagetask. Lines 3 through 6 define the files that we want included in the package; ruby globs can be used here. Lines 8 through 20 are the meat of the gem specification, and are pretty straightforward; just replace the fields with ones appropriate for your project. Finally, lines 22 through 25 define the task itself. To actually build the gem, you would now run:

$ rake gem

[1] http://libvirt.org/ruby
[2] http://ruby-doc.org/core
[3] http://ruby-doc.org/docs/ProgrammingRuby/html/ext_ruby.html
[4] https://github.com/ffi/ffi
[5] http://jasonseifer.com/2010/04/06/rake-tutorial

Update: modified some of the examples to make sure the code wasn't cut-off

Thursday, April 15, 2010

Libvirt snapshotting support

One of the oft-requested features for libvirt has been snapshotting support; that is, the ability to take a snapshot of a virtual machine at a point in time, and then later on go back in time to that snapshot. It's a pretty neat feature to see in action, and under the hoods there is a lot of trickery going on to make it work. I'm pleased to say that as of libvirt 0.8.0, we have snapshotting support for qemu/kvm, Virtualbox, and ESX in the main libvirt API's.
The design of the API's went through many iterations, until we finally settled on an API that seems to fit the snapshot model of most of the hypervisors pretty well. To take a snapshot, virDomainSnapshotCreateXML() is used with an appropriate virDomainPtr and snapshot XML. The snapshot XML looks like:

<domainsnapshot>
<name>XYZ</name>
<creationdate>...</creationdate>
<description>...</description>
<state>running</state>
<domain>
<uuid>XXXXX-XXXX-XXXX-XXXX-XXXXXXXXX</uuid>
</domain>
<parent>
<name>ABC</name>
</parent>
</domainsnapshot>

However, when creating a snapshot, only the <name> and <description> tags are settable by the user. All of the other fields are ignored and filled in by the libvirt driver at the time the snapshot is actually created. The <domainsnapshot> XML is pretty straightforward, but I'll describe the fields here.

<name> is a unique identifier for this snapshot for this domain. It's what will be used later on to lookup the snapshot to perform operations with it or on it. If the <name> is not specified at snapshot creation time, then libvirt will make one up.

<creationdate> is the time, in seconds since the Unix epoch, that the snapshot was created at. This is read-only and is automatically filled in by libvirt when the snapshot is created.

<description> is a user-editable field that can contain any unique identifying information the user wants to store along with the snapshot. If this is blank at snapshot creation time, it remains empty.

<state> is the state of the domain (running, offline, paused, etc) at the time the snapshot was taken. When a user reverts to a particular snapshot, the domain's state will be set to this state.

<domain><uuid> is the UUID corresponding to the domain that this snapshot is taken against.

<parent><name> is the name of the parent of this snapshot (if any). This tracks the parent/child relationship in "trees" of snapshots. It is important information to know when deleting snapshots, as deleting a parent snapshot has interesting repercussions for children (see virDomainSnapshotDelete() below).

Once you've created a snapshot, you can lookup the snapshot by name, query all of the snapshots for a domain, or get the currently running snapshot for a domain. At some point in the future, the user will probably want to revert back to the snapshot he has taken. To do this, a handle to the domain snapshot must be obtained with virDomainSnapshotLookupByName(). Once the handle is obtained, the domain can be reverted to the point-in-time of that snapshot by calling virDomainRevertToSnapshot(). This is pretty cool to see in action; the domain is running along, and the moment virDomainRevertToSnapshot() is called, the domain instantly travels back to the past!

Finally, I'll talk a bit about deleting snapshots. Once a user is done using a snapshot, they may want to delete that snapshot. In the simple case of a snapshot without children, a call to virDomainSnapshotDelete() will remove all traces of the snapshot. If a snapshot does have children, then things get more interesting. First, if the VIR_DOMAIN_SNAPSHOT_DELETE_CHILDREN flag is passed to virDomainSnapshotDelete(), then the current snapshot and all children of this snapshot are deleted. Second, if VIR_DOMAIN_SNAPSHOT_DELETE_CHILDREN is *not* passed, but this snapshot does have children, then all changes that are present in the to-be-deleted parent are automatically merged into the children. If you think about it, this is necessary to keep the children viable.

That's some of the fun stuff going on in libvirt. If you have questions, comments, or problems with this feature, please feel free to contact the libvirt mailing list at libvirt-users@redhat.com or libvirt-devel@redhat.com.

Monday, April 5, 2010

Ubuntu 9.10 runlevel 3

With upstart and grub2 in Ubuntu 9.10, the way to configure the default runlevel has drastically changed. I'm just writing a quick blog post so I don't forget:

1) Edit /etc/default/grub and make GRUB_CMDLINE_LINUX_DEFAULT look like:

GRUB_CMDLINE_LINUX_DEFAULT="text"

2) Run "sudo update-grub"

3) Edit /etc/inittab, and make it look like:

id:3:initdefault

That should be enough to force Ubuntu 9.10 to boot into runlevel 3

Saturday, December 5, 2009

Smugapi

My girlfriend and I use SmugMug for all of our pictures. It's actually been amazingly helpful for organizing and storing all of our photos, let's us easily show our photos to everyone else, and is pretty cheap.

However, when we first started using it there weren't a lot of great clients for uploading pictures from Linux. There are a few command-line clients, but I found that they weren't greatly documented, and I didn't like the way they were written. In particular, they were written in Python (which is great!), but they did not use an object-oriented API. Because of this, they didn't feel very natural, and seemed hard to extend.

So I decided to implement my own little library for accessing the SmugMug API's: SmugAPI. It implements a very object-oriented Python class, with the ability to manipulate albums, images, categories, and sub-categories. It also comes with a command-line client for uploading files, smugtool. All of this is fairly well documented, and I even include unittests. Finally, all of the code is under the GPLv2.

SmugAPI is available in GitHub. Let me know what you think, and if you have feature requests, bugs, or other problems, please let me know about it!