Tuesday, February 8, 2011

FUDCon 2011 - Day 2

Day 2

The second day of FUDCon was much shorter than the first day. In particular,
there were only presentations in the morning, and the afternoon was devoted to hackfests. I started out by going to a few talks.

Asterisk Hacks

This was a fun session for me to attend. I've always wanted to setup Asterisk at home, if for nothing else to play with it. Up until now, I haven't found the time, but I'm hoping to do so in the near future. This talk was about various out-of-the-way features of Asterisk.

Being a telephony system, Asterisk can do all of the normal stuff you associate with telephony: make and receive calls, do call queuing, voicemail, 3-way calling, conference calls, etc. However, there are a number of fun things that Asterisk can do.

The first one is the ability to shift the pitch of your voice. There is a PITCH_SHIFT setting in the Asterisk configuration files that allows you to change the pitch of the voice on the sending or receiving end. There don't seem to be too many practical uses, but it could be fun to be both the secretary and manager of your "company".

The next hack is the ability to avoid someone. Asterisk can do call routing based on the caller ID coming in. You can then have it automatically hang up, redirect to voice mail, or even redirect to another number. All very useful if you are trying to avoid an irate ex-girlfriend ;).

You can also use Asterisk to increase the volume on a connection automatically for certain phone numbers. This might be useful if you often talk to someone that is very soft-spoken.

Finally, Asterisk can be used to trigger calls based on calendar events. It currently has support for integrating with caldev, ical, and exchange calendars, and based on events on those calendars it can trigger a call when that event is starting. This could be very useful for remembering meetings.

GIMP as a pro photo editing tool
I then went to the session on GIMP hosted by María (tatica). She was fun and engaging, and showed off a few tools including GIMP and a couple of others for editing photos. Unfortunately I was deep in the middle of hacking, so while I looked up from time-to-time, I did not fully listen. Sorry María!

The last official talk I went to was about Matahari, which is piece of software aiming to provide remote APIs for system management. The full slide deck is here; I'll just go over some highlights.

  • Matahari is aiming to provide cross-platform, generically useful APIs over remote interfaces
  • Initial targets are for machine/guest introspection and general OS management
  • Matahari is implemented as a QMF agent; uses the AMQP protocol underneath
  • Matahari does not provide policy; needs to be driven externally
  • Initial agents are for Host, Network, Services

This was the last session of the day. The rest of the day was devoted to hackfests. For my part, I spent a bunch of time with Marek Goldmann, the maintainer of Boxgrinder. Boxgrinder and my own project, Oz, share some of the same common goals. The nice part, however, is that there isn't a lot of overlap, and we talked a lot about integrating Oz with boxgrinder and how to speed up operating system installs. I also got quite a bit of hacking done, so I thought this was a useful exercise for me.

All in all, I thought that it was a useful and productive FUDCon. I would definitely go to another one to learn about the things going on in Fedora and meet some more intelligent, fun people.

FUDCon 2011 - Day 1

(sorry this is late; it just took me a couple of weeks to get my act together and get this written)

The weekend of Jan 29 and 30 I went to FUDCon 2011 in Tempe, Arizona. Thanks very much to Robyn Bergeron and all of the rest of the people who did the planning and logistics. This was my first FUDCon, and from my point of view things went off very smoothly.

As has been mentioned elsewhere, the location of the event was excellent. Since I was coming from the snowy northeast, it was great to get a break and have some nice warm weather. The fact that both hotels were in walking distance of the conference venue was also a big plus; I didn't have to rent a car or take a taxi or anything like that.

I arrived on Friday night and headed straight to the primary hotel where the welcome event and opensource.com birthday party were happening. It was nice to meet some people I had only spoken with on IRC, and to have a few drinks and pizza :).

Day 1

On Saturday morning the conference started. My introduction to a "barcamp" style conference started when I showed up at the door to the conference, and I was enlisted to help bring some boxes down to the initial meeting room. People started filtering in, and after some logistics we headed straight into the "pitches" for the talks. Each person who wants to give a talk gives a 20 second synopsis of what they will talk about. Then each of the conference attendees gets to vote for the talks that they want to hear. I pitched my talk about "Cloud Management - Deltacloud, Aeolus, and friends", and after the voting it seemed like there was a lot of interest in the topic. While the votes were being tallied and the scheduled was being finalized, Jared Smith (the Fedora Project Leader) gave his "State of Fedora" talk. At this point we headed over to the main conference venue. My talk was scheduled for the end of the day on Saturday, so I was able to enjoy a few talks before giving mine.

Fedora on ARM

The first talk I attended was the "Fedora on ARM" talk. ARM is one of the world's most widely produced processors, and runs on everything from phones up to servers (in the near future). Why do we want to port Fedora to it? First, the OLPC XO 1.75 is using an ARM processor, and we would like to continue to be able to run Fedora on it. Second, several development boards are currently in production (GuruPlug, Panda board, etc), and running Fedora on those is a great way to get started with ARM development. Third, future tablets and netbooks will likely have ARM processors. And finally, ARM is looking to expand into the server market to compete with x86. Since this is an area that Linux has dominated, it would be good to get a head start here. Servers running ARM aren't quite available yet, but when they do become available they are projected to use 75% less power than equivalent x86 machines.

Previous efforts to port Fedora on ARM have fizzled out. This new effort has a koji build instance at arm.koji.fedoraproject.org, and any developer can submit builds there. It is hoped that this will encourage more participation in the process. The current effort basically has Fedora 13 running on ARM, though there are a couple of problems preventing a full "release". Right now the target platform in ARMv5tel, though they are looking to go to ARMv7 in the future. Now that Fedora 13 mostly works, getting Fedora 14 working is expected to be much easier. All of the patches that have been needed have been sent upstream, and in general things should move a lot faster. In theory, it should be easy to support both ARMv5 and ARMv7 in Fedora (as the instruction sets are compatible), but the reality of the situation is a bit different. ARMv5 has no required FPU, meaning that it is often the case that it has to fall back to software FPU. The ARMv7 specification requires a FPU. Additionally, in ARMv7 some parameters can be passed in FP registers, which obviously will not work on ARMv5. While you can definitely compile for ARMv5 and then run on ARMv7, the performance hit is pretty dramatic. Therefore, there will be two sub-architectures for ARM, one targeting v5 and one targeting v7.

There have been a number of challenges getting ARM building on Fedora. On the hardware side, most of the boards that are available are fairly slow. 512MB of RAM, single-core, with slow I/O seems to be the norm. There are some new "Panda boards" that have 1GB of memory and 2 cores, but the ethernet connection is slow (100Mbit), and the storage subsystem is slow. What this all means is that compiling on these machines takes a long time. On the software side, the challenges have mostly been around software that does not support ARM. For instance, OCaml in Fedora-13 does not support ARM (though this is fixed in Fedora-15). Other packages have had to be patched to deal with ARM alignment problems, compile problems, etc.

In terms of using Fedora on ARM, there are a few hurdles to overcome. The first is that for a random off-the-shelf ARM board, the bootloaders tend to be proprietary. That means that it is not always clear how to load a custom kernel onto the board. The u-boot project can help with many, but not all boards. Even if you have figured out how to load a custom kernel, the Fedora ARM port currently does not provide a packaged kernel, so it is up to you to compile one and create a minimal root filesystem to get the system bootstrapped.

The last part of this session was a quick overview of OLPC. The new version of the OLPC (1.75) will be running an ARMv7 processor, and they want to continue to be able to run Fedora on them. At the same time, there are 2 million x86-based OLPC machines out in the wild, so they want to continue to run Fedora on those machines. In particular, keeping the base distribution small and keeping dependencies down will help on the old OLPCs which only have 256MB of memory.


The next talk I attended was about Sheepdog, a distributed storage system for Qemu/KVM block storage. I had previously seen a talk about this at KVM Forum 2010, but I went in hoping to understand some more details.

What's the motivation for building Sheepdog? It has a few advantages over other distributed block storage systems. The first is that it is open-source, unlike many of the competitors. The second is that hardware-based distributed block storage systems, like SANs, are very expensive and can be a single point of failure. The third reason is that other open-source distribute block storage systems (like CEPH and Lustre) are complex to setup and administer, and don't always have the performance characteristics expected.

Sheepdog is a fully-symmetric, reliable, scalable, and well-performing distributed block storage system. There is no central node, hence no single point of failure. Instead, blocks are distributed around the machines in a clock-like fashion, so the loss of a single (or in most cases 2) machine does not cause the data to be lost. Sheepdog is intimately tied into Qemu; it uses block layer hooks in Qemu to do its work. On the machine that is running the qemu/kvm process, there is a sheep server process that qemu talks to in order to read and write blocks. When a request for a block that is not on the local node is received, the sheepdog server process reaches out to the other sheepdog process running on other machines and requests the block that it is looking for. Because the data is broken up into 4MB chunks and distributed around the nodes, the performance should be comparable to other network-based storage systems (such as NFS), with much higher reliability.

Sheepdog currently uses the Corosync Engine to do cluster-like operations, which has advantages and disadvantages. The obvious advantage is that Corosync is a well-tested piece of cluster infrastructure that uses a well-known totem ring protocol to do synchronization across nodes. The disadvantage is that corosync is limited to around 48 total nodes. Moving to larger numbers of nodes will require developing a new protocol.

Cloud: The future of computing

I next went to Mike McGrath's talk on the future of cloud computing. The first part of his talk explained the advantages that businesses see in cloud computing (vs. traditional, hardware-hosted-on-premises computing); cost, scalability, and the ability to pay as you go. In this section he also talked about the differences between IaaS, PaaS, and SaaS. Infrastructure as a service (IaaS) is the idea of running virtual machines on hardware that you do not necessarily control. Amazon EC2 is the canonical example of IaaS. Platform as a service (PaaS) is the idea of running your application on a platform that is already prepared for you. Web hosting providers fall into this category. Finally, software as a service (SaaS) is the idea of being completely divorced from the maintenance of the system. Things like Google Docs fall into this category. He then demoed "Libra", an application that he is working on to quickly deploy PaaS applications.

The second part of his talk was how cloud computing is set to replace traditional computing, and how ill-prepared free software is to deal with that new future. In particular, his points are that:

  1. HTML5 is set to replace traditional desktop toolkits with a rich interface
  2. The new model of cloud computing encourages consumption, not contribution
  3. Duplicating a cloud computing model becomes easy (given enough hardware), so there is little incentive for companies to open-source their software stack
  4. Current licenses are not prepared for cloud computing
  5. The open source crowd will have to compromise to open standards, rather than open-source

While I agree with him on some points, I don't believe that cloud computing is going to completely take over the computing landscape. Computing goes through cycles from thin clients to thick clients, and back again. While cloud computing will almost certainly be a part of the IT landscape for a long time to come, it will be just that: a part. Not every application is suitable for cloud computing. Not every user wants to outsource their concerns to a third party.


The last session I went to before my own was about Pulp, which is a software repository management system. It can manage packages, errata, and distros, along with doing repository synchronization from a feed or custom repos.

The architecture looks like:

Essentially, there is a central pulp server that can be interfaced with via REST. It uses a mongodb to hold all of the metadata about packages, errata, etc, and can push (via AMQP) requests to consumer machines. The consumer machines run a small pulp daemon that continually monitors for changes, so if modifications are done outside of pulp, the main server can be notified. The current version of pulp supports modern Fedora along with RHEL-5 and RHEL-6.

When managing repositories with pulp, there are a number of operations available. Repositories can be created, deleted, or updated based on commands from the admin. The data in the repositories can be sync'ed immediately or at a fixed schedule, and the content can come from yum repositories, RHN feeds, or locally created repositories. It also supports global package search across all repos, which may be useful for finding out-of-date packages in the repositories.

The consumers need to subscribe (bind) to certain pulp repositories. Once they are subscribed, the repositories are available through yum, so the normal yum commands will work to install additional packages. The pulp server can do remote installation of packages through the pulp API if desired. As mentioned before, the daemon running on the consumer can keep track of packages that were installed locally, and update the pulp server with that information. Finally, any actions done on the consumer are extensively audited for debugging/security purposes.

What's the difference between Pulp and Spacewalk? Pulp is a much lighter version of Spacewalk, so it can be integrated separately. The hope is to break up Spacewalk into smaller, single-purpose components that are more re-usable (and hence have more users).

Some advanced features of pulp include:
  • Batch operations on groups of repositories and groups of consumers
  • LDAP integration
  • Repository Cloning, so you can start with the same repository but use it for multiple different purposes (think development vs. production)
  • A tagging system for consumers
  • Full REST API

The future roadmap for Pulp includes:
  • Maintenance Windows (do updates at particular dates/times)
  • Scheduled installation of packages on consumers
  • Support for external content delivery systems to reduce bandwidth
  • Clone filters to filter out what packages to clone
  • HA and Failover
  • Internationalization

Cloud Management
It was then time for my talk. I'm not a great presenter, so I was a bit nervous to start with, but I eventually got into a flow with my talk. I spoke about both Deltacloud and the newly launched Aeolus project, which uses Deltacloud underneath. The outline I used is available here, so I won't go into too much additional detail here. I will say that I had quite a few people come to my presentation, and I had some good questions, so I thought it was successful from that point of view. I'll record some of the questions that I remember below; they make a nice addendum to the material from the talk itself. Any mistakes are mine, since I am working off of my imperfect memory.

Q: Why use the deltacloud API over something like the EC2 API, which seems to becoming an industry standard?
A: The Deltacloud API is very useful right now, since there is no common API between cloud providers.  While some providers are indeed implementing the EC2 API, it is not ubiquitous.  If it turns out in the future that every cloud that we care about implements the EC2 API, then deltacloud could be viewed as superflous.  Until that point, however, we think that it has significant value in avoid cloud lock-in.

Q: What about making Oz more configurable, to allow users to upload their own kickstarts?
A: This is a great question, and the answer is that yes, we should make it more configurable. In response to this question, I implemented uploadable autoinstallation files which went into the Oz 0.1.0 release.

Aeolus Conductor
Q: Looking at the full diagram of Aeolus, it is a pretty complicated structure. How are Fedora users expected to install and configure this?
A: This is a valid question, and the answer is not fully formed yet. In order to solve many of the problems that Aeolus is trying to solve, we think the complexity of the internal components is needed. That being said, we really need to make it simple to deploy a simple Aeolus Conductor. To that end, we have a set of scripts called "aeolus-configure" that can install and configure a simple configuration. This certainly needs work, but it is one way in which we can make this more consumable to end-users.

After my talk, there was FUDPub which was a lot of fun. Not much to mention here, other than the combination of drinks, bowling, pool, and friends was fantastic. I would highly recommend doing something similar at future FUDCons.

Friday, February 4, 2011

Oz version 0.1.0

I've just sent the following mail to aeolus-devel@lists.fedorahosted.org and cloud@lists.fedoraproject.org, announcing the first release of the Oz tool:

I'm pleased to announce the first release of Oz, version 0.1.0. Oz is
a tool for doing automated installation of operating systems using the native
installation tools. While Oz is part of the Aeolus umbrella, and thus intended
to be used in a cloud management environment, it is also useful as a standalone
tool to do guest installation.

This first release focusses on providing the basic infrastructure to do guest
installs. There are 3 possible phases to a guest install (and all 3 are

1) Installation of a JEOS. Oz creates an automated installation file (a
kickstart, winnt.sif, preseed, etc), and then launches a KVM virtual machine
to do the installation. By default, a minimal JEOS is always installed to
reduce the likelihood of installation failure (though this behavior can be

2) Customization. Once a guest is installed from step 1) (or otherwise
provided), the virtual machine is launched and a native tool is used to do
installation of additional packages and files. For instance, in the case of
modern linux, ssh is used to upload files and run commands to install
additional packages.

3) Manifest generation. After the guest installation and customization are
done (if any), a package manifest (and some other metadata) can be extracted
from the image.

As mentioned, all 3 steps above are optional; you can start at step 1 and run
all the way to step 3 using oz, or you can install a guest using another
method and skip directly to steps 2 or 3. Documentation of how to do these
sorts of things are on the Oz webpage at http://aeolusproject.org/oz.html.

At present, Oz can install the following types of guests:
Fedora: 7, 8, 9, 10, 11, 12, 13, 14
Fedora Core: 1, 2, 3, 4, 5, 6
RHEL 2.1: GOLD, U2, U3, U4, U5, U6
Fedora Core: 1, 2, 3, 4, 5, 6
RHEL 2.1: GOLD, U2, U3, U4, U5, U6
RHEL 3: GOLD, U1, U2, U3, U4, U5, U6, U7, U8, U9
RHEL 4: GOLD, U1, U2, U3, U4, U5, U6, U7, U8
RHEL 5: GOLD, U1, U2, U3, U4, U5, U6
RHEL 6: 0
Ubuntu: 6.10, 7.04, 7.10, 8.04.[1,2,3,4], 8.10, 9.04, 9.10
Windows: 2000, XP, 2003
RHL: 7.0, 7.1, 7.2, 7.3, 8, 9
OpenSUSE: 11.1, 11.2, 11.3

Note that not all guest types can do all 3 of the steps above. The Oz webpage
lists which backends can do which phases.

Instructions on how to download Oz are at

As usual, comments, questions, bug reports, and patches are welcome for Oz.