Thursday, January 22, 2009

Stupid Git tricks

I've been using git somewhat for a couple of years now. Until recently, though, I was only a very casual user, and I didn't know the full power of git. I'm now maintaining a git tree at work, and I'm having to learn some of the more powerful features of git. This may be an ongoing thing as I learn more about it, but for now, here are some of the commands that I've had to learn, and that are extremely useful:
  1. git checkout -b
    This is really at the core of git, so it's almost not worth talking about, but I'll start here. This creates a new branch off of the branch you are currently working on, and changes over to that new branch. git is all about working with branches, so the recommended way to do things is to have many "topic" branches; one branch for each different topic you are working on or with. Then later on it's easy to merge branches together, push and pull them remotely, etc.
  2. git commit -a -s
    Commit any outstanding changes to the current branch. This is pretty self-explanatory, except for the fact that -s adds your Signed-off-by line automatically.
  3. git rebase -i HEAD~6
    This is somewhat of a baroque command, but it's so powerful, you'll wonder how you ever lived without it. A "rebase" tells git to checkout some previous version (in this case, HEAD - 6 commits), then replay thte commits on top of that. Where this gets really interesting, however, is with the -i flag; this is an interactive rebase. This allows you to do various operations to the individual commits before they are replayed. Your three options are pick, edit, or squash. "Pick" just means to take the commit as-is. "Edit" means that you want to edit the commit in some way before replaying it. This can be as simple as editing the commit message, or as complex as adding new code into the commit. "Squash" means to take this commit, and merge it into the previous one, so they now look like one commit.
  4. git add --patch
    I actually haven't personally used this one yet, but it was pointed out to me by a co-worker. git add is the command you use when you want to add some changes to a changeset before it is committed. The "--patch" flag allows you to choose just specific hunks of the differences that git finds, so if you have some debugging or whatever left in your tree, you can just automatically throw it away. Very cool.
  5. git merge
    This is a command that lets you merge multiple branches into your current branch. It tries very hard to do automatic conflict resolution; if it has to give up, it leaves you in a place where you can fix up thte conflict by hand, and then continue the merge.
  6. git pull remote_branch local_branch
    This command lets you pull *any* remote branch onto *any* local branch. That means someone can point you to their private repository, and you can pull their changes onto your branch locally. Very handy for combining trees.

Monday, January 19, 2009

KVM performance tools

I've recently been working on tracking down some performance problems with KVM guests. There are a few tools available, but in this entry I'll stick to the two I've found most useful recently: kvm_stat and kvmtrace.

Let me start with kvm_stat, since that one is far easier to work with. kvm_stat is basically a python script that periodically collects statistics from the kernel about what KVM is up to. Unfortunately, it is not packaged in the Fedora kvm package (I should probably file a bug about this), but the good news is that it is very easy to get. You just need to check out the kvm-userspace git repository (git clone git://, and the script is at the top-level of the directory.

To get kvm_stat to actually do something useful, you first have to mount debugfs. In all likelihood, your distro kernel has this turned on, so all you really have to do is:
mount -t debugfs none /sys/kernel/debug
If you are going to do this often enough, it's probably a good idea to add that to your /etc/fstab.

Once you have debugfs mounted, you can now use kvm_stat. If you just run "kvm_stat", it periodically outputs columns of data, sort of similar to vmstat. I've found this kind of hard to look at, though. So what I've been doing instead is using the -l flag of kvm_stat, and piping the output to a text file:
kvm_stat -l >& /tmp/output
The -l flag puts kvm_stat into logging mode, which is harder to read on a second-to-second basis, but easier to read in a spreadsheet later. After I've collected data for a while (mostly during the tests I care about), I use OpenOffice to read that data into a spreadsheet (hint: use "space" as a delimiter, and tell it to "Merge Delimiters"). Now, it's fairly easy to see what your guest has been doing, from the host's POV. Note that these are cumulative numbers; if you have multiple guests running, this is all of the data from all of the guests.

There are quite a few fields that kvm_stat outputs; I'll talk about the ones I think are relevant:
  • exits - I *think* this is a combined count of all of the VMEXIT's that happened during this time period. Useful number to start with.
  • fpu_reload - The number of times a VMENTRY had to reload the FPU state (this only happens if your guest is using floating point)
  • halt_exit - This is the number of times that the guest exited due to calling "halt" (presumably because it had no work to do)
  • halt_wake - This is the number of times it was woken up from halt (it should be roughly equivalent to halt_exit)
  • host_state_reload - This is an interesting field. It counts the number of times we had to do a full reload of the host state (as opposed to the guest state). From what I can tell, this gets incremented mostly when a guest goes to read an MSR, or when we are first setting up MSR's.
  • insn_emulation - The number of instructions that the host emulated on behalf of the guest. Certain instructions (especially things like writes to MSR's, changes to page tables, etc) are trapped by the host, checked for validity, and emulated.
  • io_exits - The number of times the guest exited because it was writing to an I/O port
  • irq_exits - The number of times the guest exited because an external irq fired
  • irq_injections - The number of IRQ's "delivered" to the guest
  • mmio_exits - The number of times the guest exited for MMIO. Note that under KVM, mmio is much slower than a normal I/O exit (inb, outb), so this can make a significant difference
  • tlb_flush - The number of tlb_flush's that the guest performed.

The other tool I've started to use is kvmtrace. This tool does generally the same as the kvm_stat tool, but it does it at a much finer granualarity. From the output, you can see not only that it did a VMEXIT, but also that it did a VMEXIT because of an APIC_ACCESS (or whatever). This can be powerful, but it also generates a lot more data to sift through.

Using this tool is a little more complicated than the kvm_stat one. Luckily, it is packaged in the Fedora kvm RPM, so that part we get for free. To run this beast, you'll want to do something like:
kvmtrace -D outdir -o myguest
What this does is to tell that you want all output files to go to "outdir", and have them named "myguest.kvmtrace.?". You'll get one file for each CPU on the system. The last statement is actually quite important; generally, your best bet is going to be to pin the guest to a particular CPU on the host, so that your results don't span across multiple CPUs. Now, this is the raw, binary data for each CPU on the system. You next need to convert that into something that a human can look at. For this job, there is kvmtrace_format. You can do all kinds of clever things with kvmtrace_format, but what I've found the easiest so far is to use the "default" format file (which generates all events), and then dump that out to a file. So, for instance, I ran:
kvmtrace_format user/formats <> myguest-kvmtrace.out
Note that user/formats is from kvm-userspace at git:// (again, it's not in the Fedora kvm package, which I should probably file a bug about). That ends up dumping all of the output to myguest-kvmtrace.out, which turns it into a *huge* file. From here, I just did a bunch of processing with sed, grep, and awk to look for things that I care about.

Friday, January 16, 2009

Remote Sane

Well, this is going to be my occasional blog on technical things. Hopefully I'll keep up with it, although I'm notoriously bad for doing so. In any case, here's the first one.

Recently Jen asked me to set-up scanning at home. We have an HP PSC 1510 All-in-One Printer/Scanner/Copier. It's currently hooked to my main router via USB. Now, I could easily have done all of the scanning from the main router via the "scanimage" command. But I really wanted to be able to scan from the other computers on my network, including Jen's Windows laptop. Enter saned, a small daemon that allows remote clients to access your scanner. However, getting this working had it's share of pitfalls.

Server Configuration

To start with, my main router is a Fedora 9 i386 box. The first order of business, of course, was to get scanning working locally. Luckily this is pretty easy nowadays; I just needed to install:


A quick "scanimage -L" shows that the scanner was detected, and a quick "scanimage > image.pnm" shows that it actually scans things. Great, the first part is over.

For the next part, I needed to setup saned. While saned is packaged with the sane-backends package, it is unfortunately not well integrated into the system. To get it working, I basically had to add it to xinetd, add the IP addresses I wanted to be able to scan from, add a new user, and finally make sure that when the device was plugged in, it got set to the appropriate user. Those steps are described below; note that I had a lot of help from the following web pages:

The first order of business is to get xinetd to start up saned. This is accomplished by adding a new file /etc/xinetd.d/saned, which looks like this:
service sane-port
socket_type = stream
server = /usr/sbin/saned
protocol = tcp
user = saned
group = saned
wait = no
disable = no
Then I did a quick restart of xinetd via "service xinetd restart". Next, we have to make sure that the "saned" user exists on the system; I did this with:
useradd -M -s /sbin/nologin -r -u 491 saned
Which creates a new user without a home directory, with no login shell, and with UID 491. With that in place, I was able to test what I have going so far with a quick "telnet 6566". If this doesn't spit any errors, then we at least have the daemon up and running.

With that in place, the daemon is running, but the problem is that the permissions on the USB device won't allow the saned user to actually access it. This is where things get tricky; in order for this to work, we really want to make sure the USB device is owned by saned *every* time it gets plugged in (or powered up). In modern Linux, including Fedora, the way to do this is with a custom udev rule. In my case, I created the file /etc/udev/rules.d/10-saned.rules, and put the following in:
ACTION=="add", SUBSYSTEM=="usb", NAME="bus/usb/$env{BUSNUM}/$env{DEVNUM}", MODE="0664", OWNER="saned", GROUP="saned"
What this says is that for any add action on the usb subsystem, change the owner to saned and the group to saned. Now, this rule isn't as refined as it could be; in particular, I probably only want this rule to run when the device being plugged in is in fact my USB scanner.

After adding in the above udev rule, I just had to power off my scanner, wait a few seconds, and then power it back up. Once I did that, I had to check that the /dev device was owned by the right user. To do that, I first had to find out which USB device my printer shows up as. That's easily accomplished by a quick "lsusb", and then looking for the Bus and Device number. In my case, that was Bus 002 and Device 009, so I just did a quick "ls -l /dev/bus/usb/002/009", and ensured that that /dev node was owned by saned:saned.

Finally, I had to add which clients I wanted to be allowed to scan over the network. To do that, I just added:
to /etc/sane.d/saned.conf.

Client Configuration

Now that I had the server all set up, it was time to try out my remote clients. The first thing to try was obviously my Linux laptop. This was actually a breeze; I just had to add the internal name of my server to /etc/sane.d/net.conf, and then xsane finds the scanner when it starts up.

Windows was a bit harder, but not much. Basically I went to and downloaded the SaneTwain bridge there. After unzipping that package, I was able to start up the example executable and add my saned machine as a remote source, and it all worked!