FOSDEM 2009, Day 2

Notes for FOSDEM Day 2

Sunday, February 8
------------------

Solar Control with 1-wire open hardware
---------------------------------------
Speaker: Wookey
- Install solar thermal panels on the roof of the house to replace hot
water heater
- Solar panels get hot and do heat exchange, not electricity generation
- Roof panels cost around 1000 euros
- Use standard hot water tank
- Use thermosiphon
- Hot water is lighter than the cold water in the tank, so it comes up
a tube (without a pipe), and replaces hot water in the tank
- Can manage the system with a commercial controller, but costs 388 euros
- Instead, use a home-brew solution instead
- 500 MHz processor, 384MB memory
- I/O expansion bus with I2C, etc
- 1-wire sensors to do temperature detection
- Good to about 100 meters
- 1-wire bus is actually 3 wires - 1 data, 1 ground, 1 power
- 14 kb/s
- Multiplex 8 1-wire buses
- scan 13 devices/second
- measurement 94-750ms
- Hardware
- 5 wires - I2C data, I2CCiK, GND, +5V, IO0
- Sensors wrapped around pipes
- Sensors around tank
- Software
- Debian
- modprobe i2c-pxa, i2c-dev
- /dev/i2c-0, /dev/i2c-1
- 0 is general, 1 is for power
- I2C addresses are fixed when the board is soldered
- 8-bit addresses, but bottom bit specifies read/write (so 7 address
lines)
- i2cdump from lm-sensors or i2c-tools
- modprobe pcf8574a
- OWFS
- One Wire FS
- FUSE filesystem
- OWFS daemon
- Perl binding, tcl binding, shell
- Does temperature
- # owdir -s 4304
- # owread -s 4304 /28/temperature
- Logging data
- First cut - manual rrdtool
- rrdtool create
- rrdupdate to fill in the data
- rrdtool graph
- OK, but pretty manual. Also does all processing on embedded
board, which is very slow (floating point on ARM)
- Munin
- Hides rrdtool details
- Remote graphing; just fetches data from embedded board, does
processing offline
- Plugin system
- Munin ends up working well, nice graphs
- Control
- Interesting optimization problem; when to turn pumps on and off.
Better to run the pump continuously, or turn on and off rapidly?
- Current algorithm is simple; turn on pumps when tank is hotter than
heat exchange panel
- Reliability
- Original system uptime 87 days
- owserver crashed once
- Survived disk full condition
- Some 1-wire problems
- Reading data not always reliable
- When the sensor has problems, goes to 85C and stays there
- Unfortunate because 85C is actually a valid reading
- Nominal sensor accuracy is 0.5 C
- In practice 2C difference
- Can reach 0.5C in ideal conditions, though
- Lesson: need to clamp sensors very well to what you are measuring
- Bigger Project
- 2 tanks, 2 overflow
- RJ45 sensors
- Open hardware design for digitemp sensors
- Future?
- Reliability during power failure situations
- Local User Interface (LED, LCD)
- More information about solar tank temp, bath status
- House sensors to control heating, cooling
- Inputs - buttons to say "bath mode", "leaving house"
- Other Related Software
- DIYzoning
- Misterhouse
- temploggerd
- WT app

FreeIPA
-------
Speaker: Simo Sorce
- Free Integrity, Policy, Audit
- Goal is to make it simpler to manage all of the above
- Using standard protocols
- Target - system administrators
- Identity management is the realm of proprietary vendors
- Not fully free
- Need an open source solution for security and freedom
- Identity management problem
- Need single source for identity
- Single sign-on/single password
- Single data store for auditing and reporting
- Single point of management
- Implementation problems
- Synchronization and integration
- Windows and Unix
- Distribution of data and credentials
- Distributing change
- Single point of failure
- Integrating interfaces
- FreeIPA components
- Directory is LDAP
- Storage mechanism to perform fine-grained access control
- Organize identity and allow group relationships
- Distribute information across clients
- Replicate information on multiple servers
- Avoid single point of failure
- Chose LDAP since it is standard, extensible, flexible
- Authentication is Kerberos
- Single Sign On with delegation - carry on your credentials from
machine to machine, service to service
- Tested standard, validated secure
- Somewhat extensible - can add other authentication devices
(smartcards), and new encryption algorithms
- Once you introduce kerberos, then you need DNS and NTP
- Client and server must be within 5 minutes of each other to prevent
spoofed/expired tickets
- Host tickets based on DNS name, so client must have one
- Implementation
- Fedora Directory Server
- MIT Kerberos
- Apache + mod_nss, mod_auth_krb, mod_proxy
- Python and Turbogears
- Custom FDS plugins and CLI tools
- Clients can use nss_ldap, pam_krb5
- Avoiding synchronization problems?
- Kerberos information is in LDAP as well as KDC
- Directory structure
- Very flat structure
- Splits accounts from kerberos information in the directory
- Allows users to change some of their own information (based on ACLs)
- Management interfaces all revolve around the directory
- Web UI goes through mod_nss, mod_auth_krb, mod_proxy, GUI, xmlrpc,
then finally to LDAP
- CLI goes through mod_nss, mod_auth_krb, xmlrpc, and then LDAP
- But, installation was still complex
- ipa-server-install makes it much simpler
- Multiple servers
- Replication on directory server
- Because kerberos data in the directory, don't need to replicate
kerberos (kprop)
- 2 simple commands
- ipa-replica-prepare
- ipa-replica-install
- Future
- Add Audit Server - can use AMQP
- Add Certificate Authority
- Add Policy
- Makes interaction much more complex
- Luckily, most of the complexity is hidden from clients (and
administrators) in the FreeIPA core
- However, still a bunch of complexity, so new client agent
- System Security Services Daemon (SSSD) + IPA plugin
- caching, offline operations, etc.
- Host Based Access Control in LDAP
- Roles in LDAP
- New UI with plugin system
- DNS integration
- Dynamic updates
- Integration with Certificate Authorities

Syslinux
--------
Speaker: H. Peter Anvin
- Syslinux is a suite of bootloaders
- SYSLINUX - FAT
- PXELINUX - network PXE booting
- ISOLINUX - CD-ROMs - ISO9660
- EXTLINUX - ext2/ext3 - ext4 coming
- Only for x86/BIOS platform
- Core is in assembly
- Originally written for floppies, needed to be small
- Work underway to fix this
- Sophisticated menu systems
- Extensible via module API
- MEMDISK - disk emulator
- Emulates disk in memory
- Allows booting of legacy INT 13h OSes, mainly DOS
- Used for diagnostics
- gPXELINUX
- Collaborate with etherboot
- Enhanced capabilities for network booting
- http, ftp, nfs, AoE, iSCSI (in addition to TFTP)
- isohybrid
- Allows ISOLINUX .iso to boot from USB stick
- x86 ancient
- Released in 1981; IBM AT 1986; PS/2 1987
- Most BIOS interfaces date from this era
- Boot from floppy and hard disk only - 510 bytes
- 1993 El Torito booting (CD)
- 2 modes
- Disk image on disk, boot disk image
- Native mode, access whole CD - not widely supported until late 90's
- 1997 PXE
- Original PXE were problematic
- Nowadays, works pretty well/standard
- USB drives
- Lots of bugs still
- Tricks can help
- Syslinux history
- 1994 original syslinux implementation
- Designed for boot floppies
- Small to fit, so assembly
- Take DOS OS, make Linux boot floppy
- Add online help support
- 1999 PXELINUX support
- PXE only allows 32K for Network Boot Program (NBE)
- Re-use SYSLINUX
- 2001 ISOLINUX
- 2004 EXTLINUX - ext2/ext3 general purpose loader
- 2006 graphical menu system
- 2008 gPXELINUX, ISOLINUX "hybrid" support (iso that also works on
USB key)
- What's good?
- Designed for dynamic systems
- System discovery at boot time
- Keep to PC established principles
- Sophisticated user interface
- Problems
- Large core of assembly
- x86/BIOS only (because of assembly)
- Dynamic discovery comes at a price
- Can't read a kernel from another disk
- gPXELINUX
- gPXE plus PXELINUX in one image
- gPXE contains an extended PXE interface for PXELINUX
- Can http, ftp, nfs, AoE, iSCSI, SFP (in addition to TFTP)
- Drop-in replacement for pxelinux.0
- Needs TFTP server for initial bootstrap
- But can be skipped if NIC ROM can be reflashed
- Syslinux module API (COM32)
- Small C library (klibc)
- Similar to normal userspace C code
- Main limitation is only sequential, readonly file access
- Common modules
- UI
- Complex menu system - does everything
- Simple menu system - what most people use
- Graphics library makes the same code work for graphics, text, or
serial
- File format modules
- Loading new types of loadable binary objects
- Module describes where various bits go in memory, then syslinux
"shuffle" library puts everything in place
- Shuffle library computes a set of move operations to put things in
the right place in memory
- Example: Microsoft SDI format
- Boot WinPE with syslinux - Windows kernel + ramdisk
- 199 lines total; 139 non-blank/comment lines
- Policy modules
- Example policy:
- Boot kernel X on 64-bit machine
- Boot kernel Y on 32-bit with PAE
- Boot kernel Z otherwise
- Module is 129 lines long; 70 non-blank/comment lines
- Diagnostic modules
- Modules available to show hardware/BIOS state
- e.g. pcidump
- Move dumping data into libraries, and make available to modules
- Already code in Syslinux
- Probe PCI bus
- Map devices to modules
- Build an initramfs with the modules you need
- But...boot devices on USB, firewire? Harder to discover, because you
need a driver for the bus itself
- Ongoing work
- Lua interpreter to write policies
- readdir support - right now, no directories
- Move filesystem code out of assembly - needed to support advanced
filesystems like btrfs
- Switch core from assembly to make it portable; required for EFI
- Core components
- First stage loader, disk/network I/O, BIOS extender (protected mode),
shuffle system (rearranges RAM) need to remain in assembly
- Everything else (command-line, config parser, kernel loader/parser,
filesystem drivers) should be written in C
- Syslinux needs help
- Too large for one person side project
- Significant number of regular contributors
- Documentation help
- http://syslinux.zytor.com

Ext4
----
Speaker: Theodore T'so
- What's good about ext3?
- Most widely use linux filesystem
- People trust it
- Diverse developer community
- important because distros need to understand it well enough to
be comfortable supporting it
- What's not good about ext3
- 16TB filesystem size limitation (32-bit block numbers)
- 32000 limit on subdirectories
- Second resolution timestamps
- Performance limitation
- Traditionally cared more about data integrity than performance
- Is ext4 a new filesystem?
- ext in ext2/3/4 stands for "extended"
- Collection of new features that you can individually enable
- ext4 driver supports all features
- But can have ext3 fs and mount as ext4, which works just fine
- 2.6.29 can mount an ext2 fs with ext4 driver (code from Google)
- Google doesn't care about journals, so can mount ext4 without journal
- ext4 fork was to make sure that ext3 remained stable
- Started with ext3, added new code
- e2fsprogs supports ext2, ext3, ext4
- New features
- Extents instead of indirect blocks - most important
- Delayed allocation
- Multiblock allocation
- Persistent allocation
- Subsecond timestamps
- Greater than 32000 subdirs
- NFSv4 version id's for caching (reliable caching)
- Store file sizes in FS block size, rather than 512 sectors
- Allows huge files; 16TB files on 4k block filesystems
- ATA TRIM support - when deleting a file, can tell block device to use
for something else
- Journal and group descriptor checksums - reliably put which part of
inode table is in use (speeds up fsck)
- Ext2/3 indirect block map
- In the inode, room for 15 block pointers
- First 12 map direct blocks
- File less than 12 blocks long (48k on 4k fs), location of blocks
stored in first inode
- Bigger files allocate indirect block
- Slot 12 holds address of indirect block
- 256 blocks
- Slot 13 is double indirect block
- 256 indirect blocks
- 256 direct blocks
- slot 14 is triple indirect block
- 256 double indirect
- 256 indirect block
- 256 direct blocks
- Inefficient for large files
- Long time to delete because has to read all indirect blocks and
free block pointers in all blocks
- Ext4
- Extents
- Extents are efficient way to represent large file
- Extent is a single descriptor for a range of contiguous blocks
- Logical 0 block, length 1000 blocks, physical 200
- 12 bytes ext4_extent structure (from ClusterFS)
- Address 1EB fs (48 bit physical block number)
- Address 16TB file size (32-bit logical block number)
- Max extent 128MB (16 bit extent length)
- Up to 3 extents stored in inode directly
- Vast majority of files (99%) live in the inode
- For greater than 3 extents, convert to B-tree
- Block allocator changes
- Extents works best if files are contiguous
- Delayed allocation and allocating multiple blocks at a time makes this
much more likely
- Block allocator looks at disk to try to find free space to fit number
of blocks we want to allocate
- Makes fs more resistant to fragmentation
- Responsible for most of ext4's performance improvements
- Problem: ext3 journal mode semantics means that data is written
before inode
- Avoids security concerns
- Application programmers came to depend on this
- ext4 does this differently, because of delayed allocation, we don't
push out to disk until page allocator (30 seconds)
- And staged, so it may take a while
- In laptop mode, 2-5 minutes to write to disk
- POSIX allows this
- Open question: fsync? fdatasync?
- Persistent pre-allocation
- Useful for databases and video
- Pre-allocate 1GB on disk ahead of time, contiguousally
- Useful for package updates (rpm, deb), reduce fragmentation
- Useful for file grown by append (like logfiles); can pre-allocate
space, and then the logfile contiguous
- Available via posix_fallocate, but...
- On older FS's, will just write lots of blocks of 0 (very slow)
- Changes i_size field, meaning that it's size (as reported by stat)
looks much bigger than it actually is currently using
- Need glibc direct access to Linux system call
- Avoid i_size change
- Fail on old FS's
- e2fsck Performance
- Not explicitly engineered
- But huge improvements
- Fewer extent tree blocks to read instead of indirect blocks
- Uninitialized block groups means don't have to read portions of the
inode table

FOSDEM 2009, Day 1

Since I live in Europe now, and especially since I live about 20 minutes by train from Brussels, I decided to go to FOSDEM 2009 this year. This was good for me; I actually haven't been to a conference since 2002, so it was interesting to see how this was run. It was also good to see what is going on in the Free Software world outside of my little sphere, and maybe get me excited in some new projects. I'll write two posts with my notes, one from the first day (Saturday, February 7), and a second one about Sunday, February 8th. The notes are a little dry, so I might consider doing a third post with my overall thoughts.

Saturday, Feb 7
---------------

Relicensing SunRPC code
------------------
Speaker: Simon Phipps, Sun
Problems of success:
- Some old code in Linux
- Open Network Computing Remote Procedure Call - 29 years old
- Most liberal license 29 years ago
- Use the code, but no profit
- Debian noticed - asked Sun to re-license
- Difficult to re-license because it is so old
- As of a couple of weeks ago, got permission to re-license

Key Note
------------
Speaker: Mark Surman, Mozilla
free. open. future?
- What is Mozilla for?
- Common: build Firefox, Thunderbird, Mozilla
- Mission statement
- guard the open nature of the Internet
- build open-source software
- promote innovation
- Roadmap for 2010
- openness and participation
- data safer and more useful
- mobile
- firefox
- Roadmap for 2060
- openness and participation
- ??
- How far can free and open go?
- 2003
- Web in danger
- IE 98% share - monopoly
- online apps only for IE
- freedoms matter
- now - IE < 70%
- letting consumers change their experience, let's users "hack"
- conceptual map of free software can go far
- What's next?
- 2009 - big for free software and mobile
- mobile battle
- how can open win?
- hardware (like OpenMoko)
- software
- web
- network
- cloud
- pricing
- permissions
- Conclusions
- Need better conceptual map for the mobile space and beyond
- Strong values, freedom beyond just code
- free software, that people want to use
- Users as hackers, anyone can bend anything
- What else?
- pick something in the map
- open source and education (build into university curriculum)
- Franklin street statement
- mapping freedom to web services

Debian
------
Speaker: Bdale Garbee
- Contributor to Debian since 1994
- Debian Project
- Association of individuals
- Why does it matter?
- Freedom
- Stable, functional community despite appearances
- External appearance sometimes skewed because of a vocal minority
- Large number of architectures and packages
- Uniform process for quality testing, bug reporting, open to
contributions
- Many derivative (downstream) distributions
- History
- Started by Ian Murdock
- 1993 - 0.01
- 1994 - 0.91
- 1995 - 0.93rc5
- 1997 - Debian Social Contract
- 1998 - 2.0 released
- Debian Linux Manifesto
- Developed in the spirit of Linux and GNU
- Promise to put distribution together and maintain it
- Design process is open to ensure high quality
- Linux is not a commercial product, and never will be one, but can
compete with commercial products
(this doesn't make much sense nowadays)
- Debian Social Contract
- 100% free
- Give back to the communities
- Don't hide problems
- Prioritize users and free software
- Public Bug Tracker
- First uniform way to report bugs
- Since 1994
- Open system, anyone can open or close bugs
- Email in, web out
- Policy Manual
- Set of standards
- Tools that assist in packaging
- Constitution
- Organizational Structure
- Division of power
- Individual developers have the most power
- Observations
- Values before Vision before Strategy before Objectives
- Internal social contract potentially as useful as an external one
- Future
- Lenny
- No radical changes
- Attract new contributors
- HP
- stays connected to Free Software through Debian

OpenWRT: UCI and beyond
-------------
Speaker: John Crispin and Felix Fietkau
- UCI: Unified Configuration Interface
- Designed originally for OpenWRT
- Simple configuration system to cover 90% of cases
- Human readable/writable
- Based on typed sections, option/value pairs, and lists
- First implementation in awk
- SNMP access available
- API for C, shell, Lua
- Used (and is requirement) for all OpenWRT base packages
- LuCI
- MVC (Model, View, Controller) webapp
- OpenWRT configuration interface
- Validation framework
- Data storage
- Configuration split by packages, stored in /etc/config
- Can access through UCI pointers
- e.g. uci get network.lan.ipaddr
- Changes stored, then committed/reverted
- If you make a mistake, can just reboot the appliance to get back old
configuration
- Other backends planned (databases, etc)
- Overlays supported (commit/revert example above is example)
- API
- Shell access (uci command)
- C
- Simple wrapper
- API with limited functionality
- Regular C API
- Direct access to data structures
- libucimap
- Automatically converts configuration files to/from C data structure
- Lua binding
- Efficient scripting
- What's next?
- More application support
- Automatically generated configuration interfaces based on config files
- Remote backends
- Locking for simultaneous access

Amarok
------
Speaker: Sven Krohlas, Ian Monroe, Lydia Pintscher
- Amarok 2.0
- Context information for music - central, moved to middle in Amarok 2
- Plasma based
- New playlist on right-side
- Lots of different layouts possible
- Lots of new drag-n-drop functionality
- Podcasts from BBC
- Jamendo integration - CC licensed music
- Audiobook integration
- NPR plugin
- Radio station from friends?
- Amarok 1.4 used DCOM (interprocess communication)
- Amarok 2.0 Javascript implementation called qtscript
- Javascript QT binding
- Script console
- QT API, classes relevant to Amarok
- Javascript language is OK for development
- Integrating services into Amarok
- Scripting interface is powerful
- Lyric scripts
- Service
- Example
- Fetches lyrics - charts with only free music
- Small XML file describing data
- Spec file, tells icon, author, comment, service
- Code file
- Levels (display levels)
- Downloader class

Fedora
------
Speaker: Max Spevack
- Personal
- 4.5 years at RedHat
- Manage RedHat's global community team
- Goal of Fedora?
- Increase open source development community
- Potential population is inverse pyramid
- Top-level is users
- Next level will search and solve their problem (help themselves)
- Next level turns the user into a participant of a community (help others)
- The last gap is difficult - Fedora wants to focus there
- Fedora Four Foundations
- Freedom
- Strong commitment
- RedHat building business around community
- All the code is licensed in an OSI approved license
- 2/3 packages are maintained by community
- Release Engineering processes to make community engaged
- Distribution composition - pungi has a nice API - easy to use
- First
- Innovative community
- Examples
- plymouth - uses KMS
- nouveau driver
- virtualization
- Pulseaudio
- Friends
- Fedora Infrastructure team
- Entirely volunteer driven
- Open
- System Admin team
- New contributors
- Fedora Scholarship
- Active in open source, starting at university
- Features
- Improve Fedora QA
- Getting stuff vs. building capacity to allow people to do things
- Too much time getting the distro done, not enough time maintaining
the community
- Fedora Test Days
- Fedora 11
- 20 second startup
- DeviceKit alpha
- ext4
- DeltaRPM
- MinGW cross-compile
- Python 2.6

Hacking with Modular Hardware: the BUG
--------------------------------------
Speaker: Ken Gilmer
- Bug labs
- BUG
- Electronic building blocks for personalized hardware devices
- Examples
- GPS alarm clock (wake up based on location instead of time)
- Stereo camera system
- Hardware is all open
- Lower barriers to innovation
- Use Linux, Java, OSGi
- Base Unit
- ARM CPU, 128MB RAM, 2GB SD
- LCD
- Buttons - software programmable
- 2.6.27 kernel
- Basic interface
- Battery
- Modules available
- Accelerometer
- Touch screen LCD
- Wifi
- Audio
- von Hippel - All hardware interfaces exposed
- Poky Linux
- Includes Gnome Mobil
- Can put custom images
- OSGi
- Standard to disconnect services from implementation
- Specification for components to talk to each other
- Notifies applications when modules are added/removed
- Abstract services
- Different modules can provide the same service
- Dragonfly SDK
- Like Eclipse
- Website to publish applications
- OpenEmbedded tools for Eclipse
- BitBake Commander for building images

Intel's Graphics Projects for the Year
--------------------------------------
Speaker: Eric Anholt
- Lots of good progress, but lots of problems
- vblank broken
- No GEM support
- OpenGL 2.0, vertex shaders don't work
- High end applications are slow with GEM
- vblank
- Applications want to wait for vblank
- Copy back to front
- Copy part of back to front
- GL gives you
- Wait for n vblanks
- X gives you nothing
- Which head should we vblank?
- GL doesn't know which head to vblank either
- What happens when one of the heads turns off?
- Too slow
- Doing it right
- Schedule vblank swap ioctl
- Cancel vblank swap ioctl
- GLSL
- Mesa GLSL is really a proof of concept
- Written by a graduate student
- Mesa GLSL only implements part of GLSL
- Mesa GLSL lacks optimization
- Mesa GLSL IR is a poor mapping onto current hardware
- Written against 3 year old hardware
- New compiler needed
- In progress
- Implements all of GLSL
- New IR for modern hardware
- Designed for optimization
- Pull some Gallium ideas into Mesa (incrementally)
- Kernel Mode Setting
- Works today if you are lucky
- Slow performance in the console
- Detect hotplug interrupts
- Media
- MPEG2 support implemented today
- More codecs in progress
- Convert XVMC to GEM
- Memory Management
- GEM is good
- Performance issues on older hardware
- Flushing is painful for large applications
- Can do better by keeping objects pinned

Augeas
------
Speaker: Raphael Pinson
- Configuration management
- Sitewide configuration
- LDAP
- Database
- Puppet
- Local configuration
- Editing configuration
- Most configuration files in Linux are text files
- Easy to edit with an editor
- Human edits can conflict with graphical tool editing
- Approaches to editing
- Keyhole - one-off, like sed, awk, etc.
- Greenfield - configuration from a database, centrally managed
- Templating - configuration from database, fill in machine specific
details
- Missing pieces
- Handle configuration data directly
- Policy delegation
- Remotable API
- Design goals
- Deal with configuration data in current place
- Expose an abstract tree view of configuration data
- Preserve "unimportant" data (comments, etc)
- Describe new formats easily and safely
- Language neutral implementation
- Focus on configuration editing, and not interpretation
- Bidirectional languages
- Transform a configuration file into a tree via a "lens"
- Modify tree
- Write the tree back to the configuration file via the "lens"
- API
- init/close
- get/set
- match
- insert
- rm (a subtree)