Managing FOSS for Business Results

By CJ Fearnley

I attended DebConf10 in early August at Columbia University in New York City. I thought I’d document some of the initiatives that resulted from that event.

I attended the Debian Policy BoF. It inspired me to start reviewing Debian Policy and led to my submission of a bug report to improve the description of the archive areas in Debian. In the BoF (Birds of a Feather) session, Russ Allbery requested help from everyone including non DDs (Debian Developers) to make policy clearer and more descriptive, so I obliged. I see that the negotiated text has been accepted and included in the git repository for the next release.

During the Debian Science sessions that I attended (see especially the video from the Debian Science Round Table), the idea of engaging upstream providers (software projects that are packaged by integrators like Debian) more effectively was discussed. This led me to draft a proposal that I posted to the Debian Project mailing list: Improving coordination / support for upstreams. There was precious little feedback, but I think it is a profoundly important issue: how do we improve coordination of upstreams with projects like Debian that integrate their software. So far there is very little infrastructure or knowledge about this important issue in the management of FOSS (Free and Open Source Software). How do you think we should start to address this problem?

I helped with the herculean task of getting Sage, a FOSS mathematics system, back into Debian. After discussing the issue during the Debian Science track at DebConf10, I met Lucas Nussbaum in the hacklab and he (with help from Luca Falavigna) managed to get the old buggy version of Sage removed from unstable (apparently, this version was causing support issues for the upstream Sage community, so this was a positive step forward). I also submitted five bugs (#592349, #592354, #592425, #592426, and #592429) about new upstream versions that are needed. I posted two detailed reports of work needed on packaging Sage for Debian to appropriate mailing lists. Getting Sage into Debian is the kind of big FOSS management challenge that I’m excited about. But I will need a lot of help to make progress. Let me know if you are interested in contributing to the effort!

Finally, for the record, here are links to the sessions where I participated in the discussion (video is available): Bits from the DPL, SPI BOF, Enterprise Infrastructure BOF, Mathematical Software in Debian, Overall presentation of Debian Science, and Debian Science Round Table.

By CJ Fearnley

The November 2009 issue of Communications of the ACM (CACM) has a very interesting article by Paul Stachour and David Collier-Brown entitled “You Don’t Know Jack About Software Maintenance”. The authors argue energetically for using versioned data structures and “continuous upgrading” to improve the state of the art of software maintenance.

The piece got me thinking about FOSS (Free and Open Source Software) and “continuous upgrading”. Here are seven observations on FOSS software maintenance that occurred to me as I reflected on the CACM article:

1. FOSS projects “continuously” apply bug fixes and feature enhancements at no additional cost to their users. By applying these improvements “continuously”, the user reaps a steady stream of “interest payments” providing ever-improving security, performance, and functionality.
2. Since FOSS incurs no licensing or license management costs, upgrading FOSS is not hindered by capital expenses.
3. Typically support in FOSS projects is focused on the current stable version. Therefore, upgrading to the current stable version is the preferred way to receive the best support from FOSS communities.
4. One of the key reasons behind Debian‘s strong track record of “continuous upgrading” is its way of handling the tricky issues involved with dependent library upgrades (such as libc6, libssl.so.0.9.8, & etc). The chapter on Shared Libraries in the Debian Policy Manual details a proven method to effectively handle library upgrade issues (including its sophisticated handling of versions).
5. When upgrading is applied routinely and “continuously”, it becomes crucial to support customizations across upgrades which can be one of the biggest obstacles to a smooth upgrade (see my earlier post on customization and upgradeability). One reason for Debian’s effectiveness in this regard is its robust configuration file handling policy.
6. It is worth noting that the “continuous” implied here is not the one emphasized in dictionaries (which takes its nuances from the mathematical / physics concept of “no interruptions” and the epsilon-delta definition that students of Calculus learn). That concept of “continuous” is impossible in systems administration which is necessarily discrete as are all computer operations. The connotation required here is, perhaps, “unending”, or “eternal” or somesuch.
7. The “right” frequency for “continuous” upgrades is a complex tradeoff between business requirements and upgrade infrastructure maturity. Debian and Ubuntu provide vary mature support for “continuous upgrading”. They support the upgrade of production servers through release after release after major release with minimal downtime or risk of a glitch that could affect users. Their current release frequency of about 2 years may be the best we can do given the current state of the art of software maintenance. I hope we can learn to increase the frequency as better engineered upgrade policies are developed.

I prefer the name “eternally regenerative software administration” over “continuous upgrading”. It avoids the philosophical problems with the word “continuous” and emphasizes the active, “ecological” approach needed to envision the engineering of “regenerativity” in software. By that I mean software maintenance should involve building the system so each new version enables installation of the next while facilitating management of any customizations and integration with other software (including libraries and other “helper” applications). Regenerativity is the process of growth and change used by Nature itself. Software maintenance needs to follow similar principles.

By CJ Fearnley

Mary Hayes Weier wrote an interesting article in this week’s edition of InformationWeek on "Alternative IT: CIOs are more receptive than ever to new software models". What is great about her article is how she captured the divergent views on IT models (such as SaaS, cloud computing, etc.) and gave nice vignettes of different organizations trying different parts of various models. I especially valued her use of cognitive dissonance to leave the reader thinking … better informed but without a firm conclusion.

There are so many parts of the article that I could blog about, but the one that touched the core of my thinking about “eternally regenerative software administration” was the quote by Bill Louv, CIO at GlaxoSmithKline, who said

"And here’s the rub: When you customize software, it’s difficult to implement future upgrades from the vendor"

Louv touched the very bane of eternally regenerative software administration! Software should accommodate both customization and upgradeability: these two elements of software administration are at the heart of my notion of eternally regenerative software administration: how to preserve customizations and provide smooth (near zero downtime with almost no glitches) upgrades through major release after major release. It is a big challenge, but in our experience the Free and Open Source Software (FOSS) communities are at the leading edge in finding solutions to these conflicting objectives. Here are some of the innovative ideas from the FOSS world which should serve as models or design patterns for all software developers (if only these ideas would become commonplace!).

First, Debian (a FOSS operating system which is the root of Ubuntu, Knoppix, Xandros and many other Linux distributions) requires that their official packages, a collection of software prepared for easy administration, must adhere to a very mature policy. Debian’s policy is a marvel in the FOSS world and to a very large degree is responsible for its strong support for both customization and upgradeability. I think Debian’s reputation for stability and maintainability is almost certainly due to their decision to develop a consensus-driven policy that its software must implement.

For example, the Debian package maintainer, Luigi Gangitano, for Drupal, a FOSS content management platform, did a great job making the software both customizable and maintainable. The package supports configuration of multiple virtual hosts which can all be upgraded at once! And the Debian drupal6 package stores the look-n-feel in /etc/drupal/6/themes/ so that each site’s GUI can be customized without interfering with upgrades. If only all web applications were built to be as maintainable as Debian’s Drupal package!

Another example is the overlay support included in RT: Request Tracker, a FOSS ticket tracking system. This allows putting replacement subroutines in special files in /usr/local/share/ which overlay or substitute the upstream code. This approach is more likely to break on upgrades, but it supports minimal changes to the business logic with a decent chance that upgrades will be smooth.

There are countless more examples from the FOSS world of innovative solutions to inter-accommodate customization and upgrades in support of eternally regenerative software administration. What are some of your favorite examples?