Managing FOSS for Business Results

By CJ Fearnley

Since I’ve been involved with Debian GNU/Linux for over 15 years, it is exciting that I will be able to attend the first two and a half days of DebConf10 including Debian Day from Sunday to Tuesday August 1–3.

I am particularly looking forward to the following sessions: Pedagogical Freedom: Debian, Free Software, and Education, Beyond Sharing: Open Source Design What are the challenges for the collaborative design process?, FLOSS Manuals: A Vibrant Community for Documentation Development, Bits from the DPL, The Java Packaging Nightmare, Collaboration between Ubuntu and Debian, How We Can Be the Silver Lining of the Cloud, Enterprise Infrastructure BOF How enterprise technologies such as Kerberos, LDAP, Samba, etc can work better together in Debian, Using Debian for Enterprise Infrastructure Stanford University: A Case Study, and more (see the schedule for each day).

I’m also hoping I can also attend on Thursday when the math and science focused sessions will be held, but I’ll have to see how next week’s schedule works out in the office. If you are coming to DebConf, I’ll see you there!

By Elizabeth Krumbach

On the heels of the 5th Annual Emerging Technologies for the Enterprise Conference (ETE 2010) in Philadelphia that CJ attended last week, I’ll be attending the 4th Annual Linux Foundation Collaboration Summit tomorrow through Friday in San Francisco.

The Linux Foundation Collaboration Summit is an exclusive, invitation-only summit gathering core kernel developers, distribution maintainers, ISVs, end users, system vendors and other community organizations for plenary sessions and workgroup meetings to meet face-to-face to tackle and solve the most pressing issues facing Linux today.

My attendance will be in my capacity as a member of the Ubuntu Community Council as well as my role as a Debian Systems Administrator. As such, my attention will be split at the summit between community and governance interests, like the FOSSBazaar Workgroup and Josh Berkus’ How to Prevent Community: Making Sure Your Pond Stays Small, and talks and panels like Does Open Source Mean Open Cloud? where Ubuntu founder Mark Shuttleworth will be a panelist, and the Linux Standard Base Workgroup and Virtualization discussions.

It’s shaping up to be an exciting summit, if you are also attending be sure to say “Hello”!

By Elizabeth Krumbach

At first glance, the ecosystem in the Free and Open Source Software (FOSS) world can seem a bit complicated. There are several ways to get software: project websites where you can download it directly, use a software management tool that your Linux distribution provides, or you may also be able to install a Linux distribution that includes everything you need right out of the box! Once you understand this ecosystem, you can find where your contributions would be most useful, and why contributing is beneficial to your organization and the FOSS community.

So, where does this all begin? FOSS often originates with a project which maintains the source code for the software and provides its own development and support infrastructure.

A Linux distribution is a carefully culled collection of software from these upstream projects which makes a complete operating system and even includes a lot of application software. This collection of software is tested and prepared to run securely and maintainably together. Debian is built upon this model.

Some distributions of Linux use Debian as a source project unto itself. There are a number of Linux distributions based on Debian, including the popular KNOPPIX and Ubuntu distributions. Being “based on Debian” can mean several things, but it primarily means they draw from the software repository at some point in the release cycle, and they use the Advanced Packaging Tool (apt) to manage this software. In these cases Debian is an intermediary between the original FOSS project and the “children” distributions which may also pull from original software projects to expand upon what Debian provides to target their particular focus.

So where in this software ecosystem should your organization contribute? Why would your organization choose to contribute to Debian rather than to the original project (”upstream” of Debian) or a project like Ubuntu (”downstream” of Debian)? It really depends on your goals.

If your organization is interested in using FOSS in a way which requires rapid development, new and diverse features released quickly, or specializations that the distribution may not easily support, you will probably want to work directly on the upstream project. Frequently this requires programming experience, but many projects need other kinds of help such as bug reports in the form of feature requests which they may be able to satisfy in later releases. In these cases, contributing to development in these projects directly is the best way to meet your needs in using and building upon the software.

If your organization needs to use FOSS in a stable, maintainable and secure way, you should probably work directly with Debian. The primary duty of most developers within the Debian community is working on the “packages” which make up the operating system: creating, updating, patching, tracking their security and handling bugs, forwarding details and patches to the upstream projects when applicable. This is what maintains the solid, core operating system that makes up not only Debian, but the child distributions which depend on it, and which could not exist without it. By contributing to Debian you’re also contributing to Ubuntu, Knoppix, and dozens more, improving the tool shelf for everyone (related: Given 250,000 tools on the shelf, how do you manage them?). Contributing to Debian also helps the upstream projects, taking the burden off of them to provide installation documents and support on Debian and placing that upon you, plus making their software more readily available to users through a simple search through the Debian repository.

If the target of one of Debian’s children better meets your organization’s needs which cannot be achieved through Debian directly, then by all means contribute directly to it. Child distributions already exist which focus on everything from being an Open Source LiveCD toolbox (like KNOPPIX) to being a polished desktop operating system (like Ubuntu). As an example, even within Ubuntu’s family there are targeted projects, like Edubuntu, focused on education by packaging and shipping a collection of educational software and a project devoted to making your computer a PVR like TiVo called Mythbuntu which works with the MythTV project to easily deliver their software on a platform. Contributing to projects like these also expands the open source ecosystem and may be the preferred method to reach your organization’s goals.

Understanding the way in which these projects and distributions work together and selecting a place in the workflow for your organization to contribute is the first step. But perhaps a more important question is why you’d want to work on a FOSS project instead of doing in-house development. The benefits for the FOSS community are obvious, they will reap the benefits of having your expertise, from having the packages in Debian and beyond, but are there benefits for your organization?

I believe there are big benefits, which include:

• Peer review of packages and software now and in the future
• Processes for asking the community for assistance
• Bug reporting infrastructure, which may include patches submitted by community members
• Procedures to become informed about security problems and policy changes
• Free collaborative resources provided for FOSS projects (Alioth for Debian,  SourceForge, LaunchPad or the Apache Foundation, etc) for development, including development mailing lists and hosted revision control systems like git, bazaar, svn.
• Opportunity to learn key FOSS development strategies and industry “best practices” via freely available documentation, chat rooms, forums and mailing lists

In short, by putting the time in to releasing software, packaging for Debian or work in children distributions, you not only are doing good for the FOSS community, you get to take advantage of the plethora of tools, resources and people available to assist in the development process.

By CJ Fearnley

In February 2009 Debian released version 5.0, “Lenny” with more than 25 000 packages including many security enhancements such as PHP’s Suhosin system. LinuxForce has compiled a list of 49 news articles documenting milestones of the Debian GNU/Linux project in 2009:
http://www.linuxforce.net/debian.milestones.html#2009.

By CJ Fearnley

Although I haven’t seen a thoroughly researched study, I figure there must be at least 250,000 FOSS (Free and Open Source Software) tools available to every systems administrator on the planet (230,000 at SourceForge + 15,000 at Launchpad + 12,000 at CodePlex + 5,000 at Google Code and that doesn’t count the Linux kernel or any of the myriad other self-hosted projects). These 250,000+ resources comprise the full “toolbox” that admins can use for building solutions with FOSS; they represent the FOSS equivalent of COTS (Commercial Off-The-Shelf). Of course, if you add open source but non-free or commercial tools, the problem explodes combinatorially.

How can a systems administrator support the largest possible subset of these “on the shelf” resources to best service the next need from a stakeholder (like the boss or a new client)?

First let me emphasize the difficulty of the task with a list of items that systems administrators and systems management firms like LinuxForce are expected to do whenever a stakeholder presents a software need:

• Find and Evaluate software that can meet the need:
• Identify several candidate applications that might meet the business requirements for a given project, function, or need
• Research the options to assess their ability to meet the requirements (actually we, the systems administrators of the world, are actually expected to know which tool is “best of breed”: just from our past experience. The false assumption is, if it isn’t well known it must not be any good. The long tail applies to the 250,000+ FOSS tools also!). In our experience such research is essential, unfortunately, there is rarely enough budget to carefully explore the options.
• Install the tool(s) in a “sandbox” to allow the stakeholder to “try it out”
• Select a tool to use or look for more options
• Put the tool into production
• Read the docs to identify best practices for the software’s configuration
• Prepare an installation plan that will address (as best as possible) any upgrade glitches (yes, you have to anticipate them now or suffer the consequences later!) so that you’re prepared for when a security advisory is released (or when the stakeholder starts begging for features from a new release)
• Figure out a support plan to handle the inevitable questions that will arise during operations
• Integrate these considerations into the process of either installing a package or using the “make, configure, make install” steps that most FOSS tools provide for installation
• Carefully document the “as built” configuration including all assumptions and anticipated glitches to help yourself or future admins during the maintenance phase
• On-Going Maintenance
• Monitor the software
• Subscribe to any relevant security mailing lists for the software so that you are apprised when a security (or other major) problem is detected
• Track general trends relating to the software and its alternatives so that you are ready to respond if the project goes dormant or is eclipsed by newer, superior technology.
• Upgrade routinely

About 15 years ago I noticed that the explosion of ready to use FOSS tools plus the trend toward general purpose tools and away from custom software was leading to a combinatorial crisis in software maintenance. I saw that it was the systems administrator’s responsibility to address the situation.

It has become apparent to me that the solution would require use of convention, standards and policy to reduce the complexity of the problem to manageable proportions. I searched for the most “standardized” conventions and policy-enforcing environment that would also provide the most flexible access to the most FOSS tools. The solution I found is Debian/GNU Linux, the universal operating system (although Ubuntu and other Debian derivatives also provide most of these benefits as well).

Debian simplifies the software evaluation process (apt-get [search|show]). Debian simplifies installation (apt-get install), security and new version upgrades (apt-get [upgrade|dist-upgrade]). Debian uses conventions and packages to simplify identifying best practices for administering the software (/usr/share/doc/[package]/, /var/lib/dpkg/info/[package].postinst, and wikis, mailings lists, bug reports, etc.). But the key benefit for managing the combinatorial explosion of FOSS tools is the Debian community’s value of striving to configure each package to automatically support the most common use cases while also providing support for unusual configurations (so you save tons of time in configuring the software).

In summary, the Debian/GNU Linux system provides the infrastructure needed to manage the combinatorial explosion of off the shelf FOSS tools cost effectively. If you have to service a lot of users, customers, or clients with challenging, diverse needs, I think Debian is the most cost effective way to meet their needs and deliver quality maintenance on an on-going basis year after year after year.

By CJ Fearnley

The November 2009 issue of Communications of the ACM (CACM) has a very interesting article by Paul Stachour and David Collier-Brown entitled “You Don’t Know Jack About Software Maintenance”. The authors argue energetically for using versioned data structures and “continuous upgrading” to improve the state of the art of software maintenance.

The piece got me thinking about FOSS (Free and Open Source Software) and “continuous upgrading”. Here are seven observations on FOSS software maintenance that occurred to me as I reflected on the CACM article:

1. FOSS projects “continuously” apply bug fixes and feature enhancements at no additional cost to their users. By applying these improvements “continuously”, the user reaps a steady stream of “interest payments” providing ever-improving security, performance, and functionality.
2. Since FOSS incurs no licensing or license management costs, upgrading FOSS is not hindered by capital expenses.
3. Typically support in FOSS projects is focused on the current stable version. Therefore, upgrading to the current stable version is the preferred way to receive the best support from FOSS communities.
4. One of the key reasons behind Debian’s strong track record of “continuous upgrading” is its way of handling the tricky issues involved with dependent library upgrades (such as libc6, libssl.so.0.9.8, & etc). The chapter on Shared Libraries in the Debian Policy Manual details a proven method to effectively handle library upgrade issues (including its sophisticated handling of versions).
5. When upgrading is applied routinely and “continuously”, it becomes crucial to support customizations across upgrades which can be one of the biggest obstacles to a smooth upgrade (see my earlier post on customization and upgradeability). One reason for Debian’s effectiveness in this regard is its robust configuration file handling policy.
6. It is worth noting that the “continuous” implied here is not the one emphasized in dictionaries (which takes its nuances from the mathematical / physics concept of “no interruptions” and the epsilon-delta definition that students of Calculus learn). That concept of “continuous” is impossible in systems administration which is necessarily discrete as are all computer operations. The connotation required here is, perhaps, “unending”, or “eternal” or somesuch.
7. The “right” frequency for “continuous” upgrades is a complex tradeoff between business requirements and upgrade infrastructure maturity. Debian and Ubuntu provide vary mature support for “continuous upgrading”. They support the upgrade of production servers through release after release after major release with minimal downtime or risk of a glitch that could affect users. Their current release frequency of about 2 years may be the best we can do given the current state of the art of software maintenance. I hope we can learn to increase the frequency as better engineered upgrade policies are developed.

I prefer the name “eternally regenerative software administration” over “continuous upgrading”. It avoids the philosophical problems with the word “continuous” and emphasizes the active, “ecological” approach needed to envision the engineering of “regenerativity” in software. By that I mean software maintenance should involve building the system so each new version enables installation of the next while facilitating management of any customizations and integration with other software (including libraries and other “helper” applications). Regenerativity is the process of growth and change used by Nature itself. Software maintenance needs to follow similar principles.

By CJ Fearnley

At our Seminar last month, Managing FOSS to Lower Costs and Achieve Business Results, several participants asked about the dynamics of FOSS (Free and Open Source Software) projects that reach a crossroads (a failure, a merger, loss of key personnel, etc). I had not expected that concern because with commercial software, it seems to me, the problem is more severe. When you have the source code and the right to modify and redistribute it, the source gives many more options (and its freedoms provide many more protections) than when commercial software goes bankrupt or gets bought by a competitor for instance.

But the reason for the questions may be a lack of understanding about how FOSS projects work. They involve individual human beings, perhaps just a single person or, more likely, several people from many organizations and even different cultures around the world joined in common purpose. For various cultural reasons, the project may be “owned” by an entity — usually a non-profit, but some are for-profit or even government owned, while others may simply be an “ad hoc initiative”. Some projects have explicit constitutions and defined processes for organizing the work and handling problems others are more informal.

At any time, any human social structure can experience a crossroads that could lead it to fail suddenly or wither on the vine in a gradual descent into “oblivion”. The cause of the failure will shape the results, but a very common situation is that conflicting visions or approaches for the project result in a “fork”. Then a sub-group of the original project takes the source code and starts a “new” project to develop the code in a new direction. Sometimes the original project “dies” and sometimes both continue resulting in two projects. Since multiple FOSS projects serving the same function or market incur inefficiencies due to duplicate development, there is a strong cultural value in the FOSS world to try to find a way to accommodate everyone in the project and prevent forks. When it works, the result is great software that meets everyone’s needs. But the reality is that often it is more effective to have multiple implementations of the same functionality so that each can be optimized for distinct objectives. Frequently one cannot know which approach will be best until many years of development and evaluation have transpired.

I recently learned about a FOSS project that forked when a friend asked me to copy some files to his new “My Book Essential”, a Western Digital product that provides 1TB of USB (Universal Serial Bus) storage. The My Book uses the poorly documented, non-free NTFS (New Technology File System). Linux has three projects that support NTFS: an in kernel driver, ntfsprogs (the Linux-NTFS project), and NTFS-3G. It turns out that all three were available for my Debian Lenny (5.0.3) system. First, I tried the in kernel support and learned that it was still read-only. Then I tried ntfsprogs which failed to mount the My Book:

NTFS-fs error (device sdc1): load_system_files(): Volume is dirty. Mounting read-only. Run chkdsk and mount in Windows.

I realized that since it was a new device it probably did not ship from the factory with a dirty volume. It was probably a bug. So I tried NTFS-3G which worked very well. In my research of the situation I was able to determine that both NTFS-3G and Linux-NTFS are under active development and have features missing from the other. So each has value and I’m glad my distribution included both. In Debian Lenny, the NTFS-3G driver has better support for writing files.

This illustrates one of the benefits of a crossroads in a FOSS project: you can end up with two good tools to add to your toolbox!

By CJ Fearnley

Mary Hayes Weier wrote an interesting article in this week’s edition of InformationWeek on "Alternative IT: CIOs are more receptive than ever to new software models". What is great about her article is how she captured the divergent views on IT models (such as SaaS, cloud computing, etc.) and gave nice vignettes of different organizations trying different parts of various models. I especially valued her use of cognitive dissonance to leave the reader thinking … better informed but without a firm conclusion.

There are so many parts of the article that I could blog about, but the one that touched the core of my thinking about “eternally regenerative software administration” was the quote by Bill Louv, CIO at GlaxoSmithKline, who said

"And here’s the rub: When you customize software, it’s difficult to implement future upgrades from the vendor"

Louv touched the very bane of eternally regenerative software administration! Software should accommodate both customization and upgradeability: these two elements of software administration are at the heart of my notion of eternally regenerative software administration: how to preserve customizations and provide smooth (near zero downtime with almost no glitches) upgrades through major release after major release. It is a big challenge, but in our experience the Free and Open Source Software (FOSS) communities are at the leading edge in finding solutions to these conflicting objectives. Here are some of the innovative ideas from the FOSS world which should serve as models or design patterns for all software developers (if only these ideas would become commonplace!).

First, Debian (a FOSS operating system which is the root of Ubuntu, Knoppix, Xandros and many other Linux distributions) requires that their official packages, a collection of software prepared for easy administration, must adhere to a very mature policy. Debian’s policy is a marvel in the FOSS world and to a very large degree is responsible for its strong support for both customization and upgradeability. I think Debian’s reputation for stability and maintainability is almost certainly due to their decision to develop a consensus-driven policy that its software must implement.

For example, the Debian package maintainer, Luigi Gangitano, for Drupal, a FOSS content management platform, did a great job making the software both customizable and maintainable. The package supports configuration of multiple virtual hosts which can all be upgraded at once! And the Debian drupal6 package stores the look-n-feel in /etc/drupal/6/themes/ so that each site’s GUI can be customized without interfering with upgrades. If only all web applications were built to be as maintainable as Debian’s Drupal package!

Another example is the overlay support included in RT: Request Tracker, a FOSS ticket tracking system. This allows putting replacement subroutines in special files in /usr/local/share/ which overlay or substitute the upstream code. This approach is more likely to break on upgrades, but it supports minimal changes to the business logic with a decent chance that upgrades will be smooth.

There are countless more examples from the FOSS world of innovative solutions to inter-accommodate customization and upgrades in support of eternally regenerative software administration. What are some of your favorite examples?