How a GNU/Linux Distribution Succeeds, Part 1: Two Long-Lasting Examples
In the world of GNU/Linux distributions, many flourish like grass and then wither away. Despite many appealing elements, a lot of distributions lack staying power. But the ones that have been most successful share some key traits among their great diversity–I discovered that while researching this article.
Out of the hundreds of distributions of Linux (for short), I’m choosing for this article two that have lasted over nearly the whole history of the operating system, have grown stronger over the decades, have given rise to child distributions, and still power huge numbers of sites today: Red Hat and Debian. This article looks at why these distributions became so successful and spawned families (Fedora and CentOS out of Red Hat; Ubuntu and many others out of Debian).
Both Red Hat and Debian started in 1993, just a couple years after Linus Torwalds unveiled his kernel to the world. Red Hat and Debian quickly muscled out a handful of less professionally run projects such as Slackware and SLS. The new distributions took very different routes: Red Hat as a company with a large contributing community, and Debian as a community whose commercial aspects were quite minor.
Along with my own recollections, this article draws on interviews with the following leaders in the field, and has been reviewed by others:
- Scott McCarty, long-time Red Hat administrator and currently product manager for Red Hat Enterprise Linux Server
- Bruce Perens, one of the leading figures in open source and free software whose roles included leading the Debian project and co-founding the Open Source Initiative (OSI)
- Matt Welsh, an early leader of the Linux Documentation Project and co-author of the best-selling 1996 book Running Linux (with me as his editor)
In terms of user base, of course, Android is the most popular Linux distribution. People are even using it for purposes outside of Google phones. But I won’t cover Android here because its history and role in computing are so unique. In particular–as pointed out by Richard Stallman, founder and head of the GNU project–Android makes very little use of the commands and tools that the other distributions use, particularly the GNU compiler and libraries that make it possible for other distributions to run the operating system (hence the preferred term for distributions: GNU/Linux).
The following key factors emerged from my interviews as underlying the success of both Red Hat and Debian:
- Respecting and meeting the needs of the community
- Following a professional process to develop a reliable product
- Creating a robust packaging system
I’ll explore these and related factors in this two-part article.
Respecting and meeting the needs of the community
Numerous companies such as Caldera adopted a conventional corporate approach. They took software from various vendors and open source projects to create CDs for their customers. It was like building and selling a shirt or a bicycle. To be sure, many companies listen to customers and build on their innovations, as pointed out by MIT Sloan professor Eric von Hippel. But few companies focus on communities.
Listening to Contributors and Users
Debian was community-based from the start and has always remained so. This won it enormous trust, because no one felt that a corporate hand would distort the will of the community or profit unfairly. The Free Software Foundation (FSF), a non-profit, was an important early sponsor of Debian. Although Debian has always been a community effort, a company was set up early in the project to sell CDs of the distribution.
According to both McCarty and Welsh, Red Hat was also unusually—perhaps uniquely, for a software company—attentive to its community from the start. And this community, feeling acknowledged, accepted the company because users knew that a corporate entity was needed to manufacture CDs and get the distribution out to the public.
Throwing In One’s Lot With the Sources of Software
I do freelance work for Red Hat now, and am routinely amazed at the amount of time invested by its paid staff in community projects: compilers, Python and JavaScript packages, networking tools, and more. According to a 2005 article, Red Hat made a decision around that time to contribute their changes upstream and use the community’s source code instead of giving customers fixes specific to Red Hat. In turn, developers using the community versions—Fedora and CentOS—contribute improvements that get accepted into Red Hat Enterprise Linux, the commercial offering.
Integrating a company’s work with the upstream community is obviously good for other free software users and companies that depend on the code, but Red Hat also benefits because it doesn’t have to maintain forked versions. A video by Dave Neary provides more background on this historic decision.
James M. Whitehurst, who was chair of the board of Red Hat from 2007 through 2020, wrote an idealistic and exhortatory book The Open Organization: Igniting Passion and Performance (also the inspiration for a Red Hat website). In this book, he urges all companies to apply the principles of open source, allowing radical forms of experimentation and information sharing. No company can consistently follow these ideals, but from my vantage point as a freelancer, Red Hat employees remain committed to an egalitarian and honest work environment.
A Careful Balance With Proprietary Software
Debian had to make a very difficult decision in the mid-1990s: whether to incorporate non-free products into the distribution or make a principled commitment to including only free software (while allowing a separate repository where users could download popular proprietary tools). Anthropologist E. Gabriella Coleman, in her book Coding Freedom, explains that other distributions at that time were perceived to have an advantage because they included proprietary software that users enjoyed.
But the Debian community made an historic commitment to include only free software in the Debian distribution, partly because of its early ties to the FSF. The discussions producing this decision were led by Bruce Perens when he was Debian Project Leader. The commitment became part of a “Social Contract” with the free software community. Of course, Red Hat has the same approach, providing a totally open source distribution while making it easy to add proprietary software if you want it.
I think that the commitment to free software was a smart, forward-thinking decision that strengthened the free software community and ultimately spurred the widespread adoption of the distributions. People could trust everything in the distributions—not that everything always worked (software doesn’t operate like that), but that the community could maintain control over the software and prioritize community needs.
Perens believes that the constant concern for policy among Debian leadership is the most important factor in their success. Discussion at Debian was incorporated by Perens into the Debian Free Software Guidelines, the project’s definition of what software was free enough to be part of Debian. Less than a year later, that text became the Open Source Definition, which determines all the licenses approved by OSI and has the force of law in the United States.
Governance Is Required
The Debian leadership took a very firm direction in community building: not only vetting new maintainers, but mentoring them through what one might call a process of self-discovery to ensure their commitment to free software and its associated community processes. Coleman spends a lot of time on this New Maintainer’s Process (NMP), as it was then called. (Because her book came out in 2013, the process is probably much different now.)
The NMP involved a series of essays explaining the developer’s motivations for becoming a maintainer and explaining in the developer’s own words their relationship to free software. Logistical tasks such as joining Debian’s web of trust are also involved.
As Coleman describes the NMP, it sounds almost like initiation into a cult. But it’s better compared to attending graduate school. If you spend three to six years studying history, biology, or law, you are being forced to trace the paths taken by others before you and to absorb their thought processes. Hopefully, you pick up their habits and commitment to iron-clad research. There’s a reason that graduate school subjects are called disciplines. I think that Debian’s NMP tries to instill this kind of discipline in a relatively short time.
In any case, Debian organically developed a system for onboarding new talent, a key requirement for any project.
According to Perens, the Debian project went further and decentralized development for the 15-20 packages needed to build the operating system, so that pieces could be developed by people in different parts of the world with no manager in common.
Debian mailing lists have often been criticized for nasty postings on their mailing lists. Coleman’s book echoes these criticisms. But when I attended the 2010 Debian conference, I found a lovely group of people who enjoyed each other and cared deeply about working well as a community. I thought that they needed to improve the harshness of the online culture, but that fundamentally they were passionate volunteers who just wanted the highest level of effort from everybody.
And what about Fedora? Certainly it’s a useful distribution to load on your personal computer if you work for an organization that runs Red Hat Enterprise Linux. But McCarty tells me that Fedora is much more widespread than that. It has an independent community that prefers it on its own merits.
The Role of Conferences
As a final note on community, conferences are important for bringing communities together. Debian holds conferences on every continent, moving its annual conference each year. The Fedora Contributor Conference (Flock) and DevConf play similar roles for developers and users on Fedora and Red Hat.
Coleman offers a wonderful description of a conference that really makes you sense what it’s like to be there. She does not, however, explain another important point: that conferences are critical for summit-level discussions that set the direction of the community for the upcoming year.
The second part of the article explores the other two factors in the success of Red Hat and Debian, and compares them to a couple other Linux distributions.
About Andrew Oram:
