An outsider’s inside view on open source at AWS

As AWS continues to increase our participation in a wide variety of open source communities, we’ve been fortunate to get to work with a range of talented developers. One of these contributors is Dr. Colin Percival, a long-time FreeBSD developer perhaps best known for his work on FreeBSD Update and Portsnap. Colin is also the founder of Tarsnap, billed as “Online backups for the truly paranoid.”

Among AWS customers, however, Colin may be better known for having spent the last decade building and maintaining the FreeBSD AMIs. In large part due to that work, he has been recognized as an AWS Community Hero.

Over that time, Colin says, his own open source collaboration with AWS has gone from working with scattered AWS employees to coordinated corporate support. While “it’s always Day 1 at AWS,” including with our open source efforts, Colin’s perspective sheds some light on how AWS’ open source involvement has evolved over the past decade. In his words, the AWS embrace of open source “seemed like a fairly gradual process to me, with a combination of AWS customers wanting to use FreeBSD, Amazon using FreeBSD internally, and Amazon generally becoming more aware of the importance of open source software.”

AWS has always been a great place to run the world’s best open source software. Even so, it is absolutely the case, to Colin’s point, that our understanding of how to care for our best customers  has evolved. At AWS, our best customers aren’t always our biggest – our best customers are those that help AWS make our products better in the long run, as Colin certainly did. To help make this point clearer, it’s worth reviewing Colin’s experience with AWS, while getting a behind-the-scenes look at how AWS service teams were thinking about his requests.

FreeBSD stranger in a strange AWS land

Colin has been a FreeBSD enthusiast for more than 15 years. As he puts it, “FreeBSD is UNIX for people who like UNIX [with] sane defaults, a reasonably minimal base system…, and lots of convenient tools built in.” What FreeBSD has lacked, however, is significant market share.

When AWS first launched in 2006, Colin almost immediately started to use it. As he recounts, he needed to have storage and compute side by side for Tarsnap to work. “At the time, the combination of S3 + EC2 was unlike anything available elsewhere,” he says. He also hoped to get FreeBSD working with EC2, given that FreeBSD was already able to run under Xen. “I thought getting FreeBSD working in EC2 would be relatively straightforward,” he notes. (Spoiler alert: it wasn’t.)

Importantly, however, he could try. This “self service” aspect of creating EC2 images, coupled with the ability to build and run one’s own kernel, were early distinguishing properties of Amazon EC2 that enabled developers like Colin to “scratch their own itch.” This was arguably one of the things that made Amazon EC2 so popular with developers: It was designed to handle as many use cases as possible, with as few constraints as feasible. Against this backdrop, Colin kept working on the FreeBSD AMI.

Meanwhile, things were moving behind the AWS curtain. We released PV-GRUB images in July of 2010, because we wanted to make Amazon EC2 more broadly capable. Prior to this time, only a small number of partners could upload Linux kernel images for use in AMIs. As AWS’ Jeff Barr called out in 2010, we knew that PV-GRUB would let skilled open source developers boot other operating systems as well – “You could (if you are sufficiently adept) use this facility to launch an operating system that we don’t support directly (e.g. FreeBSD).”

PV-GRUB was useful for other operating systems, but not directly beneficial to Colin, he says. “For FreeBSD itself it was neither sufficient (there were other issues) nor necessary (I already had conversations about getting whitelisted as a kernel image publisher).” To this point, Percival stresses that while AWS didn’t offer him formal support in his efforts, behind the scenes a number of AWS employees were lending a hand:

The FreeBSD community is used to: “If you want it to work, you have to make it work”. In the early days I didn’t have anything which I would characterize as support from Amazon management; but Amazon engineers were always ready to help me. I’ve probably exchanged emails with over 50 engineers within Amazon on a range of topics including upcoming features, bugs in AWS services, internal documentation of AWS functionality, bugs in FreeBSD, and patches for FreeBSD….

What must have felt like ad hoc support from individual AWS employees to Colin was indicative of something more central to the AWS way of doing things: customer obsession. In this case, as AWS distinguished engineer Matt Wilson related to me, “We know that open source developers are some of the best customers in the world. They are collaborative, communicative, and wicked smart. They also hold a very high bar on quality and ‘taste’.” What Colin views as “individual engineers [going] out of their way to help me,” AWS viewed as a way to serve our best customers.

Colin, as a ”wicked smart“ developer, was helping us to build a better EC2 service. Over time, he stresses, he’s noticed an evolution of open source support at AWS:

In the early years of AWS, I relied on individual engineers taking time out of their days to help me get FreeBSD working; there was no clear management support at all. This completely turned around by 2016, where not only did I have advance notice of the new Elastic Network Adapter hardware, but Amazon provided a FreeBSD driver. Many hardware companies don’t give us anything, and many hardware companies “support” FreeBSD by saying: “Here’s the Linux driver; feel free to port it.” Having a working driver arrive with minimal work required by the FreeBSD team was a huge sign of Amazon recognizing the importance of contributing back to the community.

This change, however, wasn’t sudden: “It seemed like a fairly gradual process to me, with a combination of AWS customers wanting to use FreeBSD, Amazon using FreeBSD internally, and Amazon generally becoming more aware of the importance of open source software.”

While Colin has no way of knowing how broadly his AMIs are used (the only insight comes from “people who have emailed me and people who contribute to my FreeBSD/EC2 Patreon”), there are some hints of large organizations using FreeBSD. As he indicates, “There are some large companies which build appliances based on FreeBSD and have offerings inside EC2.” However, he goes on, “I don’t have any inside knowledge about them, but I’d assume that they’re using FreeBSD in their virtualized appliances in addition to their physical appliances.”

Which brings us back to open source, and whether Colin begrudges these enterprises are potentially “taking advantage” of the code he contributes.

What open source looks like

While no company (or person) is perfect in its open source engagement, Colin repeats that, in his opinion, AWS has gotten better in the decade-plus that he has worked with the company. Even so, he notes, there are things that we could be doing to help more, such as introducing a mechanism to cover the storage costs for Amazon EC2 AMI public image publishers.

In the meantime, he says, too many people misunderstand the fundamentals of how open source operates:

I’ve never understood people who release open source code and then complain about how it is used – the freedom to use open source code however you like is the most basic freedom there is in the open source world. When I see people forking projects and changing the license because they disapprove of how it has been used, all I can think is that they never understood what open source software was about in the first place.

Also, he adds, it’s rarely the case that any particular company takes without giving back:

We’ve found in FreeBSD that companies using FreeBSD almost always end up contributing back — not because they’re forced to but rather because it reduces their engineering burden if their non-core functionality is maintained upstream. I’m sure that many of the patches which have appeared unheralded in my inbox fall into the category of “Engineer tripped over a problem when building a product and didn’t want to deal with carrying around a local patch for the next decade.”

It’s a thoughtful perspective from someone who has benefited from and contributed to open source in significant ways. As Colin says, “It’s up to every company [and individual] to decide to what extent they care about maintenance burden[s], driving the direction of the code they use, keeping IP proprietary, et cetera.” AWS, for its part, has increasingly helped to lessen his load in building and maintaining the FreeBSD AMIs for Amazon EC2, even as we’ve learned from him and other open source developers about how to improve our services for them and for downstream customers.

UPDATE (3:12 PST on Friday, November 8, 2019): This post was updated to reflect Colin’s feedback (here and here). I apologize for over-emphasizing the AWS view in a post that was meant to focus on Colin’s view. 

If you’d like to help support Colin Percival’s work with FreeBSD AMIs, please visit his Patreon page here.