Wednesday, October 28, 2015

Evolution of Open Source at Netflix

When we started our Netflix Open Source (aka NetflixOSS) Program several years ago, we didn’t know how it would turn out.  We did not know whether our OSS contributions would be used, improved, or ignored; whether we’d have a community of companies and developers sending us feedback; and whether middle-tier vendors would integrate our solutions into theirs.  The reasons for starting the OSS Programs were shared previously here.


Fast forward to today.  We have over fifty open source projects, ranging from infrastructural platform components to big data tools to deployment automation.  Over time, our OSS site became very busy with more and more components piling on.  Now, even more components are on the path to being open.




While many of our OSS projects are being successfully used across many companies all over the world, we got a very clear signal from the community that it was getting harder to figure out which projects were useful for a particular company or a team; which were fully independent; and which were coupled together.  The external community was also unclear about which components we (Netflix) continued to invest and support, and which were in maintenance or sunset mode.  That feedback was very useful to us, as we’re committed in making our OSS Program a success.


We recently updated our Netflix Open Source site on Github pages.  It does not yet address all of the feedback and requests we received, but we think it’s moving us in the right direction:


  1. Clear separation of categories.  Looking for Build and Delivery tools?  You shouldn’t have to wade through many unrelated projects to find them.
  2. With the new overview section of each category we can now explain in short form how each project should be used in concert with other projects.  With the old “box art” layout, it wasn’t clear how the projects fit together (or if they did) in a way that provided more value when used together.
  3. Categories now match our internal infrastructure engineering organization.  This means that the context within each category will reflect the approach to engineering within the specific technical area.  Also, we have appointed category leaders internally that will help keep each category well maintained across projects within that area.
  4. Clear highlighting of the projects we’re actively investing and supporting.  If you see the project on the site - it’s under active development and maintenance.  If you don’t see it - it may be either in maintenance only or sunset mode.  We’ll be providing more transparency on that shortly.
  5. Support for multi-repo projects.  We have several big projects that are about to be Open Sourced.  Each one will consist of many Github repos.  The old site would list each of the repos, thus making the overall navigation even less usable.  The new site allows us to group the relevant repos together under a single project




Other feedback we’re addressing is that it was hard to get started with many of our OSS projects.  Setup / configuration was often difficult and tricky.  We’re addressing it by packaging most (not yet all) of our projects in the Docker format for easy setup.  Please note, this packaging is not intended for direct use in production, but purely for a quick ramp-up curve for understanding the open source projects.  We have found that it is far easier to help our users’ setup of our projects by running pre-built, runnable Docker containers rather than publish source code, build and setup instructions in prose on a Wiki.


The next steps we’ll be taking in our Open Source Program:


  1. Provide full transparency on which projects are archived - i.e. no longer actively developed or maintained.  We will not be removing any code from Github repos, but will articulate if we’re no longer actively developing or using a particular project.  Netflix needs change over time, and this will affect and reflect our OSS projects.
  2. Provide a better roadmap for what new projects we are planning to open, and which Open projects are still in the state of heavy flux (evolution).  This will allow the community to better decide whether particular projects are interesting / useful.
  3. Expose some of the internal metrics we use to evaluate our OSS projects - number of issues, commits, etc.  This will provide better transparency of the maturity / velocity of each project.
  4. Documentation.  Documentation.  Documentation.


While we’re continuing on our path to make NetflixOSS relevant and useful to many companies and developers, your continuous feedback is very important to us.  Please let us know what you think at netflixoss@netflix.com.  


We’re planning our next NetflixOSS Meetup in early 2016 to coincide with some new and exciting projects that are about to be open.  Stay tuned and follow @netflixoss for announcements and updates.