Open Source at Netflix
At Netflix we use a wide range of Open Source technologies. In the recent months, we also released many of our internally developed components and libraries, starting with Curator for Zookeeper, and most recently with Asgard.
We started down this path by becoming a big user of Apache licensed open source software. When we picked Apache Cassandra as our data storage solution we started to contribute fixes and extensions to optimize Cassandra’s capabilities on AWS. This led to us see the benefits of releasing our own projects and we created a central Netflix account at netflix.github.com as a home for them.
There are many reasons why we’re opening up much of our software. To highlight some of them:
- We have benefited from many other people contributing to open source, so we are paying back in kind.
- Netflix was an early cloud adopter, moving all of our streaming services to run on top of AWS infrastructure. We paid the pioneer tax — by encountering and working through many issues, corner cases and limitations. We’ve captured the patterns that work in our platform components and automation tools. We benefit from the scale effects of other AWS users adopting similar patterns, and will continue working with the community to develop the ecosystem.
- External community contributions — by opening up we enable the larger developer community to: review, comment, add test cases, bug fixes, ports and functional contributions to our components, benefiting everyone.
- Improved code and documentation quality — we’ve observed that the peer pressure from “Social Coding” has driven engineers to make sure code is clean and well structured, documentation is useful and up to date. What we’ve learned is that a component may be “Good enough for running in production, but not good enough for Github”.
- Durability — we think any code will fare better over time if it’s actively developed by an open community and used widely vs. maintained by small number of engineers for a single workload.
Brief overview of components we open sourced so far:
For Zookeeper:
- Curator — Client wrapper and rich Zookeeper framework.
- Exhibitor — Co-process for instance monitoring, backup/recovery, cleanup and visualization.
For Cassandra:
- Astyanax — High level, simple object oriented client for Cassandra.
- Priam — Co-process for backup/recovery, Token Management and centralized Configuration management for Cassandra.
- Jmeter Plugin for Cassandra — automation for running Cassandra tests.
Netflix platform and tools:
- Autoscaling scripts — Tools and documentation about using Auto Scaling part of AWS services.
- Archaius — library for managing dynamic configuration properties.
- Asgard — Web interface for application deployment and cloud management in AWS.
So far we’ve seen great community response and feedback on our Open Source efforts. Many companies are already using components listed above, and many others are evaluating and integrating them into their own software stack. We hope you find what we’ve opened so far useful. Many great libraries and components are coming soon, stay tuned! You can follow @NetflixOSS for news and updates. You can let us know whether you’re using our tools and send any feedback via one of our Mailing Lists.
If you’re interested in contributing to these and other great technologies at Netflix, check out jobs.netflix.com
See Also:
Originally published at techblog.netflix.com on July 13, 2012.
