Introducing HubCommander
by Mike Grima, Andrew Spyker, and Jason Chan
Netflix is pleased to announce the open source release of HubCommander, a ChatOps tool for GitHub management.
Why HubCommander?
Netflix uses GitHub, a source code management and collaboration site, extensively for both open source and internal projects. The security model for GitHub does not permit users to perform repository management without granting administrative permissions. Management of many users on GitHub can be a challenge without tooling. We needed to provide enhanced security capabilities while maintaining developer agility. As such, we created HubCommander to provide these capabilities in a method optimized for Netflix.
Why ChatOps?
Our approach leverages ChatOps, which utilizes chat applications for performing operational tasks. ChatOps is increasingly popular amongst developers, since chat tools are ubiquitous, provide a single context for what actions occurred when and by whom, and also provides an effective means to provide self-serviceability to developers.
How Netflix leverages GitHub:
All Netflix owned GitHub repositories reside within multiple GitHub organizations. Organizations contain the git repositories and the users that maintain them. Users can be added into teams, and teams are given access to individual repositories. In this model, a GitHub user would get invited to an organization from an administrator. Once invited, the user becomes a member of the organization, and is placed into one or more teams.
At Netflix, we have several organizations that serve specific purposes. We have our primary OSS organization “Netflix”, our “Spinnaker” organization that is dedicated to our OSS continuous delivery platform, and a skunkworks organization, “Netflix-Skunkworks”, for projects that are in rough development that may or may not become fully-fledged OSS projects, to name a few.
Challenges we face:
One of the biggest challenges with using GitHub organizations is user management. GitHub organizations are individual entities that must be separately administered. As such, the complexity of user management increases with the number of organizations. To reduce complexity, we enforce a consistent permissions model across all of our organizations. This allows us to develop tools to simplify and streamline our GitHub organization administration.
How we apply security to our GitHub organizations:
The permissions model that we follow is one that applies the principle of least privilege, but is still open enough so that developers can obtain the access they need and move fast. The general structure we utilize is to have all employees placed under an employee’s team that has “push” (write) access to all repositories. We similarly have teams for “bot” accounts to provide for automation. Lastly, we have very few users with the “owner” role, as owners are full administrators that can make changes to the organization itself.
While we permit our developers to have write access to all of our repositories, we do not directly permit them to create, delete, or change repository visibility. Additionally, all developers are required to have multi-factor authentication enabled. All of our developers on GitHub have their IDs linked in our internal employee tracking system, and GitHub membership to our organizations is removed when employees leave the company automatically (we have scripts to automate this).
We also enable third-party application restrictions on our organizations to only allow specific third party GitHub applications access to our repositories.
Why is tooling required?
We want to have self-service tooling that provides an equivalent amount of usability as providing users with administrative access, but without the risk of making all users administrators.
Our tooling provides a consistent permissions model across all of our GitHub organizations. It also empowers our users to perform privileged operations on GitHub in a consistent and supported manner, while limiting their individual GitHub account permissions.
Because we limited individual GitHub account permissions, this can be problematic for developers when creating repositories, since they also want to update the description, homepage, and even set default branches. Many of our developers also utilize Travis CI for automated builds. Travis CI enablement requires that users be administrators of their repositories, which we do not permit. Our developers also work with teams outside of Netflix to collaborate with on projects. Our developers do not have permissions to invite users to our organizations or to add outside collaborators to repositories. This is where HubCommander comes in.
The HubCommander Bot
HubCommander is a Slack bot for GitHub organizational management. It provides a ChatOps means for administering GitHub organizations. HubCommander operates by utilizing a privileged account on GitHub to perform administrative capabilities on behalf of our users. Our developers issue commands to the bot to perform their desired actions. This has a number of advantages:
- Self-Service: By providing a self-service mechanism, we have significantly reduced our administrative burden for managing our GitHub repositories. The reduction in administrative overhead has significantly simplified our open source efforts.
- Consistent and Supported: The bot performs all of the tasks that are required for operating on GitHub. For example, when creating repositories, the bot will automatically provide the correct teams access to the new repository.
- Least Privilege for Users: Because the bot can perform the tasks that users need to perform, we can reduce the GitHub API permissions on our users.
- Developer Familiarity: ChatOps is very popular at Netflix, so utilizing a bot for this purpose is natural for our developers.
- Easy to Use: The bot is easy to use by having an easily discoverable command structure.
- Secure: The bot also features integration with Duo for additional authentication.
HubCommander Features:
Out of the box, HubCommander has the following features:
- Repository creation
- Repository description and website modification
- Granting outside collaborators specific permissions to repositories
- Repository default branch modification
- Travis CI enablement
- Duo support to provide authentication to privileged commands
- Docker image support
HubCommander is also extendable and configurable. You can develop authentication and command based plugins. At Netflix, we have developed a command plugin which allows our developers to invite themselves to any one of our organizations. When they perform this process, their GitHub ID is automatically linked in our internal employee tracking system. With this linkage, we can automatically remove their GitHub organization membership when they leave the company.
Duo is also supported to add additional safeguards for privileged commands. This has the added benefit of protecting against accidental command issuance, as well as the event of Slack credentials getting compromised. With the Duo plugin, issuing a command will also trigger a “Duo push” to the employee’s device. The command only continues to execute if the request is approved. If your company doesn’t use Duo, you can develop your own authentication plugin to integrate with any internal or external authentication system to safeguard commands.
Using the Bot:
Using the bot is as easy as typing !help in the Slack channel. This will provide a list of commands that HubCommander supports:
To learn how to issue a specific command, simply issue that command without any arguments. HubCommander will output the syntax for the command. For example, to create a new repository, you would issue the !CreateRepo command:
If you are safeguarding commands with Duo (or your own authentication plugin), an example of that flow would look like this:
Contributions:
These features are only a starting point, and we plan on adding more soon. If you’d like to extend these features, we’d love contributions to our repository on GitHub.
Originally published at techblog.netflix.com on February 7, 2017.
