Wednesday, September 21, 2016

Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems

We recently made a major architectural change to Zuul, our cloud gateway. Did anyone even notice!?  Probably not... Zuul 2 does the same thing that its predecessor did -- acting as the front door to Netflix’s server infrastructure, handling traffic from all Netflix users around the world.  It also routes requests, supports developers’ testing and debugging, provides deep insight into our overall service health, protects Netflix from attacks, and channels traffic to other cloud regions when an AWS region is in trouble. The major architectural difference between Zuul 2 and the original is that Zuul 2 is running on an asynchronous and non-blocking framework, using Netty.  After running in production for the last several months, the primary advantage (one that we expected when embarking on this work) is that it provides the capability for devices and web browsers to have persistent connections back to Netflix at Netflix scale.  With more than 83 million members, each with multiple connected devices, this is a massive scale challenge.  By having a persistent connection to our cloud infrastructure, we can enable lots of interesting product features and innovations, reduce overall device requests, improve device performance, and understand and debug the customer experience better.  We also hoped the Zuul 2 would offer resiliency benefits and performance improvements, in terms of latencies, throughput, and costs.  But as you will learn in this post, our aspirations have differed from the results.

Differences Between Blocking vs. Non-Blocking Systems

To understand why we built Zuul 2, you must first understand the architectural differences between asynchronous and non-blocking (“async”) systems vs. multithreaded, blocking (“blocking”) systems, both in theory and in practice.  

Zuul 1 was built on the Servlet framework. Such systems are blocking and multithreaded, which means they process requests by using one thread per connection. I/O operations are done by choosing a worker thread from a thread pool to execute the I/O, and the request thread is blocked until the worker thread completes. The worker thread notifies the request thread when its work is complete. This works well with modern multi-core AWS instances handling 100’s of concurrent connections each. But when things go wrong, like backend latency increases or device retries due to errors, the count of active connections and threads increases. When this happens, nodes get into trouble and can go into a death spiral where backed up threads spike server loads and overwhelm the cluster.  To offset these risks, we built in throttling mechanisms and libraries (e.g., Hystrix) to help keep our blocking systems stable during these events.

Multithreaded System Architecture

Async systems operate differently, with generally one thread per CPU core handling all requests and responses. The lifecycle of the request and response is handled through events and callbacks. Because there is not a thread for each request, the cost of connections is cheap. This is the cost of a file descriptor, and the addition of a listener. Whereas the cost of a connection in the blocking model is a thread and with heavy memory and system overhead. There are some efficiency gains because data stays on the same CPU, making better use of CPU level caches and requiring fewer context switches. The fallout of backend latency and “retry storms” (customers and devices retrying requests when problems occur) is also less stressful on the system because connections and increased events in the queue are far less expensive than piling up threads.

Asynchronous and Non-blocking System Architecture

The advantages of async systems sound glorious, but the above benefits come at a cost to operations. Blocking systems are easy to grok and debug. A thread is always doing a single operation so the thread’s stack is an accurate snapshot of the progress of a request or spawned task; and a thread dump can be read to follow a request spanning multiple threads by following locks. An exception thrown just pops up the stack. A “catch-all” exception handler can cleanup everything that isn’t explicitly caught.   

Async, by contrast, is callback based and driven by an event loop. The event loop’s stack trace is meaningless when trying to follow a request. It is difficult to follow a request as events and callbacks are processed, and the tools to help with debugging this are sorely lacking in this area. Edge cases, unhandled exceptions, and incorrectly handled state changes create dangling resources resulting in ByteBuf leaks, file descriptor leaks, lost responses, etc. These types of issues have proven to be quite difficult to debug because it is difficult to know which event wasn’t handled properly or cleaned up appropriately.

Building Non-Blocking Zuul

Building Zuul 2 within Netflix’s infrastructure was more challenging than expected. Many services within the Netflix ecosystem were built with an assumption of blocking.  Netflix’s core networking libraries are also built with blocking architectural assumptions; many libraries rely on thread local variables to build up and store context about a request. Thread local variables don’t work in an async non-blocking world where multiple requests are processed on the same thread.  Consequently, much of the complexity of building Zuul 2 was in teasing out dark corners where thread local variables were being used. Other challenges involved converting blocking networking logic into non-blocking networking code, and finding blocking code deep inside libraries, fixing resource leaks, and converting core infrastructure to run asynchronously.  There is no one-size-fits-all strategy for converting blocking network logic to async; they must be individually analyzed and refactored. The same applies to core Netflix libraries, where some code was modified and some had to be forked and refactored to work with async.  The open source project Reactive-Audit was helpful by instrumenting our servers to discover cases where code blocks and libraries were blocking.

We took an interesting approach to building Zuul 2. Because blocking systems can run code asynchronously, we started by first changing our Zuul Filters and filter chaining code to run asynchronously.  Zuul Filters contain the specific logic that we create to do our gateway functions (routing, logging, reverse proxying, ddos prevention, etc). We refactored core Zuul, the base Zuul Filter classes, and our Zuul Filters using RxJava to allow them to run asynchronously. We now have two types of filters that are used together: async used for I/O operations, and a sync filter that run logical operations that don’t require I/O.  Async Zuul Filters allowed us to execute the exact same filter logic in both a blocking system and a non-blocking system.  This gave us the ability to work with one filter set so that we could develop gateway features for our partners while also developing the Netty-based architecture in a single codebase. With async Zuul Filters in place, building Zuul 2 was “just” a matter of making the rest of our Zuul infrastructure run asynchronously and non-blocking. The same Zuul Filters could just drop into both architectures.

Results of Zuul 2 in Production

Hypotheses varied greatly on benefits of async architecture with our gateway. Some thought we would see an order of magnitude increase in efficiency due to the reduction of context switching and more efficient use of CPU caches and others expected that we’d see no efficiency gain at all.  Opinions also varied on the complexity of the change and development effort. 

So what did we gain by doing this architectural change? And was it worth it? This topic is hotly debated. The Cloud Gateway team pioneered the effort to create and test async-based services at Netflix. There was a lot of interest in understanding how microservices using async would operate at Netflix, and Zuul looked like an ideal service for seeing benefits. 

While we did not see a significant efficiency benefit in migrating to async and non-blocking, we did achieve the goals of connection scaling. Zuul does benefit by greatly decreasing the cost of network connections which will enable push and bi-directional communication to and from devices. These features will enable more real-time user experience innovations and will reduce overall cloud costs by replacing “chatty” device protocols today (which account for a significant portion of API traffic) with push notifications. There also is some resiliency advantage in handling retry storms and latency from origin systems better than the blocking model. We are continuing to improve on this area; however it should be noted that the resiliency advantages have not been straightforward or without effort and tuning. 

With the ability to drop Zuul’s core business logic into either blocking or async architectures, we have an interesting apples-to-apples comparison of blocking to async.  So how do two systems doing the exact same real work, although in very different ways, compare in terms of features, performance and resiliency?  After running Zuul 2 in production for the last several months, our evaluation is that the more CPU-bound a system is, the less of an efficiency gain we see.  

We have several different Zuul clusters that front origin services like API, playback, website, and logging. Each origin service demands that different operations be handled by the corresponding Zuul cluster.  The Zuul cluster that fronts our API service, for example, does the most on-box work of all our clusters, including metrics calculations, logging, and decrypting incoming payloads and compressing responses.  We see no efficiency gain by swapping an async Zuul 2 for a blocking one for this cluster.  From a capacity and CPU point of view they are essentially equivalent, which makes sense given how CPU-intensive the Zuul service fronting API is. They also tend to degrade at about the same throughput per node. 

The Zuul cluster that fronts our Logging services has a different performance profile. Zuul is generally receiving logging and analytics messages from devices and is write-heavy, so requests are large, but responses are small and not encrypted by Zuul.  As a result, Zuul is doing much less work for this cluster.  While still CPU-bound, we see about a 25% increase in throughput corresponding with a 25% reduction in CPU utilization by running Netty-based Zuul.  We thus observed that the less work a system actually does, the more efficiency we gain from async. 

Overall, the value we get from this architectural change is high, with connection scaling being the primary benefit, but it does come at a cost. We have a system that is much more complex to debug, code, and test, and we are working within an ecosystem at Netflix that operates on an assumption of blocking systems. It is unlikely that the ecosystem will change anytime soon, so as we add and integrate more features to our gateway it is likely that we will need to continue to tease out thread local variables and other assumptions of blocking in client libraries and other supporting code.  We will also need to rewrite blocking calls asynchronously.  This is an engineering challenge unique to working with a well established platform and body of code that makes assumptions of blocking. Building and integrating Zuul 2 in a greenfield would have avoided some of these complexities, but we operate in an environment where these libraries and services are essential to the functionality of our gateway and operation within Netflix’s ecosystem.

We are in the process of releasing Zuul 2 as open source. Once it is released, we’d love to hear from you about your experiences with it and hope you will share your contributions! We plan on adding new features such as http/2 and websocket support to Zuul 2 so that the community can also benefit from these innovations.

- The Cloud Gateway Team (Mikey Cohen, Mike Smith, Susheel Aroskar, Arthur Gonigberg, Gayathri Varadarajan, and Sudheer Vinukonda)

Thursday, September 15, 2016


Why IMF?

As Netflix expanded into a global entertainment platform, our supply chain needed an efficient way to vault our masters in the cloud that didn’t require a different version for every territory in which we have our service.  A few years ago we discovered the Interoperable Master Format (IMF), a standard created by the Society of Motion Picture and Television Engineers (SMPTE). The IMF framework is based on the Digital Cinema standard of component based elements in a standard container with assets being mapped together via metadata instructions.  By using this standard, Netflix is able to hold a single set of core assets and the unique elements needed to make those assets relevant in a local territory.  So for a title like Narcos, where the video is largely the same in all territories, we can hold the Primary AV and the specific frames that are different for, say, the Japanese title sequence version.  This reduces duplication of assets that are 95% the same and allows us to hold that 95% once and piece it to the 5% differences needed for a specific use case.   The format also serves to minimize the risk of multiple versions being introduced into our vault, and allows us to keep better track of our assets, as they stay within one contained package, even when new elements are introduced.  This allows us to avoid “versionitis” as outlined in this previous blog.  We can leverage one set of master assets and utilize supplemental or additional master assets in IMF to make our localized language versions, as well as any transcoded versions, without needing to store anything more than master materials.  Primary AV, supplemental AV, subtitles, non-English audio and other assets needed for global distribution can all live in an “uber” master that can be continually added to as needed rather than recreated.  When a “virtual-version” is needed, the instructions simply need to be created, not the whole master.  IMF provides maximum flexibility without having to actually create every permutation of a master.  

OSS for IMF:

Netflix has a history of identifying shared problems within industries and seeking solutions via open source tools.  Because many of our content partners have the same issues Netflix has with regard to global versions of their content, we saw IMF as a shared opportunity in the digital supply chain space.  In order to support IMF interoperability and share the benefits of the format with the rest of the content community, we have invested in several open source IMF tools.  One example of these tools is the IMF Transform Tool  which gives users the ability to transcode from IMF to DPP (Digital Production Partnership).  Realizing Netflix is only one recipient of assets from content owners, we wanted to create a solution that would allow them to enjoy the benefits of IMF and still create deliverables to existing outlets.  Similarly, Netflix understands the EST business is still important to content owners, so we’re adding another open source transform tool that will go from IMF to an iTunes-compatible like package (when using Apple ProRes encoder). This will allow users to take a SMPTE compliant IMF and convert it to a package which can be used for TVOD delivery without incurring significant costs via proprietary tools.  A final shared problem is editing those sets of instructions we mentioned earlier.  There are many great tools in the marketplace that create IMF packages, and while they are fully featured and offer powerful solutions for creating IMFs, they can be overkill for making quick changes to a CPL (Content Play List).  Things like adding metadata markers, EIDR numbers or other changes to the instructions for that IMF can all be done in our newly released OSS IMF CPL Editor.  This leaves the fully functioned commercial software/hardware tools open in facilities for IMF creation and not tied up making small changes to metadata.

IMF Transforms

The IMF Transform uses other open source technologies from Java, ffmpeg, bmxlib and x.264 in the framework.  These tools and their source code can be found on GitHub at

IMF CPL Editor

The IMF CPL Editor is cross platform and can be compiled on Mac, Windows and/or Linux operating systems.  The tool will open a composition playlist (CPL) in a timeline and list all assets.  The essence files will be supported in .mxf wrapped .wav, .ttml or .imsc files. The user can add, edit and delete audio, subtitle and metadata assets from the timeline. The edits can be saved back to the existing CPL or saved as a new CPL modifying the Packing List (PKL) and Asset Map as well.  The source code and compiled tool will be open source and available at (

What’s Next:

We hope others will branch these open source efforts and make even more functions available to the growing community of IMF users.  It would be great to see a transform function to other AS-11 formats, XDCAM 50 or other widely used broadcast “play-out” formats.  In addition to the base package functionality that currently exists, Netflix will be adding supplemental package support to the IMF CPL Editor in October. We look forward to seeing what developers create. These solutions coupled with the Photon tool Netflix has already released create strong foundations to make having an efficient and comprehensive library in IMF an achievable goal for content owners seeking to exploit their assets in the global entertainment market.

By: Chris Fetner and Brian Kenworthy

Wednesday, September 14, 2016

Netflix OSS Meetup Recap - September 2016

Last week, we welcomed roughly 200 attendees to Netflix HQ in Los Gatos for Season 4, Episode 3 of our Netflix OSS Meetup. The meetup group was created in 2013 to discuss our various OSS projects amongst the broader community of OSS enthusiasts. This episode centered around security-focused OSS releases, and speakers included both Netflix creators of security OSS as well as community users and contributors.

We started the night with an hour of networking, Mexican food, and drinks. As we kicked off the presentations, we discussed the history of security OSS at Netflix - we first released Security Monkey in 2014, and we're closing in on our tenth security release, likely by the end of 2016. The slide below provides a comprehensive timeline of the security software we've released as Netflix OSS.

Wes Miaw of Netflix began the presentations with a discussion of MSL (Message Security Layer), a modern security protocol that addresses a number of difficult security problems. Next was Patrick Kelley, also of Netflix, who gave the crowd an overview of Repoman, an upcoming OSS release that works to right-size permissions within Amazon Web Services environments.

Next up were our external speakers. Vivian Ho and Ryan Lane of Lyft discussed their use of BLESS, an SSH Certificate Authority implemented as an AWS Lambda function. They're using it in conjunction with their OSS kmsauth to provide engineers SSH access to AWS instances. Closing the presentations was Chris Dorros of OpenDNS/Cisco. Chris talked about his contribution to Lemur, the SSL/TLS certificate management system we open sourced last year. Chris has added functionality to support the DigiCert Certificate Authority. After the presentations, the crowd moved back to the cafeteria, where we'd set up demo stations for a variety of our security OSS releases.

Patrick Kelley talking about Repoman

Thanks to everyone who attended - we're planning the next meetup for early December 2016. Join our group for notifications. If you weren't able to attend, we have both the slides and video available.

Upcoming Talks from the Netflix Security Team

Below is a schedule of upcoming presentations from members of the Netflix security team (through 2016). If you'd like to hear more talks from Netflix security, some of our past presentations are available on our YouTube channel

Automacon (Portland, OR) Sept 27-29, 2016
Scott Behrens and Andy Hoernecke
AppSecUSA 2016 (DC) - Oct 11-14, 2016
Scott Behrens and Andy Hoernecke
O'Reilly Security NYC (NYC) - Oct 30-Nov 2, 2016
Ping Identify SF (San Francisco) - Nov 2, 2016
QConSF (San Francisco) - Nov 7-11, 2016
The Psychology of Security Automation
Manish Mehta
AWS RE:invent (Las Vegas) - Nov 28-Dec 2, 2016
Solving the First Secret Problem: Securely Establishing Identity using the AWS Metadata Service
AWS RE:invent (Las Vegas) - Nov 28-Dec 2, 2016

If you're interested in solving interesting security problems while developing OSS that the rest of the world can use, we'd love to hear from you! Please see our jobs site for openings.

By Jason Chan

Thursday, September 1, 2016

Netflix Data Benchmark: Benchmarking Cloud Data Stores

The Netflix member experience is offered to 83+ million global members, and delivered using thousands of microservices. These services are owned by multiple teams, each having their own build and release lifecycles, generating a variety of data that is stored in different types of data store systems. The Cloud Database Engineering (CDE) team manages those data store systems, so we run benchmarks to validate updates to these systems, perform capacity planning, and test our cloud instances with multiple workloads and under different failure scenarios. We were also interested in a tool that could evaluate and compare new data store systems as they appear in the market or in the open source domain, determine their performance characteristics and limitations, and gauge whether they could be used in production for relevant use cases. For these purposes, we wrote Netflix Data Benchmark (NDBench), a pluggable cloud-enabled benchmarking tool that can be used across any data store system. NDBench provides plugin support for the major data store systems that we use -- Cassandra (Thrift and CQL), Dynomite (Redis), and Elasticsearch. It can also be extended to other client APIs.


As Netflix runs thousands of microservices, we are not always aware of the traffic that bundled microservices may generate on our backend systems. Understanding the performance implications of new microservices on our backend systems was also a difficult task. We needed a framework that could assist us in determining the behavior of our data store systems under various workloads, maintenance operations and instance types. We wanted to be mindful of provisioning our clusters, scaling them either horizontally (by adding nodes) or vertically (by upgrading the instance types), and operating under different workloads and conditions, such as node failures, network partitions, etc.

As new data store systems appear in the market, they tend to report performance numbers for the “sweet spot”, and are usually based on optimized hardware and benchmark configurations. Being a cloud-native database team, we want to make sure that our systems can provide high availability under multiple failure scenarios, and that we are utilizing our instance resources optimally. There are many other factors that affect the performance of a database deployed in the cloud, such as instance types, workload patterns, and types of deployments (island vs global). NDBench aids in simulating the performance benchmark by mimicking several production use cases.

There were also some additional requirements; for example, as we upgrade our data store systems (such as Cassandra upgrades) we wanted to test the systems prior to deploying them in production. For systems that we develop in-house, such as Dynomite, we wanted to automate the functional test pipelines, understand the performance of Dynomite under various conditions, and under different storage engines. Hence, we wanted a workload generator that could be integrated into our pipelines prior to promoting an AWS AMI to a production-ready AMI.

We looked into various benchmark tools as well as REST-based performance tools. While some tools covered a subset of our requirements, we were interested in a tool that could achieve the following:
  • Dynamically change the benchmark configurations while the test is running, hence perform tests along with our production microservices.
  • Be able to integrate with platform cloud services such as dynamic configurations, discovery, metrics, etc.
  • Run for an infinite duration in order to introduce failure scenarios and test long running maintenances such as database repairs.
  • Provide pluggable patterns and loads.
  • Support different client APIs.
  • Deploy, manage and monitor multiple instances from a single entry point.
For these reasons, we created Netflix Data Benchmark (NDBench). We incorporated NDBench into the Netflix Open Source ecosystem by integrating it with components such as Archaius for configuration, Spectator for metrics, and Eureka for discovery service. However, we designed NDBench so that these libraries are injected, allowing the tool to be ported to other cloud environments, run locally, and at the same time satisfy our Netflix OSS ecosystem users.

NDBench Architecture

The following diagram shows the architecture of NDBench. The framework consists of three components:
  • Core: The workload generator
  • API: Allowing multiple plugins to be developed against NDBench
  • Web: The UI and the servlet context listener
We currently provide the following client plugins -- Datastax Java Driver (CQL), C* Astyanax (Thrift), Elasticsearch API, and Dyno (Jedis support). Additional plugins can be added, or a user can use dynamic scripts in Groovy to add new workloads. Each driver is just an implementation of the Driver plugin interface.

NDBench-core is the core component of NDBench, where one can further tune workload settings.

Fig. 1: NDBench Architecture

NDBench can be used from either the command line (using REST calls), or from a web-based user interface (UI).

NDBench Runner UI

Fig.2: NDBench Runner UI

A screenshot of the NDBench Runner (Web UI) is shown in Figure 2. Through this UI, a user can select a cluster, connect a driver, modify settings, set a load testing pattern (random or sliding window), and finally run the load tests. Selecting an instance while a load test is running also enables the user to view live-updating statistics, such as read/write latencies, requests per second, cache hits vs. misses, and more.

Load Properties

NDBench provides a variety of input parameters that are loaded dynamically and can dynamically change during the workload test. The following parameters can be configured on a per node basis:
  • numKeys: the sample space for the randomly generated keys
  • numValues: the sample space for the generated values
  • dataSize: the size of each value
  • numWriters/numReaders: the number of threads per NDBench node for writes/reads
  • writeEnabled/readEnabled: boolean to enable or disable writes or reads
  • writeRateLimit/readRateLimit: the number of writes per second and reads per seconds
  • userVariableDataSize: boolean to enable or disable the ability of the payload to be randomly generated.

Types of Workload

NDBench offers pluggable load tests. Currently it offers two modes -- random traffic and sliding window traffic. The sliding window test is a more sophisticated test that can concurrently exercise data that is repetitive inside the window, thereby providing a combination of temporally local data and spatially local data. This test is important as we want to exercise both the caching layer provided by the data store system, as well as the disk’s IOPS (Input/Output Operations Per Second).

Load Generation

Load can be generated individually for each node on the application side, or all nodes can generate reads and writes simultaneously. Moreover, NDBench provides the ability to use the “backfill” feature in order to start the workload with hot data. This helps in reducing the ramp up time of the benchmark.

NDBench at Netflix

NDBench has been widely used inside Netflix. In the following sections, we talk about some use cases in which NDBench has proven to be a useful tool.

Benchmarking Tool

A couple of months ago, we finished the Cassandra migration from version 2.0 to 2.1. Prior to starting the process, it was imperative for us to understand the performance gains that we would achieve, as well as the performance hit we would incur during the rolling upgrade of our Cassandra instances. Figures 3 and 4 below illustrate  the p99 and p95 read latency differences using NDBench. In Fig. 3, we highlight the differences between Cassandra 2.0 (blue line) vs 2.1 (brown line).

Fig.3: Capturing OPS and latency percentiles of Cassandra

Last year, we also migrated all our Cassandra instances from the older Red Hat 5.10 OS to Ubuntu 14.04 (Trusty Tahr). We used NDBench to measure performance under the newer operating system. In Figure 4, we showcase the three phases of the migration process by using NDBench’s long-running benchmark capability. We used rolling terminations of the Cassandra instances to update the AMIs with the new OS, and NDBench to verify that there would be no client-side impact during the migration. NDBench also allowed us to validate that the performance of the new OS was better after the migration.

Fig.4: Performance improvement from our upgrade from Red Hat 5.10 to Ubuntu 14.04

AMI Certification Process

NDBench is also part of our AMI certification process, which consists of integration tests and deployment validation. We designed pipelines in Spinnaker and integrated NDBench into them. The following figure shows the bakery-to-release lifecycle. We initially bake an AMI with Cassandra, create a Cassandra cluster, create an NDBench cluster, configure it, and run a performance test. We finally review the results, and make the decision on whether to promote an “Experimental” AMI to a “Candidate”. We use similar pipelines for Dynomite, testing out the replication functionalities with different client-side APIs. Passing the NDBench performance tests means that the AMI is ready to be used in the production environment. Similar pipelines are used across the board for other data store systems at Netflix.
Fig.5 NDBench integrated with Spinnaker pipelines

In the past, we’ve published benchmarks of Dynomite with Redis as a storage engine leveraging NDBench. In Fig. 6 we show some of the higher percentile latencies we derived from Dynomite leveraging NDBench.
Fig.6: P99 latencies for Dynomite with consistency set to DC_QUORUM with NDBench

NDBench allows us to run infinite horizon tests to identify potential memory leaks from long running processes that we develop or use in-house. At the same time, in our integration tests we introduce failure conditions, change the underlying variables of our systems, introduce CPU intensive operations (like repair/reconciliation), and determine the optimal performance based on the application requirements. Finally, our sidecars such as Priam, Dynomite-manager and Raigad perform various activities, such as multi-threaded backups to object storage systems. We want to make sure, through integration tests, that the performance of our data store systems is not affected.


For the last few years, NDBench has been a widely-used tool for functional, integration, and performance testing, as well as AMI validation. The ability to change the workload patterns during a test, support for different client APIs, and integration with our cloud deployments has greatly helped us in validating our data store systems. There are a number of improvements we would like to make to NDBench, both for increased usability and supporting additional features. Some of the features that we would like to work on include:
  • Performance profile management
  • Automated canary analysis
  • Dynamic load generation based on destination schemas
NDBench has proven to be extremely useful for us on the Cloud Database Engineering team at Netflix, and we are happy to have the opportunity to share that value. Therefore, we are releasing NDBench as an open source project, and are looking forward to receiving feedback, ideas, and contributions from the open source community. You can find NDBench on Github at:

If you enjoy the challenges of building distributed systems and are interested in working with the Cloud Database Engineering team in solving next-generation data store problems, check out our job openings.

Authors: Vinay Chella, Ioannis Papapanagiotou, and Kunal Kundaje