At QCon San Francisco 2025, Netflix Data Platform software engineers Rajasekhar "Raj" Ummadisetty, from Online Datastores, and Ken Kurzweil, from Data Movement, presented CloudStream: their effort to move terabytes of data from offline warehouses into low-latency, random-access serving systems. InfoQ published the 49-minute recording and transcript on July 9, 2026. These are learning notes from their presentation, not a claim that the same design is appropriate for every dataset.
The headline results reported by the speakers were a 90% reduction in data deployment time and a 70% cost reduction. One large immutable-data use case did better: deployment fell from several days to under 40 minutes, a 99% reduction, with the final traffic switch taking seconds. Those figures describe Netflix's workloads and should not be treated as general benchmarks.
Presentation Notes
Confidence Is Currency
Ummadisetty and Kurzweil framed the project around stakeholder confidence rather than speed alone. They built that confidence from three properties:
- Safety: protect live systems through throttling, graceful error handling, small blast radii, and reversible changes.
- Observability: expose progress and system state rather than presenting migration as an opaque pass/fail job.
- Validation: prove that data and behavior are correct before changing the source of truth.
This framing came from an earlier Cassandra migration. Netflix had hundreds of Cassandra clusters and thousands of tables serving live traffic, but its first copying system was a black box: operators knew only whether a transfer had completed or failed. Copying too aggressively could also overload either the source or target cluster.
The team retrofitted the migration system to monitor both sides and throttle when metrics, including p90 read and write latency, departed from normal. This safety mechanism also improved throughput because the copier could consume spare capacity off-peak and slow down during peaks. Adding completion percentages and duration estimates made it possible to forecast migrations and identify clusters that needed more capacity. After dual-writing, the team copied historical data, performed parity checks, switched reads, stopped writes to the old cluster, and finally decommissioned it.
The Q&A adds an important correctness detail: all mutating APIs in the key-value abstraction carried idempotency tokens. Kurzweil said their greatest concern was not an obvious loss of millions of records, but a few missing records whose effects might remain hidden for months. Throughput, latency, and error rates were operational signals; parity checking was the data-correctness gate.
The Abstraction Layer Made Change Possible
Netflix applications generally access storage through data abstractions rather than database-specific APIs. At the time of the talk, the speakers said:
- Only 16% of the Cassandra fleet was accessed directly by applications, and that share was decreasing.
- The abstraction fleet handled about 70 million queries per second.
- The largest abstraction handled about 8 million queries per second.
- The busiest Cassandra cluster handled about 1.8 million reads and 4.9 million writes per second.
The key-value (KV) abstraction presents a distributed two-level map: a string ID maps to a sorted map of key-value items. Its high-level client provides capabilities such as SLO-based hedging, near caching, and chunking for objects too large to send safely to Cassandra. The platform team can choose and operate the backing storage, add caches for read-heavy access, and centralize authentication and authorization without exposing those decisions to application code.
This indirection had already paid for itself when Cassandra deprecated Thrift. Application teams migrated once to the KV API; Netflix could then change schemas and backing stores behind that interface. API adoption did not initially require moving data, so teams could test the abstraction against their real data and roll back easily. The later storage migration remained the platform team's problem.
The speakers acknowledged the cost. The abstraction adds another layer and therefore latency, but their application teams valued its capabilities more than that overhead. They also cautioned that a narrow abstraction cannot necessarily represent a broad interface such as SQL. The useful lesson is not "abstract every datastore," but to narrow the API only where real access patterns permit it.
Why Record-at-a-Time Loading Broke Down
Application teams generated offline datasets for advertising signals and metadata, precomputed recommendation rankings, user-activity features, embeddings, and other machine-learning uses. The conventional pipeline used batch compute such as Spark to read warehouse output and write individual records through the online datastore's front door.
For high-order-terabyte datasets, that approach could take days despite substantial spending. The ingestion traffic contended with application traffic, creating a noisy-neighbor problem that the speakers likened to teams DDoSing their own applications. Permanent over-provisioning left expensive idle capacity between loads, while backpressure made fresh versions take longer to become available. During the long load, applications also served a mixture of old and new records, visible as a gradual "sawtooth" transition between dataset versions.
The key observation was that these workloads were not all alike:
- Mutable datasets receive both application writes and batch updates.
- Immutable datasets are produced by batch jobs and only read by the serving application.
Immutable datasets did not need Cassandra's write and consensus behavior. Netflix could perform the expensive transformation offline, convert the dataset to RocksDB SST files, stage them in S3, and load them directly onto dedicated KV nodes. A new node set could bootstrap at disk and network speed without competing with production reads; routing could then move atomically to the new version and the old nodes could be removed later.
From Proof of Concept to Pathfinder
A proof of concept implemented the KV API over offline-generated data, with a coordinator supplying routing information to clients. Its performance and cost changed skeptical stakeholders' minds, illustrating the speakers' maxim that "a POC is worth 100 meetings." The production roadmap, however, required Netflix's stateless provisioning and deployment machinery to learn stateful operations.
To meet an urgent business deadline without pretending the prototype was the final platform, the team created what Netflix calls a Pathfinder: production software intentionally designed to be migrated away from. A willing application team received a capability it otherwise could not have; the platform teams learned unknown failure modes from real use.
The durable boundary came first. Netflix moved name resolution and routing behind the existing high-level client and a feature flag. The client could target conventional KV, the Pathfinder, or the future stateful KV implementation without an application code change. The team also benchmarked deployment time, throughput, and latency against the simple Pathfinder. Production KV initially failed to match it, so those benchmarks drove bottleneck removal until the production implementation caught up; some improvements benefited all KV users.
Kurzweil clarified that Netflix did not choose a trivial pilot. The participating application had a large, mission-critical workload and both a strong need for the feature and an appetite to collaborate through risk. Solving a demanding representative case gave the team more evidence than a series of easy demos would have.
Turning Stateless KV into Stateful Serving
In the original architecture, interchangeable stateless KV servers translated requests into database queries. In the stateful design, every server holds a distinct subset of the data. Netflix added:
- A partition group manager, from which each server acquires a unique lease defining the partitions it owns.
- A central routing manager, which tracks stateful namespaces and the nodes hosting their partitions.
- A cluster routing layer, which caches routing for a KV cluster and relays it to application clients.
- Client-side routing, which sends a request to a node that owns the required data.
A data deployment starts with application-level "desires," including expected reads per second and dataset size. The platform translates these into infrastructure choices such as instance type, node count, replica count, and dataset version. It then provisions an autoscaling group. Each node discovers its namespace, leases a partition assignment, downloads the corresponding files from S3, and bootstraps.
Speed changed the risk model. The old, gradual load gave teams hours to notice bad data while only part of a new version was live. The new system could expose an entire version in seconds, increasing the immediate blast radius of corruption. Netflix therefore made staging and validation part of the deployment protocol: an application-owned validator can query the staged cluster before promotion. After approval, the platform updates routing and watches requests drain from the old version. The old dataset remains available for an immediate routing rollback until the team has enough confidence to destroy it.
The Pathfinder-to-production migration used the same reversible approach. Netflix first replaced the Pathfinder coordinator with KV cluster routing while requests still went to Pathfinder nodes. It then loaded the same dataset onto production KV nodes and changed routing again. Each phase could be reversed independently, and the application team made no code or configuration changes.
Capture, Conversion, Deployment
The teams generalized CloudStream into three independently owned phases:
- Capture: produce an immutable artifact representing the source dataset at a known point. Multiple captures can also be compared to create deltas.
- Conversion: transform that artifact into a target-specific deployment format. For the immutable KV implementation, this meant RocksDB SST files.
- Deployment: provision capacity, transfer and stage artifacts, validate them, and change routing deterministically.
Separating deployment from capture and conversion makes old artifacts directly redeployable, so rollback and roll-forward do not require rebuilding data. It also gives teams explicit contracts at phase boundaries: Netflix's Data Movement, KV, and UX teams could develop and validate their components independently. A better capture method or a new target converter can be introduced without rebuilding the whole pipeline.
RocksDB was not an arbitrary choice. The team evaluated alternatives including Netflix Hollow, SQLite, Cassandra SSTables, and Memcached's extstore format. Kurzweil said RocksDB met their criteria for external support and documentation, dynamic loading, and online compaction. SQLite met most needs but did not provide the required table-merging/compaction behavior. They did not evaluate ClickHouse.
Reported Outcomes and Unfinished Work
The speakers described three outcomes:
- A massive immutable dataset that had taken days to load on infrastructure costing millions of dollars annually could be deployed in under 40 minutes and switched in seconds, reducing deployment time by 99% and cost by 70%.
- A smaller but frequently updated feature dataset loaded in under 10 minutes, with 90% operational savings and new rollback and roll-forward support.
- An application team built an ML feature platform on top of the resulting deployment capability.
The Pathfinder itself loaded the large dataset in 27 minutes. Production hardening raised that to about 40 minutes, a useful reminder that the fastest prototype is not the production target when safety and operability matter.
At presentation time, extending the method to mutable data was future work under active testing, not a reported production result. Netflix was using the Apache Cassandra Analytics project to convert captures into Cassandra SSTables and import them into live clusters. Unlike replacing an immutable node set, this path must merge new state through loading, compaction, and conflict resolution, so Kurzweil did not claim equivalent gains.
Synthesis
The following are my conclusions from the presentation rather than statements made verbatim by Netflix:
- Treat a bulk dataset as a release artifact when its access pattern allows it. Record mutation APIs are a poor deployment protocol for a complete, versioned, read-only dataset. Immutable artifacts make staging, promotion, rollback, and provenance much easier to reason about.
- Faster cutovers require stronger pre-production validation. Eliminating a slow mixed-version transition removes incidental canary time. Validation cannot disappear with it; it must move earlier and become an explicit gate.
- An abstraction earns its cost by absorbing change. The extra hop is justified here because the interface contained migrations, idempotency, routing, chunking, and datastore selection. An abstraction that only renames a database API would not provide the same leverage.
- Progress reporting is part of correctness for long operations. A migration can copy every record correctly and still be operationally unusable if nobody can forecast completion, capacity, or risk. Observability turned a black box into something stakeholders could plan around.
- Design temporary production systems around an exit seam. The Pathfinder worked because client-side indirection existed before the temporary backend was adopted. "Temporary" without a tested migration boundary tends to become permanent.
- Optimize from the access pattern, not the incumbent datastore. The decisive step was recognizing that immutable serving did not require the mutable store's guarantees. The mutable extension remains harder precisely because those guarantees return.
- Talk to stakeholders before choosing the optimization. The speakers found the mutable/immutable distinction by interviewing application teams about real use cases. Their explicit lesson was that understanding access patterns and stakeholder constraints is where the innovation begins.
Canonical source: Accelerating Netflix Data: a Cross-Team Journey from Offline to Online, presented by Rajasekhar Ummadisetty and Ken Kurzweil at QCon San Francisco 2025.