Postgres Bulk Loading: From 8 Hours to 5 Minutes, Databricks Bypasses the Single-Writer Bottleneck

Postgres excels at high-concurrency, low-latency online queries, but bulk loading has always been a pain point. Pour several terabytes of data into the primary node and CPU spikes, memory gets tight, WAL writes surge, and online applications suffer. Teams can only schedule imports for the middle of the night and manually manage checkpoints and backfills.

Databricks changes this with the LTAP architecture in Lakebase Postgres. LTAP (Lake Transactional/Analytical Processing) allows the transaction engine and analytical engine to operate directly on the same copy of data in the lake. The core idea in one sentence: bulk data no longer writes through the primary node.

Spark builds valid Postgres data pages directly on separate compute, writes them to object storage, and the primary node is only responsible for final publication. Below is the official Direct writes flow diagram.

Databricks' official blog provides specific numbers.

In beta, a customer loads about 1 billion rows per day into Lakebase via Synced Tables. This batch used to take more than 8 hours, with CPU and memory completely saturated. Under the LTAP architecture, loading no longer competes for primary-node resources, and online queries are unaffected.

In internal benchmarks, loading 1 TB now takes under 5 minutes. Compared with traditional bulk loading, that is up to a 147x speedup.

Load throughput can scale approximately linearly as compute resources increase.

Why is traditional Postgres so slow?

The native COPY command is fine for small data volumes, but once data volumes grow, all imported rows must enter through the single primary writer. Even if Spark runs distributed tasks, Spark only reads the source data in parallel; every row ultimately still has to go through the same Postgres write process. The primary node turns rows into heap pages and index pages, writes the write-ahead log (WAL), flushes to durable storage, and then commits. The target side is a single point and cannot scale horizontally.

The bigger problem is that this primary node is also serving online queries. Imports and business traffic compete for CPU, memory, I/O, connections, and WAL bandwidth. When latency rises, teams either over-provision resources or schedule imports into off-peak periods. Resources are often sized for the maximum import scale rather than daily traffic.

LTAP changes this.

Postgres transaction compute is stateless; persistent state is not on the primary node's local disk but in the distributed storage layer. The primary node is no longer the only way to create persistent state. Put another way: Postgres becomes a client of storage. It serves queries and transactions, but it does not necessarily have to materialize every new page itself.

During bulk loading, Spark is responsible for building Postgres state and writing it to storage, and the primary node only publishes the result. This brings several practical changes.

Bulk loading and OLTP no longer compete for resources. Loading runs on separate Spark compute, while application traffic continues to get the CPU, I/O, and WAL bandwidth it needs.

Multiple large loads can run concurrently. At the end of each import, the primary node writes only one compact WAL record; imports do not have to queue behind each other's COPY streams.

The primary node does not need to be scaled for load size. A small primary node can serve traffic while waiting for Spark to load billions of rows, and Spark compute is released once it is done.

Next are the technical details: how Spark builds valid, safe Postgres pages.

Building heap pages

Each Spark executor starts a sandbox Postgres running in binary-upgrade mode. This is the same mechanism pg_upgrade uses to preserve catalog OIDs across major-version upgrades. The target database's catalog OIDs are transplanted into the sandbox to ensure that the OIDs embedded in generated pages match those of the target database. The Spark driver assigns non-overlapping OID ranges to sandboxes, so concurrently created objects do not collide.

Each sandbox runs a binary COPY plus FREEZE. FREEZE marks imported tuples as committed, so Postgres can consider these tuples visible without consulting transaction history.

Each worker also computes a checksum for the generated pages. Lakebase's pageserver verifies them when receiving files, so corrupted data is detected before it takes effect.

Sandboxes are disposable and thrown away after use, which brings an additional benefit: UNLOGGED auxiliary tables can be used to reduce WAL and lock contention. This is a performance optimization, not a safety mechanism.

Is this still "just Postgres"?

Yes. What Spark writes are standard Postgres pages, not an import format that the primary node later converts. Databricks added extensions and table access methods along this path without changing the Postgres core. This is precisely a demonstration of Postgres extensibility.

Building indexes

A B-tree is an ordered structure over the entire key space, and leaf entries contain (key, ctid) pairs that point to specific locations in the heap. Normally Postgres obtains these pairs by scanning the heap. But here the heap is spread across uploaded slices; downloading everything to the indexing worker would cancel out the benefits of parallel building.

Index building does not actually need heap contents, only a stream of (key, ctid). Each heap worker exports the key columns and tuple ID to object storage files while building slices. These records feed into an auxiliary table containing only key columns and tuple IDs. During CREATE INDEX, a custom table access method scans the auxiliary table like heapam, but writes index entries using the original heap's ctid rather than the auxiliary table row's own physical location. When loading records, each slice-local ctid is offset by the cumulative size of preceding heap slices, ultimately pointing to the correct position in the concatenated heap.

The result is that the standard Postgres B-tree builder scanned a heap it never downloaded and built the index. Only the (key, ctid) representation moves, far smaller than the entire heap.

Handing off to Postgres

Once heap and index slices are ready, the Spark driver writes a manifest and calls a SQL function on the target side. The primary node records the import as one compact WAL record. Traditional COPY sends the full data volume to the safekeeper quorum; here only the import description is sent.

The pageserver claims the uploaded files as authoritative pages, verifies checksums, and then warms them to local SSD so initial reads do not need cold access to object storage. Finally, the primary node atomically switches over, and the staged data becomes a user-visible Synced Table.

Before commit, imported tables are fully isolated. After commit, what the primary node sees is ordinary heap and B-tree pages, in standard Postgres format and interfaces.

Currently, the LTAP architecture is used for Lakebase Synced Tables, rapidly syncing Gold-level Lakehouse data to applications. The company says it will next extend this protocol to large-scale index builds and database migrations.

In the future, running large bulk imports may no longer require waiting for a midnight window.