Lakehouse

Iceberg maintenance: four jobs, in order

, 2 minute read

A streaming writer committing every minute creates 1,440 snapshots a day and far more small files. Iceberg tolerates that for a while. Then planning gets slow, queries open thousands of files, and object storage keeps growing even though the logical data doesn't.

We run four procedures, in this order, from one nightly Spark job.

Compact data files

CALL lake.system.rewrite_data_files(
  table   => 'db.events',
  where   => 'ts < current_date()',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5',
    'partial-progress.enabled', 'true'
  )
);

The where clause keeps the job out of today's partition, which the streaming writer is still committing to. Rewriting it invites commit conflicts for no gain. Partial progress commits in groups, so a failure halfway through doesn't throw away an hour of work.

We started with the sort strategy and went back to binpack. Sorting helped a few queries and doubled the runtime; a write order on the table gave us most of the benefit for free.

Expire snapshots

CALL lake.system.expire_snapshots(
  table       => 'db.events',
  older_than  => TIMESTAMP '2026-08-18 00:00:00',  -- the job computes now() - 3 days
  retain_last => 50
);

Files replaced by compaction only become deletable once no snapshot references them. Three days of history covers our time-travel needs and leaves room to debug a bad write.

Remove orphan files

CALL lake.system.remove_orphan_files(
  table      => 'db.events',
  older_than => TIMESTAMP '2026-08-18 00:00:00'
);

Orphans are files that no metadata points to: leftovers from failed writes and aborted compactions. The margin must be longer than your longest-running write, or you will delete files a job is about to commit.

Rewrite manifests

CALL lake.system.rewrite_manifests('db.events');

Cheap and often skipped. After a month of minute-level commits, planning was reading hundreds of tiny manifests; one rewrite brought it from seconds back to milliseconds.

The order matters. Compaction makes old files unreferenced by new snapshots, expiry makes them deletable, and orphan removal catches what expiry can't see.

Updated 2 September 2026: the orphan cleanup margin went from one day to three after a backfill job ran longer than expected.