Why DriftDB 0.9 changes the on-disk format
Format v2 shrinks files by 40% and makes crash recovery safe. Here is why we broke compatibility and how to migrate.
On this page3 sections
DriftDB 0.9 writes a new on-disk format. Files written by 0.9 can’t be opened by 0.8, and 0.9 opens 0.8 files read-only until you migrate them. We don’t break formats lightly, so this post explains why we did.
What was wrong with v1
Format v1 stored one header per block with the sensor id written out in full. With short blocks and long ids, headers took up to 30% of the file. Worse, the block index was only written on clean shutdown. After a crash, DriftDB rebuilt it by scanning the whole file, which took minutes on large databases.
What v2 changes
- Series dictionary. Series ids are stored once per file and blocks point to them by number. Files are 40% smaller on average.
- Write-ahead index. Index changes go to a small log before the block is written. Recovery after a crash now takes under a second, whatever the file size.
- Checksums on every block. A corrupted block is skipped and reported instead of failing the whole query.
How to migrate
Upgrade the crate
Run cargo add driftdb@0.9, or set the version in
Cargo.tomlby hand.Back up your data directory
Migration rewrites files in place. Copy the directory first.
Run the migration
Call
Db::migrateonce at startup, or use the CLI. It runs at about 1 GB per second.
Rust
use driftdb::Db;
let db = Db::open("./data")?;
if db.format_version() < 2 {
db.migrate()?;
}CLI
driftdb migrate ./dataThe full list of changes is in the 0.9.0 release notes.