On this page3 sections

DriftDB 0.9 writes a new on-disk format. Files written by 0.9 can’t be opened by 0.8, and 0.9 opens 0.8 files read-only until you migrate them. We don’t break formats lightly, so this post explains why we did.

What was wrong with v1

Format v1 stored one header per block with the sensor id written out in full. With short blocks and long ids, headers took up to 30% of the file. Worse, the block index was only written on clean shutdown. After a crash, DriftDB rebuilt it by scanning the whole file, which took minutes on large databases.

What v2 changes

  • Series dictionary. Series ids are stored once per file and blocks point to them by number. Files are 40% smaller on average.
  • Write-ahead index. Index changes go to a small log before the block is written. Recovery after a crash now takes under a second, whatever the file size.
  • Checksums on every block. A corrupted block is skipped and reported instead of failing the whole query.

How to migrate

  1. Upgrade the crate

    Run cargo add driftdb@0.9, or set the version in Cargo.toml by hand.

  2. Back up your data directory

    Migration rewrites files in place. Copy the directory first.

  3. Run the migration

    Call Db::migrate once at startup, or use the CLI. It runs at about 1 GB per second.

Rust

use driftdb::Db;

let db = Db::open("./data")?;
if db.format_version() < 2 {
    db.migrate()?;
}

CLI

driftdb migrate ./data

The full list of changes is in the 0.9.0 release notes.