File format

file-formatcompatibilityversioninginternals

If a .witdb file is only ever a scratch database for your test suite, none of this matters: it is created and thrown away in the same minute. It matters when the file is a document your users keep, because then it outlives the version of your application that wrote it.

What is in the file

A database using the B+Tree store is one file, divided into pages. The default page size is 4 KB, and it can be any power of two from 512 bytes to 64 KB, fixed when the database is created.

Page 0 is the header, 128 bytes of it followed by unused space:

Bytes Holds
0 to 15 The magic, WitDB Format 1
16 to 17 Format version, major and minor
18 to 19 Page size
20 to 31 Page count, first free page, free page count
32 to 35 Root page of the catalogue
36 to 47 Transaction counter, flags, checkpoint counter
48 to 127 Which providers the database was created with

Everything after page 0 is the database: the catalogue that describes tables, indexes, views, triggers and routines, then the pages holding your rows and index entries.

That last header region is worth knowing about. A database records the storage engine, the encryption, the page cache and the journal it was made with, as text keys rather than as numbers. Text because the provider registry is open: a cache or a journal you registered yourself has a key of its own, and an enumeration would have quietly closed the door on it.

Versions

The format version is a major and a minor. The rule they follow is short.

A file whose major version is higher than the build understands is refused, with an error that says so. The alternative is worse than a refusal: the magic bytes would still match, and the build would read a layout it does not know as though it were the layout it does, arriving at page numbers and offsets that mean something else.

Minor versions read in both directions. When the provider region grew to record the cache and the journal, the minor went up and nothing else changed. An older build reads the part it knows and ignores the rest. A newer build reading an older file finds zeros where the new fields would be, reads that as "not recorded", and falls back to the defaults that file was always opened with.

Encrypted databases

An encrypted database has a second header in front of the first: a plaintext preamble occupying a physical page of its own, carrying the salt, the iteration count, the nonce sequence and the wrapped data key. Every logical page is shifted one physical page along to make room. See encryption.

The first sixteen bytes tell three cases apart without needing a password. The crypto magic means an encrypted database in the current format. The database magic means an unencrypted one. Anything else means a database encrypted before 13.1.0, which the engine refuses since 14.0.0 unless the old scheme is asked for by name. See encryption.

Encrypted databases written by 13.1.0 or later cannot be opened by 12.8.0 or earlier, which is what the major version bump in that release records. An older build meets a preamble page where it expects ciphertext. Unencrypted databases are unaffected and byte for byte identical.

The other layouts

The LSM store is a folder rather than a file. Sorted string tables accumulate in it and compactions merge them. There is no page 0 to put a crypto header in, so an encrypted LSM store keeps one in crypto.hdr beside the tables.

Write-ahead logging adds .witdb-wal next to the database. It holds committed work not yet folded into the main file, and a checkpoint folds it in. Keep the two files together: a database separated from its log has lost whatever had not been checkpointed.

In-memory databases have no format, since nothing is written down.

Making a copy

A database is a file, and copying it is copying the file. Two things have to be true first.

Nothing may be holding it. Copy with the database closed, or through Studio, which checkpoints and flushes before it copies.

The journal has to come with it, or be folded in first. app.witdb-wal holds committed work not yet written into the main file, so a database separated from its log has lost whatever had not been checkpointed.

From code, fold the log in and flush the cache, then copy one file:

csharp
db.Checkpoint();
db.Flush();

File.Copy("app.witdb", "backup.witdb");

For an LSM database, copy the folder, and force a compaction first so there is less of it.

Practical advice for document formats

If your application saves user work in a .witdb file, three things are worth deciding early.

Keep a copy before an upgrade writes to the file. Once your application has opened a document with a newer engine, an older version of your application may no longer be able to open it. That is only a problem if a user can downgrade, and users do downgrade.

Record which engine version wrote a document, in a table of your own. The format version answers what the layout is; it does not tell you which of your releases produced the file, which is the question you will have during a support call.

Do not depend on the byte layout. These pages describe what a database contains so that you can reason about compatibility and file size. The layout changes between releases, and reading it yourself means owning that.

Reading a file you have been sent

WitDatabase Studio opens a .witdb and reports the format version, the storage engine, whether it is encrypted and which providers it was created with, before it opens anything. When a customer sends you a document that will not load, that is the fastest way to find out why. No third-party database client can open the file at all.

Where to go next