Encryption

Encryption covers the whole database: pages, indexes, the catalogue, the journal. Add a password to the connection string and everything that reaches the disk is ciphertext.

Data Source=app.witdb;Encryption=aes-gcm;Password=your-password

Or through the builder:

var db = new WitDatabaseBuilder()
    .WithFilePath("app.witdb")
    .WithEncryption("your-password")
    .WithBTree()
    .WithTransactions()
    .Build();

What the file carries

An encrypted database starts with a 128-byte plaintext preamble sitting in a physical page of its own, in front of everything else. It holds a magic of its own, a format version, the key derivation in use, the iteration count, sixteen random bytes of salt, the next unused nonce sequence number, and the data key wrapped under your password.

All of it is public. The password is the only secret.

Having those values in the file rather than in your connection string is what makes the rest work. The database knows how its own key was made, so opening it needs the password and nothing else. No setting can be forgotten, and a file written with fewer iterations for a slow environment opens anywhere without being told.

Two keys, not one

Pages are encrypted under a random 256-bit data key. Your password never touches them. It derives a wrapping key through PBKDF2-SHA256, and that unwraps the data key from the header.

The arrangement pays off in three places.

Changing a password rewrites 60 bytes. The data key stays as it is, so every page of the database stays as it is. On a file of any size the operation finishes immediately.

Raising the iteration count is the same operation. New databases are created at 600,000 iterations, the current OWASP figure for PBKDF2-HMAC-SHA256, and an existing database can be moved up without being rewritten.

A wrong password gets an honest answer. The wrapping key fails to unwrap, the authentication tag on the wrapped key says so, and the error names the password. Nothing further in the engine is consulted, so you never see a decryption failure from a layer that knows nothing about passwords.

Bringing your own key

When key material comes from somewhere else, a vault, an HSM, a derivation your organisation mandates, pass the bytes instead of a password:

var key = await vault.GetKeyAsync("app-database");   // 32 bytes

var db = new WitDatabaseBuilder()
    .WithFilePath("app.witdb")
    .WithEncryption(key)
    .Build();

The header then carries a salt and a nonce sequence and no wrapped key, since there is nothing to wrap. You own the key and its lifetime, and the engine says so if someone tries to open that database with a password.

Choosing an algorithm

AES-GCM is the default and is the faster of the two wherever the processor implements AES in hardware. Without that, software AES is slow enough to notice, and ChaCha20-Poly1305 is designed to run well on general-purpose instructions. It arrives with OutWit.Database.Core.BouncyCastle:

var db = new WitDatabaseBuilder()
    .WithFilePath("app.witdb")
    .WithBouncyCastleEncryption("your-password")
    .Build();

Both are authenticated. A modified page fails its tag and the read fails rather than returning altered data.

Nonces

Every encryption under a key needs a nonce it has never used before. Under AES-GCM, reusing one is not a degradation, it is a break: an attacker holding two ciphertexts written under the same nonce recovers the second plaintext without the key.

WitDatabase builds each nonce from the page number and a sequence number. The sequence alone makes it unique, and the page number is bound in so that a page lifted to a different offset is refused before its tag is examined.

The sequence belongs to the file. Opening the database reserves a block of numbers by advancing the header on disk and flushing it before handing out a single one, so a process that is killed loses the unused remainder of its block and nothing else. The next session starts above every number the last one could have used. The remainder is never handed back on close, because writing a smaller number than the one on disk is the one operation whose torn write could cause reuse.

An encrypted database holds at most 2^32 pages, which is 17 TB at the default page size. Past that the engine refuses the write rather than letting two pages share a nonce prefix.

Performance

Encryption costs roughly 10 to 15 per cent on reads and writes with hardware AES. Deriving the key happens once when the database opens and takes a few hundred milliseconds at 600,000 iterations. For environments where that is felt, mainly WebAssembly, a database can be created at 10,000 iterations, and it will open anywhere without the caller knowing.

Two things reduce the cost more than any tuning. A larger page cache means fewer pages read from disk, and every page that stays in memory is decrypted once. And a bigger page size amortises the per-page tag and nonce over more data.

Databases encrypted before 13.1.0

Such a database is refused, and the refusal is about its encryption rather than its age. Three things are wrong with it that reading it cannot repair:

• its salt is SHA256(password + suffix), a pure function of the password, so one password means one key across every database ever created with it;
• that salt is also the file's first eight bytes in the clear, which makes the head of the file a password verifier costing one SHA-256, about 41,000 times cheaper than PBKDF2 at 100,000 iterations on one core and far more on a GPU;
• its nonce counter is set to zero on every open, so two sessions encrypt different plaintext under one nonce, which is the failure AES-GCM has no recovery from.

Version 13.1.0 kept opening such files. Since 14.0.0 the engine refuses and says so, naming both ways forward.

Read it as it is, which is what a converter does and what gets your data out:

new WitDatabaseBuilder()
    .WithFilePath(path)
    .WithBTree()
    .WithEncryption(password)
    .WithLegacyEncryption()
    .Build();

Data Source=app.witdb;Password=your-password;Legacy Encryption=true

Then change the password, which rewrites the database in the current format and ends the need for the opt-in. Studio does this from its Database tab and offers the opt-in itself when it meets such a file.

The refusal is narrow. A database written by 13.1.0 or later opens with no opt-in, and an unencrypted database is untouched, which matters because an unencrypted file looks exactly like a legacy one to the preamble check. The branch is only reached when encryption was asked for.

LegacyEncryptionException is a type of its own, so a caller can act on the refusal without matching its message.

Older builds and newer databases

A database written by 13.1.0 or later has its pages shifted one physical page along by the preamble, so an earlier build cannot read it. Keep a copy before upgrading a database you may need to open with an older version of your application.

What encryption does not do

It protects a file at rest. Someone who takes the file gets ciphertext, and someone who alters it gets a failed authentication tag rather than altered data.

While the database is open, pages are decrypted in memory, and anything that can read your process's memory can read them. Encryption is also not access control: it is one password for the whole database, with no notion of users or of rights over particular tables. If you need those, they belong in your application.

Where to go next

• Architecture, replacing ICryptoProvider with your own
• Connection strings, every encryption parameter
• File format, the preamble and how format versions relate