Architecture

WitDatabase is six interfaces stacked on top of each other, each with implementations registered under a key, and each replaceable on its own without disturbing the rest.

Browser support and ChaCha20 encryption ship as separate packages, written through the public interfaces on this page and living outside the core. Nothing in the engine was changed to accommodate either of them.

The layers

Read the table downward and you have the path a byte takes. A row goes into the storage engine, which asks the page cache for a page, which reads it through the storage backend, which may be encrypting on the way. The journal sits alongside, recording enough to undo or replay. Each of those steps is a call through an interface, which is why each of them can be answered by something you wrote.

For what each layer does in detail, and for the checklist of things an implementation has to get right, see the article for that layer. This page is about the wiring.

The registry

Every pluggable component implements IProvider, which asks for one thing:

public interface IProvider
{
    string ProviderKey { get; }
}

Implementations are registered against their interface and a key, and created back out by the same pair. The registry is a thread-safe singleton, and keys are compared without regard to case.

using OutWit.Database.Core.Providers;

ProviderRegistry.Instance.Register<ICryptoProvider>(
    "my-crypto",
    parameters => new MyCryptoProvider(parameters.GetRequired<byte[]>("key")));

var keys = ProviderRegistry.Instance.GetRegisteredKeys<ICryptoProvider>();
// aes-gcm, chacha20-poly1305, my-crypto

Register throws if the key is already taken. RegisterOrReplace overwrites, which is how you substitute one of the built-ins.

Parameters arrive as a small typed bag rather than a constructor signature, because the registry cannot know what any given implementation needs:

var parameters = new ProviderParameters()
    .Set("bucket", "prod-database")
    .Set("region", "eu-west-1");

var bucket = parameters.GetRequired<string>("bucket");
var region = parameters.Get("region", "us-east-1");

Get returns the default when a parameter is absent, and throws when a parameter is present but unreadable as the type asked for.

Registering without being asked

A provider in its own assembly can register itself when that assembly loads, which means the consuming application only has to reference the package.

using System.Runtime.CompilerServices;

public static class MyProviderRegistration
{
    private static bool m_initialized;

    [ModuleInitializer]
    public static void Initialize()
    {
        if (m_initialized)
            return;

        ProviderRegistry.Instance.Register<IStorage>(
            "s3",
            p => new S3Storage(p.GetRequired<string>("bucket"),
                               p.Get("region", "us-east-1")));

        m_initialized = true;
    }
}

This is how the built-in providers register themselves, and how the IndexedDB and BouncyCastle packages do it.

One trap, and it catches everybody once. A module initializer runs when the assembly is loaded, and an assembly nothing has touched is never loaded. Calling an extension method the package defines loads it, so WithBouncyCastleEncryption(...) works. Naming the same provider in a connection string does not: nothing has referenced the assembly, so the initializer has not run and Open refuses with Encryption provider 'chacha20-poly1305' is not registered.

Call the package's registration once at startup when you configure by string:

BouncyCastleProviderRegistration.EnsureRegistered();

The same applies to any provider registered from a module initializer, including your own.

A key that is not registered is refused with the keys that are, so the message says what was available rather than only what was missing.

Selecting one

Either by key, with parameters, or by handing over an instance you built yourself:

var db = new WitDatabaseBuilder()
    .WithStorage("s3", new ProviderParameters().Set("bucket", "prod-database"))
    .WithEncryption("azure-keyvault", new ProviderParameters()
        .Set("vaultUrl", "https://myvault.vault.azure.net/")
        .Set("keyName", "db-key"))
    .WithBTree()
    .WithTransactions()
    .Build();

The same choices exist as WithStorage(IStorage), WithEncryption(ICryptoProvider), WithCache(IPageCache), WithJournal(ITransactionJournal) and WithStore(IKeyValueStore) when you would rather construct the object and pass it in. For the built-in providers, most of this can be said in a connection string instead; see connection strings.

What people actually replace

The last row is the one the LSM store demonstrates. It is a completely different data structure answering the same interface as the B+Tree store, in the same process, under the same SQL. Your own store would go in the same way.

Where the seams stop

Everything from the key-value store downward is selectable. The SQL layer above it is not: the parser, the planner, the executor, the catalogue and the transaction semantics are fixed, and there is no registry entry for them.

The engine's promise is that an application written for a full database server keeps working when the server is swapped out, and the SQL layer is what makes that promise. A pluggable dialect would be a pluggable promise.

Where to go next

• Storage engines, what the three stores do and when each is right
• Encryption, the algorithms, the key derivation, and ICryptoProvider
• Caching, implementing IPageCache
• Write-ahead logging, implementing ITransactionJournal
• Platform support, implementing IStorage