WitDatabase is built on a modular, provider-based architecture. Every major component — storage, encryption, caching, journaling, and indexing — is defined by an interface and can be replaced with a custom implementation.
1. Overview
Architecture Diagram
Pluggable Interfaces
| Interface | Purpose | Built-in Implementations |
|---|---|---|
IKeyValueStore |
Storage engine | StoreBTree, StoreLsm, StoreInMemory |
IStorage |
Page-level storage backend | StorageFile, StorageMemory, StorageIndexedDb |
ICryptoProvider |
Encryption algorithm | EncryptorProviderAesGcm, BouncyCastleCryptoProvider |
IPageCache |
Page caching strategy | PageCacheLru, PageCacheShardedClock |
ITransactionJournal |
Transaction durability | RollbackJournal, WalTransactionJournal |
ISecondaryIndexFactory |
Secondary index creation | SecondaryIndexFactoryKeyValueStore |
Provider Keys
Each provider has a unique key used for registration and lookup:
| Interface | Provider Keys |
|---|---|
IKeyValueStore |
btree, lsm, inmemory |
IStorage |
file, memory, encrypted, indexeddb |
ICryptoProvider |
aes-gcm, chacha20-poly1305 |
IPageCache |
clock, lru |
ITransactionJournal |
rollback, wal |
Why Extensibility?
Common use cases for custom providers:
| Scenario | Custom Provider |
|---|---|
| HSM integration | ICryptoProvider → Hardware Security Module |
| Cloud storage | IStorage → S3, Azure Blob, GCS |
| Enterprise key management | ICryptoProvider → Azure Key Vault, HashiCorp Vault |
| Custom caching | IPageCache → Redis-backed cache |
| Specialized indexes | ISecondaryIndexFactory → Full-text, spatial |
| Custom durability | ITransactionJournal → Network-replicated journal |
2. Provider System
IProvider Interface
All pluggable components implement IProvider:
public interface IProvider
{
/// <summary>
/// Gets the unique key identifying this provider type.
/// Examples: "aes-gcm", "chacha20-poly1305", "file", "memory", "btree"
/// </summary>
string ProviderKey { get; }
}ProviderRegistry
The global ProviderRegistry manages provider factories:
using OutWit.Database.Core.Providers;
// Register a provider factory
ProviderRegistry.Instance.Register<ICryptoProvider>("my-crypto",
parameters => new MyCryptoProvider(parameters.GetRequired<byte[]>("key")));
// Create a provider instance
var crypto = ProviderRegistry.Instance.Create<ICryptoProvider>("my-crypto",
new ProviderParameters().Set("key", myKey));
// Check if registered
bool exists = ProviderRegistry.Instance.IsRegistered<ICryptoProvider>("my-crypto");
// Get all registered keys
var keys = ProviderRegistry.Instance.GetRegisteredKeys<ICryptoProvider>();
// Returns: ["aes-gcm", "chacha20-poly1305", "my-crypto"]ProviderParameters
Parameters are passed to provider factories during creation:
// Create parameters
var parameters = new ProviderParameters()
.Set("key", encryptionKey)
.Set("iterations", 100000)
.Set("user", "tenant1");
// Get required parameter (throws if missing)
var key = parameters.GetRequired<byte[]>("key");
// Get optional parameter with default
var iterations = parameters.Get("iterations", 10000);
// Check if parameter exists
if (parameters.Has("user"))
{
var user = parameters.Get<string>("user");
}Auto-Registration with ModuleInitializer
Use [ModuleInitializer] for automatic registration when your assembly loads:
using System.Runtime.CompilerServices;
using OutWit.Database.Core.Interfaces;
using OutWit.Database.Core.Providers;
namespace MyCompany.WitDatabase.CustomProviders;
public static class MyProviderRegistration
{
private static bool _initialized;
[ModuleInitializer]
public static void Initialize()
{
if (_initialized) return;
// Register your custom providers
ProviderRegistry.Instance.Register<ICryptoProvider>("my-hsm",
p => new HsmCryptoProvider(p.GetRequired<string>("keyId")));
ProviderRegistry.Instance.Register<IStorage>("s3-storage",
p => new S3Storage(
p.GetRequired<string>("bucket"),
p.Get("region", "us-east-1")));
_initialized = true;
}
}When your assembly is loaded, providers are automatically registered.
Using Custom Providers
// Via builder with provider key
var db = new WitDatabaseBuilder()
.WithStorage("s3-storage", new ProviderParameters()
.Set("bucket", "my-database-bucket")
.Set("region", "eu-west-1"))
.WithEncryption("my-hsm", new ProviderParameters()
.Set("keyId", "production-key-001"))
.Build();
// Or pass instance directly
var db = new WitDatabaseBuilder()
.WithStorage(new S3Storage("my-bucket", "us-east-1"))
.WithEncryption(new HsmCryptoProvider("key-001"))
.Build();Provider Persistence
Some provider settings are stored in the database header:
| Setting | Persisted | Notes |
|---|---|---|
Store type (btree/lsm) |
✅ Yes | Auto-detected on reopen |
| Encryption enabled | ✅ Yes | Requires password on reopen |
| Encryption provider key | ✅ Yes | e.g., aes-gcm |
| Transactions enabled | ✅ Yes | Auto-detected |
| Page size | ✅ Yes | Cannot change after creation |
| Cache type/size | ❌ No | Can change on reopen |
| Journal mode | ❌ No | Can change on reopen |
// Inspect database without opening
var info = WitDatabase.GetDatabaseInfo("app.witdb");
Console.WriteLine($"Store: {info.StoreProvider}"); // "btree"
Console.WriteLine($"Encrypted: {info.RequiresEncryption}"); // true
Console.WriteLine($"Encryption: {info.EncryptionProvider}"); // "aes-gcm"3. Custom Crypto Provider
ICryptoProvider Interface
public interface ICryptoProvider : IProvider, IDisposable
{
/// <summary>
/// Encrypts plaintext using AEAD (Authenticated Encryption with Associated Data).
/// </summary>
/// <param name="nonce">Unique nonce (typically 12 bytes).</param>
/// <param name="plaintext">Data to encrypt.</param>
/// <param name="ciphertext">Output buffer for encrypted data.</param>
/// <param name="tag">Output buffer for authentication tag (typically 16 bytes).</param>
void Encrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> plaintext,
Span<byte> ciphertext, Span<byte> tag);
/// <summary>
/// Decrypts ciphertext and verifies authentication tag.
/// </summary>
/// <returns>True if authentication succeeded; false if tampered.</returns>
bool Decrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> ciphertext,
ReadOnlySpan<byte> tag, Span<byte> plaintext);
/// <summary>
/// Creates a clone of this provider (for concurrent operations).
/// </summary>
ICryptoProvider Clone();
/// <summary>
/// Size of nonce in bytes (typically 12).
/// </summary>
int NonceSize { get; }
/// <summary>
/// Size of authentication tag in bytes (typically 16).
/// </summary>
int TagSize { get; }
}Example: HSM Crypto Provider
using OutWit.Database.Core.Interfaces;
using System.Security.Cryptography;
namespace MyCompany.WitDatabase.Hsm;
/// <summary>
/// Crypto provider that delegates encryption to a Hardware Security Module.
/// </summary>
public sealed class HsmCryptoProvider : ICryptoProvider
{
private readonly IHsmClient _hsmClient;
private readonly string _keyId;
private bool _disposed;
public HsmCryptoProvider(string keyId, IHsmClient? hsmClient = null)
{
_keyId = keyId ?? throw new ArgumentNullException(nameof(keyId));
_hsmClient = hsmClient ?? HsmClientFactory.Create();
}
public string ProviderKey => "my-hsm";
public int NonceSize => 12;
public int TagSize => 16;
public void Encrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> plaintext,
Span<byte> ciphertext, Span<byte> tag)
{
ThrowIfDisposed();
// Call HSM for encryption
var result = _hsmClient.Encrypt(_keyId, nonce.ToArray(), plaintext.ToArray());
result.Ciphertext.AsSpan().CopyTo(ciphertext);
result.Tag.AsSpan().CopyTo(tag);
}
public bool Decrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> ciphertext,
ReadOnlySpan<byte> tag, Span<byte> plaintext)
{
ThrowIfDisposed();
try
{
var result = _hsmClient.Decrypt(_keyId, nonce.ToArray(),
ciphertext.ToArray(), tag.ToArray());
result.AsSpan().CopyTo(plaintext);
return true;
}
catch (CryptographicException)
{
return false; // Authentication failed
}
}
public ICryptoProvider Clone()
{
return new HsmCryptoProvider(_keyId, _hsmClient);
}
public void Dispose()
{
if (!_disposed)
{
// HSM client may be shared, don't dispose
_disposed = true;
}
}
private void ThrowIfDisposed()
{
ObjectDisposedException.ThrowIf(_disposed, this);
}
}Example: Azure Key Vault Provider
using Azure.Identity;
using Azure.Security.KeyVault.Keys;
using Azure.Security.KeyVault.Keys.Cryptography;
public sealed class AzureKeyVaultCryptoProvider : ICryptoProvider
{
private readonly CryptographyClient _cryptoClient;
private readonly string _keyId;
public AzureKeyVaultCryptoProvider(string vaultUrl, string keyName)
{
var keyClient = new KeyClient(new Uri(vaultUrl), new DefaultAzureCredential());
var key = keyClient.GetKey(keyName);
_keyId = key.Value.Id.ToString();
_cryptoClient = new CryptographyClient(key.Value.Id, new DefaultAzureCredential());
}
public string ProviderKey => "azure-keyvault";
public int NonceSize => 12;
public int TagSize => 16;
public void Encrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> plaintext,
Span<byte> ciphertext, Span<byte> tag)
{
// Azure Key Vault uses AES-GCM with nonce prepended
var input = new byte[nonce.Length + plaintext.Length];
nonce.CopyTo(input);
plaintext.CopyTo(input.AsSpan(nonce.Length));
var result = _cryptoClient.Encrypt(EncryptionAlgorithm.A256Gcm, input);
// Extract ciphertext and tag from result
result.Ciphertext.AsSpan(0, plaintext.Length).CopyTo(ciphertext);
result.Ciphertext.AsSpan(plaintext.Length, TagSize).CopyTo(tag);
}
public bool Decrypt(ReadOnlySpan<byte> nonce, ReadOnlySpan<byte> ciphertext,
ReadOnlySpan<byte> tag, Span<byte> plaintext)
{
try
{
var input = new byte[ciphertext.Length + tag.Length];
ciphertext.CopyTo(input);
tag.CopyTo(input.AsSpan(ciphertext.Length));
var result = _cryptoClient.Decrypt(EncryptionAlgorithm.A256Gcm, input);
result.Plaintext.AsSpan().CopyTo(plaintext);
return true;
}
catch
{
return false;
}
}
public ICryptoProvider Clone() => new AzureKeyVaultCryptoProvider(_keyId);
public void Dispose() { }
}Registration and Usage
// Register the provider
ProviderRegistry.Instance.Register<ICryptoProvider>("azure-keyvault", p =>
new AzureKeyVaultCryptoProvider(
p.GetRequired<string>("vaultUrl"),
p.GetRequired<string>("keyName")));
// Use in database
var db = new WitDatabaseBuilder()
.WithFilePath("secure.witdb")
.WithBTree()
.WithEncryption("azure-keyvault", new ProviderParameters()
.Set("vaultUrl", "https://my-vault.vault.azure.net/")
.Set("keyName", "database-key"))
.Build();4. Custom Storage Provider
IStorage Interface
public interface IStorage : IProvider, IDisposable
{
/// <summary>
/// Reads a page from storage.
/// </summary>
void ReadPage(long pageNumber, Span<byte> buffer);
/// <summary>
/// Writes a page to storage.
/// </summary>
void WritePage(long pageNumber, ReadOnlySpan<byte> buffer);
/// <summary>
/// Reads a page asynchronously.
/// </summary>
ValueTask ReadPageAsync(long pageNumber, Memory<byte> buffer,
CancellationToken cancellationToken = default);
/// <summary>
/// Writes a page asynchronously.
/// </summary>
ValueTask WritePageAsync(long pageNumber, ReadOnlyMemory<byte> buffer,
CancellationToken cancellationToken = default);
/// <summary>
/// Ensures all writes are persisted to storage.
/// </summary>
void Flush();
/// <summary>
/// Flushes asynchronously.
/// </summary>
ValueTask FlushAsync(CancellationToken cancellationToken = default);
/// <summary>
/// Gets the page size in bytes.
/// </summary>
int PageSize { get; }
/// <summary>
/// Gets the current number of pages.
/// </summary>
long PageCount { get; }
}Example: S3 Storage Provider
using Amazon.S3;
using Amazon.S3.Model;
using OutWit.Database.Core.Interfaces;
namespace MyCompany.WitDatabase.CloudStorage;
/// <summary>
/// Storage provider backed by Amazon S3.
/// Stores each page as a separate S3 object.
/// </summary>
public sealed class S3Storage : IStorage
{
private readonly IAmazonS3 _s3Client;
private readonly string _bucket;
private readonly string _prefix;
private readonly int _pageSize;
private long _pageCount;
private bool _disposed;
public S3Storage(string bucket, string region, string prefix = "witdb/",
int pageSize = 4096)
{
_bucket = bucket;
_prefix = prefix;
_pageSize = pageSize;
_s3Client = new AmazonS3Client(Amazon.RegionEndpoint.GetBySystemName(region));
// Load page count from metadata
_pageCount = LoadPageCount();
}
public string ProviderKey => "s3-storage";
public int PageSize => _pageSize;
public long PageCount => _pageCount;
public void ReadPage(long pageNumber, Span<byte> buffer)
{
ReadPageAsync(pageNumber, buffer.ToArray(), default).AsTask().Wait();
}
public async ValueTask ReadPageAsync(long pageNumber, Memory<byte> buffer,
CancellationToken cancellationToken = default)
{
ThrowIfDisposed();
var key = $"{_prefix}page_{pageNumber:D10}";
try
{
var response = await _s3Client.GetObjectAsync(new GetObjectRequest
{
BucketName = _bucket,
Key = key
}, cancellationToken);
using var stream = response.ResponseStream;
await stream.ReadExactlyAsync(buffer, cancellationToken);
}
catch (AmazonS3Exception ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
{
// Page doesn't exist, return zeros
buffer.Span.Clear();
}
}
public void WritePage(long pageNumber, ReadOnlySpan<byte> buffer)
{
WritePageAsync(pageNumber, buffer.ToArray(), default).AsTask().Wait();
}
public async ValueTask WritePageAsync(long pageNumber, ReadOnlyMemory<byte> buffer,
CancellationToken cancellationToken = default)
{
ThrowIfDisposed();
var key = $"{_prefix}page_{pageNumber:D10}";
using var stream = new MemoryStream(buffer.ToArray());
await _s3Client.PutObjectAsync(new PutObjectRequest
{
BucketName = _bucket,
Key = key,
InputStream = stream
}, cancellationToken);
if (pageNumber >= _pageCount)
{
_pageCount = pageNumber + 1;
await SavePageCount(cancellationToken);
}
}
public void Flush()
{
// S3 writes are immediately durable
}
public ValueTask FlushAsync(CancellationToken cancellationToken = default)
{
return ValueTask.CompletedTask;
}
private long LoadPageCount()
{
try
{
var response = _s3Client.GetObjectAsync(new GetObjectRequest
{
BucketName = _bucket,
Key = $"{_prefix}metadata"
}).Result;
using var reader = new StreamReader(response.ResponseStream);
return long.Parse(reader.ReadToEnd());
}
catch
{
return 0;
}
}
private async Task SavePageCount(CancellationToken cancellationToken)
{
using var stream = new MemoryStream();
using var writer = new StreamWriter(stream);
await writer.WriteAsync(_pageCount.ToString());
await writer.FlushAsync(cancellationToken);
stream.Position = 0;
await _s3Client.PutObjectAsync(new PutObjectRequest
{
BucketName = _bucket,
Key = $"{_prefix}metadata",
InputStream = stream
}, cancellationToken);
}
public void Dispose()
{
if (!_disposed)
{
_s3Client.Dispose();
_disposed = true;
}
}
private void ThrowIfDisposed()
{
ObjectDisposedException.ThrowIf(_disposed, this);
}
}Registration and Usage
// Register the provider
ProviderRegistry.Instance.Register<IStorage>("s3-storage", p =>
new S3Storage(
p.GetRequired<string>("bucket"),
p.Get("region", "us-east-1"),
p.Get("prefix", "witdb/"),
p.Get("pageSize", 4096)));
// Use in database
var db = new WitDatabaseBuilder()
.WithStorage("s3-storage", new ProviderParameters()
.Set("bucket", "my-database-bucket")
.Set("region", "eu-west-1")
.Set("prefix", "production/db/"))
.WithBTree()
.WithEncryption("password")
.Build();Performance Considerations
Cloud storage has different characteristics than local files:
| Aspect | Local File | Cloud Storage (S3) |
|---|---|---|
| Latency | ~0.1 ms | ~50-100 ms |
| Throughput | Limited by SSD | Limited by network |
| Cost | Fixed | Per-operation |
| Durability | Single disk | Multi-AZ replication |
Recommendations for cloud storage:
- Use large page sizes (16-64 KB) to reduce operation count
- Implement local caching layer
- Use eventually-consistent reads where possible
- Consider batch operations for bulk writes
5. Custom Cache Provider
IPageCache Interface
public interface IPageCache : IProvider, IDisposable
{
/// <summary>
/// Gets a page from cache, loading from storage if necessary.
/// </summary>
CachedPage GetPage(long pageNumber);
/// <summary>
/// Creates a new page in cache (for newly allocated pages).
/// </summary>
CachedPage CreatePage(long pageNumber);
/// <summary>
/// Marks a page as dirty (modified).
/// </summary>
void MarkDirty(long pageNumber);
/// <summary>
/// Releases a reference to a page.
/// </summary>
void ReleasePage(long pageNumber);
/// <summary>
/// Evicts a specific page from cache.
/// </summary>
void Evict(long pageNumber);
/// <summary>
/// Flushes all dirty pages to storage.
/// </summary>
void FlushAll();
/// <summary>
/// Clears all pages from cache.
/// </summary>
void Clear();
/// <summary>
/// Gets the number of cached pages.
/// </summary>
int Count { get; }
/// <summary>
/// Gets the number of dirty pages.
/// </summary>
int DirtyCount { get; }
}Example: Redis-Backed Cache
using StackExchange.Redis;
using OutWit.Database.Core.Cache;
using OutWit.Database.Core.Interfaces;
namespace MyCompany.WitDatabase.DistributedCache;
/// <summary>
/// Page cache backed by Redis for distributed scenarios.
/// </summary>
public sealed class RedisCacheProvider : IPageCache
{
private readonly IStorage _storage;
private readonly IDatabase _redis;
private readonly string _keyPrefix;
private readonly int _pageSize;
private readonly TimeSpan _expiry;
public RedisCacheProvider(IStorage storage, string redisConnection,
string keyPrefix = "witdb:", TimeSpan? expiry = null)
{
_storage = storage;
_keyPrefix = keyPrefix;
_pageSize = storage.PageSize;
_expiry = expiry ?? TimeSpan.FromMinutes(30);
var connection = ConnectionMultiplexer.Connect(redisConnection);
_redis = connection.GetDatabase();
}
public string ProviderKey => "redis-cache";
public CachedPage GetPage(long pageNumber)
{
var key = $"{_keyPrefix}{pageNumber}";
var cached = _redis.StringGet(key);
if (cached.HasValue)
{
var page = new CachedPage(pageNumber, _pageSize);
((byte[])cached!).AsSpan().CopyTo(page.Data);
return page;
}
// Load from storage
var newPage = new CachedPage(pageNumber, _pageSize);
_storage.ReadPage(pageNumber, newPage.Data);
// Cache in Redis
_redis.StringSet(key, newPage.Data.ToArray(), _expiry);
return newPage;
}
public CachedPage CreatePage(long pageNumber)
{
var page = new CachedPage(pageNumber, _pageSize);
page.Data.Clear();
page.MarkDirty();
return page;
}
public void MarkDirty(long pageNumber)
{
// Track dirty pages locally or in Redis set
_redis.SetAdd($"{_keyPrefix}dirty", pageNumber);
}
public void ReleasePage(long pageNumber)
{
// No-op for Redis cache
}
public void Evict(long pageNumber)
{
_redis.KeyDelete($"{_keyPrefix}{pageNumber}");
}
public void FlushAll()
{
var dirtyPages = _redis.SetMembers($"{_keyPrefix}dirty");
foreach (var pageNum in dirtyPages)
{
var pageNumber = (long)pageNum;
var key = $"{_keyPrefix}{pageNumber}";
var data = _redis.StringGet(key);
if (data.HasValue)
{
_storage.WritePage(pageNumber, (byte[])data!);
}
}
_redis.KeyDelete($"{_keyPrefix}dirty");
_storage.Flush();
}
public void Clear()
{
// Clear all keys with prefix
// (In production, use SCAN instead of KEYS)
}
public int Count => 0; // Would need to track
public int DirtyCount => (int)_redis.SetLength($"{_keyPrefix}dirty");
public void Dispose() { }
}6. Custom Transaction Journal
ITransactionJournal Interface
public interface ITransactionJournal : IProvider, IDisposable
{
/// <summary>
/// Logs the beginning of a transaction.
/// </summary>
void BeginTransaction(long transactionId);
/// <summary>
/// Logs a Put operation within a transaction.
/// </summary>
void LogPut(long transactionId, ReadOnlySpan<byte> key,
ReadOnlySpan<byte> value, ReadOnlySpan<byte> oldValue);
/// <summary>
/// Logs a Delete operation within a transaction.
/// </summary>
void LogDelete(long transactionId, ReadOnlySpan<byte> key,
ReadOnlySpan<byte> oldValue);
/// <summary>
/// Logs the commit of a transaction.
/// </summary>
void CommitTransaction(long transactionId);
/// <summary>
/// Logs the rollback of a transaction.
/// </summary>
void RollbackTransaction(long transactionId);
/// <summary>
/// Ensures all pending writes are flushed to disk.
/// </summary>
void Sync();
/// <summary>
/// Recovers transactions after a crash.
/// </summary>
/// <returns>Number of operations recovered.</returns>
int Recover(IKeyValueStore store);
/// <summary>
/// Creates a checkpoint, truncating the journal.
/// </summary>
void Checkpoint();
}Example: Network-Replicated Journal
namespace MyCompany.WitDatabase.ReplicatedJournal;
/// <summary>
/// Transaction journal that replicates to remote servers for high availability.
/// </summary>
public sealed class ReplicatedJournal : ITransactionJournal
{
private readonly ITransactionJournal _localJournal;
private readonly IReplicationClient[] _replicas;
private readonly int _requiredAcks;
public ReplicatedJournal(string localPath, string[] replicaUrls, int requiredAcks = 1)
{
_localJournal = new WalTransactionJournal(localPath);
_replicas = replicaUrls.Select(url => new ReplicationClient(url)).ToArray();
_requiredAcks = requiredAcks;
}
public string ProviderKey => "replicated-wal";
public void BeginTransaction(long transactionId)
{
_localJournal.BeginTransaction(transactionId);
ReplicateAsync("BEGIN", transactionId).Wait();
}
public void LogPut(long transactionId, ReadOnlySpan<byte> key,
ReadOnlySpan<byte> value, ReadOnlySpan<byte> oldValue)
{
_localJournal.LogPut(transactionId, key, value, oldValue);
// Replicate asynchronously for performance
_ = ReplicateAsync("PUT", transactionId, key.ToArray(), value.ToArray());
}
public void LogDelete(long transactionId, ReadOnlySpan<byte> key,
ReadOnlySpan<byte> oldValue)
{
_localJournal.LogDelete(transactionId, key, oldValue);
_ = ReplicateAsync("DELETE", transactionId, key.ToArray());
}
public void CommitTransaction(long transactionId)
{
_localJournal.CommitTransaction(transactionId);
// Wait for required acknowledgments before returning
var acks = ReplicateAsync("COMMIT", transactionId).Result;
if (acks < _requiredAcks)
{
throw new InvalidOperationException(
$"Failed to replicate commit: got {acks} acks, required {_requiredAcks}");
}
}
public void RollbackTransaction(long transactionId)
{
_localJournal.RollbackTransaction(transactionId);
_ = ReplicateAsync("ROLLBACK", transactionId);
}
public void Sync() => _localJournal.Sync();
public int Recover(IKeyValueStore store) => _localJournal.Recover(store);
public void Checkpoint() => _localJournal.Checkpoint();
private async Task<int> ReplicateAsync(string operation, long transactionId,
byte[]? key = null, byte[]? value = null)
{
var tasks = _replicas.Select(r => r.ReplicateAsync(operation, transactionId, key, value));
var results = await Task.WhenAll(tasks);
return results.Count(r => r);
}
public void Dispose()
{
_localJournal.Dispose();
foreach (var replica in _replicas)
{
replica.Dispose();
}
}
}7. Quick Reference
Interface Summary
| Interface | Methods | Built-in Providers |
|---|---|---|
IProvider |
ProviderKey |
Base for all providers |
ICryptoProvider |
Encrypt, Decrypt, Clone, NonceSize, TagSize |
aes-gcm, chacha20-poly1305 |
IStorage |
ReadPage, WritePage, Flush, PageSize, PageCount |
file, memory, indexeddb |
IPageCache |
GetPage, CreatePage, MarkDirty, FlushAll, Count |
clock, lru |
ITransactionJournal |
BeginTransaction, LogPut, CommitTransaction, Recover |
rollback, wal |
IKeyValueStore |
Get, Put, Delete, Scan, BeginTransaction |
btree, lsm, inmemory |
ISecondaryIndexFactory |
CreateIndex |
Built-in B+Tree |
ProviderRegistry API
// Registration
ProviderRegistry.Instance.Register<T>(key, factory);
ProviderRegistry.Instance.RegisterOrReplace<T>(key, factory);
// Creation
T provider = ProviderRegistry.Instance.Create<T>(key, parameters);
// Query
bool exists = ProviderRegistry.Instance.IsRegistered<T>(key);
var keys = ProviderRegistry.Instance.GetRegisteredKeys<T>();ProviderParameters API
var p = new ProviderParameters()
.Set("key", value)
.Set<int>("count", 100);
var value = p.Get<string>("key");
var count = p.Get("count", defaultValue: 50);
var required = p.GetRequired<byte[]>("key");
bool has = p.Has("key");Custom Provider Checklist
When implementing a custom provider:
ICryptoProvider:
- Implement AEAD encryption (Encrypt + authentication tag)
- Return
falsefromDecrypton authentication failure (don't throw) - Make
Clone()return independent instance - Use standard nonce (12 bytes) and tag (16 bytes) sizes
- Clear sensitive data in
Dispose()
IStorage:
- Handle page reads for non-existent pages (return zeros)
- Implement both sync and async methods
- Ensure
Flush()persists all writes - Track
PageCountaccurately
IPageCache:
- Load pages from storage on cache miss
- Track dirty pages for flushing
- Handle eviction when cache is full
- Implement reference counting if needed
ITransactionJournal:
- Persist commit marker durably before returning from
CommitTransaction - Implement
Recover()to replay/undo after crash - Support
Checkpoint()to truncate journal
Builder Methods for Custom Providers
var db = new WitDatabaseBuilder()
// Storage
.WithStorage(IStorage instance)
.WithStorage(string providerKey, ProviderParameters parameters)
// Encryption
.WithEncryption(ICryptoProvider instance)
.WithEncryption(string providerKey, ProviderParameters parameters)
// Cache
.WithCache(IPageCache instance)
.WithCache(string providerKey, ProviderParameters parameters)
// Journal
.WithJournal(ITransactionJournal instance)
.WithJournal(string providerKey, ProviderParameters parameters)
// Store
.WithStore(IKeyValueStore instance)
.Build();Example: Full Custom Configuration
// Register all custom providers
public static class CustomProviders
{
[ModuleInitializer]
public static void Register()
{
ProviderRegistry.Instance.Register<ICryptoProvider>("azure-keyvault",
p => new AzureKeyVaultCryptoProvider(
p.GetRequired<string>("vaultUrl"),
p.GetRequired<string>("keyName")));
ProviderRegistry.Instance.Register<IStorage>("s3",
p => new S3Storage(
p.GetRequired<string>("bucket"),
p.Get("region", "us-east-1")));
ProviderRegistry.Instance.Register<IPageCache>("redis",
p => new RedisCacheProvider(
p.GetRequired<IStorage>("storage"),
p.GetRequired<string>("connection")));
}
}
// Use custom providers
var db = new WitDatabaseBuilder()
.WithStorage("s3", new ProviderParameters()
.Set("bucket", "prod-database")
.Set("region", "eu-west-1"))
.WithEncryption("azure-keyvault", new ProviderParameters()
.Set("vaultUrl", "https://myvault.vault.azure.net/")
.Set("keyName", "db-key"))
.WithBTree()
.WithTransactions()
.Build();