Home - trueai-org/ultra-kv GitHub Wiki

UltraKV DataBase Comprehensive API Documentation

Language: 中文 | English

Table of Contents

  1. Project Overview
  2. Quick Start
  3. Core Components
  4. Configuration Details
  5. API Reference
  6. Advanced Features
  7. Performance Optimization
  8. Best Practices
  9. Code Examples

Project Overview

UltraKV is a high-performance C# key-value database engine featuring an in-memory index with disk storage architecture. It supports generic key-value pair storage, data compression, encryption, batch operations, and more.

Core Features

  • High Performance: In-memory indexing + disk storage for fast read/write operations
  • Type Safety: Generic design supporting any type of key-value pairs
  • Data Security: AES256-GCM encryption and multiple compression algorithms
  • Batch Operations: Efficient batch insert and delete operations
  • Auto Maintenance: Automatic compaction and index rebuilding
  • Multi-Engine Management: Support for managing multiple database instances

Quick Start

1. Basic Usage

// Create database engine
using var engine = new UltraKVEngine<string, string>("./test.ultrakv");

// Insert data
engine.Put("user:1001", "John Doe");
engine.Put("user:1002", "Jane Smith");

// Read data
var user = engine.Get("user:1001");
Console.WriteLine($"User: {user}"); // Output: User: John Doe

// Check if key exists
bool exists = engine.ContainsKey("user:1001");

// Delete data
engine.Delete("user:1002");

// Persist to disk
engine.Flush();

2. Multi-Engine Management

// Create engine manager
using var manager = new UltraKVManager<string, object>("./databases");

// Get engines for different purposes
var userEngine = manager.GetEngine("users");
var sessionEngine = manager.GetEngine("sessions");
var cacheEngine = manager.GetEngine("cache");

// Use different engines
userEngine.Put("user:1001", new { Name = "John", Age = 30 });
sessionEngine.Put("session:abc123", new { UserId = 1001, LoginTime = DateTime.Now });
cacheEngine.Put("cache:data1", "cached_value");

// Batch flush all engines
manager.FlushAll();

Core Components

1. UltraKVEngine<TKey, TValue>

The main database engine class responsible for key-value storage, retrieval, and management.

public class UltraKVEngine<TKey, TValue> : IDisposable where TKey : notnull

2. UltraKVManager<TKey, TValue>

Multi-engine manager for managing multiple database instances.

public class UltraKVManager<TKey, TValue> : IDisposable where TKey : notnull

3. UltraKVConfig

Database configuration class controlling various behaviors and performance parameters.

public class UltraKVConfig

Configuration Details

UltraKVConfig Options

Basic Configuration

Parameter Type Default Description
MemoryModeEnabled bool false Enable memory mode
ValueCacheEnabled bool false Enable value caching
ValueCacheSeconds int 3600 Value cache duration (seconds)
UpdateValidationEnabled bool false Enable update validation
MaxKeyLength int 4096 Maximum key length (bytes)

Storage Configuration

Parameter Type Default Description
CompressionType CompressionType None Compression algorithm type
EncryptionType EncryptionType None Encryption algorithm type
HashType HashType XXH3 Hash algorithm type
EncryptionKey string? null Encryption key

Performance Configuration

Parameter Type Default Description
FileStreamBufferSizeKB int 64 File stream buffer size (KB)
WriteBufferEnabled bool true Enable write buffering
WriteBufferSizeKB int 1024 Write buffer size (KB)
WriteBufferTimeThresholdMs int 5000 Write buffer time threshold (ms)

Maintenance Configuration

Parameter Type Default Description
AutoCompactEnabled bool false Enable auto compaction
AutoCompactThreshold byte 50 Auto compaction threshold (%)
FlushInterval ushort 5 Flush interval (seconds)
IndexRebuildThreshold byte 20 Index rebuild threshold (%)
FileUpdateMode FileUpdateMode Append File update mode

Predefined Configurations

// Default configuration
var defaultConfig = UltraKVConfig.Default;

// Minimal configuration (for resource-constrained environments)
var minimalConfig = UltraKVConfig.Minimal;

// Secure configuration (with encryption and compression)
var secureConfig = UltraKVConfig.Secure("MySecureKey32BytesLong!@#$%^&*()");

// Debug configuration (enables all validations)
var debugConfig = UltraKVConfig.Debug;

// SSD optimized configuration
var ssdConfig = UltraKVConfig.SSD;

// HDD optimized configuration
var hddConfig = UltraKVConfig.HDD;

// Strict mode configuration
var strictConfig = UltraKVConfig.Strict;

Advanced Configuration Example

var config = new UltraKVConfig
{
    // Performance settings
    FileStreamBufferSizeKB = 1024,     // 1MB buffer
    WriteBufferSizeKB = 2048,          // 2MB write buffer
    FlushInterval = 10,                // Auto-flush every 10 seconds
    
    // Compression and encryption
    CompressionType = CompressionType.LZ4,
    EncryptionType = EncryptionType.AES256GCM,
    EncryptionKey = "MySecureKey32BytesLong!@#$%^&*()",
    
    // Maintenance settings
    AutoCompactEnabled = true,         // Enable auto compaction
    AutoCompactThreshold = 30,         // 30% fragmentation triggers compaction
    IndexRebuildThreshold = 25,        // 25% deletions trigger index rebuild
    
    // File update mode
    FileUpdateMode = FileUpdateMode.Append  // Append mode (high performance)
};

var engine = new UltraKVEngine<string, object>("./database.ultrakv", config);

API Reference

UltraKVEngine<TKey, TValue> Methods

Basic Operations

Put(TKey key, TValue value)

Insert or update a key-value pair.

public void Put(TKey key, TValue value)

Parameters:

  • key: Key, cannot be null
  • value: Value

Example:

engine.Put("user:1001", "John Doe");
engine.Put("config:timeout", 30);
Set(TKey key, TValue value)

Insert or update a key-value pair (alias for Put).

public void Set(TKey key, TValue value)
Get(TKey key)

Retrieve value by key.

public TValue? Get(TKey key)

Parameters:

  • key: Key to search for

Returns:

  • Returns the corresponding value if found, otherwise returns default(TValue)

Example:

var user = engine.Get("user:1001");
if (user != null)
{
    Console.WriteLine($"Found user: {user}");
}
ContainsKey(TKey key)

Check if key exists.

public bool ContainsKey(TKey key)

Parameters:

  • key: Key to check

Returns:

  • Returns true if exists, false otherwise

Example:

if (engine.ContainsKey("user:1001"))
{
    Console.WriteLine("User exists");
}
Remove(TKey key)

Delete record with specified key.

public bool Remove(TKey key)

Parameters:

  • key: Key to delete

Returns:

  • Returns true if successfully deleted, false if key doesn't exist
Delete(TKey key)

Delete record with specified key (alias for Remove).

public bool Delete(TKey key)

Batch Operations

SetBatch(Dictionary<TKey, TValue> items, bool skipDuplicates = true)

Batch insert key-value pairs.

public int SetBatch(Dictionary<TKey, TValue> items, bool skipDuplicates = true)

Parameters:

  • items: Dictionary of key-value pairs to insert
  • skipDuplicates: Whether to skip duplicate keys (default true)

Returns:

  • Number of successfully inserted records

Example:

var batch = new Dictionary<string, string>
{
    ["user:1001"] = "John Doe",
    ["user:1002"] = "Jane Smith",
    ["user:1003"] = "Bob Wilson"
};

int inserted = engine.SetBatch(batch);
Console.WriteLine($"Inserted {inserted} records");
SetBatch(KeyValuePair<TKey, TValue>[] items, bool skipDuplicates = true)

Batch insert key-value pair array.

public int SetBatch(KeyValuePair<TKey, TValue>[] items, bool skipDuplicates = true)
SetBatch(IEnumerable<KeyValuePair<TKey, TValue>> items, bool skipDuplicates = true)

Batch insert key-value pair collection.

public int SetBatch(IEnumerable<KeyValuePair<TKey, TValue>> items, bool skipDuplicates = true)
DeleteBatch(List batchKeys)

Batch delete.

public int DeleteBatch(List<TKey> batchKeys)

Parameters:

  • batchKeys: List of keys to delete

Returns:

  • Number of successfully deleted records

Example:

var keysToDelete = new List<string> { "user:1001", "user:1002" };
int deleted = engine.DeleteBatch(keysToDelete);
Console.WriteLine($"Deleted {deleted} records");

Maintenance Operations

Flush()

Force flush data to disk.

public void Flush()

Example:

// Force flush after inserting data
engine.Put("important:data", "critical_value");
engine.Flush(); // Ensure data is immediately written to disk
Clear()

Clear all data.

public void Clear()

Example:

engine.Clear(); // Delete all data
Compact(bool allCompact = false)

Execute compaction operation to eliminate file fragmentation.

public void Compact(bool allCompact = false)

Parameters:

  • allCompact: Whether to perform full compaction (default false, preserves index expansion space)

Example:

// Standard compaction (preserves index space)
engine.Compact();

// Full compaction (minimizes file size)
engine.Compact(true);
ShouldCompact()

Check if compaction operation is needed.

public bool ShouldCompact()

Returns:

  • Returns true if compaction is needed

Query Operations

Keys

Get collection of all keys.

public IEnumerable<TKey> Keys { get; }

Example:

foreach (var key in engine.Keys)
{
    Console.WriteLine($"Key: {key}");
}
Values

Get collection of all values.

public IEnumerable<TValue> Values { get; }
GetAllKeys()

Get list of all keys.

public List<TKey> GetAllKeys()

Example:

var allKeys = engine.GetAllKeys();
Console.WriteLine($"Total keys: {allKeys.Count}");
Count

Get total record count.

public int Count { get; }
this[TKey key]

Indexer, supports direct value access through key.

public TValue? this[TKey key] { get; set; }

Example:

// Set value
engine["user:1001"] = "John Doe";

// Get value
var user = engine["user:1001"];

Statistics

GetStats()

Get database statistics.

public string GetStats()

Returns:

  • String containing database header information and file size

Example:

var stats = engine.GetStats();
Console.WriteLine($"Database stats: {stats}");

UltraKVManager<TKey, TValue> Methods

GetEngine(string name, UltraKVConfig? config = null)

Get or create database engine with specified name.

public UltraKVEngine<TKey, TValue> GetEngine(string name, UltraKVConfig? config = null)

Parameters:

  • name: Engine name
  • config: Configuration object (optional)

Returns:

  • Database engine instance

Example:

var manager = new UltraKVManager<string, object>("./databases");

// Use default configuration
var userEngine = manager.GetEngine("users");

// Use custom configuration
var cacheEngine = manager.GetEngine("cache", new UltraKVConfig 
{ 
    MemoryModeEnabled = true,
    FlushInterval = 30 
});

FlushAll()

Flush all engines' data to disk.

public void FlushAll()

CloseEngine(string name)

Close engine with specified name.

public void CloseEngine(string name)

GetEngineNames()

Get all engine names.

public IEnumerable<string> GetEngineNames()

Advanced Features

1. Memory Mode

Memory mode loads all data into memory, providing extremely high read/write performance.

var config = new UltraKVConfig
{
    MemoryModeEnabled = true  // Enable memory mode
};

var engine = new UltraKVEngine<string, string>("./memory.ultrakv", config);

// All operations are performed in memory
engine.Put("key1", "value1");
var value = engine.Get("key1"); // Read directly from memory

// Manual flush to disk
engine.Flush();

2. Value Caching

Enabling value caching improves read performance for hot data.

var config = new UltraKVConfig
{
    ValueCacheEnabled = true,    // Enable value caching
    ValueCacheSeconds = 1800     // Cache for 30 minutes
};

var engine = new UltraKVEngine<string, string>("./cached.ultrakv", config);

// First read loads from disk
var value1 = engine.Get("hot_key");

// Subsequent reads from cache (faster)
var value2 = engine.Get("hot_key");

3. Data Encryption

Supports AES256-GCM encryption to protect sensitive data.

var config = new UltraKVConfig
{
    EncryptionType = EncryptionType.AES256GCM,
    EncryptionKey = "MySecureKey32BytesLong!@#$%^&*()"
};

var engine = new UltraKVEngine<string, string>("./encrypted.ultrakv", config);

// Data is automatically encrypted when stored
engine.Put("sensitive:data", "secret_information");

4. Data Compression

Supports multiple compression algorithms to reduce storage space.

var config = new UltraKVConfig
{
    CompressionType = CompressionType.LZ4  // Or Gzip, Brotli
};

var engine = new UltraKVEngine<string, string>("./compressed.ultrakv", config);

// Data is automatically compressed when stored
engine.Put("large:data", largeStringValue);

5. Auto Maintenance

Configure automatic compaction and index rebuilding to maintain database health.

var config = new UltraKVConfig
{
    AutoCompactEnabled = true,        // Enable auto compaction
    AutoCompactThreshold = 30,        // 30% fragmentation triggers compaction
    IndexRebuildThreshold = 25,       // 25% deletions trigger index rebuild
    FlushInterval = 5                 // Auto-flush every 5 seconds
};

var engine = new UltraKVEngine<string, string>("./auto.ultrakv", config);

// Database maintains itself automatically, no manual intervention needed

Performance Optimization

1. Batch Operations

Using batch operations can significantly improve performance.

// Avoid: Individual inserts
for (int i = 0; i < 10000; i++)
{
    engine.Put($"key_{i}", $"value_{i}");
}

// Recommended: Batch insert
var batch = new Dictionary<string, string>();
for (int i = 0; i < 10000; i++)
{
    batch[$"key_{i}"] = $"value_{i}";
}
int inserted = engine.SetBatch(batch);

2. Configuration Optimization

Optimize configuration for different storage devices and usage scenarios.

// SSD configuration
var ssdConfig = new UltraKVConfig
{
    FileStreamBufferSizeKB = 1024,    // Large buffer
    WriteBufferSizeKB = 2048,         // Large write buffer
    FlushInterval = 1,                // Frequent flushing
    AutoCompactEnabled = true,
    AutoCompactThreshold = 20
};

// HDD configuration
var hddConfig = new UltraKVConfig
{
    FileStreamBufferSizeKB = 256,     // Small buffer
    WriteBufferSizeKB = 512,          // Small write buffer
    FlushInterval = 10,               // Less frequent flushing
    AutoCompactEnabled = true,
    AutoCompactThreshold = 50
};

3. Memory Usage Optimization

// For frequently accessed small datasets, use memory mode
var memoryConfig = new UltraKVConfig
{
    MemoryModeEnabled = true
};

// For large datasets, use value caching
var cacheConfig = new UltraKVConfig
{
    ValueCacheEnabled = true,
    ValueCacheSeconds = 3600  // 1 hour cache
};

Best Practices

1. Resource Management

// Use using statement to ensure proper resource disposal
using var manager = new UltraKVManager<string, object>("./databases");
using var engine = new UltraKVEngine<string, string>("./test.ultrakv");

// Or manual disposal
try
{
    var engine = new UltraKVEngine<string, string>("./test.ultrakv");
    // Use engine...
}
finally
{
    engine?.Dispose();
}

2. Error Handling

try
{
    var engine = new UltraKVEngine<string, string>("./test.ultrakv", config);
    engine.Put("test:key", "test_value");
    engine.Flush();
}
catch (InvalidOperationException ex)
{
    Console.WriteLine($"Configuration error: {ex.Message}");
}
catch (InvalidDataException ex)
{
    Console.WriteLine($"Data format error: {ex.Message}");
}
catch (Exception ex)
{
    Console.WriteLine($"Unknown error: {ex.Message}");
}

3. Performance Monitoring

var stopwatch = Stopwatch.StartNew();

// Execute operations
var batch = GenerateTestData(10000);
int inserted = engine.SetBatch(batch);

stopwatch.Stop();

Console.WriteLine($"Inserted {inserted} records in: {stopwatch.ElapsedMilliseconds} ms");
Console.WriteLine($"Average speed: {inserted / Math.Max(stopwatch.Elapsed.TotalSeconds, 1):F0} records/sec");

// Check database status
var stats = engine.GetStats();
Console.WriteLine($"Database status: {stats}");

4. Data Backup

// Regularly flush data
engine.Flush();

// Check if compaction is needed
if (engine.ShouldCompact())
{
    Console.WriteLine("Performing database compaction...");
    engine.Compact();
}

// Get statistics for monitoring
var stats = engine.GetStats();
Console.WriteLine($"Database statistics: {stats}");

Code Examples

1. User Management System

public class UserManager
{
    private readonly UltraKVEngine<string, User> _userEngine;
    private readonly UltraKVEngine<string, UserSession> _sessionEngine;

    public UserManager(string basePath)
    {
        var userConfig = new UltraKVConfig
        {
            CompressionType = CompressionType.LZ4,
            EncryptionType = EncryptionType.AES256GCM,
            EncryptionKey = "UserDataKey32BytesLong!@#$%^&*()",
            AutoCompactEnabled = true,
            AutoCompactThreshold = 30
        };

        var sessionConfig = new UltraKVConfig
        {
            MemoryModeEnabled = true,  // Session data uses memory mode
            FlushInterval = 30         // Flush every 30 seconds
        };

        _userEngine = new UltraKVEngine<string, User>(
            Path.Combine(basePath, "users.ultrakv"), userConfig);
        _sessionEngine = new UltraKVEngine<string, UserSession>(
            Path.Combine(basePath, "sessions.ultrakv"), sessionConfig);
    }

    public void CreateUser(string userId, User user)
    {
        _userEngine.Put($"user:{userId}", user);
        _userEngine.Flush(); // Immediately persist user data
    }

    public User? GetUser(string userId)
    {
        return _userEngine.Get($"user:{userId}");
    }

    public void CreateSession(string sessionId, UserSession session)
    {
        _sessionEngine.Put($"session:{sessionId}", session);
        // Session data in memory, periodically flushed
    }

    public UserSession? GetSession(string sessionId)
    {
        return _sessionEngine.Get($"session:{sessionId}");
    }

    public void DeleteSession(string sessionId)
    {
        _sessionEngine.Delete($"session:{sessionId}");
    }

    public void Dispose()
    {
        _userEngine?.Dispose();
        _sessionEngine?.Dispose();
    }
}

public class User
{
    public string Name { get; set; } = string.Empty;
    public string Email { get; set; } = string.Empty;
    public DateTime CreatedAt { get; set; }
}

public class UserSession
{
    public string UserId { get; set; } = string.Empty;
    public DateTime LoginTime { get; set; }
    public string IpAddress { get; set; } = string.Empty;
}

2. Cache Service

public class CacheService<T>
{
    private readonly UltraKVEngine<string, CacheItem<T>> _engine;

    public CacheService(string cachePath)
    {
        var config = new UltraKVConfig
        {
            MemoryModeEnabled = true,      // Full memory mode
            ValueCacheEnabled = false,     // No additional cache needed in memory mode
            FlushInterval = 60,            // Flush every minute
            CompressionType = CompressionType.LZ4
        };

        _engine = new UltraKVEngine<string, CacheItem<T>>(cachePath, config);

        // Start expiration cleanup task
        _ = Task.Run(CleanupExpiredItems);
    }

    public void Set(string key, T value, TimeSpan? expiration = null)
    {
        var item = new CacheItem<T>
        {
            Value = value,
            CreatedAt = DateTime.UtcNow,
            ExpiresAt = expiration.HasValue ? DateTime.UtcNow.Add(expiration.Value) : null
        };

        _engine.Put(key, item);
    }

    public T? Get(string key)
    {
        var item = _engine.Get(key);
        if (item == null)
            return default;

        // Check if expired
        if (item.ExpiresAt.HasValue && DateTime.UtcNow > item.ExpiresAt.Value)
        {
            _engine.Delete(key);
            return default;
        }

        return item.Value;
    }

    public void Remove(string key)
    {
        _engine.Delete(key);
    }

    public void Clear()
    {
        _engine.Clear();
    }

    private async Task CleanupExpiredItems()
    {
        while (true)
        {
            try
            {
                var expiredKeys = new List<string>();
                var now = DateTime.UtcNow;

                foreach (var key in _engine.Keys)
                {
                    var item = _engine.Get(key);
                    if (item?.ExpiresAt.HasValue == true && now > item.ExpiresAt.Value)
                    {
                        expiredKeys.Add(key);
                    }
                }

                if (expiredKeys.Count > 0)
                {
                    _engine.DeleteBatch(expiredKeys);
                    Console.WriteLine($"Cleaned up {expiredKeys.Count} expired cache items");
                }

                await Task.Delay(TimeSpan.FromMinutes(5)); // Clean every 5 minutes
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Cache cleanup error: {ex.Message}");
                await Task.Delay(TimeSpan.FromMinutes(1));
            }
        }
    }

    public void Dispose()
    {
        _engine?.Dispose();
    }
}

public class CacheItem<T>
{
    public T Value { get; set; } = default!;
    public DateTime CreatedAt { get; set; }
    public DateTime? ExpiresAt { get; set; }
}

3. Configuration Manager

public class ConfigurationManager
{
    private readonly UltraKVEngine<string, ConfigValue> _engine;

    public ConfigurationManager(string configPath)
    {
        var config = new UltraKVConfig
        {
            EncryptionType = EncryptionType.AES256GCM,
            EncryptionKey = "ConfigKey32BytesLong!@#$%^&*()",
            UpdateValidationEnabled = true,  // Configuration data needs validation
            FlushInterval = 0,               // Configuration immediately persisted
            AutoCompactEnabled = false       // Configuration data doesn't need auto compaction
        };

        _engine = new UltraKVEngine<string, ConfigValue>(configPath, config);
    }

    public void SetString(string key, string value)
    {
        _engine.Put(key, new ConfigValue { Type = "string", Value = value });
        _engine.Flush(); // Save configuration immediately
    }

    public void SetInt(string key, int value)
    {
        _engine.Put(key, new ConfigValue { Type = "int", Value = value.ToString() });
        _engine.Flush();
    }

    public void SetBool(string key, bool value)
    {
        _engine.Put(key, new ConfigValue { Type = "bool", Value = value.ToString() });
        _engine.Flush();
    }

    public string? GetString(string key, string? defaultValue = null)
    {
        var config = _engine.Get(key);
        return config?.Type == "string" ? config.Value : defaultValue;
    }

    public int GetInt(string key, int defaultValue = 0)
    {
        var config = _engine.Get(key);
        return config?.Type == "int" && int.TryParse(config.Value, out var value) 
            ? value : defaultValue;
    }

    public bool GetBool(string key, bool defaultValue = false)
    {
        var config = _engine.Get(key);
        return config?.Type == "bool" && bool.TryParse(config.Value, out var value) 
            ? value : defaultValue;
    }

    public void RemoveConfig(string key)
    {
        _engine.Delete(key);
        _engine.Flush();
    }

    public Dictionary<string, string> GetAllConfigs()
    {
        var result = new Dictionary<string, string>();
        foreach (var key in _engine.Keys)
        {
            var config = _engine.Get(key);
            if (config != null)
            {
                result[key] = config.Value;
            }
        }
        return result;
    }

    public void Dispose()
    {
        _engine?.Dispose();
    }
}

public class ConfigValue
{
    public string Type { get; set; } = string.Empty;
    public string Value { get; set; } = string.Empty;
}

Language: 中文 | English

This documentation provides comprehensive coverage of the UltraKV C# Database Engine. For the latest updates and additional resources, please visit the project repository.

⚠️ **GitHub.com Fallback** ⚠️