GGUF Model Editor (utils) - Mungert69/GGUFModelBuilder GitHub Wiki

What is redis_utils.py?

redis_utils.py provides a robust, thread-safe, and atomic interface to a Redis-backed model catalog for tracking the status, metadata, and operations on machine learning models in your pipeline. It is the central data store for all model metadata, conversion status, quantization info, error logs, and more.

It is used by all major scripts in the codebase (conversion, web UI, automation, etc.) to add, update, query, and manage models.


Key Features

  • Atomic operations using Redis pipelines and watch/multi/exec for safe concurrent access.
  • Model catalog as a Redis hash (model:catalog), with each model stored as a JSON blob.
  • CRUD operations: Add, get, update, and delete models.
  • Field-level updates with optional conditions.
  • Batch import/export and backup/restore to/from JSON files.
  • Counter incrementing for tracking attempts, etc.
  • Import models from a list (with default values).
  • Command-line interface for direct catalog manipulation.
  • Retry logic for transient Redis errors.
  • SSL/TLS support for secure Redis connections.

Main Class: RedisModelCatalog

Initialization

catalog = RedisModelCatalog(host, port, password, user, ssl=True)

Connects to Redis with SSL, disables cert verification for testing.

Core Methods

  • load_catalog(): Returns the entire catalog as a dict {model_id: model_data}.
  • get_model(model_id): Returns a single model’s data as a dict.
  • add_model(model_id, model_info): Atomically adds a new model (returns False if it exists).
  • update_model_field(model_id, field, value, condition=None): Atomically updates a field, with optional condition (e.g., only if another field matches).
  • delete_model(model_id): Deletes a model from the catalog.
  • increment_counter(model_id, field): Atomically increments a counter field.
  • backup_to_file(file_path): Saves the entire catalog to a JSON file.
  • initialize_from_file(file_path): Loads a catalog from a JSON file (overwrites existing).
  • import_models_from_list(model_ids, defaults=None): Imports a list of model IDs, marking them as converted (or with custom defaults).

Helper/Utility

  • _safe_operation: Retries Redis operations up to 3 times on failure.
  • Value normalization for booleans, JSON, etc.

Usage

In Python Code

from redis_utils import init_redis_catalog

catalog = init_redis_catalog(host, port, password, user, ssl=True)

# Add a model
catalog.add_model("company/model", {"converted": False, ...})

# Get a model
model = catalog.get_model("company/model")

# Update a field
catalog.update_model_field("company/model", "converted", True)

# Delete a model
catalog.delete_model("company/model")

# Load all models
all_models = catalog.load_catalog()

Command-Line Interface

python redis_utils.py --host <host> --port <port> --user <user> --password <password> <command> [args...]

Commands:

  • add_model --model_id <id> --model_info '<json>'
  • get_model --model_id <id>
  • load_catalog
  • update_model_field --model_id <id> --field <field> --value <value> [--condition '<json>']
  • increment_counter --model_id <id> --field <field>
  • backup_to_file --file_path <path>
  • initialize_from_file --file_path <path>
  • import_models_from_list --model_ids <comma-separated> [--defaults '<json>']

Example:

python redis_utils.py add_model --model_id "foo/bar" --model_info '{"converted": false, "parameters": 0}'
python redis_utils.py get_model --model_id "foo/bar"
python redis_utils.py update_model_field --model_id "foo/bar" --field "converted" --value true
python redis_utils.py backup_to_file --file_path backup.json

Integration

  • Model conversion pipeline: Tracks conversion status, attempts, errors, quantizations, etc.
  • Web UI: Reads and updates catalog for search, edit, import/export, restore.
  • Automation scripts: Add new models, update status, etc.

Best Practices

  • Always use atomic methods (add_model, update_model_field) to avoid race conditions.
  • Use backup/restore before making bulk changes.
  • Use the CLI for manual fixes or batch operations.
  • Monitor Redis connection health (ping on startup).

Example: Add and Update a Model

from redis_utils import init_redis_catalog

catalog = init_redis_catalog("localhost", 6379, "password", "user", ssl=False)
catalog.add_model("test/model", {"converted": False, "parameters": 1000000})
catalog.update_model_field("test/model", "converted", True)
print(catalog.get_model("test/model"))

Error Handling

  • Retries up to 3 times on Redis errors.
  • Graceful fallback for JSON decode errors, missing models, etc.
  • Logs errors using Python’s logging module.

Summary Table

Method Purpose
add_model Add a new model (atomic, fails if exists)
get_model Get a model’s data
update_model_field Update a field (atomic, with optional condition)
delete_model Remove a model
increment_counter Atomically increment a field
backup_to_file Save catalog to JSON
initialize_from_file Load catalog from JSON
import_models_from_list Import models from a list
load_catalog Get all models as a dict

ant me to generate a Mermaid diagram or any other additions, just ask!

⚠️ **GitHub.com Fallback** ⚠️