GGUF Model Editor (utils) - Mungert69/GGUFModelBuilder GitHub Wiki

redis_utils.py provides a robust, thread-safe, and atomic interface to a Redis-backed model catalog for tracking the status, metadata, and operations on machine learning models in your pipeline. It is the central data store for all model metadata, conversion status, quantization info, error logs, and more.

It is used by all major scripts in the codebase (conversion, web UI, automation, etc.) to add, update, query, and manage models.

• Atomic operations using Redis pipelines and watch/multi/exec for safe concurrent access.
• Model catalog as a Redis hash (model:catalog), with each model stored as a JSON blob.
• CRUD operations: Add, get, update, and delete models.
• Field-level updates with optional conditions.
• Batch import/export and backup/restore to/from JSON files.
• Counter incrementing for tracking attempts, etc.
• Import models from a list (with default values).
• Command-line interface for direct catalog manipulation.
• Retry logic for transient Redis errors.
• SSL/TLS support for secure Redis connections.

catalog = RedisModelCatalog(host, port, password, user, ssl=True)

Connects to Redis with SSL, disables cert verification for testing.

• load_catalog(): Returns the entire catalog as a dict {model_id: model_data}.
• get_model(model_id): Returns a single model’s data as a dict.
• add_model(model_id, model_info): Atomically adds a new model (returns False if it exists).
• update_model_field(model_id, field, value, condition=None): Atomically updates a field, with optional condition (e.g., only if another field matches).
• delete_model(model_id): Deletes a model from the catalog.
• increment_counter(model_id, field): Atomically increments a counter field.
• backup_to_file(file_path): Saves the entire catalog to a JSON file.
• initialize_from_file(file_path): Loads a catalog from a JSON file (overwrites existing).
• import_models_from_list(model_ids, defaults=None): Imports a list of model IDs, marking them as converted (or with custom defaults).

• _safe_operation: Retries Redis operations up to 3 times on failure.
• Value normalization for booleans, JSON, etc.

from redis_utils import init_redis_catalog

catalog = init_redis_catalog(host, port, password, user, ssl=True)

# Add a model
catalog.add_model("company/model", {"converted": False, ...})

# Get a model
model = catalog.get_model("company/model")

# Update a field
catalog.update_model_field("company/model", "converted", True)

# Delete a model
catalog.delete_model("company/model")

# Load all models
all_models = catalog.load_catalog()

python redis_utils.py --host <host> --port <port> --user <user> --password <password> <command> [args...]

Commands:

• add_model --model_id <id> --model_info '<json>'
• get_model --model_id <id>
• load_catalog
• update_model_field --model_id <id> --field <field> --value <value> [--condition '<json>']
• increment_counter --model_id <id> --field <field>
• backup_to_file --file_path <path>
• initialize_from_file --file_path <path>
• import_models_from_list --model_ids <comma-separated> [--defaults '<json>']

Example:

python redis_utils.py add_model --model_id "foo/bar" --model_info '{"converted": false, "parameters": 0}'
python redis_utils.py get_model --model_id "foo/bar"
python redis_utils.py update_model_field --model_id "foo/bar" --field "converted" --value true
python redis_utils.py backup_to_file --file_path backup.json

• Model conversion pipeline: Tracks conversion status, attempts, errors, quantizations, etc.
• Web UI: Reads and updates catalog for search, edit, import/export, restore.
• Automation scripts: Add new models, update status, etc.

• Always use atomic methods (add_model, update_model_field) to avoid race conditions.
• Use backup/restore before making bulk changes.
• Use the CLI for manual fixes or batch operations.
• Monitor Redis connection health (ping on startup).

from redis_utils import init_redis_catalog

catalog = init_redis_catalog("localhost", 6379, "password", "user", ssl=False)
catalog.add_model("test/model", {"converted": False, "parameters": 1000000})
catalog.update_model_field("test/model", "converted", True)
print(catalog.get_model("test/model"))

• Retries up to 3 times on Redis errors.
• Graceful fallback for JSON decode errors, missing models, etc.
• Logs errors using Python’s logging module.

ant me to generate a Mermaid diagram or any other additions, just ask!