Architecture Overview - theweebcoders/CommercialBreaker GitHub Wiki
Architecture Overview
This document provides a technical overview of the CommercialBreaker & Toonami Tools system architecture, component relationships, and data flow.
System Overview
CommercialBreaker & Toonami Tools is a modular Python application designed to automate the creation of Toonami-style anime marathon channels. The system processes anime episodes, detects commercial break points, and integrates authentic Toonami bumps to create seamless viewing experiences.
Core Philosophy
- Modular Design: Each tool handles a specific aspect of the pipeline
- Database-Driven: SQLite manages metadata and processing state
- Interface Flexibility: Multiple UIs for different use cases
- Platform Agnostic: Works with various media servers and channel platforms
Application Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Entry Point (main.py) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
βInterface Selection: --tom | --webui | --clydes | --combreak β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββ¬ββββββββββββββββββ¬ββββββββββββββββββ
β GUI Layer β Web Layer β CLI Layer β
β β β β
β βββββββββββββββ β βββββββββββββββ β βββββββββββββββ β
β β TOM.py β β βAbsolution.pyβ β β clydes.py β β
β β (Tkinter) β β β (REMI/Web) β β β (Console) β β
β βββββββββββββββ β βββββββββββββββ β βββββββββββββββ β
βββββββββββ¬ββββββββ΄ββββββββββ¬ββββββββ΄ββββββββββ¬ββββββββ
β β β
βββββββββββββββββββΌββββββββββββββββββ
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β FrontEndLogic.py (Orchestrator API) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β’ LogicController class - Central API for all UIs β
β β’ State management via SQLite database via DataManager β
β β’ Threading for background operations β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β β² β β²
β β β β β
β β β βΌ β
β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β β β Supporting API Modules β
β β β ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββ€
β β β β FlagManager.py β MessageBroker.py β
β β β β β’ Platform compatibility β β’ Real-time communication β
β β β β β’ Global Flags β β’ In-memory pub/sub β
β β β ββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββ€
β β β β NetworkManager.py β ErrorManager.py β
β β β β β’ Network validation β β’ Centralized error handling β
β β β β β’ Config persistence β β’ Error history tracking β
β β β β (update config.py) β β’ UI error broadcasting β
β β β ββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββ€
β β β β DatabaseValidator β FilenameParser β
β β β β β’ S.A.R.A. orchestration β β’ Centralized parsing logic β
β β β β β’ 20 step validators β β’ Episode name extraction β
β β β β β’ Integrity validation β β’ Year handling (2002) β
β β β β β’ Phase-aware tracking β β’ Consistent across tools β
β β β ββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ€
β β β β NetworkUtils.py β
β β β β β’ Curl-based HTTP client β’ Wikipedia table parser β
β β β β β’ Response interface β’ Request exceptions β
β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β β β²
β βΌ β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Database β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββ€ β
β β DatabaseManager.py β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββ€ β
β β β’ Thread-safe connections β’ Transaction supportβ β
β β β’ Automatic retry logic β’ Simplified API β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β² β β
β β β βββββββββββββββββββββββββββββββ
βΌ β βΌ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Core Processing Layer β
βββββββββββββββββββ¬ββββββββββββββββββ¬ββββββββββββββββββββββββββ€
β ComBreak/ β ToonamiTools/ β ExtraTools/ β
β β β β
β β’ Commercial β β’ Show Detectionβ β’ Manual Tools β
β Detection β β’ Lineup β β’ Utilities β
β β’ File Cutting β Generation β β’ Debugging β
β β’ Cutless Mode β β’ Bump Encoding β β
βββββββββββββββββββ΄ββββββββββββββββββ΄ββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Platform Integration Layer β
βββββββββββββββββββ¬ββββββββββββββββββββββ¬ββββββββββββββββββββββββ€
β Plex API β Platform APIs β ComBreakDirect β
β β β β
β β’ OAuth Auth β β’ External REST β β’ Self-contained β
β β’ Library Scan β β’ Channel Creation β β’ Direct streaming β
β β’ Timestamps β β β’ M3U/XMLTV gen β
β β’ Smart Retry β β β’ Plex discovery β
βββββββββββββββββββ΄ββββββββββββββββββββββ΄ββββββββββββββββββββββββ
Plex Integration
CommercialBreaker includes a Plex client implementation located in API/utils/:
Plex Client Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PlexConnectionHelper.py β
β β’ Smart connection retry (local β direct β relay) β
β β’ Server discovery and connection management β
β β’ Automatic failover between connection URLs β
ββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββ
β
ββββββββββββ΄βββββββββββ
β β
βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ
β PlexClient.py β β PlexServer.py β
β β β β
β β’ PlexAuthClient β β β’ SimplePlexServ.β
β β’ OAuth PIN flow β β β’ Library access β
β β’ Account client β β β’ Section queriesβ
β β’ Resource list β β β’ Episode data β
ββββββββββββββββββββ ββββββββββββββββββββ
β β
ββββββββββββ¬ββββββββββββ
βΌ
ββββββββββββββββββββββββββ
β CurlHttpClient β
β (ToonamiTools/) β
β β’ HTTP operations β
β β’ Stdlib based β
ββββββββββββββββββββββββββ
Key Benefits:
- Minimal Dependencies: Uses Python stdlib
urllibandCurlHttpClient - Connection Reliability: Automatically tries all available server URLs
- Lightweight: Only implements features actually used by CommercialBreaker
- Stability: Direct control over Plex API interactions
Integration Points:
LoginToPlex.py- OAuth authentication and server selectionGetTimestampPlex.py- Fetch "Skip Intro" timestampsPlexAutoSplitter.py- Split merged episodesRenameSplitPlex.py- Rename split episodes to match file names
In-Memory Message Broker
A unified in-memory message broker is responsible for all real-time communication between the LogicController and the various user interfaces (GUI, Web, CLI).
- Channel-Based Communication: All UIs subscribe to relevant channels for updates and publish user actions/events.
- Decoupled Integration: Interfaces interact with the LogicController exclusively through the message broker, ensuring modularity and testability.
Database Management
The CommercialBreaker uses a centralized DatabaseManager for all database operations, providing:
- Thread-safe connections: Each thread gets its own database connection
- Automatic retry logic: Handles database locks with exponential backoff
- Transaction support: Atomic operations with automatic commit/rollback
- Simplified API: Common operations wrapped in convenient methods
- Dictionary-based operations: Queries return lists of dictionaries (
fetchall_as_dicts(),bulk_insert_dicts(), etc.)
All modules access the database through get_db_manager() from API.utils.DatabaseManager, ensuring consistent error handling and preventing database lock issues in multi-threaded scenarios.
Centralized Error Handling
The system employs a centralized error handling mechanism via ErrorManager.py, which provides:
- Global Error Tracking: All errors are logged and can be retrieved by any interface
- UI Broadcasting: Errors can be sent to all interfaces for real-time user feedback
- Error History: Maintains a history of errors for debugging and user support
- Modular Error Handling: Each module can raise errors that are caught and processed by the ErrorManager
- Custom Error Types: Allows for specific error handling based on module needs
S.A.R.A. Validation System
The S.A.R.A. (System Analysis and Reporting Assistant) validation system provides comprehensive database validation and diagnostics. It follows the philosophy: "Validate everything, trust nothing."
Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DatabaseValidator β
β (Main Orchestration Layer) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β’ Coordinates 20 validators (18 step + 2 integrity) β
β β’ Consolidates issues intelligently β
β β’ Reports to ErrorManager β broadcasts to all UIs β
β β’ Phase-aware progress tracking β
ββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β BaseValidator β
β (Abstract Base for All Validators) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β’ Database access utilities (via DatabaseManager) β
β β’ Bump vs anime detection (consistent with main tools) β
β β’ Mode detection (cutless vs traditional) β
β β’ Platform detection (DizqueTV, Tunarr, ComBreakDirect) β
β β’ Issue creation and management β
ββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
ββββββββββββββββΌβββββββββββββββ¬βββββββββββββββ
β β β β
βΌ βΌ βΌ βΌ
ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ
βStep Validatorβ β Integrity β βPhaseTracker β βValidationRes.β
β (18 total) β β Validators β β β β β
β β β (2 total) β ββ’ Maps steps β ββ’ Result β
ββ’ Platform β β β β to phases β ββ’ Issue β
ββ’ ToonamiCheckβ ββ’ Lineup β ββ’ Progress β ββ’ Level β
ββ’ LineupPrep β β Integrity β β tracking β ββ’ Status β
ββ’ CommercialBrβ ββ’ Referential β ββ’ Conditional β β β
ββ’ (15 more) β β Integrity β β step logic β β β
ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ
Key Features
Step Validation:
- Each pipeline step has a dedicated validator
- Validates table existence, structure, and data quality
- Checks cross-table relationships and referential integrity
- Detects platform compatibility issues
Integrity Validation:
- LineupIntegrityValidator: Validates bump placement rules (multibumps β anime, intro β matching BLOCK_ID, etc.)
- ReferentialIntegrityValidator: Validates cross-table consistency (BLOCK_IDs exist, file paths valid, etc.)
Phase-Aware Tracking:
- Maps individual steps to logical workflow phases (0-6)
- Tracks which phase is complete vs in-progress
- Accounts for conditional steps (PlexAuth, BumpCalculator, etc.)
Issue Consolidation:
- Groups related issues to prevent overwhelming users
- Three consolidation strategies:
- Version-based (e.g., "Version 2: missing data, Version 3: invalid timestamps")
- List-items (e.g., "3 folder issues: β’ ANIME_FOLDER missing β’ BUMPS_FOLDER missing β’ ...")
- Generic numbered list for other multi-issue scenarios
Mode Detection:
- Automatically detects cutless vs traditional mode
- Identifies platform (DizqueTV, Tunarr, ComBreakDirect)
- Warns about incompatibilities (e.g., cutless mode with Tunarr)
UI Integration: Page8 Diagnostics
S.A.R.A. is accessible through Page8 in the GUI interfaces (TOM and Absolution):
Access Method:
- Hidden Panic Button: Click any page title 5 times in 2 seconds
Features:
- Pipeline status with completion checklist (β/β for each step)
- Run full validation button
- Refresh status button
- Copy results to clipboard (for bug reports)
- Real-time status updates during validation
- Grouped issues by severity (CRITICAL, ERROR, WARNING, INFO)
Hidden Panic Button
Implementation: Track clicks on page title labels. 5 clicks within 2 seconds β navigate to Page8.
Purpose: Provides quick access to diagnostics without memorizing menu structure, especially useful during critical errors.
Location: Implemented on all pages except Page5 in TOM (complex layout).
Validation Output
Each validation issue includes:
- Where: Which step, table, and row (if applicable)
- What: User-friendly message describing the problem
- Why: Technical details for developers
- How to Fix: Actionable suggestion
- When: Timestamp
Example Issue:
[ERROR] CommercialBreaker β cuts
Message: 3 episodes missing commercial break timestamps
Details: Files: Naruto S01E05.mkv, Naruto S01E06.mkv, Bleach S02E03.mkv
Suggestion: Re-run CommercialBreaker in normal mode (not low power) to detect breaks
Integration with FrontEndLogic
Methods Added to LogicController:
def run_full_validation(self):
"""Run comprehensive database validation (background thread)"""
def get_pipeline_status(self) -> PipelineStatus:
"""Get quick pipeline status without full validation"""
For complete S.A.R.A. documentation, see S.A.R.A. Validation System.
Filename Parsing
The FilenameParser utility provides centralized, consistent episode filename parsing across all modules.
Location: /ToonamiTools/utils/FilenameParser.py
Features:
- Automatic year stripping from show names (e.g., "Naruto (2002)" β "Naruto")
- Comprehensive SxxExx pattern extraction
- Season and episode number parsing (handles single and double digits)
- Validation and error reporting
Used By:
- VirtualCut (cutless mode operations)
- DirectoryScanner (file discovery)
- BlockMaker (BLOCK_ID generation)
- PlexToDizqueTV (channel creation)
- All validators (consistency check)
Benefits:
- Single source of truth for filename parsing
- Consistent behavior across entire codebase
- Automatic year handling from show names
- Easier to maintain and test
Plex Connection Reliability
Enhanced Plex connection reliability with retry logic and progressive timeouts.
Improvements:
- Retry Logic: Up to 3 attempts with configurable delay
- Progressive Timeout: Increases with each retry (base + 20s per attempt)
- Status Broadcasting: Real-time updates to user during retries
- Configuration-Driven:
config.PLEX_RETRY_ATTEMPTS,config.PLEX_RETRY_DELAY, timeouts
Modified Files:
LoginToPlex.py- PlexLibraryManager and PlexLibraryFetcherPlexServer.py- SimplePlexServer with configurable timeout
Why: Intermittent failures on slower networks or when servers are starting up. Retry logic makes connection more reliable without user intervention.
Data Flow
1. Initialization Phase
User Input β Interface Selection β Configuration Loading β Platform Selection β Plex Authentication (conditional)
Note: Platform selection (DizqueTV, Tunarr, or ComBreakDirect) determines whether Plex authentication is required. ComBreakDirect users skip Plex authentication entirely.
2. Content Discovery Phase
Plex Library Scan β Show Detection (IMDB/Wikipedia) β Bump Analysis β Database Population
3. Processing Phase
Episode Selection β Commercial Detection β File Processing β Metadata Generation
4. Integration Phase
Lineup Generation β Platform Channel Creation β Playback Optimization
Platform Options:
- External Platforms: Traditional platform integration via REST APIs
- ComBreakDirect: Self-contained streaming server (new)
5. ComBreakDirect Streaming Phase (Optional)
ComBreakToComBreakDirect β ComBreakDirect Server β Live MPEG-TS Stream β Plex/Clients
β
βββΆ LineupExtender (per channel)
β threading.Timer
β fires INFINITE_EXTEND_LEAD_MS
β before channel end
βΌ
InfiniteChannelExtender
(ShowScheduler + CutlessFinalizer)
β
βΌ
FactoryFloor.extend_channel
(append-only, persists channels.json)
ComBreakDirect channels are never finite. The LineupExtender's per-channel timer fires on wall-clock time (independent of whether a client is streaming) and appends a fresh ~40 hour chunk before the existing one runs out. The channel survives indefinitely until the LineupExtender is explicitly disabled or _infinite_meta.enabled is set to false on the channel.
ComBreakDirect Architecture
ComBreakDirect is an optional self-contained streaming server using Studio β FIFO β Broadcast FFmpeg β BroadcastTower architecture. It provides:
- Continuous MPEG-TS Streaming: True broadcast-style streaming with seamless program transitions
- BroadcastTower Multi-Client: Efficient distribution to multiple simultaneous viewers
- WebUI: Toonami-themed landing page with setup instructions
- M3U8 Generation: Dynamic playlist creation for Plex/Jellyfin
- XMLTV Guides: Full EPG data with show metadata
- Plex/Jellyfin Discovery: HDHomeRun-style auto-discovery
- Commercial Injection: Server-side ad break management with pre-rendering
- Intelligent Audio Selection: Configurable audio track selection (defaults to English for Toonami)
- Auto Lifecycle: Studio and broadcast FFmpeg start/stop based on client connections
- Infinite Channel Extension: Per-channel
threading.Timerarms ahead of channel end, runs ShowScheduler+CutlessFinalizer to append the next chunk, then reschedules β wall-clock based
ComBreakDirect Components
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ComBreakDirect Server (Flask + Studio + BroadcastTower) β
β Port 8083 β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β WebUI (UI/) β β
β β β’ Landing page (setup instructions, copy-to-clipboard) β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββΌββββββββββββββββββ
βΌ βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββ ββββββββββββββββββββ
β Loading Dock β βFactory Floor β β Unloading Dock β
β (docks/) β β (docks/) β β (docks/) β
β β β β β β
β β’ Process lineup β β β’ Store data β β β’ Studio thread β
β β’ Inject ads β β β’ Generate β β β’ Broadcast β
β β’ Pre-render β β M3U8/XMLTV β β FFmpeg β
β breaks β β β’ Persist β β β’ BroadcastTower β
β β β channels β β β’ Multi-client β
ββββββββββββββββββββ ββββββββββββββββ ββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Supporting Utilities (utilities/) β
β β’ BroadcastTower - Multi-client streaming engine β
β β’ AudioTrackSelector - Intelligent audio selection β
β β’ CommercialBreakRenderer - Pre-rendering system β
β β’ CleanupManager - Automatic file maintenance β
β β’ configuration - Path resolution β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Infinite Channel Extension (docks/ + ToonamiTools/) β
β β’ LineupExtender (docks/) - Per-channel threading.Timer β
β watchdog, fires before channel end β
β β’ InfiniteChannelExtender (ToonamiTools/) - Builds one β
β extension chunk via ShowScheduler + CutlessFinalizer β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Broadcasting Architecture (Unloading Dock):
Studio Thread β FIFO β Broadcast FFmpeg β BroadcastTower β Antennas
(full tape, (pipe) (+genpts, (distributes (per-client
normalizes continuous) to all) buffers)
programs)
Key Features:
- Loading Dock (
docks/LoadingDock.py): Processes cutless lineup data, injects commercials between bumps, pre-renders breaks.format_extensionreuses the program-builder to splice extension chunks onto the existing channel timeline. - Factory Floor (
docks/FactoryFloor.py): Generates M3U8 playlists and XMLTV guides, stores channel configurations.extend_channelappends new programs under the factory lock;_maybe_spawn_extenderarms a LineupExtender per infinite channel atstore_channeland on_load_channelsrestart. - Unloading Dock (
docks/UnloadingDock.py): Manages studio threads, broadcast FFmpeg processes, and BroadcastTower distribution - BroadcastTower (
utilities/BroadcastTower.py): Multi-client streaming engine with Antenna pattern - WebUI (
UI/WebUI.py): Landing page with setup instructions and copy-to-clipboard buttons - Audio Selector (
utilities/AudioTrackSelector.py): Configurable audio track selection (defaults to English) - Commercial Renderer (
utilities/CommercialBreakRenderer.py): Pre-renders breaks to eliminate startup delays - Cleanup Manager (
utilities/CleanupManager.py): Automatically removes old pre-rendered breaks - Lineup Extender (
docks/LineupExtender.py): Per-channelthreading.Timerscheduler, firesINFINITE_EXTEND_LEAD_MS(default 3h) before the last program ends, then reschedules after each extension - Infinite Channel Extender (
ToonamiTools/InfiniteChannelExtender.py): One-shot orchestrator the LineupExtender calls β runs ShowScheduler withcontinue_from_last_used_episode_block=Trueagainst a per-extension SQLite table, then CutlessFinalizer, then returns the rows for LoadingDock to format
Technical Architecture:
- Studio Thread: Calculates channel position, spawns FFmpeg per program with normalization, writes to FIFO
- Broadcast FFmpeg: Reads FIFO with
+genpts, creates continuous stream, rate limits broadcasting - BroadcastTower: Receives stream, distributes to all Antenna objects (one per client)
- Antenna Buffer: Per-client 132-chunk buffer (~2 seconds) with rate limiting
- Auto Lifecycle: Studio and broadcast FFmpeg only run when clients connected
- Real-time Sync: All clients synchronized to same channel position
- Wall-Clock Extension: LineupExtender's timer fires on real time, independent of playback β channels grow even when no client is streaming. The watchdog reads the last program's
stopISO timestamp, not Studio'scurrent_index. - Cursor-Driven Continuation:
ComBreakToComBreakDirect._seed_episode_cursorpopulateslast_used_episode_blockfrom the initial lineup before the channel is POSTed, so the first extension picks up at E+1 of each show (not the same E0 again, not a random jump) - Append-Only Growth:
FactoryFloor.extend_channelonly ever callsprograms.extend()β never replaces or reorders β so the Studio'sprograms[i % len(programs)]reads stay coherent across the seam without read-side locking
Integration Points:
ComBreakToComBreakDirect- Pushes cutless lineup to server via REST API; setsinfinite=True+infinite_metaon the payload for ComBreakDirect channelsLogicController._ensure_combreakdirect_server()- Auto-starts server- Flask REST API for channel management and streaming
- HDHomeRun discovery for Plex DVR integration
- WebUI served at root path for easy access
config.INFINITE_EXTEND_LEAD_MS- Tunable lead time (default 10 800 000 ms = 3h) for when the LineupExtender fires ahead of channel end
For detailed ComBreakDirect documentation, see ComBreakDirect.md.
This architecture supports the system's goals of modularity, reliability, and extensibility while maintaining the performance necessary for processing large media libraries efficiently. The addition of ComBreakDirect provides an alternative deployment model that eliminates external platform dependencies.