Volnux: Execution You Can Reason About - nshaibu/volnux GitHub Wiki

Volnux: Execution You Can Reason About

Most workflow systems fail at the moment they’re needed most.

  • A pipeline crashes mid-run.
  • Some tasks succeeded, others didn’t.
  • Retries run blindly.
  • State is lost. And no one is quite sure what actually happened or what should happen next.

Volnux was born from that frustration.

Volnux is an open-source workflow execution platform designed to make execution explicit, recoverable, and governable—not hidden inside Python control flow or scheduler internals.

It is not just another scheduler or orchestrator. Volnux treats workflows as first-class execution graphs, with clear semantics, observable state, and resilient runtime behaviour.

The Core Idea

Execution should be visible, resumable, and understandable—by humans, not just machines.

Most orchestrators focus on when tasks run. Volnux focuses on how execution flows, what state exists, and how systems recover.

To achieve this, Volnux combines:

  • A readable graph-based DSL (Pointy-lang)
  • A resilient execution runtime
  • A collaboration-first model for defining and sharing operations

1. Workflows as Explicit Execution Graphs

Volnux workflows are defined using Pointy-lang, a simple, arrow-based language that makes control flow obvious at a glance.

ValidateInput -> ProcessData || EnrichData -> PersistResults

This is not “Python code that happens to be a DAG.” It is a declarative execution graph with clear meaning for:

  • Sequencing
  • Parallelism
  • Conditional branching
  • Error paths
  • Retries and fallbacks

Because execution is explicit:

  • Engineers understand what runs and why
  • Operators can debug without reading source code
  • Non-engineers can review and reason about workflows, even if they don’t implement them

Pointy-lang workflows double as executable documentation.

2. Events as Governed, Reusable Execution Units

In Volnux, workflows are composed from Events.

An Event is not just a function call. It is a versioned, governed execution template.

Events can:

  • Be versioned ([email protected], @v1, @latest)
  • Be deprecated with changelogs
  • Define retry policies and result evaluation
  • Exist locally, in PyPI packages, or in shared GitHub registries
  • Be reused across teams and workflows

At runtime, Events become tasks—but the Event remains the source of truth.

This cleanly separates:

  • Implementation (engineers)
  • Workflow design (analysts, platform teams)
  • Execution governance (operations)

3. Resilient Execution, Not Blind Retries

Volnux is designed for failure—because real systems fail.

Its runtime includes:

  • Rehydration: restore execution state after crashes and continue safely
  • Adaptive execution: scale workers and queues based on actual load
  • Hybrid concurrency: threads for I/O, processes for CPU, optional native bindings
  • GPU-aware execution with automatic CPU fallback

Execution state is preserved and inspectable. Retries are intentional, not guesswork.

This makes Volnux especially well-suited for:

  • MLOps pipelines
  • Computer vision workloads
  • Large-scale data processing
  • Long-running, partially successful jobs

4. Built for Scale: Batch Pipelines

Volnux supports Batch Pipelines as a first-class concept.

Define a pipeline once. Volnux automatically:

  • Splits large inputs into batches
  • Spawns multiple pipeline instances
  • Executes them in parallel
  • Tracks success and failure independently

This turns massive jobs into many small, fault-tolerant executions—without rewriting your workflow.

5. Observability Is Not an Afterthought

Volnux workflows are observable by design.

  • Execution graphs are visualizable (XDOT integration)
  • Runtime emits signals and metrics
  • OpenTelemetry support is built in
  • Success and error paths are explicit, not hidden in logs

You don’t just know that something failed. You know where, why, and what can safely resume.

Why Volnux Exists

Airflow made scheduling popular—but execution opaque. Prefect improved ergonomics—but still hides control flow in code. Dagster added structure—but workflows remain developer-centric.

Volnux takes a different path.

It treats execution as a governed system, not a side effect of running Python.

The result is a platform that is:

  • Powerful enough for large-scale ML and data systems
  • Explicit enough for teams to reason about together
  • Resilient enough to survive real-world failure

Volnux in One Sentence

Volnux is an execution platform where workflows are readable, recoverable graphs—not just scheduled tasks.