Firmware Management and OTA - Kush-Kelaiya22/imPress GitHub Wiki

Firmware Management and OTA

How firmware gets from a build to classroom devices: the registry, approval, staged deployments, what each device does, rollback, and releasing a new version. References: docs/guides/ota-updates.md, docs/firmware/OTA_ARCHITECTURE.md, docs/firmware/FIRMWARE_VERSIONING.md.

Contents


What can be updated over the air

Device Over the air How
Gateway (C6) ✅ it downloads the image itself over Wi-Fi
Hub (S3) ✅ prompted through its gateway, it briefly joins Wi-Fi (the "OTA hop"), downloads, installs and reboots; if nothing is pending it returns to the mesh without rebooting
Student module ❌ no transport: ESP-NOW frames are at most 250 bytes, and modules have no Wi-Fi credentials. Update over USB (Firmware: Build and Flash)

First v2.1 install on gateways and hubs: over USB, once. v2.1 changed the hub–gateway SPI format, and the gateway's bootloader gained rollback. After that, both update over the air.

The big picture

flowchart LR
    B["Build<br/>impress_class_c6.bin"] --> U["Upload<br/>(Firmware page)"]
    U --> V{"valid imPress image?<br/>chip · project · X.Y.Z ·<br/>SHA-256 · size · signature"}
    V -- no --> R["refused with the reason"]
    V -- yes --> A["registry: uploaded"]
    A --> AP["Approve"] --> D["Deploy<br/>(devices + strategy)"]
    D --> E["deployment engine<br/>(every 5 s)"]
    E -- "offer + prompt" --> G["gateway"]
    G -- "SPI prompt" --> S3["hub"]
    G -- "check · download · status" --> E
    S3 -- "check · download (Wi-Fi hop)" --> E
    S3 -- "result after reboot (via gateway)" --> E
Loading

The firmware registry

Every uploaded image is a row in firmware_artifacts, plus a file named by its SHA-256 (backend/firmware_bins/<sha256>.bin):

Property Rule
Facts come from the image target (c6, s3, student), chip, project, version, ESP-IDF version, build date and ELF hash are all read from the binary, never from the file name or a form field
Immutable a (target, version) pair names exactly one set of bytes. Uploading the same bytes again is a no-op; different bytes under an existing version are refused (409): bump the version instead
Content-addressed stored as <sha256>.bin, so no user input ever reaches a file path
Status uploaded → approved → deprecated (and back to approved if needed)
Channel stable or beta: a label for your own use
Release notes free text; editable
Signer which key signed it, or unsigned (Security)

Uploading an image

Firmware → Upload an image: drop build/impress_<project>.bin on the box. The upload is checked:

Check Refuses
magic byte 0xE9, a plausible segment count, a minimum size random files
chip id ∈ {ESP32, ESP32-S3, ESP32-C6} images for other chips
an app descriptor (magic 0xABCD5432) bootloader or partition-table binaries
project ∈ {impress_class_c6, impress_class_s3, impress_student}, built for its own chip other applications; a gateway project built for the wrong chip
version is X.Y.Z builds without a proper version.txt
the image's appended SHA-256 is present and correct corrupted or truncated files
the image fits the target's OTA slot images that couldn't be installed
a valid signature, if any is attached; a signature by the site key, if one is configured tampered images; unsigned or foreign images when signing is required
at most 4 MB oversized uploads (413)

Every refusal names the problem, for example must be built for esp32c6, image version '1' is not X.Y.Z, or the image is not signed with this server's firmware key.

To check an image before uploading:

scripts/verify_firmware.py firmware/class_c6/build/impress_class_c6.bin [--key site.pub]

Approve, deprecate, delete

Action Effect
Approve the image can be deployed. Only approved images are offered anywhere
Deprecate no new deployments use it. Devices already running it, or about to install it, are unaffected, and it stays downloadable for them
Delete only when the image is not approved, and no device runs it or has it pending: those images are the way back. It removes the row and the file
History its upload, approval, pushes and every result devices reported

The page groups images by device type, newest first, and shows how many devices run each image or have it pending. The highest approved version is marked latest.

Deploying

Deploy on an approved image opens the rollout dialog:

  1. Choose devices (the filters combine):
    • specific devices;
    • every device of the image's type;
    • devices running a given version;
    • a classroom's devices (its gateway and the hub it relays).
  2. Set the strategy (below).
  3. Allow downgrade only for a deliberate rollback.
  4. Click Preview. It lists every device that will be included, with its stage, and every one that won't, with the reason.
  5. Click Start. The canary stage begins at once.

Push OTA on the Modules page is the same thing for one device: a one-device deployment with the same checks.

Strategy: canary, batches, limits

Setting Default Meaning
Canary 1 devices in the first stage
Batch size 5 devices in each later stage
At once (max concurrent) 5 devices updating at the same time (network load)
Failures allowed per stage 0 a finished stage with more failures than this pauses the deployment
Attempts 2 tries per device before it counts as failed
Timeout 900 s per attempt
flowchart LR
    C["stage 0: canary"] --> H1{"failures ≤ allowed?"}
    H1 -- yes --> B1["stage 1: next batch"] --> H2{"…"} --> BN["last batch"] --> DONE["completed"]
    H1 -- no --> P["paused<br/>(the note says which stage)"]
    P -- "Resume (accept the failures)" --> B1
    P -- Cancel --> X["cancelled"]
Loading

So with the defaults, one device updates first. Only if it succeeds do the others follow, five at a time.

Who is excluded, and why

Reason in the preview Meaning
incompatible: a s3 device can't run a c6 image wrong device type
no over-the-air path to student modules yet (update over serial) student modules
device is disabled an admin switched it off
image is not signed with the site firmware key (#66) signing is required, and this image isn't signed by the site key
already running 2.2.0 nothing to do
would downgrade 2.2.0 → 2.1.0 (confirm a rollback) tick Allow downgrade if that's intended
already in deployment #7 busy in another rollout

Following a deployment

The Deployments list shows each rollout's image, state and counts per state, and refreshes itself while a rollout runs. Details shows every device: its stage, state, attempts, versions before and after, and the error if any.

Control Effect
Pause no new stage starts; devices already updating finish
Resume continues; accepts the failures that caused a pause
Cancel devices still waiting, queued or precheck are cancelled (nothing has been written to their flash); devices further along finish and report

Durability:

  • Restarts: everything is stored in the database. The scheduler runs every 5 s and is idempotent, so after a backend restart running deployments carry on where they were.
  • Double submits: an idempotency key makes a double submit return the same deployment instead of starting two.

The per-device state machine

stateDiagram-v2
    [*] --> waiting: in a later stage
    [*] --> queued: stage active
    waiting --> queued: stage starts (concurrency permitting)
    queued --> precheck: device asks firmware/check
    precheck --> downloading
    downloading --> verifying
    verifying --> installing
    installing --> rebooting
    rebooting --> health_check
    health_check --> success: running the expected version
    queued --> cancelled
    precheck --> cancelled
    waiting --> cancelled
    queued --> unreachable: timeout
    downloading --> timed_out: timeout
    rebooting --> rolled_back: the bootloader reverted
    health_check --> rolled_back
    downloading --> failed
    installing --> failed
    failed --> waiting: retry (attempts left)
    timed_out --> waiting: retry
    unreachable --> waiting: retry
Loading
Rule Why
Forward only. A device may skip states (the hub reports only its final result) but never go back: 409. Repeating the current state is harmless relayed duplicates and lost acknowledgements can't confuse it
Success needs proof. success counts only when the device reports the expected version running after the reboot a finished download or a reboot isn't success
Rollback is recorded as such. rolled_back keeps the device's old version, and the bad image isn't offered again
Bounded retries. failed, timed_out and unreachable are retried while attempts remain
Timeouts. An attempt that never started becomes unreachable; one that stalled becomes timed_out power loss, network trouble
The server observes progress too. Asking firmware/check moves a device to precheck; serving the download moves it to downloading progress is visible even from older clients

What the gateway does during an update

sequenceDiagram
    participant BE as Backend
    participant C6 as Gateway (running v1)
    participant BL as Bootloader
    BE->>C6: WS ota_update {device_type: c6, mac}
    C6->>BE: firmware/check → v2, sha256, size (precheck)
    C6->>BE: GET firmware/download (downloading)
    Note over C6: writes into the inactive slot,<br/>hashing the stream (SHA-256)
    C6->>BE: ota/status verifying
    Note over C6: digest == offered sha256?<br/>esp_ota_end (chip, segments, hash, signature if required)
    C6->>BE: ota/status installing → rebooting
    C6->>BL: reboot (NVS: ota_target = v2)
    BL->>C6: boot v2 (pending verification)
    Note over C6: SPI up, Wi-Fi up, registration answered 200
    C6->>BE: ota/status health_check
    Note over C6: mark the image valid
    C6->>BE: ota/status success (version v2)
    Note over C6,BL: no Wi-Fi or backend within 120 s → reboot →<br/>bootloader reverts → v1 reports rolled_back
Loading

The gateway also checks for an update at every boot, so it picks up prompts it missed while offline. It ignores an offer it can't verify: a missing or short SHA-256, a non-semver version, or a zero size. It writes nothing to flash if the download's HTTP status or length is wrong.

What the hub does during an update

  1. The backend sends the prompt to the hub's gateway, which forwards it to the hub over SPI.
  2. The hub joins Wi-Fi (the mesh keeps running), registers, and asks firmware/check.
  3. Nothing pending: it disconnects and goes back to the mesh channel, without rebooting.
  4. Update pending: it downloads (checking the HTTP status and size), installs and reboots.
  5. The new image marks itself valid once the mesh and SPI are up. A crash before that makes the bootloader revert.
  6. After the reboot, the hub reports applied (or rolled_back, or failed with an error code) through the gateway.

Rollback

Kind How
Automatic, on the device a new image must prove itself after the reboot. Otherwise the bootloader boots the previous image, which reports rolled_back. The gateway's proof: SPI, Wi-Fi and a successful registration within 120 s. The hub's proof: mesh and SPI start up
Operator, for a fleet deploy an older approved image with Allow downgrade ticked. It is recorded as a rollback deployment
Manual, for one stuck device flash it over USB (Recovery Procedures)

Releasing a new version

  1. Bump the version in firmware/<project>/version.txt for each project that changed, and in VERSION for a release. A repository test requires all four to match.
  2. Build: idf.py build, or use the CI artifact (each image comes with SHA256SUMS). With signing on: scripts/build_signed.sh (Security).
  3. Check it: scripts/verify_firmware.py <image> [--key site.pub].
  4. Upload it, and check the version, target and chip the page shows.
  5. Approve it.
  6. Deploy it with a canary.
  7. Write release notes.

What a version number doesn't tell you:

  • Bootloader and partitions: two devices on 2.1.0 can have different bootloaders or partition tables, because OTA never changes those.
  • Build profile: an image built with the reduced-flash or signed profile reports the same version as a normal build. Deploy such builds only to the devices built the same way.

Safety properties

Property Mechanism
A crashing image can't brick a gateway or hub app rollback and self-verification after reboot
Only an image an admin deployed can be downloaded firmware/download serves only that device's pending version (403 otherwise)
A corrupted download is never installed SHA-256 against the offer (gateway); esp_ota_end's own checks (both)
A foreign image is never installed (with signing) the signature check in esp_ota_end, and the backend refusing it at upload
A failed canary stops the rollout stage failure limits pause the deployment
A no-op prompt doesn't disturb a class the hub returns to the mesh without rebooting
A restart doesn't lose a rollout durable state, idempotent scheduler

Troubleshooting updates

Symptom Check
The hub never starts a hop Is the hub linked to a gateway (gateway_id)? Is that gateway connected (online, WebSocket up)?
OTA aborted — could not connect to WiFi (hub log) the hub's Wi-Fi settings (NVS wifi_ssid, wifi_pass)
No pending update (hub log) nothing pending for it, or it's already on that version
Download 403 the requested version isn't the device's pending one: deploy again
Download 404 / 410 no registered image for that type and version / its file was deleted from the store
failed with sha256 mismatch corruption on the way. It is retried while attempts remain
failed from esp_ota_end on a signed device the image isn't signed with the key the device trusts: check with verify_firmware.py --key
rolled_back the new image couldn't prove itself. Read its serial log, fix, bump the version, deploy again
The deployment is paused open Details, read the failed device's error; Resume to accept, or Cancel
A device stays queued it never asked firmware/check: offline, no WebSocket, or (hub) no gateway. It becomes unreachable at the timeout
⚠️ **GitHub.com Fallback** ⚠️