What to do when something has really gone wrong: a failed update, a device that won't boot, a lost key, a bad upgrade. The procedures run from the most automatic to the most manual, and each says how to tell it worked. Reference: docs/firmware/RECOVERY_PROCEDURE.md.

None of these procedures has been rehearsed on hardware yet. Rehearse them on one bench device before you need them.

1. A device rolled itself back
2. A deployment paused or timed out
3. Roll a fleet back to an earlier version
4. A device can't update at all: reflash over USB
5. The firmware signing key is lost
6. A device was reflashed or replaced
7. A device was lost or stolen
8. Undo a backend upgrade
9. Restore the database from a backup
10. The admin password is lost

You see: the device shows rolled_back on the deployment, its health is ERROR (last OTA update rolled back), and it still runs its old version.

What happened: the new image couldn't prove itself after rebooting, so the bootloader went back to the previous one:

• gateway: no SPI, Wi-Fi and successful registration within 120 s, so its watchdog rebooted;
• hub: a crash before its mesh and SPI were up.

Do:

1. Open the deployment's Details and read the device's error.
2. The rollout paused itself (the default allows no failures per stage), so no other device got the image. Don't resume until you know why.
3. If you can, reproduce it on a bench device with a serial log. Typical causes: the new image's Wi-Fi or backend settings, TLS turned on before the server serves TLS, or a crash.
4. To fix the image: bump the version, rebuild, upload, approve, deploy.
5. To abandon it: Cancel the deployment and Deprecate the image.

It worked when: the device's health returns to ONLINE after a later successful update. A rollback leaves ERROR showing until then.

• Resume accepts the failures so far and continues with the next stage.
• Cancel stops devices that haven't started installing. Devices already installing finish and report.

Use this for a version that installed fine but behaves badly. Device rollback only covers the update that just failed; this is a deliberate downgrade.

1. Firmware page: Approve the older image again if it was deprecated.
2. Deploy it to the devices, and tick Allow downgrading devices on a newer version (rollback). Without it they are excluded as would downgrade.
3. The deployment is recorded as a rollback, and goes through the same canary and stages.

It worked when: the devices report success with the older version.

Use this when a device boots neither image, is stuck in a reset loop (the boot count rising while its uptime stays short), or must move to a different partition layout.

1. Check out the version you want, and build:

   git checkout <tag or commit>
   cd firmware/<project> && idf.py build

   For a signed fleet: IMPRESS_SIGNING_KEY=… scripts/build_signed.sh <project>, and flash from build/signed.
2. Flash everything: bootloader, partition table, initial OTA data and app.

   idf.py -p <port> flash monitor

   idf.py flash writes fresh OTA data, so the device boots the image you just flashed, whichever slot ran before.
3. If its stored configuration is suspect, erase everything first. That loses Wi-Fi, backend address, keys and class:

   idf.py -p <port> erase-flash flash monitor

   The device then starts from its compiled-in defaults (idf.py menuconfig → the imPress configuration menu).
4. If its flash was erased, it lost its device key: Reset key for it on the Modules page, so it can enrol again (below).

It worked when: the device registers, the Modules page shows the expected version, and its health is ONLINE.

Devices on the signed profile accept only images signed with that key, so they can no longer be updated over the air.

1. Create a new key in a new directory: scripts/firmware_key.sh ~/.impress/firmware-signing-2.
2. Point IMPRESS_FIRMWARE_SIGNING_KEY at the new public key, and restart the backend.
3. Reflash each signed device over USB with an image built with the new key (above).

Prevent it: keep the private key offline, backed up in two places.

Reflashed with its flash erased: the device lost its device key, but the server still holds that key as active, so the device's registrations with the shared key are refused (403, "has its own key").

1. Modules → Diagnostics → Reset key.
2. The device registers again on its next attempt (restart it to hurry it), and is issued a new key.

Replaced by a new board: the new board has a new MAC address and registers as a new device.

1. Link it to the room (its room code, or as an admin).
2. Disable the old device's row.

1. Disable it: Modules → access off. Every request for its MAC is then refused, whichever key it presents, and it is excluded from deployments.

2. Rotate the shared key if the device might be read out (its flash holds the shared key, unless flash encryption is on):

   1. set a new IMPRESS_DEVICE_API_KEY on the server;
   2. put it into any device that still has no key of its own.

   Devices with an active key of their own aren't affected.

3. If it was a hub or gateway with Wi-Fi credentials, consider changing the Wi-Fi password.

1. Stop the service: sudo systemctl stop impress.
2. Restore the database from the backup the migration wrote before upgrading (below). Anything written after the upgrade is lost.
3. Restore the code: git checkout <previous release>, then scripts/install.sh.
4. Start: sudo systemctl start impress. GET /health shows the old version and schema_version.

Devices don't need to change:

• The v2.1 web app works with v2.1 firmware as before.
• A v2.1 gateway against a v2.0 backend still does presence and quizzes, but not OTA updates.
• Device keys: a v2.0 backend doesn't know device keys, so devices that hold one get refused. Before rolling the backend back to before #66, reset every device's key, or flash firmware without device keys.

Before applying a migration, the backend copies the database next to itself as impress.db.bak-<from>-to-<to>-<timestamp>.

sudo systemctl stop impress
cd /opt/imPress/backend
ls impress.db.bak-*                                   # pick the one taken before the upgrade
cp impress.db impress.db.failed-upgrade               # keep the evidence
cp impress.db.bak-6-to-7-20261009101500 impress.db
# deploy the code that matches that schema, then:
sudo systemctl start impress

• Running the new code again: a restored backup has the old migration state, so starting the new code on it runs the migration again. Fix the cause first, for example a duplicate section that stopped the upgrade.
• Clean up: backups contain personal data and are git-ignored. Delete them once an upgrade is confirmed.
• Your own backups: for scheduled backups, see Installation → Backups.

• Another admin exists: they use Users → Reset pwd. An admin can't reset a super admin's password; another super admin can.
• The only super admin is locked out: there is no reset link. With shell access to the server, run a short Python one-off against the database that sets a new bcrypt hash for the admin user (stop the server first, and back up impress.db). Or start a new database: IMPRESS_INITIAL_ADMIN_PASSWORD only applies when the database has no users.