01 — Problem

The Fragility of Single-Image Systems

In high-radiation or electrically noisy environments, a single bit-flip in the flash memory (SEU) or a partial write during an OTA update can render a device completely bricked. Standard recovery mechanisms often rely on a single backup, but if both the primary and backup are corrupted, the system fails catastrophically.

Triple Redundant OTA partitions (ota_0, ota_1, ota_2) to ensure a "Golden Image" always exists
3×3 Matrix Two-layer TMR voting for RAM and CPU operations to eliminate transient faults
Zero Downtime. The system repairs itself in the background while the main control loop continues
02 — Architecture

Autonomous Self-Healing Loop

The system implements a "Watchdog-Reparation" cycle. Upon boot and periodically during runtime, the agent scans all available OTA slots. If an invalid image is detected, the system dynamically selects the healthiest available slot as the source to rebuild the corrupted partition.

PARTITION SCAN FAULT DETECTED SELF-HEAL REPAIR OK

Technical Detail: The "Golden Image" is not fixed. Any partition that passes the ESP_ERR_IMAGE_VALID check can act as the source, allowing the system to evolve its firmware across all slots while maintaining a safety net.

03 — Mitigation Matrix

From Random Failure to Deterministic Recovery

The system treats hardware instability as a data-corruption problem, applying a tiered defense strategy across different memory and compute domains.

Failure Scenario Mitigation Strategy System Outcome
Flash Bit-Flip (OTA Partition) Triple Redundancy and SHA-256 Autonomous repair from valid slot
RAM Corruption (Critical Var) Two-Layer TMR (3×3 Matrix) Correct value recovered via voting
Transient CPU Fault (Math) Symmetric Redundant Compute Majority vote eliminates outliers
System Hang (Flash Op) Task Watchdog Timer (TWDT) Deterministic reboot to valid image
04 — TMR Implementation

Combatting Compiler Optimization

A naive implementation of redundancy (a+b, a+b, a+b) is often collapsed into a single operation by the -O2 or -Os compiler flags. To ensure true physical redundancy, I used volatile qualifiers, compiler memory barriers, and expression diversity. 

// TMR-wrapped addition with expression diversity
static float add(float a, float b) {
    float res[3][3];
    for (int i = 0; i < 3; i++) {
        for (int j = 0; j < 3; j++) {
            volatile float tmp;
            if ((i + j) % 3 == 0)      tmp = a + b; // standard
            else if ((i + j) % 3 == 1) tmp = b + a; // swapped
            else                      tmp = a; tmp += b; // accumulation
            res[i][j] = tmp;
            asm volatile("" ::: "memory"); // Memory Barrier
        }
    }
    return twoStageTMR(res);
}
a+b b+a a;+b a+b b+a a;+b a+b b+a a;+b 3×3 Matrix Layer 1 Voter Layer 2 Voter VALID FIG 2.0 — Two-Stage TMR Voting Logic: Eliminating Single Points of Failure
05 — Validation

Fault Injection and Recovery Log

Verification was performed using a Python-based fault injector that writes garbage data to specific OTA flash offsets. The logs show the system detecting an ESP_ERR_IMAGE_INVALID state and initiating a background repair from a secondary valid partition. 

# Fault Injection Sequence
[TEST]       Corrupting non-running partition ota_0...
[OK]           Corrupted ota_0

[INFO]       Scanning all partitions.
[ERROR]      ota_0 is INVALID (err: ESP_ERR_IMAGE_INVALID)
[INFO]       ota_1 is VALID
[INFO]       ota_2 is VALID
[INFO]       Using ota_2 as source for repairs
[INFO]       Attempting to repair invalid partition: ota_0
[INFO]       Repairing ota_0 <- ota_2 (size: 1.3 MB)
[INFO]       Repair successful, ota_0 is now VALID

<Normal Task> System active - 0ms interruption observed.

Verification: The baseOperation task maintained 100% uptime during the corruption and repair of ota_0, proving the architecture is truly fail-operational.

06 — Impact

From Fragile to Fail-Operational

By decoupling boot validity from a single memory location and implementing computational redundancy, the system mitigates the primary risks of unattended operation in radiation-heavy or electrically noisy environments.

  • Zero-Touch Recovery: No manual reset or JTAG intervention required after catastrophic flash corruption.
  • TMR Precision: Mathematical operations are protected against transient CPU faults.
  • Continuous Availability: Repair cycles happen in parallel to the main control task.
  • Scalable Redundancy: Triple-slot architecture allows for simultaneous "Golden Image" and "A/B Update" patterns.
Technical Repository