An embedded prototype can run for hours on a bench and still fail within days after deployment. The reason is simple: bench testing usually exercises the nominal path, while the real environment adds power disturbances, temperature, weak networks, timing collisions, mechanical stress, unexpected resets and long-term state accumulation.
Deployment reliability therefore requires testing the fault model, not only the feature list.
1. Power quality changes outside the lab
Bench supplies are usually cleaner than real USB adapters, batteries, long cables and shared industrial rails. Radio transmission, motors, relays and displays can create short current peaks that pull the supply below the MCU’s safe operating voltage.
Many MCUs include brownout protection for exactly this reason. ESP-IDF, for example, documents an ESP32 brownout detector that resets the system when the supply falls below a configured safe level.
If a board resets only when Wi-Fi, a motor or another heavy load becomes active, inspect the power rail at the MCU pins with an oscilloscope rather than assuming firmware crashed. See our ESP32 brownout troubleshooting guide.
2. Temperature exposes marginal designs
Oscillators, regulators, sensors, batteries and semiconductors all change behavior with temperature. A design that is stable at 23 °C may violate timing or voltage margins in a hot enclosure or cold outdoor installation.
Validate the complete operating-temperature range specified for the product, not only the room-temperature behavior of one board.
3. A watchdog is not useful if you never test it
A watchdog exists to recover from a system that stops making progress. Espressif documents separate watchdog mechanisms for interrupt starvation, task starvation and boot-time problems.
Do not simply “feed the watchdog everywhere.” Define what healthy progress means and feed it only after the relevant subsystem proves that it is alive. Otherwise a deadlocked system can keep petting the watchdog and remain broken forever.
4. Network code assumes the internet is always available
Real Wi-Fi disappears. Routers reboot, DHCP leases change, DNS fails and cloud endpoints time out.
Firmware should distinguish between:
- Temporary link loss.
- Authentication failure.
- DNS failure.
- Server rejection.
- Application-level invalid responses.
Use bounded retries and backoff instead of tight reconnect loops. Keep local control functions working when cloud connectivity is unavailable. For ESP32 specifically, use our guide to automatic Wi-Fi recovery.
5. Long uptime reveals memory leaks and fragmentation
A firmware image that runs correctly for 30 minutes can still fail after thousands of repeated allocations, network reconnects or message-processing cycles.
Track free heap and, where the platform exposes it, the largest free block. Repeatedly exercise the paths that allocate and free memory. Do not accept “the heap comes back approximately” without measuring whether fragmentation is accumulating.
6. Stack usage changes under rare call paths
Error logging, recursive parsing, large local arrays and deep callback chains can use far more stack than the normal path.
In an RTOS, inspect task high-water marks during worst-case scenarios. In bare-metal systems, use linker information and stack instrumentation where available. Stack corruption can look like random hardware behavior because it damages return addresses and state.
7. Race conditions appear when timing shifts
Adding network traffic, interrupts or another task changes scheduling. Shared-state code that “never collided” on the bench can suddenly fail in the field.
Audit variables shared between ISRs, tasks and cores. volatile alone does not make compound operations atomic or provide general synchronization. See C bugs that become hardware failures for common examples.
8. Sensors drift or encounter inputs never seen in calibration
Production data can leave the range used during calibration. Sensors may warm up, age, become contaminated, or be mounted differently from the prototype.
Firmware should detect out-of-range, disconnected and implausible data instead of silently converting every ADC number into a “valid” measurement.
9. Mechanical connections fail after vibration and handling
Loose headers, poorly supported wires, marginal solder joints and connectors without strain relief may survive a desk test but fail after repeated movement.
Inspect the mechanical load path. Solder is an electrical interconnect; it should not be the only structure resisting repeated cable force. Our soldering-defect guide covers several failures that pass an initial continuity test.
10. Persistent state becomes inconsistent after resets
Power can disappear while flash or EEPROM state is being updated. A robust design must consider what happens if a reset occurs between every pair of persistent-storage operations.
Use versioned records, checksums/CRCs where appropriate, and update schemes that can distinguish a complete record from an interrupted write. Do not treat flash as if it were transactional RAM.
11. Firmware updates can brick devices
OTA is not production-ready merely because downloading a new image works. The device also needs image validation, power-loss behavior and a recovery strategy.
ESP-IDF’s OTA subsystem, for example, supports app rollback so a new application can be marked invalid and the device can return to the previous working image. The exact mechanism differs by platform, but the principle is universal: never make one unverified image the only bootable path.
12. Error logs disappear exactly when you need them
A production system should preserve enough evidence to distinguish brownout, watchdog, software panic, deliberate restart and power-on reset.
At minimum, record:
- Reset reason.
- Firmware version/build ID.
- Last major state or operation.
- Key fault counters.
- Network status.
- Relevant supply/sensor telemetry if available.
A device that “recovered automatically” but leaves no diagnostic trail is still hard to maintain.
13. Timeouts are tuned for the bench
Fast local networks hide bad timeout design. Field networks may have seconds of delay or brief outages. The correct timeout depends on the operation, not on one successful bench measurement.
Separate connect timeout, request timeout and application deadline where practical. Make retries bounded and observable.
14. The product has no safe degraded mode
Ask what the device should do when a sensor fails, cloud access disappears or configuration is corrupted. “Reset repeatedly” is not a useful system state.
A safe product often needs a degraded mode: local control continues, outputs move to a defined state, and the fault is reported for service.
15. Only one board was tested
One prototype cannot represent manufacturing variation. Component tolerance, regulator behavior, oscillator accuracy, PCB assembly quality and sensor offsets vary across units.
Before release, test multiple boards and deliberately include edge conditions such as minimum supply, maximum load and temperature extremes relevant to the design.
A deployment test plan that catches real failures
- Power-cycle the device hundreds of times at different points in operation.
- Interrupt power during configuration and firmware-update operations.
- Disconnect and restore the network repeatedly.
- Restart routers/brokers/servers while the device remains powered.
- Run maximum electrical load and monitor supply voltage.
- Exercise watchdog recovery deliberately.
- Run for days while tracking heap, stack and fault counters.
- Test invalid sensor data and disconnected peripherals.
- Verify the safe state after every critical failure.
- Confirm logs survive long enough to diagnose the incident.
Primary references
- Espressif ESP-IDF: watchdog timers
- Espressif ESP-IDF: brownout and fatal-error diagnostics
- Espressif ESP-IDF: OTA updates and rollback
Bottom line
Bench success proves that the nominal path works. Deployment reliability requires proving recovery from bad power, bad timing, bad networks, resets, corrupted state and long uptime. Design the failure behavior before the environment designs it for you.