IoT specifications are hard for a reason that has nothing to do with complexity: the system under test is not in the room, and it is not online when you ask it to be.
The behaviour that matters most — what happens when the network drops, when a device is slow, when half the fleet is on old firmware — is precisely the behaviour that is hardest to reproduce and easiest to get wrong.
This guide covers the four areas where fleet specifications break.
Offline behaviour is the default case, not the exception
Consumer devices are offline a meaningful fraction of the time. A specification that assumes connectivity is a specification of the easy path.
Feature: Command delivery to a device
Scenario: A command issued to an offline device is queued
Given device "DEV-4412" has been offline for 12 minutes
When the operator sends command "restart_sensor" to device "DEV-4412"
Then the command is queued for delivery
And the command is reported as "pending" in the device's status
And no delivery attempt has been made yet
Scenario: A queued command is delivered when the device reconnects
Given device "DEV-4412" has been offline for 12 minutes
And command "restart_sensor" is queued for device "DEV-4412"
When device "DEV-4412" reconnects
Then command "restart_sensor" is delivered within 30 seconds
And the command is reported as "acknowledged"
The first scenario’s third clause — "no delivery attempt has been made yet" — is the one that distinguishes a queue from a retry loop. A system that starts retrying against an offline device burns battery and, on a metered cellular connection, money. Whether the platform attempts delivery immediately or waits for a reconnect signal is a rule.
Retry semantics need a full specification
"Retry three times" is a quarter of a specification. What is needed is the schedule, the backoff, and what happens when the retries are exhausted.
Scenario: A command is retried with exponential backoff
Given device "DEV-4412" is online but not acknowledging
When command "restart_sensor" is sent to device "DEV-4412"
Then delivery attempts occur at 0, 5, 15 and 35 seconds
And each attempt waits twice as long as the previous attempt
Scenario: A command that is never acknowledged is marked as failed
Given device "DEV-4412" is online but not acknowledging
When command "restart_sensor" is sent to device "DEV-4412"
And 60 seconds elapse
Then the command is reported as "failed"
And the failure reason is "acknowledgement_timeout"
The backoff schedule is stated as absolute offsets rather than as "exponentially". Exponential backoff has many variants — with jitter, without, capped, uncapped — and the jitter alone changes the schedule enough to matter when you are debugging a fleet that is not behaving.
The jitter question deserves an explicit clause when you use jitter, because un-jittered retry storms across a large fleet are a real operational failure:
Scenario: Retries across the fleet are not synchronised
Given 10,000 devices are online but not acknowledging
When command "collect_diagnostics" is sent to all devices
Then no two devices retry within the same 500 ms window
Acknowledgement is not delivery
The distinction between "the command reached the device" and "the device confirmed it acted" is the single most important distinction in a fleet specification, and it is usually absent.
Scenario: A command is reported as sent but not acknowledged
Given device "DEV-4412" is online
When command "set_interval" is sent to device "DEV-4412"
Then the command is reported as "sent" immediately
And the command remains "sent" and not "acknowledged" until the device confirms
And the device's configuration is unchanged until the acknowledgement arrives
Note the last clause. "Sent" is a statement about the transport; "acknowledged" is a statement about the device. An implementation that updates the desired configuration at send time — rather than at acknowledgement — will show the fleet’s configuration as correct while the devices are doing something else entirely. That mismatch is invisible until someone queries a device directly.
Firmware rollout is a staged operation with rules
Rollouts have stages, a pause between them, health criteria, and a rollback. All of it is specification, and most of it is left to a configuration file that nobody reviews.
Scenario: A rollout pauses when the error rate exceeds the threshold
Given a firmware rollout is in stage 2 of 4
And 5% of devices in stage 2 report a boot failure
When the rollout health check runs
Then the rollout is paused
And no further devices are updated
And the rollout reports "paused_health_threshold"
Scenario: A paused rollout is rolled back
Given a firmware rollout is paused with reason "paused_health_threshold"
When an operator requests a rollback
Then all devices in stage 2 are returned to the previous firmware version
And devices in stages 3 and 4 are not updated
The second scenario matters: the rollback covers only the devices that were updated. Rolling back everything, including devices that never received the bad firmware, wastes an enormous amount of time on a large fleet and can itself cause an outage.
Also state the thresholds. "The error rate exceeds the threshold" is not checkable; "5% of devices in stage 2 report a boot failure" is, and it can be checked by the platform rather than by a human reading a dashboard.
Fleet-scale operations need their own clauses
Any operation that touches every device has a blast radius, and the specification is where the blast radius is bounded.
Scenario: A fleet-wide command is applied in batches
Given the fleet contains 250,000 devices
When command "rotate_credentials" is sent to the whole fleet
Then devices are updated in batches of 1,000
And at most 50 batches are in flight at any time
A fleet-wide operation that is not batched is an outage waiting for a reason. The batch size and the concurrency limit are exactly the kind of numbers that belong in a clause a reviewer can look at.
What to leave implicit
- The transport. Whether commands travel over MQTT, HTTP, or a vendor protocol is not a business rule, and it changes often.
- Device-side data structures. How the device stores the command locally, and how it deduplicates, is a device concern unless it is externally observable.
- Hardware-specific behaviour. Sensor drift, sampling rates, and calibration are physical, not contractual.
The review question for this domain
For each clause, ask:
Does this hold when the network is bad?
If the answer is "we think so, but it was written assuming connectivity", the clause is not ready. Fleet behaviour under adverse conditions is the whole domain; a specification that describes only the happy path has described about 5% of the system.
Related
- Gherkin feature files — format and conventions
- Gherkin for game backend APIs — state machines under different pressure
- Templates — a starting point