Skip to main content

Devices stop communicating with the backend after a DNSSEC root key rollover

Update the DNSSEC root trust anchors on the device to include the current root Key Signing Key.

Written by Jakub Kubinski

Environment

Issue

  • Devices stop communicating with our backend, but the SIMs show as Online in the Onomondo app.

  • Devices stop communicating shortly after a DNSSEC root Key Signing Key (KSK) rollover.

  • Devices send a high volume of repeated DNS queries but never connect to the backend.

  • Traffic Monitor shows successful DNS responses (NOERROR) with valid A records. It shows no TCP/TLS traffic to the backend.

  • DNS resolution on the device fails with: DNSSEC validation failed: missing-key.

Solution

Update the DNSSEC root trust anchors on the device so they include the root KSK currently signing the root zone. Affected devices cannot reach your backend for over-the-air updates, so apply the fix remotely over Onomondo OpenVPN.

  1. Connect to the affected device:

    1. Set up your OpenVPN client.

    2. Enable VPN access for your user in the Onomondo app.

    3. Get the IPv4 address of the SIM from SIMs or GET SIMs API endpoint.

    4. Connect to the device over SSH using that IP address.

  2. Confirm the cause. Resolve your backend domain on the device, fx. with resolvectl query <your-backend-domain>. The output shows a DNSSEC validation error, fx. missing-key.

  3. Update the root trust anchors following the instructions of your device’s OS or DNS resolver vendor. Use the current root key data published by IANA.

  4. Resolve your backend domain again. The domain resolves, and the device reconnects to the backend.

  5. Repeat steps 1–4 on each affected device.

  6. Release a firmware update with the current root trust anchors for your whole fleet.

Warning: Test the trust anchor update on a single device before applying it to the rest of your fleet.

If you are still experiencing this issue, reach out to us at [email protected].

Cause

DNSSEC validation starts from a trust anchor: the public root Key Signing Key (KSK) stored on the validating resolver. The root KSK is replaced periodically in a process called a root key rollover. After a rollover, the root zone signs its DNSKEY record set with the new key.

Onomondo DNS resolvers resolve queries and do not perform DNSSEC validation. Devices that validate locally send queries with the Checking Disabled (CD) flag set. This leaves validation to the device.

A device with an outdated trust anchor cannot validate the new root signature. It rejects the DNS answer with a validation error, fx. missing-key. The application never receives an IP address and keeps retrying, which causes a spike in DNS queries.

The SIM stays Online because cellular registration and the data session are not affected. Only name resolution on the device fails. This is also why remote access over OpenVPN works.

The fix must be applied on the device. The trust anchor is part of the device’s own configuration.

To avoid this issue in future rollovers:

  • Enable automated trust anchor updates (RFC 5011) on your devices where supported.

  • Keep root trust anchors current in your firmware.

  • Follow root key rollover announcements on the IANA Trust Anchors and Rollovers page.

Did this answer your question?