Inconsistent propagation between authoritative nameservers during ACME DNS-01 challenge

[philip.raschke]

Hello deSEC Team,

I am experiencing a reproducible issue when obtaining Let’s Encrypt certificates via ACME DNS-01 using a deSEC-managed zone.

Environment

• DNS provider: deSEC

• Domain: example.com

• ACME clients tested:

  • Caddy 2.11.4 (caddy-dns/desec v1.1.0)

  • lego

The issue occurs with both clients, so it does not appear to be specific to Caddy.

Observed behaviour

Both ACME clients successfully create the TXT record via the deSEC API.

However, the two authoritative nameservers appear to become inconsistent for some time afterward.

Immediately after the TXT record is created:

dig @ns1.desec.io TXT _acme-challenge.immich.nas.example.com +short

returns the expected challenge token, whereas

dig @ns2.desec.org TXT _acme-challenge.immich.nas.example.com +short

returns no record (NXDOMAIN).

Because lego verifies propagation against the authoritative nameservers before proceeding, it eventually fails with:

NS ns2.desec.org. returned NXDOMAIN for _acme-challenge.immich.nas.example.com

Second ACME attempt

If I immediately retry the certificate request, the behavior changes:

• ns1.desec.io already serves the new challenge token.

• ns2.desec.org now serves the previous challenge token.

Let’s Encrypt then fails with:

During secondary validation:
Incorrect TXT record "<old token>" found at
_acme-challenge.immich.nas.example.com

This behavior is reproducible.

Conclusion

It appears that TXT record updates become visible on ns1.desec.io significantly earlier than on ns2.desec.org.

During ACME DNS-01 validation, this results in inconsistent answers from the authoritative nameservers:

1. First attempt:

   • ns1.desec.io → new token

   • ns2.desec.org → NXDOMAIN

2. Second attempt:

   • ns1.desec.io → new token

   • ns2.desec.org → previous token

Since the issue can be reproduced with both Caddy and lego, it does not appear to be client-specific.

Could you please investigate whether there is a replication or synchronization delay between the authoritative nameservers?

Thank you very much in advance.

Best regards,

Philip

[peter]

Hi Philip,

Thanks for your message, and welcome to deSEC!

ns1 and ns2 do not use different software, so the behavior you’re observing likely is a network connectivity issue etc.

Note that ns1 and ns2 each are 7 or 8 servers. It’s possible this is happening at only one of the 15 locations total, which happens to be the one you see as ns2.

There is no technical reason why ns2 should lag one update behind ns1. Rather, the issue seems to be that the ns2 instance you’re hitting is catching up less quickly than other instances. If you wait a little longer, they should be in sync.

We’re aware of suboptimal replication timing, and are planning to improve the situation by replacing our replication mechanism (but it will not be done tomorrow).

Stay secure,
Peter

[philip.raschke]

Hi Peter,

Thanks for your quick and detailed response and for welcoming me to deSEC.

That makes sense and also matches what I observed during testing. Shortly after my post, the propagation had completed, and the ACME validation succeeded (it had consistently failed all day before). Eventually, I successfully obtained the certificate!

My takeaway is that in such situations, it is best to let the ACME client continue retrying automatically rather than triggering new validation attempts. The authoritative nameservers will converge over time, and the validation will succeed without further intervention.

Thank you also for the information about the planned improvements to the replication mechanism.

Best regards,

Philip

[Forst]

Hi folks,

I’ve been gradually working on getting certbot-dns-desec a bit more up to date in a fork, with the ultimate goal of building a snap package and having it tested (and upstreaming all of it, of course).

(commit messages in the fork are intentionally lazy, I want to get all of this working first before I make them clean and atomic, so please ignore for now)

Running the end-to-end tests in GitHub Actions I’m seeing the same problem, where from time to time DNS records won’t propagate in the default 80 seconds:

Certbot failed to authenticate some domains (authenticator: dns-desec). The Certificate Authority reported these problems:
  Domain: certbot-2-11-1-python-3-10.***
  Type:   unauthorized
  Detail: Incorrect TXT record "IzuZMlYb1Fl3UHDJ9TjZdHQToI0HUBcVz6NLRB_iBfE" (and 1 more) found at _acme-challenge.certbot-2-11-1-python-3-10.***

Job log: Bump actions/checkout and action/setup-python and pin to specific SHAs · Forst/certbot-dns-desec@9e0935f · GitHub

At the same time there’s a commit by Nils in a pending PR that bumps the propagation time to 5 minutes: Run action on demand and once a month by nils-wisiol · Pull Request #27 · desec-io/certbot-dns-desec · GitHub

Do we know what the actual propagation delay is nowadays, do you per chance keep any statistics?

I’m a bit hesitant to bump the delay by this much, especially with me limiting the parallelism in the Python/certbot version matrix, as I was otherwise hitting Let’s Encrypt’s rate limits.

Thanks!

[nils]

I don’t have up-to-date numbers but at GitHub - desec-io/desec-debug: Collection of Tools for Debugging Distributed DNS Systems · GitHub we got some - not officially maintained - tooling that can help collecting statistics. Let us know if you need help debugging.

Best,

Nils

[Forst]

Hello Nils,

This was useful, thank you! The tool you linked inspired me to write a slightly different one: GitHub - Forst/desec-heatmap: Unofficial tool to test deSEC change propagation · GitHub

It creates a random TXT record on a given domain/subname through the deSEC API, and then periodically (2 seconds by default) pokes the known geographic servers until they all return the same new random TXT value. The output looks like this:

Each vertical column is 2 seconds. Green means “good”/done, “yellow” means it hasn’t synced yet (no record, or multiple records in the RRset), “red” means error (timeout, refusal etc).

Made a few runs, and here are some observations:

• Most *.a.desec.io servers sync very quickly, typically within 4-6 seconds.
• jnb-1.a.desec.io consistently and significantly lags behind. In the bottom run it took 156 seconds.
• All of *.b.desec.io aren’t responding to external queries, and hence were excluded.
• *.c.desec.io are also quite slow to sync changes, though the purpose of these is not known to me. These weren’t listed in the desec-io/desec-debug list of frontend servers, I found them through exploration. Thought I’d include those anyway. The worst run I’ve seen so far was 170 seconds for sin-1.c.desec.io.

These values are approximate, but should be enough to get a general idea of how it’s doing.

With these kind of figures, it wouldn’t be surprising that certbot plugiun’s default 80-second propagation window may sometimes not be enough.