Ns1.desec.io replication issues

dynDNS at dedyn.io
1

Hi!

Currently I’m experiencing ns1.desec.io and ns2.desec.org out of sync for quite a while. I updated IPv4 and IPv6 with dyndns, the correct records are showing on ns2.desec.org but ns1.desec.io does not seem to be updated correctly. Googles name server on 8.8.8.8 answers sometimes with the correct records, sometimes with the old ones, and sometimes with a mix of the two on IPv4 and IPv6. Here is an example (IPs and domain redacted for privacy, I will provide them in direct message if necessary).

Correct IPs: <redacted>.244.210 and <redacted>:7f03:<redacted>; Outdated IPs: <redacted>.81.163 and <redacted>:7f0e:<redacted>.

root@OpenWrt:~# nslookup <my-domain>.de ns2.desec.org
Server:         ns2.desec.org
Address:        2607:f740:e00a:deec::2#53

Name:   <my-domain>.de
Address: <redacted>.244.210
Name:   <my-domain>.de
Address: <redacted>:7f03:<redacted>

root@OpenWrt:~# nslookup <my-domain>.de ns1.desec.io
Server:         ns1.desec.io
Address:        2607:f740:e633:deec::2#53

Name:   <my-domain>.de
Address: <redacted>.81.163
Name:   <my-domain>.de
Address: <redacted>:7f0e:<redacted>

root@OpenWrt:~# nslookup <my-domain>.de 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   <my-domain>.de
Address: <redacted>.244.210
Name:  <my-domain>.de
Address: <redacted>:7f0e:<redacted>

The web interface shows me the correct IP addresses.

Cheers!

2

Hi,

the same issue seems to exist for acme challenge. ns2.desec.org answers correctly and quickly. Unfortunately ns1.desec.io does not seem to get updated with the correct entry.

someOne@server:~$ dig TXT @ns2.desec.org _acme-challenge.<my-domain>
;; BADCOOKIE, retrying.

; <<>> DiG 9.16.1-Ubuntu <<>> TXT @ns2.desec.org _acme-challenge.<my-domain>
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43452
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1400
; COOKIE: (good)
;; QUESTION SECTION:
;_acme-challenge.<my-domain> IN        TXT

;; ANSWER SECTION:
_acme-challenge.<my-domain> 3600 IN TXT "<the-acme-challenge>"

;; Query time: 0 msec
;; SERVER: 157.53.224.1#53(157.53.224.1)
;; WHEN: So Jan 07 17:31:19 CET 2024
;; MSG SIZE  rcvd: 148

someOne@server:~$ dig TXT @ns1.desec.io _acme-challenge.<my-domain>
;; BADCOOKIE, retrying.

; <<>> DiG 9.16.1-Ubuntu <<>> TXT @ns1.desec.io _acme-challenge.<my-domain>
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41658
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1400
; COOKIE: (good)
;; QUESTION SECTION:
;_acme-challenge.<my-domain> IN        TXT

;; AUTHORITY SECTION:
<my-domain>.          300     IN      SOA     get.desec.io. get.desec.io. 2023125320 86400 3600 2419200 3600

;; Query time: 4 msec
;; SERVER: 45.54.76.1#53(45.54.76.1)
;; WHEN: So Jan 07 17:31:31 CET 2024
;; MSG SIZE  rcvd: 140
3

It might be a regional problem. DeSec routes requests to the closest server, and when I look up my domain on https://mxtoolbox.com, the answer from ns1.desec.io is correct. Queried from Germany, however, ns1.desec.io still serves the wrong records.
Maybe it is only the Frankfurt server which is out of sync.

4

Currently all is back to normal. Let’s hope it was only a one-time glitch.

5

Hi all,

this is my first post to this community – sorry that I start my contribution with a problem report.

I experience the same problems described above. A MX record for one of my domains (created 16 hours ago) is available at ns2.desec.org, but unavailable at ns1.desec.io.

I do my tests from the Munich region.

Edit:

Updating the MX record does not solve the issue, the updated record is available at ns2, but ns1 does not show the record at all.

All tests are done with dig, like so:

dig @ns2.desec.org mydomain.example.org MX
6

Hi,

Thank you for your report. Indeed, we are currently experiencing some replication problems. One aspect of our replication works by using an interface of the namseserver software (PowerDNS) to get the list of domains and their current state, both on each location and on the primary instance. We then compute a diff and initiate synchronization for those domains that are part of the diff.

For some reason, the PowerDNS interface used to collect the list of domains (and state) sometimes freezes, leading to a timeout in the replication system. We haven’t been able to identify the root cause yet. The problem indeed primarily affects out ns1 instance in Frankfurt.

It’s a bit confusing, because other locations which run the same software and version are not affected.

In any case, we made some adjustments, which we hope will improve things. We expect to fully address the problem over the next few days.

Thank you for understanding!

Stay secure,
Peter

4 Likes
7

Hi Peter,
thanks for the explanation of the problem. This already helps a lot.
Staying secure :slight_smile:
Hans

1 Like
8

Currently ns1 is out of sync again. As a workaround I blocked the ns1 IPs in my home network’s recursive resolver (relying solely on ns2). Now at least all my local devices avoid random connection issues whenever ns1 happens to be asked. But of course this is not a satisfactory solution.

Any updates on fixing the underlying issue with the replication?

9

Hi,
I’m also experiencing issues since about 24 hrs in the way that although my current IPv4 got recognized for my domains in the desec domain mgmt, however, when doing some DNS lookup, I get returned the previous (obsolete) IPv4 as a reply.
Hence my internal services are not reachable from the internet.

Hope you can fix it quite soon.

Thanks a lot.

10

I’m considering to actively use desec.io for several domains, but wondering about the above. Has the root cause ever been found @peter ?

11

HI Maarten,

Thanks for your message, and welcome to deSEC. :slight_smile:

I have to admit that we did not identify the root cause, but we also did not chase it much. Instead, we managed to fix it by making adjustments to some config settings, and then spent our energy on migrating half of our instances to Knot DNS, so that the deployment is now diversified between both PowerDNS and Knot DNS.

We have not experienced the above problem since it happened in January.

Stay secure,
Peter

3 Likes