As I am working on an improved external-dns webhook, external-dns-desec-webhook-rs, the desec-rs library (also available on docs.rs/desec) is maturing. Its current version, v0.0.2, is symbolic in the sense that I don’t run production workloads on it yet. It is so far very tested and featureful:
- Respects rate limits at multiple scopes; avoid HTTP 429s that throttle, handle the rate limit using configurable waiting periods, which can be set to zero for upstream clients that require fail-fast semantics (e.g. external-dns).
- Supports efficient pagination using lazy streams
The motivation for making was that the Go webhook kept breaking in old and new ways. A common pattern was that the webhook would spam the API, the API would throttle the webhook, and the webhook would stay throttled partly because of whatever reason made it get throttled, and partly because the throttling became self-reinforced.
One category for getting throttled is keep trying to add a record that either can’t be added, or was already added, but went unnoticed:
- One reason a record can fail to get added is e.g. that it’s malformed; external-dns defaults to using TXT records to keep track of what records are being managed, and when the record-under-management lives directly on the apex (example.com, rather than foo.example.com), then the “ownership TXT record” becomes something like a-example.com, which the deSEC API correctly rejects. Not a bug in the provider per se, but a faulty default in external-dns that a webhook provider should help avoid by documenting ideal usage.
- Another reason a record can fail to get recognized as being added is record normalization. Scenario: external-dns says “create this record”, API says “OK!”, external-dns polls and does not recognize that it happened, because the created record looks different, external-dns proceeds to ask again, and again, and again, exhausting the rate-limit. I’ve only had problems with this in production with TXT records, but since I’m remaking the library and the webhook, I’ve been digging into what types of records get normalized by the deSEC API.
The desec-rs library is made to support the external-dns webhook, but I will also eventually port the terraform provider, where a rate-limiter is just as important, if not moreso. Having a strong foundation means the webhook can focus on high-level stuff and let the client library deal with low-level stuff.
For example, the original Go webhook was almost required to set `–interval 15m`, which means to only poll the webhook every 15 minutes instead of the default every 1 minute, because it would eat the daily rate limit in 5 hours. The Rust webhook doesn’t care what you set the interval to, because it doesn’t poll the API more than necessary, and it doesn’t forward every single request from external-dns.
Both crates live on the newly established:
I will be spending my spare time in the next weeks maturing the webhook before I move onto creating a fixed cert-manager webhook. Judging from how deep you can rabbit-hole a single problem, it might take a while before I start, since I expect certificates have their own edge cases.