Request was throttled

Managed DNS Hosting
1

Hi,
we use lego to request a single letsencrypt certificate that contains 5 hostnames (SANs) and we consistently run into a request limit.

lego error message:

   desec: failed to create records: domainName=acme-dns-challenge.applied-privacy.net, recordName=doh-appliedprivacy-net: error: 429: {\"detail\":\"Request was throttled. Expected available in 1 second.\"}"]}

Since we do not hit any of these documented LE limits


I assume this is a limit by the deSEC API.

What is the best option to workaround this issue since 5 SANs should not be that problematic, right?

I was also not able to find any documentation at
https://desec.readthedocs.io/
about the rate limits of the deSEC API.

2

Hi,

That’s our DNS API rate limit (for write operations). The limits we put into effect initially are indeed quite conservative, and we intended to relax them on demand. So, here you go! We’ll deploy it once the internal review went through (should be early next week).

Stay secure,
Peter

3

Thanks, for your reply.

Looking at:

I assume that the limit got increased from 3 to 6 API write calls per token/per account?

So after adding two more SANs (7 in total) I’d run into the same issue again?

Is one bulk operation for N records counted as one or N requests towards this rate limit?
https://desec.readthedocs.io/en/latest/dns/rrsets.html#bulk-operations

Should lego use bulk requests instead? (a bulk request can only cover records in a single domain)

4

Hi,

The rate limit is for DNS write operations per second per account. Bulk requests are counted only once.

It is correct that all requests are domain-specific, so you will need to do several requests when answering ACME challenges for several domains.

Our backend works with PowerDNS, and there is currently no way to make cross-domain requests reliable, in the sense that they either all succeed, or the requests fail completely if one subtask fails. For the details, see this PR discussion. Thus, no matter how we set the rate limit, the general problem persists.

At the moment, there are two ways forward:

  • create certificates per domain (is there are compelling reason not to do that?)
  • sleep for ~0.2 second between sending requests

Stay secure,
Peter

5

Thanks, I’ve now filed a lego issue for this suggesting the two options:

1 Like
6

Hello,

@peter maybe those limitations have to be added in the API documentation, because the issue on lego was a bit surprising to me.

I’m sure that you have reasons behind this rate limit, but 200ms is very slow from the point of view of an HTTP client and for a DNS API.

Also this kind of rate limiting per second can be an issue with concurrency calls or multiple API clients: in this case it’s impossible to sleep 200ms between two calls.

Could you explain this rate limit per second instead of only a rate limit per minute and per hour?

7

@peter I learned to the go desec client to follow the rate limits: read 5/s and write 3/s.

Are the new limits active? (read 10/s, write 6/s)

8

Hi ldez,

Thank you for your feedback, it is highly appreciated.

The new (slightly increased) rate limits are active, yes. I also created draft documentation on the topic (about to be merged this week). We added a chapter on rate limiting to the documentation.

The rate limit is per second because we would like to minimize the chance of concurrent requests for changes on the same domain, as PowerDNS doesn’t handle that scenario very well (we ran into related problems in the past). It would, of course, be better to enforce limits by domain (not by account), but we haven’t gotten around to implementing that. I created an issue for it.

Stay secure,
Peter

1 Like
9

Currently we are unable to renew certificates using lego 4.4.0.
429: body: {“detail”:“Request was throttled. Expected available in 1 second.”}

@ldez what do you think about making the rate limit a lego cli option so we can instruct it to be kind without changing the code?

10

Hello,

It’s an API rate limit issue, it should be fixed with lego v4.5.0

11

Hi all,

On this occasion: Rate limits for DNS write operations have been changed to be per-domain in the meantime (they are no longer per account).

Regarding the per-second rate limits, we think that it’s feasible for a smart client to know which requests it is going to make within the next second, and to aggregate these requests into a bulk request. As far as the rate limit is concerned, such bulk requests are counted once (per domain). This means that you can limit requests to ~1/s by making proper use of bulk requests, and as a result, the rate limit won’t get in your way.

Stay secure,
Peter