The problem: the updater never lets the limit reset
A container auto-updater runs on a loop. Every pass it asks the registry whether each image has a newer digest, and pulls when it does. That is fine until the registry answers 429 toomanyrequests. Docker Hub rate limits pulls per IP, and anonymous pulls get the lowest allowance.
In DockWarden, an open-source Docker auto-update and health monitor written in Go, a 429 was treated as an ordinary pull failure, not as a reason to back off. The next pass retried the pull, so the host kept hitting the anonymous limit and never got out from under it. The fix shipped in v2.1.1 (2026-11-15), and the commit message states the cause plainly: a 429 is not an outage.
What we changed
- A 429 puts the whole registry host on hold. The hold starts at 10 minutes and doubles on each consecutive 429, capped at 1 hour. The Docker daemon API does not expose the rate-limit reset header, so the updater cannot wait for an exact reset time and backs off instead.
- A 401, 403, "denied" or unknown-repository pull holds that one image, not the host: 15 minutes doubling to 6 hours, warned once in the log. A private image with a bad token should not stop public images from updating.
- Held images keep answering with the local image. A container whose image was already pulled still rolls, and the health watcher is not affected, so a rate limit does not pause unhealthy-container restarts the way a registry outage does.
- docker.io images are checked at most once an hour by default (
DOCKWARDEN_HUB_INTERVAL,0means every pass). A new Hub push therefore rolls within that interval. The API's "update now" clears holds and the throttle when you need an immediate check.
The follow-up bug: a good check must not clear a 429 hold
The first version had a subtle flaw. A cheap manifest HEAD request can succeed while pulls are still limited, and that success cleared the host's hold. The escalation from 10 minutes to 1 hour never happened. The fix (v2.1.1, second commit) is that only a successful pull clears a 429 hold, and the Docker Hub throttle timestamp is written only when the digest was unchanged or the pull actually succeeded.
How we tested it
Unit tests cover the hold arithmetic, the clearing rules, and that a held image which was already pulled still rolls. The end-to-end script (test/e2e/pull-holds.sh) runs against a real Docker daemon: refused images (nonexistent repo, private repo without credentials) must be held after one attempt instead of looping, an image that is fine must keep being checked, and nothing may be recreated. A real 429 cannot be provoked from CI, so that path is covered by stub tests, including a fake registry that answers the manifest HEAD with a new digest and returns 429 for everything else. That split is what proves a good HEAD does not clear the hold.
What to do on your own hosts
- Authenticate pulls. Authenticated pulls get a higher allowance than anonymous ones, and DockWarden supports private-registry credentials through Docker secrets rather than a mounted
config.json. - Do not poll every minute. Check each image on a schedule that matches how often you actually need updates.
- Treat 429 and 401/403 differently from network errors. One means "slow down", one means "this image will not work until someone fixes credentials", and one means "try again soon".
DockWarden is MIT-licensed on GitHub. If your containers need a hands-off update and health routine built around your own stack, talk to Nexis.