Rate limiters are infrastructure, not products — users never interact with them directly. This changes how you gather requirements. Instead of asking "what does the user do?", you ask "what does the system protect?" and "what behavior counts as excess?" The scope of the rate limiter is defined by the APIs it protects and the limits placed on each.
The first critical scope decision is where the rate limiter lives. Three options were considered: embedded inside each API server, as a standalone service before the API servers, or at the API gateway layer. API gateway was chosen because it is centralized (all traffic flows through it), independently scalable (can be bought off-shelf or built), and avoids the coordination problem of multiple API servers each running their own counter. If each API server maintained its own limit, a user could send 5 requests to server A and 5 to server B and exceed the limit without either server knowing.
The second scope decision is identifier type. IP-based limiting (block by IP address) vs user ID-based limiting. User ID was chosen for this session to keep scope manageable — IP-based adds complexity around proxies, shared IPs, and VPNs. The architecture supports both; user ID is the cleaner starting point.
| Requirement | Decision |
|---|---|
| Placement | API gateway layer — centralized, independently scalable, avoids multi-server counter sync |
| Identifier | User ID based for this session. IP-based is extensible but adds VPN/proxy complexity. |
| Limit: login | 5 login requests per minute per user |
| Limit: post | 3 post requests per minute per user |
| Response: allowed | Pass through. Return X-RateLimit-Remaining: N header on every response. |
| Response: throttled | HTTP 429 Too Many Requests. Return X-RateLimit-Reset: <timestamp> header. |
| IP-based limiting | Out of scope for this session — extensible later |
| Per-endpoint limiting | Login and post are the two types covered. Other endpoints extensible. |
| DAU | 10M daily active users |
| Peak RPS | 1M requests per second at the API gateway |
| Latency | Rate limiter must add <1ms to each request. Must not be user-perceptible. |
| Availability | Fail open — if rate limiter is down, allow requests through rather than blocking all traffic |
| Accuracy | Slight over-counting acceptable at boundaries. Burst control more important than exact precision. |
If the rate limiter goes down and we fail closed (block all requests), we've taken down our own system to prevent abuse. That's worse than the abuse itself. Fail open means a brief window of uncontrolled traffic while the rate limiter recovers — acceptable. In practice, this is mitigated by adding a fallback local counter at each gateway node (see edge cases in step 6). The organization can also run a backup application-level rate limiter as a secondary defense for critical APIs.
Rate limiters are unusual because the most important persona is not the legitimate user — it's the bad actor. This is one of the rare systems where a negative persona (someone trying to abuse the system) is the primary design driver. Without the bad actor, there's no reason to build a rate limiter at all.
- Makes normal API calls within limits
- Expects fast response (sub-millisecond gate)
- Should never see a 429 under normal use
- May benefit from seeing X-RateLimit-Remaining to manage API usage
- Makes automated, high-frequency requests
- Credential stuffing (repeated login attempts)
- DDoS — flood specific endpoints
- Scraping — systematically fetch all content
- Must be blocked quickly with 429 + reset time
Most system design sessions only consider happy-path users. For security infrastructure like a rate limiter, the threat model IS the design spec. Credential stuffing (bots trying username/password combos at scale) is why login limits exist. DDoS is why overall request limits exist. Naming the bad actor persona explicitly forces you to design the system for its actual purpose, not just its normal usage. This same thinking applies to any security-adjacent system — fraud detection, auth services, content moderation APIs.
The key estimation for a rate limiter is different from a data service. You are not estimating media storage or read/write ratios — you are estimating Redis memory for counter storage and Redis operation throughput. These two numbers determine whether a single Redis node suffices or whether you need a cluster.
A critical reframe from the session: a rate limiter does not store the full request. It stores a tiny counter per user per request type. The initial instinct of "500KB per request" was immediately wrong — a counter entry is closer to 70–100 bytes. At 10M active users, this means Redis needs under 1GB of memory. That's the order-of-magnitude difference between a correct and incorrect estimation.
At 1M requests/sec and 1M Redis ops/sec capacity per node, a single Redis node is at its operational limit. The practical design needs either: (1) Redis Cluster to distribute load across multiple nodes, or (2) local counters at each gateway node that sync to Redis periodically (trading accuracy for throughput). For the base design, centralized Redis is correct. The distributed counting deep dive covers how to handle node-level throughput constraints.
The rate limiter does not expose user-facing APIs — it intercepts existing ones. There is no "call the rate limiter" endpoint. Instead, the rate limiter adds headers to every response and a specific HTTP status code when throttled. These are the contracts that matter.
| Scenario | HTTP Status | Response Headers |
|---|---|---|
| Request allowed | Pass through (200/201/etc) | X-RateLimit-Remaining: N — how many requests are left in this window |
| Request throttled | 429 Too Many Requests | X-RateLimit-Reset: <unix_timestamp> — when the window resets and requests will be allowed again |
The X-RateLimit-Remaining header on every successful response lets clients manage their own usage. A well-behaved client can throttle itself before hitting 429. The X-RateLimit-Reset timestamp on a 429 tells the client exactly when to retry — preventing retry storms where clients hammer the endpoint immediately after being blocked.
HTTP 429 is the correct and specific status code for rate limiting (defined in RFC 6585). Using 503 (Service Unavailable) would imply the server is overloaded or down — wrong signal. Using 400 (Bad Request) implies the client sent an invalid request — also wrong. 429 signals specifically "your request was syntactically correct, but you've sent too many." Clients and monitoring systems know exactly what to do with a 429.
The rate limiter sits at the API gateway — every request passes through it before reaching the application servers. The design has three core components: the rate limiting logic itself (at the gateway), Redis for counter storage, and a rules engine that tells the rate limiter what the limits actually are.
- Client sends request to API gateway. The rate limiter intercepts before routing.
- Rate limiter loads rules from local cache. Rules are pre-loaded from Redis/config — no synchronous call to rules service on every request.
- Rate limiter increments counter in Redis. Atomic
INCR+ check against limit. If under limit, allow. Counter has a TTL that matches the window size (e.g., 60s for per-minute limits). - Request passes through to app server. Response includes
X-RateLimit-Remaining: Nheader calculated from Redis counter. - Fire-and-forget analytics event to Kafka. Request type, userId, allowed/blocked, region, timestamp. Used for operational intelligence — where to scale Redis, which regions see the most throttling.
When the counter exceeds the limit, the rate limiter returns HTTP 429 directly to the client — the app server is never reached. This is the whole point. Blocked requests must not consume application server resources. The short-circuit path is what makes the rate limiter effective as a protection mechanism rather than just a measurement tool.
The algorithm determines how "excess" is defined and how accurately burst traffic is controlled. Five algorithms exist. The session covered all four widely known ones; the fifth (sliding window counter) was explained and understood by session end. Knowing the toolbox and being able to defend one choice is what matters.
Token Bucket
Each user has a bucket of tokens. Each request consumes one token. Tokens refill at a fixed rate up to a max bucket size. If the bucket is empty, the request is rejected.
- Simple to understand and implement
- Memory efficient — just two numbers (count + refill timestamp)
- Allows bursts up to bucket size
- Two parameters to tune (bucket size + refill rate) — hard to calibrate for varying load
- Allows burst at window boundaries: empty bucket end of min 1, refilled at start of min 2 → double burst in short window
Leaking Bucket
Requests enter a FIFO queue. The queue processes N requests per second at a fixed rate regardless of arrival pattern. Overflow is rejected.
- Smooth output — processes at constant rate
- Protects backend from bursts (absorbs spikes)
- More memory than token bucket (stores queue, not just counter)
- Under burst: old requests fill the queue, new bursts rejected even after old requests expire
- Queue adds latency to all requests
Fixed Window Counter
Time divided into fixed windows (e.g., per second, per minute). Each window has its own counter. Counter resets at the window boundary.
- Very simple — one counter per window per user
- Memory efficient
- Easy to understand and debug
- Boundary burst: 3 requests at end of window + 3 at start of next = 6 in 2 seconds on a 3/min limit
- Can allow 2× the intended rate at window boundaries
Sliding Window Log
Store a timestamp for every request in a sorted set. On each new request, remove timestamps outside the window and count what remains. Reject if count ≥ limit.
- Accurate — no boundary burst problem
- Exact window tracking regardless of when requests arrive
- Memory expensive — stores every timestamp even for rejected requests
- At 1M requests/sec, billions of timestamps → Redis memory explodes
Sliding Window Counter (Hybrid)
Combines accuracy of sliding window log with memory efficiency of fixed window counter. Instead of storing timestamps, stores just two integers: previous window count and current window count. Uses a weighted formula to estimate traffic in the current sliding window.
requests_in_window = current_count + (previous_count × % of previous window still overlapping) Example: Limit: 3 requests per minute Previous window count: 2 requests Current window (30 seconds in): 1 request Overlap: 50% of previous window still within the rolling window requests_in_window = 1 + (2 × 0.5) = 2.0 → under limit → allow
- Only 2 integers per user per window — memory efficient at 1M requests/sec
- Smooths boundary bursts — no double-hit at window transitions
- Close enough to sliding window log accuracy for production use
- Twitter and Cloudflare use this approach in production
- Small remaining edge case: if all previous window requests clustered at the very end, overlap calculation slightly undercounts them
- Slightly more complex formula than fixed window
"I'd use sliding window counter — two integers per user per window in Redis, weighted formula for boundary overlap. Memory efficient at 1M req/sec, accurate enough for production, no boundary burst problem. Twitter and Cloudflare use this. The remaining edge case at window boundaries is an acceptable known tradeoff."
With multiple gateway nodes handling requests, each node could maintain its own local counter. This creates a subtle accuracy problem: if a user makes 3 requests to gateway A and 3 requests to gateway B, each server sees only 3 requests and allows all of them — even though the user made 6 total. The limit is violated without either server knowing.
All gateway nodes read and write to the same Redis cluster. Every request does an atomic INCR on the shared counter. No local caching — every request hits Redis. This is the clean, correct baseline solution. At sub-millisecond latency per Redis operation and Redis's throughput capacity, this handles the stated scale. The counter is always accurate across all nodes.
At very high scale, centralized Redis becomes the bottleneck. An alternative: each gateway node maintains a local in-memory counter and syncs to Redis every 100ms. Requests check the local counter first. If local counter is under limit, allow. Sync pushes local deltas to Redis periodically.
The tradeoff: a user could send 3 requests to node A and 3 to node B within the same 100ms sync window. Both nodes see only 3 locally, both allow. The user makes 6 requests on a limit of 5. This over-counting is bounded and small — the "error" is at most N_nodes × limit for a brief window. For most use cases (login limiting, spam prevention), this is acceptable. For financial or safety-critical APIs, it is not.
Centralized Redis — default for most systems. Accurate, simple, proven. Start here. Local counters + eventual sync — only when Redis throughput is the measured bottleneck in production. Don't over-engineer preemptively. The cost of slight over-counting must be weighed against the benefit of reduced Redis load per request.
The rate limiter needs to know what the limits are. Hardcoding limits into the rate limiter service means a code deployment every time limits change. That's operationally painful and creates unnecessary risk — deploying code to change a number. The rules should be externalized.
Limits are stored in a YAML/JSON config file: login: 5/min, post: 3/min, search: 100/min. A polling worker reads this file periodically and pushes the values into Redis. The rate limiter reads limits from Redis at startup and caches them in-process. Rule updates are eventually consistent — there's a lag between updating the config file and the new limit taking effect across all nodes.
rate_limits:
login:
limit: 5
window: 60s # 5 per minute
algorithm: sliding_window_counter
post:
limit: 3
window: 60s # 3 per minute
algorithm: sliding_window_counter
search:
limit: 100
window: 60s # 100 per minute (example)
algorithm: token_bucket # bursts OK for search
- Worker runs periodically (every few hours, configurable). Reads the config file, compares to current Redis values, pushes any changes to Redis.
- Rule changes are not immediate. If you lower the login limit from 10 to 5, it takes up to one polling cycle to propagate. For most limit changes, this lag is fine — you're not doing emergency response with a config file.
- Deployment of the config file is independent — infrastructure deployment, not code deployment. Much lower risk. Can be rolled back instantly by reverting the file.
- Future improvement: replace config file with a rules database and admin UI for real-time updates without any deployment. The worker would poll the DB instead of a file. This is how Kong, AWS API Gateway, and most enterprise rate limiters work.
Externalized configuration with a polling worker is a widely used production pattern — not just for rate limiters, but for any service that needs runtime configurability without code deploys. Feature flags, A/B test weights, circuit breaker thresholds all use similar patterns. The key principle: the service reads config from a fast local store (Redis or in-process cache), never from the config file directly. The config file is the source of truth; the fast store is the runtime view. The worker bridges them asynchronously.
The rate limiter's data lives entirely in Redis — no persistent database. Redis is the right choice because counters are transient (they expire with each window), must be read and written at sub-millisecond latency, and Redis's native atomic operations (INCR, EXPIRE) map directly to counter management.
| Key pattern | Value | TTL | Notes |
|---|---|---|---|
| rl:{userId}:{requestType}:current | INT — current window count | window size (e.g., 60s) | Main counter. INCR on each request. |
| rl:{userId}:{requestType}:previous | INT — previous window count | 2× window size | Stored when current window expires and rolls over. Used in weighted formula. |
| rl:{userId}:{requestType}:reset | UNIX timestamp — when current window resets | window size | Returned as X-RateLimit-Reset header on 429. |
| Key pattern | Value | TTL | Notes |
|---|---|---|---|
| rl:{userId}:{requestType}:tokens | INT — current token count | bucket refill period | Decremented on each allowed request. |
| rl:{userId}:{requestType}:refill_at | UNIX timestamp | same as tokens key | When tokens were last refilled. Used to calculate how many tokens to add on next request. |
This is the most important failure mode to reason through. If Redis is unavailable, the rate limiter cannot check or increment counters. Two options:
- Fail open (chosen): Allow the request to pass. Start a local in-memory counter as a temporary fallback on each gateway node. Accept that limits will be imprecise during the outage window. This maintains availability.
- Fail closed: Reject all requests with 429. This is correct behavior but takes down the system — unacceptable for most products.
When Redis recovers, the local counters accumulated during the outage are flushed to Redis. The key insight: the system is designed with headroom — if the limit is 5 requests/min, the backend can handle 10 comfortably. The brief period of uncontrolled traffic during a Redis outage will not overwhelm the backend.
Importantly, every Redis outage event should be fired as a Kafka analytics event. The analytics pipeline detects frequent Redis failures and triggers an alert. The operational response: add a Redis replica or switch to Redis Cluster. The detection mechanism (Kafka events from the outage) is already built into the system.
When limits are changed in the config file, the new values don't propagate until the polling worker runs (every few hours). During this window, old limits are enforced. This is acceptable for routine limit changes. For emergency changes (a DDoS attack requiring immediate limit reduction), a manual trigger to force a polling worker run — or a direct Redis write — should be available as an operational escape hatch.