Design a News Feed

The problem statement is: "Design a news feed system like Facebook — users can publish posts and see a ranked feed of posts from people they follow." That is deliberately ambiguous. The first job is to narrow it to a workable scope and surface the constraints that will drive every subsequent decision.

The most important scope call in this step is the follow model. One-way follow (Twitter-style: User A can follow User B without B following back) vs. two-way friendship (Facebook-style: both must accept). This matters enormously for the fan-out logic — one-way follow allows celebrities with millions of followers, which changes the architecture fundamentally. Two-way friendship has natural limits on fan-out because the social graph is denser but smaller per user.

The second critical call is whether feed is ranked or chronological. Chronological is trivial to implement — just sort by timestamp. Ranked requires an engagement model, a ranking service, and a more complex read path. Agreeing on ranked up front sets the expectation that step 5 will go deep on ranking logic.

For a news feed, personas matter more than they did for TinyURL because the user type directly drives the architectural split. Two fundamental user behaviors produce two completely different system paths.

Estimation starts from DAU and post frequency and chains through to the numbers that constrain architecture. The key insight for news feed: this is a massively read-heavy system. The ratio of reads to writes is not 10:1 like TinyURL — it's more like 100:1 because every active user opens their feed multiple times per day, but only a fraction of users post anything.

Media storage is the number that surprises people. At 10MB per post (assuming video), the numbers are enormous. The session corrected this to 1MB average after accounting for the fact that most posts are text or images, not video. Even at 1MB average, media storage over 10 years is measured in petabytes — which is why object storage (S3, OSS) is mandatory, not optional. You cannot put 18PB in a relational database.

Two core endpoints cover the full functional scope: one for publishing, one for reading. The publish endpoint is straightforward. The read endpoint has one important detail — pagination.

Infinite scroll requires the client to tell the server where it left off. Cursor-based pagination uses the ID of the last post seen as the starting point for the next call. This is far superior to offset-based pagination for infinite scroll because:

• New posts don't shift results: If you're on "page 3" using offset and 5 new posts arrive, your page 3 is now different. With a cursor, you always get the next 20 posts after the one you last saw — no drift.
• Works with Cassandra's data model: Cassandra supports efficient range queries by partition key + clustering key. A cursor maps directly to a Cassandra range query. Offset-based pagination requires a full scan and skip — inefficient at scale.
• First call behavior: On first load, cursor is null — return the most recent posts. After each scroll, cursor = the last postId returned. If the app is closed and reopened, cursor resets to null and the feed starts fresh from the top.

The news feed system has two completely different paths: the write path (publishing a post) and the read path (loading a feed). They share infrastructure but are optimized for opposite workloads — write for reliability and async throughput, read for speed and pre-computation. Drawing and explaining both paths separately is essential.

• Post Service — receives the post, writes to post cache and post DB synchronously, then drops ONE event into Kafka. Returns success to the user immediately. Does not wait for fan-out to complete. This is what keeps post latency low regardless of follower count.
• Kafka (post.published topic) — decouples post service from fan-out service. The event carries userId and postId. Kafka provides durability, ordering within a partition, and fan-out service can be scaled by adding more consumers without changing post service.
• Fan-out Service (Kafka consumer) — consumes the event, queries Graph DB to get the list of followers, applies the active/inactive filter (skip inactive users to avoid waste), checks if user is celebrity (skip fan-out if >1M followers), then writes one record per eligible follower into the Feed Table.
• Graph DB — stores the social graph: who follows whom. Fan-out service queries this to get the follower list for a given userId. Separate from user DB because graph traversal queries are different in structure from relational queries.
• Feed Table — the most important table in the system. Schema: (userId, postId, createdAt). Indexed by userId. Fan-out service writes one row per follower per post. The get service reads from here on the read path.

Phase 1 — Instant from cache: The get service immediately queries the feed cache for this user's pre-computed feed. This returns within milliseconds. The user sees their first posts almost instantly. This handles the initial scroll without any database calls.

Phase 2 — Async enrichment: Simultaneously, an async job fetches from the feed DB (Cassandra) to get the full set of recent posts for regular friends. In parallel, the get service queries Graph DB to identify which followees are celebrities (high-volume users, flagged in user table). For each celebrity in the user's follow list, it calls the Hybrid Get Service, which fetches recent posts directly from the post cache. These celebrity posts are recent enough to almost always be cache-hot. Both result sets — regular friend posts from feed DB + celebrity posts from post cache — are sent to the Feed Ranking Service.

Feed Ranking Service: Receives the merged pool of posts from both sources, applies the engagement scoring model (see deep dive 5B), sorts by score, and returns the top 20. This ranked result is also written back to the feed cache so the next scroll is even faster.

The fan-out problem is the central challenge of news feed design. When a user posts, every one of their followers needs to see it. At small scale this is trivial. At Facebook scale — where a single celebrity can have 100 million followers — doing this synchronously would take hours and generate millions of writes in a second.

The key insight is that fan-out is not urgent. Users can wait 3–5 seconds for a post to appear in their feed — we agreed to this in requirements. This acceptable lag is what enables the entire async architecture. If the requirement were "post visible within 100ms to all followers," the design would be fundamentally different and much harder.

The session debated this threshold — it started at 500 followers, moved to 1M. The right number depends on your fan-out service capacity and your SLA. At 58 write QPS for posts, with an average of say 500 followers each, fan-out service handles 29,000 feed writes per second. That's manageable. But one celebrity with 100M followers posting would generate 100M writes — taking hours even with parallelized workers. The threshold is the point at which fan-out becomes a capacity problem, not a latency problem. 1M is a reasonable starting point that can be tuned.

Before writing to the feed table, the fan-out service filters out inactive followers. A user who hasn't opened the app in 30 days doesn't need their feed pre-populated — when they return, their first request will trigger a fresh pull from the feed DB. Skipping inactive users reduces fan-out volume significantly (inactive users are often a large fraction of total followers) and avoids polluting the feed table with records that will never be read. The notification/analytics service maintains the active/inactive flag in the user table, updated from analytics events.

• Kafka consumer group: Fan-out service runs as multiple consumer instances in a Kafka consumer group. Each post event is processed by exactly one consumer. Adding more consumers (horizontal scaling) increases fan-out throughput without changing post service.
• Batch writes to Feed Table: Instead of writing one row at a time to Cassandra for each follower, batch multiple writes into a single Cassandra batch. This significantly reduces write overhead at high follower counts.
• Async fan-out within fan-out: For users with large but sub-threshold follower counts (say 500K followers), the fan-out service can spawn parallel worker threads to write to feed table concurrently rather than serially, completing within seconds.

Once the read path has assembled posts from two sources — regular friend posts from the feed table and celebrity posts from the post cache — something must merge them into a single ranked list of 20. This is the feed ranking service's job.

The key question is: what does "ranked by relevance" actually mean in implementation? Purely chronological is easy — sort by timestamp. But Facebook's feed and Twitter's algorithm are not chronological. They surface posts the user is more likely to engage with, which means the system needs a signal about engagement between users.

The approach discussed: maintain a per-user-pair engagement score. Every time User A likes, comments on, or shares User B's post, increment a counter stored in the user relationship record (or a separate engagement table). This score reflects the strength of User A's interest in User B's content. Higher score = higher ranking for B's posts in A's feed.

score(post) = 
  engagement_score(viewer, author)    // how much viewer likes this author
  × recency_weight(post.createdAt)    // newer posts score higher
  × post_engagement(likes + comments) // popular posts score higher

// engagement_score: incremented on like, comment, share
// decays over time — interaction 6 months ago matters less than last week
// stored in graph DB on the user-follows-user edge

The feed ranking service receives two lists:

• List A: Posts from regular friends — fetched from feed table (Cassandra), already sorted by createdAt. Maybe 50–200 posts from the last 24 hours.
• List B: Posts from celebrities the user follows — fetched from post cache via hybrid get service. 5–10 celebrity posts from recent hours.

The ranking service combines both lists, scores each post using the engagement model, sorts descending by score, and returns the top 20. The remaining posts (position 21+) are discarded — the next scroll call will re-fetch and re-rank with updated scores and newer posts.

Infinite scroll works best when the next batch is ready before the user reaches the bottom. When the user reaches the 10th post of their current 20, the app triggers a background prefetch call to fetch posts 21–40. These arrive before the user finishes reading posts 11–20, making the scroll feel seamless. The get service writes the prefetched batch to the feed cache tagged by cursor position. This is a client-side trigger — the app knows when to fire it, the server just processes it like any other feed request.

This system uses five different databases — each chosen for a specific access pattern. The instinct to consolidate to fewer databases is understandable (operational simplicity), but each choice is justified by the workload. Using the wrong database type for a workload doesn't just cost performance — it can make certain queries functionally impossible at scale.

The critical distinction to understand first: OLTP vs OLAP. These are two completely different types of database read workloads, often confused because both are described as "read-heavy."

Five tables, five different database types. Each table's schema is driven by the access patterns defined in the design — not by habit or convenience.

A nightly batch job aggregates this raw event data into summary tables: active user counts by region, post engagement rates, feed load latencies by device type. These summaries feed the analytics dashboard and update the is_active flag in the User table. The engagement score on FOLLOWS edges in Graph DB is also updated from this pipeline — likes and comments in the analytics events translate into edge weight increments.