11: DESIGN A NEWS FEED SYSTEM - swchen1234/systemDesign GitHub Wiki

Step 1 - Understand the problem and establish design scope

• mobile app/web app?
• what are the important features?
• what's the sorting logic?
• how many friends?
• traffic volume?
• contain images, video or just text?

Step 2 - Propose high-level design and get buy-in

Newsfeed APIs

• Feed publishing API
• Newsfeed retrieval API

Feed publishing

• Post service: persist post in the database and cache.
• Fanout service: push new content to friends’ news feed. Newsfeed data is stored in the cache for fast retrieval.
• Notification service: inform friends that new content is available and send out push notifications.

Newsfeed building

Step 3 - Design deep dive

Feed publishing Deep Dive

Fanout service

Fanout is the process of delivering a post to all friends. Two Modes:

• fanout on write(aka. push model)

• 一旦publish，马上被写入朋友的cache
• pros:
  • news feed generated in real-time
  • Fetching is fast
• cons:
  • hotkey problem: 如果用户朋友很多，对所有朋友生成newsfeed耗时

• fanout on read(aka. pull model)

• The news feed is generated during read time
• pros:
• 对于不活跃用户，fanout on read 能节约大量资源
• cons:
• fetching is slow

方案 => hybrid approach

• 对于大多数用户： push model
• 对于celebrities: pull model, 防止system overload

• 减少 hotkey problem： consistent hashing

Newsfeed retrieval deep dive

Cache architecture

Step 4 - Wrap up

If there are a few minutes left, you can talk about scalability issues.

Scaling the database:

• Vertical scaling vs Horizontal scaling
• SQL vs NoSQL
• Master-slave replication
• Read replicas
• Consistency models
• Database sharding

Other talking points: • Keep web tier stateless • Cache data as much as you can • Support multiple data centers • Lose couple components with message queues • Monitor key metrics(e.g. peak QPS, latency)