1. SCALE FROM ZERO TO MILLIONS OF USERS - swchen1234/systemDesign GitHub Wiki

Basic Setup:

  1. Users access websites through domain names, such as api.mysite.com. Usually, the Domain Name System (DNS) is a paid service provided by 3rd parties and not hosted by our servers.
  2. Internet Protocol (IP) address is returned to the browser or mobile app. In the example, IP address 15.125.23.214 is returned.
  3. Once the IP address is obtained, Hypertext Transfer Protocol (HTTP) [1] requests are sent directly to your web server.
  4. The web server returns HTML pages or JSON response for rendering.

Web server 的traffic 主要来自 web application 和 mobile application.

改进 1: 分离 web/mobile traffic (web tier) and database (data tier) servers allows them to be scaled independently.

DataBase的选择：

  • Relational DB
    • e.g. MySQL
    • 更常见，使用广泛
  • Non-Relational DB
    • Popular ones are CouchDB, Neo4j, Cassandra, HBase, Amazon DynamoDB, etc
    • 适用情况
      • Your application requires super-low latency.
      • Your data are unstructured, or you do not have any relational data.
      • You only need to serialize and deserialize data (JSON, XML, YAML, etc.).
      • You need to store a massive amount of data.

改进 2: 处理更多request => Load Balancer

处理更多request:

  • Vertical scaling

    • adding more power (CPU, RAM, etc.) to your servers.
    • 缺点： (1) It is impossible to add unlimited CPU and memory to a single server. (2) Vertical scaling does not have failover and redundancy. If one server goes down, the website/app goes down with it completely.

  • Horizontal scaling

    • adding more servers into your pool of resources.
    • 更广泛，使用load balancer来实现

A load balancer 将request平均分配给各个web server that are defined in a load-balanced set.

  • User 无法直接连到web servers.
  • Load Balancer与web servers间用private IP交流，增加私密性。

改进 3: Database Replication => Master/Slave Database

  • Master Database支持write
  • Slave Database支持read

优势：

  • Better performance: In the master-slave model, all writes and updates happen in master nodes; whereas, read operations are distributed across slave nodes. This model improves performance because it allows more queries to be processed in parallel.
  • Reliability: If one of your database servers is destroyed by a natural disaster, such as a typhoon or an earthquake, data is still preserved. You do not need to worry about data loss because data is replicated across multiple locations.
  • High availability: By replicating data across different locations, your website remains in operation even if a database is offline as you can access data stored in another database server.

failure情况

  • 如果slave database下线了，原来分给它的任务占时分给master；若有多个offline, 任务也可能分给其它slave
  • 如果master database下线了，讲一个slave晋升为master. In production systems, promoting a new master is more complicated as the data in a slave database might not be up to date. The missing data needs to be updated by running data recovery scripts.

Let us take a look at the design: • A user gets the IP address of the load balancer from DNS. • A user connects the load balancer with this IP address. • The HTTP request is routed to either Server 1 or Server 2. • A web server reads user data from a slave database. • A web server routes any data-modifying operations to the master database. This includes write, update, and delete operations.

改进4: improve the load/response time

Can be done by adding a cache layer and shifting static content (JavaScript/CSS/image/video files) to the content delivery network (CDN)

Cache

A cache is a temporary storage area that stores the result of expensive responses or frequently accessed data in memory so that subsequent requests are served more quickly.

  • 处于server 和 database中

Cache tier

The cache tier is a temporary data store layer, much faster than the database. The benefits of having a separate cache tier include better system performance, ability to reduce database workloads, and the ability to scale the cache tier independently.

  • cache的使用注意事项
  • 什么时候使用：数据经常被读，而较少被修改。因为cache restart后就会清空，所以不适合放重要数据
  • 对数据能在cache中存储时间设置expiration policy
  • 确保cache和database中的数consistent
  • 避免single point of failure(SPOF), 对于每个设置多个cache
  • Eviction Policy: 多用LRU

Content Delivery Network(CDN)

A CDN is a network of geographically dispersed servers used to deliver static content. CDN servers cache static content like images, videos, CSS, JavaScript files, etc.

  • 优势是地理位置更近。
  • 只存储static data(e.g. image, video, etc..)
  • 处于user 和 server中

Stateless Web Tier

将state data(e.g. user session data)从web tier 抽出放入shared storage(e.g. relationship database, Memcached/Redis, NoSQL)中，这样cluster中的每个web server都可以从database中抓去state data. NoSQL 因为其容易scale所以最常用。

  • Autoscaling means adding or removing web servers automatically based on the traffic load. After the state data is removed out of web servers, auto-scaling of the web tier is easily achieved by adding or removing servers based on traffic load.

Data centers

In normal operation, users are geoDNS-routed, also known as geo-routed, to the closest data center, with a split traffic of x% in US-East and (100 – x)% in US-West. geoDNS is a DNS service that allows domain names to be resolved to IP addresses based on the location of a user.

Message Queue

A message queue is a durable component, stored in memory, that supports asynchronous communication. It serves as a buffer and distributes asynchronous requests. 主要解决了asynchronous requests的问题

Logging, metrics, automation

对于小型应用非必须

  • Logging: Monitoring error logs
  • Metrics:
  • Host level metrics: CPU, Memory, disk I/O, etc.
  • Aggregated level metrics: for example, the performance of the entire database tier, cache tier, etc.
  • Key business metrics: daily active users, retention, revenue, etc.

  • Automation: When a system gets big and complex, we need to build or leverage automation tools to improve productivity. Continuous integration is a good practice, in which each code check-in is verified through automation, allowing teams to detect problems early. Besides, automating your build, test, deploy process, etc. could improve developer productivity significantly.

  • Updated Design Due to the space constraint, only one data center is shown in the figure.

Database Scaling

Vertical Scaling

adding more power (CPU, RAM, DISK, etc.) to an existing machine. 缺点：spof, 贵

Horizontal Scaling(aka Sharding)

Sharding separates large databases into smaller, more easily managed parts called shards. Each shard shares the same schema, though the actual data on each shard is unique to the shard.

  • Anytime you access data, a hash function is used to find the corresponding shard.
  • The most important factor to consider when implementing a sharding strategy is the choice of the sharding key, 最重要的考量标准是evenly distributed data.
  • 注意事项：
  • Resharding Data: 当1）某个shard数据增长过快 2)某个shard exhuastion
  • Celebrity Problem: 解决方法为每个celebrity设置个shard
  • Join and de-normalization: Once a database has been sharded across multiple servers, it is hard to perform join operations across database shards. 解决方法： de- normalize the database so that queries can be performed in a single table.