System Desing Basics - manishKapasiya/Object-Oriented-Design-and-design-Principles GitHub Wiki
Buying larger machine -> Vertical Scaling Buying more machines -> Horizontal Scaling
Horizaontal Scaling :
- Load Balancing
- Resilient -> + Point
- Network calls between machines (RPC Remote Procedure Calls)
- Data Inconsistency.
- Scales well as Users increase. -> + Point
Vertical Scaling :
- No Load Balancing
- Single Point of Failure
- Inter process communication -> + point
- Data Consistant -> + point
- Hardware Limit.
- Vertical scaling
- Preprocessing using cron jobs
- Backup servers
- Horizontal scaling
- Microservices
- Distributed Systems
- Load Balancing
- Decoupling
- Logging and metrics calculation
- Extensibility
- Low level system design
Load Balancing:
- Consistent Hashing :
Message Queue:
Monolith vs Microservices: Sharding :
- Horizontal Partitioning
- Vertical Partitioning
- Joins is a problem
- Memcached
- Hierarchical Sharding
- Index
- Master Slave Architecture
- Open Connect box in Netflix.
File vs Blob
- CDN
- HTTP vs XMPP protocol
- Web Socket Connection TCP
- Thrift
- Cache : LRU , Window Sliding policy
- Disadvantages of Cache : 1) Thrashing 2) Increased calls 3) Data Inconsistency
- In-place cache vs Global Cache like Redis
- Write-through vs Write-back cache
- Hardware requirements of Youtube
------STORAGE Let's start with storage requirements: About 1 billion active users. I assume 1/1000 produces a video a day. Which means 1 million new videos a day.
What's the size of each video? Assume the average length of a video to be 10 minutes. Assume a 10 minute video to be of size 1 GB. Or... A video is a bunch of images. 10 minutes is 600 seconds. Each second has 24 frames. So a video has 25*600 = 150,000 frames. Each frame is of size 1 MB. Which means (1.5 * 10^5) * (10^6) bytes = 150 GB. This estimate is very inaccurate, and hence we must either revise our estimate or hope the interviewer corrects us. Normal video of 10 minutes is about 700 MB.
As each video is of about 1GB, we assume the storage requirement per day is 1GB * 1 million = 1 PB.
This is the bare minimum storage requirement to store the original videos. If we want to have redundancy for fault tolerance and performance, we have to store copies. I'll choose 3 copies. That's 3 petabytes of raw data storage. What about video formats and encoding? Let's assume a single type of encoding, mp4, and the formats will take a 720p video and store it in 480, 360, 240 and 144p respectively. That means approximately half the video size per codec.
If X is the original storage requirement = 1 PB, We have X + X/2 + X/4 + X/8 == 2X. With redundancy, that's 2X * 3 = 6X.
That's 6 PB(processed) + 3PB (raw) == 10 PB of data. About 100 hard drives. The cost of this system is about 1 million per day.
For a 3 year plan, we can expect a 1 billion dollar storage price.
Now let's look at the real numbers: Video upload speed = 3 * 10^4 minutes per minute. That's 3 * 10^4 *1440 video footage per day = 4.5 * 10^7 minutes. Video encoding can reduce a 1 hour film to 1 GB. So 1 million GB is the requirement. That's 1 PB.
So the original cost is similar to what the real numbers say.
If we are off by an order of magnitude, it's good. However, being off by 3 or more orders of magnitude is too much. We can then highlight: Where our assumption was wrong, or Which factor we didn't take into account.
References: Designing Data Intensive Applications - https://amzn.to/2yQIrxH http://highscalability.com/youtube-ar... https://www.youtube.com/watch?v=w5WVu... Numbers everyone should know: https://www.youtube.com/watch?v=modXC... http://www.youtube.com/watch?v=G-lGCC... https://en.wikipedia.org/wiki/Back-of... Capacity planning with AWS: https://youtu.be/-3qetLv2Yp0
Single Point of Failure :
- More Nodes
- Master Slave Architecture. Replication
- Multiple Regions
SQL vs NOSQL :
- Cassandra : Load Balancing, Replication , Distributed Consensus Quorum
Master Slave Architecture : Split Brain Problem : Distributed consensus : 2PC, 3PC, MVCC (Postgress), SAGA,