Distributed Systems - Overview

Updated: 2021-11-19

Working on any non-trivial projects in any modern software companies would require some knowledge about distributed systems. With data at today's scale, everything is distributed. This is an attempt to create a mind map to help you navigate.

To some extent, "distributed systems" is about building reliable software on top of unreliable hardware.

The Stack

• Servers: to run the applications and backends
  • bare metal,
  • VMs
  • containers,
  • serverless functions
  • orchestration, e.g. Kubernetes
• Databases to store data and make them readily available for appliations, and indexes to speed up searches and filters.
• Caches to speed up reads by remembering results of expensive operations
• Data Warehouse to store historical data for analytics
• Storage to store files and objects (can also be used to serve static websites)
• Message queues to communicate between processes and enable async operations.
• Logging: AWS Kinesis, Fluentd, GCP Cloud Logging
• Monitoring to monitor system health and key business metrics, and send out alerts: GCP Cloud Monitoring, Datadog
• Service Discovery, Configs and Secrets: Consul/Vault
• Orchestration / Provision: Kubernetes, Terraform
• Package format: a common way to package all the applications, e.g. Docker, Java archives, etc.
• Code management: e.g. git or hg
• CI/CD: continuous integration and deployment
• Testing Infra:
  • Test data: logging -> capture -> filter / sample / anonymization -> diff test / query test / ad-hoc debugging

More Stacks

• Stack Share: https://stackshare.io/
• Cloud Native Computing Foundation: https://www.cncf.io/