3 types of storage: Block Storage, File Storage, Object Storage (blob=binary large object).
- Block Storage:
- Evenly sized chunks.
- No file systems.
- File Storage:
- A hierarchy of files in folders.
- With file systems.
- Blob / Object Storage:
- Immutable and unstructured objects, e.g. images, audio or other multimedia objects;
- Sometimes binary executable code is stored as a blob.
- No hierachy.
- Access data and metadata using an HTTP API.
Example of storage systems
- Online Transaction Processing Databases (OLTP)
- Facebook Graph, mission critical, strong consistency, core services
- Semi-online Light Transaction Processing Databases (SLTP)
- Facebook Messages and Facebook Time Series
- Immutable DataStore
- Photos, videos, etc
- Analytics DataStore
- Data Warehouse, Logs storage
Facebook example. This is adapted from this slide
|Facebook Graph||MySQL/TAO||Random read IOPS||few ms||quickly consistent across data centers||no data loss|
|Messages and Time Series||HBase and HDFS||Write IOPS/storage capacity||< 200 ms||consistent within a data center||no data loss|
|Photos / Videos||Haystack||storage capacity||< 250 ms||immutable||no data loss|
|Data Warehouse||Hive / Presto / HDFS||storage capacity||< 1min||not consistent across data centers||no silent data loss|
The core of a distributed storage system
- sharding strategy
- metadata storage
Distributed File Systems
Distributed file systems: GFS, Colossus, Alluxio, CephFS, HDFS
- Cluster level, fault tolerant, distributed file systems:
- append only
- not for structured data(use database instead)
- not optimized for small files
- cluster level, not data center level, data destroyed after the cluster turns down
- HDFS is the open source version of GFS(Google File System)
- Colossus is the successor of GFS
- Spanner uses Colossus to store its tablets
Software Defined Storage
- Ceph: by Red Hat; object store at its core, but support all 3 types (object, block, file). For shorter-term stoarge and more frequent user access.
- CephFS: a POSIX-compliant network file system
- Gluster: scalable file storage with object capabilities; also by Red Hat. Should not be used for something transactional, like a database or something that depends on really strict locking.
- Alluxio: a virtual distributed storage system
- MinIO: S3 compatible object storage, k8s native.
- Rook: as a storage orchestrator (can be used with Ceph where Ceph is a storage provider, Rook: Ceph Operator + Discovery)
- VMware vSAN: creates shared storage for VMs.
- HDFS: part of Hadoop.
Also known as virtual provisioning or thin storage, is a method of on-demand storage allocation based on user requirements in storage area networks (SAN), centralized storage disks, and storage virtualization systems.
Thin provisioning allows space to be easily allocated to servers, on a just-enough and just-in-time basis.