logo
CHEATSHEETSTOOLSABOUT中文

Data Formats and Serialization

Data can be saved on disk or transmitted across network boundaries. The physical serialization format on disk or wire differs fundamentally from runtime in-memory objects:

  • Serialization: Converting structured in-memory data structures into a flat stream of bytes for disk persistence or network transmission.
  • Deserialization: Reconstructing in-memory data structures from a raw byte stream.

Text Formats

Examples: JSON, XML, CSV, YAML

  • Pros: Universally human-readable, debuggable with standard Unix tools (cat, jq), self-describing schemas.
  • Cons: High storage space overhead (redundant field keys in every record), expensive CPU parsing latency, ambiguity around numbers and binary payloads.

Binary Wire Formats

Examples: Protocol Buffers (Protobuf), Apache Thrift, Apache Avro

  • Pros: Compact byte payloads, rapid serialization/deserialization, strict schema evolution and backwards compatibility guarantees.
  • Cons: Not human-readable without schema definitions and decoding tools.

For more details on RPC and wire formats, see API Design and gRPC.

Columnar Storage Formats

Traditional databases store data row-by-row (OLTP: fast writes and lookups by primary key). Analytical engines store data column-by-column (OLAP: reading only the requested columns across millions of rows).

Apache Parquet

  • An open-source columnar storage format created by Cloudera and Twitter.
  • Highly compressed using dictionary encoding, run-length encoding (RLE), and Snappy/ZSTD.
  • Optimized for batch processing with Spark, DuckDB, Presto/Trino, and BigQuery.

Apache Arrow

  • An open-source, language-agnostic in-memory columnar data format specification.
  • Parquet vs. Arrow:
    • Parquet: Designed for efficient on-disk persistent storage.
    • Arrow: Designed for zero-copy, CPU-cache-friendly in-memory analytics and cross-process data transport.

Google Capacitor & Ressi

  • Capacitor: Google BigQuery's proprietary columnar file format, succeeding ColumnIO.
  • Ressi: Google Spanner's hybrid columnar format, optimized for combined transactional (OLTP) and analytical (OLAP) queries.

SequenceFile & ORC

  • SequenceFile: Flat file of binary key-value pairs used in Hadoop MapReduce temporary stages.
  • Optimized Row Columnar (ORC): Highly efficient columnar format popular in the Apache Hive and Hadoop ecosystems.

SSTable (Sorted String Table)

An SSTable is a simple yet foundational storage abstraction used by BigTable, Cassandra, RocksDB, LevelDB, and ScyllaDB:

  • Append-Only & Immutable: SSTable files on disk are never modified in-place. Updates and deletes are written to a MemTable (in-memory write-ahead tree) and later flushed as new SSTables. Background compactions merge and purge obsolete tombstones.
  • Sorted Keys: Keys are strictly ordered. Lookups use a small in-memory index or sparse index pointing to block offsets, allowing O(log N) lookups without loading entire files into RAM.
  • Bloom Filters: Paired with in-memory Bloom filters to avoid costly disk reads for non-existent keys.
Loading...