logo

Google Internal vs Open Source

Last Updated: 2022-06-26

Some of Google's open source projects are born open-source, but others are siblings to the original internal projects, their names, even technical details may be different; also some famous projects are not open sourced by Google, but as an implementation of Google's published papers.

Google Open-sourced

Blaze and Bazel

  • Blaze: Google's internal build system.
  • Bazel: the open source version.

Borg and Kubernetes

  • Borg: Google's internal cluster manager.
    • package format: MPM.
  • Kubernetes: modeled after Borg. Written in Go.
    • package format: containers.

Public info about Borg and MPM:

Knative vs Cloud Run

  • Knative is an open-source serverless platform on top of Kubernetes, now part of CNCF.
  • Cloud Run is a managed Knative service.

Flume vs Apache Beam

  • Flume: Google's internal data pipelines; originally on top of MapReduce, now moved to a faster underlying engine.
  • Apache Beam: the open-source version of Flume.

Google's Flume is totally different from Apache Flume.

Stubby vs gRPC

  • Stubby: Google's internal RPC framework.
  • gRPC: the next generation of Stubby, and open sourced.

Both depend on protobuf.

https://cloud.google.com/blog/products/gcp/grpc-a-true-internet-scale-rpc-framework-is-now-1-and-ready-for-production-deployments

Google Influenced

GFS and HDFS

  • GFS: Google's (previous generation) distributed file system, and Google published a paper about it.
  • HDFS: part of Hadoop, an open source implementation of that paper.

More

https://github.com/jhuangtw/xg2xg