logo

Testing

Different types of tests:

Diff Testing

Diff testing: compare the outputs of two comparable systems under test (SUTs); often reviewed manually by engineers.

This is useful to determine how the behavior of a system has changed based on changes in the source code.

Hermetic Tests

Hermetic Servers: "Server in a box", quote from Google's Testing blog:

start up the entire server on a single machine that has no network connection AND the server works as expected.

They will increase your test’s runtime since you have to start the entire SUT each time you run the end-to-end test.

Fuzzing

Fuzzing is a testing technique that feeds auto-generated inputs to a target code in an attempt to crash it.

Test Flakiness

Automated tests do not provide consistent Pass / Fail results.

A good read: https://testing.googleblog.com/2020/12/test-flakiness-one-of-main-challenges.html

Fake vs Mock

  • Fake: a lightweight implementation. It actually works, but not with full implementation and not for production.
  • Mock: used to make assertions; "given input X, return Y"

Test data

logging -> capture -> filter / sample / anonymization -> diff test / query test / ad-hoc debugging

Binary Logging

Log incoming and outgoing RPC requests.

Use production-derived data for tests, ensuring the test data accurately represents the full spectrum of data flowing through the production system

A good binary logging platform should:

  • log request / response (e.g. from the envelope)
  • collect the logs
  • take care of privacy and security concerns: encryption, ACL, retention, auditing, etc
  • make the logs available for tests (diff tests, load tests, etc)

Proposal for gRPC: https://github.com/grpc/proposal/blob/master/A16-binary-logging.md