Testing

Last Updated: 2023-09-04

Different types of tests:

Diff Testing

Diff testing: compare the outputs of two comparable systems under test (SUTs); often reviewed manually by engineers.

This is useful to determine how the behavior of a system has changed based on changes in the source code.

Hermetic Servers: "Server in a box", quote from Google's Testing blog:

start up the entire server on a single machine that has no network connection AND the server works as expected.

They will increase your test’s runtime since you have to start the entire SUT each time you run the end-to-end test.

Fuzzing is a testing technique that feeds auto-generated inputs to a target code in an attempt to crash it.

Automated tests do not provide consistent Pass / Fail results.

Fake: a lightweight implementation. It actually works, but not with full implementation and not for production.
Mock: used to make assertions; "given input X, return Y"

logging -> capture -> filter / sample / anonymization -> diff test / query test / ad-hoc debugging

Log incoming and outgoing RPC requests.

Use production-derived data for tests, ensuring the test data accurately represents the full spectrum of data flowing through the production system

A good binary logging platform should:

log request / response (e.g. from the envelope)
collect the logs
take care of privacy and security concerns: encryption, ACL, retention, auditing, etc
make the logs available for tests (diff tests, load tests, etc)