all: Document full integration testing

[jiping-s]

Document our full integration testing once we have a prototype setup

The scope of full integration testing should cover:

• Input: mixed normal log records (200b) and huge log records (~50K), with a sizable part of them to be dropped
• Processing: hit every transforms except deprecated or not-recommended-for-production
• Output: network output
• Upstream: fake server from https://github.com/relex/fluentlib with random failures to trigger recovery
• Kill agent in the middle and restart to test shutdown & recovery: note this only works for catchable signals such as SIGTERM

Automatic verification needs to cover:

• Final output logs from upstream in JSON
  • Order of output logs should be the same as input, per each of pipeline (key-set)
    • Except when recovery happens, which should be resent and possibly duplicated in the original order. For example logs 1 2 3 4 5 may be sent as 1 2 3 | 2 3 | 4 5
• Metric values except buffer metrics regarding on-disk vs in-memory chunks