Signed-off-by: Omar Khasawneh <[email protected]>

Signed-off-by: Omar Khasawneh <[email protected]>

Signed-off-by: Omar Khasawneh <[email protected]>

Signed-off-by: Omar Khasawneh <[email protected]>

Signed-off-by: Omar Khasawneh <[email protected]>

Signed-off-by: Omar Khasawneh <[email protected]>

These patches introduce some new interfaces and rename some existing JSON transformers to make new transformation types more distinguishable.
I've also moved around some fields between classes as thinking about how to handle auth headers has started to create some new types of patterns.

Signed-off-by: Greg Schohn <[email protected]>

…igning all the way back to the command line.

The command line now takes a service and region as opposed to just a zero arity boolean flag as both are required for SigV4 calculations.  These values should refer to the region and service that the replayer is making calls to (e.g., the 'target' server).
Notice that an AuthTransformerFactory is passed around that returns an AuthTransformer for a specific http message.
There could be different strategies dependent upon the original message.  For example, unauthorized users might simply be mapped to no transformation, meaning that we don't need to go through the added effort of buffering and signing the messages.
The old style http header transformation via jolt is also gone and has been replaced with an (HeaderOnly)AuthTransformer class that manipulates headers.  I'm not happy with the care that I had to put into the  NettyDecodedHttpRequestHandler class (now NettyDecodedHttpRequestPreliminaryConvertHandler) to do the auth transformation in two spots while also passing more complicated auth transformations into the pipeline construction helper methods.
I've also converted the SigV4Signer to be an AuthTransformer and have tied it into the netty handler to host it, though none of that has been tested in this commit.

Unrelated changes include converting the packets of an HttpMessage(AndTimestamp) from an ArrayList to a named extension (ie. typedef).  I've never been comfortable with the vagueness of the ArrayList.  I had also been working on edits to take the response into the AuthTransformer so that 403s could be ignored - but decided that would be a bad idea.  1) Not every request will have a response and 2) it will allow future requests that should have been blocked to never be tested and the time to test them will be within the target server.

Signed-off-by: Greg Schohn <[email protected]>

…of the replayer.

I've got a unit test to verify that the contents of the signature match a mock key and mock date.  To put that together, I've extracted another helper function from a similar test and placed it into TestUtils.
There was one other bugfix to remove a spurious "toString()" that followed a headers.put call for the content-length.  The string wasn't used and put() will return null when the header wasn't previously present, creating an NPE situation.

Signed-off-by: Greg Schohn <[email protected]>

Signed-off-by: Greg Schohn <[email protected]>

…ayload streams.

A couple other minor improvements and cleanups were thrown in here too, most notably, tracking the shadow request packets that are sent, which is very helpful to debug signing problems.

Signed-off-by: Greg Schohn <[email protected]>

* main: (27 commits)
  Switch to size-based rollover policy
  MIGRATIONS-1200 - Update python test cluster_migration_core path
  MIGRATIONS-1200 - Fix github workflow actions
  MIGRATIONS-1200 - Fix python requirement paths
  MIGRATIONS-1200 - Fix flake8 line too long error
  MIGRATIONS-1200 update python requirements files with new path
  MIGRATIONS-1200 - Move cluster_traffic_capture to experimental
  MIGRATIONS-1200 - Move cluster_migration_core to experimental
  MIGRATIONS-1200 - Move knowledge_base to experimental
  MIGRATIONS-1200 - Create experimental subdirectory and place upgrades within experimental
  Revert "MIGRATIONS-1200 - Remove historical capture CDK stack"
  Update to hourly, fix filename
  Add logging config so output tuples go to shared filesystem
  Add option to use AWS secret in auth header for Replayer requests (opensearch-project#265)
  [Fetch Migration] Added total document count to report (opensearch-project#261)
  MIGRATIONS-1200 - Rename 'upgrades' directory to 'experimental' and replace references
  MIGRATIONS-1200 - Remove deprecated docker files
  MIGRATIONS-1200 - Remove historical capture CDK stack
  MIGRATIONS-1200 - Remove unused TrafficReplayer directory
  Soften assertions for aspects of the test that are independent of the strict contract that the ExpiringSubstitutableItemPool defines.
  ...

Signed-off-by: Greg Schohn <[email protected]>

…e isn't already one present.

This pushes the success rate for signatures to 100% in the first 120 messages from the opensearch benchmark workload when rerun against a cluster that uses sigv4.

Signed-off-by: Greg Schohn <[email protected]>

* capture-and-replay-v0.1.0:
  Update runTestBenchmarks with params and default values
  Update README.md
  MIGRATIONS-1200 - Remove plugins
  MIGRATIONS-1200 - Remove FetchMigration
  MIGRATIONS-1200 - Remove datastash
  MIGRATIONS-1200 - Capture and Replay v.0.1.0 branch

Signed-off-by: Greg Schohn <[email protected]>

Signed-off-by: Greg Schohn <[email protected]>

* sigv4excision_andsigner:
  PR feedback around command line arguments

Signed-off-by: Greg Schohn <[email protected]>

* capture-and-replay-v0.1.0:
  Add script to transform tuples to human readable format

Signed-off-by: Greg Schohn <[email protected]>