Repeated GET calls not being played back properly

[aisrael]

Background

Am writing a spec for an API client. To verify the client is working I'm trying to:

1. GET /api to fetch the current state/value
2. POST /api to set the new state/value
3. GET /api again to get the new state/value

Issue

When first running the spec and recording the cassette, the spec passes.

When running the spec again and playing back the recorded interactions, the spec fails in step 3. The result of the first GET is being returned as the result of the second GET.

Sample code demonstrating the issue

Please see: https://github.com/aisrael/hi8-test

In particular, note the different results of two successive GET requests at repeated_calls_failing.yml#17 and repeated_calls_failing.yml#L33

The test in hi8-test_spec.cr goes like:

    HI8.use_cassette("repeated_calls_failing") do
      get1 = HTTP::Client.get "http://localhost:3000"
      get1.body.should eq("1")
      get2 = HTTP::Client.get "http://localhost:3000"
      get2.body.should eq("2")
    end

Expected result

The spec passes

Actual result

The spec fails with:

  1) Hi8::Test demonstrates the issue
     Failure/Error: get2.body.should eq("2")

       Expected: "2"
            got: "1"

[aisrael]

So, I've traced it down to WebMock. When we set up the episodes to be stubbed by WebMock, we call WebMock.stub()... for each. WebMock then creates its internal Stub objects for each and appends them to its StubRegitry.

The problem occurs when WebMock tries to find the matching stub for any given request. WebMock will sequentially go through all stubs attempting to find a match. Since we are performing identical requests, the first stub will match the second request.

Will see if there's an easy way to have WebMock "consume" a stub after the first time it matches a request.

[alexanderadam]

Hi Alistair,

if I understood this correctly you expect different results when you make the same request?
If so: this library records and replays same looking requests. So if GET /foo returned bar it will be recorded and every other request that looks like GET /foo will return bar from this moment on.
This is usually expected and the reason why this library is used.

However, you can control this behaviour with different ways, like switching to another cassette (and therefore another result) for example or setting an on_playback block (where you can change request matching as you prefer it).

But maybe I just misunderstood your issue!?

[aisrael]

Hi, Alexander,

When I use vcr gem for Ruby, if I make a GET request the first time vcr records that as one request.

If I issue the same GET request a second time, that then gets recorded as another request, whether the server responds with the same response or not (in my case, since I POST in between GETs, I'm expecting a new value).

I can put up a Ruby gist to demonstrate.

[aisrael]

https://gist.github.com/aisrael/c389569b83ac2c12e1cbc7aec1032a5a

I think in this case hi8 essentially performs caching of HTTP requests and matching based on the "signature". Identical request signatures then yield the same result.

But, as demonstrated, this is not necessarily the desired (nor the expected, based on vcr) behaviour. In my case, I'm asserting that, while initially the value is "1", after I POST, then new value should be "2".

[alexanderadam]

In that case the easiest solution could be to set the on_playback and add a request counter to compare that as well. So besides URL and headers compare for that as well (so you would expect that the "second" call to /foo would be independent from the first call).

[aisrael]

That might work if the Web service being mocked would just ignore the request counter, right? I mean, if I just append ?request_count=1 (and so on) to the URL then Webmock would be able to find the correct interaction. But in my case the Web service is beyond my control so I can't think of a way to safely add a counter to the requests to distinguish repeated calls from each other...

Anyway, I remember now this is really a limitation of current Webmock so if they don't act on the ticket I opened, I'm inclined to just patch Webmock myself.

[aisrael]

Just waiting on manastech/webmock.cr#21

Meanwhile, we're using my fork.