Parameter mapping

[Gallaecio]

Resolves #143.

• _params.py now contains a dictionary of top-level Zyte API request params with metadata about them, and it should be extended as new parameters are added upstream in the future. Related changes:
  • "jobPosting" is now recognized as an extraction type (affects whether or not httpResponseBody and httpResponseHeaders are enabled by default).
  • "GET" is now recognized as the default value of "httpRequestMethod", and stripped from request parameters if manually defined.
• Fingerprinting changes:
  • *Options.extractFrom is now taken into account to determine whether or not to keep the URL fragment. The fragment is no longer kept just because automatic extraction is used, browserHtml must also be explicitly requested as extraction source.
  • Matching "httpRequestBody" and "httpRequestText" values do not cause different fingerprints.
  • The following additional keys are ignored for the sake of fingerprints: sessionContext, sessionContextParameters, cookieManagement, requestCookies, responseCookies.
  • ⚠️ device remains taken into account for fingerprinting (the default behavior for any unknown key). I am not completely sure, though. If it only affects requests (and not e.g. the viewport), maybe it would be best to ignore it during fingerprinting.
  • ⚠️ I wonder if we should also ignore httpResponseHeaders (True/False), since we already take into account other outputs it can be combined with, in line how responseCookies is now ignored.

To do:

• Tests for new scenarios.
• Update after Use Sphinx and ReadTheDocs for the documentation #150.

[codecov]

Codecov Report

Merging #151 (4943f47) into main (b892453) will increase coverage by 0.08%.
Report is 2 commits behind head on main.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #151      +/-   ##
==========================================
+ Coverage   98.83%   98.92%   +0.08%     
==========================================
  Files          10       10              
  Lines         687      744      +57     
==========================================
+ Hits          679      736      +57     
  Misses          8        8              

Files	Coverage Δ
scrapy_zyte_api/_params.py	100.00% <100.00%> (ø)
scrapy_zyte_api/_request_fingerprinter.py	97.56% <100.00%> (+0.19%)	⬆️
scrapy_zyte_api/responses.py	99.00% <ø> (+0.04%)	⬆️

... and 3 files with indirect coverage changes