Code

Avoid duplicate requests#105

Code

Merged

Gallaecio merged 117 commits intoscrapy-plugins:provider-fix-indirect-duplicationscrapy-plugins/scrapy-zyte-api:provider-fix-indirect-duplicationfrom Gallaecio:provider-fix-indirect-duplicationGallaecio/scrapy-zyte-api:provider-fix-indirect-duplicationCopy head branch name to clipboardDec 27, 2023

+3,392-1,181

Commits

Commits on Jul 3, 2023

• Move the provider deps into an optional feature.

  wRAR

  committed
  5908b6c

  Copy full SHA for 5908b6c

• Run the provider tests on CI.

  wRAR

  committed
  eaa3409

  Copy full SHA for eaa3409

Commits on Jul 4, 2023

• Merge pull request scrapy-plugins#98 from scrapy-plugins/scrapy-poet-optional

  Show description for a3f3e15
  kmike

  authored
  a3f3e15

  Copy full SHA for a3f3e15

Commits on Jul 12, 2023

• ZyteApiProvider: support more data types

  kmike

  committed
  5ad4d79

  Copy full SHA for 5ad4d79

• pre-commit fixes

  kmike

  committed
  914d599

  Copy full SHA for 914d599

• document requirements for scrapy-poet integration

  kmike

  committed
  3c81296

  Copy full SHA for 3c81296

• cleanup tools configuration

  Show description for be25b79
  kmike

  committed
  be25b79

  Copy full SHA for be25b79

• fix black configuration

  kmike

  committed
  edd2ed2

  Copy full SHA for edd2ed2

• document new dependencies which scrapy-zyte-api can provide

  kmike

  committed
  44c3a5f

  Copy full SHA for 44c3a5f

• proper mypy config

  kmike

  committed
  8fd2138

  Copy full SHA for 8fd2138

• Update README.rst

  Show description for 7ac7f12
  

  kmike

  and

  Gallaecio

  authored
  7ac7f12

  Copy full SHA for 7ac7f12

• make SCRAPY_POET_PROVIDERS configuration copy-pasteable (scrapy-plugins#102)

  kmike

  authored
  aeca861

  Copy full SHA for aeca861

• Merge pull request scrapy-plugins#101 from scrapy-plugins/cleanup-ci

  Show description for fd91fbf
  kmike

  authored
  fd91fbf

  Copy full SHA for fd91fbf

• Merge pull request scrapy-plugins#100 from scrapy-plugins/more-data-types

  Show description for bd4bd6b
  kmike

  authored
  bd4bd6b

  Copy full SHA for bd4bd6b

• add build/ folder to gitignore (scrapy-plugins#103)

  kmike

  authored
  6d549b7

  Copy full SHA for 6d549b7

• add a quick start section to readme

  kmike

  committed
  226620c

  Copy full SHA for 226620c

• fix rst syntax in readme

  kmike

  committed
  139f55e

  Copy full SHA for 139f55e

• Merge pull request scrapy-plugins#104 from scrapy-plugins/quickstart

  Show description for 267d689
  kmike

  authored
  267d689

  Copy full SHA for 267d689

Commits on Jul 13, 2023

• WIP

  Gallaecio

  committed
  23c4940

  Copy full SHA for 23c4940

• Support driving Zyte API requests through a proxy

  Gallaecio

  committed
  1c7db87

  Copy full SHA for 1c7db87

• Add release notes for 0.10.0.

  wRAR

  committed
  4e4c829

  Copy full SHA for 4e4c829

Commits on Jul 14, 2023

• Merge pull request scrapy-plugins#109 from scrapy-plugins/relnotes-0.10.0

  Show description for 0c8b648
  wRAR

  authored
  0c8b648

  Copy full SHA for 0c8b648

• Reword the behind-a-proxy readme entry to minimize confusion

  Gallaecio

  committed
  46ae308

  Copy full SHA for 46ae308

• Merge remote-tracking branch 'scrapy-plugins/main' into trust-env

  Gallaecio

  committed
  24a4914

  Copy full SHA for 24a4914

• CHANGES:rst: cover ZYTE_API_USE_ENV_PROXY

  Gallaecio

  committed
  4361ca7

  Copy full SHA for 4361ca7

• Merge pull request scrapy-plugins#106 from Gallaecio/trust-env

  Show description for 28927c0
  kmike

  authored
  28927c0

  Copy full SHA for 28927c0

• Bump version: 0.9.0 → 0.10.0

  wRAR

  committed
  23762b6

  Copy full SHA for 23762b6

Commits on Jul 19, 2023

• introduce ZYTE_API_MAX_REQUESTS

  BurnzZ

  committed
  febe99f

  Copy full SHA for febe99f

• update tests to include non-ZAPI requests

  BurnzZ

  committed
  5a8ce24

  Copy full SHA for 5a8ce24

• refactor conditional check on max request reached

  BurnzZ

  committed
  561d9ff

  Copy full SHA for 561d9ff

Commits on Jul 25, 2023

• Warn about requestCookies=[] (scrapy-plugins#115)

  Gallaecio

  authored
  276c7f4

  Copy full SHA for 276c7f4

Commits on Aug 2, 2023

• improve IgnoreRequest message when max ZAPI requests is reached

  BurnzZ

  committed
  660cdb2

  Copy full SHA for 660cdb2

• document new ZYTE_API_MAX_REQUESTS setting

  BurnzZ

  committed
  63a9d44

  Copy full SHA for 63a9d44

Commits on Aug 3, 2023

• Styling improvements

  Show description for 2820d56
  

  BurnzZ

  and

  Gallaecio

  authored
  2820d56

  Copy full SHA for 2820d56

• Merge pull request scrapy-plugins#114 from scrapy-plugins/max-zapi-requests

  Show description for 3c09dfc
  kmike

  authored
  3c09dfc

  Copy full SHA for 3c09dfc

Commits on Aug 4, 2023

• CHANGES.rst: Cover 0.11.0

  Gallaecio

  committed
  8438007

  Copy full SHA for 8438007

• Clarify that ZYTE_API_MAX_REQUESTS limits successful requests

  Gallaecio

  committed
  1097650

  Copy full SHA for 1097650

Commits on Aug 7, 2023

• Merge pull request scrapy-plugins#117 from Gallaecio/release-notes

  Show description for 6da9a39
  kmike

  authored
  6da9a39

  Copy full SHA for 6da9a39

• Bump version: 0.10.0 → 0.11.0

  Gallaecio

  committed
  b2d251d

  Copy full SHA for b2d251d

• CHANGES.rst: set the 0.11.0 release date

  Gallaecio

  committed
  654aaba

  Copy full SHA for 654aaba

• Fix version bumping

  Gallaecio

  committed
  e11265e

  Copy full SHA for e11265e

• Support scrapy-poet 0.10.0 again

  Gallaecio

  committed
  30c91d0

  Copy full SHA for 30c91d0

• Hint at potential issues when switching reactors on existing projects (scrapy-plugins#116)

  Gallaecio

  authored
  d3e7115

  Copy full SHA for d3e7115

Commits on Aug 8, 2023

• Merge pull request scrapy-plugins#121 from Gallaecio/fix-version-bumping

  Show description for 5708e0e
  kmike

  authored
  5708e0e

  Copy full SHA for 5708e0e

• Test the minimum dependencies of the provider extra (scrapy-plugins#122)

  Gallaecio

  authored
  75808e3

  Copy full SHA for 75808e3

Commits on Aug 10, 2023

• Clarify how status codes are reflected in stats

  Gallaecio

  authored
  a330b4b

  Copy full SHA for a330b4b

• Merge pull request scrapy-plugins#123 from scrapy-plugins/stat-status-codes

  Show description for 51823c9
  kmike

  authored
  51823c9

  Copy full SHA for 51823c9

Commits on Aug 21, 2023

• Do not warn about dropping the default Accept-Encoding header

  Gallaecio

  committed
  8b1411b

  Copy full SHA for 8b1411b

Commits on Aug 23, 2023

• Merge pull request scrapy-plugins#126 from scrapy-plugins/dont-warn-on-default-headers

  Show description for b8a2fa4
  kmike

  authored
  b8a2fa4

  Copy full SHA for b8a2fa4

Commits on Aug 24, 2023

• Cover 0.11.1 in the release notes

  Gallaecio

  committed
  a82ccbb

  Copy full SHA for a82ccbb

• Fix a typo

  Gallaecio

  authored
  deb1826

  Copy full SHA for deb1826

• Merge pull request scrapy-plugins#127 from Gallaecio/release-notes

  Show description for 20046c1
  kmike

  authored
  20046c1

  Copy full SHA for 20046c1

Commits on Aug 25, 2023

• Set a release date for 0.11.1

  Gallaecio

  committed
  6fd143b

  Copy full SHA for 6fd143b

• Bump version: 0.11.0 → 0.11.1

  Gallaecio

  committed
  cd47399

  Copy full SHA for cd47399

• Track top-level request args in stats

  Gallaecio

  committed
  b9eb5d1

  Copy full SHA for b9eb5d1

Commits on Aug 30, 2023

• Apply special treatment for experimental features

  Gallaecio

  committed
  21926eb

  Copy full SHA for 21926eb

Commits on Aug 31, 2023

• Merge pull request scrapy-plugins#128 from Gallaecio/request-arg-stats

  Show description for d61cb68
  kmike

  authored
  d61cb68

  Copy full SHA for d61cb68

Commits on Sep 26, 2023

• Implement ZYTE_API_PROVIDER_PARAMS

  Gallaecio

  committed
  0282e00

  Copy full SHA for 0282e00

• Change the code-block syntax for one that works on GitHub

  Gallaecio

  committed
  af1e223

  Copy full SHA for af1e223

• Merge pull request scrapy-plugins#133 from Gallaecio/provider-geolocation

  Show description for c5f02e0
  kmike

  authored
  c5f02e0

  Copy full SHA for c5f02e0

• Add release notes for 0.12.0

  Gallaecio

  committed
  f2dac4e

  Copy full SHA for f2dac4e

• Merge pull request scrapy-plugins#134 from Gallaecio/release-notes

  Show description for 207fed4
  kmike

  authored
  207fed4

  Copy full SHA for 207fed4

• Set a release date for 0.12.0

  Gallaecio

  committed
  c60c48a

  Copy full SHA for c60c48a

• Bump version: 0.11.1 → 0.12.0

  Gallaecio

  committed
  0918370

  Copy full SHA for 0918370

Commits on Sep 28, 2023

• change ellipsis to 3 dots

  BurnzZ

  committed
  b4ce6e0

  Copy full SHA for b4ce6e0

Commits on Sep 29, 2023

• Identify scrapy-zyte-api usage via custom user-agent (scrapy-plugins#130)

  Show description for d50b14d
  

  PyExplorer

  and

  Gallaecio

  authored
  d50b14d

  Copy full SHA for d50b14d

• updates for 0.12.1 release (scrapy-plugins#137)

  Show description for 6de738a
  

  PyExplorer

  and

  Gallaecio

  authored
  6de738a

  Copy full SHA for 6de738a

• update date for the release 0.12.1

  PyExplorer

  committed
  1bd2e25

  Copy full SHA for 1bd2e25

• Bump version: 0.12.0 → 0.12.1

  PyExplorer

  committed
  b110833

  Copy full SHA for b110833

Commits on Oct 4, 2023

• Merge pull request scrapy-plugins#136 from scrapy-plugins/ellipsis

  Show description for 3db30f1
  kmike

  authored
  3db30f1

  Copy full SHA for 3db30f1

• remove unused extration options

  kmike

  committed
  60e2265

  Copy full SHA for 60e2265

Commits on Oct 11, 2023

• Initial support for typing.Annotated.

  wRAR

  committed
  2cabbe0

  Copy full SHA for 2cabbe0

Commits on Oct 13, 2023

• Switch to scrapy_poet.AnnotatedResult, improve the test.

  wRAR

  committed
  ab34b93

  Copy full SHA for ab34b93

Commits on Oct 16, 2023

• Forbid multiple extractFrom.

  wRAR

  committed
  ecc42d4

  Copy full SHA for ecc42d4

Commits on Oct 18, 2023

• document the new behavior

  kmike

  committed
  7f15a9c

  Copy full SHA for 7f15a9c

Commits on Oct 19, 2023

• Update README.rst

  Show description for 8ee4e24
  

  kmike

  and

  Gallaecio

  authored
  8ee4e24

  Copy full SHA for 8ee4e24

• Merge pull request scrapy-plugins#138 from scrapy-plugins/remove-extraction-options

  Show description for ad39749
  kmike

  authored
  ad39749

  Copy full SHA for ad39749

• 0.12.2 release notes

  kmike

  committed
  bb54a28

  Copy full SHA for bb54a28

• Bump version: 0.12.1 → 0.12.2

  kmike

  committed
  1370505

  Copy full SHA for 1370505

Commits on Oct 23, 2023

• log Zyte API request id on errors

  BurnzZ

  committed
  fbf37f0

  Copy full SHA for fbf37f0

Commits on Oct 25, 2023

• use new RequestError.request_id attribute

  BurnzZ

  committed
  600a571

  Copy full SHA for 600a571

Commits on Nov 2, 2023

• bump zyte-api dep: 0.4.7 → 0.4.8

  BurnzZ

  committed
  80ae0a0

  Copy full SHA for 80ae0a0

Commits on Nov 3, 2023

• Use repr when logging request-id

  Show description for 63228ec
  

  BurnzZ

  and

  Gallaecio

  authored
  63228ec

  Copy full SHA for 63228ec

• Merge pull request scrapy-plugins#142 from scrapy-plugins/log-request-id

  Show description for 5e205ef
  BurnzZ

  authored
  5e205ef

  Copy full SHA for 5e205ef

Commits on Nov 6, 2023

• Test that different extraction outputs generate different fingerprints (scrapy-plugins#145)

  Gallaecio

  authored
  0b58307

  Copy full SHA for 0b58307

Commits on Nov 7, 2023

• Clarify the max requests docs

  Gallaecio

  committed
  438df61

  Copy full SHA for 438df61

• Update README.rst

  Gallaecio

  authored
  b73bfc2

  Copy full SHA for b73bfc2

Commits on Nov 8, 2023

• Merge pull request scrapy-plugins#146 from Gallaecio/max-request-docs

  Show description for b395711
  kmike

  authored
  b395711

  Copy full SHA for b395711

Commits on Nov 14, 2023

• Set a close reason for bad key and suspended account scenarios

  Gallaecio

  committed
  e55e99c

  Copy full SHA for e55e99c

• Merge remote-tracking branch 'origin/main' into annotated-support

  wRAR

  committed
  87bd23a

  Copy full SHA for 87bd23a

• Fix removing *Options.

  wRAR

  committed
  4b5c088

  Copy full SHA for 4b5c088

• Update the pinned zyte-api version in CI (scrapy-plugins#149)

  wRAR

  authored
  aa3f77b

  Copy full SHA for aa3f77b

• Fixes and improvements.

  wRAR

  committed
  b63ad66

  Copy full SHA for b63ad66

• Fix CI issues.

  wRAR

  committed
  15fc06e

  Copy full SHA for 15fc06e

• More fixes.

  wRAR

  committed
  08a5f48

  Copy full SHA for 08a5f48

• Install more deps for the mypy test.

  wRAR

  committed
  e0ba154

  Copy full SHA for e0ba154

• Merge pull request scrapy-plugins#148 from Gallaecio/bad-auth

  Show description for 37e3517
  kmike

  authored
  37e3517

  Copy full SHA for 37e3517

Commits on Nov 15, 2023

• Fix an old typing issue.

  wRAR

  committed
  2d936fd

  Copy full SHA for 2d936fd

Commits on Nov 20, 2023

• Use Sphinx and ReadTheDocs for the documentation (scrapy-plugins#150)

  Gallaecio

  authored
  b892453

  Copy full SHA for b892453

Commits on Nov 24, 2023

• Drop the Set-Cookie header (scrapy-plugins#132)

  Gallaecio

  authored
  493a48c

  Copy full SHA for 493a48c

Commits on Nov 28, 2023

• Improve forbidden domain handling (scrapy-plugins#147)

  Gallaecio

  authored
  dc09ac3

  Copy full SHA for dc09ac3

Commits on Nov 30, 2023

• Implement a parameter map (scrapy-plugins#151)

  Gallaecio

  authored
  702cc63

  Copy full SHA for 702cc63

• Fail on SPM + transparent mode (scrapy-plugins#152)

  Gallaecio

  authored
  d701f48

  Copy full SHA for d701f48

Commits on Dec 12, 2023

• Merge remote-tracking branch 'origin/main' into annotated-support

  wRAR

  committed
  cad9c3a

  Copy full SHA for cad9c3a

• Bump the andi version.

  wRAR

  committed
  e9be1a3

  Copy full SHA for e9be1a3

• Fix and improve extractFrom tests.

  wRAR

  committed
  3ee24f0

  Copy full SHA for 3ee24f0

• Roll back capture_exceptions.

  wRAR

  committed
  2b0d659

  Copy full SHA for 2b0d659

• Bump the scrapy-poet version.

  wRAR

  committed
  de01fc2

  Copy full SHA for de01fc2

• Bump the web-poet version.

  wRAR

  committed
  412475e

  Copy full SHA for 412475e

• Move ExtractFrom into scrapy_zyte_api/_annotations.py.

  wRAR

  committed
  5305b46

  Copy full SHA for 5305b46

• Add docs for ExtractFrom.

  wRAR

  committed
  f5bb894

  Copy full SHA for f5bb894

• Merge pull request scrapy-plugins#141 from scrapy-plugins/annotated-support

  Show description for 11a9b03
  wRAR

  authored
  11a9b03

  Copy full SHA for 11a9b03

Commits on Dec 13, 2023

• Release notes for 0.13.0.

  wRAR

  committed
  a80fc0b

  Copy full SHA for a80fc0b

• Merge pull request scrapy-plugins#153 from scrapy-plugins/relnotes-0.13.0

  Show description for fab5e58
  wRAR

  authored
  fab5e58

  Copy full SHA for fab5e58

• Bump version: 0.12.2 → 0.13.0

  wRAR

  committed
  0536f42

  Copy full SHA for 0536f42

Commits on Dec 27, 2023

• Merge remote-tracking branch 'scrapy-plugins/main' into provider-fix-indirect-duplication

  Gallaecio

  committed
  a5f909c

  Copy full SHA for a5f909c

• Require scrapy-poet >= 0.19.0

  Gallaecio

  committed
  857b51c

  Copy full SHA for 857b51c