DummyResponse

[bulatbulat48]

When we add a type hint DummyResponse to the response as it mentioned here:

The scrapy doesn't send the actions, experimental, cookies or other additional settings added to the settings or the the request:

Maybe it's a cloudflare related issue, idk.

cookies = {
    'bh_cookies_accepted': True,
    'bh_cookies_allow_all': True,
    'bh_cookies_allow_marketing': True,
    'bh_cookies_allow_statistic': True,
    'bh_cookies_allow_thirdcountries': True,
},

@classmethod
def update_settings(cls, settings: BaseSettings) -> None:
    settings["ZYTE_API_EXPERIMENTAL_COOKIES_ENABLED"] = True
    settings["ZYTE_API_LOG_REQUESTS"] = True
    settings["ZYTE_API_LOG_REQUESTS_TRUNCATE"] = 1000
    return super().update_settings(settings)

2024-09-26 20:40:50 [scrapy_poet.downloadermiddlewares] DEBUG: Using DummyResponse instead of downloading <GET https://www.example.com/>
2024-09-26 20:40:50 [scrapy_zyte_api.handler] DEBUG: Sending Zyte API extract request: {"browserHtml": true, "url": "https://www.example.com"}

[kmike]

@bulatbulat48 could you please also provide the example of the callback you're using? The signature is enough.

[bulatbulat48]

@kmike sure:

async def parse_product(
        self, response: DummyResponse, product_dpo: ProductPageObject
):

[kmike]

What are the dependencies of ProductPageObject?

[bulatbulat48]

@kmike

from web_poet import BrowserResponse, HttpResponse, ItemPage, PageParams, field

@attrs.define(auto_attribs=True)
class ProductPageObject(ItemPage[ProductItem]):
    response: BrowserResponse
    page_params: PageParams

[Gallaecio]

The behavior here is as designed, Zyte API params for provider dependencies can only be altered with the zyte_api_provider request metadata key or the ZYTE_API_PROVIDER_PARAMS setting.

However, I wonder if for some parameters, like cookies, we should make things behave like automap, or at least make it easier for them to do that, instead of requiring those cookies to be specified in those settings as Zyte API params.