Feature: Configuration Parser

[siddiqkaithodu]

Have you searched if there an existing feature request for this?

• I have searched the existing requests

Feature description

Needs a feature that configure the scraper from an external configuration file.

This configuration can include default proxies , default headers for different domain names that may found in the scraper with each of the domains also have seperate page types .
eg:
(listing, product page,store api etc for each domain names).

The page types will be auto determined using regex matching of the url found and the appropriate headers and proxies with the specified order will be used.
eg:

{domains:[
{
url:example.com,
types:[{
type:product,
url: example.com/prd/*,
proxies:[0.0.0.0,9.9.9.9],
header:{
..........
}
},
{
type:listing,
url: example.com/listing/*,
proxies:[1.0.0.0,9.8.8.9],
header:{
..........
}
}
]
}
]
}

[D4Vinci]

Thanks for the idea, I will try to add it with version 0.3