Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Scraper Matching System #368

Open
WingofaGriffin opened this issue Feb 24, 2024 · 1 comment
Open

Improve Scraper Matching System #368

WingofaGriffin opened this issue Feb 24, 2024 · 1 comment
Labels
enhancement New feature or request topic:scraper Issues or pull requests related to game information scrapers
Milestone

Comments

@WingofaGriffin
Copy link

I've noticed that the scraper is getting hung up on special characters in file names, most notably things in parenthesis, brackets, and braces. This leaves things such as versions and country codes in the scraping text, leading to fail to find matches.

Additionally, may I suggest adding integration with IGDB as a database? It tends to have much less rate limiting than ScreenScraper, and even includes some entries they don't have. Applications like Daijisho use it as their primary scraper.

@rsubtil
Copy link
Member

rsubtil commented Feb 24, 2024

Yes, there needs to be some cleanup system when using raw file names due to that extra garbage. I'm still not sure of a reliable way to do so, but I'll investigate options and detail them here in the future.

As for IGDB, I've considered using it too as it's way more reliable. However, the main issue is how their API works, which requires me to generate a Twitch token, which has an expiration time. Embedding this in the app would force me (and users) to update the app every ~2 months. And their suggestion of setting up a proxy service for requests is likely to incur in costs I can't have ATM.

I want to eventually support more scrapers in the future and IGDB is indeed a good candidate, but I'll need to figure out a good solution for this problem.

@rsubtil rsubtil added this to the v1.0.2 milestone Feb 28, 2024
@rsubtil rsubtil added enhancement New feature or request topic:scraper Issues or pull requests related to game information scrapers labels Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request topic:scraper Issues or pull requests related to game information scrapers
Projects
None yet
Development

No branches or pull requests

2 participants