Newegg 403 and Amazon 504 Fixes

[Utherellus]

I hope this tiny guide will help newcomers who are getting banned from Newegg and Amazon. I finally solved this issue and want to share with you guys.

First of all, I recommend using Google Cloud VM for this. Setting up streetmerchant on your personal daily machine will not be convenient. Google has 90 days free promo.

Newegg simply bans IP. They dont really care about Captcha. I bypassed that by installing Tor and using it's proxy to change IP every 2 minutes. https://youtu.be/wjBO0HM_JuU This video will help you understand how it works. You just adapt this scenario to linux by writing proxy settings in .env and adding MaxCircuitDirtiness NUM(minutes you want IP to be changed) to TORRC file(Located in etc/tor)

Amazon on the other hand blacklists you because of Captcha. To avoid this, you can clone StreetMerchant repo folder entirely and use slightly different .env config than in case of Newegg: We dont need proxies and IP randomizer here(though you can if you want), just headless must be set to false in .env file.
This will trigger opening Chromium everytime repo checks the stock. But it never failed me while testing it whole day.

I ended up with two terminals open: one for Newegg, one for Amazon. They work with different settings and so far never failed to check the stock.

Hope this helps folks around here.

[airsharkie]

I wasn't able to get this to work in Windows 10. I had no problem setting:
in TORRC:
MaxCircuitDirtiness 10
(I was going to change to "2" after I got it working)

In .env for Streetmerchant, I set:
PROXY_ADDRESS="127.0.0.1"
PROXY_PROTOCOL="socks5"
PROXY_PORT="9150"

Start Tor.
Start streetmerchant "NPM RUN START"

I can open another window through Chromium when it is is invoked and see my IP addresses are definitely going through Tor and they are changing at intervals.

Setting HEADLESS: "false"
I can see tons of stores blocking me and/or telling me that "STORE" isn't available in my country (US).

Then in TORCC I added:
ExitNodes {us}
StrictNodes 1

Sure enough, I ONLY get US based exit nodes, but the same warnings appear in Chromium (US based, blocked, etc) as it's running through it's automation.

I figured I would try this from home as I have an extra computer, plus I was worried about the massive amount of traffic being billed if I ran this from GCP. In just one week I've used 1TB of data. I have unlimited Xfinity here in Oregon.

I've given this my level best and am not sure how to go farther. If you have any ideas, I'm open to hearing them.

[ghost]

Hey, can you elaborate on how you integrated MaxCircuitDirtiness to the .env file. Sorry I'm new to this and it would really help if you tell me how to do it a little bit elaborately.

[airsharkie]

MaxCircuitDirtiness NUM goes in the 'TORRC' file in the Tor installation folder.

'Path'\Tor Browser\Browser\TorBrowser\Data\Tor\torrc

Launch Tor once, then close it. Edit torrc using a text editor like Notepad++.

Add:
MaxCircuitDirtiness NUM
to the end of the file. Change 'NUM' to the value in minutes you'd like to auto switch proxies.

The next time you start up the Tor browser, you should start seeing your IP change at the interval you specified.

This is what my entire trrc file looks like:

---

This file was generated by Tor; if you edit it, comments will not be preserved

The old torrc file was renamed to torrc.orig.1 or similar, and Tor will ignore it

ClientOnionAuthDir C:\Users\xxxx\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor\onion-auth
DataDirectory C:\Users\xxxx\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\Users\xxxx\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\Users\xxxx\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6
ExitNodes {us}
StrictNodes 1
MaxCircuitDirtiness 2

[ghost]

Ok this makes sense, but how will the chromium know the change the ip address. This only tells tor to change the ip address every NUM. How could this setting change affect chromium and .env files on streetmerchant to avoid ip ban on different sites?

[airsharkie]

In Streetmerchant's .env file, you edit to tell it to use the local machine Tor proxy:
PROXY_ADDRESS="127.0.0.1"
PROXY_PROTOCOL="socks5"
PROXY_PORT="9150"

Open Tor.
Open streetmerchant and it will use the proxy when Chromium is invoked.

But....Just like I stated earlier in this thread, a lot of the exit node IP's being used seem to be blocked by merchant websites.
I enabled US Only IP's as well in Tor and it didn't help.

Not sure how @Utherellus is making it work.

[Utherellus]

yep. I got it working exactly that way.

[Utherellus]

P.S. If someone's getting errors with
PROXY_ADDRESS="127.0.0.1"
PROXY_PROTOCOL="socks5"
PROXY_PORT="9150"

in .env file try 9050 port.

[mss-repo]

How hard or easy would it be to directly implement this in StreetMerchant itself? Instead of using the single proxy option in the .env file itself, Couldn't you just provide it a list of proxies that it rotates through for every time it scans a store/item? This way every scan/update uses a new IP/Proxy.

I tried the update that was added where you can add a list of proxies for each individual store but it seemed to have overwhelmed it and it crashes right away (I created 5 individual .proxies files for each of my 5 stores I was scanning and I put 25 proxies in each). Instead of this, It'd be better to provide StreetMerchant itself this list of 25 proxies that it rotates through for every scan for ALL the stores.

[holeyness]

For amazon, do you mean INCOGNITO and not HEADLESS? If it's running on a remote server, I don't think it can run not headless mode

[Utherellus]

not sure what you mean but headless=false opens browser physically, which apparently kinda helps bypassing captcha