Got banned after 4 requests (3 sec gap)

[shinkuan]

每次測試都在4次請求後被禁, 請求與請求之間有三秒間隔, 以下為代碼

`import twstock
import time

// Create a Stock array
stock = []

// stock[i] is stock which sid = i
for i in range(0, 10000):
sid = str(i)
# check if stock i exist
if sid in twstock.twse or sid in twstock.tpex:
stock.append(twstock.Stock(sid))
# print(stock[i].data)
print('sid' + stock[i].sid + 'get!')
time.sleep(3)
# if not, its NaN
else:
stock.append('NaN')
print('stock' + sid + 'does not exist')`

以下為輸出結果:

stock0does not exist
stock1does not exist
stock2does not exist
stock3does not exist
.
.
.
stock1097does not exist
stock1098does not exist
stock1099does not exist
stock1100does not exist
sid1101get!
sid1102get!
sid1103get!
sid1104get!
stock1105does not exist
stock1106does not exist
stock1107does not exist
Traceback (most recent call last):
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse
response.begin()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\util\retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse
response.begin()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/USER/PycharmProjects/StockTrade/app.py", line 11, in
stock.append(twstock.Stock(sid))
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 154, in init
self.fetch_31()
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 183, in fetch_31
self.fetch_from(before.year, before.month)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 175, in fetch_from
self.raw_data.append(self.fetcher.fetch(year, month, self.sid))
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 58, in fetch
r = requests.get(self.REPORT_URL, params=params,
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Process finished with exit code 1

有試過用更長時間 (10 sec) 為間隔 依然被禁

[joytsay]

我發現可以隨機給予3-10秒sleep (抓取中華電信20年成功):

in stock.py (line 55)

    def fetch(self, year: int, month: int, sid: str, retry: int=5):
        params = {'date': '%d%02d01' % (year, month), 'stockNo': sid}
        for retry_i in range(retry):
            randNum = random.randrange(3,10)    ##增加這兩行##
            time.sleep(randNum)                 ##增加這兩行##
            r = requests.get(self.REPORT_URL, params=params,
                             proxies=get_proxies())

[rach-lyn]

請問這裡的self是要填什麼啊？謝謝> 我發現可以隨機給予3-10秒sleep (抓取中華電信20年成功):

in stock.py (line 55)

    def fetch(self, year: int, month: int, sid: str, retry: int=5):

        params = {'date': '%d%02d01' % (year, month), 'stockNo': sid}

        for retry_i in range(retry):

            randNum = random.randrange(3,10)    ##增加這兩行##

            time.sleep(randNum)                 ##增加這兩行##

            r = requests.get(self.REPORT_URL, params=params,

                             proxies=get_proxies())

[joytsay]

self 只是 python 語言裡面 class 的 instance
這裡為了要避掉 class TWSEFetcher(BaseFetcher)中
fetch fucntion 抓太快被台股 server 鎖IP的問題
在twstock/stock.py#L57 的下一行
for 迴圈裏面 增加 以下兩行 code 即可

randNum = random.randrange(3,10)   
time.sleep(randNum) 

呼叫方法可以參考我的Jupyter Notebook
stock_from = stock.fetch_from(2000,10) # 獲取 2010 年 01 月至今日之股票資料