Auto update parquets #20
Conversation
…tion_errors [ADD] Added support for catching other types of connection errors
…tion_errors [FIX] added a timeout of 30 seconds by default
Added progressbar to data retrieval.
Removed unecessary prints. Refactored code further. Implemented direct-to-parquet candles update.
Compressed (parquet) and data (csv) folders specified with top level settings. Removed code related to development.
…eded the local time and would cause a progressbar exception.
|
I have noticed a few minor issues that I will attempt to correct. Please wait with the merge. |
|
Wooooow this has been on my list for so long (#13), really appreciate this! I wouldn't even mind getting rid of the csv files completely—they just take up space. I will wait for your final PR and dig through the code. Impression so far is awesome! |
New setting: Optional skipping of inactive pairs. User options are printed when program runs.
|
Happy you like it. A few more issues to iron out. |
|
Okay, it's ready on my end. There were no additional issues that had to be fixed. |
gosuto-inzasheru
changed the base branch from
master
to
dev-nup002-auto-parquets
| if bar is not None: | ||
| time_covered = datetime.fromtimestamp(last_timestamp / 1000) - start_datetime | ||
| minutes_covered = int(time_covered.total_seconds()/60) | ||
| bar.max_value = max(int((datetime.now() - start_datetime).total_seconds()/60), minutes_covered) |
There was a problem hiding this comment.
@nup002 can I ask what you are doing here? It seems to introduce a bug where the max of the bar can become higher than what is communicated in line 154 (total_minutes_of_data).
Wouldn't it be better for consistency to stick to max(bar.max_value, minutes_covered)?
There was a problem hiding this comment.
Yes, max(bar.max_value, minutes_covered) is a much more elegant solution.
This pull request adds new functionality:
Optional direct update of parquet files
Parquet files can be updated directly without the need for intermediate CSV files. This makes it much more convenient for users who have downloaded the parquet files from kaggle and wish to keep the up to date.
Improved output
A progressbar is displayed during the fetching of candles, showing the user how many candles will be retrieved, and how long it is expected to take.
Additional user settings
The user can set global variables on the top of Main.py to control how the script behaves.
The code has also been refactored in order to more easily implement the changes listed above. The functionality of the code has not been changed, and the script can be used just as before this pull request.