Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new data download command to lean-cli for data retrieval #447

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
9237490
feat: download file in json format
Romazes Apr 11, 2024
9f45262
feat: download no-/interactive with new command data-provider-historical
Romazes Apr 11, 2024
2c5598b
feat: constant data types
Romazes Apr 12, 2024
0fe8412
feat: new property in JsonModule
Romazes Apr 12, 2024
6f01c93
feat: validate of no-/interactive in download command
Romazes Apr 12, 2024
8171b2f
feat: constant data folder path
Romazes Apr 12, 2024
cc4f7bd
feat: entry point to new download project
Romazes Apr 12, 2024
6b3c158
feat: working_dir in docker
Romazes Apr 15, 2024
76aa61a
feat: get full title with namespace
Romazes Apr 15, 2024
2fa5da0
feat: get basic docker config without path
Romazes Apr 17, 2024
97d44db
refactor: split up get_basic_docker_config on small parts
Romazes Apr 17, 2024
044e8c4
revert: test_downloader file
Romazes Apr 17, 2024
1e97930
feat: support additional config in command line when user run `data d…
Romazes Apr 17, 2024
26f96f6
rename: remove historical prefix in command names
Romazes Apr 17, 2024
89559b8
feat: download run test
Romazes Apr 18, 2024
2ea806f
refactor: switch off interactive mode for `data-provider` param
Romazes Apr 18, 2024
4474502
remove: extra resolution
Romazes Apr 18, 2024
752f242
refactor: download json return like Dict[str, Any]
Romazes Apr 18, 2024
0432392
refactor: no-/interactive data download params
Romazes Apr 18, 2024
6cded4b
feat: test mock REST get request
Romazes Apr 18, 2024
ac482d2
refactor: put local methods above main method
Romazes Apr 18, 2024
c67d2f2
feat: new config param options `download`
Romazes Apr 18, 2024
586dada
remove: extra log.info
Romazes Apr 18, 2024
7b36bac
fix:bug: if provider has more than 2 download providers
Romazes Apr 18, 2024
1d31bb6
feat: additional test download process
Romazes Apr 18, 2024
fbd2d7e
fix: wrong condition tabulation
Romazes Apr 19, 2024
e339661
refactor: use provider_config live internal variable
Romazes Apr 19, 2024
fe4a76e
Revert "refactor: use provider_config live internal variable"
Romazes Apr 19, 2024
b3a8fc9
fix: test initialization of new property in mock config file
Romazes Apr 19, 2024
6acb4f2
feat: add market param
Romazes Apr 19, 2024
57496a2
refactor:test: data download
Romazes Apr 19, 2024
c7707dc
feat: create config file to run download provider project + mount
Romazes Apr 23, 2024
fe1b9b6
feat: mock get_organization
Romazes Apr 23, 2024
0d4da4e
refactor: styling
Romazes Apr 23, 2024
b55d4c3
fix: test_dataset_requirements
Romazes Apr 23, 2024
7b66686
refactor: input key's DataProvider interactive=False
Romazes Apr 24, 2024
1e8f5a6
rename: help message Select a historical Provider
Romazes Apr 24, 2024
e5cbf81
feat: interactive param test
Romazes Apr 24, 2024
97f792e
feat: new class QCDataType
Romazes Apr 24, 2024
49c2236
refactor: use QC default types
Romazes Apr 24, 2024
0767582
remove: extra argument data-download-name in pass to exe dll
Romazes Apr 24, 2024
8db7928
refactor: --no-update flag in local using
Romazes Apr 24, 2024
4dee041
fix: back compatibility assert test
Romazes Apr 24, 2024
c08a48c
rename: DownloaderDataProvider.dll
Romazes Apr 25, 2024
4569009
feat: update readme
Romazes Apr 26, 2024
f0a70d2
feat: description of data download command
Romazes Apr 26, 2024
01c541c
fix: mounting path of Data Providers' config
Romazes Apr 26, 2024
29037d9
feat: if lean_config without environment use lean_config in mount_paths
Romazes Apr 26, 2024
89ca968
feat: info to debug log in mount
Romazes Apr 26, 2024
21eb2c5
feat: user friendly prompt end-date
Romazes Apr 26, 2024
a902ac6
fix: get all config for data downloader
Romazes Apr 29, 2024
14c4638
refactor: specifications_url (like property)
Romazes Apr 29, 2024
451ca0f
rename: module specification properties
Romazes Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 100 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -781,12 +781,26 @@ _See code: [lean/commands/create_project.py](lean/commands/create_project.py)_

### `lean data download`

Purchase and download data from QuantConnect Datasets.
Purchase and download data directly from QuantConnect or download from Support Data Providers

```
Usage: lean data download [OPTIONS]

Purchase and download data from QuantConnect Datasets.
Purchase and download data directly from QuantConnect or download from Support Data Providers

1. Acquire Data from QuantConnect Datasets: Purchase and seamlessly download data directly from QuantConnect.

2. Streamlined Access from Support Data Providers:

- Choose your preferred historical data provider.

- Initiate hassle-free downloads from our supported providers.

We have 2 options:

- interactive (follow instruction in lean-cli)

- no interactive (write arguments in command line)

An interactive wizard will show to walk you through the process of selecting data, accepting the CLI API Access and
Data Agreement and payment. After this wizard the selected data will be downloaded automatically.
Expand All @@ -799,12 +813,90 @@ Usage: lean data download [OPTIONS]
https://www.quantconnect.com/datasets

Options:
--dataset TEXT The name of the dataset to download non-interactively
--overwrite Overwrite existing local data
-y, --yes Automatically confirm payment confirmation prompts
--lean-config FILE The Lean configuration file that should be used (defaults to the nearest lean.json)
--verbose Enable debug logging
--help Show this message and exit.
--data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit]
The name of the downloader data provider.
--ib-user-name TEXT Your Interactive Brokers username
--ib-account TEXT Your Interactive Brokers account id
--ib-password TEXT Your Interactive Brokers password
--ib-weekly-restart-utc-time TEXT
Weekly restart UTC time (hh:mm:ss). Each week on Sunday your algorithm is restarted at
this time, and will require 2FA verification. This is required by Interactive Brokers.
Use this option explicitly to override the default value.
--oanda-account-id TEXT Your OANDA account id
--oanda-access-token TEXT Your OANDA API token
--oanda-environment [Practice|Trade]
The environment to run in, Practice for fxTrade Practice, Trade for fxTrade
--bitfinex-api-key TEXT Your Bitfinex API key
--bitfinex-api-secret TEXT Your Bitfinex API secret
--coinbase-api-key TEXT Your Coinbase Advanced Trade API key
--coinbase-api-secret TEXT Your Coinbase Advanced Trade API secret
--binance-exchange-name [Binance|BinanceUS|Binance-USDM-Futures|Binance-COIN-Futures]
Binance exchange name [Binance, BinanceUS, Binance-USDM-Futures, Binance-COIN-Futures]
--binance-api-key TEXT Your Binance API key
--binanceus-api-key TEXT Your Binance API key
--binance-api-secret TEXT Your Binance API secret
--binanceus-api-secret TEXT Your Binance API secret
--kraken-api-key TEXT Your Kraken API key
--kraken-api-secret TEXT Your Kraken API secret
--kraken-verification-tier [Starter|Intermediate|Pro]
Your Kraken Verification Tier
--iqfeed-iqconnect TEXT The path to the IQConnect binary
--iqfeed-username TEXT Your IQFeed username
--iqfeed-password TEXT Your IQFeed password
--iqfeed-version TEXT The product version of your IQFeed developer account
--iqfeed-host TEXT The IQFeed host address
--polygon-api-key TEXT Your Polygon.io API Key
--factset-auth-config-file FILE
The path to the FactSet authentication configuration file
--iex-cloud-api-key TEXT Your iexcloud.io API token publishable key
--iex-price-plan [Launch|Grow|Enterprise]
Your IEX Cloud Price plan
--alpha-vantage-api-key TEXT Your Alpha Vantage Api Key
--alpha-vantage-price-plan [Free|Plan30|Plan75|Plan150|Plan300|Plan600|Plan1200]
Your Alpha Vantage Premium API Key plan
--coinapi-api-key TEXT Your coinapi.io Api Key
--coinapi-product [Free|Startup|Streamer|Professional|Enterprise]
CoinApi pricing plan (https://www.coinapi.io/market-data-api/pricing)
--thetadata-ws-url TEXT The ThetaData host address
--thetadata-rest-url TEXT The ThetaData host address
--thetadata-subscription-plan [Free|Value|Standard|Pro]
Your ThetaData subscription price plan
--terminal-link-connection-type [DAPI|SAPI]
Terminal Link Connection Type [DAPI, SAPI]
--terminal-link-server-auth-id TEXT
The Auth ID of the TerminalLink server
--terminal-link-environment [Production|Beta]
The environment to run in
--terminal-link-server-host TEXT
The host of the TerminalLink server
--terminal-link-server-port INTEGER
The port of the TerminalLink server
--terminal-link-openfigi-api-key TEXT
The Open FIGI API key to use for mapping options
--bybit-api-key TEXT Your Bybit API key
--bybit-api-secret TEXT Your Bybit API secret
--bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
Your Bybit VIP Level
--dataset TEXT The name of the dataset to download non-interactively
--overwrite Overwrite existing local data
-y, --yes Automatically confirm payment confirmation prompts
--data-type [Trade|Quote|OpenInterest]
Specify the type of historical data
--resolution [Tick|Second|Minute|Hour|Daily]
Specify the resolution of the historical data
--security-type [Equity|Index|Forex|Cfd|Future|Crypto|CryptoFuture|Option|IndexOption|Commodity|FutureOption]
Specify the security type of the historical data
--market TEXT Specify the market name for tickers (e.g., 'USA', 'NYMEX', 'Binance')
--tickers TEXT Specify comma separated list of tickers to use for historical data request.
--start-date TEXT Specify the start date for the historical data request in the format yyyyMMdd.
--end-date TEXT Specify the end date for the historical data request in the format yyyyMMdd. (defaults
to today)
--image TEXT The LEAN engine image to use (defaults to quantconnect/lean:latest)
--update Pull the LEAN engine image before running the Downloader Data Provider
--no-update Use the local LEAN engine image instead of pulling the latest version
--lean-config FILE The Lean configuration file that should be used (defaults to the nearest lean.json)
--verbose Enable debug logging
--help Show this message and exit.
```

_See code: [lean/commands/data/download.py](lean/commands/data/download.py)_
Expand Down
Loading
Loading