Common data formats are CSV, JSON, XML, and ZIP but you can use any file type that can be read over the internet. For Excel files, double check the raw data format for parsing in the data reader, since data will be formatted for convenient visualization in Excel application view. To avoid confusion of data format, save the spreadsheet as a CSV file and open it in a text editor to confirm the raw data format.
-The data in the file must be in chronological order. If you import from a remote file provider, each request has a one-second overhead, so package your custom data to minimize requests. Bundle dates together where possible to speed up execution. The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution.
+If you import from a remote file provider, each request has a one-second overhead, so package your custom data to minimize requests. Bundle dates together where possible to speed up execution. The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution.
diff --git a/03 Writing Algorithms/16 Importing Data/02 Streaming Data/01 Key Concepts/06 Unsorted Data.html b/03 Writing Algorithms/16 Importing Data/02 Streaming Data/01 Key Concepts/06 Unsorted Data.html new file mode 100644 index 0000000000..d2a7058ae9 --- /dev/null +++ b/03 Writing Algorithms/16 Importing Data/02 Streaming Data/01 Key Concepts/06 Unsorted Data.html @@ -0,0 +1,34 @@ +
+ By default, LEAN expects the data in chronological order. If the data is unsorted, set the Sortsort property of the SubscriptionDataSource object to trueTrue.
+
public class MyCustomDataType : BaseData
+{
+ public override SubscriptionDataSource GetSource(
+ SubscriptionDataConfig config,
+ DateTime date,
+ bool isLiveMode)
+ {
+ var source = $"http://my-ftp-server.com/{config.Symbol.Value}/{date:yyyyMMdd}.csv";
+ return new SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile)
+ {
+ Sort = true
+ };
+ }
+}
+ class MyCustomDataType(PythonData):
+ def get_source(self,
+ config: SubscriptionDataConfig,
+ date: datetime,
+ is_live_mode: bool) -> SubscriptionDataSource:
+
+ source = f"http://my-ftp-server.com/{config.symbol.value}/{date:%Y%M%d}.csv"
+ subscription = SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)
+ subscription.sort = True
+ return subscription
+
+ LEAN uses EndTimeend_time property of your custom data to sort it.
+