Importing Unsorted Data

03 Writing Algorithms/16 Importing Data/02 Streaming Data/01 Key Concepts/02 Data Formats.html

@@ -1,3 +1,3 @@
 <p>Common data formats are <span class="public-file-name">CSV</span>, <span class="public-file-name">JSON</span>, <span class="public-file-name">XML</span>, and <span class="public-file-name">ZIP</span> but you can use any file type that can be read over the internet. For Excel files, double check the raw data format for parsing in the data reader, since data will be formatted for convenient visualization in Excel application view. To avoid confusion of data format, save the spreadsheet as a <span class="public-file-name">CSV</span> file and open it in a text editor to confirm the raw data format.</p>
 
-<p>The data in the file must be in chronological order. If you import from a remote <a href='/docs/v2/writing-algorithms/importing-data/key-concepts#02-File-Providers'>file provider</a>, each request has a one-second overhead, so package your custom data to minimize requests. Bundle dates together where possible to speed up execution. The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution.</p>
+<p>If you import from a remote <a href='/docs/v2/writing-algorithms/importing-data/key-concepts#02-File-Providers'>file provider</a>, each request has a one-second overhead, so package your custom data to minimize requests. Bundle dates together where possible to speed up execution. The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution.</p>

03 Writing Algorithms/16 Importing Data/02 Streaming Data/01 Key Concepts/06 Unsorted Data.html

@@ -0,0 +1,34 @@
+<p>
+    By default, LEAN expects the data in chronological order. If the data is unsorted, set the <code class="csharp">Sort</code><code class="python">sort</code> property of the <code>SubscriptionDataSource</code> object to <code class="csharp">true</code><code class="python">True</code>.
+</p>
+
+<div class="section-example-container">
+    <pre class="csharp">public class MyCustomDataType : BaseData
+{
+    public override SubscriptionDataSource GetSource(
+        SubscriptionDataConfig config,
+        DateTime date,
+        bool isLiveMode)
+    {
+        var source = $"http://my-ftp-server.com/{config.Symbol.Value}/{date:yyyyMMdd}.csv";
+        return new SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile)
+        { 
+            Sort = true
+        };
+    }
+}</pre>
+    <pre class="python">class MyCustomDataType(PythonData):
+    def get_source(self,
+            config: SubscriptionDataConfig,
+            date: datetime,
+            is_live_mode: bool) -&gt; SubscriptionDataSource:
+
+        source = f"http://my-ftp-server.com/{config.symbol.value}/{date:%Y%M%d}.csv"
+        subscription = SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)
+        subscription.sort = True
+        return subscription</pre>
+</div>
+
+<p>
+    LEAN uses <code class="csharp">EndTime</code><code class="python">end_time</code> property of your custom data to sort it.
+</p>

03 Writing Algorithms/16 Importing Data/02 Streaming Data/02 Custom Securities/02 CSV Format Example/99 Demonstration Algorithms.html

@@ -7,10 +7,18 @@
         CustomDataNIFTYAlgorithm.py <span class="badge-python pull-right">Python</span>
     </a>
 
+    <a class="python example-algorithm-link" href="https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/DescendingCustomDataObjectStoreRegressionAlgorithm.py" target="_BLANK">
+        DescendingCustomDataObjectStoreRegressionAlgorithm.py <span class="badge-python pull-right">Python</span>
+    </a>
+
     <a class="csharp example-algorithm-link" href="https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/BubbleAlgorithm.cs" target="_BLANK">
         BubbleAlgorithm.cs <span class="badge-csharp pull-right">C#</span>
     </a>
     <a class="csharp example-algorithm-link" href="https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/CustomDataNIFTYAlgorithm.cs" target="_BLANK">
         CustomDataNIFTYAlgorithm.cs <span class="badge-csharp pull-right">C#</span>
     </a>
+
+    <a class="csharp example-algorithm-link" href="https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/DescendingCustomDataObjectStoreRegressionAlgorithm.cs" target="_BLANK">
+        DescendingCustomDataObjectStoreRegressionAlgorithm.py <span class="badge-csharp pull-right">Python</span>
+    </a>
 </div>