Add read_<format> functions to the etna.datasets module for loading TSDataset directly from files.

[ArtemLiA]

Description

Hi! I suggest to add to the etna.datasets module functions read_<format> , where format = csv, excel and other popular file formats for storing tabular data, for reading files into TSDataset objects. What do you think about that?

Thoughts

It would be perfect to add support of all formats available in pandas. But as first three it seem to me optimal to add support of reading of csv, excel and sql formats.

Related

No response

[d-a-bunin]

Hi! I'm not sure that we should support 3 formats for loading data from disk and I'm not sure that these formats should be human-readable.

But I agree that we probably need 2 things:

• Ability to save dataset on disk using some format. We could think how to store all the inner components of a dataset into some kind of archive like we do for pipelines for example.
• Ability to load saved dataset from disk.

This format could be different from the format we use in load_datasets I suppose.

[ArtemLiA]

Do you suggest using two files to store TSDataset: the first one to store pandas dataframes, the second in json or pickle format for the rest of the object?

[d-a-bunin]

There are at least 3 pd.DataFrame attributes:

• raw_df
• df
• df_exog (it could also be None)
  They could be stored using pd.to_<format>.

I'm unsure how hierarchical_structure should be stored but in theory it is json-serializable.

It seems like every other attribute of TSDataset is json-serializable.

The most simple approach is to handle raw_df, df, df_exog using to_<format> and handle the rest using pickle. We already do smth like this with Pipeline: link. However, as a result we have binary format and there are some security concerns about pickle.

[ArtemLiA]

Do you want to store TSDataset object as several separated files or in a single archive/folder?

[d-a-bunin]

I suggest that this could be an archive that contains several files inside.

[ArtemLiA]

Let's implement the possibility of storing TSDataset objects in zip-archieve, where fields df and df_exog will be stored as two different files in any format supported by pandas (csv, sql, excel, e.t.c.) and other fields of object will be stored in json file. So, I suggest to implement the functions to_zip and read_zip, which will save and read TSDataset objects into above format respectively.

What do you think?