Different order of columns after TSDataset.to_pandas() and after TSDataset.__init__()

[Mr-Geekman]

Issue by GooseIt
Wednesday Feb 08, 2023 at 15:55 GMT
Originally opened as tinkoff-ai#1106

---

Description

🚀 Feature Request

It may be useful to impose the same order on both the return dataframe of TSDataset.to_dataset() and the dataframe df constructed during TSDataset.__init__() as the order imposed on the return dataframe of TSDataset.to_flatten() for the sake of consistency.
Current order of columns in both the return dataframe of TSDataset._to_dataset() and TSDataset.df places "target" along other features in alphabetical order, while order of columns in the return dataframe of TSDataset.to_flatten() places "target" after "timestamp" and "segment" and prior to other features in alphabetical order.
The order after TSDataset.to_flatten() makes observing "target" value more convenient (as it is not hidden among many other features) and emphasises its special role.

Proposal

I propose the following order of columns:

• timestamp,
• segment,
• target,
• other columns in alphabetical order.

How it can be done for TSDataset.to_dataset():

1. Find line df_copy = df_copy.pivot(index="timestamp", columns="segment") in etna.datasets.tsdataset.py
2. Prior to it reorder columns of df_copy in a way that puts "target" prior to other features, if said "target" is provided. It should look like feature_columns.remove("target") and in the next line df_copy = df_copy[["timestamp, "segment", "target"] + feature_columns]

How it can be done for TSDataset.__init__():

1. Find line df = pd.concat((df, self.df_exog), axis=1).loc[df.index].sort_index(axis=1, level=(0, 1)) in etna.datasets.tsdataset.py
2. Correct it in a way that puts "target" before other columns, still sorted in alphabetical order.

Test cases

1. Fix doctest of TSDataset.to_dataset().
2. Make sure current tests pass.
3. Add tests on order of columns for both modified methods to etna.tests.test_datasets.test_dataset.py:

• test_to_dataset_correct_column_order for TSDataset.to_dataset()
• test_init_with_exog_correct_column_order for TSDataset.__init__() with df_exog != None

Additional context

See issue tinkoff-ai#873 for similar issue for TSDataset.to_flatten()

It was converted from #25.

Related

• Not defined order of segments in `TSDataset` #369