Poor model performance

[AFilipaVEL]

Hi again :)

I'm fitting AutoETS to a bunch of series. My trainning dataframe is a Spark dataframe with 12 partitions (the number of workers from my cluster), each partition contains about 92k unique_id's. When i execute the code (through a display command) it takes forever to see the results. Any idea on how to improve the performance? Here's my code for reference:

d_train = df\
    .select('unique_id', 'ds', 'y')\
    .repartition(12,'unique_id')

models = [AutoETS()] 

fcst = StatsForecast(models=models,
                     freq='M',
                     n_jobs=1)

pred_sup = fcst.forecast(df = d_train, 
                         h=1,
                         fitted = True, 
                         level=[90])

pred_sup.display()

Thanks in advance!

[jmoralez]

Hey. I suggest saving the forecast result instead of displaying it because if you then want to save it you'll have to run it again. About the time you have to keep in mind that a model is trained for each serie, so if it took 1s to train one model it'd then take 92k seconds to train all models in a partition, which is around 25 hours. If you have that many series I'd suggest getting a bigger cluster to better distribute the work, in terms of $ it'll be about the same because the compute time will be the same but you won't have to wait that much.

[AFilipaVEL]

OK... once again, thank you very much!