Predicting with Time-LLM using GPT2 has awful performance

[odinhorizontal]

I follow the script in https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac to Predicting with Time-LLM using GPT2.
My code is exactly the same as the script. But got the terrible result.

My result:

script:
https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac

code:

import time
import numpy as np
import pandas as pd
import pytorch_lightning as pl
import matplotlib.pyplot as plt

from neuralforecast import NeuralForecast
from neuralforecast.models import TimeLLM
from neuralforecast.losses.pytorch import MAE
from neuralforecast.tsdataset import TimeSeriesDataset
from neuralforecast.utils import AirPassengers, AirPassengersPanel, AirPassengersStatic, augment_calendar_df

from transformers import GPT2Config, GPT2Model, GPT2Tokenizer

AirPassengersPanel, calendar_cols = augment_calendar_df(df=AirPassengersPanel, freq='M')

Y_train_df = AirPassengersPanel[AirPassengersPanel.ds<AirPassengersPanel['ds'].values[-12]]
Y_test_df = AirPassengersPanel[AirPassengersPanel.ds>=AirPassengersPanel['ds'].values[-12]].reset_index(drop=True)

gpt2_config = GPT2Config.from_pretrained('openai-community/gpt2')
gpt2 = GPT2Model.from_pretrained('openai-community/gpt2',config=gpt2_config)
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('openai-community/gpt2')

prompt_prefix = "The dataset contains data on monthly air passengers. There is a yearly seasonality"

timellm = TimeLLM(h=12,
                 input_size=36,
                 llm=gpt2,
                 llm_config=gpt2_config,
                 llm_tokenizer=gpt2_tokenizer,
                 prompt_prefix=prompt_prefix,
                 max_steps=100,
                 batch_size=24,
                 windows_batch_size=24)

nf = NeuralForecast(
    models=[timellm],
    freq='M'
)

nf.fit(df=Y_train_df, val_size=12)
forecasts = nf.predict(futr_df=Y_test_df)

[odinhorizontal]

Can somebody tell me why my code has such awful outcome? qaq

[odinhorizontal]

Really thank you for your reply! I also feel that this project is indeed difficult to replicate the results reported in the paper. I'm wondering if it's because the authors haven't disclosed some of their tricks or techniques. But my issue #161 is an easy work，I 've got such poor result. It doesn't feel like it should be this way. Have you tried the performance of timellm on simple tasks? How did it perform?

…

---- Replied Message ---- | From | ***@***.***> | | Date | 11/27/2024 11:08 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [KimMeen/Time-LLM] Predicting with Time-LLM using GPT2 has awful performance (Issue #161) | #109 related issue. It is observed that their results are worse than reported in the paper. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[mgaral]

您好，我也遇到了这个问题，在我的数据集上预测效果很差，甚至不如lstm，请问您解决这个问题了吗

[odinhorizontal]

您好，我也遇到了这个问题，在我的数据集上预测效果很差，甚至不如lstm，请问您解决这个问题了吗

我使用NeuralForecast中集成的timellm预测自己的数据时效果很差，但在这个github项目上修改data_factory.py和data_loader.py后，应用自己的数据集效果还可以，不知道为什么。