Models

This notebook covers the following topics:

Converting datasets.Dataset into other popular time series data formats.
Adding a wrapper for your model to fev/examples.
Submitting the results for your model to the fev-leaderboard.

Dataset adapters¶

Unfortunately, different time series forecasting libraries use very different data formats.

Luckily, fev comes with various adapters that make it easy to convert the data associated with each Task into an appropriate format for the different libraries.

In [1]:

Copied!





import fev

# Define a task with a mix of static & dynamic features
task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    horizon=30,
    target="price_mean",
    past_dynamic_columns=["distance_mean", "surge_mean"],
    known_dynamic_columns=["api_calls", "temp", "rain", "humidity", "clouds", "wind"],
    static_columns=["source_location", "provider_name", "provider_service"],
)
import fev

# Define a task with a mix of static & dynamic features
task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    horizon=30,
    target="price_mean",
    past_dynamic_columns=["distance_mean", "surge_mean"],
    known_dynamic_columns=["api_calls", "temp", "rain", "humidity", "clouds", "wind"],
    static_columns=["source_location", "provider_name", "provider_service"],
)

By default, window.get_input_data() returns two datasets.Dataset objects:

past_data contains all past data including target, timestamps, and covariates
future_data contains future values of timestamps and known covariates

In [2]:

Copied!





window = task.get_window(0)
past_data, future_data = window.get_input_data()
print(past_data)
print(future_data)
window = task.get_window(0)
past_data, future_data = window.get_input_data()
print(past_data)
print(future_data)

Dataset({
    features: ['id', 'timestamp', 'price_mean', 'api_calls', 'clouds', 'humidity', 'rain', 'temp', 'wind', 'distance_mean', 'surge_mean', 'provider_name', 'provider_service', 'source_location'],
    num_rows: 156
})
Dataset({
    features: ['id', 'timestamp', 'api_calls', 'clouds', 'humidity', 'rain', 'temp', 'wind', 'provider_name', 'provider_service', 'source_location'],
    num_rows: 156
})

You can use the fev.convert_input_data() method to convert the past & future data into formats expected by other frameworks.

Pandas¶

In [3]:

Copied!





from IPython.display import display

train_df, future_df, static_df = fev.convert_input_data(window, adapter="pandas")
print("train_df")
display(train_df.head())
print("future_df")
display(future_df.head())
print("static_df")
display(static_df.head())
from IPython.display import display

train_df, future_df, static_df = fev.convert_input_data(window, adapter="pandas")
print("train_df")
display(train_df.head())
print("future_df")
display(future_df.head())
print("static_df")
display(static_df.head())

train_df

	id	timestamp	price_mean	api_calls	clouds	humidity	temp	wind	distance_mean	surge_mean
0	T000000	2018-11-26 06:00:00	16.555555	9.0	0.990667	0.913333	40.627335	1.350667	1.726667	1.055556
1	T000000	2018-11-26 07:00:00	17.299999	10.0	0.970000	0.920000	41.137501	1.735000	1.690000	1.100000
2	T000000	2018-11-26 08:00:00	13.500000	1.0	0.980000	0.923333	40.919998	1.330000	1.380000	1.000000
3	T000000	2018-11-26 09:00:00	17.954546	11.0	1.000000	0.927500	40.937500	1.365000	1.920909	1.113636
4	T000000	2018-11-26 10:00:00	18.625000	12.0	0.995000	0.940000	40.695000	1.895000	2.122500	1.083333

future_df

	id	timestamp	api_calls	clouds	humidity	temp	wind
0	T000000	2018-12-17 13:00:00	10.0	0.97	0.90	35.169998	7.22
1	T000000	2018-12-17 14:00:00	7.0	0.92	0.90	36.299999	6.87
2	T000000	2018-12-17 15:00:00	13.0	0.88	0.87	37.250000	7.58
3	T000000	2018-12-17 16:00:00	12.0	1.00	0.84	39.000000	6.28
4	T000000	2018-12-17 17:00:00	9.0	0.95	0.81	40.009998	6.46

static_df

	id	provider_name	provider_service	source_location
0	T000000	Lyft	Lux	Back Bay
1	T000001	Lyft	Lux Black	Back Bay
2	T000002	Lyft	Lux Black XL	Back Bay
3	T000003	Lyft	Lyft	Back Bay
4	T000004	Lyft	Lyft XL	Back Bay

GluonTS¶

Data is stored in a PandasDataset.

The train_dataset contains only the historic data; the prediction_dataset additionally contains future values of the dynamic features.

In [4]:

Copied!





train_dataset, prediction_dataset = fev.convert_input_data(window, adapter="gluonts")
print("train_dataset")
print(train_dataset)
print("prediction_dataset")
print(prediction_dataset)
train_dataset, prediction_dataset = fev.convert_input_data(window, adapter="gluonts")
print("train_dataset")
print(train_dataset)
print("prediction_dataset")
print(prediction_dataset)

train_dataset
PandasDataset<size=156, freq=h, num_feat_dynamic_real=6, num_past_feat_dynamic_real=2, num_feat_static_real=0, num_feat_static_cat=3, static_cardinalities=[ 2. 13. 12.]>
prediction_dataset
PandasDataset<size=156, freq=h, num_feat_dynamic_real=6, num_past_feat_dynamic_real=2, num_feat_static_real=0, num_feat_static_cat=3, static_cardinalities=[ 2. 13. 12.]>

AutoGluon¶

Converts historic & future values to TimeSeriesDataFrame objects.

In [5]:

Copied!





train_df, known_covariates = fev.convert_input_data(window, adapter="autogluon")
print("train_df")
display(train_df)
print("train_df.static_features")
display(train_df.static_features)
print("known_covariates")
display(known_covariates)
train_df, known_covariates = fev.convert_input_data(window, adapter="autogluon")
print("train_df")
display(train_df)
print("train_df.static_features")
display(train_df.static_features)
print("known_covariates")
display(known_covariates)

train_df

		target	api_calls	clouds	humidity	rain	temp	wind	distance_mean	surge_mean
item_id	timestamp
T000000	2018-11-26 06:00:00	16.555555	9.0	0.990667	0.913333	0.000	40.627335	1.350667	1.726667	1.055556
	2018-11-26 07:00:00	17.299999	10.0	0.970000	0.920000	0.000	41.137501	1.735000	1.690000	1.100000
	2018-11-26 08:00:00	13.500000	1.0	0.980000	0.923333	0.000	40.919998	1.330000	1.380000	1.000000
	2018-11-26 09:00:00	17.954546	11.0	1.000000	0.927500	0.000	40.937500	1.365000	1.920909	1.113636
	2018-11-26 10:00:00	18.625000	12.0	0.995000	0.940000	0.000	40.695000	1.895000	2.122500	1.083333
...	...	...	...	...	...	...	...	...	...	...
T000155	2018-12-17 08:00:00	9.454545	11.0	1.000000	0.920000	0.000	37.279999	10.670000	2.230909	1.000000
	2018-12-17 09:00:00	9.700000	15.0	1.000000	0.930000	0.000	36.189999	9.760000	2.447333	1.000000
	2018-12-17 10:00:00	9.300000	10.0	1.000000	0.930000	0.003	34.750000	9.950000	2.203000	1.000000
	2018-12-17 11:00:00	9.400000	15.0	1.000000	0.930000	0.009	34.180000	9.240000	2.139333	1.000000
	2018-12-17 12:00:00	9.593750	16.0	0.990000	0.930000	0.000	34.209999	8.380000	1.958750	1.000000

79716 rows × 9 columns

train_df.static_features

	provider_name	provider_service	source_location
item_id
T000000	Lyft	Lux	Back Bay
T000001	Lyft	Lux Black	Back Bay
T000002	Lyft	Lux Black XL	Back Bay
T000003	Lyft	Lyft	Back Bay
T000004	Lyft	Lyft XL	Back Bay
...	...	...	...
T000151	Uber	Taxi	West End
T000152	Uber	UberPool	West End
T000153	Uber	UberX	West End
T000154	Uber	UberXL	West End
T000155	Uber	WAV	West End

156 rows × 3 columns

known_covariates

		api_calls	clouds	humidity	rain	temp	wind
item_id	timestamp
T000000	2018-12-17 13:00:00	10.0	0.97	0.90	0.0	35.169998	7.22
	2018-12-17 14:00:00	7.0	0.92	0.90	0.0	36.299999	6.87
	2018-12-17 15:00:00	13.0	0.88	0.87	0.0	37.250000	7.58
	2018-12-17 16:00:00	12.0	1.00	0.84	0.0	39.000000	6.28
	2018-12-17 17:00:00	9.0	0.95	0.81	0.0	40.009998	6.46
...	...	...	...	...	...	...	...
T000155	2018-12-18 14:00:00	17.0	0.48	0.47	0.0	26.190001	13.89
	2018-12-18 15:00:00	15.0	0.34	0.46	0.0	27.219999	15.03
	2018-12-18 16:00:00	15.0	0.31	0.47	0.0	28.700001	14.60
	2018-12-18 17:00:00	9.0	0.15	0.46	0.0	30.049999	13.55
	2018-12-18 18:00:00	12.0	0.00	0.46	0.0	30.790001	13.09

4680 rows × 6 columns

Nixtla¶

Similar to pandas, but ID, timestamp and target columns are renamed to unique_id, ds and y respectively.

In [6]:

Copied!





train_df, future_df, static_df = fev.convert_input_data(window, adapter="nixtla")
print("train_df")
display(train_df.head())
print("future_df")
display(future_df.head())
print("static_df")
display(static_df.head())
train_df, future_df, static_df = fev.convert_input_data(window, adapter="nixtla")
print("train_df")
display(train_df.head())
print("future_df")
display(future_df.head())
print("static_df")
display(static_df.head())

train_df

	unique_id	ds	y	api_calls	clouds	humidity	temp	wind	distance_mean	surge_mean
0	T000000	2018-11-26 06:00:00	16.555555	9.0	0.990667	0.913333	40.627335	1.350667	1.726667	1.055556
1	T000000	2018-11-26 07:00:00	17.299999	10.0	0.970000	0.920000	41.137501	1.735000	1.690000	1.100000
2	T000000	2018-11-26 08:00:00	13.500000	1.0	0.980000	0.923333	40.919998	1.330000	1.380000	1.000000
3	T000000	2018-11-26 09:00:00	17.954546	11.0	1.000000	0.927500	40.937500	1.365000	1.920909	1.113636
4	T000000	2018-11-26 10:00:00	18.625000	12.0	0.995000	0.940000	40.695000	1.895000	2.122500	1.083333

future_df

	unique_id	ds	api_calls	clouds	humidity	temp	wind
0	T000000	2018-12-17 13:00:00	10.0	0.97	0.90	35.169998	7.22
1	T000000	2018-12-17 14:00:00	7.0	0.92	0.90	36.299999	6.87
2	T000000	2018-12-17 15:00:00	13.0	0.88	0.87	37.250000	7.58
3	T000000	2018-12-17 16:00:00	12.0	1.00	0.84	39.000000	6.28
4	T000000	2018-12-17 17:00:00	9.0	0.95	0.81	40.009998	6.46

static_df

	unique_id	provider_name	provider_service	source_location
0	T000000	Lyft	Lux	Back Bay
1	T000001	Lyft	Lux Black	Back Bay
2	T000002	Lyft	Lux Black XL	Back Bay
3	T000003	Lyft	Lyft	Back Bay
4	T000004	Lyft	Lyft XL	Back Bay

Adding a wrapper for your model to fev/examples.¶

To add a wrapper for your library to fev/examples, you need to create a folder under fev/examples/{YOUR_MODEL_NAME} that contains:

A Python file evaluate_model.py that contains a method predict_with_model with signature

def predict_with_model(task: fev.Task, **kwargs) -> tuple[list[datasets.Dataset], float, dict]:
    """Returns model predictions, inference time and potentially extra information about the model."""
    ...

requirements.txt file containing the required dependencies for your model.

Defining the method predict_with_model is the most complex part of this process. We recommend looking at the implementations of some existing models to see how this can be done.

The only hard requirement for this method is that it should return a tuple consisting of 3 elements:

predictions (list[datasets.Dataset]) object containing the model predictions for each evaluation window.
inference_time (float) inference time of the model for the entire task (in seconds).
extra_info (dict | None) optional information about the model such as model configuration.

Predictions should follow the schema provided by task.predictions_schema.

Each entry of predictions must contain a list of length task.horizon

In [7]:

Copied!





task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
)
task.predictions_schema
task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
)
task.predictions_schema

Out[7]:

{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None)}

For a probabilistic forecasting task (if task.quantile_levels are provided), each entry of predictions must additionally contain the quantile forecasts. For example

In [8]:

Copied!





task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
    quantile_levels=[0.1, 0.5, 0.9],
)
task.predictions_schema
task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
    quantile_levels=[0.1, 0.5, 0.9],
)
task.predictions_schema

Out[8]:

{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.1': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.5': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.9': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None)}

The predictions cannot contain any missing values represented by NaN, otherwise an exception will be raised.

Other than what's described above, there are no hard restrictions on how the predict_with_model method needs to be implemented. For example, it's completely up to you whether the method uses any datasets columns except the target or how the data is preprocessed.

Still, here is some general advice:

If your model is capable of generating probabilistic forecasts, make sure that you correct the "optimal" forecast for the task.eval_metric. For example, metrics like "MSE" or "RMSSE", the mean forecast is preferred, while metrics like "MASE" are optimized by the median forecast.
Use fev.convert_input_data() to take advantage of the adapters and reduce the boilerplate preprocessing code.
Make sure that your wrapper can deal with missing values (or at least imputes them before passing the data to your model).
Make sure that your wrapper takes advantage of the extra features of the task. For example, the following attributes might be useful:

In [9]:

Copied!





print(f"{task.static_columns=}")
print(f"{task.dynamic_columns=}")
print(f"{task.known_dynamic_columns=}")
print(f"{task.past_dynamic_columns=}")
# Attributes available after `task.load_full_dataset` is called
task.load_full_dataset()
print(f"{task.freq=}")
print(f"{task.static_columns=}")
print(f"{task.dynamic_columns=}")
print(f"{task.known_dynamic_columns=}")
print(f"{task.past_dynamic_columns=}")
# Attributes available after `task.load_full_dataset` is called
task.load_full_dataset()
print(f"{task.freq=}")

task.static_columns=[]
task.dynamic_columns=[]
task.known_dynamic_columns=[]
task.past_dynamic_columns=[]
task.freq='h'

Submitting the results for your model to the fev-leaderboard ¶

After you've implemented the wrapper for your model in fev/examples, complete the following steps:

Fork autogluon/fev and clone your fork to your machine.
Implement your model's wrapper in fev/examples.
Run the model on all tasks from the benchmark and save the results to fev/benchmarks/chronos_zeroshot/results/{model_name}.csv.
Open a pull request to autogluon/fev containing the following files:
- fev/examples/{model_name}/evaluate_model.py
- fev/examples/{model_name}/requirements.txt
- fev/benchmarks/chronos_zeroshot/results/{model_name}.csv
We will independently reproduce the results using the code you provided and add the results to the leaderboard.

# Example code from fev/examples/my_amazing_model/evaluate_model.py

def predict_with_model(task: fev.Task, **kwargs) -> tuple[list[datasets.DatasetDict], float, dict]:
    """Wrapper for my_amazing_model"""
    ...
    return predictions_per_window, inference_time, extra_info

if __name__ == "__main__":
    model_name = "my_amazing_model"
    benchmark = fev.Benchmark.from_yaml(
        "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/tasks.yaml"
    )

    summaries = []
    for task in benchmark.tasks:
        predictions_per_window, inference_time, extra_info = predict_with_model(task)
        evaluation_summary = task.evaluation_summary(
            predictions_per_window,
            model_name=model_name,
            inference_time_s=inference_time,
            extra_info=extra_info,
            trained_on_this_dataset=True,  # True if model has seen this dataset during training, False otherwise. Please try to be honest!
        )
        summaries.append(evaluation_summary)

    summary_df = pd.DataFrame(summaries)
    print(summary_df)
    summary_df.to_csv(f"{model_name}.csv", index=False)