Models

This notebook covers the following topics:

1. Adding a wrapper for your model to fev/examples.
2. Submitting the results for your model to the fev-leaderboard.

Adding a wrapper for your model to fev/examples.¶

To add a wrapper for your library to fev/examples, you need to create a folder under fev/examples/{YOUR_MODEL_NAME} that contains:

• A Python file evaluate_model.py that contains a method predict_with_model with signature

  def predict_with_model(task: fev.Task, **kwargs) -> tuple[list[datasets.DatasetDict], float, dict]:
      """Returns model predictions, inference time and potentially extra information about the model."""
      ...
  

• requirements.txt file containing the required dependencies for your model.

Defining the method predict_with_model is the most complex part of this process. We recommend looking at the implementations of some existing models to see how this can be done.

The only hard requirement for this method is that it should return a tuple consisting of 3 elements:

1. predictions (list[datasets.DatasetDict]) object containing the model predictions for each evaluation window.
2. inference_time (float) inference time of the model for the entire task (in seconds).
3. extra_info (dict | None) optional information about the model such as model configuration.

Predictions should follow the schema provided by task.predictions_schema.

Each entry of predictions must contain a list of length task.horizon

In [1]:

import fev

task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
)
task.predictions_schema

import fev task = fev.Task( dataset_path="autogluon/chronos_datasets", dataset_config="monash_rideshare", target="price_mean", horizon=30, ) task.predictions_schema

/home/shchuro/envs/fev/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Out[1]:

{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None)}

For a probabilistic forecasting task (if task.quantile_levels are provided), each entry of predictions must additionally contain the quantile forecasts. For example

In [9]:

task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="monash_rideshare",
    target="price_mean",
    horizon=30,
    quantile_levels=[0.1, 0.5, 0.9],
)
task.predictions_schema

task = fev.Task( dataset_path="autogluon/chronos_datasets", dataset_config="monash_rideshare", target="price_mean", horizon=30, quantile_levels=[0.1, 0.5, 0.9], ) task.predictions_schema

Out[9]:

{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.1': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.5': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None),
 '0.9': Sequence(feature=Value(dtype='float64', id=None), length=30, id=None)}

The predictions cannot contain any missing values represented by NaN, otherwise an exception will be raised.

Other than what's described above, there are no hard restrictions on how the predict_with_model method needs to be implemented. For example, it's completely up to you whether the method uses any datasets columns except the target or how the data is preprocessed.

Still, here is some general advice:

• If your model is capable of generating probabilistic forecasts, make sure that you correct the "optimal" forecast for the task.eval_metric. For example, metrics like "MSE" or "RMSSE", the mean forecast is preferred, while metrics like "MASE" are optimized by the median forecast.
• Use fev.convert_input_data() to take advantage of the adapters and reduce the boilerplate preprocessing code.
• Make sure that your wrapper can deal with missing values (or at least imputes them before passing the data to your model).
• Make sure that your wrapper takes advantage of the extra features of the task. For example, the following attributes might be useful:

In [10]:

print(f"{task.static_columns=}")
print(f"{task.dynamic_columns=}")
print(f"{task.known_dynamic_columns=}")
print(f"{task.past_dynamic_columns=}")
# Attributes available after `task.load_full_dataset` is called
task.load_full_dataset()
print(f"{task.freq=}")

print(f"{task.static_columns=}") print(f"{task.dynamic_columns=}") print(f"{task.known_dynamic_columns=}") print(f"{task.past_dynamic_columns=}") # Attributes available after `task.load_full_dataset` is called task.load_full_dataset() print(f"{task.freq=}")

task.static_columns=[]
task.dynamic_columns=[]
task.known_dynamic_columns=[]
task.past_dynamic_columns=[]
task.freq='h'

Submitting the results for your model to the fev-leaderboard¶

After you've implemented the wrapper for your model in fev/examples, complete the following steps:

1. Fork autogluon/fev and clone your fork to your machine.
2. Implement your model's wrapper in fev/examples.
3. Run the model on all tasks from the benchmark and save the results to fev/benchmarks/fev_bench/results/{model_name}.csv.
4. Open a pull request to autogluon/fev containing the following files:
   • fev/examples/{model_name}/evaluate_model.py
   • fev/examples/{model_name}/requirements.txt
   • fev/benchmarks/fev_bench/results/{model_name}.csv
5. We will independently reproduce the results using the code you provided and add the results to the leaderboard.

# Example code from fev/examples/my_amazing_model/evaluate_model.py

def predict_with_model(task: fev.Task, **kwargs) -> tuple[list[datasets.DatasetDict], float, dict]:
    """Wrapper for my_amazing_model"""
    ...
    return predictions_per_window, inference_time, extra_info

if __name__ == "__main__":
    model_name = "my_amazing_model"
    benchmark = fev.Benchmark.from_yaml(
        "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/fev_bench/tasks.yaml"
    )

    summaries = []
    for task in benchmark.tasks:
        predictions_per_window, inference_time, extra_info = predict_with_model(task)
        evaluation_summary = task.evaluation_summary(
            predictions_per_window,
            model_name=model_name,
            inference_time_s=inference_time,
            extra_info=extra_info,
            trained_on_this_dataset=True,  # True if model has seen this dataset during training, False otherwise. Please try to be honest!
        )
        summaries.append(evaluation_summary)

    summary_df = pd.DataFrame(summaries)
    print(summary_df)
    summary_df.to_csv(f"{model_name}.csv", index=False)