Dataset format¶

This notebook covers:

1. The simple option: pass a long-format DataFrame (or parquet file) directly to fev.
2. The data schema that fev expects.
3. The native format used internally by fev, and why we use it.
4. Formats that are not supported, and the reasoning.

In [1]:

import warnings
import datasets
import pandas as pd

import fev
import fev.utils

warnings.simplefilter("ignore")
datasets.disable_progress_bars()

import warnings import datasets import pandas as pd import fev import fev.utils warnings.simplefilter("ignore") datasets.disable_progress_bars()

The simple option: a long-format DataFrame¶

If your data is already in a long-format pandas DataFrame (one row per (id, timestamp) observation), you can use it with fev without any manual conversion.

Here is an example dataset:

In [2]:

df = pd.read_csv("https://autogluon.s3.us-west-2.amazonaws.com/datasets/timeseries/grocery_sales/merged.csv")
df.head()

df = pd.read_csv("https://autogluon.s3.us-west-2.amazonaws.com/datasets/timeseries/grocery_sales/merged.csv") df.head()

Out[2]:

	item_id	timestamp	scaled_price	promotion_email	promotion_homepage	unit_sales	product_code	product_category	product_subcategory	location_code
0	1062_101	2018-01-01	0.879130	0.0	0.0	636.0	1062	Beverages	Fruit Juice Mango	101
1	1062_101	2018-01-08	0.994517	0.0	0.0	123.0	1062	Beverages	Fruit Juice Mango	101
2	1062_101	2018-01-15	1.005513	0.0	0.0	391.0	1062	Beverages	Fruit Juice Mango	101
3	1062_101	2018-01-22	1.000000	0.0	0.0	339.0	1062	Beverages	Fruit Juice Mango	101
4	1062_101	2018-01-29	0.883309	0.0	0.0	661.0	1062	Beverages	Fruit Juice Mango	101

You have two ways to use a long-format DataFrame:

Option A — save to parquet and point fev.Task at the file. fev detects long-format data automatically and converts it on load.

df.to_parquet("data.parquet")

task = fev.Task(
    dataset_path="data.parquet",
    id_column="item_id",
    timestamp_column="timestamp",
    target="unit_sales",
    known_dynamic_columns=["scaled_price", "promotion_email"],
    static_columns=["product_code", "product_category", "product_subcategory", "location_code"],
    horizon=8,
)

Option B — convert in-memory with fev.utils.convert_long_df_to_hf_dataset (useful when you want to inspect, save, or push the converted dataset to the Hugging Face Hub):

In [3]:

ds = fev.utils.convert_long_df_to_hf_dataset(
    df,
    id_column="item_id",
    timestamp_column="timestamp",
    static_columns=["product_code", "product_category", "product_subcategory", "location_code"],
)
ds.features

ds = fev.utils.convert_long_df_to_hf_dataset( df, id_column="item_id", timestamp_column="timestamp", static_columns=["product_code", "product_category", "product_subcategory", "location_code"], ) ds.features

Out[3]:

{'item_id': Value('string'),
 'product_code': Value('int64'),
 'product_category': Value('string'),
 'product_subcategory': Value('string'),
 'location_code': Value('int64'),
 'timestamp': List(Value('timestamp[ns]')),
 'scaled_price': List(Value('float64')),
 'promotion_email': List(Value('float64')),
 'promotion_homepage': List(Value('float64')),
 'unit_sales': List(Value('float64'))}

You can save the converted dataset to a parquet file or push it to the Hugging Face Hub for reuse:

ds.to_parquet("data.parquet")
# or
ds.push_to_hub(repo_id=YOUR_REPO_ID, config_name=CONFIG_NAME)

To verify that a dataset is well-formed, use fev.utils.validate_time_series_dataset:

In [4]:

fev.utils.validate_time_series_dataset(ds, id_column="item_id", timestamp_column="timestamp")

fev.utils.validate_time_series_dataset(ds, id_column="item_id", timestamp_column="timestamp")

Data schema¶

fev expects time series datasets to follow this schema:

• each row represents a single (univariate or multivariate) time series
• each row contains
  1. a field of type Sequence(timestamp) with the timestamps of observations
  2. at least one field of type Sequence(float) that can be used as the target time series
  3. a field of type string with the unique ID of the time series
• all fields of type Sequence have the same length
• additional fields are allowed:
  • extra dynamic columns of type Sequence (covariates)
  • static features of type Value, Image, or any other Hugging Face Features type

The names of the ID, timestamp, and target fields are arbitrary and configured on the fev.Task.

Note that the dataset itself contains no information about the forecasting task. The same dataset can be reused across multiple tasks (e.g. different targets, horizons, or covariate setups) without data duplication.

When you load a long-format DataFrame (Option A or B above), fev produces a dataset matching this schema. Here is an example:

In [5]:

ds = datasets.load_dataset("autogluon/chronos_datasets", "monash_kdd_cup_2018", split="train")
ds.set_format("numpy")
ds.features

ds = datasets.load_dataset("autogluon/chronos_datasets", "monash_kdd_cup_2018", split="train") ds.set_format("numpy") ds.features

Out[5]:

{'id': Value('string'),
 'timestamp': List(Value('timestamp[ms]')),
 'target': List(Value('float64')),
 'city': Value('string'),
 'station': Value('string'),
 'measurement': Value('string')}

In [6]:

ds[0]

ds[0]

Out[6]:

{'id': np.str_('T000000'),
 'timestamp': array(['2017-01-01T14:00:00.000', '2017-01-01T15:00:00.000',
        '2017-01-01T16:00:00.000', ..., '2018-03-31T13:00:00.000',
        '2018-03-31T14:00:00.000', '2018-03-31T15:00:00.000'],
       dtype='datetime64[ms]'),
 'target': array([453., 417., 395., ..., 132., 158., 118.], dtype=float32),
 'city': np.str_('Beijing'),
 'station': np.str_('aotizhongxin_aq'),
 'measurement': np.str_('PM2.5')}

Why the native format?¶

fev stores datasets using the schema above (we'll call this the native format) rather than long-format DataFrames. The main reason is flexibility:

• Multimodal static features. Static metadata is stored once per series, with full type information from datasets.Features. This naturally accommodates non-tabular content like images, text documents, or embeddings — things that don't fit cleanly into a long-format DataFrame.
• Clean static/dynamic separation. Sequence vs Value types make it explicit which columns vary over time and which don't, with no need to duplicate static values across rows or maintain a separate file.
• Sharding-friendly. One time series per row means rows can be sharded freely without splitting a single series across files.
• No expensive groupby. Individual time series are accessed by row index, not by grouping the data on the fly.
• Future-proof. New feature types from the Hugging Face datasets ecosystem (audio, video, etc.) can be incorporated without changing the format.

What is not supported: the GluonTS format¶

The GluonTS format is another popular choice for storing time series data (e.g. used in LOTSA). Each entry follows a fixed schema:

{
    "start": "2024-01-01",
    "freq": "1D",
    "target": [0.5, 1.2, ...],
    "feat_dynamic_real": [[...]],
    "past_feat_dynamic_real": [[...]],
    "feat_static_cat": [...],
    "feat_static_real": [...]
}

fev does not support this format, for the following reasons:

• Hard-codes the task definition. The schema bakes in which columns are the target, which are known in the future, and which are only known in the past. Evaluating the same data under different assumptions (e.g. with vs. without future weather covariates) requires duplicating the dataset.
• Numeric-only. Multimodal static features (text, images) cannot be represented natively, even though they are increasingly relevant for time series tasks (Multimodal Time Series Analysis: A Tutorial and Survey).
• Tied to pandas frequency aliases. The freq field relies on pandas frequency strings, which have changed over time and are not stable across versions.

---

To convert a datasets.Dataset into formats consumed by other forecasting libraries, see 04-adapters.ipynb.