Creating MetroFiles

This page explains how to add a new MetroFile to the Pymetropolis codebase. It assumes you are already familiar with the Pipeline (Steps, MetroFiles, Parameters) as a user of Pymetropolis.

A MetroFile is declared as a Python class, in src/pymetropolis/metro_pipeline/file.py for the base classes described below, or in a files.py module of the relevant package for everything else (e.g. src/pymetropolis/metro_demand/population/files.py). Every MetroFile subclass must declare a path (a string, relative to main_directory) and should set a human-readable description, used in the generated MetroFiles reference.

from pymetropolis.metro_pipeline.file import MetroTxtFile


class MyNewFile(MetroTxtFile):
    path = "my_package/my_file.txt"
    description = "A short, one-sentence description of what this file contains."

Once your class is defined, add it to the FILES list exported by your package’s __init__.py. This list is what feeds the generated MetroFiles reference page; a MetroFile class that a Step actually uses will still work correctly even if you forget this step, but it will not appear in the documentation.

File types

Never subclass MetroFile directly: pick the concrete base class that matches how your data should be serialized.

Each base class implements read()/write(value) (and, for MetroDataFrameFile, a lazy scan() returning a polars.LazyFrame) appropriately for its storage format — you do not need to implement serialization yourself. If none of these fit (e.g. a genuinely new file format), subclass MetroFile directly and override read/write; see their base implementations in file.py, which simply raise MetropyError("Unimplemented").

Schema

MetroDataFrameFile and MetroGeoDataFrameFile can declare a schema: list[Column], validated on every write(). Columns not listed in schema are dropped (with a warning) unless you set discard_extra_columns = False on the class. You can also cap the number of rows with max_rows.

from pymetropolis.metro_pipeline.file import Column, MetroDataFrameFile, MetroDataType


class TripsDistancesFile(MetroDataFrameFile):
    path = "demand/population/trips/distances.parquet"
    description = "Euclidean distance of each trip."
    schema = [
        Column(
            "trip_id",
            MetroDataType.ID,
            description="Identifier of the trip.",
            unique=True,
            nullable=False,
        ),
        Column(
            "od_distance",
            MetroDataType.FLOAT,
            description="Distance between origin and destination, in meters.",
            nullable=False,
        ),
    ]

If schema is left as None (the default), no validation is performed at all — this is acceptable for files whose columns are inherently dynamic, but a schema is strongly preferred whenever the set of columns is known ahead of time: it documents the file (in the generated reference) and catches mistakes in run() early.

Tip

MetroTxtFile/MetroPlotFile/MetroMLEstimatorFile do not support a schema — the Schema section below only applies to MetroDataFrameFile/MetroGeoDataFrameFile.

Columns

Column(name, dtype, optional=False, nullable=True, unique=False, description=None)

• name: the column name.
• dtype: a MetroDataType value (see below).
• optional: if True, the column may be entirely absent from the DataFrame. If False (the default), write() fails when the column is missing.
• nullable: whether the column is allowed to contain nulls. Only enforced when the column is present — an optional column that is absent is not checked against nullable.
• unique: whether every value in the column must be distinct. Also only enforced when present.
• description: a short, human-readable description, used in the generated reference.

Datatypes

MetroDataType is an enum; each value is validated differently depending on whether the file is a MetroDataFrameFile (checked against the polars dtype) or a MetroGeoDataFrameFile (checked against the pandas dtype, since geopandas is pandas-based) — Column handles this distinction for you.

Note

TIME and DURATION columns are only supported in MetroDataFrameFile (polars): geopandas has no native time-of-day or duration dtype, so these two values always fail validation on a MetroGeoDataFrameFile.

Population-specific files

If your file’s content is inherently specific to one demand population (see the Multiple-demand concept page) — for example, it is produced by a PopulationStep, or consumed by an all_populations=True input — mix in PopulationFile alongside your storage base class, with PopulationFile listed last:

from pymetropolis.metro_pipeline.file import Column, MetroDataFrameFile, MetroDataType, PopulationFile


class ToursFile(MetroDataFrameFile, PopulationFile):
    path = "demand/{population}/tours/tours.parquet"
    description = "Variables at the tour-level."
    schema = [
        Column("tour_id", MetroDataType.ID, description="Identifier of the tour.", nullable=False),
        # ...
    ]

The only requirement is that path contains a literal {population} placeholder. Pymetropolis substitutes it with the actual population name (or "population" for the main population) the first time the file is resolved for that population; forgetting the placeholder raises a MetropyError immediately, rather than letting every population silently collide on the same physical path.

Note

PopulationFile only affects where the file lives on disk. How Steps produce or consume a PopulationFile is covered on the Creating Steps with Parameters page.