Skip to content
PMetal

pmetal dataset

Utilities for working with training datasets.

Subcommands

Section titled “Subcommands”

analyze

Section titled “analyze”

Analyze a local dataset — shows format, row count, token statistics, and column info.

Terminal window
pmetal dataset analyze train.jsonl
pmetal dataset analyze data.parquet

download

Section titled “download”

Download a dataset from HuggingFace Hub.

Terminal window
pmetal dataset download squad --output ./data/

convert

Section titled “convert”

Convert between dataset formats.

Terminal window
# Parquet to JSONL
pmetal dataset convert data.parquet --format jsonl --output data.jsonl


# ShareGPT to Alpaca
pmetal dataset convert sharegpt.json --format alpaca --output alpaca.jsonl

Supported Formats

Section titled “Supported Formats”
FormatExtensionsReadWrite
JSONL.jsonlYesYes
JSON.jsonYesYes
Parquet.parquetYesYes
CSV.csvYesYes

See Also

Section titled “See Also”