For the complete documentation index, see llms.txt. This page is also available as Markdown.

🗃️Structure and files

This page describes how the Data Hub is physically organized on S3: prefix layout, file format, and segmentation.

Format

  • Files: JSONL (JSON Lines) — one JSON object per line, UTF-8 encoded.

  • Each line is an independent record; files are processed in streaming without loading everything into memory.

General layout

Storage is S3-compatible (Amazon S3 or DigitalOcean Spaces). The layout is identical in both; only the endpoint changes when connecting (see Data access).

Each dataset lives under its own prefix, usually with a snapshot in latest/ and, depending on the dataset, history by date.

s3://<listing-bucket>/<Family>/<dataset>/latest/<files>.jsonl

Examples of path patterns:

# Streaming Availability — stats by countryxplatform
s3://<listing-bucket>/SA/Stats/latest/
    <country>_<platform>.jsonl        

# Release dates
s3://<listing-bucket>/Sources/Nexus/ReleaseDate/latest/release_date.jsonl

Segmentation

Files are usually split by countryxplatform, so you can download only the subset you need by filtering on the prefix.

Dataset
Typical segmentation

Streaming Availability / Stats

one file per countryxplatform

Release dates (Nexus)

single file in latest/

Demand / Hits

by date (weekly)

Top Ten

by country and date (weekly)

Talent

split files in latest/ (weekly)

Querying with Athena

JSONL can be queried with Athena by creating an external table with a JSON SerDe pointing to the dataset prefix:

For large volumes and repeated queries it's worth converting the JSONL to your own columnar format (CTAS to Parquet) in the client's lake — see Consumption best practices.

Update frequency

The cadence at which each latest/ is refreshed depends on the dataset — see Update frequency. In general:

  • Presence / catalog / stats: daily and weekly.

  • Demand / Top Ten: weekly.

  • Prices: monthly / quarterly depending on the feed.


We hope you found this article helpful. If you have a question this article doesn't address, reach out on the Service Desk.

Last updated