🗃️Structure and files
This page describes how the Data Hub is physically organized on S3: prefix layout, file format, and segmentation.
Format
Files: JSONL (JSON Lines) — one JSON object per line, UTF-8 encoded.
Each line is an independent record; files are processed in streaming without loading everything into memory.
General layout
Storage is S3-compatible (Amazon S3 or DigitalOcean Spaces). The layout is identical in both; only the endpoint changes when connecting (see Data access).
Each dataset lives under its own prefix, usually with a snapshot in latest/ and, depending on the dataset, history by date.
s3://<listing-bucket>/<Family>/<dataset>/latest/<files>.jsonlExamples of path patterns:
# Streaming Availability — stats by countryxplatform
s3://<listing-bucket>/SA/Stats/latest/
<country>_<platform>.jsonl
# Release dates
s3://<listing-bucket>/Sources/Nexus/ReleaseDate/latest/release_date.jsonlSegmentation
Files are usually split by countryxplatform, so you can download only the subset you need by filtering on the prefix.
Streaming Availability / Stats
one file per countryxplatform
Release dates (Nexus)
single file in latest/
Demand / Hits
by date (weekly)
Top Ten
by country and date (weekly)
Talent
split files in latest/ (weekly)
Querying with Athena
JSONL can be queried with Athena by creating an external table with a JSON SerDe pointing to the dataset prefix:
For large volumes and repeated queries it's worth converting the JSONL to your own columnar format (CTAS to Parquet) in the client's lake — see Consumption best practices.
Update frequency
The cadence at which each latest/ is refreshed depends on the dataset — see Update frequency. In general:
Presence / catalog / stats: daily and weekly.
Demand / Top Ten: weekly.
Prices: monthly / quarterly depending on the feed.
We hope you found this article helpful. If you have a question this article doesn't address, reach out on the Service Desk.
Last updated

