> For the complete documentation index, see [llms.txt](https://knowledgebase.fabricdata.com/insights/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://knowledgebase.fabricdata.com/insights/data-hub/consumption-best-practices.md).

# Consumption best practices

Recommendations for consuming the S3 Data Hub (**JSONL** files) efficiently.

## Download only the subset you need

Files are segmented by prefix (platform×country and/or date — see [Structure and files](/insights/data-hub/structure-and-files.md)). Filter by prefix when listing/downloading so you don't pull the whole dataset:

```bash
# Only Netflix US from the current snapshot
aws s3 sync s3://<bucket>/SA/Stats/latest/ ./datos/ \
  --exclude "*" --include "us_netflix.jsonl"

# DigitalOcean Spaces: add the S3-compatible endpoint
aws s3 sync s3://<space>/SA/Stats/latest/ ./datos/ \
  --endpoint-url https://<region>.digitaloceanspaces.com \
  --exclude "*" --include "us_netflix.jsonl"
```

## Read JSONL in streaming

Each line is an independent JSON object, so you can process it line by line without loading the whole file:

```python
import json
with open("us_netflix.jsonl") as f:
    for line in f:
        rec = json.loads(line)
        ...
```

```python
# pandas
import pandas as pd
df = pd.read_json("us_netflix.jsonl", lines=True)
```

```python
# Spark / Databricks
df = spark.read.json("s3://<bucket>/SA/Stats/latest/")
```

## For repeated analysis, convert to columnar

JSONL is row-based: every query reads the full object. If you're going to query it a lot, **convert it once to Parquet** in your own lake/warehouse (CTAS in Athena, or `df.write.parquet(...)` in Spark) and query from there. It's much cheaper and faster.

## Join datasets by UID

The UID is the key used to link presence, metadata, demand, and awards for the same title. After loading the JSONL files into tables, the join can be performed directly using that field.

```sql
-- 1. Join content by UID: presence + metadata
SELECT m.title, m.genres, p.platform_id, p.packages_type
FROM content_presence p
JOIN content_metadata m USING (uid)
WHERE p.platform_country = 'US'
  AND p.out_on IS NULL;   
```

## Handling arrays and nested objects

Fields like `genres`, `directors`, `cast`, or `deeplinks` may come as **arrays or nested JSON objects**. When consuming:

* In Athena with JSON SerDe, declare the field as `array<string>` or `struct<...>` as appropriate.
* In Spark/pandas, normalize (`explode` / `json_normalize`) if you need one row per element.

## Current presence vs. historical

* **Current:** `exit_date` empty/null (the title is still in the catalog).
* **Historical / gaps:** use `entry_date` and `exit_date` to reconstruct availability windows.

## Data freshness

The Data Hub is updated in batches depending on the dataset (see [Update frequency](/insights/origin-insights-dashboards/general-reference/frequency.md)). The `latest/` prefix always holds the latest snapshot; it's not a real-time feed.

## What not to expect

* It's not real time: latency depends on the batch (daily/weekly/monthly).
* It doesn't include audience or individual user consumption (see [Data coverage](/insights/origin-insights-dashboards/general-reference/data-coverage.md)).

***

<details>

<summary>Related Articles</summary>

* [Getting started](/insights/data-hub/getting-started.md)
* [Data access](/insights/data-hub/data-access.md)
* [Structure and files](/insights/data-hub/structure-and-files.md)

</details>

> We hope you found this article helpful. If you have a question this article doesn't address, reach out on the [Service Desk](https://fabric.atlassian.net/servicedesk/customer/portal/336).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://knowledgebase.fabricdata.com/insights/data-hub/consumption-best-practices.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.