> For the complete documentation index, see [llms.txt](https://knowledgebase.fabricdata.com/fabric-knowledge-base/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://knowledgebase.fabricdata.com/fabric-knowledge-base/support/studio-onboarding-process/best-practices-for-large-databases-migration.md).

# Best Practices for Large Databases Migration

One of the most significant challenges during Fabric Studio implementations is onboarding clients with large and complex databases. These clients often bring millions of records, deep hierarchies, or massive attribute sets — all of which can make the migration process complex and time-consuming.

Over the years, Fabric Studio has successfully onboarded clients across the full data spectrum — from small catalogs to enterprise-level datasets containing tens of millions of records. Through that experience, we’ve developed a clear and proven framework and a set of best practices for managing large-scale migrations efficiently and reliably.

## Understanding the Challenge

Large and complex databases introduce challenges beyond just data volume. The complexity typically stems from two key factors:

1. **Scale:** Client has millions of records that must be migrated into Fabric Studio.
2. **Structure:** Each record may have dozens or even hundreds of attributes, relationships, and dependencies.

Together, these increase processing demands and elevate the risk of data quality or linkage issues if not carefully managed. Fabric Studio’s approach focuses on scalability, accuracy, and control — ensuring each stage of the migration is validated and optimized for performance.

## 1️⃣ Preparation and Readiness: Laying the Groundwork

Before any data migration begins, **preparation** is the most critical success factor.\
For large datasets, upfront planning and setup directly influence the performance, accuracy, and stability of the entire process.

The following preparatory steps are essential to ensure migration stability and data fidelity:

<table><thead><tr><th width="233.49798583984375">Activity</th><th>Description</th></tr></thead><tbody><tr><td><strong>Field Configuration</strong></td><td>Defined collaboratively between the Fabric Professional Services team and the client based on discovery sessions. Ensures the client’s structure has been mapped to the Fabric Studio data model accurately.</td></tr><tr><td><strong>Data Extraction and Quality Checks</strong></td><td>The client provides their extracted dataset, which is then validated against Fabric Studio’s Record Builder rules. This ensures compliance with expected field types, formats, and structures before ingestion.</td></tr><tr><td><strong>Data Mapping</strong></td><td>Connecting the client’s source data structure with Fabric Studio’s destination model. This step ensures that every source field is correctly aligned to a Fabric Studio's field, maintaining relational integrity and consistency during ingestion.</td></tr><tr><td><strong>Controlled Vocabulary (</strong>or we call it <strong>Managed list</strong>!<strong>) Setup</strong></td><td>Pre-loading the entire catalog, dropdown values and controlled lists prevents ingestion errors due to missing or mismatched vocabularies.</td></tr><tr><td><strong>Environment Readiness</strong></td><td>Ensures the Fabric Studio ingest environment is properly configured with sufficient concurrency and performance capacity for large data volumes.</td></tr><tr><td><strong>Stakeholder Alignment</strong></td><td>Establishes clear ownership, communication channels, and approval points across the Fabric Studio and client teams.</td></tr></tbody></table>

> 🧠 **Tip:** In large migrations, more than half the success depends on preparation. Investing time here reduces rework, improves ingest efficiency, and minimizes post-load cleanup.

## 2️⃣ Test Load and Validation

The test load is a crucial checkpoint before the main migration. A small, representative sample of records into the target environment to validate key aspects of the migration process.

During the test load, the focus is on verifying:

* **Field mappings and data relationships:** Confirm that source fields are correctly mapped to target fields and that all relational dependencies (such as parent-child or lookup relationships) are maintained.
* **Record count accuracy:** Ensure the number of records migrated matches expectations for the sample set, indicating that filters, joins, and transformations are functioning correctly.
* **Data integrity and transformation logic:** Validate that data values are accurate, complete, and properly transformed according to the defined business rules.

## 3️⃣ Migration Execution Framework: Structured and Sequential

For large and complex datasets, Fabric Professional Service's team uses a phased, dependency-aware migration framework. Rather than loading everything at once, data is ingested in logical, hierarchical stages that allow validation between each layer.

The below is the example of how it looks like:

<table><thead><tr><th width="118.99667358398438">Orde</th><th>Focus Area</th><th>Objective</th></tr></thead><tbody><tr><td><strong>1</strong></td><td>Load <em>Show</em> (highest-level parent records)</td><td>Establish parent-level structure for dependent records.</td></tr><tr><td><strong>2</strong></td><td>Validate <em>Show</em>, Load <em>Features</em> (standalone records)</td><td>Confirm parent data and begin loading independent categories.</td></tr><tr><td><strong>3</strong></td><td>Load <em>Season</em>, Validate <em>Features</em></td><td>Add hierarchical depth while confirming previous loads.</td></tr><tr><td><strong>4</strong></td><td>Validate <em>Season</em>, Load <em>Special Categories</em></td><td>Reinforce structure and add secondary or optional data types.</td></tr><tr><td><strong>5</strong></td><td>Load <em>Episodes</em>, Cross-Check Categories</td><td>Final ingestion of dependent-level data with linkage checks.</td></tr></tbody></table>

### Scaling the Data Load: Performance and Concurrency

A critical success factor for large-scale migrations is managing **throughput** — the rate at which data can be processed and loaded into Fabric Studio.\
To optimize speed without sacrificing integrity, Fabric Studio leverages **Concurrent API Workers (C)** to run multiple ingestion threads in parallel.

For example, in a scenario involving **5 million records**:

| Metric                                   | Value        | Unit            |
| ---------------------------------------- | ------------ | --------------- |
| Total Number of Records (N)              | 5,000,000    | Records         |
| Estimated Processing Time per Record (T) | 5            | Seconds/Record  |
| Concurrent API Workers (C)               | 60           | Workers         |
| **Estimated Bulk Load Duration**         | **4–5 Days** | Processing Time |

> 🕐 **Note:** The above projection covers only the **data ingestion window**. Preparatory steps such as configuration, mapping, and validation occur before ingestion and extend the total project timeline.

## :four: Post-Migration Activities: Validating Success

Once all records have been ingested, **post-migration activities** confirm the completeness, accuracy, and integrity of the data.\
This phase typically spans **2–3 days** and involves close collaboration between Fabric Studio and the client.

| Activity                             | Description                                                                                          |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| **Reconciliation and Validation**    | Verify record counts, field-level completeness, and linkage accuracy between source and destination. |
| **Defect Remediation**               | Identify and correct mismatched or failed records.                                                   |
| **Validation Reports**               | Generate reports summarizing data completeness and ingestion metrics.                                |
| **Final Client Review and Sign-Off** | Present validation outcomes to the client for sign-off before production handover.                   |

> ✅ **Goal:** Ensure the client’s database within Fabric Studio matches expected structures, counts, and relationships — fully validated and ready for use.

***

## Fabric Studio’s Proven Best Practices

Based on extensive experience with enterprise-scale migrations, the following practices consistently deliver successful results:

1. **Validate Early, Validate Often** — Catch data issues before ingestion, not after.
2. **Map Once, Test Twice** — Spend time refining mapping; confirm it through multiple validation cycles.
3. **Leverage Concurrency Smartly** — Scale workers based on infrastructure and dataset size.
4. **Stage the Load** — Follow a structured sequence to protect data relationships.
5. **Reserve Cleanup Time** — Always include buffer for post-load validation and remediation.
6. **Collaborate Constantly** — Keep both client and Fabric Professional Service teams aligned across all phases.
7. **Document Everything** — Maintain clear mapping documentation, validation logs, and migration reports for auditability.

<details>

<summary><span data-gb-custom-inline data-tag="emoji" data-code="1f44d">👍</span> Sample End-to-End Timeline (Example: 5 Million Records)</summary>

| Phase                           | Estimated Duration |
| ------------------------------- | ------------------ |
| Preparation & Configuration     | 7–9 Days           |
| Data Mapping & Test Load        | 3–5 Days           |
| Bulk Data Load (60 Concurrency) | 4–5 Days           |
| Post-Load QA & Cleanup          | 3–5 Days           |
| **Total Estimated Duration**    | **17–24 Days**     |

</details>