25 TB SharePoint migration: Why ShareGate and SPMT are not enough here

In the first part of this series, we explained why large migrations should rarely be one-to-one. Now we get specific: why did we build a custom migration pipeline for a 25 TB migration from SharePoint 2016 to SharePoint Online instead of using ShareGate or the Microsoft SharePoint Migration Tool (SPMT)?

Spoiler: not because these tools are bad. The reason is that our requirements were not copy requirements. They were transformation requirements.

The starting point

• More than 25 terabytes of documents in SharePoint 2016 on-premises
• More than 130 major projects, each with its own site collection, structures and sometimes hundreds of libraries
• Millions of files, and even more versions
• Target: a group-wide shared SharePoint Online tenant with a unified, newly designed information architecture

The real requirement was this: content should not arrive at the old place in the cloud. It should arrive in a new structure, with new content types, new metadata and consolidated document classes.

What standard tools are very good at

ShareGate, SPMT and comparable products are excellent when the mapping essentially says: source -> target, same structure, same fields. They provide ready-made user interfaces, scheduling, reports and solid error handling. For most migrations, they are the right choice, and that is what we tell our customers too.

Where it broke for us

1. Field-level transformation with fallback logic

Our document classes existed in dozens of variants in the source: historical spellings, project-specific special cases, renamed terms. In the target, there is a consolidated global vocabulary. The mapping is not a simple table, but a rule chain with fallbacks:

1. Exact match against consolidated target value       -> use it
2. Match via historical synonym table                  -> map + log
3. Match via normalization (casing, special chars)     -> map + log
4. No match                                            -> defined fallback value
                                                        + entry in remediation register

Point 4 is decisive: no document may fail because of mapping, but every fallback must be auditable so the business team can remediate targeted cases later. This requirement, a complete mapping protocol as an acceptance artifact, is not provided by standard tools.

2. Taxonomy becomes text, choice and lookup

Millions of project-specific terms do not fit into the shared Term Store. We cover the details in part 4. Managed metadata fields therefore had to change field type during migration, including cleanup of wire-format artifacts such as ;# prefixes and |guid suffixes that would otherwise appear as text garbage in the target. Field type transformation at this level is, at best, rudimentary in standard tools.

3. Version deduplication as a business case

In the source, every metadata change created a new file version, even if the file content was identical. Across 25 TB, this means terabytes of byte-identical duplicates that would create permanent storage cost in SharePoint Online. Our pipeline identifies identical version chains via hash comparison and migrates only versions whose content actually changed. We cover this in part 5. No standard tool we know performs content-based version deduplication.

4. Throughput and control at 25 TB

At this data volume, every pipeline stage matters: parallelized source downloads with connection gating, batched migration packages with budgets for items, versions and bytes per package, parallel blob upload and throttling telemetry. We needed the control knobs in code, not hidden behind a UI.

5. Acceptance with evidence

We do not answer the question “Did everything really arrive?” with a tool report. We answer it with a custom parity check: every source file is compared against the target with missing=0 and SizeMismatch=0 per project, and SHA256 forensics for suspicious cases. These reports are part of formal project acceptance. A migration job that reports “green” is not proof. String-coded counters in job reports and silent partial failures taught us that more than once.

The architecture of our pipeline

Some design decisions that proved useful:

• Staging as truth: everything is fully extracted from the source first and stored locally with versions. Transformation and packaging work against staging. The source is touched read-only, and every step is repeatable.
• Resume everywhere: with 25 TB, something will fail. Every phase must be idempotent and resumable, including re-attaching to running migration jobs after a process restart.
• Fail fast on authentication: repeated auth retries against on-premises systems can lock accounts. An auth error stops immediately instead of entering a retry loop.
• Verification as a forced gate: the acceptance check is not an optional script. It is built into the pipeline. Without mandatory checks passing, there is no “done”.

When is a custom build worth it?

Honest answer: rarely. A dedicated pipeline is worth considering when at least two of these points apply:

1. Transformation instead of copying structure, field types or vocabulary
2. Data volume where storage or runtime optimization saves real money
3. Compliance requirements for provable completeness
4. Limit conflicts that force redesigns, such as Term Store, lookups or thresholds

If only one applies, first check whether a standard tool plus pre- or post-processing scripts is enough. A custom build is powerful, but it also means you must understand the failure modes of the Migration API yourself. That is the topic of the next part.

---

Your migration exceeds what classic tools can handle? We built a complete enterprise-scale migration pipeline and are happy to share the experience, from feasibility analysis to implementation. Talk to us.

All parts of the series:

1. Why not a one-to-one migration
2. 25 TB SharePoint migration: Why standard tools are not enough (this article)
3. The SharePoint Migration API in practice
4. Term Store at the limit: Taxonomy migration in a shared tenant
5. Migrate SharePoint version history intelligently
6. Provisioning at enterprise scale: PnP, throttling, permissions