Migrate SharePoint version history intelligently: Less storage, same traceability

SharePoint Online storage costs money permanently, per gigabyte, every month. If you migrate 25 terabytes (our series starts with part 1), you should ask an uncomfortable question first: how many of these bytes are actually identical copies of each other?

In our case, the answer was: a surprisingly large number. The reason is a classic one.

The problem: Every metadata change becomes a “new file”

In the SharePoint 2016 source environment, almost every metadata update created a new file version: document class added, new version. Responsibility changed, new version. Workflow status updated, new version. The file content itself was byte-identical.

A typical version chain looked like this:

v1.0  Upload                    50 MB   <- real content
v2.0  Metadata added            50 MB   <- identical bytes
v3.0  Status changed            50 MB   <- identical bytes
v4.0  Content revision          52 MB   <- real new content
v5.0  Metadata corrected        52 MB   <- identical bytes

Five versions, 254 MB, but only two content-distinct documents (102 MB). Multiplied across millions of files and ten years of project history, an estate like this contains terabytes of redundant bytes that would be paid for in SharePoint Online forever.

The solution: Hash-based version deduplication

During extraction, our migration pipeline compares the content of each version chain by hash and only migrates versions whose content changed:

Source: v1 ──> v2 ──> v3 ──> v4 ──> v5
Hash:    A      A      A      B      B

Target: v1 (content A) ──> v2 (content B)

Conceptually:

var toMigrate = new List<SourceVersion>();
string? lastHash = null;

foreach (var version in chain.OrderBy(v => v.VersionNumber))
{
    var hash = await ComputeContentHashAsync(version);
    if (hash != lastHash)          // only versions with NEW content
    {
        toMigrate.Add(version);
        lastHash = hash;
    }
}
// The latest version in each hash group carries the final metadata,
// so the business-current metadata state is preserved.

Important: the metadata of the newest version wins. The skipped intermediate metadata states are almost always maintenance noise, but that must be explicitly agreed with the business team and not decided silently by technology.

What needs to be agreed with the business team

Deduplication is a business decision with a technical implementation, not the other way around:

1. Version numbers shift. Source v5.0 may become target v2.0. If documents were referenced in reports or approvals by “version 5.0”, you need a communication rule.
2. Audit requirements must be checked. Some document classes may require a complete version chain for regulatory reasons. Those are excluded. Deduplication is a rule per library or class, not a global switch.
3. Traceability. Every dedup decision is logged: which source versions were consolidated into which target version, including hashes. When more than 2,000 “different version labels” appeared during acceptance, we could prove each one as intentional hash-backed deduplication. The difference between “feature” and “finding” is the protocol.

The Migration API trap: the two-version bug

Ironically, deduplication triggers one of the most annoying service bugs in the Migration API: files with exactly two versions lose their history during import, and both arrive as v1.0. With one version and with three or more versions, it works.

And what does hash deduplication produce especially often? Chains with exactly two remaining versions.

Our answer is an explicit two-version strategy as a configurable option. In affected cases, we deliberately decide whether the older version must remain as separate history, in which case the import is adjusted, or whether the current state is sufficient from a business perspective. The decision is documented instead of being left to a service bug.

Result

• Significantly reduced target volume, with the storage business case carrying a relevant part of the project cost.
• Faster migration, because every skipped version saves download, encryption, upload and import time.
• Clean acceptance, with file-level parity checks (missing=0) and hash protocols at version level.

Recommendations for your project

1. Analyze versions before migration: what percentage of your estate consists of byte-identical version duplicates? That number decides whether the effort is worth it. In our case, it was double-digit.
2. Agree dedup rules with the business and define exceptions for compliance classes.
3. Build the protocol as a first-class artifact, because it is your acceptance evidence.
4. Handle two-version chains explicitly, otherwise the Microsoft bug handles them for you.

---

Would you like to know how much SharePoint Online storage your migration estate contains that you should never have to pay for? We can analyze it and bring the pipeline that implements it. Get in touch.

All parts of the series:

1. Why not a one-to-one migration
2. 25 TB SharePoint migration: Why standard tools are not enough
3. The SharePoint Migration API in practice
4. Term Store at the limit: Taxonomy migration in a shared tenant
5. Migrate SharePoint version history intelligently (this article)
6. Provisioning at enterprise scale: PnP, throttling, permissions