The SharePoint Migration API in practice: What Microsoft does not document

If a large SharePoint migration cannot be handled with standard tools (why we reached that point is covered in part 2), you sooner or later end up with the SharePoint Online Migration API. It is the same interface used under the hood by ShareGate and SPMT. It is the only practical way to import content with version history, authors and timestamps into SharePoint Online at high throughput, without dripping item by item through CSOM or REST.

The official documentation only covers half the story. This article consolidates what we learned in more than 50 reverse-engineering iterations against live SharePoint Online, with every point verified through successful end-to-end imports.

How the Migration API basically works

1. Provision containers: ProvisionMigrationContainers returns two Azure Blob containers (content and manifests), SAS URLs and an AES-256 key.
2. Build the package: the files plus 8 manifest XML files such as Manifest.xml, UserGroup.xml and SystemData.xml describe what is imported where, including field values, versions and authors.
3. Upload encrypted blobs: every blob is uploaded with AES-256-CBC encryption.
4. Start the job: CreateMigrationJobEncrypted passes containers and key to SharePoint; SharePoint imports server-side.
5. Read progress: an Azure Queue emits JobProgress and JobEnd events.

Sounds simple. The traps are in the details.

Trap 1: Encryption - never put the IV into the content

The most common dead end, including in Microsoft Q&A threads, is writing the initialization vector before the ciphertext. Wrong. The IV belongs into the blob metadata. The blob body contains only the ciphertext:

internal static (byte[] ciphertext, byte[] iv) EncryptAesCbcPureCiphertext(
    byte[] plainBytes, byte[] key)
{
    using var aes = Aes.Create();
    aes.Mode      = CipherMode.CBC;
    aes.Padding   = PaddingMode.PKCS7;
    aes.KeySize   = 256;
    aes.Key       = key;          // key from ProvisionMigrationContainers
    aes.GenerateIV();             // fresh IV per blob
    using var encryptor = aes.CreateEncryptor();
    var cipher = encryptor.TransformFinalBlock(plainBytes, 0, plainBytes.Length);
    return (cipher, aes.IV);
}

The upload flow per blob, and every step is mandatory:

1. PUT <blob>                    Body = ONLY ciphertext
                                 Content-MD5 = MD5 over CIPHERTEXT (not plaintext)
2. PUT <blob>?comp=metadata      x-ms-meta-IV = Base64(IV)
3. PUT <blob>?comp=snapshot      no body

Two consequences cost us hours:

• MD5 over the ciphertext, not over the plaintext. Otherwise Azure rejects the PUT with 400 Content-MD5 mismatch.
• The snapshot is not optional. Without a snapshot, the migration worker does not read the blob at all, and the job silently runs into a timeout. The snapshot must happen after the metadata PUT, otherwise the worker does not see the IV.

The error Data at the root level is invalid. Line 1, position 1. almost always means: IV in the blob body instead of in metadata. The worker decrypts garbage and tries to parse it as XML.

Trap 2: Manifest.xml and the strict deserializer

The schema documentation shows many attributes that the import worker simply ignores or fails on. Lessons from practice:

• SPFile.Id must not equal SPListItem.Id, otherwise file and list item collapse into one entry.
• SystemData.xml needs ObjectsProcessed, otherwise the job does not start cleanly.
• Folder hierarchies must be listed parents first in the manifest.
• Field values need exact wire formats: user fields as id;#login with the user present in UserGroup.xml, lookups as id;#text, taxonomy values with real term GUIDs.

Trap 3: The two-version bug

Our favorite service bug: a file with exactly two versions loses its history during import, and both versions arrive as v1.0. With one version, and with three or more versions, the <Versions> pattern works correctly. If, like us, you deduplicate version chains (part 5), you create exactly those two-version chains all the time and need an explicit strategy.

Also worth documenting:

• Author preservation is best-effort: if a historical user cannot be resolved anymore, the worker silently falls back to the system account and only reports a warning.
• JobEnd counters sometimes arrive as strings: if you parse FilesCreated naively as a number, you may read 0 and treat successful jobs as empty. Only tolerant parsing makes job telemetry reliable.
• Historical versions without the current version in the <Versions> block cause the worker to discard the entire history.

Trap 4: Green does not mean done

The most important strategic lesson: a successful migration job does not prove that everything arrived. Partial failures, discarded versions, silent fallbacks: all of that may only appear as warnings, if at all. We therefore built our own acceptance gate: after each project, a parity check compares every source file against the target (missing=0, SizeMismatch=0), with SHA256 sampling for suspicious cases. This report, not the job status, is our acceptance artifact.

Error dictionary (excerpt)

Symptom	Actual cause
Data at the root level is invalid	IV in blob body instead of x-ms-meta-IV
Job runs into timeout without error	Blob snapshot missing
400 Content-MD5 mismatch during upload	MD5 calculated over plaintext instead of ciphertext
Version history missing in target	Two-version bug or <Versions> without current version
All documents belong to the system account	User missing in UserGroup.xml or not resolvable anymore
FilesCreated=0 although files are visible	String-coded counters in the JobEnd event

Conclusion

The Migration API is powerful: versions, authors, timestamps and high throughput. But it is an API for tool builders, not an end-user interface. If you use it directly, you own encryption, manifest correctness and, most importantly, your own verification of the result.

---

You are evaluating the Migration API for your project or are stuck in one of these failure modes? We have already climbed that learning curve and can help from architecture review to the finished pipeline. Get in touch.

All parts of the series:

1. Why not a one-to-one migration
2. 25 TB SharePoint migration: Why standard tools are not enough
3. The SharePoint Migration API in practice (this article)
4. Term Store at the limit: Taxonomy migration in a shared tenant
5. Migrate SharePoint version history intelligently
6. Provisioning at enterprise scale: PnP, throttling, permissions