Provisioning at enterprise scale: PnP templates, throttling and race conditions across thousands of libraries

If you treat migration as transformation (part 1 of this series), you first have to build the target structure: in our case hundreds of project sites, each with dozens of libraries, unified content types, project-specific views and a multi-level permission model. Manually, this is unthinkable. So we used PnP provisioning via Azure Function, driven by templates.

In small environments, this is a solved problem. At enterprise scale, three additional opponents appear: throttling, missing idempotency and parallelism.

The architecture

Our provisioning runs as an Azure Function with certificate-based app-only authentication (PnP.Core + Key Vault). A PnP provisioning template defines content types, fields, libraries and views declaratively, while project- and construction-section-specific structures are added dynamically. The separation mattered:

• Template = group-wide standard (fields, content types, base views)
• Code = project-specific dynamics (number and naming of structures, navigation, permission levels)

This keeps the standard reviewable and versioned without requiring a separate template for every project variant.

Opponent 1: Throttling (HTTP 429)

Creating thousands of objects means tens of thousands of REST/CSOM calls. SharePoint Online throttles aggressively, and it does so per app per tenant. Your provisioning therefore competes with everything else your app registration does.

What proved useful:

1. Respect Retry-After, always. Retrying immediately after a 429 only extends the block. PnP.Core provides handling; custom HTTP calls need it explicitly.
2. Connection gating at process level. We limit parallel connections process-wide (12 in our case) instead of per task. Otherwise parallel work packages silently multiply into hundreds of simultaneous calls.
3. Write throttle telemetry. A simple JSON log with timestamp, endpoint and Retry-After duration shows which call patterns throttle. Optimization without measurement is guessing.
4. Batch where the API supports it. Views, fields and items can be grouped into $batch requests. That reduces call count and throttling pressure.

Opponent 2: Missing idempotency

With hundreds of sites, a run will fail in the middle: function timeout, throttling cascade, deployment. The most important property of enterprise provisioning is therefore: every run must be safely repeatable.

In practice, that means: before creating any object, check whether it exists. But do not blindly skip it. Check whether it matches the desired state (drift detection). “Already exists” can also mean “exists incorrectly”. One example from our project: a case-sensitivity bug in the view comparison made views look present although they were missing. The run was green, but the structure was incomplete. Since then, comparison logic is tested like business logic.

Opponent 3: Parallelism and the lost-update race

The most subtle bug in the project: when several structures of the same site were provisioned in parallel, navigation entries disappeared sporadically. Out of 19 expected links, sometimes only 6 arrived. Not reproducible, never the same ones.

The cause was a classic read-modify-write race: two parallel workers read the same navigation configuration, each adds its own entry, and both write back. The second writer overwrites the first one. Lost update.

The solution: optimistic concurrency with ETag/IF-MATCH and a retry loop:

// Lost-update-safe update of a shared property
for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
    var (value, etag) = await ReadWithETagAsync(url);      // GET returns ETag

    var updated = AddEntryIdempotent(value, newEntry);      // by business key
    if (updated is null) return;                            // already there

    var request = new HttpRequestMessage(HttpMethod.Put, url)
    {
        Content = new StringContent(updated)
    };
    request.Headers.TryAddWithoutValidation("IF-MATCH", etag); // NOT "*"

    var response = await client.SendAsync(request);
    if (response.IsSuccessStatusCode) return;                // won

    if (response.StatusCode == HttpStatusCode.PreconditionFailed)
        continue;                                            // 412: lost -> reread

    response.EnsureSuccessStatusCode();
}

Three details make the difference:

• IF-MATCH: * does not protect you. It only says “overwrite whatever is there”, which is exactly the race we want to prevent. You need the specific ETag of the state you read. Be careful: some client libraries internally force *, in which case only a raw REST call helps.
• Idempotency by business key: the update step checks whether the entry logically exists already, making retries and repeated runs safe.
• 412 is not an error, it is the system working: Precondition Failed means “someone else was faster”. Reread, reapply, write again.

Lessons learned

1. Provisioning is software, not a script. At a certain scale, it needs tests, telemetry, idempotency and concurrency design, the same standards as any backend component.
2. Design parallelism first, not afterwards. Every shared resource, such as navigation, property bags or central lists, is a race candidate.
3. Verify green runs. A provisioning run without desired-state comparison is a hope, not a result. It is the same lesson as in the migration itself.

---

You provision SharePoint structures across hundreds of sites and fight throttling, aborted runs or hard-to-reproduce bugs? These are exactly the problems we solved. We can help.

All parts of the series:

1. Why not a one-to-one migration
2. 25 TB SharePoint migration: Why standard tools are not enough
3. The SharePoint Migration API in practice
4. Term Store at the limit: Taxonomy migration in a shared tenant
5. Migrate SharePoint version history intelligently
6. Provisioning at enterprise scale: PnP, throttling, permissions (this article)