Перейти к содержимому

When Switching Server Roles «Fixes» Your Bug — It Doesn’t

Опубликовано

Or: how a single attribute, two server roles, and one very helpful accident cost us weeks of confusion.

We recently spent the better part of two weeks chasing a bug that, at every step, looked like it simply could not be ours. Same binaries. Same configuration. Same request. One server returned data, the other returned a 500. In the end, the fix was deleting one attribute from one class — and the journey there taught us more about the ASP.NET Web API + Newtonsoft.Json + Sitecore role plumbing than we ever wanted to know.

This is the write-up we wish we could have googled on day one.

The setup

  • Sitecore XP 10.4, scaled topology
  • Hosted on Azure App Services
  • SXA with headless services
  • A custom ASP.NET Web API (`System.Web.Http`) controller exposing a search endpoint for the front end

The controller itself was as boring as Web API gets:

[HttpPost, Route("results")]
public async Task<IHttpActionResult> GetResults([FromBody] SearchRequestModel request)
{
    var validationMessage = ValidateRequestModel(request);

    if (!string.IsNullOrEmpty(validationMessage))
    {
        return BadRequest(validationMessage);
    }

    return Json(await searchService.GetSearchEntriesAsync(request));
}

And the request model — pay attention to the decoration, it's the villain of this story:

[Serializable]
public class SearchRequestModel : FilteredSearchRequestModel
{
    // properties with sensible C# initializers,
    // [JsonProperty] names, the usual DTO stuff
}

The symptom

A `POST` to the endpoint on CM worked perfectly. The exact same request — same JSON body, same headers, verified with `curl` from the same machine — against CD blew up with an HTTP 500. The CD logs showed:

ERROR Exception during fetching user profile
Exception: System.NullReferenceException
at FilteredSearchRequestModel.GetHashCode() ...

Exception System.InvalidOperationException: A null value was returned
where an instance of IHttpActionResult was expected.

(That second error was self-inflicted — our `catch` block returned `null` instead of a proper error result, which muddied the logs nicely. Own your anti-patterns; they *will* come back to confuse you at the worst time.)
So the model arrived at the action method… empty. Not `null` — the parameter itself was populated — but every property inside it was `null` or `0`.

The "impossible" observation

Here's where it got weird. While bisecting the differences between the instances, we changed one line in the CD instance's `web.config`:

<add key="role:define" value="ContentManagement" />

… (also added required connectionstrings.config lines to point to `master` database) and the endpoint instantly started working. Switch it back to `ContentDelivery` — broken again. Fully reproducible, every time.

Same code. Same `bin` folder. Same physical config files. The only variable was the role name. At this point we were fairly convinced this was a platform bug, and we opened a Sitecore support ticket titled, fittingly, "Strange ContentDelivery role behaviour."

Spoiler: it was not a platform bug. But that role-switch observation — the most convincing piece of "evidence" we had — turned out to be the most misleading clue of the whole investigation. More on that soon.

The red herring tour

Before the breakthrough, we (together with Sitecore support) ruled out an impressive collection of suspects. If you're debugging something similar, here's the checklist we burned through:

  1. A proxy or CDN truncating the request body. There's a known ASP.NET issue about silently truncated request bodies. We enabled Failed Request Tracing and queried App Insights (`requests | project timestamp, url, requestBody`) to prove the body arrived intact. It did.
  2. SXA site resolution. The endpoint used `sc_site`/`sxa_site` parameters, and "site not found" errors are a classic CM/CD discrepancy. Not it.
  3. Effective configuration differences. We diffed `/sitecore/admin/showconfig.aspx` output from both roles for anything Web API or HTTP-pipeline related. Identical where it mattered.
  4. Dependency injection differences. Checked. Identical.
  5. Deployment drift between CM and CD `bin` folders. Support actually found a small namespace difference in our deployed assemblies here — a refactor that had reached one role before the other. Embarrassing, worth fixing (and fixed right away), and completely unrelated — the underlying types were consistent and the bug reproduced regardless.

Every avenue closed. Same request in, different behavior out, and the only lever was the role name.

The clue that cracked it

The breakthrough was almost insultingly simple: we logged the bound model from inside the action, serialized back to JSON:

Log.Error($"Request object: '{JsonConvert.SerializeObject(request, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Include })}'", this);

On CD it printed:

{"EntryTemplateId":null,"filters":null,"sortingId":null,"term":null,"keyword":null,"count":0,"page":0,"lang":null}

Look closer. Several of those properties have C# initializers — default template IDs, a default page size, an empty filter list. They can't be `null`… unless the constructor never ran.

There's exactly one mainstream way to create a .NET object without running its constructor: `FormatterServices.GetUninitializedObject` — the mechanism used by *runtime serializers* for types marked `[Serializable]`.

That attribute suddenly looked a lot less innocent.

The actual mechanics

Sitecore support confirmed the theory in style: they took memory dumps of the `w3wp` processes on both roles and compared the cached Newtonsoft.Json contracts for our model.

On CD, the Web API JSON formatter was using its stock contract resolver — `System.Net.Http.Formatting.JsonContractResolver`. Here's the relevant bit of its source code:

public JsonContractResolver(MediaTypeFormatter formatter)
{
    _formatter = formatter;
    // Need this setting to have [Serializable] types serialized correctly
    IgnoreSerializableAttribute = false;
}

Newtonsoft's own `DefaultContractResolver` ignores `[Serializable]` by default. But Web API's resolver deliberately opts back in. And when Json.NET honors `[Serializable]` on a type that has no `[JsonObject]` or `[DataContract]` override, it switches that type to field-based serialization (`MemberSerialization.Fields`):

  • The expected JSON member names become the compiler-generated backing fields — `<Term>k__BackingField`, `<Keyword>k__BackingField`…
  • The instance is created uninitialized, BinaryFormatter-style — no constructor, no property initializers.

The memory dump made it beautifully concrete: CD's Newtonsoft name table contained `<Term>k__BackingField`-style entries, while CM's contained plain `term` and `keyword`.

So on CD, our perfectly valid JSON — `{"term":"", "keyword":"", "count":10, "page":1}` — was deserialized into a contract expecting field names that no sane client would ever send. Nothing matched. Result: an uninitialized object with every member at its default value, handed politely to our action method.

But why did CM work?

This is the part that makes the story worth telling. Our code was equally wrong on both roles. CM should have failed identically.

It didn't, because of an accidental guardian angel: **Experience Profile**. On ContentManagement instances, Sitecore's Contact Intelligence plumbing runs an `initialize` pipeline processor — `Sitecore.Cintel.Endpoint.Plumbing.InitializeRoutes` — which, as a side effect, replaces the global Web API contract resolver:

// effectively what happens on CM during initialize:
GlobalConfiguration.Configuration.Formatters.JsonFormatter
    .SerializerSettings.ContractResolver = new DefaultContractResolver();

A fresh `DefaultContractResolver` has `IgnoreSerializableAttribute = true`. So on CM — and only on CM — our `[Serializable]` attribute was being ignored, and property-based binding worked exactly as we expected.

flowchart TD
    A["POST /api/search/results<br/>{ term, keyword, count, page }"] --> B{Which role?}
    B -->|CM| C["Cintel InitializeRoutes ran at startup<br/>→ global resolver replaced with<br/>DefaultContractResolver<br/>(IgnoreSerializableAttribute = true)"]
    B -->|CD| D["Stock Web API JsonContractResolver<br/>(IgnoreSerializableAttribute = false)"]
    C --> E["[Serializable] ignored<br/>→ property-based contract<br/>→ model binds ✅"]
    D --> F["[Serializable] honored<br/>→ field-based contract<br/>→ expects <Term>k__BackingField<br/>→ uninitialized, empty model ❌"]

And that fully explains the "impossible" observation: flipping the CD instance's role to `ContentManagement` didn't fix our bug — it enabled Experience Profile's route initialization, which happened to overwrite the global resolver, which happened to mask our bug. We weren't bisecting one variable; we were flipping dozens of pipeline processors at once and reading tea leaves.

A role switch is a clue, never a diagnosis. The roles differ in which code runs at startup, and some of that code mutates global state — including, apparently, the JSON serialization behavior of your custom APIs.

The fix

After two weeks of proxies, dumps, and existential doubt, the fix was this:

-    [Serializable]
     public class SearchRequestModel : FilteredSearchRequestModel

One line. We then swept the rest of our API DTOs for the same attribute and removed it everywhere it had no business being.

Why was it there in the first place? Honestly: habit. The attribute had been on that class since the day it was created, seven months earlier — pure muscle memory, probably copied from a neighboring class. (Sitecore developers grow this reflex naturally: plenty of platform types -- custom validators, for instance -- genuinely require `[Serializable]`.) It cost nothing for seven months, because the endpoint was only ever exercised against CM and local single-instance sandboxes… right up until real traffic hit a real ContentDelivery server.

If you genuinely need `[Serializable]` on a type that also travels through Web API JSON binding (say, it's shared with session state), you don't have to choose. Any of these force a property-based JSON contract regardless of resolver settings:

[Serializable]
[JsonObject(MemberSerialization.OptOut)]   // wins over [Serializable] for Json.NET
public class SearchRequestModel { ... }

There's also a no-code workaround we considered and rejected: patching the Cintel `InitializeRoutes` processor to run on CD as well, making both roles behave "like CM". It works -- but it couples your public API's serialization behavior to the internal plumbing of Experience Profile, which is exactly the kind of accidental dependency that produced this bug in the first place. Fix the model, not the symptom.

Prevention checklist

  • Never put `[Serializable]` on Web API / JSON DTOs. It's for runtime binary serialization, not JSON. If a DTO must carry it, add `[JsonObject(MemberSerialization.OptOut)]` alongside.
  • Know that Web API's JSON formatter ≠ plain Json.NET. `JsonContractResolver` sets `IgnoreSerializableAttribute = false`. Your own `JsonConvert.SerializeObject` diagnostics use different defaults than the model binder — which is why our logging showed friendly property names while the binder was hunting for backing fields.
  • Treat CM and CD as different runtimes, not different configs. Identical code and config still produce different startup pipelines, and platform modules can mutate global state like `GlobalConfiguration`. Your custom API runs inside that.
  • Test custom endpoints against a CD-role instance early. A local Docker compose with a real CD container would have caught this on day one of the feature, not month seven.
  • Don't return `null` from catch blocks in Web API actions. "A null value was returned where an instance of IHttpActionResult was expected" buries the actual exception one log entry deeper.
  • When a model arrives mysteriously empty, serialize it back and check initializer-backed properties. If even those are null, no constructor ran — you're looking at a serialization-contract problem, not a binding problem.
  • Memory dumps are underrated. Comparing the cached Newtonsoft contracts across two `w3wp` dumps is what turned theories into proof.

Just a little bit of explanation that I owe on this

Let's be honest about the emotional arc here, because the technical write-up above makes it sound much tidier than it felt.

For days, we were certain this was a platform bug. We had the perfect proof: same code, same config, different roles, different behavior. We'd built a fortress of evidence — request traces, config diffs, bin comparisons — and every brick of it was solid. And the whole time, the bug was a single attribute one of us had typed without thinking, seven months earlier, on a Thursday.

That's the thing about the "impossible" bugs: they're never impossible, they're just misattributed. The role switch felt like a smoking gun pointing at Sitecore. It was actually a smoke machine pointing away from us.

A sincere thank-you to the Sitecore support engineers on this case — first for patiently ruling out our theories without ever making us feel silly, and then for the memory-dump analysis that nailed the root cause with the kind of rigor you can't argue with. (Fun fact from the modern era of support: the first responder on the ticket was an AI agent that politely declined to read our `.txt` attachments. The humans, when we reached them, more than made up for it.)

If you take one thing from our two weeks: when a bug "can't be your code", go read your code again — slower this time, and with special suspicion for the lines that have been there so long they've become invisible.

The diff was one line. The lesson, hopefully, is permanent.