Diagnosing a Redis session timeout that only happened in production

I recently hit a Redis session timeout that only appeared on the live site under real traffic. It never showed up on staging, on my admin server, or in any of my automated tests. It took two failed production deploys, some digging around in AWS, and a load testing tool built with Claude Code before I found the cause. Here's what happened and what fixed it.

I look after a large ASP.NET Core Umbraco site that uses Redis as its distributed session cache. The Redis side is an AWS ElastiCache cluster running Valkey, the open source Redis fork. Session state and a few other cached lookups go through Microsoft's Microsoft.Extensions.Caching.StackExchangeRedis package, which wraps the StackExchange.Redis client.

As part of normal dependency maintenance I moved StackExchange.Redis from 2.x to the new 3.x release. Keeping dependencies current matters for security patches and new features, so this wasn't something I wanted to put off. Version 3 is a substantial rewrite of the client internals rather than a small bump, but I treated it like any other update: run the tests, deploy, keep an eye on it.

<PackageVersion Include="StackExchange.Redis" Version="3.1.31" />

Not long after the production deploy, the logs filled with this:

Session cache read exception, Key:"08a0af64-0349-d48c-b082-1d0bb0ed33bb"
The message timed out in the backlog attempting to send because no connection
became available (10000ms) - Last Connection Exception: It was not possible to
connect to the redis server(s)... ConnectTimeout, command=HMGET, ...
WORKER: (Busy=155,Free=32612,Min=2,Max=32767), ...

I also saw a smaller number of "Error closing the session" errors with the same underlying exception. Both were StackExchange.Redis.RedisConnectionException, and both meant visitors were getting failed or corrupted sessions on a live site.

I rolled back to the version 2 client that evening and the errors stopped straight away.

A rollback tells you what made the symptom go away, not what caused it, so I carried on looking. One thing stood out: the production ElastiCache cluster was several months behind on engine patches, while staging had them all applied. That looked like a plausible cause, so I booked a maintenance window, applied every outstanding patch to production, and tried the version 3 upgrade again.

Same errors, same night. Useful to know, but it cost me a second evening of live errors before rolling back again.

Sitting on version 2 indefinitely wasn't realistic. It blocks every future security fix and feature in the client, and I'd have to make the same jump later, probably under worse circumstances. The rollback bought me stability, not an answer, so the investigation stayed open.

The answer was in the exception text the whole time: WORKER: (Busy=155,Free=32612,Min=2,Max=32767).

The .NET ThreadPool minimum thread count defaults to the number of processors. My production web servers have 2 vCPUs, so the pool started with a floor of 2 worker threads. That's fine under light load. Once demand goes past the minimum, the pool only adds roughly one thread every 500ms. Getting from 2 to the 100 or more threads the app actually needed under a burst could take over a minute, and every session read or write queued behind that ramp up looked, from Redis's point of view, like a connection timeout.

This matches the StackExchange.Redis documentation on diagnosing timeouts, which calls out exactly this: seredis.dev/Timeouts.

The fix is one line at startup:

// The runtime default (processor count) is far too low a floor for an
// I/O-heavy app under load. Raising it removes the ramp-up window instead
// of just raising the eventual ceiling.
ThreadPool.SetMinThreads(200, 200);

Replacing a blocking cache call in the async request pipeline

While looking at what was running on those threads, I found one cache lookup calling the synchronous IDistributedCache.GetString and SetString methods instead of the async versions. That's a blocking call sharing a connection with all the async session traffic. StackExchange.Redis 3.x ships an analyzer rule specifically to flag this pattern, which suggested I was in the right area. I switched it to the async API:

public async Task<bool> HasPurchasesAsync(Guid memberKey)
{
    if (memberKey == Guid.Empty) { return false; }
    if (_requestCache.TryGetValue(memberKey, out var cached)) { return cached; }
 
    var stored = await _cache.GetStringAsync(KeyPrefix + memberKey);
    // ...
}

Keeping session middleware off the health check endpoint

I also found the load balancer health check endpoint was going through the same session middleware as every other request, purely because of where it sat in the pipeline. A Redis hiccup could fail a health check on a perfectly healthy server and pull it out of rotation, which makes a bad situation worse. I moved the health check to the front of the pipeline so it never touches session:

public class HealthCheckMiddleware
{
    private readonly RequestDelegate _next;
 
    public HealthCheckMiddleware(RequestDelegate next) => _next = next;
 
    public async Task InvokeAsync(HttpContext context)
    {
        if (context.Request.Path.Equals("/healthcheck", StringComparison.OrdinalIgnoreCase))
        {
            context.Response.StatusCode = StatusCodes.Status200OK;
            return;
        }
 
        await _next(context);
    }
}

Changing the Umbraco SessionIdLogging setting

While I was in that area I changed Umbraco's session logging mode. By default Umbraco adds the real ASP.NET Core session id to log entries, which means loading the session from the store on every logged request. Umbraco's configuration docs recommend switching this off when session lives in a distributed cache: SessionIdLogging settings.

"Umbraco": {
  "CMS": {
    "Logging": {
      "SessionIdLogging": "CookieHash"
    }
  }
}

None of this would have been convincing without reproducing the problem somewhere other than live production, and that was the hardest part of the whole job. The bug would not appear anywhere except the real site under real traffic.

My single admin server never saw it because it carries almost no concurrent load. Staging never saw it under manual testing or the automated test suite, because staging traffic is a fraction of production's. I built a small console app that hammered Redis directly with lots of concurrent session style reads and writes, on the theory that pure throughput was the trigger. It ran clean every time, at volumes well above what production sees, so command volume on its own wasn't the problem.

The breakthrough came from using Claude Code to build a Playwright load testing tool that drove real browser sessions through the actual site: browsing pages, opening products, adding items to a basket, with many of these journeys running concurrently against staging rather than against Redis on its own. That was the first time I reproduced the exact session cache errors outside production. The failure needed the full stack: real HTTP request handling, real session middleware, and real concurrent load on the same thread pool, not just a lot of Redis commands.

With a reliable way to trigger it on staging, I deployed the fixes above and ran the same load test again. The Redis session errors were gone.

The rollback to the older client hid the symptom without explaining anything, and it would have quietly stopped me taking security and feature updates on that library. The real cause was a combination of a thread pool floor that was too low for the load I was putting through it, a blocking cache call on that same starved pool, and a health check that should never have depended on session. None of it was specific to version 3 of the client. They were latent problems that the rewritten connection handling in version 3 had far less tolerance for.

The wider lesson is about how I test this kind of change. A bug that only appears under real concurrent load won't show up in a staging smoke test, however careful. Putting realistic concurrent traffic through staging before a risky dependency upgrade goes live is now part of my standard process, not something I only do once there's an incident to chase.