Runpod Serverless Cold Starts: A Thousand Invocations, Three Weeks Later

Serverless GPU computing promises elastic scalability: pay only for what you use, spin up and down as demand dictates. The catch, of course, is the ‘cold start’—the time it takes for your container to pull, initialize, and become ready to process a request. For many, this delay is the primary reason to avoid serverless entirely. We wanted to see what kind of real-world cold start performance one could expect from Runpod’s Serverless platform, so we set up a controlled experiment.

Our Methodology: A Thousand Calls to Reality

Over a period of three weeks, we performed exactly 1,000 invocations against a Runpod Serverless endpoint. Our setup was straightforward: a Python script triggered a request to our deployed endpoint, then measured the precise time from initiating the HTTP POST request to receiving the first byte of the response. We configured the endpoint to use a NVIDIA T4 GPU, a common and cost-effective choice for smaller inference tasks, and allocated 16GB of VRAM. The model itself was a slightly modified Stable Diffusion 1.5 checkpoint, roughly 5.2GB in size, loaded into PyTorch.

We ensured sufficient idle time between invocations to guarantee a cold start for each test—typically waiting 15-20 minutes, well beyond Runpod’s usual container shutdown threshold. The tests were distributed throughout the day and night to account for potential network or platform load variations. Our measurement script ran from a dedicated server located in a major European data center, ensuring consistent network conditions to Runpod’s European regions.

The Numbers: Variance Is the Name of the Game

Across our 1,000 invocations, the cold start times were, to put it mildly, inconsistent. This wasn’t entirely unexpected for a serverless platform, but the range proved instructive. Here’s what we observed:

Metric	Time (seconds)
Minimum	7.8
Maximum	46.1
Average	18.2
90th Percentile	26.5
99th Percentile	41.3

The fastest cold start, at 7.8 seconds, was likely a scenario where a warm worker was available or a container image was heavily cached on the host. The slowest, 46.1 seconds, indicated a complete spin-up, including image pull and initial model loading, potentially on a freshly provisioned host. The average of 18.2 seconds might sound acceptable on paper, but a 90th percentile of over 26 seconds means that one out of ten users would be waiting nearly half a minute just for the service to become ready. For anything user-facing, this is a non-starter.

We also noted that around 3% of our invocations (31 instances, to be precise) incurred cold start times exceeding 35 seconds. This suggests that while the platform generally performs within a certain range, there are occasional outliers that could significantly impact user experience or batch processing SLAs.

Cost Implications and Workarounds

Runpod Serverless bills by the second for GPU usage, with a minimum billing increment. A significant portion of your billed time for short inference tasks will be spent on the cold start. If your actual inference time for our Stable Diffusion model was, say, 10 seconds, and the cold start was 18 seconds, you’re paying for 28 seconds of GPU time. Almost two-thirds of that is overhead. For bursty workloads with short runtimes, this quickly erodes the cost benefits of serverless.

One common workaround is to minimize your Docker image size. Our 5.2GB model, while compressed, still needed to be pulled. A smaller base image and a more compact model could shave off a few seconds. Another strategy, if your workload allows, is to keep a container ‘warm’ by sending periodic dummy requests, although this defeats the purpose of serverless cost efficiency for truly idle periods. Runpod does offer a ‘min workers’ setting for pods, but for Serverless endpoints, the primary draw is the auto-scaling to zero.

For those looking to optimize, ensure your Dockerfile is lean, and consider using multi-stage builds to reduce the final image size. Pre-loading models into the image, rather than downloading at runtime, is essential, though it doesn’t solve the image pull time itself.

When is Runpod Serverless a Good Fit?

Despite the cold start variability, Runpod Serverless isn’t without its merits. For asynchronous batch processing, where users aren’t waiting interactively, these cold starts are often acceptable. If you’re queuing up a few hundred image generations or document analyses, an 18-second delay per job initialization is a minor inconvenience compared to managing dedicated instances that might sit idle for hours. The cost model—pay-per-second, scaling to zero—remains compelling for workloads that are highly unpredictable or bursty over longer durations.

However, for real-time applications, user-facing APIs, or anything requiring sub-10-second response times, the current cold start performance of Runpod Serverless on a T4 GPU presents a significant hurdle. You’ll likely need to either provision always-on dedicated pods (which negates the serverless advantage) or explore alternative architectures. If you’re setting up GPU inference and need to balance cost with responsiveness, understanding these cold start numbers is critical. For those exploring options, we’ve found their bare-metal offerings to be more consistent for dedicated workloads, and you can investigate their range of services at https://runpod.io/?ref=8vbo5oc9.

Our three weeks of testing confirm that serverless GPU cold starts are a real engineering challenge. While Runpod provides a convenient abstraction, the underlying realities of container orchestration and large model loading mean you’ll still be waiting. It’s a trade-off: convenience and cost efficiency for sporadic tasks versus raw, consistent speed. Choose wisely based on your application’s tolerance for latency.