Long-running run getting cancelled by aggressive workers scale down

LangSmith Product Help Deployment
1

Hi LangChain team, recently we are experiencing some issues related to some of our graph runs getting cancelled (asyncio.CancelledError) in the LangSmith tracing in our Production deployment.

After some debugging using the server log, we found out that during the time these runs getting cancelled, there were quite a few shutdown logs that appeared.

For example, at 7/7/2026, 12:09:26 PM:

[ERROR] The queue surpassed the grace period while shutting down. Some runs may still be running and will be canceled and retried on a healthy instance. The grace period is currently set to 180 seconds. If that is too low, adjust the BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS environment variable to an appropriate value. Signaling shutdown
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/api/langgraph_api/worker.py", line 505, in worker
  File "/api/langgraph_api/worker.py", line 269, in worker
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/api/langgraph_api/worker.py", line 207, in wrap_user_errors
  File "/api/langgraph_api/stream.py", line 639, in consume
  File "/api/langgraph_api/stream.py", line 665, in _consume_inline
  File "/api/langgraph_api/stream.py", line 501, in astream_state
  File "/api/langgraph_api/asyncio.py", line 86, in wait_if_not_done
  File "/usr/local/lib/python3.12/asyncio/taskgroups.py", line 71, in __aexit__
    return await self._aexit(et, exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/taskgroups.py", line 153, in _aexit
    raise propagate_cancellation_error
  File "/api/langgraph_api/asyncio.py", line 94, in wait_if_not_done
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/main.py", line 3455, in astream
    async for _ in runner.atick(
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/_runner.py", line 482, in atick
    done, inflight = await asyncio.wait(
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 550, in _wait
    await waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/storage/langgraph_runtime_postgres/lifespan.py", line 249, in queue_with_signal
  File "/storage/langgraph_runtime_postgres/queue.py", line 404, in queue
  File "/storage/langgraph_runtime_postgres/queue.py", line 197, in shutdown_queue
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
    async with timeouts.timeout(timeout):
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

We tried to locate the monitoring during that time period. Our hypothesis was that, for a brief moment, since the number of replicas were increasing very substantially and then also went down extremely fast after that, some workers that were executing the runs were getting shutdown. Some of these runs can take quite a decent amount of time (much longer than three minutes), they were getting cancelled and retried on another worker.

Screenshot 2026-07-09 at 10.55.28
Screenshot 2026-07-09 at 10.55.281842×1342 270 KB

We can try to increase the BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS but it won’t help much as a lot of our runs can take up to 20 mins to finish. These runs deal with a lot of user input data and thus the LLM take substantial amount of time to process.

I’m not sure if our diagnosis is correct. Please help us on this and any help is appreciated.

Thank you.

2

Your diagnosis is correct — aggressive scale-down is terminating workers mid-run, and your long runs are outliving the shutdown grace period. Two things you’re missing, though, and they change the fix.

How it actually works: each in-flight run records a heartbeat, and on a graceful shutdown (SIGTERM from a scale-down or rollout) the worker stops taking new runs, gives the in-flight ones a limited window to finish, and if they don’t finish it cancels them (asyncio.CancelledError) and puts them back on the queue. That’s exactly your TimeoutError → CancelledError trace: the 180s grace period expiring on a run that needed longer. (Scalability & resilience)

Missing piece #1 — those runs aren’t lost; they’re retried and resumed from the last checkpoint. BG_JOB_MAX_RETRIES (default 3) re-queues a shutdown-cancelled run and resumes it from the last checkpointed step, not from scratch. So the CancelledError is a cancelled attempt, not necessarily a failed run. Key question: are your runs ultimately completing after retry, or actually ending in failed? That tells us whether this is scary-but-recovering noise or real data loss. (BG_JOB_MAX_RETRIES)

Missing piece #2 — the grace period can cover your 20-min runs. BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS maxes out at 3600s (1 hour), so 20 min (1200s) fits. The catch most people hit: on Kubernetes your pod’s terminationGracePeriodSeconds must be ≥ that value, or the kubelet sends SIGKILL and the graceful drain never finishes. Those two settings have to move together. (BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS)

What I’d do, in order (assuming you manage the data plane on K8s):

  1. Tame the scale-down — this is the real trigger. Raise the HPA behavior.scaleDown.stabilizationWindowSeconds and cap the scale-down rate so a traffic spike followed by a fast drop doesn’t yank busy workers. Add a PodDisruptionBudget to limit simultaneous evictions.
  2. Align the grace periods. Set BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS toward your p95 run time (e.g. 1200–1500s) and bump pod terminationGracePeriodSeconds to match. Trade-off: slower rollouts/scale-downs, so pair it with #1 rather than relying on it alone.
  3. Make retries cheap via checkpoint granularity. If a 20-min run is one big node, a resumed attempt redoes a lot of work and can re-hit the cancellation. Splitting long LLM work into more nodes = finer checkpoints = a resume that redoes minimal work.
  4. Keep pre-checkpoint side effects idempotent so a resume doesn’t double-execute writes. (Idempotency guidance)

To point you at the right subset, two questions:

  • Cloud, Hybrid, or fully self-hosted? It decides whether you own the HPA (self-hosted → #1/#2 are yours) or we tune it (managed → focus on grace period + an escalation).
  • What’s driving the spike? Scale-up-then-immediate-scale-down usually means the autoscaler is reacting to a bursty metric — worth smoothing at the source.

Happy to go deeper once we know the deployment modality.

3

Thank you. That does match what we found. A few clarifications and questions up front:

  1. Cloud, Hybrid, or fully self-hosted?
    We’re on LangGraph Cloud (managed), so several of your suggestions, scaleDown.stabilizationWindowSeconds, terminationGracePeriodSeconds, and PodDisruptionBudget, aren’t knobs we control.

  2. Are runs completing after retry, or ending in failed? We observed both.
    In our worst window, (~2.5h of heavy testing):

  • Most succeed on the first attempt.
  • The one that do failed (7 runs) were cancelled by the scale down and were resumed. Of those, 2 runs (which take longer than 20-30 min) were cancelled and resumed multiple times and resulted in failed state due to exceeded max attempts (3).
 [WARNING] RETRYING
    Traceback (most recent call last):
      File "/api/langgraph_api/worker.py", line 269, in worker
      File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
        return await fut
               ^^^^^^^^^
      File "/api/langgraph_api/worker.py", line 207, in wrap_user_errors
      File "/api/langgraph_api/stream.py", line 639, in consume
      File "/api/langgraph_api/stream.py", line 665, in _consume_inline
      File "/api/langgraph_api/stream.py", line 501, in astream_state
      File "/api/langgraph_api/asyncio.py", line 86, in wait_if_not_done
      File "/usr/local/lib/python3.12/asyncio/taskgroups.py", line 71, in __aexit__
        return await self._aexit(et, exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/taskgroups.py", line 153,
        raise propagate_cancellation_error
      File "/api/langgraph_api/asyncio.py", line 94, in wait_if_not_done
      File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/main.py", line 3455, in astream
        async for _ in runner.atick(
      File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/_runner.py", line 482, in atick
        done, inflight = await asyncio.wait(
                         ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/tasks.py", line 464, in wa
        return await _wait(fs, timeout, return_when, loop)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/tasks.py", line 550, in _wait
        await waiter
    asyncio.exceptions.CancelledError
    7/7/2026, 1:02:34 PM
    [WARNING] Background run failed, will retry. Exception: <class 'asyncio.exceptions.CancelledError'>()
    7/7/2026, 1:02:33 PM
    [ERROR] Background run failed. Exception: <class 'RuntimeError'>(Run 019f3af3-163f-7410-a9bb-9e0b62b5dcdd exceeded max attempts (3).
    This usually means the pod processing the run became unhealthy and the run was re-queued repeatedly. Check for OOM kills, pod restarts, liveness/readiness probe failures, or other signs of pod instability in your deployment's log
  1. What’s driving the spike? The spike was due to our internal testing sessions with multiple people. This might skew the result a bit extreme however our product is also quite bursty in nature (usage might not be the same throughout the day). We also will be shipping new features that might consist of AI agent running for a very long time where sometimes a model invocation can take even up to 20 minute and thus the potential of these model invocation being cancelled mid-run will be quite undesirable.

We also have some questions about the Cloud Environment:

  1. On Cloud, can we set BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS and BG_JOB_MAX_RETRIES ourselves via deployment env vars? And critically, will a raised BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS actually be honored, i.e. does Cloud set the pod’s terminationGracePeriodSeconds to match, or is it capped platform-side (e.g. at 180s)?
  2. Since we can’t set HPA/PDB on Cloud: can you raise our scale-down stabilization window and/or set a min-replica floor for our deployment (at least during active testing)? What’s our current autoscaling policy? Scaling metric, min/max replicas, scale-down window?
  3. For runs that legitimately approach or exceed the 3600s grace cap, what pattern do you recommend on Cloud?
4

Hi, @arthurtran-drova! Thanks for the updates and clarifications.

Regarding the cloud environment, here’s what I could gather:

1. BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS and BG_JOB_MAX_RETRIES are both settable as deployment env vars. Defaults are 180s and 3. The grace period is capped at 3600s on Cloud (terminationGracePeriodSeconds). BG_JOB_MAX_RETRIES technically has no fixed max that I could see. (env vars for Cloud)

2. We only have two autoscaling options: Development (up to 1 replica) and Production (up to 10). Production scales on pending runs (target 10 per container) plus CPU/memory at 75%, and scale-down already waits 30 minutes specifically to avoid the thrashing you described. Given that 30-min wait, the mid-run SIGTERMs you saw may not be scale-down at all — rollouts, node recycling, or spot reclamation hit the same shutdown path. (Cloud scaling · deployment types)

3. On Cloud, 3600s is the hard cap. The mitigation: split the work and long LLM calls across more graph nodes. A resumed attempt restarts from the last checkpoint, so finer nodes mean a rescued run redoes far less instead of starting from the beginning. Depending on your use case, this may or may not work for you, but could be worth exploring. (Durable execution)

Finally, if long-running agents are central to your use case, you might also consider self-hosting the Agent Server. You still get all the benefits of tracing and evals in LangSmith Cloud, but you would own your own Agent Server pod and control the termination grace, autoscaler, and eviction policies. Trade-off of course is that you’d be managing your own infrastructure, and it requires an Enterprise license. But if that is something you’d like to explore, I’d be happy to connect you with our team. (Self-hosted options)

5

Thanks for the detailed reponse. We might consider increasing BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS for now as a temporary measure.

We also might consider the self-hosted options once we have a more reliable stream of users.

One thing that is not clear is whether or not we can raise the MIN-REPLICA from 1 to a bigger number in Production environment to ensure that that the auto-scaling can respond immediately to traffic spikes without the latency of cold-starting new pods.

6

Unfortunately, on Cloud, the replica counts (min and max) are not currently customer-configurable. Right now the autoscaling is fully managed.

We do always keep at least one warm replica (min of 1), so you’re not cold-starting from nothing. And we are looking at ways to make this more configurable.

For the time being, self-hosting the Agent Server is going to give you the most flexibility here.