Hi Isaac, sorry, we had to delete the deployment to free up our pipeline. Find below the logs of that deployment:
10/30/2025, 9:16:42 AM Application startup failed. Exiting.
10/30/2025, 9:16:42 AM Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/starlette/routing.py", line 694, in lifespan
async with self.lifespan_context(app) as maybe_state:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/lifespan.py", line 67, in lifespan
await database.start_pool()
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/database.py", line 221, in start_pool
await _pg_pool.open(wait=True)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 385, in open
await self.wait(timeout=timeout)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 173, in wait
raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec
10/30/2025, 9:16:27 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:16:19 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:16:15 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:16:13 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:16:12 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:16:12 AM Starting Postgres runtime with langgraph-api=0.4.47
10/30/2025, 9:16:12 AM Waiting for application startup.
10/30/2025, 9:16:12 AM Started server process [1]
10/30/2025, 9:16:12 AM Using auth of type=langsmith
10/30/2025, 9:16:12 AM Using langgraph_runtime_postgres
10/30/2025, 9:15:57 AM psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec
10/30/2025, 9:15:57 AM raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 173, in wait
10/30/2025, 9:15:57 AM await self.wait(timeout=timeout)
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 385, in open
10/30/2025, 9:15:57 AM await _pg_pool.open(wait=True)
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/database.py", line 221, in start_pool
10/30/2025, 9:15:57 AM await database.start_pool()
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/lifespan.py", line 67, in lifespan
10/30/2025, 9:15:57 AM ^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:15:57 AM return await anext(self.gen)
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
10/30/2025, 9:15:57 AM File "/api/langgraph_api/queue_entrypoint.py", line 177, in combined_lifespan
10/30/2025, 9:15:57 AM ^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:15:57 AM return await anext(self.gen)
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
10/30/2025, 9:15:57 AM File "/api/langgraph_api/queue_entrypoint.py", line 190, in entrypoint
10/30/2025, 9:15:57 AM File "/api/langgraph_api/queue_entrypoint.py", line 226, in main
10/30/2025, 9:15:57 AM File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
10/30/2025, 9:15:57 AM ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:15:57 AM return self._loop.run_until_complete(task)
10/30/2025, 9:15:57 AM File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
10/30/2025, 9:15:57 AM ^^^^^^^^^^^^^^^^
10/30/2025, 9:15:57 AM return runner.run(main)
10/30/2025, 9:15:57 AM Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/api/langgraph_api/queue_entrypoint.py", line 258, in <module>
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
10/30/2025, 9:15:57 AM Cancelling queue entrypoint task
10/30/2025, 9:15:44 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:15:35 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:15:30 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:15:28 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:15:27 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:15:27 AM Starting Postgres runtime with langgraph-api=0.4.47
10/30/2025, 9:13:18 AM Application startup failed. Exiting.
10/30/2025, 9:13:18 AM Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/starlette/routing.py", line 694, in lifespan
async with self.lifespan_context(app) as maybe_state:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/lifespan.py", line 67, in lifespan
await database.start_pool()
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/database.py", line 221, in start_pool
await _pg_pool.open(wait=True)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 385, in open
await self.wait(timeout=timeout)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 173, in wait
raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec
10/30/2025, 9:13:18 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:13:03 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:55 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:51 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:49 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:48 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:48 AM Starting Postgres runtime with langgraph-api=0.4.47
10/30/2025, 9:12:48 AM Waiting for application startup.
10/30/2025, 9:12:48 AM Started server process [1]
10/30/2025, 9:12:48 AM Using auth of type=langsmith
10/30/2025, 9:12:48 AM Using langgraph_runtime_postgres
10/30/2025, 9:12:39 AM psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec
10/30/2025, 9:12:39 AM raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 173, in wait
10/30/2025, 9:12:39 AM await self.wait(timeout=timeout)
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 385, in open
10/30/2025, 9:12:39 AM await _pg_pool.open(wait=True)
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/database.py", line 221, in start_pool
10/30/2025, 9:12:39 AM await database.start_pool()
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/lifespan.py", line 67, in lifespan
10/30/2025, 9:12:39 AM ^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:12:39 AM return await anext(self.gen)
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
10/30/2025, 9:12:39 AM File "/api/langgraph_api/queue_entrypoint.py", line 177, in combined_lifespan
10/30/2025, 9:12:39 AM ^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:12:39 AM return await anext(self.gen)
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
10/30/2025, 9:12:39 AM File "/api/langgraph_api/queue_entrypoint.py", line 190, in entrypoint
10/30/2025, 9:12:39 AM File "/api/langgraph_api/queue_entrypoint.py", line 226, in main
10/30/2025, 9:12:39 AM File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
10/30/2025, 9:12:39 AM ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10/30/2025, 9:12:39 AM return self._loop.run_until_complete(task)
10/30/2025, 9:12:39 AM File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
10/30/2025, 9:12:39 AM ^^^^^^^^^^^^^^^^
10/30/2025, 9:12:39 AM return runner.run(main)
10/30/2025, 9:12:39 AM Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/api/langgraph_api/queue_entrypoint.py", line 258, in <module>
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
10/30/2025, 9:12:39 AM Cancelling queue entrypoint task
10/30/2025, 9:12:25 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:17 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:12 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:10 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:09 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:12:09 AM Starting Postgres runtime with langgraph-api=0.4.47
10/30/2025, 9:11:17 AM Application startup failed. Exiting.
10/30/2025, 9:11:17 AM Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/starlette/routing.py", line 694, in lifespan
async with self.lifespan_context(app) as maybe_state:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/lifespan.py", line 67, in lifespan
await database.start_pool()
File "/usr/lib/python3.12/site-packages/langgraph_runtime_postgres/database.py", line 221, in start_pool
await _pg_pool.open(wait=True)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 385, in open
await self.wait(timeout=timeout)
File "/usr/lib/python3.12/site-packages/psycopg_pool/pool_async.py", line 173, in wait
raise PoolTimeout(f"pool initialization incomplete after {timeout} sec")
psycopg_pool.PoolTimeout: pool initialization incomplete after 30.0 sec
10/30/2025, 9:11:17 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:11:02 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:10:54 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:10:50 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:10:48 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:10:47 AM error connecting in 'pool-1': connection failed: connection to server at "172.26.64.123", port 5432 failed: FATAL: password authentication failed for user "postgres"
10/30/2025, 9:10:47 AM Starting Postgres runtime with langgraph-api=0.4.47
10/30/2025, 9:10:47 AM Waiting for application startup.
10/30/2025, 9:10:47 AM Started server process [1]
We managed to temporarily fix it by deploying using “production” deployment, which is a bit counterproductive given the pricing difference and the purpose: being able to QA a deployment from a branch before merging it.