Skip to main content

Waiting for a job

Analyzing a part, analyzing its features, creating a plan, calculating toolpaths, tessellating a part for display, inspecting a file, and importing a tool holder all run in the background. Each of those operations answers 202 Accepted straight away with a jobId, and the result exists only once that job has finished.

Wait for the job by streaming it, with GET /v1/jobs/{id}/events. Do not wait by calling GET /v1/jobs/{id}, or the result endpoint, in a loop.

How the stream works​

The response is a server-sent event stream. It stays open while the job runs and carries two kinds of event:

  • A job event as soon as the stream opens, holding the job as it is now. Its data is the same JSON object GET /v1/jobs/{id} returns.
  • Another job event every time the job changes: its status, its progress percentage, its retriesRemaining, or its error.

The stream may also contain comment lines (starting with :) that keep the connection open; ignore them. SSE parsers skip them for you.

event: job
data: {"jobUuid":"0195f02c-…","status":"running","progress":40,"retriesRemaining":2,"error":null,…}

: keepalive

event: job
data: {"jobUuid":"0195f02c-…","status":"succeeded","progress":100,"retriesRemaining":2,"error":null,…}

A job is queued until a worker starts it, then running until it finishes — including while it waits to be retried after a failed attempt. succeeded and failed are final: a job in either state never changes again, and the server closes the stream right after sending one. On succeeded, read the result. On failed, report the job's error; there is nothing more to wait for.

The worker enforces a hard execution deadline (default 300 seconds). A job past that deadline fails without retry, with an error such as timed out after 300s.

If queue deliveries are exhausted without a recorded outcome — for example, the parent dies on the last delivery or its final database write fails — a consumer of the dead-letter queue fails the job with job_delivery_exhausted:. With healthy consumers, recovery normally follows within a few minutes after the last delivery; it waits for workers and infrastructure to recover during an outage. A stopped worker on an earlier delivery can instead be followed by a successful retry.

A job still queued after 24 hours fails with job_dispatch_failed:. This covers a request that stopped after creating the row but before sending its message, and also a message that was sent but never started in time. Recovery does not infer whether sending occurred. A final failure can retain the last recorded retriesRemaining, including a positive count or null if no attempt was recorded.

A job fails without retry, with an error beginning source_changed:, if its source changed after the job was queued. It is not charged. Repeating the request is refused too, with 409 and a code such as part_source_changed: create a new part, and upload to that.

A stream that ends without a final status lost its connection, or the server could not read the job. Open it once more: a new stream starts by sending the job as it is now, so nothing is missed. If that one ends the same way too, or answers 410 because the job has expired (or 404 because it was not found), stop and report it rather than opening it in a loop.

To try a failed job again, make its request again. Where the operation takes an Idempotency-Key, use a new one: the key the job was made with answers with the job that failed. Analyze part features takes none, so making its request again starts a new job.

When the job can't be queued​

An operation that queues work can answer 503 with the problem code job_dispatch_failed: the job could not be sent to the processing queue, so it was not created and there is nothing to wait for. Retry the request. Reusing its Idempotency-Key, where it has one, is safe.

Why not ask repeatedly?​

Asking for the job in a loop — polling — does eventually see the job finish, but:

  • every call counts against your API key's rate limit, while a stream is one request however long the job takes;
  • you learn the job has finished only at your next call, not the moment it does;
  • choosing the interval is a guess: too short and you burn requests, too long and you wait for nothing.

The stream has none of these problems, and it is what the SDKs use.

With the SDKs​

The TypeScript SDK, @toolpath/api, does all of the above in waitForJob. It resolves with the finished job when it succeeds, throws a JobFailedError carrying the job when it fails, and calls onUpdate with every job event along the way:

import { createToolpathClient, uploadToPresignedUrl, waitForJob } from '@toolpath/api'

const client = createToolpathClient({ apiKey: process.env.TOOLPATH_API_KEY! })
const part = await client.parts.createPart({ filename: 'bracket.step' })
await uploadToPresignedUrl(part.uploadUrl, cadFileBytes)

const processing = await client.parts.updatePart({ id: part.partId })
await waitForJob(client, processing.jobId, {
onUpdate: (job) => console.log(job.status, job.progress),
})
const analyzed = await client.parts.getPart({ id: part.partId, jobId: processing.jobId })

The Python SDK in the Toolpath SDK repository has the same helper, as wait_for_job:

import asyncio

from toolpath import AuthenticatedClient, wait_for_job


async def main() -> None:
client = AuthenticatedClient("https://api.toolpath.com", token=api_key)
job = await wait_for_job(client, job_id, on_update=lambda job: print(job.status, job.progress))


asyncio.run(main())

SDK usage examples uses both in complete programs.

Reconnecting​

By default each helper reads one stream and rejects if it ends early. Pass reconnect to have it open the stream again itself: when the stream ends early, when the connection fails, and on a 429, after its Retry-After, or a 5xx. A refused key (401, 403) or a job the key cannot reach (404, 410) still ends the wait, since asking again will not change it. reconnect: true (reconnect=True in Python) waits 1, 2, 4, then every 8 seconds between streams, starting over after a stream that sent the job.

These options tune it. TypeScript takes milliseconds and Python, in WaitForJobReconnect, seconds:

  • delaysMs (delays): the waits before each new stream; the last repeats.
  • giveUpAfterMs (give_up_after): gives up once reconnecting has failed for that long, after one last attempt.
  • jitter: a fraction from 0 to 1. Each wait varies by up to that much, so streams lost together do not reconnect together. A delay may shrink or grow by it; a Retry-After wait only grows, so it is spread too but never shorter than the server asked. Off by default.
  • stableMs (stable_after): starts the delays over only after a stream that stayed open that long, rather than after any stream that sent the job, so a stream that opens and drops at once still backs off.
  • quietMs (quiet_after): treats a stream that has sent nothing for that long, keep-alives included, as lost, and opens another. The server sends a keep-alive every 15 seconds, so 45 seconds is a sensible value. TypeScript counts it from when the response arrives. Python's quiet_after is the stream's read timeout, so it also bounds the wait for the response's headers: a timeout there is a stream that never opened, backed off with no 'lost' and no look.
  • lookOnLoss (look_on_loss): reads the job once with GET /v1/jobs/{id} when a stream ends or is lost early. A final job settles the wait. A 401, 403, 404, 410 or any other 4xx but a 429 ends it, while a 429, a 5xx or a failed connection goes on reconnecting, waiting the look's Retry-After where it is longer than the next delay. Off by default.

A jitter outside 0 to 1, or not finite, is refused before anything is requested: a RangeError in TypeScript, a ValueError in Python.

The client argument may also be a function that returns the client. It is called before each stream and each look, so a long wait can switch to another key. A refused key still ends the wait: whether to start again with a different key is your decision. An error from the function itself also ends the wait, and is not retried; in TypeScript, a call still pending sees signal only once it settles. Keep one client per key and return it, rather than making one each call: in Python each client holds an httpx.AsyncClient that stays open until it is closed.

onStream (on_stream) tells you when a stream is listening: 'open' each time a stream's response arrives, and, with reconnect, 'lost' when an open stream ends, drops or goes quiet before the job is final, before any look and the wait to reconnect. It is not called for a stream that could not be opened, nor when the wait settles or is stopped.

Running a server that watches many jobs​

A server that keeps a wait open for every job its users are watching holds many streams at once, and they tend to fail together: a deploy or a network blip drops them all in the same second. Turn on all four of jitter, stableMs, quietMs and lookOnLoss there. Jitter spreads the reconnects out, stableMs backs off a stream that keeps opening and dropping, quietMs notices a connection that died without closing, and lookOnLoss settles a job that finished while its stream was down without opening another. onStream tells the server which jobs are between streams, for example to read them itself when someone asks meanwhile:

// One client per key, made once: the function is called before every stream and look.
const clients = new Map<string, ToolpathClient>()
const clientFor = (apiKey: string): ToolpathClient => {
let client = clients.get(apiKey)
if (!client) {
client = createToolpathClient({ apiKey })
clients.set(apiKey, client)
}
return client
}

const job = await waitForJob(() => clientFor(currentApiKey()), jobId, {
reconnect: { jitter: 0.2, stableMs: 30_000, quietMs: 45_000, lookOnLoss: true },
signal,
})
# One client per key, made once and closed at shutdown: each holds an open httpx.AsyncClient.
clients: dict[str, AuthenticatedClient] = {}


def client_for_current_key() -> AuthenticatedClient:
api_key = current_api_key()
if api_key not in clients:
clients[api_key] = AuthenticatedClient("https://api.toolpath.com", token=api_key)
return clients[api_key]


job = await wait_for_job(
client_for_current_key,
job_id,
reconnect=WaitForJobReconnect(jitter=0.2, stable_after=30, quiet_after=45, look_on_loss=True),
)

Without an SDK​

Any HTTP client that can read a response as it arrives can read the stream. To watch one from a terminal, curl -N turns off buffering:

curl -N -H "Authorization: Bearer $TOOLPATH_API_KEY" \
https://api.toolpath.com/v1/jobs/<jobId>/events

In a browser, use fetch, not EventSource. EventSource cannot send an Authorization header, so it cannot authenticate to the API. Read the fetch response body with an SSE parser instead. This one uses version 3 of eventsource-parser (npm install eventsource-parser@3) and runs in a browser or in Node.js 20 or newer. Like the Python reader below, it gives up after a minute with nothing read, because a connection that drops silently would otherwise leave read() waiting forever:

import { createParser } from 'eventsource-parser'

type Job = { status: string; progress: number | null; error: string | null }

// The stream sends a keep-alive comment every 15 seconds while the job runs, so a minute with
// nothing read means the connection is gone.
const IDLE_TIMEOUT_MS = 60_000

const waitForJob = async (jobId: string, apiKey: string): Promise<Job> => {
const idle = new AbortController()
let timer = setTimeout(() => idle.abort(), IDLE_TIMEOUT_MS)
try {
return await readJobStream(jobId, apiKey, idle.signal, () => {
clearTimeout(timer)
timer = setTimeout(() => idle.abort(), IDLE_TIMEOUT_MS)
})
} catch (error) {
if (idle.signal.aborted) {
throw new Error('The connection to the job event stream was lost')
}
throw error
} finally {
clearTimeout(timer)
}
}

const readJobStream = async (
jobId: string,
apiKey: string,
signal: AbortSignal,
onRead: () => void,
): Promise<Job> => {
const response = await fetch(`https://api.toolpath.com/v1/jobs/${jobId}/events`, {
headers: { Authorization: `Bearer ${apiKey}`, Accept: 'text/event-stream' },
signal,
})
if (!response.ok || !response.body) {
throw new Error(`Could not open the job's event stream: HTTP ${response.status}`)
}

const jobs: Array<Job> = []
const parser = createParser({
onEvent: (event) => {
if (event.event === 'job') {
jobs.push(JSON.parse(event.data))
}
},
})
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader()
// Not every fetch errors the body on abort, so cancel the reader too: read() then settles.
signal.addEventListener('abort', () => void reader.cancel(), { once: true })
for (;;) {
const { done, value } = await reader.read()
onRead()
if (done) {
throw new Error('The connection to the job event stream was lost')
}
parser.feed(value)
const job = jobs.at(-1)
if (job?.status === 'succeeded') {
await reader.cancel()
return job
}
if (job?.status === 'failed') {
await reader.cancel()
throw new Error(job.error ?? 'The job failed')
}
}
}

A key used from a browser is readable by anyone who opens the page. Restrict it to your site's origins, as Authentication describes.

In Python, read the stream's bytes with httpx and split them into lines at each newline. Each job event's JSON arrives on a single data: line. Don't use iter_lines() here: it also splits at Unicode line separators, which can appear inside the JSON.

import json

import httpx


def wait_for_job(job_id: str, api_key: str) -> dict:
url = f"https://api.toolpath.com/v1/jobs/{job_id}/events"
headers = {"Authorization": f"Bearer {api_key}", "Accept": "text/event-stream"}
# The stream stays open for as long as the job runs, sending keep-alive comments while it
# does, so a minute with nothing read means the connection is gone (httpx.ReadTimeout).
with httpx.stream("GET", url, headers=headers, timeout=httpx.Timeout(10, read=60)) as response:
response.raise_for_status()
event = None
pending = b""
for chunk in response.iter_bytes():
*lines, pending = (pending + chunk).split(b"\n")
for raw in lines:
line = raw.removesuffix(b"\r").decode()
if line.startswith("event:"):
event = line.removeprefix("event:").strip()
elif line.startswith("data:") and event == "job":
job = json.loads(line.removeprefix("data:"))
if job["status"] == "succeeded":
return job
if job["status"] == "failed":
raise RuntimeError(job["error"])
elif line == "":
event = None
raise RuntimeError("The connection to the job event stream was lost")