Errors, retries and writing a durable integration

The API uses conventional HTTP status codes. The distinction that matters most when writing automation is between failures you should retry and failures you should not, because getting it backwards produces either a silent gap or a self-inflicted outage.

A 401 means the token is invalid or expired. Do not retry — retrying a bad credential in a loop achieves nothing and looks exactly like an attack in the login-attempt record. Fail loudly instead so somebody fixes the token.

A 403 means authenticated but not permitted. This is a scoping problem rather than a transient one, and it is often correct: an operator token scoped to three clients querying a fourth should be refused. Do not retry.

A 404 means the resource does not exist, which in a long-running integration usually means something was deleted or reassigned since you last ran. Handle it as a normal outcome rather than an exception; estates change.

A 5xx or a network failure is the retry case. Back off exponentially with a cap, and give up after a bounded number of attempts rather than retrying forever.

# Bounded retry with backoff — the minimum any scheduled integration should do
attempt=0
until curl -fsS "$AEGISONE_HOST/api/agents" \
        -H "Authorization: Bearer $AEGISONE_TOKEN" -o agents.json; do
  attempt=$((attempt + 1))
  [ "$attempt" -ge 5 ] && { echo "giving up after $attempt attempts" >&2; exit 1; }
  sleep $((2 ** attempt))
done

Two habits separate integrations that age well from ones that need rewriting. First, make your integration idempotent — if it runs twice because a scheduler misfired, the second run should be harmless. Second, never let a failure be silent. An integration that has been failing quietly for three weeks is worse than one that never existed, because everyone downstream has been trusting data that stopped arriving.

Be conservative with request volume in loops. Insert a small delay between iterations, particularly for agent-directed calls, which do real work on real machines.