Rate-Limit Friendly Batch Jobs: A Practical Design for Reliable API Automations

Most “simple” automations start the same way: you need to pull a bunch of records from one system, transform them a bit, then push them into another system via an API. It works fine in testing, then fails in production when the dataset grows, the API gets slower, or you hit a rate limit.

The goal is not to build a complicated distributed system. The goal is to build a batch job that is predictable: it runs to completion, it can be restarted safely, and it behaves politely toward the API it depends on.

This post describes a practical design you can implement with almost any stack. It focuses on the mechanics that actually reduce pager noise: work partitioning, throttling, idempotency, and recovery.

Why batch jobs break (and why rate limits are only half the story)

Rate limits are the most visible failure mode, but batch jobs usually fail for more ordinary reasons:

• Time: a job that used to finish in 10 minutes now takes 2 hours, and a deployment, server restart, or scheduled window interrupts it.
• Partial failure: item 4,237 fails due to a transient timeout, and the entire run aborts, leaving you unsure what was applied.
• Duplication: you re-run the job to “fix it” and accidentally create duplicates or overwrite newer data.
• Invisible progress: logs say “processing…” but you cannot answer “how many left?” or “where did it stop?”

A rate-limit friendly design addresses all of these by treating the batch job as a series of small, restartable transactions, not a single fragile marathon.

Define the job contract first

Before you choose queue tech, cron, GitHub Actions, or a server, write a one-page “job contract”. This prevents scope creep and clarifies what “done” means.

Include:

• Inputs: what set of records should be processed (example: “all orders updated since last successful run”).
• Outputs: what the job produces (example: “create or update invoices in the accounting system”).
• Correctness rules: what must be true after completion (example: “every source order has exactly one target invoice; status fields match mapping”).
• Constraints: rate limits, payload size limits, and any daily quotas.
• Failure policy: whether to stop on first error, skip and continue, or retry with limits.

A job contract turns an automation into a maintainable system. It also makes it easier to explain behavior to non-engineers when something goes wrong.

Partition work into restartable chunks

The single biggest improvement you can make is to stop thinking in terms of “run the whole thing” and instead process chunks. A chunk is small enough to retry and large enough to be efficient.

Real-world example: nightly invoice sync

Imagine a small ecommerce operation that wants a nightly job to sync orders into an accounting tool. Some days there are 50 orders, some days 5,000. The accounting API allows 60 requests per minute and occasionally times out.

A robust design would:

• Fetch orders updated since the last successful checkpoint, in pages (example: 200 at a time).
• For each order, build an “upsert invoice” operation with an idempotency key.
• Process in batches with a deliberate throttle and bounded retries.
• Persist progress after each page (or after each invoice) so a restart continues, not repeats.

One simple way to model this is to treat each run as a run record with a cursor. The cursor could be a timestamp, an incrementing ID, or a page token from the source API. Persist it somewhere durable.

{
  "runId": "2026-08-14T02:00Z",
  "status": "running",
  "sourceCursor": "updated_at>=2026-08-13T02:00Z",
  "pageCursor": "token_17",
  "processedCount": 3400,
  "errorCount": 3,
  "lastHeartbeat": "2026-08-14T02:21Z"
}

This structure is intentionally boring. The key is that your job is now resumable. If it stops, you can restart using the last persisted cursor.

Throttle on purpose: a simple strategy that stays polite

Many batch jobs “throttle” accidentally by being slow. That fails the moment you optimize something or add concurrency. Instead, implement an explicit limiter and make it part of the job contract.

A practical throttling approach

• Set a target request rate lower than the published limit (example: 45 per minute when the limit is 60). Leave headroom for retries and for other systems using the same API key.
• Bound concurrency. Even with a limiter, too much parallelism creates bursts that trigger short-window limits.
• Back off on 429 and 5xx. Treat “too many requests” and transient server errors as a signal to slow down temporarily.
• Prefer steady pacing over bursts. Smooth traffic is easier on APIs and more predictable for you.

Keep the throttle configuration easy to change without a redeploy if you can. Limits change, and you want to respond quickly.

Idempotency and deduplication: the difference between safe and scary

If you take only one concept from this post, take this: a batch job must be safe to run twice. The world will force you to re-run it. Someone will click “retry,” a server will reboot, credentials will rotate mid-run, or a network blip will happen.

To make re-runs safe, you need one (or both) of these:

• Idempotency keys: a unique key per operation that the target API uses to deduplicate creates.
• Upserts: you create-or-update a target record based on a stable external identifier.

Even if the API does not support an official idempotency feature, you can often simulate it by storing a mapping table: source ID to target ID. On each run, look up the mapping first, then update if it exists, create if it does not, and store the mapping atomically when creation succeeds.

Also decide what counts as “the same operation.” A good idempotency key usually includes:

• Source system name
• Source record ID
• Operation type (create-invoice, update-invoice)
• Optional version marker (like the source updated_at) if updates are meaningful

Observability and recovery: make partial success boring

When a job fails at 80 percent, your first question should not be “what happened?” It should be “which items failed, and what is the next safe action?” You get there by recording structured progress.

At minimum, track:

• Run status: running, succeeded, failed, cancelled.
• Cursor: what you last completed (page token, timestamp, ID).
• Counts: processed, succeeded, failed, retried.
• Error samples: store the first N unique error messages with context (source ID, endpoint, status code).
• Heartbeat: a periodic “still alive” marker to detect stuck runs.

Recovery then becomes a decision tree:

1. If the failure is transient (timeouts, 429, 5xx), restart from the last persisted cursor.
2. If the failure is deterministic (validation errors, missing required fields), export the failed items to a review list and continue with the rest.
3. If the failure indicates a contract change (API schema change), stop and fix the integration, then re-run safely thanks to idempotency.

A small but powerful practice is to implement a “dry run” mode that validates inputs and estimates work (how many records, expected requests) without writing to the target. This catches surprises early and helps you size your schedule window.

Common mistakes (and what to do instead)

• Mistake: one giant transaction. If you only “commit” at the end, you guarantee pain. Do instead: commit per chunk and persist cursors often.
• Mistake: retrying forever. Infinite retries look like resilience but behave like outages. Do instead: bounded retries with a clear terminal error state.
• Mistake: concurrency without pacing. Parallel requests can create bursts that exceed short-window limits. Do instead: keep concurrency low and enforce a steady rate.
• Mistake: no idempotency. Re-runs create duplicates or corrupt state. Do instead: use idempotency keys, upserts, or a mapping table.
• Mistake: logs only. Plain text logs are hard to summarize and action. Do instead: store structured run metadata plus a small error ledger.

When not to use a batch job

Batch jobs are great for periodic syncing and backfills, but they are not the answer to every integration problem.

Consider alternatives when:

• You need near real-time updates. A periodic batch introduces inherent delay. Event-driven triggers or incremental webhooks can fit better.
• Source data is extremely large and changes constantly. You may need streaming, change data capture, or a dedicated ETL tool.
• The target API cost model punishes repeated reads. If you must poll expensively to discover changes, redesign around a change feed or push mechanism.
• You cannot make writes idempotent. If the target system cannot safely dedupe and you cannot implement your own mapping, re-runs become risky.

If you still choose batch, reduce risk by shrinking scope: sync fewer fields, fewer endpoints, and only the subset of records that truly needs automation.

Copy-paste checklist for your next automation

Use this as a build spec you can share with your team.

• Job contract
  • Defined input set (how to find records)
  • Defined output actions (create, update, upsert)
  • Correctness rules and “done” definition
• Chunking
  • Choose a cursor type (timestamp, ID, page token)
  • Persist cursor and run status in durable storage
  • Commit progress at least once per page or per N items
• Rate limiting
  • Set target rate below published limit
  • Bound concurrency (start small)
  • Backoff on 429 and transient failures
• Safety
  • Idempotency keys or upsert based on stable external ID
  • Mapping table if needed (source ID to target ID)
  • Bounded retries with terminal failures recorded
• Visibility
  • Run metrics: processed, succeeded, failed, retried
  • Error ledger with context (which record, which endpoint)
  • Heartbeat to detect stuck runs
• Operations
  • Manual restart procedure (what to do, what not to do)
  • Optional dry-run mode for validation and estimation
  • Clear alerting threshold (example: “failures > 1%”) if you have alerts

Conclusion

Reliable API batch jobs are less about clever code and more about disciplined structure. If you partition work, throttle deliberately, and make writes idempotent, most failures become routine: you restart, it resumes, and nothing breaks.

The payoff is compounding. The same design supports backfills, migrations, and new integrations, without reinventing safety mechanisms each time.

FAQ

How big should a chunk be?

Pick a chunk size that completes in a predictable time window and can be retried without pain. Many teams start with 100 to 500 items per page, then adjust based on API latency and payload limits. The “right” size is the one that keeps retries fast and progress frequent.

Should I stop the whole job when one item fails?

Usually no. For large batches, it is better to record per-item failures, continue processing, and produce a concise failure report. Stop the whole job when the error indicates a systemic issue, like invalid credentials, a breaking schema change, or a consistently failing endpoint.

What if the API rate limit is undocumented or inconsistent?

Start with a conservative fixed rate and observe responses. If you see 429s, slow down and add backoff. Keep your limiter configuration easy to change, and prefer steady pacing rather than bursts.

Do I need a queue to do this well?

Not necessarily. Many reliable jobs are a single scheduled process with persisted cursors and a limiter. A queue becomes useful when you need to distribute work across workers, isolate failures per item more strongly, or support user-triggered on-demand runs.