The File-Based Job Queue: Reliable API Automations Without Extra Infrastructure

Most “small” automations start the same way: a script runs on a schedule, calls an API, and writes results somewhere. It works for weeks, then you hit a bad network day, an API timeout, a rate limit, or a partial failure that leaves your data half updated.

The usual response is to add more complexity: a message broker, a workflow engine, or a full background job system. Those tools are great, but they can be heavy for a small team, especially when the automation is the only thing that needs the new infrastructure.

A middle path exists: treat work items as durable files on disk, then process them with a simple worker loop. You get reliability patterns like retries, audits, and re-runs while keeping the operational footprint small.

Why small automations fail (and what a queue fixes)

One-shot scripts tend to combine “decide what to do” and “do it” in a single run. When something goes wrong, you often have no clean way to resume. You either re-run the whole script and risk duplicates, or you manually patch the data.

A job queue separates the concerns:

• Enqueue: create durable job records that describe the work to do.
• Process: a worker reads jobs and attempts them safely.
• Record outcomes: success, failure, retries, and timestamps are captured.

This separation creates practical benefits even if you only have one worker on one machine:

• Durability: if your script crashes mid-run, jobs are still there.
• Re-runs: you can retry failed jobs without redoing everything.
• Audit trail: you can answer “what happened?” without guesswork.
• Back-pressure: you can slow processing to respect rate limits.

The pattern: an append-only job folder

The core idea is simple: every unit of work is a file in a directory. To enqueue work, you write a new file. To process work, you move files between folders that represent states (pending, in-progress, done, failed).

You can store job files as JSON or another structured format. Keep them small, stable, and easy to inspect with basic tools.

A minimal job shape

A job should describe what to do and provide enough context to make it safe to retry. The two fields that matter most are an idempotency key and a retry counter.

queue/
  pending/
  processing/
  done/
  failed/

job file (JSON-like):
{
  "jobId": "2026-08-24T020001Z-00123",
  "type": "sync_invoice",
  "idempotencyKey": "invoice:INV-10492",
  "payload": { "invoiceId": "INV-10492" },
  "attempt": 0,
  "maxAttempts": 5
}

Why the idempotency key matters: it is the stable identifier for “this operation should happen at most once,” even if you attempt it multiple times. Your worker uses it to avoid duplicates and to detect already-completed work.

Why append-only matters: enqueue should only add new job files, never rewrite old ones. Rewrites are where corruption and confusing history tend to appear.

Processing jobs safely: idempotency, retries, and state

The worker loop can be very small conceptually: pick the next pending job, move it to processing, attempt the operation, then move it to done or failed. But the reliability comes from a few disciplined rules.

Rule 1: make state transitions atomic

Whenever possible, use a filesystem move or rename as your “atomic” action. The move from pending/ to processing/ is your lock. If your worker crashes, you can recover by inspecting the processing/ folder and deciding which items to re-queue.

Rule 2: design for safe re-runs

Retries are only safe if the operation is idempotent, or effectively idempotent with safeguards. There are several ways to do this without complex infrastructure:

• Write-once destination: if the destination has a unique key (for example, an external record ID), perform upsert operations rather than inserts.
• Local receipt log: maintain a simple local “receipts” store mapping idempotencyKey to outcome. Before executing, check if the job already succeeded.
• External idempotency support: if an API supports idempotency keys, pass your key through consistently.

Even if you can only implement one safeguard, choose a consistent idempotency key and record success receipts. That one step is often the difference between “we can re-run safely” and “we are afraid to touch it.”

Rule 3: retries should be limited and spaced out

A tight retry loop can turn a temporary outage into a self-inflicted rate-limit problem. Instead:

• Cap attempts with maxAttempts.
• Increase wait time between attempts (for example, 1 minute, 5 minutes, 15 minutes).
• Record why it failed (HTTP status, timeout, validation error) in a small sidecar note or in the job file when moving it to failed/.

In this pattern, a “retry later” mechanism can be as simple as adding a notBefore timestamp field and skipping jobs until that time.

A concrete example: nightly invoice sync

Imagine a small agency that needs invoices copied from a billing tool to an internal CRM. The initial script runs nightly and fetches “all invoices from the last 24 hours,” then posts them to the CRM. It works until:

• The billing API times out halfway through the list.
• The CRM API rejects one invoice due to a missing field.
• The script is re-run and duplicates appear in the CRM.

With a file-based queue, the flow becomes clearer and safer:

1. Enqueue step: list invoice IDs that need syncing and write one job per invoice into pending/. Each job has idempotencyKey = invoice:<invoiceId>.
2. Process step: the worker takes one job at a time, fetches the invoice details, then upserts into the CRM using the invoice ID as a unique key.
3. Receipt: on success, record that invoice:INV-10492 succeeded and move the job file to done/.
4. Failure handling: if the CRM rejects the record due to validation, move the job to failed/ and include a short note like “missing purchase order number” so a human can fix the data source or add a mapping rule.

The payoff is not speed, it is clarity. When someone asks “did INV-10492 sync?”, you can answer by checking the queue state and the receipt, instead of re-running a whole nightly script and hoping for the best.

Copyable checklist: implement in a weekend

If you want to introduce this pattern without boiling the ocean, use this checklist as your baseline.

• Define job types: write down 1 to 3 job types you need (for example, sync_invoice, sync_contact).
• Choose idempotency keys: one stable key per job, derived from business identifiers (invoice ID, customer ID).
• Create folders: pending/, processing/, done/, failed/.
• Implement enqueue: a routine that writes new job files only, never overwriting existing ones.
• Implement the worker loop: move one job at a time to processing/, attempt, then move to done/ or failed/.
• Add receipts: store a simple “completed keys” record so re-runs skip already-successful operations.
• Retry policy: set maxAttempts and a backoff schedule, and record failure reasons.
• Recovery rule: decide how to handle stale processing/ jobs (for example, older than 30 minutes gets moved back to pending/ with incremented attempt).
• Operational habit: review failed/ regularly and either fix the root cause or explicitly mark jobs as permanently failed.

If you do nothing else, implement idempotency keys plus receipts. That is the foundation that turns “retry” from risky into routine.

Common mistakes to avoid

• Using timestamps as the idempotency key: timestamps change across runs, so retries become duplicates. Use business IDs whenever possible.
• Batching too much into one job: a job should fail and retry independently. If one job represents 500 records, one bad record blocks everything.
• Not recording failure causes: if you only know “failed,” you will waste time reproducing and diagnosing. Capture a short, human-readable reason.
• Infinite retries: “retry forever” is how you accumulate silent debt. Cap attempts and surface failures.
• Multiple workers without coordination: this pattern can support multiple workers, but only if your locking is correct. Start with one worker until you need more throughput.

When not to use a file-based queue

This approach is intentionally lightweight, which means it has limits. Consider a more robust queueing system when:

• You need high throughput or strict latency: thousands of jobs per second, or near real-time processing with tight guarantees.
• You need distributed processing across many machines: shared filesystem coordination can become fragile depending on your environment.
• You need exactly-once semantics across multiple services: most teams can achieve “effectively once” with idempotency, but true exactly-once is a deeper problem.
• Jobs contain sensitive data: storing full payloads in plaintext on disk may not meet your security requirements. In that case, store only references and fetch sensitive details at runtime.

If any of these are true, you can still keep the same mental model, but you may want a dedicated queue or workflow tool. The good news is that your job schema and idempotency decisions will carry over.

Conclusion

A file-based job queue is not fancy, and that is the point. It gives small teams a durable, inspectable way to run automations with retries and re-runs, without adding a new always-on service just to push a few API calls through safely.

If you want to build confidence in your automation first, start with a single job type, one worker, and a receipt log. Once you trust the pattern, you can expand it gradually or migrate it to a more advanced system later.

For more posts on practical systems like this, browse the Archive or learn how this site is produced on the About page.

FAQ

Is a file-based queue safe if my worker crashes?

It can be, as long as you treat moving a job file between folders as the authoritative state transition. After a crash, you can inspect processing/ and either retry jobs that look stale or reconcile them using receipts and destination state.

How do I prevent duplicates if I re-run the enqueue step?

Generate job files with deterministic names or check for an existing receipt before enqueuing. The key idea is that “enqueue” should be repeatable without creating multiple independent jobs for the same idempotency key.

Where should I store receipts?

For a small setup, a simple local store works: a lightweight database file, a line-delimited log, or even a folder of “receipt files” keyed by idempotency key. The important property is quick lookup before executing a job.

Can I run more than one worker?

Yes, but do it only after your state transitions are correct. Multiple workers need a reliable lock mechanism, and the simplest lock is an atomic move from pending/ to processing/. If that move is not atomic in your environment, stick to one worker.

How big should each job be?

Small and focused. Ideally one external object per job (one invoice, one customer, one email). Small jobs make retries safer, failures easier to isolate, and progress more visible.