> ## Documentation Index > Fetch the complete documentation index at: https://formbricks.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Job Runner > Configure and operate the BullMQ background-job worker in self-hosted Formbricks Formbricks uses a shared BullMQ Job Runner for asynchronous work such as response processing, survey scheduling, and [Workflows](https://formbricks.com/docs/workflows/overview.md). Formbricks starts the worker inside the web application by default, so a standard self-hosted deployment does not need a separate worker container. > [!NOTE] > > Every Formbricks web process starts one worker with a concurrency of one by default. You can tune these > values with environment variables after you have measured your workload. ## Runtime architecture The web application creates jobs and consumes them from a shared Redis-backed queue: * Queue name: `background-jobs` * Redis key prefix: `{formbricks:jobs}` * Worker startup: Next.js server instrumentation * Default workers per web process: `1` * Default concurrent jobs per worker: `1` Redis is also used to coordinate recurring jobs. The web process retries worker and recurring-job registration after 30 seconds if startup fails. > [!NOTE] > > Changing a recurring job's cron pattern, interval or time zone — for example by setting > `SURVEY_SCHEDULING_TIME_ZONE` — updates the schedule as soon as the process restarts, but takes effect from > the **following** run: the run already queued keeps its previous timing. For the daily sweeps that can be up > to 24 hours. Inspecting the schedule in Redis shows the new value immediately, so a job firing once more at > the old hour after a restart is expected rather than a sign the setting was ignored. ## Current background jobs Current workloads on the shared queue include: | Workload | Job names | Purpose | | - | - | - | | Response pipeline | `response-pipeline.process` | Processes asynchronous response pipeline events and fans out one delivery job per webhook. | | Webhook delivery | `webhook-delivery.process` | Delivers one response event to one webhook, with its own retry budget. | | Survey scheduling | `survey-scheduling.reconcile` | Reconciles scheduled survey state. | | Survey archive | `survey-archive-purge.process` | Permanently deletes surveys past the archive retention window. | | Usage telemetry | `usage-telemetry.process` | Sends the instance's daily usage update. | | Workflows | `workflow-run.process`, `workflow-run.reconcile` | Executes workflow runs and recovers stalled runs. | These are current examples rather than a fixed list. Future Formbricks features can use the same Job Runner for additional background work. ## Configure the Job Runner **1. Connect Redis** Set `REDIS_URL` to a Redis instance reachable by every Formbricks web replica. ```env theme={null} REDIS_URL=redis://redis:6379 ``` All replicas must use the same Redis instance so they share the BullMQ queue and recurring schedules. **2. Keep the in-process worker enabled** `BULLMQ_WORKER_ENABLED` is enabled by default outside test environments. You can omit it or set it explicitly: ```env theme={null} BULLMQ_WORKER_ENABLED=1 BULLMQ_WORKER_COUNT=1 BULLMQ_WORKER_CONCURRENCY=1 ``` Restart every Formbricks web process after changing these values. **3. Verify startup** Check the web process logs for the exact message `BullMQ runtime started`. Its structured fields include the queue name, Redis prefix, worker count, and worker concurrency. ## Scale across replicas Each web replica starts its own in-process workers and connects to the same queue. BullMQ coordinates job claims so one available worker ordinarily processes a queued job. Total processing capacity is approximately: ```text theme={null} web replicas x BULLMQ_WORKER_COUNT x BULLMQ_WORKER_CONCURRENCY ``` Increase concurrency gradually and monitor database, Redis, and downstream service capacity. Adding replicas also adds workers, so account for both scaling controls. Every response now produces one job per matching webhook in addition to the pipeline job itself, and each delivery holds a worker slot for up to `WEBHOOK_DELIVERY_TIMEOUT_MS` (5 seconds by default). Because those jobs share the queue's concurrency, the deliveries for one response run **as concurrency allows rather than all at once**: at the default of one, ten webhooks are delivered one after another, so the last receiver is contacted later than it would have been when the pipeline job posted to all of them itself. Raise `BULLMQ_WORKER_CONCURRENCY` if your workspaces have several webhooks per survey, or receivers that are slow to answer — otherwise a burst of deliveries also delays pipeline and workflow jobs behind it. Each concurrent slot needs a database connection, so raise the connection pool alongside it. ## Retry behavior BullMQ applies the following defaults to background jobs: | Behavior | Current default | | - | - | | Attempts | 3 | | Backoff | Exponential, starting at 5 seconds | | Completed job retention | Up to 24 hours or 1,000 jobs | | Failed job retention | Up to 7 days or 5,000 jobs | Individual workloads can add their own recovery behavior on top of these queue-level retries. Webhook deliveries override the queue defaults with **5 attempts** and an **exponential backoff starting at 30 seconds** (retries after roughly 30 seconds, 1, 2 and 4 minutes). A receiver that returns a 4xx status other than 408 or 429, or a webhook URL that fails the SSRF policy, fails the job immediately without retrying; network errors, timeouts, DNS failures, 408, 429 and 5xx statuses are retried. Deliveries to a webhook that was deleted or unsubscribed from the event while the job was queued complete without a request. Redirects depend on `DANGEROUSLY_ALLOW_WEBHOOK_INTERNAL_URLS`. Left unset — the default, and the safe setting — a 3xx response is refused rather than followed, and the attempt is retried in case the receiver's configuration is fixed. With the variable set, redirects are followed instead: note that following one turns the POST into a GET, which many receivers answer with `405`, and a 405 is then treated as a permanent rejection. ## Workflows Workflows use the shared Job Runner to execute runs and reconcile runs that lost their queue or worker handoff. No additional worker process is required. ### Configure action dependencies Configure the services required by the workflow actions you use. For example, the **Send email** action uses your deployment's SMTP transport and sender. Configure `SMTP_HOST`, `SMTP_PORT`, `MAIL_FROM`, `MAIL_FROM_NAME`, and any authentication or TLS variables required by your provider. See [SMTP Configuration](https://formbricks.com/docs/self-hosting/configuration/smtp.md) for the complete setup. ### Retry and recovery behavior > [!WARNING] > > Workflow delivery supports retries and recovery; it is not an exactly-once guarantee. Formbricks uses > deterministic job IDs and guarded workflow run and step transitions to make redispatch safe, but operators > should still expect jobs to be retried after transient failures. A recurring reconciliation job runs every three minutes to recover workflow runs that lost their queue or worker handoff: * Queued runs older than two minutes are eligible for redispatch. * Queued runs older than 24 hours are marked as failed instead of being redispatched indefinitely. * Running runs with no executor activity for one hour are marked as failed and unfinished steps are skipped. * Each reconciliation sweep processes at most 250 queued and 250 running candidates. Larger backlogs drain over subsequent sweeps. When reconciliation changes runs, Formbricks writes `Workflow run reconciliation acted` with structured counts for redispatched, aged-out, and recovered runs. ### Troubleshoot workflow jobs If workflow runs stay queued after the Job Runner is healthy: 1. Search for `BullMQ job failed` entries where `jobName` is `workflow-run.process` or `workflow-run.reconcile`. 2. Search for `Workflow run reconciliation acted` and inspect the queued and stuck-running counts. 3. Open the workflow's **Runs** view to inspect the run status and step-level error. ## Use a custom external worker The default in-process worker is the supported setup. Formbricks does not ship a standalone worker command or container. The external-worker flags are only for custom deployments that provide their own consumer and register the complete Formbricks application job handlers. In such a custom deployment, configure the web/API processes to keep enqueueing without starting local workers: ```env theme={null} BULLMQ_WORKER_ENABLED=0 BULLMQ_EXTERNAL_WORKER_ENABLED=1 REDIS_URL=redis://redis:6379 ``` > [!WARNING] > > `BULLMQ_EXTERNAL_WORKER_ENABLED=1` only keeps queueing enabled. It does not provide, start, or deploy a > worker. Only use this configuration after your custom consumer registers the complete Formbricks job > handlers and is already consuming the same queue. Otherwise, background jobs will remain queued. ## Troubleshoot the Job Runner Formbricks writes structured logs to stdout and stderr. Search the `msg` field for these exact messages: | Log message | What to check | | - | - | | `BullMQ runtime started` | The worker is connected. Verify its `workerCount`, `workerConcurrency`, `queueName`, and `prefix` fields. | | `BullMQ Redis connection error` | Verify `REDIS_URL`, DNS, network access, Redis credentials, and TLS settings. | | `BullMQ queue error` | The producer queue emitted an error. Review the attached `err` field; usually the same causes as above. | | `Failed to start BullMQ runtime` | Review the attached `err` field, then confirm Redis is reachable from the web process. | | `BullMQ worker registration failed` | Worker startup failed. Formbricks schedules another attempt after 30 seconds. | | `BullMQ worker registration retry scheduled` | A startup retry is pending. Fix the underlying Redis or configuration error before it runs. | | `BullMQ recurring job registration failed` | Recurring schedules could not be registered. Check Redis connectivity. | | `BullMQ recurring job registration retry scheduled` | A recurring-schedule registration retry is pending for 30 seconds. | | `BullMQ job failed` | Inspect `jobName`, `jobId`, `attemptsMade`, and `err`. The job may retry until all its attempts are used (three by default, five for `webhook-delivery.process`). | | `Webhook delivery failed; retry scheduled` | One webhook delivery attempt failed and BullMQ will retry it. Inspect `webhookId`, `webhookUrlHost`, `statusCode`, `reason`, and `attempt` of `maxAttempts`. Usually the receiver is down or slow; nothing to do unless it keeps failing. | | `Webhook delivery failed; retries exhausted` | The last attempt for a webhook delivery failed; the event was not delivered. Same fields as above. Check the receiver, then re-send from the source system if needed — Formbricks does not replay exhausted deliveries. | | `Webhook delivery failed permanently` | The receiver rejected the request (4xx) or the webhook URL failed the SSRF policy, so no retry was attempted. Fix the receiver or the webhook URL; `reason` names the cause. | | `Webhook delivery skipped: webhook deleted` | The webhook was deleted after the event was queued. Expected after deleting a webhook with deliveries in flight. | | `Webhook delivery skipped: webhook no longer subscribed to this event` | The webhook's triggers or surveys were changed after the event was queued. Expected after editing a webhook with deliveries in flight. | | `Webhook delivery could not load the webhook and will be retried` | The database refused the webhook lookup, so no request was made and the attempt is retried. Check database health and the connection pool; the companion message names pool exhaustion explicitly. | | `Dropping BullMQ job with no registered processor: its schedule outlived the code that handled it` | A schedule in Redis outlived the build that handles it — typically after a rollback to a version without that feature. The job is completed rather than retried, so it does not accumulate in the failed set. Redeploy the build that owns `jobName`, or remove the stale schedule from Redis. | | `Dropping BullMQ job with no registered processor` for `webhook-delivery.process` | Seen only while upgrading to the first version that fans webhook delivery out into its own job: replicas still on the older build share the queue but do not know the name, so they complete the job **without delivering it** — the event is not retried and cannot be replayed. It stops as soon as every replica runs the new build. Upgrade during a quiet period if losing a few minutes of webhook events matters, or scale to a single replica for the upgrade. | If background jobs stay queued: 1. Confirm `REDIS_URL` is present and identical on all web replicas. 2. Confirm at least one in-process worker is enabled, or that a compatible external worker is running. 3. Look for `BullMQ runtime started` after the latest deployment or restart. 4. Check Redis connection and `BullMQ job failed` logs. 5. Use the `jobName` field to identify the affected workload and continue in its feature-specific logs or UI. When OpenTelemetry metrics are enabled (`PROMETHEUS_ENABLED=1` or `OTEL_EXPORTER_OTLP_ENDPOINT`), webhook deliveries also emit `formbricks.webhook.delivery.total` (a counter with `outcome`, `event` and `status_class` attributes) and `formbricks.webhook.delivery.duration` (a histogram in milliseconds), so delivery failure rates and receiver latency can be graphed and alerted on without reading logs. See [Monitoring](https://formbricks.com/docs/self-hosting/setup/monitoring.md) for how to scrape them. See [Environment Variables](https://formbricks.com/docs/self-hosting/configuration/environment-variables.md) for every Job Runner setting and [Monitoring](https://formbricks.com/docs/self-hosting/setup/monitoring.md) for log collection guidance. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.