Skip to main content

Background Processes

TestPlanit uses several background workers and scheduled jobs to handle asynchronous tasks and improve application performance. This document explains how to set up and manage these processes.

Overview​

The application uses the following background processes:

  1. Notification Worker - Handles creating notifications and sending notification emails
  2. Email Worker - Processes email sending tasks
  3. Forecast Worker - Updates test case forecasting data
  4. Sync Worker - Synchronizes issues with external integrations
  5. Testmo Import Worker - Processes large Testmo JSON imports
  6. Elasticsearch Reindex Worker - Reindexes entities for search functionality
  7. Auto Tag Worker - Runs AI-powered automatic tagging on test cases and other entities
  8. Audit Log Worker - Persists audit log entries for user and system actions
  9. Budget Alert Worker - Evaluates the AI budget alert rules after AI usage
  10. Repo Cache Worker - Automatically refreshes expired code repository caches for QuickScript
  11. Magic Select Worker - Processes AI-powered test case selection jobs for test runs
  12. Webhook Dispatch Worker - Delivers outbound webhooks to subscribed external endpoints
  13. Webhook Outbox Worker - Polls the outbox table and fans events out to webhook configs
  14. Webhook Retention Worker - Daily purge of webhook delivery / dedup / outbox rows older than 30 days
  15. DataChangeLog Retention Worker - Daily purge of processed audit change-capture rows older than 30 days
  16. Audit Log Retention Worker - Daily archive or deletion of audit log entries older than AUDIT_LOG_RETENTION_DAYS (off unless set)
  17. Dataset Lease Sweep Worker - Reaps expired dataset-row leases (test-data reservations) roughly once a minute
  18. Alert Worker - Evaluates alert rules when results land, when runs complete, when automated executions fail and sessions complete, and on hourly and daily scheduled passes
  19. Webhook Actions Worker - Runs an issue-tracker webhook's actions (importing the issue, creating test runs, drafting test cases with AI) after the webhook has been answered
  20. Failure Triage Worker - Groups automated failures into failure patterns, asks the project's LLM for a root cause and releases recovered quarantines
  21. Risk Score Worker - Keeps every test case's risk score current after results land and in a nightly sweep
  22. Scheduler - Sets up recurring jobs (cron jobs)

Workers​

Notification Worker​

  • Creates in-app notifications
  • Sends immediate notification emails
  • Processes daily digest emails (scheduled at 8 AM daily)
  • Default concurrency: 5 (lightweight operations)
  • Location: workers/notificationWorker.ts

Email Worker​

  • Handles all email sending operations
  • Processes email queue jobs
  • Default concurrency: 3 (I/O-intensive)
  • Location: workers/emailWorker.ts

Forecast Worker​

  • Updates forecasting data for test cases
  • Runs scheduled updates (3 AM daily)
  • Runs the abandoned automated-run sweep every 15 minutes: closes incomplete automated runs that have been idle past the configured threshold (see Abandoned automation cleanup); off by default
  • Default concurrency: 5 (CPU-intensive but parallelizable)
  • Location: workers/forecastWorker.ts

Sync Worker​

  • Synchronizes issues with external integrations (Jira, Linear, etc.)
  • Handles individual issue refreshes and bulk project syncs
  • Default concurrency: 2 (I/O-intensive, API rate-limited)
  • Location: workers/syncWorker.ts

Testmo Import Worker​

  • Processes large Testmo JSON export files
  • Imports test cases, runs, and related data
  • Default concurrency: 1 (memory-intensive operations)
  • Location: workers/testmoImportWorker.ts

Elasticsearch Reindex Worker​

  • Reindexes entities into Elasticsearch for search functionality
  • Handles full and partial reindexing operations
  • Processes batches of documents for efficient indexing
  • Default concurrency: 2 (I/O-intensive, balanced for ES performance)
  • Location: workers/elasticsearchReindexWorker.ts

Supported Operations:

  • Full reindex of all entities
  • Selective reindex by entity type (repository-cases, test-runs, sessions, etc.)
  • Project-specific reindexing
  • Batch processing with configurable batch sizes

Auto Tag Worker​

  • Runs AI-powered automatic tagging on test cases and other entities
  • Processes tagging jobs triggered by content changes
  • Default concurrency: 3 (one per entity type)
  • Location: workers/autoTagWorker.ts

Audit Log Worker​

  • Persists audit log entries for user and system actions
  • High throughput, independent operations
  • Default concurrency: 10 (lightweight, independent writes)
  • Also runs the CDC correlation loop (Loop B): drains each database's DataChangeLog into the readable AuditLog; in multi-tenant mode it polls every configured tenant per cycle, re-reading the tenant list so runtime additions are picked up
  • Location: workers/auditLogWorker.ts

Budget Alert Worker​

  • Runs after each recorded AI usage or failure and evaluates the AI model alert rules (budget used, request error rate, provider rate limiting)
  • Default concurrency: 2
  • Location: workers/budgetAlertWorker.ts

Repo Cache Worker​

  • Automatically refreshes expired code repository caches used by QuickScript AI generation and Impact Analysis; for Impact Analysis repositories, each refresh also scans repository markers and recent commit messages for ticket keys into Code Pins, importing named tickets the project does not hold yet
  • Runs the manual ticket scans queued from the Impact Analysis settings page (Rescan Recent Commits and Scan Full History), writing progress into the config so the page can follow a long walk; walked commits are cached per branch so later scans read only what is new and a capped full walk resumes where it stopped
  • Runs a daily sweep (4 AM) to find configs with expired or missing caches and re-fetches from git
  • Only refreshes caches that have actually expired — configs with valid caches are skipped
  • Default concurrency: 1 (serial processing to avoid hammering git provider APIs)
  • Location: workers/repoCacheWorker.ts

Magic Select Worker​

  • Processes AI-powered test case selection (Magic Select) jobs in the background
  • Analyzes test cases against test run metadata using LLM batching
  • Reports batch progress and tracks truncated responses per batch
  • Default concurrency: 1 (each job involves multiple LLM batch calls)
  • Location: workers/magicSelectWorker.ts

Impact Analysis Worker​

  • Processes Impact Analysis jobs: fetches the commit diff, matches Code Pins, resolves tickets named in the commits to their linked cases, searches the case library by keyword, scores run history, and asks the LLM to rank the candidates those signals leave undecided
  • Reports a phase per step and persists the affected-test list to the analysis record
  • Default concurrency: 1 (each job can involve provider API calls and multiple LLM batches)
  • Location: workers/impactAnalysisWorker.ts

Failure Triage Worker​

  • Consumes the failure-triage BullMQ queue: a debounced job per test run whenever automated FAILURE/ERROR results arrive, a per-project backfill started from the project's settings, and on-demand classification jobs
  • Fingerprints each failure (exception type, message with volatile values masked, top application stack frames), attaches it to the matching FailureSignature in the project or creates one, recomputes the signature's occurrence/case/run counters, and flags signatures whose cases flip between pass and fail
  • When the project allows it and an LLM provider resolves for the failure_triage feature, asks the model to pick a root cause for each new pattern from the Root Causes catalogue
  • Releases quarantined cases that have passed the project's configured number of consecutive times since they were quarantined
  • Default concurrency: 2 (overridable via FAILURE_TRIAGE_CONCURRENCY)
  • Location: workers/failureTriageWorker.ts

Risk Score Worker​

  • Consumes the risk-score BullMQ queue: a debounced job per test run whenever manual or automated results arrive (rescoring just the cases whose results landed recently), a per-project recompute, and a nightly sweep over every project with risk scoring enabled
  • Scores each test case 0-100 from its failure history, flakiness, staleness, linked requirement criticality (including suspect links), and linked defects, using the project's configured component weights
  • Persists the score with a per-component breakdown; the debounced and nightly passes only write (and re-index) cases whose score or inputs actually changed, while the on-demand per-project recompute rewrites everything so configuration changes propagate
  • Default concurrency: 2 (overridable via RISK_SCORE_CONCURRENCY)
  • Location: workers/riskScoreWorker.ts

Webhook Dispatch Worker​

  • Consumes the webhook-dispatch BullMQ queue and POSTs each event to its subscribed endpoints
  • Honors per-config retry curve (0s, 30s, 5m, 30m, 2h, 6h, 12h) and writes a WebhookDelivery row per attempt
  • Caps response-body buffering to prevent a misbehaving consumer from exhausting worker memory
  • Default concurrency: 5 (overridable via WEBHOOK_DISPATCH_CONCURRENCY)
  • Location: workers/webhookDispatchWorker.ts

Execution Dispatch Worker​

  • Consumes the execution-dispatch BullMQ queue: starts the CI job for a requested automated execution (GitHub Actions workflow_dispatch, GitLab pipeline, or a signed generic webhook) and records the outcome on the TestRunExecution row
  • Never retries a dispatch (starting a job twice is worse than a recorded failure a person can retry); the scheduler's poll-active-executions job on the same queue asks each provider for job status every minute with a per-execution backoff, times out executions past their target's limit, and fails requests still PENDING after ten minutes so a stalled dispatch never blocks the run
  • Default concurrency: 3 (overridable via EXECUTION_DISPATCH_CONCURRENCY)
  • Location: workers/executionDispatchWorker.ts

Webhook Outbox Worker​

  • Polls WebhookOutboxEvent every 2s using FOR UPDATE SKIP LOCKED and fans matched events into the dispatch queue
  • Multi-tenant aware: in MULTI_TENANT_MODE=true it iterates every configured tenant per cycle and stamps tenantId onto each enqueued dispatch job
  • Multiple replicas can run concurrently without double-claiming
  • Location: workers/webhookOutboxWorker.ts

Webhook Retention Worker​

  • Wakes once per day and purges WebhookDelivery, WebhookEventDedup, and dispatched WebhookOutboxEvent rows older than 30 days
  • Multi-tenant aware: runs the purge against every configured tenant database per cycle and emits one audit row per (tenant, run)
  • Batched LIMIT 1000 deletes to avoid lock contention
  • Location: workers/webhookRetentionWorker.ts

DataChangeLog Retention Worker​

  • Wakes once per day and batch-deletes processed DataChangeLog rows (the audit change-capture substrate) older than 30 days
  • Never deletes unprocessed rows — the audit log worker must correlate them into AuditLog first; the append-only enforcement triggers (tpl_dcl_no_delete / tpl_dcl_no_update) enforce this at the database level too
  • Batched LIMIT 1000 deletes to avoid lock contention with the capture path; emits one DCL_RETENTION_PURGED audit event per run
  • Multi-tenant aware: runs the purge against every configured tenant database per cycle and emits one audit row per (tenant, run)
  • Standalone daily loop (no BullMQ queue) — self-schedules internally rather than via the scheduler
  • Location: workers/dataChangeLogRetentionWorker.ts

Audit Log Retention Worker​

  • Off unless AUDIT_LOG_RETENTION_DAYS (or a tenant's auditLogRetentionDays) is set; see Retention
  • Wakes once per day and removes AuditLog rows older than the window
  • archive mode (default): writes them to object storage as gzipped NDJSON, one calendar month (UTC) at a time, up to 25,000 rows per file. Each file is read back and its SHA-256 checked before anything is deleted; the file's manifest row and the deletion of its rows commit in one transaction. Emits one AUDIT_LOG_ARCHIVED event per run
  • delete mode: batched LIMIT 1000 deletes, nothing kept. Emits one AUDIT_LOG_PURGED event per run
  • Multi-tenant aware: applies each tenant's own retention per cycle and skips tenants with none; archives go under a per-tenant key prefix
  • Standalone daily loop (no BullMQ queue) — self-schedules internally rather than via the scheduler
  • Location: workers/auditLogRetentionWorker.ts

Dataset Lease Sweep Worker​

  • Wakes roughly once a minute and clears dataset-row leases (the test-data reservation API) whose TTL has lapsed
  • Observability + hygiene only — correctness comes from lazy expiry (the acquire query already treats an expired lease as free), so a delayed sweep never blocks reservations
  • Emits a dataset.row.released webhook (reason: expired) per reaped lease; a FOR UPDATE ... SKIP LOCKED re-check means a row re-leased before the sweep runs is simply skipped, never falsely released
  • Multi-tenant aware: sweeps every configured tenant database per cycle
  • Standalone polled loop (no BullMQ queue) — self-schedules internally rather than via the scheduler
  • Location: workers/datasetLeaseSweepWorker.ts

Alert Worker​

  • Evaluates alert rules: one job per recorded result (manual and automated), one deduplicated job per run when results arrive or the run completes, one job per failed or timed-out automated execution, one per completed session, an hourly job per tenant for stalled runs and webhook failure rates, and a daily job per tenant for the milestone, untested-share and project health metrics
  • Records a AlertEvent row per firing, notifies the rule's subscribers through the notification worker, and optionally writes a alert.triggered outbox event for webhooks
  • Default concurrency: 2 (ALERT_CONCURRENCY)
  • Location: workers/alertWorker.ts

Webhook Actions Worker​

  • Runs the actions of an issue-tracker inbound webhook, one job per delivery: refreshes the issue from the tracker, imports it when an Import Issues action takes it, and creates a test run for each Create Test Run action the event triggers
  • Each Generate Test Cases with AI action the event triggers gets a job of its own on the same queue (retried three times with backoff), so the model call never holds up the delivery's other actions; the drafts are persisted, linked to the issue, and the action's reviewers notified
  • A job whose issue is being synced by another process retries with backoff
  • Default concurrency: 2 (WEBHOOK_ACTIONS_CONCURRENCY)
  • Location: workers/webhookActionsWorker.ts

Scheduler​

  • Sets up recurring jobs using cron patterns
  • Configures daily digest emails (8 AM)
  • Configures forecast updates (3 AM)
  • Configures the nightly risk-score sweep (3:30 AM)
  • Configures code repository cache refresh (4 AM)
  • Configures the abandoned automated-run sweep (every 15 minutes)
  • Location: scheduler.ts

Running Workers​

Development Mode​

For development, you can run workers in the foreground:

# Run all workers together (foreground)
pnpm start:workers

# Or run individual workers
pnpm worker:notification
pnpm worker:email
pnpm worker:forecast

# Run scheduler separately
pnpm scheduler

Production Mode (Background Processes)​

For production environments, use PM2 to run workers as background daemons:

Start all workers​

pnpm pm2:start

This command:

  1. Runs the scheduler once to set up cron jobs
  2. Starts all three workers as background processes
  3. Automatically restarts workers if they crash

Stop all workers​

pnpm pm2:stop

Restart all workers​

pnpm pm2:restart

Check worker status​

pnpm pm2:status

View worker logs​

pnpm pm2:logs

# View logs for specific worker
pm2 logs notification-worker
pm2 logs email-worker
pm2 logs forecast-worker

Remove all workers from PM2​

pnpm pm2:delete

PM2 Configuration​

The PM2 configuration is defined in ecosystem.config.js. Each worker is configured with:

  • Automatic restart on failure
  • Per-worker max_memory_restart and --max-old-space-size tuned to that worker's footprint
  • Production environment variables

Worker memory tiers​

Every worker's --max-old-space-size is 75% of its max_memory_restart ceiling, leaving the rest for buffers and native memory. Most workers run with a 512 MB ceiling and a 384 MB old-space limit, which is sufficient for lightweight queue consumers; raise all of them at once with WORKER_MAX_MEMORY_RESTART (see Environment Variables). The following workers run with elevated ceilings because they load a heavier dependency tree at idle and/or carry larger payloads:

Workermax_memory_restart--max-old-space-sizeWhy
Sync Worker1G768MLoads integration adapters + Elasticsearch sync extensions
Testmo Import Worker4G (configurable)3072M (configurable)Streams and analyzes multi-GB Testmo JSON exports; the default 512 MB tier OOM-killed large imports. Override via TESTMO_IMPORT_MAX_MEMORY_RESTART for very large exports (e.g. 18G), host RAM permitting
Forecast Worker2G1536MRecomputes run/case forecasts over large historical result sets; the default 512 MB tier was OOM-killed under production data volumes
Elasticsearch Reindex Worker2G (configurable)1536M (configurable)A full reindex sweeps every project's cases/runs/sessions/issues/milestones and bulk-loads them into Elasticsearch; the default 512 MB tier OOM-killed full-DB reindexes mid-job — PM2 SIGKILLs the worker, BullMQ redelivers the same job, and it restarts from the top, so runs never finish indexing. Override via ELASTICSEARCH_REINDEX_MAX_MEMORY_RESTART for very large tenants (many projects / large run history), host RAM permitting
SCIM Access Recompute Worker2G1536MLoads the full ZenStack runtime to recompute User.access tiers from IdP group mappings; boots to ~1.4 GB RSS at idle, so the default 512 MB tier triggered a tight PM2 SIGINT/restart loop rather than a real OOM
Audit Log Worker3G2304MThe CDC correlation loop (Loop B) caches one raw Prisma client per tenant in multi-tenant mode — one Rust query engine per tenant, the same per-tenant footprint as the webhook outbox worker. Harmless headroom in single-tenant mode (a single client).
Webhook Dispatch Worker3G2304MLoads ZenStack runtime + ES sync services + audit log service; carries full test_run.completed payloads under concurrency=5; observed steady-state RSS in multi-tenant clusters sits near 1.9 GB
Webhook Outbox Worker3G2304MSame heavy dependency tree as dispatch; caches one Prisma client per tenant in multi-tenant mode
Webhook Retention Worker3G2304MIterates every tenant's database per pass; headroom protects against batched-delete loops on tenants with large retention backlogs. Tenant Prisma clients are disconnected after each pass to release Rust query engine buffers, so steady-state should drop substantially after the first few passes — these ceilings can be lowered once production telemetry confirms it.
DataChangeLog Retention Worker3G2304MLoads the audit log service plus the raw Prisma base client and runs a batched-delete loop over the high-volume change-capture table; headroom mirrors the webhook retention worker so a large purge backlog (e.g. after a heavy import) does not trip the ceiling mid-pass.
Dataset Lease Sweep Worker3G2304MSweeps every configured tenant's database on a ~60s loop to reap lapsed dataset-row leases; the per-tenant client footprint is the same as the retention workers'
Execution Dispatch Worker3G2304MDispatches automated-execution jobs and polls active executions per tenant each minute, caching one client per tenant in multi-tenant mode
Audit Log Retention Worker3G2304MHolds one 25,000-row batch and its compressed file in memory while uploading and verifying it; headroom mirrors the retention workers.

Every other worker runs on the default 512 MB tier, including the newer Alert, Webhook Actions and Failure Triage workers.

The Repo Cache Worker stays on the default 512 MB tier, but a cache refresh holds a repository's whole archive and its cached file contents in memory at once. Caching a very large repository can push it over the ceiling: PM2 restarts the worker, BullMQ redelivers the refresh job, and it starts over on the same repository. Override via REPO_CACHE_MAX_MEMORY_RESTART (e.g. 2G), host RAM permitting, or exclude binaries and build output from the configuration's path patterns.

Persistence Across Reboots​

To ensure workers start automatically after server reboots:

  1. Save the current PM2 process list:

    pm2 save

  2. Generate startup script:

    pm2 startup

    Follow the instructions provided by this command.

Prerequisites​

All workers require:

  • Valkey server running (for BullMQ job queues)
  • PostgreSQL database configured
  • Environment variables properly set

The Repo Cache Worker also needs git on its PATH to read Bitbucket Cloud repositories. The workers Docker image includes it. See Bitbucket Cloud requirements.

Monitoring​

You can monitor worker health and performance using:

  1. PM2 Monitoring

    pm2 monit

  2. PM2 Web Dashboard (optional)

    pm2 install pm2-web

  3. Application Logs

    • Workers log to PM2 logs
    • Check pm2 logs for detailed output

Troubleshooting​

Workers not starting​

  • Check Valkey connection: Ensure Valkey server is running
  • Verify database connection: Check PostgreSQL credentials
  • Review logs: pnpm pm2:logs

Jobs not processing​

  • Check queue health in Valkey
  • Verify worker is running: pnpm pm2:status
  • Check for errors in worker logs

Memory issues​

  • Most workers run with a 512 MB ceiling; the sync worker runs with a 1 GB ceiling; the forecast, SCIM access recompute, and Elasticsearch reindex workers run with a 2 GB ceiling; the audit log worker, the three webhook workers (dispatch, outbox, retention), the DataChangeLog retention worker, and the audit log retention worker run with a 3 GB ceiling. See the Worker memory tiers table above for the rationale on each elevated tier.
  • If a worker is being killed and restarted by PM2 in a tight loop (visible as repeated restart events in pm2 status), raise its ceiling before assuming there is a real leak: TESTMO_IMPORT_MAX_MEMORY_RESTART, ELASTICSEARCH_REINDEX_MAX_MEMORY_RESTART or REPO_CACHE_MAX_MEMORY_RESTART for those workers, or WORKER_MAX_MEMORY_RESTART for the 512 MB tier. Other elevated workers are set in ecosystem.config.js.
  • Monitor with pm2 monit

Environment Variables​

Workers use the same environment variables as the main application. Ensure these are set:

  • DATABASE_URL - PostgreSQL connection string
  • VALKEY_URL - Valkey connection string
  • AWS_* - AWS credentials for file storage
  • EMAIL_* - Email service configuration

Configuring Worker Concurrency​

Worker concurrency controls how many jobs each worker processes simultaneously. Higher concurrency can improve throughput on powerful machines but increases CPU and memory usage.

Available Concurrency Settings​

You can configure concurrency for each worker using environment variables:

# Testmo Import Worker (memory-intensive, default: 1)
# Keep this low (1-2) as imports consume significant memory
TESTMO_IMPORT_CONCURRENCY=1

# Memory ceiling for every worker that defaults to 512M. The heap follows at
# 75% unless WORKER_MAX_OLD_SPACE_MB is set. Host RAM permitting.
# WORKER_MAX_MEMORY_RESTART=1G

# Testmo Import Worker memory ceiling (default 4G; heap follows at 75%).
# Raise it for very large multi-GB exports, host RAM permitting.
# TESTMO_IMPORT_MAX_MEMORY_RESTART=18G

# Sync Worker (I/O-intensive, API rate-limited, default: 2)
# Moderate values (2-5) work well for external API calls
SYNC_CONCURRENCY=2

# Email Worker (I/O-intensive, default: 3)
# Can handle moderate concurrency (3-10) for email sending
EMAIL_CONCURRENCY=3

# Auto Tag Worker (default: 3)
# One per entity type; increase cautiously as AI calls are expensive
AUTO_TAG_CONCURRENCY=3

# Notification Worker (lightweight operations, default: 5)
# Can handle higher concurrency (5-20) for lightweight tasks
NOTIFICATION_CONCURRENCY=5

# Forecast Worker (CPU-intensive but parallelizable, default: 5)
# Set based on available CPU cores, typically 1-2x CPU count
FORECAST_CONCURRENCY=5

# Budget Alert Worker (default: 2)
BUDGET_ALERT_CONCURRENCY=2

# Alert Worker (default: 2)
ALERT_CONCURRENCY=2

# Elasticsearch Reindex Worker (I/O-intensive, default: 2)
# Balanced for Elasticsearch performance; increase for faster reindexing
ELASTICSEARCH_REINDEX_CONCURRENCY=2

# Elasticsearch Reindex Worker memory ceiling (default 2G; heap follows at 75%).
# Raise it for very large tenants (many projects / large run history), host RAM permitting.
# ELASTICSEARCH_REINDEX_MAX_MEMORY_RESTART=4G

# Repo Cache Worker memory ceiling (default 512M; heap follows at 75%).
# Raise it when caching very large repositories, host RAM permitting.
# REPO_CACHE_MAX_MEMORY_RESTART=2G

# Audit Log Worker (lightweight independent writes, default: 10)
# Can safely be set higher on powerful machines
AUDIT_LOG_CONCURRENCY=10

# Webhook Dispatch Worker (I/O-intensive, default: 5)
# Each job POSTs to one external endpoint with a 10s timeout; raise on
# beefier hosts, lower if external endpoints are slow or rate-limited
WEBHOOK_DISPATCH_CONCURRENCY=5

Setting Concurrency Values​

Add the desired concurrency values to your .env file:

# Example: High-performance server configuration
FORECAST_CONCURRENCY=10
NOTIFICATION_CONCURRENCY=10
EMAIL_CONCURRENCY=5
SYNC_CONCURRENCY=3
TESTMO_IMPORT_CONCURRENCY=2
# Example: Low-resource server configuration
FORECAST_CONCURRENCY=2
NOTIFICATION_CONCURRENCY=3
EMAIL_CONCURRENCY=2
SYNC_CONCURRENCY=1
TESTMO_IMPORT_CONCURRENCY=1

Applying Changes​

After updating concurrency settings in .env:

  1. Restart workers for changes to take effect:

    pnpm pm2:restart

  2. Verify settings in the Admin UI:

    • Navigate to Admin > Job Queues
    • Check the "Concurrency" column to confirm new values

Recommendations​

  • Start with defaults and monitor performance
  • Monitor resource usage (CPU, memory, database connections)
  • Consider your workload:
    • High-volume test imports: Increase FORECAST_CONCURRENCY
    • Frequent email notifications: Increase EMAIL_CONCURRENCY and NOTIFICATION_CONCURRENCY
    • Many external integrations: Increase SYNC_CONCURRENCY (but respect API rate limits)
  • Database connection pool: Ensure your database can handle sum(all concurrency values) + web server connections
  • Memory limits: Each concurrent Testmo import can use 500MB-1GB of memory

Monitoring​

Check worker performance and queue health:

# View real-time worker status
pm2 monit

# Check queue statistics
pnpm pm2:logs

# Admin UI monitoring
# Navigate to Admin > Job Queues for live queue statistics

Security Considerations​

  • Workers run with the same permissions as the Node.js process
  • Ensure proper file permissions on production servers
  • Keep PM2 and dependencies updated
  • Monitor worker logs for suspicious activity