Jobs

The Jobs module runs long work, such as postings, imports and reports, as durable background jobs. A job is a YesSql document that a tenant-wide dispatcher picks up and a runner executes in bounded batches. Each batch commits together with its checkpoint, so a job survives restarts, shell reloads and failures without repeating or losing work. The module also adds the Tools > Jobs list and details screens.

Feature ID	Name	EnabledByDependencyOnly	Dependencies
CrestApps.Almohaseb.Jobs	Almohaseb Jobs	true	CrestApps.Almohaseb.Admin.UI

The feature is never enabled on its own. Features that run jobs depend on it; for example Accounting (CrestApps.Almohaseb.Ledger, see Ledger) depends on it, so the business setup recipes that enable Accounting also enable Jobs. The list filters come from Admin UI.

What the module adds

• The Job document, stored in its own YesSql collection Jobs, with the JobIndex map index.
• IJobManager to enqueue jobs and to cancel, pause, resume and requeue them.
• IJobHandler, the extension point a module implements for each job type, registered with AddJobHandler.
• A per-tenant dispatcher and a runner that runs jobs in time slices of paced, checkpointed batches.
• Retries with exponential back-off and jitter, failure classification, and adaptive batch sizes driven by a memory budget and the memory pressure of the process.
• The Almohaseb jobs sweeper background task, which requeues abandoned jobs, deletes finished jobs past retention and starts due jobs.
• Resume on tenant activation.
• The Tools > Jobs list (AdminList List and Grid layouts, search terms, status drop-down, bulk Cancel and Requeue) and the details page (progress, attempts, last error, item errors, Pause, Resume, Cancel, Requeue).
• A diagnostic job type, queued from Run diagnostic job, to check that jobs run on a deployment.

How people use the screens is described in the User Manual: Background Jobs.

The job model

CrestApps.Almohaseb.Jobs.Models.Job derives from the CrestApps CatalogItem (its id is ItemId). The main properties:

Property	Purpose
JobType	Selects the IJobHandler that runs the job.
DisplayText	The title shown in the list.
Status	A JobStatus (see below).
Priority	Scheduled (0), Interactive (1, the default of a JobRequest) or DeadlineCritical (2). Higher values start first.
Parameters	A JsonObject holding ids and options only, never payloads. Large inputs are read from their own store in batches.
Checkpoint	The position the next batch continues from, usually the last processed key (a keyset cursor).
ProcessedCount, TotalCount	Progress. TotalCount is optional; without it the list shows a count instead of a percentage.
AttemptCount, MaxAttempts	Failed attempts since the job was last queued by a user, and the limit taken from the job type.
BatchSizeOverride	A smaller batch size the runner chose after batches went over the memory budget.
CancelRequested, PauseRequested	Flags a user sets on a running job; the runner acts on them before the next batch.
IdempotencyKey	Prevents enqueuing the same work twice while an earlier job for it is active.
OwnerUserId, OwnerUserName	Who started the job; empty for scheduled jobs.
CreatedUtc, NextAttemptUtc, StartedUtc, HeartbeatUtc, CompletedUtc	Timing. HeartbeatUtc is set only while the job runs.
SliceCount	The number of slices run. A slice is the work done under one lock.
ErrorSummary	The last error, truncated to 2,000 characters.
WarningCount, ItemErrorCount, ItemErrors	Warnings and per-item errors. Only the first ItemErrorLimit item errors are kept; the rest are counted.

Lifecycle

Every status change goes through the static JobLifecycle class.

Status	Value	Final	Entered when
Queued	0	No	The job is enqueued, yields at the end of a slice, is resumed or requeued, or a transient failure schedules a retry.
Running	1	No	A runner starts a slice.
Paused	2	No	A user pauses a queued job (at once) or a running job (after its current batch).
Completed	3	Yes	A batch reports IsComplete and the job has no item errors or warnings.
CompletedWithWarnings	4	Yes	A batch reports IsComplete and the job has item errors or warnings.
Failed	5	Yes	A permanent failure, including a missing handler.
NeedsAttention	6	No	A transient failure reaches MaxAttempts.
Canceled	7	Yes	A user cancels a job that is not running (at once) or a running job (after its current batch).

User actions and the statuses they accept:

Action	IJobManager method	Accepted from	Result
Cancel	CancelAsync	Any non-final status	Canceled, or CancelRequested on a running job.
Pause	PauseAsync	Queued, Running	Paused, or PauseRequested on a running job.
Resume	ResumeAsync	Paused	Queued, due now.
Requeue	RequeueAsync	NeedsAttention, Failed	Queued from the checkpoint, with AttemptCount reset and the error cleared.

Each method returns false when the job does not exist or the action does not apply. Changes are saved with the caller's session; EnqueueAsync, ResumeAsync and RequeueAsync signal the dispatcher in a deferred task, after that session commits.

How jobs run

Dispatcher

JobDispatcher is a tenant singleton. Signal() returns at once and runs a dispatch loop in the background; only one loop runs at a time, and a signal that arrives meanwhile makes it go round again. Each pass asks the store for due jobs (Status = Queued and NextAttemptUtc <= now, highest Priority first, then oldest NextAttemptUtc) and starts as many as the free slots allow.

Two limits apply:

• Per tenant: at most MaxConcurrentJobsPerTenant slices of one tenant run at once.
• Per instance: every slice also waits for JobInstanceGate, a process-wide semaphore of MaxConcurrentJobsPerInstance slots shared by all tenants, so no tenant can take every worker.

The dispatcher is signalled when a job is enqueued, resumed or requeued, when a slice ends, when the tenant is activated, and every minute by the sweeper.

Runner, slices and batches

JobRunner.RunSliceAsync takes the distributed lock CrestApps.Almohaseb.Jobs:{jobId} (expiring after LockExpiration) and starts the job if it is due, or if it is Running with a heartbeat older than StaleAfter (its previous runner died). It then runs batches until one of these happens:

• the job completes, fails, or is canceled or paused by a request;
• the slice has run for SliceDuration; the job yields (Queued, due now) so other jobs get a turn;
• the process is under critical memory pressure; the job yields with a delay of MemoryPressureDelay;
• the tenant or application is stopping; the interrupted batch is rolled back and the job yields from its checkpoint.

Each batch runs in a fresh shell scope with its own YesSql session (IJobScopeRunner). In that scope the runner loads the job, honors CancelRequested and PauseRequested, resolves the handler, calls IJobHandler.RunBatchAsync, applies the JobBatchResult to the job (checkpoint, counts, warnings, item errors, completion) and saves the job. The handler's changes and the new checkpoint commit in the same session, or not at all. A batch that fails is rolled back and the job runs again from the previous checkpoint, so batches run exactly once.

Every batch goes through the CrestApps BackgroundWorkPacer, which runs bulk batches of all tenants one at a time per process and pauses after each batch in proportion to how long it took (see CrestApps:BackgroundWork:Pacing under Configuration).

ShellJobScopeRunner gets a scope of the tenant's current shell for each call, so a job that outlives a shell reload (a feature change) continues on the reloaded shell.

Retries and failure classification

When a batch throws, the handler's ClassifyFailure decides first; when it returns null, JobFailureClassifier applies the default rules:

Kind	Default for	Effect
Conflict	YesSql.ConcurrencyException (the job document changed, for example a cancel request)	Requeued due now. The attempt is not counted.
Transient	DbException, TimeoutException, HttpRequestException, IOException, JobTransientException, or an exception whose inner exception is transient	AttemptCount increases. Below MaxAttempts the job is requeued after a back-off; at MaxAttempts it becomes NeedsAttention.
Permanent	Anything else	Failed. A user can requeue it once the cause is fixed.

The back-off is exponential with "equal jitter": the delay after attempt n is RetryBaseDelay × 2^(n−1), capped at RetryMaxDelay; half of it is fixed and half is random, so jobs that failed together (for example during a database restart) do not retry in lock step.

A job whose handler is not registered (its feature was disabled) fails with No handler is registered for the job type '…'. Enable the feature that provides it, then requeue the job. EnqueueAsync throws InvalidOperationException for a job type with no handler.

Memory pressure and adaptive batch size

GcMemoryPressureMonitor (the default IMemoryPressureMonitor) reads GC.GetGCMemoryInfo():

Pressure	When	Effect
Critical	The managed heap of the process is at least 85% of the memory available to it (the container limit or GCHeapHardLimit when set, otherwise physical memory).	Batches drop to the minimum size, and the job yields after its current batch and waits MemoryPressureDelay.
High	The heap is at least 70% of the available memory, or the machine memory load reaches the collector's high-load threshold.	Batches are halved.
Normal	Otherwise.	The configured or adjusted batch size.

Machine load alone never makes pressure critical, so other programs on the same machine cannot stall the jobs.

JobBatchSizer also measures the bytes each batch allocates. A batch that allocates more than the job type's MemoryBudgetBytes halves the next batch (stored in BatchSizeOverride); batches under a quarter of the budget double it again, up to the configured BatchSize. The smallest size chosen is 10, or the configured size when it is smaller.

Sweeper

The Almohaseb jobs sweeper background task (schedule * * * * *, every minute) runs JobSweeper, then signals the dispatcher. Each pass handles up to SweepBatchSize jobs per duty:

• Abandoned jobs: a Running job whose heartbeat is older than StaleAfter, and whose lock is no longer held, is treated as a transient failure (The job stopped responding and was queued again from its last checkpoint.) and retried after a back-off. A runner that still holds the lock is left alone.
• Retention: final jobs (Completed, CompletedWithWarnings, Failed, Canceled) whose CompletedUtc is older than Retention are deleted. NeedsAttention jobs are not final and are kept.
• Due jobs: the signal starts due jobs, including jobs whose retry delay has just ended.

The sweeper never runs job work itself.

Resume on activation and shutdown

JobsTenantEvents.ActivatedAsync signals the dispatcher as soon as a running tenant starts, after an application restart or a shell reload, so queued and interrupted jobs resume without waiting for the sweeper.

When the shell is released, the dispatcher cancels its running slices: the current batch is rolled back and the job yields from its checkpoint. If the yield cannot be saved, the sweeper requeues the job once its heartbeat is stale.

Idempotency

A JobRequest with an IdempotencyKey returns the existing active job (Queued, Running, Paused or NeedsAttention) with the same JobType and key instead of adding another. Once that job reaches a final status, the same key enqueues a new job.

Permissions

Permission (role editor)	Key	Grants	Default roles
Manage background jobs	ManageAlmohasebJobs	Cancel, pause, resume and requeue every job, bulk actions, and Run diagnostic job. Implies View background jobs.	Administrator
View background jobs	ViewAlmohasebJobs	See Tools > Jobs, every job of the tenant and its details.	Administrator

Both are defined in JobsPermissions (Core project). Every controller action calls IAuthorizationService.AuthorizeAsync and returns Forbid() on failure; the menu item only hides when the user lacks View background jobs.

Configuration

Both sections are bound from the tenant's shell configuration when the tenant starts, so changes take effect after the tenant restarts. Time spans use the hh:mm:ss or d.hh:mm:ss format.

Jobs

Section	CrestApps:Almohaseb:Jobs
Options class	JobsOptions

Key	Default	What it controls
MaxConcurrentJobsPerTenant	2	Slices one tenant runs at once.
MaxConcurrentJobsPerInstance	8	Slices one application instance runs at once across all tenants. The first tenant that starts the dispatcher fixes it for the process, so set it in the host configuration.
SliceDuration	00:02:00	How long a job works before it yields its slot to other jobs.
LockExpiration	00:10:00	How long the lock of a slice lasts. It must be longer than a slice plus its longest batch.
StaleAfter	00:12:00	How long a running job may go without a heartbeat before the sweeper treats it as abandoned.
RetryBaseDelay	00:00:30	The delay before the first retry of a transient failure.
RetryMaxDelay	00:30:00	The longest delay between retries.
MemoryPressureDelay	00:00:30	The delay before a job that yielded under critical memory pressure runs again.
ItemErrorLimit	100	How many item errors are kept on a job; the rest are only counted.
Retention	30.00:00:00	How long finished jobs are kept before the sweeper deletes them.
SweepBatchSize	100	Jobs the sweeper handles per pass for each of its duties.

{
  "OrchardCore": {
    "CrestApps": {
      "Almohaseb": {
        "Jobs": {
          "MaxConcurrentJobsPerTenant": 2,
          "MaxConcurrentJobsPerInstance": 8,
          "SliceDuration": "00:02:00",
          "Retention": "90.00:00:00"
        }
      }
    }
  }
}

OrchardCore__CrestApps__Almohaseb__Jobs__MaxConcurrentJobsPerTenant=2
OrchardCore__CrestApps__Almohaseb__Jobs__MaxConcurrentJobsPerInstance=8
OrchardCore__CrestApps__Almohaseb__Jobs__SliceDuration=00:02:00
OrchardCore__CrestApps__Almohaseb__Jobs__Retention=90.00:00:00

For one tenant only, use OrchardCore:{TenantName}:CrestApps:Almohaseb:Jobs in the host configuration, or put the section without the OrchardCore wrapper in the tenant's App_Data/Sites/{TenantName}/appsettings.json.

Pacing

Section	CrestApps:BackgroundWork:Pacing
Options class	BackgroundWorkPacingOptions (CrestApps.OrchardCore)

Key	Default	What it controls
DatabaseShare	0.25	The share of time, from 0.05 to 1, bulk batches spend working against the database. With 0.25, a batch that took one second is followed by a three-second pause. 1 turns pacing off.
MaxPause	00:00:30	The longest pause after one batch.

{
  "OrchardCore": {
    "CrestApps": {
      "BackgroundWork": {
        "Pacing": {
          "DatabaseShare": 0.5
        }
      }
    }
  }
}

OrchardCore__CrestApps__BackgroundWork__Pacing__DatabaseShare=0.5

The pacer is shared with the other CrestApps bulk jobs (such as content imports) in the same process. See also the configuration reference.

Per job type

JobTypeOptions (BatchSize default 200, MaxAttempts default 5, MemoryBudgetBytes default 256 MB) are named options keyed by job type. They are set in code when the job type is registered with AddJobHandler, and are not bound from configuration. MaxAttempts is copied onto the job when it is enqueued.

Defining a job type

A module that runs long work adds a job type:

1. Make the feature that owns the job type depend on CrestApps.Almohaseb.Jobs.
2. Implement IJobHandler.
3. Register it with AddJobHandler in the feature's startup.
4. Enqueue jobs through IJobManager.

Implement the handler

public interface IJobHandler
{
    string JobType { get; }

    Task<JobBatchResult> RunBatchAsync(JobBatchContext context, CancellationToken cancellationToken);

    JobFailureKind? ClassifyFailure(Exception exception)
        => null;
}

A batch must:

• read the next items after context.Checkpoint with keyset paging (never Skip), and process at most context.BatchSize of them;
• make its changes through the session resolved from context.Services, so they commit with the checkpoint;
• be idempotent and finish in seconds, and never load a whole table or file;
• stop between items when cancellationToken is cancelled (the application or tenant is stopping).

JobBatchContext also exposes JobId, JobType, Parameters, ProcessedCount, AttemptCount and GetParameter<T>(name).

The batch returns a JobBatchResult: Checkpoint, Processed (items in this batch), Total (optional), IsComplete, Warnings and ItemErrors (JobItemError(Key, Message) records for items that could not be processed without failing the job; Key identifies the item for the user, such as a row number).

The diagnostic handler is the smallest complete example:

public sealed class DiagnosticJobHandler : IJobHandler
{
    public const string Type = "CrestApps.Almohaseb.Jobs.Diagnostic";

    public const string ItemsParameter = "items";

    public string JobType => Type;

    public async Task<JobBatchResult> RunBatchAsync(JobBatchContext context, CancellationToken cancellationToken)
    {
        var total = Math.Max(0, context.GetParameter<int>(ItemsParameter));
        var start = context.Checkpoint is null ? 0 : int.Parse(context.Checkpoint, CultureInfo.InvariantCulture);
        var count = Math.Min(context.BatchSize, total - start);

        await Task.Delay(TimeSpan.FromMilliseconds(250), cancellationToken);

        var next = start + count;

        return new JobBatchResult
        {
            Checkpoint = next.ToString(CultureInfo.InvariantCulture),
            Processed = count,
            Total = total,
            IsComplete = next >= total,
        };
    }
}

To have a provider's "slow down" or "try later" response retried with back-off, throw JobTransientException, or override ClassifyFailure for exceptions specific to the job type.

Register the handler

services.AddJobHandler<DiagnosticJobHandler>(DiagnosticJobHandler.Type, options => options.BatchSize = 50);

The jobType argument must equal the handler's JobType. The optional delegate sets BatchSize, MaxAttempts and MemoryBudgetBytes for that job type. The handler is registered as scoped.

Enqueue a job

var job = await _jobManager.EnqueueAsync(new JobRequest
{
    JobType = DiagnosticJobHandler.Type,
    DisplayText = S["Diagnostic job"],
    Parameters = new JsonObject { [DiagnosticJobHandler.ItemsParameter] = DiagnosticItems },
    OwnerUserName = User.Identity?.Name,
}, HttpContext.RequestAborted);

JobRequest also accepts Priority (default Interactive), IdempotencyKey, NotBeforeUtc (the earliest time the job may run) and OwnerUserId. The job is saved with the caller's session and starts after that session commits.

Storage

Jobs are stored through the CrestApps DocumentCatalog<Job, JobIndex> (JobStore, behind IJobStore) in the YesSql collection Jobs, so a busy job history never grows the shared document table. The collection is declared through StoreCollectionOptions, and its tables are Jobs_Document and Jobs_JobIndex (after the tenant's table prefix).

JobIndex is a map index with these columns: ItemId (26), JobType (255), Status (int), Priority (int), OwnerUserId (26), IdempotencyKey (255), CreatedUtc, NextAttemptUtc, HeartbeatUtc (nullable) and CompletedUtc (nullable). JobMigrations creates these SQL indexes on it:

Index	Columns	Used by
IDX_JobIndex_ItemId	ItemId	Loading a job by id.
IDX_JobIndex_Due	Status, NextAttemptUtc, Priority	The dispatcher's due-jobs query.
IDX_JobIndex_Heartbeat	Status, HeartbeatUtc	The sweeper's abandoned-jobs query.
IDX_JobIndex_Completed	Status, CompletedUtc	The sweeper's retention query.
IDX_JobIndex_Idempotency	JobType, IdempotencyKey, Status	Idempotent enqueue.
IDX_JobIndex_Created	CreatedUtc, Status	The jobs list, newest first.

Every store query is index-driven and bounded. IJobStore.Query() returns an IQuery<Job, JobIndex> over the collection for the admin list filters.

Admin list filters

The Tools > Jobs search box is built on the Admin UI list filter pattern. JobsAdminListFilterProvider registers these terms for Job:

Term	Kind	Values
type	Default term (text without a term name)	Job types that contain the text (ManyCondition with a Contains and a NotContains predicate).
status	Named, single value; mapped to and from the status drop-down	queued, running, paused, completed, completed-with-warnings, failed, needs-attention, canceled, or active (Queued, Running, Paused and NeedsAttention).
queued	Named, single value	Any DateRangeTerm value, applied to CreatedUtc in the site time zone through ILocalClock.
sort	Named, single value	newest (default), oldest, priority.

Operations

• Check a deployment: Run diagnostic job on Tools > Jobs queues a 1,000-item job (batch size 50) that does no work. Watch it go from Queued to Completed on its details page; if it stays Queued while no other job runs, the dispatcher is not starting jobs on that tenant.
• Several instances: the per-job lock uses Orchard Core's IDistributedLock. When the application runs on more than one instance, enable a distributed lock provider (such as Orchard Core's Redis lock) so two instances never run the same job. MaxConcurrentJobsPerInstance and the pacing gate apply per process.
• Timing settings: keep LockExpiration longer than SliceDuration plus the longest batch, and StaleAfter longer than LockExpiration, or the sweeper may requeue live jobs.
• Logs: the runner logs batch failures as errors (with the failure kind), conflicts at debug level, and a warning when a job yields under critical memory pressure; the sweeper logs how many jobs it requeued and deleted; the dispatcher logs when due jobs could not be dispatched.
• Background task: the sweeper appears in Orchard Core's background tasks as Almohaseb jobs sweeper. If background tasks are disabled for a tenant, due retries and abandoned jobs are only picked up when another signal (enqueue, resume, requeue, a slice ending or tenant activation) arrives, and finished jobs are not deleted.

See also the feature reference and the architecture overview.