The asynchronous service executes long-running operations independent of the main Microsoft Dataverse core operation. Executing long-running operations this way results in improved overall system performance and improved scalability. The asynchronous service is a managed first-in, first-out (FIFO) queue for the execution of asynchronous registered plug-ins, workflows, and operations such as bulk mail, bulk import, and campaign activity propagation. These operations are registered with the asynchronous service and executed periodically when the service processes its queue.
After an event occurs and any synchronous extensions are processed, Dataverse serializes the context for any asynchronous extensions and saves it in the System Job (AsyncOperation) table. The system job defines and tracks the execution of the asynchronous operation. As resources become available, Dataverse processes system jobs and executes the operations they define. Dataverse processes any data operations defined in the extension again in the event execution pipeline, but this time as a synchronous operation.
Execution order and dependencies
System jobs are evaluated as a queue using the CreatedOn date. If there are no conditions to defer execution, they're executed as soon as resources are available. Execution isn't always performed in the order set by the CreatedOn date because different types of operations require different resources.
A system job can be dependent on another system job so that it will begin only after the other system job completes. The DependencyToken column value establishes this dependency when a system job is created. If the DependencyToken value is null, the system job has no dependencies. Dependent system jobs have the same DependencyToken value and are executed in the order they were created. If a system job is postponed, all subsequent dependent system jobs continue to wait until the postponed system job executes.
Note
This dependency system cannot be used by plug-ins registered to run asynchronously because the system jobs for them are created by the system.
Security considerations for system jobs
The AsyncOperation (system job) table has a layered permission model as described below.
Entity-level privileges: Users with the Write privilege (prvWrite) on
AsyncOperation can update the supported lifecycle columns (StateCode,
StatusCode, PostPoneUntil) to manage system jobs. This is the intended
mechanism for canceling, pausing, resuming, or postponing jobs.
Platform-level protection for internal columns: Columns that contain
internal execution data, such as the serialized job payload, are protected
by additional server-side authorization checks that are enforced independently
of entity-level privileges. These protections prevent unauthorized modification
of job execution data even by users who have Write access to the
AsyncOperation table.
Read access: Users with the Read privilege (prvRead) on AsyncOperation can
retrieve system job records, including metadata such as job name, status,
and timestamps. This is by design to allow users to monitor job progress
and status.
This layered model ensures that users can manage the lifecycle of their system
jobs, an intended and supported operation, while internal execution data remains
protected by the platform.
Managing system jobs
You can perform the following operations to manage system jobs using the AsyncOperation Table.
Retrieve system jobs
Delete system jobs
Manage system job states
Postpone system jobs
Note
Creating system jobs with code is not supported. Although the AsyncOperation table supports several writeable columns and create operations, only the following columns are supported for update:
These columns allow users with the appropriate entity-level
Write privilege (prvWrite) on AsyncOperation to manage the lifecycle of system jobs
(cancel, pause, resume, and postpone).
Other columns in the AsyncOperation table, including internal execution data,
are subject to additional platform-level authorization checks and cannot be
modified through standard update operations even if the table schema indicates
they are writable. The schema-level write attribute reflects internal platform
capabilities, not end-user permissions.
Retrieve system jobs
You can view system jobs in the application by navigating to Settings > System > System Jobs and you can also search them using Advanced find in model-driven apps.
Using code, you can retrieve system jobs like any other table. The following table lists selected columns that are important in understanding system jobs:
Column
Description
AsyncOperationId
Unique identifier of the system job.
CompletedOn
Date and time when the system job was completed.
CreatedBy
Unique identifier of the user who created the system job.
With the Web API there is a single-valued navigation property for each table that supports system jobs. The name of this navigation property follows the pattern regardingobjectid_
Every table that supports system jobs has a listed Many-to-One relationship with the AsyncOperation table via the RegardingObjectId lookup column. The name of this relationship follows the pattern
Recurring system jobs require information about when they should start and how often to recur. These values are stored in the AsyncOperation table RecurrenceStartTime and RecurrencePattern columns.
Because you can't create AsyncOperation records directly with code, you need to interpret these values when you query the data. These property values are set indirectly by using messages that create new system jobs. The BulkDelete and BulkDeleteDuplicates messages both include parameters or properties in the corresponding Web API actions or SDK for .NET request classes. More information: BulkDetectDuplicatesRequest Class, BulkDetectDuplicates Action, BulkDeleteRequest Class, and BulkDelete Action
The RecurrenceStartTime is simply a datetime value to indicate when the system job should start. If it isn't set, the system job was expected to start immediately.
The RecurrencePattern column stores information about how frequently recurring system jobs occur. The platform sometimes sets this value when a new asyncoperation record is created. You can set this value to change the pattern.
SELECT statecode,
statuscode,
operationtype,
Count(*) AS count
FROM asyncoperation
WHERE statecode = 1
GROUP BY statecode,
statuscode,
operationtype
ORDER BY Count DESC
Quantify Suspended Jobs: The query specifically targets jobs that are Suspended (statecode = 1). The results give you a count of all such jobs, categorized by their statuscode and operationtype.
Operation Type Breakdown: The count of Suspended jobs grouped operationtype, showing you the types of operations are most commonly in a Suspended state.
Identifying Potential Bottlenecks: A high count of Suspended jobs for extended periods of time might be due to resource limitations, dependencies on other processes, or system misconfigurations.
Capacity and Resource Management: If certain jobs are consistently in the Suspended state, it could indicate that the system lacks the necessary resources to process these jobs efficiently.
System Health Check: The jobs in a Suspended state serve as a health indicator. A healthy system should ideally have minimal jobs in a Suspended state or at least show a quick turnover from Suspended to active processing.
Workflow Efficiency: The results can shed light on workflow efficiency. If a particular operationtype has a high count of Suspended jobs, it might indicate inefficiencies or the need for optimization within that workflow.
Workflows by count
This query provides a detailed breakdown of workflow-related jobs, filtered by an operationtype value for workflows. Use the query results to gain a comprehensive view of workflow jobs, manage system resources more effectively, and ensure that workflows are running smoothly and efficiently.
SELECT statecode,
statuscode,
operationtype,
Count(*) AS count
FROM asyncoperation
WHERE operationtype = 10
GROUP BY statecode,
statuscode,
operationtype
ORDER BY Count DESC
Workflow-Specific Jobs: The operationtype = 10 filter limits the results to workflow jobs. You can see how many jobs are related to workflows and their current statuses. You can also apply this example query with other types of operations. Learn more about Operation Types
Job State Distribution: The statecode tells you the current state of these workflow jobs, such as whether they're Ready, Suspended, Locked, or Completed.
Status Code Analysis: The statuscode can give specific insights into the reason behind a job's current state. For example, it could indicate if a job is Waiting for Resources, Waiting, In Progress, Pausing, Cancelling, Succeeded, Failed, or Cancelled.
Count of Each Category: The total number of jobs for each statecode and statuscode combination. This helps in identifying the most common outcomes of workflow operations.
Identifying Common Outcomes: With the results ordered by count in descending order, you can identify the most frequent outcomes or bottlenecks in workflow processing.
Troubleshooting and Optimization: High counts in Failed status or Suspended states can highlight areas where workflows might be failing or getting stuck, signaling a need for troubleshooting or process optimization.
Performance Metrics: Understanding which workflows are most common and how they're distributed across different states can help in assessing the performance and reliability of the workflow management system.
Capacity Planning: Consistently high number of In Progress or Waiting workflows could suggest that more resources are needed to handle the load, or that there's a need to optimize the workflow execution environment.
Workflow Management: The query results can guide administrators on managing workflows more effectively, such as deciding which workflows to prioritize or identifying workflows that can be optimized or deactivated/disabled.
System Health Check: The overall results can serve as a health check for your workflow system, indicating whether the system is performing optimally or if there are areas that require attention.
Jobs waiting for system resources to become available
Use this query to retrieve a detailed analysis of jobs from the AsyncOperation table that are in a specific state of readiness but are pending execution due to unavailability of system resources. This query can identify factors contributing to slowness and making decisions for efficiency and better handling of backlog.
SELECT statecode,
statuscode,
operationtype,
Count(*) AS count
FROM asyncoperation
WHERE statecode = 0 AND statuscode = 0
GROUP BY statecode,
statuscode,
operationtype
ORDER BY Count DESC
Filter for Ready Jobs Waiting for Resources: Limiting results where statecode = 0 and statuscode = 0 filters for jobs that are in a Ready and Waiting For Resources. This combination indicates jobs that are queued up and prepared to run but are on hold.
Optimizing Job Scheduling: Identifying patterns in job readiness and wait times can inform improvements in job scheduling, possibly leading to a more balanced distribution of system load.
Identifying Underlying Issues: In some cases, jobs waiting for resources might not be solely a resource issue but could be indicative of underlying problems such as deadlocks or inefficient resource locking mechanisms.
Queries for file storage
Depending on the size of the Data column of the AsyncOperation table, the data in that column may be saved in file storage. The DataBlobId column has a value when the row uses file storage. To save space, you might want to identify and delete these records. Use the following queries to discover these records
AsyncOperation file storage datablobid count
Use this query to count the number of records in the AsyncOperation table where the datablobid column isn't null.
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((datablobid ne null))/aggregate($count as FileStorageCount)
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Data Storage Implications:
You may want to delete the records in using file storage to save space so this is helpful to know. Large numbers might suggest significant space used by these blobs, which can be important for database size managemement.
System Performance Considerations:
If the FileStorageCount is unexpectedly high, you may want to take further action such as Bulk delete and clean up.
AsyncOperations not in blob storage
Use this query to count the number of records in the AsyncOperation table where the datablobid field is NULL.
Understanding Async Operations not in Blob Storage: The DBCount result will indicate the volume of async operations that do not have associated data blobs. This shows us the storage status when not accounting for the blobs.
Identifying inefficiencies Unless intended, high count here may suggest the need to schedule cleanup and run bulk delete. Low count in blob storage and high count here would attribute this as the primary volume contributor.
Find names of jobs using file storage
The results of this query shows the job types, the name of the jobs, and the number of times this job exists on the table consuming file storage. Use this to identify the specific job names that have the greatest impact on file consumption and create a bulk delete job for records having that name.
This will enable the identification of the specific job names that have the greatest impact on file consumption. As a result, the customer can initiate a bulk delete process for that particular job by targeting its name.
This query doesn't order by the jobs column decending. You may want to use FetchXml with Web API instead. Query data using FetchXml
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((datablobid ne null))/groupby((operationtype,name,friendlymessage),aggregate($count as jobs))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
SELECT operationtypename
,name
,friendlymessage
,count(*) AS Jobs
FROM asyncoperation
WHERE datablobid IS NOT NULL
GROUP BY operationtypename, name, friendlymessage
ORDER BY Jobs DESC
SELECT count(*) as RecordCount
,sum(filesizeinbytes) as TotalSize
,owningextensionid
,owningextensionidname
,statecode
,statuscode
,operationtype from fileattachment
join asyncoperation on fileattachment.fileattachmentid = asyncoperation.datablobid
where fileattachment.objectidtypecode = 4700
group by owningextensionid, owningextensionidname, statecode, statuscode, operationtype
order by TotalSize desc
Record Count:RecordCount tells you how many records are being returned for each grouping of system job records. This will give you an idea of the volume of asynchronous operations being performed and which types are most common.
Total File Size:TotalSize tells you the amount of data these operations are handling. This can help you identify if there are any unusually large files that could be affecting system performance.
Grouping by Owning Entities: The query groups results by owningextensionid, owningextensionidname, statecode, statuscode, and operationtype. Look at these groupings to pinpoint which extensions are generating the most activity and if there are specific operation types that are predominant.
Operation States and Statuses: The inclusion of statecode and statuscode in the grouping will help you determine the current state and status of these operations, such as which ones are pending, in progress, or completed.
Ordering by TotalSize: Since the results are ordered by TotalSize in descending order, pay attention to the top results as they will highlight the operations that are consuming the most storage. This could be important for identifying potential areas for optimization or cleanup.
Delete system jobs
You can delete system jobs in the application or in code just like any other table if you have the necessary privileges to do so.
Note
When registering asynchronous plug-ins, there is an option to automatically delete successful operations. It is recommended that you use it. Learn to write a plug-in
The status of the system job changes multiple times until the operation is completed. The following are the StateCode and StatusCode options that represent the available state and status reason values:
StateCode Value
StateCode Label
StatusCode Value
StatusCode Label
0
Ready
0
Waiting For Resources
1
Suspended
10
Waiting
2
Locked
20
In Progress
2
Locked
21
Pausing
2
Locked
22
Canceling
3
Completed
30
Succeeded
3
Completed
31
Failed
3
Completed
32
Canceled
You can change the status of system jobs in the application by navigating to Settings > System > System Jobs and using commands available in the More Actions menu.
Note
Any action you can perform via this UI can also be performed using code. You cannot perform cancel, pause, or resume operations on system jobs generated by the platform. Learn about Operation types
Together with options to manage views, the following options to manage system jobs are available:
Option
Description
Delete
Using the command. Deletes a system job
Bulk Delete
Using the More Actions menu. Opens a wizard to define conditions and create a new Bulk Deletion system job to delete the matching system jobs.
Cancel
Using the More Actions menu. Cancels the system job.
Resume
Using the More Actions menu. Resumes a paused system job.
Postpone
Using the More Actions menu. Reschedules a system job
Pause
Using the More Actions menu. Pauses a system job.
Whether the requested operation occurs depends on the state of the system job. For example, you can't pause a job that is completed or hasn't started yet. The following table describes the conditions for each change and what happen when they're selected.
Option
Valid StateCode values
Change
Delete
any
System Job is deleted
Cancel
0 (Ready) 1 (Suspended) 2 (Locked)
StateCode changed to 3 (Completed) and StatusCode changed to 32 (Cancelled), or StateCode changed to 3 (Completed) and StatusCode changed to 31 (Failed)
The PostPoneUntil column contains a datetime value when the system job changes state from 1 (Suspended) to 0 (Ready). The PostPoneUntil, StateCode, and StatusCode columns are the only AsyncOperation table columns supported for update.
command.