A way to process more than 5000 records in Cloud Flow execution

If you need to process a large number of records, for example in Dataverse, what we really do is create a scheduled Cloud Flow/ Power Automate. If the process is complex, we can encapsulate the logic in a Custom API and call from Cloud Flow. 

To fetch the records to process, you need to use List Rows operation of Dataverse Connector which returns only 5000 records as once. This is a limitation. In order to overcome this, we need to do below trick.

1) Introduce a Boolean Variable

2) Create a Do Until loop to run until this variable is False

3) Inside the Loop

    - Create List Rows operation to retrieve records

    - Set Variable to True as far as List Rows returns none zero records

4) Add a For Each control to call a Bound Operation against each record in the batch.

Caution

1) List Rows fetch should contains a specific condition for the records being fetched. (perhaps a flag)

2) Within the Custom API it is important to change this flag value once its processed either its successful or failure. 

This way, any record, once processed is omitted in next fetch resulting only one processing per a given record. Otherwise, this will go to an infinite loop which could cause issues. 

Anyway, Cloud flows can handle badly designed loops and it will anyway stop at default maximum number of loops.. still this should be avoided.

Write a Custom API call from Clout Flow

In a previous article, we check how we Write a Custom API and calling it from classic workflow. It worked well, yet we encountered some hiccups in deployment. Actually, recommended way is to call it from a Cloud Flow than a workflow.

Lets check that using a simple sample Cloud Flow. Implementation of the Custom API remains the same. Please refer that part from previous post. 

Lets do the calling part from a Cloud Flow. I would select a flow that triggered on demand from the custom entity (in our case so_office).

First I am selecting a task to trigger when selecting a record from Office where I need to select the environment along with entity.

Then I need to select Perform a bound action from list of actions of Microsoft Dataverse.

Now interestingly, we can see our Custom API within the Action lists shown, which we would select as below. Then we need to pass the row Id passed by trigger from the first action.

Now if you go open and office record, you are able to runt he Flow on Demand as any other workflow is triggered. 

Here I demonstrated only simple on demand flow for ease of demonstration, but what we need to learn is, in any flow we can call out Custom API using Performa Bound Action of Microsoft Dataverse. Most importantly, this is the recommended way than calling from a workflow. 

Write a Custom API and call from Workflow

A Custom API is comparatively a new mechanism to create your own API in Dataverse. This is like a Action where you can package few steps to accomplish something. Advantage is you can then call from anywhere, either client side or server side etc. Obviously it can be called via a workflow or Cloud Flow too. Custom API is more flexible since coding is possible.

In this scenario we will see how to create a simple Custom API and call it from a classic workflow.

Writing a Custom API

1) We need to write a code which is pretty similar to a Plug-in. This is where we define our logic. In my case I am accepting a parameter string (i.e. DetailStr). Logic is to update Description (i.e. so_description) field of the custom entity called Office (i.e. so_office) with parameter value.

namespace AccAdmin.Plugins.CustomApi
{
    public class UpdateOffice : IPlugin
    {
        public void Execute(IServiceProvider serviceProvider)
        {
            IPluginExecutionContext context;
            IOrganizationServiceFactory factory;
            IOrganizationService service;

            try
            {
                context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
                ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));

                if (!context.InputParameters.Contains("DetailStr"))
                    throw new InvalidPluginExecutionException($"DetailStr not designated or Invalid..");

                if (context.InputParameters.Contains("Target") && context.InputParameters["Target"] is EntityReference target)
                {
                    if (target.LogicalName != "so_office") return;
                    factory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
                    service = factory.CreateOrganizationService(context.UserId);

                    Entity entity = new Entity();
                    entity.LogicalName = "so_office";
                    entity.Id = context.PrimaryEntityId;
                    entity["so_description"] = (string)context.InputParameters["DetailStr"];
                    service.Update(entity);
                }
            }
            catch (FaultException<OrganizationServiceFault> e)
            {
                throw e;
            }
            finally
            {
                service = null;
                factory = null;
                context = null;
            }
        }
    }
}

2) Once register the Plug-in, Custom API step will be visible as a step as below.

3) Now we start configuration. First add new Custom API to your solution.

    

Once save my Custom API was seen like below. It will have an unique name which we use to call this. We have set Enable for Workflow = Yes since we plan to call this from a Workflow. Most importantly, we have set our Plug-in step as the Plugin Type which is the linking point of our logic to the API. 

* Idea of all other settings

4) Same way we added the Custom API, we now add a parameter to our solution. Parameter is optional. In our case we plan to pass a string parameter.

Once created, parameter was seen as below. Nothing complex about this. You may notice we have selected Custom API we defined in previous step is set here for Custom API field.

Now we are done with our Custom API

Calling the Custom API from Workflow

Here I am create just an On Demand workflow so we can easily check it.

5) I am initiating a very simple Async workflow against Office entity.

6) Now I am adding a step of performing an Action. Interestingly, when I check the list of available Actions, I am seeing my Custom API by its unique name.

I see two parameters here. One is Target where we need to pass the current office record. Other one is what we define in Custom API which is needed for our logic.

7) Now its a matter of running our Workflow On Demand for any selected Office record.

* 

Unique Name: Unique name for the Custom API (Start with the publisher’s prefix which you have specified on your solution).

Name: Name for the Custom API.

Display Name: Display Name for this Custom API. In case we have enabled multi-languages on the System, we can provide a specific name for the custom API based on the language.

Description: This field is to store the description of this Custom API.

Binding Type: The Binding Type is the Options set field and could be set as Global, Entity, and Entity Collection. This should be specified as per your operation requirement (Entity Collection is only supported in Function Type Custom API).

Bound Entity Logical Name: This field is required when we select the Binding Type as Entity or Entity Collection. For the binding type as Global, it can be empty.

Is Function: It defines whether your custom API is a Function (this can only be called using Get Method) or Action (this can only be called using POST method).

Is Private: To define the Custom API is private or public (setting it private makes it accessible to only the owner of the Custom API).

Allowed Custom Processing Step Type: Allowed Custom Processing Step Type is another Option set field with options like None, Async Only, Sync, and Async. These options let you define whether the other plugins could be registered on this Custom Message and also lets you define its behavior. Like Async Only allows the plugins to register only onPost operation with Execution Mode to Asynchronous, and same for the Sync and Async option.

Execute Privilege Name: We can define the privilege that allows execution of the custom API. As per Microsoft docs, we can also create custom privileges and it is currently in development. We can use OOB privileges, for example,prvCreateLead, prvWriteLead, etc.

Plugin Type: Set the reference of your plugin for this API.

Warning

This example works fine. Anyway, there has been some deployment issues reported as per now we write this. So it is always advisable to call your Custom API from a Cloud Flow than a classic workflow. That worked perfectly.