Appending to Array

@Liz DixonEversole
i'm assuming you are talking about Power Automate.

While I don't know what Emma API is, Power Automate can handle 20K records in an array without issue. The question is, what are you doing to append. Is the “array” you talking about a “simple” array of only string (email address) or is it a “complex” array of object?

example of simple array of string ["abc@aol.com", “def@yahoo.com”]

example of complex array of object
[
{
"id": "131",
"address": "JPdiresta@net.com",
"constituent_id": "185",
"date_added": "1600-01-01T00:00:00Z",
"date_modified": "2016-10-19T19:42:38.775Z",
"do_not_email": false,
"inactive": false,
"primary": false,
"type": "Email"
}, {
"id": "135",
"address": "adiresta@aol.net.com",
"constituent_id": "333",
"date_added": "1600-01-01T00:00:00Z",
"date_modified": "2016-10-19T19:42:38.775Z",
"do_not_email": false,
"inactive": false,
"primary": false,
"type": "Email"
}
]

You cannot append an array with different “type” of “item” (simple or complex)

so to better help you troubleshoot, you will need to provide a little more screenshots of the actions where you init the array, append to the array, the data you are trying to append.

@Liz DixonEversole
There is a logic error here. what you current have:

• Do until hasMorePages = false
  • HTTP get records based on page = variable page
  • Parse JSON of the result from HTTP
  • Apply to each (appears to be applying to each of the records returned)
    • append to array variable allMembers with the records returned
  • Condition on length of Parse JSON # of records < 500
    • if yes
      variable hasMorePages assign to false
    • if no
      increment variable page

The red above is wrong logic: what you want is to append all records from Parse JSON into the array variable allMembers; what you did though is to append all records from Parse JSON X times into the array variable allMembers; (meaning, if the first run has 500 records, you are appending the same 500 records, 500 times into the array variable allMembers.

Instead, the logic you want is:

• Do until hasMorePages = false
  • HTTP get records based on page = variable page
  • Parse JSON of the result from HTTP
  • append to array variable allMembers with the records returned
  • Condition on length of Parse JSON # of records < 500
    • if yes
      variable hasMorePages assign to false
    • if no
      increment variable page

Other note:

• click on Change limits in the Do until action, screenshot what is there
• the returned JSON (looking at your schema) is a complex array, even though you may think it is “simple” b/c it only has 1 number: member_id
  • what may be better to do is use Data action Select to select the member_id from the array, which will create an simple array of number (i.e. [1, 2, 3, 4]), then append to array variable allMembers using the output from the Select action.

@Liz DixonEversole
shouldn't take this long. my guess is you still have unneeded loop

screenshot what you got now to get help

@Liz DixonEversole
your flow is too slow b/c of the Apply to each inside your Do Until loop. Your Apply to each is adding ONE of the 500 records at a time into the array. This is super slow.

Instead, use union() expression to combine the array varible with the new array from HTTP and assign it back into the variable.

The Select action will select ONLY the member_id number from the Parse JSON. Then the Compose Combine Array uses union() expression to combine what was in the array with the new info from Parse JSON, and finally Set Variable of New Combined Array back into allMembers array variable.

union expression:

union(variables('allMembers'), body('Select'))

@Liz DixonEversole
Do Until has 2 options as safe guard from an infiinity running loop:

What is the MAX number of times the loop can run?
and/or
What is the MAX duration the loop can run?

By default, if you do not click “Change limits” to change it, the Count=60 and Timeout=PT1H means your do until will ONLY run for 60x max OR 1 hour, whichever comes first.

Only one of the option need to be specified, not both, so I suspect your flow ran over 1 hour and flow just “stops” the run.

Remove PT1H and your do until loop will run for a max of 60x. I do not recommend using Timeout as it leaves your flow logic up in the air as to what is done / not done. Use Count instead. Count specify the “max”, so it is not like the flow will definitely run 60x, if your logic only calls for the loop to run ~40x, then it will only run ~40x.

While this will fix your issue with flow not running all records pull, you don't want your flow to be running 1+ hour, implement what I replied earlier and your flow will finish in less than 10 minutes. (assuming what you said on OP, 20K records, 500 per pull, and per pull will take ~10 seconds, then the flow can finish in ~6-7 minutes, vs 1-2hr.

@Liz DixonEversole
i'll need to see what your flow currently looks like to help. some screenshot in edit mode and run history will help understand the problem

Also, what exactly are you trying to do, it will be easier to suggest once I know what your intention for the flow is.

@Liz DixonEversole
it sounds like your google sheet has the unsubscribe, which is a lot less records than the emma list. so you are correct to “loop” (apply to each) through the google sheet returned rows.

• Provide screenshot of your Compose action.
• provide a screenshot of the raw data returned by Get rows 2 and one of the Parse JSON inside your Do until loop