Parallel Execution with ForEach in PowerShell

The PowerShell’s way of executing parallel instructions has been somewhat complicated till now using Jobs, Runspaces, etc. With PowerShell 7 and above, the team has greatly simplified this requirement with improvement in ForEach-Object cmdlet. You would need two extra parameters: Parallel and Throttlelimit to execute a set of instructions in parallel.

The Parallel parameter specifies the script block that is run in parallel for each input item. The new ThrottleLimit parameter limits the number of script blocks running in parallel at a given time. The default is 5.

For example, consider below code:

$items = 1..500  # defines an array of 500 items

$items | foreach-object -parallel {
  Write-Host "$(Get-Date) | Sleeping for 30 seconds..."
  Start-Sleep 30
} -throttlelimit 5

With this, we have specified throttle limit to 5. Consider the run output as below:

PS /home/cloud_user/pwsh-ws> ./foreach.ps1
03/17/2021 11:28:42 | Sleeping for 30 seconds...
03/17/2021 11:28:42 | Sleeping for 30 seconds...
03/17/2021 11:28:42 | Sleeping for 30 seconds...
03/17/2021 11:28:42 | Sleeping for 30 seconds...
03/17/2021 11:28:42 | Sleeping for 30 seconds...
03/17/2021 11:29:12 | Sleeping for 30 seconds...
03/17/2021 11:29:12 | Sleeping for 30 seconds...
03/17/2021 11:29:12 | Sleeping for 30 seconds...
03/17/2021 11:29:12 | Sleeping for 30 seconds...
03/17/2021 11:29:12 | Sleeping for 30 seconds...
03/17/2021 11:29:42 | Sleeping for 30 seconds...
03/17/2021 11:29:42 | Sleeping for 30 seconds...
03/17/2021 11:29:42 | Sleeping for 30 seconds...
03/17/2021 11:29:42 | Sleeping for 30 seconds...
03/17/2021 11:29:42 | Sleeping for 30 seconds...
03/17/2021 11:30:12 | Sleeping for 30 seconds...
03/17/2021 11:30:12 | Sleeping for 30 seconds...
03/17/2021 11:30:12 | Sleeping for 30 seconds...
03/17/2021 11:30:12 | Sleeping for 30 seconds...
03/17/2021 11:30:12 | Sleeping for 30 seconds...
03/17/2021 11:30:42 | Sleeping for 30 seconds...
03/17/2021 11:30:42 | Sleeping for 30 seconds...
03/17/2021 11:30:42 | Sleeping for 30 seconds...
03/17/2021 11:30:42 | Sleeping for 30 seconds...
03/17/2021 11:30:42 | Sleeping for 30 seconds...
03/17/2021 11:31:12 | Sleeping for 30 seconds...

Using parallel execution, we can reduce the time it takes to complete our script. For example, compare the time taken to complete below instructions:

PS /home/cloud_user/pwsh-ws> Measure-Command {1..100 | Foreach-Object {Start-Sleep 1}}

Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 40
Milliseconds      : 55
Ticks             : 1000551117
TotalDays         : 0.00115804527430556
TotalHours        : 0.0277930865833333
TotalMinutes      : 1.667585195
TotalSeconds      : 100.0551117
TotalMilliseconds : 100055.1117


PS /home/cloud_user/pwsh-ws> Measure-Command {1..100 | Foreach-Object -Parallel {Start-Sleep 1} -ThrottleLimit 10}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 10
Milliseconds      : 241
Ticks             : 102415479
TotalDays         : 0.000118536434027778
TotalHours        : 0.00284487441666667
TotalMinutes      : 0.170692465
TotalSeconds      : 10.2415479
TotalMilliseconds : 10241.5479

We can use the $_ variable to represent the current input object in the script block and $using: scope to pass variable references to the running script block.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s