How to manage concurrent jobs in PowerShell
There had been many times in the past two months where I had to write scripts that run multiple jobs concurrently. Since PowerShell doesn’t have a built-in way to limit the number of jobs run at a time, this task falls on us, mere mortals. (On a side note: I’ve had my fair share of PowerShell Workflow and foreach -parallel, it is not ideal!).
Manage the concurrent jobs means, if there is more than X amount of jobs, wait for at least 1 job to finish its execution before starting another. Sounds simple enough. In the past, I wrote a snippet, which was a bulky chunk of code with over 100 lines of code and also did many other “generic” stuff.
Since then I wrote scripts that implement that idea but I didn’t have a solid snippet to use or a simple pattern to follow.
Now I’ve written one that is simple and designed to do this one thing:
$ConcurrentJobsThreshold = 20
$Servers = @()
foreach($Server in $Servers)
{
$RunningJobs = @(Get-Job | Where-Object{ $_.Status -eq 'Running' })
if($RunningJobs.count -ge $ConcurrentJobsThreshold){
$RunningJobs | Wait-Job -Any
}
Start-Job -Name "CopyTo $Server" -ScriptBlock {
mkdir "\\$using:Server\C`$\Updates" -Force
Copy-Item -Path "C:\KB123456.msu" -Destination "\\$using:Server\C`$\Updates\KB123456.msu"
}
}
# Waiting for remaining jobs to finish (Change state to Completed, Failed, Stopped, Suspended or Disconnected)
Get-Job | Wait-JobLets go over it.
There are 3 inputs to this snippet:
$ConcurrentJobsThresholdinteger variable — number of jobs that will run concurrently.$Serverswhich represents a collection, we would like to start a job for each item in this collection.- The ScriptBlock value in the
Start-Jobcmdlet — What we do on each item.
There is a foreach loop, iterating through the collection.
First thing in the loop, we get all the jobs that are running. If the amount of running jobs is smaller than the threshold we set at the beginning, a new job will be created.
Otherwise, if the amount of running jobs is greater or equal (-ge) to the threshold we will wait for one of these jobs to finish.
Wait-Job gets an array of job objects and waits for them to change status to Completed, Failed, Stopped, Suspended or Disconnected. (See the Notes section in the help page of Wait-Job).
By itself, Wait-Job will wait for all the given jobs to finish (change its state to one of the specified states), and with the -Any switch it will wait for the first job that will finish (or if it’s already finished), and return it.
Finally, we are waiting for all the remaining jobs to finish (Wait-job without -Any), and we’re done.
Now, this is just a template, you can add more stuff like:
- Progress bar using
Write-Progress. - Stop the Script execution if there are more than Y amount jobs with state of ‘Failed’.
- Collect the jobs outputs to a log file: After the concurrency threshold was reached
$RunningJobs | Wait-Job -Any | Receive-Job -Wait -AutoRemoveJob | Out-File 'log.txt' -Append
and at the end:
Get-Job | Wait-Job | Receive-Job | Out-File 'log.txt' -Append - Add a Pause functionality.
In one of the recent scripts, where the line $RunningJobs | Wait-Job -Any I wrote Get-Job | Wait-Job -Any instead. In this variation of the script after the first job failed there was no limitation on the number of jobs that were running concurrently. Instead of 20 jobs, we had more than 200, that took too many resources (the horrors!!!). What happened was something like that: After a job finished its execution (changed state to Failed or Completed) Get-Job hands over Wait-Job -Any an array of jobs that looks like this (for illustration only the jobs statuses are listed):
@(Failed, Running, Completed, Running, Running,…) | Wait-Job -Any
Wait-Job -Any immediately returns the first finished job, which in this example could be either the first job (with status ‘Failed’) or the third job (with status ‘Completed’). And while the amount of running jobs is still the same, a new job is created which exceeds the threshold that was set. I could make sure to remove finished jobs with Remove-Job or Receive-Job -Wait -AutoRemoveJob, which is actually what I did for jobs in ‘Completed’ state only and forgot the failed ones… >_>
For reference please don’t hesitate to use Get-Help, and Get-Help -Online
Start-Job
Wait-Job
Receive-Job
Stop-Job
Remove-Job
or to ask me 🙂
Have a great week!
Amir.