How to batch with PNP to pull back 50k+ personal urls to avoid timeouts

[dgoldman-msft]

Hey everyone! I see that PNP is the replacement for the older SharePoint module. With testing the older module version we could run Get-SPOSite -IncludePersonalSite $true -Filter "Url -like '-my.sharepoint.com/personal/a'", Get-SPOSite -IncludePersonalSite $true -Filter "Url -like '-my.sharepoint.com/personal/b'" and kind of separate the queries, however not batch them due to the restrictions of session limit and reconnects. Trying to do one large batch will eventually timeout.

With testing It seems the script will pull back the OneDrive URLs, however will timed-out at a certain threshold. Before we pre-provisioned all users, we were able to pull 11k URLs without any issues.

Question: With the new PNP using RestAPI how can we batch out the queries so we can get around the long running transaction timeout?

TIA
Dave

[ToddKlindt]

It doesn't look like Get-PnPTenantSite supports the -Batch parameter like other PnP cmdlets do. For now, you can use a similar approach to what you used in the past. From help Get-PnPTenantSite -Examples :
Get-PnPTenantSite -IncludeOneDriveSites -Filter "Url -like '-my.sharepoint.com/personal/'"

I don't know if Get-PnPTenantSite lends itself well to the -Batch parameter. I'll let smarter people answer that. :)

[dgoldman-msft]

Thanks Todd!! I will test this today and see if this will time out. If it does I would hope that we could possibly take a PR to add support for the -Batch functionality.

[ToddKlindt]

It would probably make more sense for them to add the -PageSize parameter if they underlaying calls make that possible. I confused -Batch with -PageSize in my previous post.

[veronicageek]

@dgoldman-msft - Just to clarify that PowerShell PnP is not a replacement for the (Microsoft) SharePoint module (with *-SPO cmdlets). It's the replacement of the SharePointPnPPowerShell module, and the new one is now running on PowerShell Core & is cross-platform.

There's no -Batch functionality (yet) on cmdlets other than:

• Add-PnPListItem
• Set-PnPListItem
• Remove-PnPListItem

This article from Matthew Yarlett is not what you're looking for, but might be a starting point for batching if you can adapt to your needs.

[dgoldman-msft]

Thanks for that and much appreciated. The pnp commands did the trick and got us past the timeout issues.