Some Antivirus (BeyondTrust/Avecto DefendPoint) may cause PowerShell Extension to stop responding

[cveld]

Issue Description

Visual Studo Code freezes completely for 4 minutes while starting up the PowerShell extension and then the extension fails to start up. It shows the infamous dialog 'The window is no longer responding' multiple times:

After analyzing your troubleshooting guide (https://github.com/PowerShell/vscode-powershell/blob/master/docs/troubleshooting.md) my hypothesis is that the language server is simply not able to start in the allotted time window and therefore the client will not find a session file in time. I guess BeyondTrust/Avecto is delaying the process for some reason. Refer to the log file:

vscode-powershell.log

28-11-2020 16:33:14 [NORMAL] - Language server starting --
28-11-2020 16:33:14 [NORMAL] -     PowerShell executable: C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe
28-11-2020 16:33:14 [NORMAL] -     PowerShell args: -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command Import-Module 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\modules\PowerShellEditorServices\PowerShellEditorServices.psd1'; Start-EditorServices -HostName 'Visual Studio Code Host' -HostProfileId 'Microsoft.VSCode' -HostVersion '2020.9.0' -AdditionalModules @('PowerShellEditorServices.VSCode') -BundledModulesPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\modules' -EnableConsoleRepl -StartupBanner '=====> PowerShell Preview Integrated Console v2020.9.0 <=====
' -LogLevel 'Normal' -LogPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\logs\1606577594-22dda057-f65c-444d-b276-7edc00d0a1d11606577351760\EditorServices.log' -SessionDetailsPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\sessions\PSES-VSCode-24032-911679' -FeatureFlags @() 
28-11-2020 16:33:14 [NORMAL] -     PowerShell Editor Services args: Import-Module 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\modules\PowerShellEditorServices\PowerShellEditorServices.psd1'; Start-EditorServices -HostName 'Visual Studio Code Host' -HostProfileId 'Microsoft.VSCode' -HostVersion '2020.9.0' -AdditionalModules @('PowerShellEditorServices.VSCode') -BundledModulesPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\modules' -EnableConsoleRepl -StartupBanner '=====> PowerShell Preview Integrated Console v2020.9.0 <=====
' -LogLevel 'Normal' -LogPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\logs\1606577594-22dda057-f65c-444d-b276-7edc00d0a1d11606577351760\EditorServices.log' -SessionDetailsPath 'c:\Users\caveld\.vscode-insiders\extensions\ms-vscode.powershell-preview-2020.9.0\sessions\PSES-VSCode-24032-911679' -FeatureFlags @() 
28-11-2020 16:33:14 [NORMAL] - powershell.exe started.
28-11-2020 16:33:14 [NORMAL] - Waiting for session file
28-11-2020 16:37:14 [NORMAL] - Timed out waiting for session file to appear.
28-11-2020 16:37:14 [NORMAL] - Language server startup failed.
28-11-2020 16:37:14 [ERROR] - The language service could not be started: 
28-11-2020 16:37:14 [ERROR] - Error: Timed out waiting for session file to appear.

I would like to understand why this is happening. The PowerShell terminal window inside vscode is immediately usable. So how is the creation of this PowerShell terminal window different than the PowerShell session the extension is trying to create?

Additionally, when I start the language server from this PowerShell terminal window, using the executed command shown in the log, it just immediately creates the session file.

So again the question, why is BeyondTrust/Avecto trusting this immediately?

Interestingly, when starting vscode elevated, BeyondTrust/Avecto immediately pops up with to a dialog to confirm 'powershell.exe'. This immediately enables the PowerShell terminal window to run.
The extension takes longer, eventually (1 minute?) BeyondTrust/Avecto pops up with a dialog to confirm 'powershell.exe' again. After confirmation, eventually (after 15 seconds?) the process proceeds.

Interestingly, when I wait with confirming the dialog, the same dialog 'The window is no longer responding' pops up 😉

I see this behavior only occurring with PowerShell 5.1 64-bit. With PowerShell 5.1 32-bit there is no observable delay, as well as with PowerShell 7.1.

Can you shed some light in which process is being delayed by BeyondTrust/Avecto?

Environment Information

Visual Studio Code

Name	Version
Operating System	Windows_NT x64 10.0.18363
BeyondTrust/Avecto DefendPoint	5.3.219.0
VSCode	1.52.0-insider
PowerShell Extension Version	2020.9.0

PowerShell Information

Name	Value
PSVersion	5.1.18362.1171
PSEdition	Desktop
PSCompatibleVersions	1.0 2.0 3.0 4.0 5.0 5.1.18362.1171
BuildVersion	10.0.18362.1171
CLRVersion	4.0.30319.42000
WSManStackVersion	3.0
PSRemotingProtocolVersion	2.3
SerializationVersion	1.1.0.1

Visual Studio Code Extensions

Visual Studio Code Extensions(Click to Expand)

Extension	Author	Version
powershell-preview	ms-vscode	2020.9.0
remote-wsl	ms-vscode-remote	0.51.4

[jbockle]

Can confirm, I have the same issue with defendpoint and ps 5.1 desktop. Installed ps 7.1 core and the blocking issue isn't replicated.

[mmascolino]

Have Avecto and PS5.1 and version 2020.1.0 of the VSCode extension is the last one that works for me.

[steviecoaster]

If I had to guess, your AV has started to more strictly monitor user space. You could attempt to add an exclusion for c:\Users\caveld\.vscode-insiders\extensions and see if the behavior remains the same.

[rjmholt]

Howdy @cveld, thanks for opening an issue.

The UI lock up in VSCode is, as far as I know, #2377. That's unrelated to Avecto/BeyondTrust, and is caused by a bad interaction between PSReadLine and VSCode's terminal's implementation of certain APIs.

As far as the Avecto/BeyondTrust issues go, we would also like to understand what's going on. We've found through a number of startup issue threads that the usual common element is the presence of these security solutions. But trying to get more information on the topic has been difficult. We would love to know exactly when and why Avecto/BeyondTrust triggers a scan of the extension at startup and what we can do to alleviate it, but so far we haven't had much to go on there.

Architecturally, the PowerShell extension is a client/server pair that talk to each other using LSP, which is pretty typical of VSCode extensions. VSCode + the PowerShell extension form the client (VSCode handles some LSP interactions directly, the extension handles some others). The PowerShell extension, when loaded, also starts the server, which is PowerShellEditorServices. This is a PowerShell process run to load in the PSES module and execute the Start-EditorServices cmdlet, which starts the language server. This process then hosts all the language services plus the integrated console, so that all the context is shared in the same process.

So with this in mind, some responses to your questions:

The PowerShell terminal window inside vscode is immediately usable. So how is the creation of this PowerShell terminal window different than the PowerShell session the extension is trying to create?

The PowerShell terminal window simply starts PowerShell. The extension's PowerShell process begins at that point, then loads the PSES module, then executes Start-EditorServices, then creates the required named pipes, then creates the session file, then waits for the client to connect, then starts the Integrated Console and language services. In order to effectively share context between editor completions and the Integrated Console, these two functions are actually managed by the same thread running the same PowerShell runspace, meaning there is extra complexity here too. I've been working on improving that when I find time in PowerShell/PowerShellEditorServices#1295.

Interestingly, when starting vscode elevated, BeyondTrust/Avecto immediately pops up with to a dialog to confirm 'powershell.exe'. This immediately enables the PowerShell terminal window to run.
The extension takes longer, eventually (1 minute?) BeyondTrust/Avecto pops up with a dialog to confirm 'powershell.exe' again. After confirmation, eventually (after 15 seconds?) the process proceeds.

This is very interesting information. Thanks for sharing it.

I see this behavior only occurring with PowerShell 5.1 64-bit.

Yes, I've also noticed this, but it's completely unclear as to why.

Can you shed some light in which process is being delayed by BeyondTrust/Avecto?

Unfortunately it's not something we know. If we did, we'd hopefully be able to do something about it.

Have Avecto and PS5.1 and version 2020.1.0 of the VSCode extension is the last one that works for me.

This is also interesting. As far as I know there should be no significant change there. However, we did migrate from a startup script to a direct invocation of a cmdlet for a number of reasons. I have thought about adding an option to instead use a script.

So to summarise:

• The UI freeze issue is something we know about, but we think is unrelated to Avecto/BeyondTrust. It was supposed to be fixed by VSCode at some point, but I'm not clear if the ostensible fix made it in. We're following up on that in VSCode UI locks up occassionally when using "Powershell: Restart Current Session" #2377.
• The Avecto/BeyondTrust startup issue is something we know about at a high level by piecing together various user reports. We've looked through crashdumps and other information but haven't been able to get any better information than "everyone who reports this has Avecto/BeyondTrust installed"
• The extension module and scripts are all signed with Microsoft certificates
• I've tried to contact BeyondTrust, so hopefully they'll get back to me

[mmascolino]

i'm willing to hop into a Teams screenshare if you want to poke around.

…

On Tue, Dec 1, 2020 at 5:45 PM Robert Holt ***@***.***> wrote: Howdy @cveld <https://github.com/cveld>, thanks for opening an issue. The UI lock up in VSCode is, as far as I know, #2377 <#2377>. That's unrelated to Avecto/BeyondTrust, and is caused by a bad interaction between PSReadLine and VSCode's terminal's implementation of certain APIs. As far as the Avecto/BeyondTrust issues go, we would also like to understand what's going on. We've found through a number of startup issue threads that the usual common element is the presence of these security solutions. But trying to get more information on the topic has been difficult. We would love to know exactly when and why Avecto/BeyondTrust triggers a scan of the extension at startup and what we can do to alleviate it, but so far we haven't had much to go on there. Architecturally, the PowerShell extension is a client/server pair that talk to each other using LSP, which is pretty typical of VSCode extensions. VSCode + the PowerShell extension form the client (VSCode handles some LSP interactions directly, the extension handles some others). The PowerShell extension, when loaded, also starts the server, which is PowerShellEditorServices <https://github.com/powershell/powershelleditorservices>. This is a PowerShell process run to load in the PSES module and execute the Start-EditorServices cmdlet, which starts the language server. This process then hosts all the language services plus the integrated console, so that all the context is shared in the same process. So with this in mind, some responses to your questions: The PowerShell terminal window inside vscode is immediately usable. So how is the creation of this PowerShell terminal window different than the PowerShell session the extension is trying to create? The PowerShell terminal window simply starts PowerShell. The extension's PowerShell process begins at that point, then loads the PSES module, then executes Start-EditorServices, then creates the required named pipes, then creates the session file, then waits for the client to connect, then starts the Integrated Console and language services. In order to effectively share context between editor completions and the Integrated Console, these two functions are actually managed by the same thread running the same PowerShell runspace, meaning there is extra complexity here too. I've been working on improving that when I find time in PowerShell/PowerShellEditorServices#1295 <PowerShell/PowerShellEditorServices#1295>. Interestingly, when starting vscode elevated, BeyondTrust/Avecto immediately pops up with to a dialog to confirm 'powershell.exe'. This immediately enables the PowerShell terminal window to run. The extension takes longer, eventually (1 minute?) BeyondTrust/Avecto pops up with a dialog to confirm 'powershell.exe' again. After confirmation, eventually (after 15 seconds?) the process proceeds. This is very interesting information. Thanks for sharing it. I see this behavior only occurring with PowerShell 5.1 64-bit. Yes, I've also noticed this, but it's completely unclear as to why. Can you shed some light in which process is being delayed by BeyondTrust/Avecto? Unfortunately it's not something we know. If we did, we'd hopefully be able to do something about it. Have Avecto and PS5.1 and version 2020.1.0 of the VSCode extension is the last one that works for me. This is also interesting. As far as I know there should be no significant change there. However, we did migrate from a startup script to a direct invocation of a cmdlet for a number of reasons. I have thought about adding an option to instead use a script. So to summarise: - The UI freeze issue is something we know about, but we think is unrelated to Avecto/BeyondTrust. It was supposed to be fixed by VSCode at some point, but I'm not clear if the ostensible fix made it in. We're following up on that in #2377 <#2377>. - The Avecto/BeyondTrust startup issue is something we know about at a high level by piecing together various user reports. We've looked through crashdumps and other information but haven't been able to get any better information than "everyone who reports this has Avecto/BeyondTrust installed" - The extension module and scripts are all signed with Microsoft certificates - I've tried to contact BeyondTrust, so hopefully they'll get back to me — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3077 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYCRNPNY24M6XXW7FNNZV3SSVWXXANCNFSM4UGYQ3FQ> .

[cveld]

My BeyondTrust/Avecto enterprise support team just upgraded my client to 5.6.148.0 and the problem went away!

[rjmholt]

Ok, I've gotten in contact with BeyondTrust support. I'm not sure how far it will go, but they've asked for a PGCaptureConfig and a Service/Hook/Driver test.

Given that I'm an intermediary here, I'm not sure how well this will go (I would encourage anyone seeing issues here to contact BeyondTrust support directly and discuss the issue with them, because I asked them directly in the email what we can do to prevent the issue and they've told me there are no best practices), but I'd like to give it a shot.

Please email [email protected] if you want to follow up and I can send you instructions on how to capture logs (instructions I can't read since I'm not a BeyondTrust customer).

EDIT: The BeyondTrust support representative I contacted told me that it's best that anyone experiencing this issue contact BeyondTrust directly. Probably at [email protected].

[ghost]

This issue is being closed as inactive, if this issue is still occurring it will be re-opened