[Bug] Unable to Get Console Handle of Windows PowerShell ISE

[msJinLei]

Library version used

N/A

.NET version

.net standard 2.0

Scenario

PublicClient - desktop app

Is this a new or an existing app?

The app is in production, and I have upgraded to a new version of MSAL

Issue description and reproduction steps

The customer reports that WAM window cannot popup when use Windows PowerShell ISE console

We find the cause is GetConsoleWindow() are return nullptr in the following cod snippet.

enum GetAncestorFlags
{   
    GetParent = 1,
    GetRoot = 2,
    /// <summary>
    /// Retrieves the owned root window by walking the chain of parent and owner windows returned by GetParent.
    /// </summary>
    GetRootOwner = 3
}

/// <summary>
/// Retrieves the handle to the ancestor of the specified window.
/// </summary>
/// <param name="hwnd">A handle to the window whose ancestor is to be retrieved.
/// If this parameter is the desktop window, the function returns NULL. </param>
/// <param name="flags">The ancestor to be retrieved.</param>
/// <returns>The return value is the handle to the ancestor window.</returns>
[DllImport("user32.dll", ExactSpelling = true)]
static extern IntPtr GetAncestor(IntPtr hwnd, GetAncestorFlags flags);

[DllImport("kernel32.dll")]
static extern IntPtr GetConsoleWindow();

// This is your window handle!
public IntPtr GetConsoleOrTerminalWindow()
{
    IntPtr consoleHandle = GetConsoleWindow();
    IntPtr handle = GetAncestor(consoleHandle, GetAncestorFlags.GetRootOwner );
    
    return handle;
}

from https://learn.microsoft.com/en-us/entra/msal/dotnet/acquiring-tokens/desktop-mobile/wam#parent-window-handles

Relevant code snippets

[DllImport("kernel32.dll")]
static extern IntPtr GetConsoleWindow();

// This is your window handle!
public IntPtr GetConsoleOrTerminalWindow()
{
    IntPtr consoleHandle = GetConsoleWindow();
    IntPtr handle = GetAncestor(consoleHandle, GetAncestorFlags.GetRootOwner );
    
    return handle;
}

Expected behavior

GetConsoleWindow() can return the current window handler but actually return nullptr

Identity provider

Microsoft Entra ID (Work and School accounts and Personal Microsoft accounts)

Regression

No response

Solution and workarounds

No response

Related issue

          In ISE, on Windows 11 mine says the following, does this correlate?

# 12.0.0    Az
# 3.0.0     Az.Accounts
# 6.0.0     Az.KeyVault
Connect-AzAccount -Tenantid [TenantID] -Subscription [Subscription]
# > Connect-AzAccount : InteractiveBrowserCredential authentication failed: A window handle must be configured. See https://aka.ms/msal-net-wam#parent-window-handles 
# > Could not find tenant id for provided tenant domain '[TenantID]'. Please ensure that the provided user is found in the provided tenant domain.

Originally posted by @weyCC81 in Azure/azure-powershell#25005 (comment)