[release/8.0][no-merge][testing][infra] Try Apple mobile platforms on iOS 18 and XCode 16

[matouskozak]

No description provided.

[matouskozak]

/azp run runtime-ioslike

[azure-pipelines]

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

[dotnet-policy-service]

Tagging subscribers to this area: @directhex, @matouskozak
See info in area-owners.md if you want to be subscribed.

[matouskozak]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[matouskozak]

iOS/tvOS error:

Signing /tmp/helix/working/9F130938/w/A8BB099B/e/tvOS.Device.Aot.Test.app
    "/Users/helix-runner/Library/Keychains/signing-certs.keychain-db"
     0 valid identities found
No valid signing identities found in the keychain

[matouskozak]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[matouskozak]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[carlossanlop]

I know this is a draft but I wanted to share the friendly reminder that Code Complete for the October Release is September 9. If we want this fix to be included in that release, please merge this PR before that date. Otherwise, it will have to wait until November.

[matouskozak]

I know this is a draft but I wanted to share the friendly reminder that Code Complete for the October Release is September 9. If we want this fix to be included in that release, please merge this PR before that date. Otherwise, it will have to wait until November.

This is NO-MERGE PR just for testing purposes, not going to merge it.

[matouskozak]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[matouskozak]

@oleksandr-didyk I tried re-running the CI on the scouting queues from dotnet/dnceng#3953 and I'm getting this error currently:

[02:34:25] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 104
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, Boolean pairedDevicesOnly, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 74
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.OrchestrateTest(GetAppBundleInfoFunc getAppBundlePath, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 189
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145

Do you think it could be still related or perhaps it's an error on xharness side?

[oleksandr-didyk]

@oleksandr-didyk I tried re-running the CI on the scouting queues from dotnet/dnceng#3953 and I'm getting this error currently:

[02:34:25] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 104
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, Boolean pairedDevicesOnly, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 74
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.OrchestrateTest(GetAppBundleInfoFunc getAppBundlePath, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 189
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145

Do you think it could be still related or perhaps it's an error on xharness side?

From the exception it seems unrelated, my guess would be its an xharness issue

[ivanpovazan]

That is most probably an xharness issue which is missing a newer mlaunch which will get fixed in: dotnet/xharness#1277 and will take some time to flow in into arcade and runtime.

[ivanpovazan]

@matouskozak new version of xharness got published, you can try bumping the version here to test it out

[matouskozak]

@matouskozak new version of xharness got published, you can try bumping the version here to test it out

I updated the xharness version and currently the listing of tvOS devices is failing with

[00:41:54.7580180] Xamarin.Hosting: Connected to DNCENGTVOS-091 (92fa8094fe76a5aa79e1dbba10f415eb2967b29c) in 00:00:00.0029934
[00:41:55.1149700] error MT0000: Unexpected error - Please file a bug report at https://github.com/xamarin/xamarin-macios/issues/new
[00:41:55.1750630] System.NullReferenceException: Object reference not set to an instance of an object.
[00:41:55.1750990]    at Xamarin.Hosting.RealDevice.ConnectToken() in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:line 121
[00:41:55.1751070]    at System.Threading.Tasks.Task.InnerInvoke()
[00:41:55.1751130]    at System.Threading.Tasks.Task.<>c.<.cctor>b__281_0(Object obj)
[00:41:55.1751300]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread, ExecutionContext, ContextCallback, Object)
[00:41:55.1751360] --- End of stack trace from previous location ---
[00:41:55.1751410]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread, ExecutionContext, ContextCallback, Object)
[00:41:55.1751460]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task&, Thread )
[00:41:55.1751520] --- End of stack trace from previous location ---
[00:41:55.1751570]    at Xamarin.Hosting.RealDevice.IsUnlockedAsync() in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:line 192
[00:41:55.1751700]    at Xamarin.Launcher.DevController.GetDeviceInfoAsync(RealDevice, Boolean) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/controller-device.cs:line 695
[00:41:55.1859940] Process mlaunch exited with 1
[00:41:55.2098420] Failed to parse device list: System.Exception: Failed to list devices.
   at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 104

The iOS queues are timing out with no tests being run

[ivanpovazan]

I suggest testing on physical devices with xharness CI to check if the problem is with the tooling or devices.