Durable activity failures are not propagated to orchestrator in fan out/fan in pattern

[YaroBear]

At one point #572 was fixed with this pull request #571.

If I do a single activity execution, the exception is propagated back up to the orchestrator. Good 👍

If I implement a fan out/fan in pattern like below, I don't get errors and only the activities that succeeded populate the $tasks variable.

$ErrorActionPreference = "Stop"

$ParallelTasks = foreach ($file in $FailedFiles) {
    # Create $FileDetails object
    Invoke-DurableActivity -FunctionName "ProcessAuditFileActivity" -Input $FileDetails -NoWait
}

$tasks = Wait-DurableTask -Task $ParallelTasks

# 1 out of 2 tasks fails, $tasks only has one result and no errors from the failed activity, or exceptions thrown

[cgillum]

Pinging @davidmrdavid for a quick triage.

[ruankrTs]

I am also having issues with this. The orchestrator knows about the failure but does not act on it at all. My example is as follow:

Trigger (Service Bus)

param([string] $SbMsg, $TriggerMetadata)

<#
Service bus message received:
{
  "inPutData": {
    "customerName": [
      "testcustomer1",
      "testcustomer2"
    ],
    "systemSet": "varied"
  }
}
#>

# Declare 5 cities to be randomly picked later
$city = @("London", "Paris", "New York", "Tokyo", "Sydney")

# Create a hashtable the orchestrator could use to iterate over
foreach ($item in $TriggerMetadata.inPutData.customerName) {
    $hashtableout += @{
        $item = @{
            city     = $city | Get-Random
            set      = $TriggerMetadata.inPutData.systemSet
            customer = $item
        }
    }
}

# Start the orchestration
$InstanceId = Start-DurableOrchestration -FunctionName "of-test" -Input $hashtableout -ErrorAction Stop
Write-Host "Started orchestration with ID = '$InstanceId'"

Orchestration Function

param($Context)
$ErrorActionPreference = "Stop"

$endResults = @{}

$orchResults =
foreach ($cusobj in $Context.Input) {
    try {
        Invoke-DurableActivity -FunctionName 'af-test' -Input $cusobj -NoWait
    }
    catch {
         # Gather Failed activities
        $endResults += @{
            $cusobj.customer = "Failed"
        }
    }
}

$Outputs1 = Wait-DurableTask -Task $orchResults

# Gather Successful activities
foreach ($output in $Outputs1) {
    $endResults += @{
        $output.customer = "Success"
    }
}

# Output all
$endResults

Activity Function

param($name)


if ($name.customer -eq "testcustomer1") {
    throw "This is a test error"
}
else {
    Write-Information -MessageData "Success for $name from $($name.city)"
    # Add the results so the orchestrator can show it
    $name[0].Add("result", "Success")
}

$name

So if at least 1 activity succeeds, the orchestrator outputs success but it is not catching the failed activity. Idea is to set a custom status within the catch block if there was a failure on any activities.

Looking at the orchestration instance the failure shows, but it never enters the catch block at all.

As a result of this, if there is no success in any of the activities it throws out a non-deterministic

Would be good if we can get a way to catch these failures in the fan-in/fan-out orchestrator, or have a way to interrogate the instance to establish the outcome of the activities. I notice there is a durable monitor, however I do not particularly want to raise more activities just to get the status of others.

Hoping there is a workaround or something that can be done here?

[davidmrdavid]

My apologies, I thought this had been resolved already. I've opened a PR fixing exactly this: #918

The challenge with fixing this bug is that it's hard to fix without potentially introducing unexpected behavioral changes on running apps. I need to discuss this with the team, but I suspect that I'll have to make this an opt-in feature to avoid breaking existing apps. Off the top of my head, we will most likely add a "EnableFailurePropagation" flag of sorts that fixes this issue for existing apps, and make that the default behavior starting in PS7.4.

I'll keep this issue posted on updates, and I will explore whether there are workarounds that do not require waiting for a deployment.

[ruankrTs]

Been quite some time since the last response so was wondering if there has been any movement on this yet @davidmrdavid ?? Alternatively is there a workaround or suggested method to achieve this we can consider at present?

[ruankrTs]

The new preview SDK also seems to be having issues with error logging in the orchestrator.. Not sure if anyone is even bothered to look into it anymore as not getting any responses anywhere. Such a shame as it's the only issue preventing us from properly using this atm.

[davidmrdavid]

My apologies for the delayed response, @Ruankr. I replied on the other thread: Azure/azure-functions-durable-powershell#65 (comment)