helm test errors in Mojaloop helm v15.0.0.0 (can't test 3ppi or bulk with helm test)

[tdaly61]

helm test --logs <=== has errors and appears to abort with error

Error: unable to get pod logs for ml-ml-ttk-test-cleanup: pods "ml-ml-ttk-test-cleanup" not found
This appears to imply that 3ppi and bulk tests can't be tested with helm test as documented i.e. with helm test facility

To ensure that mini-loop is not introducing any variables and to aid re-reproducibility I extracted the current Mojaloop v15.0.0.0 readme instructions into a small script called ml-default-test.sh

#!/usr/bin/env bash 
# script to test vanilla deploy of Mojaloop from package repo 
# assumes correct version of kubernetes , helm, ingress etc already installed and configured.
# April 2023 
# see install instructions in readme.md at https://github.com/mojaloop/helm and https://github.com/mojaloop/helm/blob/master/thirdparty/README.md

helm repo update 
helm repo list
helm delete ml
helm delete be 

helm install be --wait --timeout 300s mojaloop/example-mojaloop-backend
helm install ml --wait --timeout 2400s mojaloop/mojaloop \
  --set account-lookup-service.account-lookup-service.config.featureEnableExtendedPartyIdType=true \
  --set account-lookup-service.account-lookup-service-admin.config.featureEnableExtendedPartyIdType=true \
  --set thirdparty.enabled=true \
  --set ml-ttk-test-setup-tp.tests.enabled=true \
  --set ml-ttk-test-val-tp.tests.enabled=true 
   
helm test ml --logs

Results

ubuntu@miniloop-vm ~/mini-loop/util/test (v5.0) $ ./ml-default-testing.sh
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "codecentric" chart repository
...Successfully got an update from the "kiwigrid" chart repository
...Successfully got an update from the "elastic" chart repository
...Successfully got an update from the "mojaloop" chart repository
...Successfully got an update from the "charts" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME URL
kiwigrid https://kiwigrid.github.io
elastic https://helm.elastic.co
bitnami https://charts.bitnami.com/bitnami
mojaloop http://mojaloop.io/helm/repo/
charts https://docs.mojaloop.io/charts/repo
codecentric https://codecentric.github.io/helm-charts
release "ml" uninstalled
Error: uninstall: Release not loaded: be: release: not found
NAME: be
LAST DEPLOYED: Mon Apr 24 07:10:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
coalesce.go:220: warning: cannot overwrite table with non table for mojaloop.thirdparty.consent-oracle.env (map[])
NAME: ml
LAST DEPLOYED: Mon Apr 24 07:12:50 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing mojaloop.

Your release is named: ml.

Mojaloop Official Website: https://mojaloop.io

To learn more about Mojaloop: https://docs.mojaloop.io
To learn more about Mojaloop deployments: https://docs.mojaloop.io/legacy/deployment-guide
To learn more about this release: https://github.com/mojaloop/helm/releases/tag/v15.0.0

This Helm chart lives @ http://github.com/mojaloop/helm

To learn more about this deployment, try:

$ helm status ml

Use the following command to execute Test cases:

$ helm -n default test ml

Use the following command to execute Test cases and print logs to console:

$ helm -n default test ml --logs

Use the following command to execute a specific Test case:

$ helm -n default test ml --filter "name=ml-<TEST_NAME_1>,name=ml-<TEST_NAME_2>"

Use the following command to ignore a specific Test case:

$ helm -n default test ml --filter "!name=ml-<TEST_NAME_1>"

View Test logs with the following commands:
$ kubectl -n default logs pod/ml-ml-ttk-test-setup
$ kubectl -n default logs pod/ml-ml-ttk-test-val-gp
$ kubectl -n default logs pod/ml-ml-ttk-test-setup-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-val-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-cleanup

Copyright © 2020 Mojaloop Foundation
NAME: ml
LAST DEPLOYED: Mon Apr 24 07:12:50 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: ml-ml-ttk-test-setup
Last Started: Mon Apr 24 07:29:13 2023
Last Completed: Mon Apr 24 07:31:49 2023
Phase: Succeeded
TEST SUITE: ml-ml-ttk-test-val-gp
Last Started: Mon Apr 24 07:31:49 2023
Last Completed: Mon Apr 24 07:36:34 2023
Phase: Failed
NOTES:
Thank you for installing mojaloop.

Your release is named: ml.

Mojaloop Official Website: https://mojaloop.io

To learn more about Mojaloop: https://docs.mojaloop.io
To learn more about Mojaloop deployments: https://docs.mojaloop.io/legacy/deployment-guide
To learn more about this release: https://github.com/mojaloop/helm/releases/tag/v15.0.0

This Helm chart lives @ http://github.com/mojaloop/helm

To learn more about this deployment, try:

$ helm status ml

Use the following command to execute Test cases:

$ helm -n default test ml

Use the following command to execute Test cases and print logs to console:

$ helm -n default test ml --logs

Use the following command to execute a specific Test case:

$ helm -n default test ml --filter "name=ml-<TEST_NAME_1>,name=ml-<TEST_NAME_2>"

Use the following command to ignore a specific Test case:

$ helm -n default test ml --filter "!name=ml-<TEST_NAME_1>"

View Test logs with the following commands:
$ kubectl -n default logs pod/ml-ml-ttk-test-setup
$ kubectl -n default logs pod/ml-ml-ttk-test-val-gp
$ kubectl -n default logs pod/ml-ml-ttk-test-setup-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-val-tp
$ kubectl -n default logs pod/ml-ml-ttk-test-cleanup

Copyright © 2020 Mojaloop Foundation

Error: unable to get pod logs for ml-ml-ttk-test-cleanup: pods "ml-ml-ttk-test-cleanup" not found
ubuntu@miniloop-vm ~/mini-loop/util/test (v5.0) $ kubectl -n default logs pod/ml-ml-ttk-test-val-tp
Error from server (NotFound): pods "ml-ml-ttk-test-val-tp" not found