CASMTRIAGE-5835 - worker rebuild stuck at "canu validate network bgp --verbose"

[spillerc-hpe]

Summary and Scope

In the event that the management network spine switch password is not set in vault, the guidance given by the check_bgp_neighbors_established.sh script contains a syntax error and fails.

20230809_142217_574952377 ## executable starting ##
###########################################################################
NAME DATA AGE
metallb 1 70d
CHN
No value found at secret/net-creds/switch_admin
command terminated with exit code 2
ERROR: Detected Vault is running. Missing switch admin password from vault. Please run the following commands:
VAULT_PASSWD=$(kubectl -n vault get secrets cray-vault-unseal-keys -o json | jq -r .data["vault-root"] | base64 -d)
alias vault="kubectl -n vault exec -i cray-vault-0 -c vault – env VAULT_TOKEN=\"$VAULT_PASSWD\" VAULT_ADDR=http://127.0.0.1:8200/
VAULT_FORMAT=json vault"
vault kv put secret/net-creds/switch_admin admin=SWITCH_ADMIN_PASSWORD
FAIL
###########################################################################
20230809_142218_831707098 ## executable completed (exit code=20) ##

The jq command to extract the vault-root token is missing quotes.

ncn-m001:~ # VAULT_PASSWD=$(kubectl -n vault get secrets cray-vault-unseal-keys -o json | jq -r .data["vault-root"] | base64 -d)
jq: error: vault/0 is not defined at <top-level>, line 1:
.data[vault-root]
jq: error: root/0 is not defined at <top-level>, line 1:
.data[vault-root]
jq: 2 compile errors

Issues and Related PRs

• Resolves CASMTRIAGE-5835

Testing

Tested on:

• fanta
• Customer system B
• surtur

Test description:

ncn-m001:~/cspiller # ./check_bgp_neighbors_established.sh
NAME      DATA   AGE
metallb   1      8d
CHN
No value found at secret/net-creds/switch_admin
command terminated with exit code 2
ERROR: Detected Vault is running.  Missing switch admin password from vault.  Please run the following commands:
        VAULT_PASSWD=$(kubectl -n vault get secrets cray-vault-unseal-keys -o json | jq -r '.data["vault-root"]' |  base64 -d)
        alias vault="kubectl -n vault exec -i cray-vault-0 -c vault -- env VAULT_TOKEN=\"$VAULT_PASSWD\" VAULT_ADDR=http://127.0.0.1:8200 VAULT_FORMAT=json vault"
        vault kv put secret/net-creds/switch_admin admin=SWITCH_ADMIN_PASSWORD
FAIL

ncn-m001:~/cspiller # VAULT_PASSWD=$(kubectl -n vault get secrets cray-vault-unseal-keys -o json | jq -r '.data["vault-root"]' |  base64 -d)
ncn-m001:~/cspiller # alias vault="kubectl -n vault exec -i cray-vault-0 -c vault -- env VAULT_TOKEN=\"$VAULT_PASSWD\" VAULT_ADDR=http://127.0.0.1:8200 VAULT_FORMAT=json vault"
ncn-m001:~/cspiller # vault kv put secret/net-creds/switch_admin admin=testpass
ncn-m001:~/cspiller # vault kv get secret/net-creds/switch_admin
{
  "request_id": "a7a2f1fc-4159-95ed-47dc-ac0347b97a1b",
  "lease_id": "",
  "lease_duration": 2764800,
  "renewable": false,
  "data": {
    "admin": "testpass"
  },
  "warnings": null
}

Risks and Mitigations

Pull Request Checklist

• Version number(s) incremented, if applicable
• Copyrights updated
• License file intact
• Target branch correct
• CHANGELOG.md updated
• Testing is appropriate and complete, if applicable
• HPC Product Announcement prepared, if applicable

[denniswalker]

Holding for CSM 1.4.3