CRAYSAT-956: Replace CAPMC calls with PCS calls

[jack-stanek-hpe]

Summary and Scope

Replace calls to the CAPMC API with equivalent calls to the PCS API.

I tried to keep the software API of PCSClient mostly compatible with CAPMCClient, though there may be differences that aren't covered in the tests.

This hasn't been tested on a real system with PCS installed yet.

Issues and Related PRs

• Resolves CRAYSAT-956

Testing

Tested on:

• Local development environment

Test description:

Run unit tests.

Pull Request Checklist

• Version number(s) incremented, if applicable
• Copyrights updated
• License file intact
• Target branch correct
• CHANGELOG.md updated
• Testing is appropriate and complete, if applicable
• HPC Product Announcement prepared, if applicable

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

<sup><sub>🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!</sub></sup>

[haasken-hpe]

I'm just leaving some comments based on a code review with @shivaprasad-metimath and team that we did this morning.

[haasken-hpe]

I haven't reviewed the whole PR, but I've found some issues in the new PCSClient class that need to be addressed.

[haasken-hpe]

There are some functional issues that need to be fixed before you try to test this with sat swap blade.

[shivaprasad-metimath]

Test output:
sat_swap_validation.txt

• sat swap performed on Baldar
• tested on xname=x8000c0s0

Observations:

During enable of blade, time out was observed;
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
WARNING: Waiting for "cronjob hms-discovery in namespace services" timed out, recreating cronjob.
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
ERROR: Could not add blade: Timed out waiting for hms-discovery cron job to resume

During the power on of the nodes, timeout was observed
INFO: Waiting for BOS session 68fb2303-1fa6-4aa6-9b06-0426140824d8 to reach target state complete. Session template: compute-23.11.0-beta.6.x86_64shs-b2.2.0-6
INFO: Waiting for BOS session 68fb2303-1fa6-4aa6-9b06-0426140824d8 to reach target state complete. Session template: compute-23.11.0-beta.6.x86_64shs-b2.2.0-6
ERROR: BOS boot timed out after 900 seconds for session template: compute-23.11.0-beta.6.x86_64shs-b2.2.0-6.
INFO: Session 68fb2303-1fa6-4aa6-9b06-0426140824d8: 0% components succeeded, 0% components failed
ERROR: Boot failed or timed out for session template: compute-23.11.0-beta.6.x86_64shs-b2.2.0-6

[haasken-hpe]

One minor issue was discovered with PCSError class. This issue has been present since the original PR was raised. Shiva and I just noticed in while doing a code review on a meeting. It should be easy to address.

[haasken-hpe]

Please do file a CRAYSAT Jira for us to look at this problem you noted as well:

During enable of blade, time out was observed;

Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
WARNING: Waiting for "cronjob hms-discovery in namespace services" timed out, recreating cronjob.
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
ERROR: Could not add blade: Timed out waiting for hms-discovery cron job to resume

And please mark it related to CRAYSAT-1623. I was expecting that Jira to have resolved this issue.

[shivaprasad-metimath]

Please do file a CRAYSAT Jira for us to look at this problem you noted as well:

During enable of blade, time out was observed;

Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
WARNING: Waiting for "cronjob hms-discovery in namespace services" timed out, recreating cronjob.
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
ERROR: Could not add blade: Timed out waiting for hms-discovery cron job to resume

And please mark it related to CRAYSAT-1623. I was expecting that Jira to have resolved this issue.

sure Ryan, will have new one created and add a ref to the old ticket

[prasanthkurian]

Please do file a CRAYSAT Jira for us to look at this problem you noted as well:

During enable of blade, time out was observed;

Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
WARNING: Waiting for "cronjob hms-discovery in namespace services" timed out, recreating cronjob.
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
ERROR: Could not add blade: Timed out waiting for hms-discovery cron job to resume

And please mark it related to CRAYSAT-1623. I was expecting that Jira to have resolved this issue.

sure Ryan, will have new one created and add a ref to the old ticket

CRAYSAT-1818 is filed by Harold recently. Can we use this jira instead of filing new one.

[shivaprasad-metimath]

Please do file a CRAYSAT Jira for us to look at this problem you noted as well:

During enable of blade, time out was observed;

Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
WARNING: Waiting for "cronjob hms-discovery in namespace services" timed out, recreating cronjob.
Waiting for condition "HMS Discovery Scheduled" timed out after 109 seconds
ERROR: Could not add blade: Timed out waiting for hms-discovery cron job to resume

And please mark it related to CRAYSAT-1623. I was expecting that Jira to have resolved this issue.

sure Ryan, will have new one created and add a ref to the old ticket

CRAYSAT-1818 is filed by Harold recently. Can we use this jira instead of filing new one.

Let me take a look at it Prasanth, it is same error reported, seems like a documentation changes are added