[upgrade] During upgrade process create bucket using S3 aws api throws wrong error

[PravinRanjan10]

Environment info

• NooBaa Version: VERSION
  Nobaa version:4.18.0

Actual behavior

s3-u1 mb s3://bucket-api-122
urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host '192.168.4.102'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host '192.168.4.102'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host '192.168.4.102'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
make_bucket failed: s3://bucket-api-122 An error occurred (InternalError) when calling the CreateBucket operation (reached max retries: 2): We encountered an internal error. Please try again.

Expected behavior

Expection is to throw some proper error. Atleast remove eached max retries: 2 from error log

Steps to reproduce

More information - Screenshots / Logs / Other output

[romayalon]

Hey @PravinRanjan10

1. please share logs
2. Did you run this command from an upgraded node/a node that wasn't upgraded yet? if the first one I would expect this behavior because doing changes to the config directory, like creating buckets is not allowed from an upgraded node.

[romayalon]

@PravinRanjan10 please check my last comment

[PravinRanjan10]

Hey @PravinRanjan10

1. please share logs
2. Did you run this command from an upgraded node/a node that wasn't upgraded yet? if the first one I would expect this behavior because doing changes to the config directory, like creating buckets is not allowed from an upgraded node.

running aws s3 api command runs on --endpoint so it doesn't matter from which node this command is executed.

Let me explain the scenario:
let's say i have 2 node cluster and upgraded the noobaa-rpm and restarted the service(just to note, till now didn't run noobaa-cli upgrade command). So bucket/account creation using noobaa-cli is NOT expected and it also throw erroe that system.json is locked etc.

But in case of aws s3 api, it doesn't throw the proper error.[It is expected to throw error because upgrde is not completed yet]. It throw general error like.... error occurred (InternalError) reached max retries: 2).

So my expectation is can we improve the error output

[romayalon]

@PravinRanjan10 The --endpoint matters, when node upgraded its RPM but the config directory is not upgraded, this node is not allowed to do changes to the config directory, regardless if it's coming from noobaa-cli or S3 api, this is the expected behavior, you can't expect to do changes to an old version of the config directory from a new version source code.