saml2aws CLI fails to call AWS APIs in opt-in regions

[DanielOved93]

Hello,
I am encountering an issue when using the saml2aws CLI tool to call AWS APIs in the opt-in region il-central-1 (Tel Aviv) or ap-southeast-4 (Melbourne) . The error message received is:

(InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

Steps to Reproduce:

1. Configure saml2aws as per the instructions on the GitHub page.
2. Set the following environment variables:

export AWS_DEFAULT_REGION=il-central-1
export AWS_STS_REGIONAL_ENDPOINTS=regional

3. Execute saml2aws login --force to generate new credentials.
4. Attempt to call AWS APIs in the il-central-1 or ap-southeast-4 region. using aws sts get-caller-identity.

Expected Behavior:

The saml2aws CLI tool should be able to call AWS APIs in the il-central-1 or ap-southeast-4 regions (or another opt-in region) using regional STS endpoints without any errors.

Actual Behavior

The tool fails with the error:

(InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

No AssumeRoleWithSAML events are logged in CloudTrail for the il-central-1 or ap-southeast-4 regions, indicating that the STS endpoints (sts.il-central-1.amazonaws.com or sts.ap-southeast-4.amazonaws.com) are not being reached. However, calling AWS APIs using the AWS CLI with IAM Role from an EC2 instance in the same regions works correctly with no issues.

Environment

• saml2aws version: 2.36.16
• AWS CLI version: aws-cli/2.16.11
• OS: mac, ubuntu, amazon linux2

Additional Information

I have confirmed that the configuration is correct as per the GitHub instructions. Using saml2aws works in default regions like us-east-1 or eu-west-1, but fails in the opt-in regions like il-central-1 and ap-southeast-4. The same environment variables and credentials work with the AWS CLI in these regions.

Support Case Details

I have opened a support case with AWS Premium Support. The support engineer suggested that this might be an issue with the saml2aws tool itself, as it seems to be configured to use Global STS endpoints despite the environment variables being set to use regional endpoints.

Request

Could you please investigate this issue and provide a fix or workaround to allow saml2aws to work with regional STS endpoints in opt-in regions like il-central-1 and ap-southeast-4?

Thank you for your assistance!

[tinaboyce]

Thank you for raising this issue.

A few questions on this.

1. I have noticed the aws cli version you have is 1 minor version behind but it shouldn't impact what you are trying to do. Are you able to update and see if the problem still persist?
2. How are you calling aws sts get-caller-identity? by default, it should already call regional endpoint since you are using version 2.
3. Is the same behaviour occurs if you explicitly specify the STS regional endpoint of the aws cli sts get-caller-identity command with the --endpoint-url flag?
4. has the --profile been passed with the profile also had its region set through aws configure? (one way to find out is to look into ~/.aws/config

The support engineer suggested that this might be an issue with the saml2aws tool itself, as it seems to be configured to use Global STS endpoints despite the environment variables being set to use regional endpoints.

Quite possible, though I haven't had the time yet to look deeper into the code to give a definite answer. We are calling STS to obtain the credentials and it could be calling from global STS since it could be ignoring the env variables.

[DanielOved93]

Thank you for your response.

1. I have updated the AWS CLI to the latest versions available, tested multiple versions, and unfortunately, the issue persists unchanged.
2. When running aws sts get-caller-identity, it correctly utilizes regional endpoints by default, as expected with the AWS CLI versions I tested. I have verified this behavior and can confirm it consistently uses regional endpoints.
3. I explicitly specified the STS regional endpoint using the --endpoint-url flag with aws sts get-caller-identity, and it consistently behaves with the default regional endpoint for the respective regions (il-central-1 and ap-southeast-4).

aws sts get-caller-identity --profile TEST --region il-central-1 --endpoint-url=https://sts.il-central-1.amazonaws.com

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

4. The AWS profile used with saml2aws has its region explicitly set through aws configure. I confirmed this by checking the ~/.aws/credentials file, which correctly reflects the regional configuration for the profiles used. I also tried running saml2aws without setting the region in the configuration and attempted to specify the region in the AWS CLI using the --region flag or the AWS_DEFAULT_REGION environment variable, I also tried to set the region in the .saml2aws config file, but encountered the same behavior.

• Regarding the suspicion that saml2aws might be ignoring environment variables for regional endpoints:

Based on these tests, it appears that saml2aws may indeed be disregarding the environment variables (AWS_DEFAULT_REGION and AWS_STS_REGIONAL_ENDPOINTS) intended to enforce the use of regional STS endpoints. Despite setting these variables appropriately, the tool continues to exhibit behavior where it does not access regional STS endpoints in opt-in regions such as il-central-1 and ap-southeast-4.