Fully tested and working

content/rosa/cross-account-efs/index.md

@@ -30,6 +30,8 @@ This is a guide to enable cross-account EFS mounting on ROSA.
    | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
    export AWS_ACCOUNT_A_ID="Account ID that holds your ROSA cluster"
    export AWS_ACCOUNT_B_ID="Account ID that will hold your EFS filesystem"
+   export AWS_ACCOUNT_A_VPC_CIDR="CIDR of the VPC of your ROSA cluster"
+   export AWS_ACCOUNT_B_VPC_CIDR="CIDR of the VPC of your EFS filesystem"
    export SCRATCH_DIR=/tmp/scratch
    export AWS_PAGER=""
    mkdir -p $SCRATCH_DIR
@@ -266,6 +268,63 @@ In this account, we need to allow certain permissions to allow the EFS operator
 
 - To be done, but essentially you need to peer your VPCs together, and ensure that DNS resolving is enabled. Also set up the route tables. Working on this now.
 
+### Set up Account A
+
+1. Swap to your Account A profile
+
+   ```bash
+   export AWS_DEFAULT_PROFILE=aws_account_a
+   ```
+
+1. Export the VPC IDs from Account A (with your ROSA Cluster) and Account B (where your EFS will live)
+
+   ```bash
+   export ACCOUNT_A_VPC_ID="Your VPC ID here"
+   export ACCOUNT_B_VPC_ID="Your VPC ID here"
+   ```
+
+1. Start a peering request to Account B from Account A
+
+   ```bash
+   PEER_REQUEST_ID=$(aws ec2 create-vpc-peering-connection --vpc-id "${ACCOUNT_A_VPC_ID}" --peer-vpc-id "${ACCOUNT_B_VPC_ID}" --peer-owner-id "${AWS_ACCOUNT_B_ID}" --query VpcPeeringConnection.VpcPeeringConnectionId --output text)
+   ```
+
+1. Accept the peering request from Account B
+
+   ```bash
+   export AWS_DEFAULT_PROFILE=aws_account_b
+   aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id "${PEER_REQUEST_ID}"
+   ```
+
+1. Get the route table IDs for Account A and add route to Account B VPC
+
+   ```bash
+   export AWS_DEFAULT_PROFILE=aws_account_a
+   for NODE in $(oc get nodes --selector=node-role.kubernetes.io/worker | tail -n +2 | awk '{print $1}')
+   do 
+     SUBNET=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].NetworkInterfaces[*].SubnetId' | jq -r '.[0][0][0]')
+     echo SUBNET is ${SUBNET}
+     ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=${SUBNET}" --query 'RouteTables[*].RouteTableId' | jq -r '.[0]')
+     echo Route table ID is $ROUTE_TABLE_ID
+     aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_B_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
+   done
+   ```
+   
+1. Get the route table IDS for Account B and add route to Account A VPC
+
+   ```bash
+   export AWS_DEFAULT_PROFILE=aws_account_b
+   export ROUTE_TABLE_ID="Put your Route table ID here"
+   echo Route table ID is $ROUTE_TABLE_ID
+   aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_A_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
+   ```
+
+1. Enable DNS resolution for Account A to read from Account B's VPC
+
+   ```bash
+   aws ec2 modify-vpc-peering-connection-options --vpc-peering-connection-id ${PEER_REQUEST_ID} --accepter-peering-connection-options AllowDnsResolutionFromRemoteVpc=true
+   ```
+
 ## Deploy and test the AWS EFS Operator
 
 1. Create a Secret to tell the AWS EFS Operator which IAM role to request.
@@ -369,23 +428,12 @@ In this account, we need to allow certain permissions to allow the EFS operator
      --query 'Reservations[*].Instances[*].{VpcId:VpcId}' \
      --region $AWS_REGION \
      | jq -r '.[0][0].VpcId')
-   CIDR=$(aws ec2 describe-vpcs \
-     --filters "Name=vpc-id,Values=$VPC" \
-     --query 'Vpcs[*].CidrBlock' \
-     --region $AWS_REGION \
-     | jq -r '.[0]')
    SG=$(aws ec2 describe-instances --filters \
      "Name=private-dns-name,Values=$NODE" \
      --query 'Reservations[*].Instances[*].{SecurityGroups:SecurityGroups}' \
      --region $AWS_REGION \
      | jq -r '.[0][0].SecurityGroups[0].GroupId')
-   echo "CIDR - $CIDR,  SG - $SG"
-   ```
-
-1. Export the CIDR of the VPC in Account B, that will contain the EFS Filesystem
-
-   ```bash
-   export ACCOUNT_B_CIDR="your VPC cidr"
+   echo "SG - $SG"
    ```
 
 1. Update the Security Groups in Account A to allow NFS traffic to your nodes from EFS
@@ -395,7 +443,7 @@ In this account, we need to allow certain permissions to allow the EFS operator
     --group-id $SG \
     --protocol tcp \
     --port 2049 \
-    --cidr $ACCOUNT_B_CIDR | jq .
+    --cidr $AWS_ACCOUNT_B_VPC_CIDR | jq .
    ```
 
 > At this point you can create either a single Zone EFS filesystem, or a Region wide EFS filesystem. To simplify this document, we're going to give only an example of a Region wide EFS filesystem.
@@ -421,12 +469,11 @@ In this account, we need to allow certain permissions to allow the EFS operator
 
    ```bash
    for SUBNET in $(aws ec2 describe-subnets \
-     --filters Name=vpc-id,Values=$VPC Name='tag:kubernetes.io/role/internal-elb',Values='*' \
      --query 'Subnets[*].{SubnetId:SubnetId}' \
      --region $AWS_REGION \
      | jq -r '.[].SubnetId'); do \
        MOUNT_TARGET=$(aws efs create-mount-target --file-system-id $EFS \
-          --subnet-id $SUBNET --security-groups $SG \
+          --subnet-id $SUBNET \
           --region $AWS_REGION \
           | jq -r '.MountTargetId'); \
        echo $MOUNT_TARGET; \