Skip to content

Installing Istio with a Custom Envoy Proxy

Jump to bottom
Peter Hao Chen edited this page Mar 11, 2020 · 3 revisions

As of Authservice 0.2.0, there is no need to install Istio with a Custom Envoy Proxy. This docs will be deleted soon.

Version of Istio

At the time of writing, the team targeted Istio 1.4.0 as the version to build the custom proxy sidecar docker image against.

Create a VM to build the custom Envoy/Proxy image

Create an N1 VM with 96 cpus (high cpu type) and with 120 GB SSD drive and selected Ubuntu 18.04 LTS as the operating system.

Connect to the VM in preparation of acquiring the necessary build tools. Use this ssh command:

gcloud beta compute --project "gcp-project-name" ssh --zone "us-central1-b" "vm-name"

Install the required build tools

# Install Bazel
sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v0.0.8/bazelisk-linux-amd64
sudo chmod +x /usr/local/bin/bazel

# Install some basic build tools
sudo apt-get update
sudo apt-get install -y libtool cmake automake autoconf make ninja-build \
    curl wget unzip virtualenv build-essential llvm clang libc++-dev libc++abi-dev lld

# Install golang
sudo add-apt-repository -y ppa:longsleep/golang-backports
sudo apt-get update
sudo apt-get install -y golang-go

# Install docker (see https://docs.docker.com/install/linux/docker-ce/ubuntu)
sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
# Allow non-root users to run docker commands (see https://docs.docker.com/install/linux/linux-postinstall)
sudo usermod -aG docker $USER
sudo systemctl enable docker
# Log out and log back in so that group membership is re-evaluated
exit

After logging out, log back into the VM and continue installing build tools.

# Verify that docker is installed correctly:
docker run hello-world

# Uninstall gcloud, because this version does not allow installing plugins
sudo snap remove google-cloud-sdk
# Then reinstall gcloud (see https://cloud.google.com/sdk/docs/#linux)
wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-271.0.0-linux-x86_64.tar.gz # or latest
tar xvf google-cloud-sdk-271.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh \
  --quiet \
  --command-completion true \
  --path-update true \
  --additional-components docker-credential-gcr
# Log out and log back in so that the path is updated
exit

Log back in to be ready to continue.

Clone the repositories needed to build the proxy image

On the build server:

mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/istio/envoy.git envoy
git clone https://github.com/istio/proxy.git proxy
git clone https://github.com/istio/istio.git istio
  • The envoy project is the Istio community's fork of Envoy used to provide the envoy proxy server. This project is needed because custom changes will be made in this project.
  • The proxy project is used to build Istio's version of Envoy which includes Istio's custom extensions. This will be used to build the binary that includes the custom changes.
  • The istio project is the top-level repo for Istio. It will be used to build the docker image for the sidecar, which will include the proxy binary built from the proxy repo.

Use istio/proxy and istio/envoy together to build the envoy binary

  1. cd ~/workspace/envoy
  2. git checkout release-1.4
  3. Apply these changes (also shown in the diff below):
    1. Manually apply this PR, because the GitHub repo was deleted. This PR points the code to a mirror of the original repo: https://github.com/envoyproxy/envoy/pull/9072/commits/05f9c66748bba3ef60fb7ffc3511ecb3e42e59b4
    2. Manually apply this PR, as it contains the bug fix: https://github.com/envoyproxy/envoy/pull/8668/commits/d4db053425274b8d7778868c6dc29122fc9002bf 
diff --git a/bazel/repositories.bzl b/bazel/repositories.bzl
index 46978aaa4..f6b3466a5 100644
--- a/bazel/repositories.bzl
+++ b/bazel/repositories.bzl
@@ -128,7 +128,7 @@ def envoy_dependencies(skip_targets = []):
     _com_github_circonus_labs_libcircllhist()
     _com_github_cyan4973_xxhash()
     _com_github_datadog_dd_opentracing_cpp()
-    _com_github_eile_tclap()
+    _com_github_mirror_tclap()
     _com_github_envoyproxy_sqlparser()
     _com_github_fmtlib_fmt()
     _com_github_gabime_spdlog()
@@ -243,14 +243,14 @@ def _com_github_envoyproxy_sqlparser():
         actual = "@com_github_envoyproxy_sqlparser//:sqlparser",
     )

-def _com_github_eile_tclap():
+def _com_github_mirror_tclap():
     _repository_impl(
-        name = "com_github_eile_tclap",
+        name = "com_github_mirror_tclap",
         build_file = "@envoy//bazel/external:tclap.BUILD",
     )
     native.bind(
         name = "tclap",
-        actual = "@com_github_eile_tclap//:tclap",
+        actual = "@com_github_mirror_tclap//:tclap",
     )

 def _com_github_fmtlib_fmt():
diff --git a/bazel/repository_locations.bzl b/bazel/repository_locations.bzl
index 0d298f3bc..e664a4396 100644
--- a/bazel/repository_locations.bzl
+++ b/bazel/repository_locations.bzl
@@ -73,10 +73,10 @@ REPOSITORY_LOCATIONS = dict(
         strip_prefix = "sql-parser-5f50c68bdf5f107692bb027d1c568f67597f4d7f",
         urls = ["https://github.com/envoyproxy/sql-parser/archive/5f50c68bdf5f107692bb027d1c568f67597f4d7f.tar.gz"],
     ),
-    com_github_eile_tclap = dict(
+    com_github_mirror_tclap = dict(
         sha256 = "f0ede0721dddbb5eba3a47385a6e8681b14f155e1129dd39d1a959411935098f",
         strip_prefix = "tclap-tclap-1-2-1-release-final",
-        urls = ["https://github.com/eile/tclap/archive/tclap-1-2-1-release-final.tar.gz"],
+        urls = ["https://github.com/mirror/tclap/archive/tclap-1-2-1-release-final.tar.gz"],
     ),
     com_github_fmtlib_fmt = dict(
         sha256 = "4c0741e10183f75d7d6f730b8708a99b329b2f942dad5a9da3385ab92bb4a15c",
diff --git a/source/extensions/filters/http/ext_authz/ext_authz.cc b/source/extensions/filters/http/ext_authz/ext_authz.cc
index 3878c2ba9..a6ea4a43d 100644
--- a/source/extensions/filters/http/ext_authz/ext_authz.cc
+++ b/source/extensions/filters/http/ext_authz/ext_authz.cc
@@ -216,10 +216,14 @@ void Filter::onComplete(Filters::Common::ExtAuthz::ResponsePtr&& response) {
           ENVOY_STREAM_LOG(trace,
                            "ext_authz filter added header(s) to the local response:", callbacks);
           for (const auto& header : headers) {
-            ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
+            // ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
             response_headers.remove(header.first);
-            response_headers.addCopy(header.first, header.second);
+            // response_headers.addCopy(header.first, header.second);
           }
+         for (const auto& header : headers) {
+            ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
+           response_headers.addCopy(header.first, header.second);
+         }
         },
         absl::nullopt, RcDetails::get().AuthzDenied);
     callbacks_->streamInfo().setResponseFlag(StreamInfo::ResponseFlag::UnauthorizedExternalService);
  1. cd ~/workspace/proxy
  2. git checkout 1.4.0
  3. Apply these changes (also shown in the diff below):
    1. Mount the local clone of istio/envoy into the build container, and do not remove the container when the build finishes
    2. Edit WORKSPACE to use the local clone of istio/envoy as the envoy source code instead of downloading it from github
    3. In Makefile.overrides.mk, configure a different container image for use with BUILD_WITH_CONTAINER 
diff --git a/Makefile b/Makefile
index 027b86f..da09cce 100644
--- a/Makefile
+++ b/Makefile
@@ -67,7 +67,7 @@ $(info Building with the build container: $(IMG).)
 # the path of the file.
 TIMEZONE=`readlink $(READLINK_FLAGS) /etc/localtime | sed -e 's/^.*zoneinfo\///'`

-RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker --rm \
+RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker \
        -e IN_BUILD_CONTAINER="$(BUILD_WITH_CONTAINER)" \
        -e TZ="$(TIMEZONE)" \
        -e TARGET_ARCH="$(TARGET_ARCH)" \
@@ -76,6 +76,7 @@ RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker --rm \
        -e HUB="$(HUB)" \
        -e TAG="$(TAG)" \
        -v /etc/passwd:/etc/passwd:ro \
+       -v /home/pivotal/workspace/envoy:/sw-rr-envoy \
        $(DOCKER_SOCKET_MOUNT) \
        $(CONTAINER_OPTIONS) \
        --mount type=bind,source="$(PWD)",destination="/work" \
diff --git a/Makefile.overrides.mk b/Makefile.overrides.mk
index a29486e..c9beff8 100644
--- a/Makefile.overrides.mk
+++ b/Makefile.overrides.mk
@@ -14,3 +14,4 @@

 # this repo is not on the container plan by default
 BUILD_WITH_CONTAINER ?= 0
+IMG = gcr.io/istio-testing/build-tools-proxy:master-2019-11-08T17-21-18
diff --git a/WORKSPACE b/WORKSPACE
index d8d64db..21cab53 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -38,25 +38,25 @@ bind(
 # 2. Update .bazelrc and .bazelversion files.
 #
 # envoy commit date: 11/11/2019
-ENVOY_SHA = "429ee52b1f45b57420db15f434a36ad9efd81828"
+#ENVOY_SHA = "429ee52b1f45b57420db15f434a36ad9efd81828"

-ENVOY_SHA256 = "144aac08d1c5491e948546f2780ba938d92ba59f079bbef038e8174a5a877588"
+#ENVOY_SHA256 = "144aac08d1c5491e948546f2780ba938d92ba59f079bbef038e8174a5a877588"

-LOCAL_ENVOY_PROJECT = "/PATH/TO/ENVOY"
+LOCAL_ENVOY_PROJECT = "/sw-rr-envoy"

-http_archive(
-    name = "envoy",
-    sha256 = ENVOY_SHA256,
-    strip_prefix = "envoy-" + ENVOY_SHA,
-    url = "https://github.com/istio/envoy/archive/" + ENVOY_SHA + ".tar.gz",
-)
+#http_archive(
+#    name = "envoy",
+#    sha256 = ENVOY_SHA256,
+#    strip_prefix = "envoy-" + ENVOY_SHA,
+#    url = "https://github.com/istio/envoy/archive/" + ENVOY_SHA + ".tar.gz",
+#)

 # TODO(silentdai) Use bazel args to select envoy between local or http
 # Uncomment below and comment above http_archive to depends on local envoy.
-#local_repository(
-#     name = "envoy",
-#     path = LOCAL_ENVOY_PROJECT,
-#)
+local_repository(
+     name = "envoy",
+     path = LOCAL_ENVOY_PROJECT,
+)

 load("@envoy//bazel:api_binding.bzl", "envoy_api_binding")
  1. cd ~/workspace/proxy
  2. bazel clean --expunge --async && BUILD_WITH_CONTAINER=1 make build to build
  3. docker container ls to find the container that was just used to build
  4. docker cp $CONTAINER_ID:/work/bazel-out/k8-fastbuild/bin/src/envoy/envoy . to copy the built binary out of the container to the current directory

Use istio/istio to build the container image

  1. cd workspace/istio
  2. git checkout 1.4.0
  3. export GOPATH=$HOME/go The following steps will use this directory as a cache. It does not need to exist before running the following steps. Feel free to run rm -rf $HOME/go if to start clean.
  4. This will copy the envoy binary from the istio/proxy repo into the appropriate place for this repo to use: 
    rm -f /home/pivotal/go/out/linux_amd64/release/istio_is_init
USE_LOCAL_PROXY=1 ISTIO_ENVOY_LOCAL_PATH=/home/pivotal/workspace/proxy/envoy make init
  5. Build the docker container which will include the custom built envoy binary: USE_LOCAL_PROXY=1 ISTIO_ENVOY_LOCAL=/home/pivotal/workspace/proxy/envoy make docker.proxyv2
  6. Confirm that the envoy executable can be started and has the right git sha, run "envoy --version" inside the container: docker run --entrypoint envoy <the-built-image-id> --version
  7. tag and push the container image that was just built: 
    docker tag <the-built-image-id> gcr.io/gcp-project-name/patched-sidecar:istio-envoy-1.4.0-plus-patch
gcloud auth login
gcloud auth configure-docker
docker push gcr.io/gcp-project-name/patched-sidecar:istio-envoy-1.4.0-plus-patch
  8. Once the container image is pushed, the build server is not needed anymore. Log out of glcoud using gcloud auth revoke and then stop the VM to save money.

Download the Official Istio Release

Target Istio 1.4.0 as the version to build the custom proxy sidecar docker image against.

  1. Download the official Istio release: 
    cd workspace
wget https://github.com/istio/istio/releases/download/1.4.0/istio-1.4.0-osx.tar.gz
tar xvf istio-1.4.0-osx.tar.gz
  2. Edit /install/kubernetes/istio-demo.yaml to specify that the injected sidecars should use the custom built container:
diff --git a/install/kubernetes/istio-demo.yaml b/install/kubernetes/istio-demo.yaml
index a4c97a6..38fcc95 100644
--- a/install/kubernetes/istio-demo.yaml
+++ b/install/kubernetes/istio-demo.yaml
@@ -20561,11 +20561,7 @@ data:
       {{- end }}
       containers:
       - name: istio-proxy
-      {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }}
-        image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}"
-      {{- else }}
-        image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.hub }}/{{ .Values.global.proxy.image }}:{{ .Values.global.tag }}"
-      {{- end }}
+        image: "gcr.io/gcp-project-name/patched-sidecar:istio-envoy-1.4.0-plus-patch"
         ports:
         - containerPort: 15090
           protocol: TCP
  1. Add the istioctl to the path 
    cd istio-1.4.0
echo 'export PATH=$PWD/bin:$PATH' > .envrc
direnv allow

Create a cluster and install Istio

  1. Create a GKE cluster:
    1. Go to the GCP Console and select the project to create a k8s cluster in.
    2. Install the gcloud CLI and the kubectl CLI
    3. Click “Create Cluster”, choose “Standard Cluster”, and fill in a name for the cluster, e.g. “ryan-cluster”
    4. Note the k8s version option. See the Istio install docs for the supported versions of k8s. “Istio 1.4 has been tested with Kubernetes releases 1.13, 1.14, 1.15.”.
    5. Otherwise, accept the defaults.
    6. Click “Create”.
  2. When it is finished creating, install Istio into the cluster
    1. Prepare the k8s cluster
      1. For testing purposes, skip enabling the Istio CNI feature
      2. Login: gcloud auth login [email protected]
      3. Retrieve credentials for kubectl: gcloud container clusters get-credentials <your_cluster_name> --zone <your_cluster_zone> --project gcp-project-name
      4. If kubectl commands do not work, grant cluster administrator permissions to the current user: kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)
    2. Follow the setup instructions for Istio
      1. Install Istio 
        for i in install/kubernetes/helm/istio-init/files/crd*yaml; do
  kubectl apply -f ${i};
done
kubectl apply -f install/kubernetes/istio-demo.yaml
      2. Verify the installation by ensuring the pods are running. Then verify that the Istio Kubernetes services are deployed and verify they all have an appropriate CLUSTER-IP except the jaeger-agent service. Also wait for the external IP of the istio-ingressgateway to appear in the watch command below. 
        kubectl get pods -n istio-system
watch kubectl get svc -n istio-system
      3. Enable automatic sidecar injection in the default namespace: kubectl label namespace default istio-injection=enabled

Install the authcode demo app and protect it using the authservice

  1. cd ~/workspace/istio-authn-authz-demo
  2. Check the image name for the authservice container in the authcode-sample-app deployment in ~/workspace/istio-authn-authz-demo/sample-istio-policies/authcode/authcode-sample-app.yaml. Make sure it is set to the version of the authservice to be tested.
  3. Get the ingress hostname: source ./scripts/sample-apps/print-gateway-info-and-sample-app-url.sh
  4. Update the callback.hostname and landing_page IP addresses in this file: ~/workspace/istio-authn-authz-demo/sample-istio-policies/authcode/authcode-sample-app-authservice-configmap.yaml Use the value of $INGRESS_HOSTNAME as the new IP address.
  5. Generate and update certs for the authcode-app-gateway: ./scripts/istio/generate-self-signed-certs-for-gateway.sh
  6. Deploy the authcode and resource server app to the k8s cluster, and set up the gateway for them: ./scripts/sample-apps/deploy-sample-apps.sh
  7. Wait for the deployment to finish: watch kubectl get pods
Clone this wiki locally