Various improvements (#146)

* Add comments regarding app_workloads and additional_workloads * Use double braces for template substitution and update variables * Automatically bind app to secrets policy in setup-app command * Add --run-release-phase option to deploy-image command * Refresh local API token when it is about to expire * Exit with non-zero code if failed to apply any templates * Update docs per suggestions * Show deprecated warning for old template variables * Exit with number codes instead of booleans * Show error message when identity or policy doesn't exist * Do not swallow error messages
shakacode · Mar 21, 2024 · d3a3b7d · d3a3b7d
1 parent 68ec968
commit d3a3b7d
Show file tree

Hide file tree

Showing 32 changed files with 346 additions and 84 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -14,6 +14,19 @@ Changes since the last non-beta release.
 
 _Please add entries here for your pull requests that are not yet released._
 
+### Added
+
+- Added new template substitution variables (used by `apply-template` and `setup-app` commands): `{{APP_LOCATION_LINK}}`, `{{APP_IMAGE_LINK}}`, `{{APP_IDENTITY}}`, `{{APP_IDENTITY_LINK}}`, `{{APP_SECRETS}}` and `{{APP_SECRETS_POLICY}}`. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Added `--run-release-phase` option to `deploy-image` command to run release script before deploying (same step as in `promote-app-from-upstream` command). [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+
+### Changed
+
+- Template substitution (used by `apply-template` and `setup-app` commands) now uses double braces (e.g., `APP_ORG` -> `{{APP_ORG}}`). This change is backwards compatible. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Renamed template substitution variable `APP_GVC` to `{{APP_NAME}}` (used by `apply-template` and `setup-app` commands). This change is backwards compatible. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `setup-app` command now automatically binds the app to the secrets policy, as long as both the identity and the policy exist. Added `--skip-secret-access-binding` option to prevent this behavior. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Local API token is now refreshed when it is about to expire. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `apply-template` command now exits with non-zero code if failed to apply any templates. [PR 146](https://github.com/shakacode/heroku-to-control-plane/pull/146) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+
 ## [1.3.0] - 2024-03-19
 
 ### Fixed

diff --git a/Gemfile.lock b/Gemfile.lock
@@ -4,6 +4,7 @@ PATH
     cpl (1.3.0)
       debug (~> 1.7.1)
       dotenv (~> 2.8.1)
+      jwt (~> 2.8.1)
       psych (~> 5.1.0)
       thor (~> 1.2.1)
 
@@ -13,6 +14,7 @@ GEM
     addressable (2.8.4)
       public_suffix (>= 2.0.2, < 6.0)
     ast (2.4.2)
+    base64 (0.2.0)
     childprocess (4.1.0)
     crack (0.4.5)
       rexml
@@ -29,6 +31,8 @@ GEM
       rdoc
       reline (>= 0.4.2)
     json (2.6.3)
+    jwt (2.8.1)
+      base64
     overcommit (0.60.0)
       childprocess (>= 0.6.3, < 5)
       iniparse (~> 1.4)

diff --git a/README.md b/README.md
@@ -190,8 +190,20 @@ aliases:
 
     # Allows running the command `cpl setup-app`
     # instead of `cpl apply-template gvc redis postgres memcached rails sidekiq`.
+    #
+    # Note:
+    # 1. These names correspond to files in the `./controlplane/templates` directory.
+    # 2. Each file can contain many objects, such as in the case of templates that create a resource, like `postgres`.
+    # 3. While the naming often corresponds to a workload or other object name, the naming is arbitrary. 
+    #    Naming does not need to match anything other than the file name without the `.yml` extension.
     setup_app_templates:
       - gvc
+
+      # These templates are only required if using secrets.
+      - identity
+      - secrets
+      - secrets-policy
+
       - redis
       - postgres
       - memcached
@@ -202,11 +214,16 @@ aliases:
     one_off_workload: rails
 
     # Workloads that are for the application itself and are using application Docker images.
+    # These are updated with the new image when running the `deploy-image` command,
+    # and are also used by the `info`, `ps:`, and `run:cleanup` commands in order to get all of the defined workloads.
+    # On the other hand, if you have a workload for Redis, that would NOT use the application Docker image
+    # and not be listed here.
     app_workloads:
       - rails
       - sidekiq
 
     # Additional "service type" workloads, using non-application Docker images.
+    # These are only used by the `info`, `ps:` and `run:cleanup` commands in order to get all of the defined workloads.
     additional_workloads:
       - redis
       - postgres

diff --git a/cpl.gemspec b/cpl.gemspec
@@ -17,6 +17,7 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "debug",    "~> 1.7.1"
   spec.add_dependency "dotenv",   "~> 2.8.1"
+  spec.add_dependency "jwt",      "~> 2.8.1"
   spec.add_dependency "psych",    "~> 5.1.0"
   spec.add_dependency "thor",     "~> 1.2.1"
 

diff --git a/docs/commands.md b/docs/commands.md
@@ -21,10 +21,14 @@ This `-a` option is used in most of the commands and will pick all other app con
 **Preprocessed template variables:**
 
 ```
-APP_GVC      - basically GVC or app name
-APP_LOCATION - default location
-APP_ORG      - organization
-APP_IMAGE    - will use latest app image
+{{APP_ORG}}           - organization name
+{{APP_NAME}}          - GVC/app name
+{{APP_LOCATION}}      - location, per YML file, ENV, or command line arg
+{{APP_LOCATION_LINK}} - full link for location, ready to be used for the value of `staticPlacement.locationLinks` in the templates
+{{APP_IMAGE}}         - latest app image
+{{APP_IMAGE_LINK}}    - full link for latest app image, ready to be used for the value of `containers[].image` in the templates
+{{APP_IDENTITY}}      - default identity
+{{APP_IDENTITY_LINK}} - full link for identity, ready to be used for the value of `identityLink` in the templates
 ```
 
 ```sh
@@ -111,6 +115,8 @@ cpl delete -a $APP_NAME
 ### `deploy-image`
 
 - Deploys the latest image to app workloads
+- Optionally runs a release script before deploying if specified through `release_script` in the `.controlplane/controlplane.yml` file and `--run-release-phase` is provided
+- The deploy will fail if the release script exits with a non-zero code or doesn't exist
 
 ```sh
 cpl deploy-image -a $APP_NAME
@@ -251,8 +257,9 @@ cpl open-console -a $APP_NAME
 - Copies the latest image from upstream, runs a release script (optional), and deploys the image
 - It performs the following steps:
   - Runs `cpl copy-image-from-upstream` to copy the latest image from upstream
-  - Runs a release script if specified through `release_script` in the `.controlplane/controlplane.yml` file
   - Runs `cpl deploy-image` to deploy the image
+  - If `.controlplane/controlplane.yml` includes the `release_script`, `cpl deploy-image` will use the `--run-release-phase` option
+  - The deploy will fail if the release script exits with a non-zero code
 
 ```sh
 cpl promote-app-from-upstream -a $APP_NAME -t $UPSTREAM_TOKEN
@@ -399,6 +406,8 @@ cpl run:detached -a $APP_NAME --use-local-token -- rails db:migrate:status
 - Creates an app and all its workloads
 - Specify the templates for the app and workloads through `setup_app_templates` in the `.controlplane/controlplane.yml` file
 - This should only be used for temporary apps like review apps, never for persistent apps like production (to update workloads for those, use 'cpl apply-template' instead)
+- Automatically binds the app to the secrets policy, as long as both the identity and the policy exist
+- Use `--skip-secret-access-binding` to prevent the automatic bind
 
 ```sh
 cpl setup-app -a $APP_NAME

diff --git a/docs/tips.md b/docs/tips.md
@@ -85,7 +85,17 @@ For storing ENVs in the source code, we can use a level of indirection so that y
 code like `cpln://secret/my-app-review-env-secrets.SECRET_KEY_BASE` and then have the secret value stored at the org
 level, which applies to your GVCs mapped to that org.
 
-Here is how you do this:
+You can do this during the initial app setup, like this:
+
+1. Add the templates for `identity`, `secrets` and `secrets-policy` to `.controlplane/templates`
+2. Ensure that the templates are listed in `setup_app_templates` for the app in `.controlplane/controlplane.yml`
+3. Run `cpl setup-app -a $APP_NAME`
+4. The identity, secrets and secrets policy will be automatically created, along with the proper binding
+5. In the upper left "Manage Org" menu, click on "Secrets"
+6. Find the created secret (it will be in the `$APP_PREFIX-secrets` format) and add the secret env vars there
+7. Use `cpln://secret/...` in the app to access the secret env vars (e.g., `cpln://secret/$APP_PREFIX-secrets.SOME_VAR`)
+
+You can also do it manually after. Here is how you do this:
 
 1. In the upper left "Manage Org" menu, click on "Secrets"
 2. Create a secret with `Secret Type: Dictionary` (e.g., `my-secrets`) and add the secret env vars there

diff --git a/examples/controlplane.yml b/examples/controlplane.yml
@@ -25,8 +25,20 @@ aliases:
 
     # Allows running the command `cpl setup-app`
     # instead of `cpl apply-template gvc redis postgres memcached rails sidekiq`.
-    setup:
+    #
+    # Note:
+    # 1. These names correspond to files in the `./controlplane/templates` directory.
+    # 2. Each file can contain many objects, such as in the case of templates that create a resource, like `postgres`.
+    # 3. While the naming often corresponds to a workload or other object name, the naming is arbitrary. 
+    #    Naming does not need to match anything other than the file name without the `.yml` extension.
+    setup_app_templates:
       - gvc
+
+      # These templates are only required if using secrets.
+      - identity
+      - secrets
+      - secrets-policy
+
       - redis
       - postgres
       - memcached
@@ -37,11 +49,16 @@ aliases:
     one_off_workload: rails
 
     # Workloads that are for the application itself and are using application Docker images.
+    # These are updated with the new image when running the `deploy-image` command,
+    # and are also used by the `info`, `ps:`, and `run:cleanup` commands in order to get all of the defined workloads.
+    # On the other hand, if you have a workload for Redis, that would NOT use the application Docker image
+    # and not be listed here.
     app_workloads:
       - rails
       - sidekiq
 
     # Additional "service type" workloads, using non-application Docker images.
+    # These are only used by the `info`, `ps:` and `run:cleanup` commands in order to get all of the defined workloads.
     additional_workloads:
       - redis
       - postgres

diff --git a/lib/command/apply_template.rb b/lib/command/apply_template.rb
@@ -20,10 +20,14 @@ class ApplyTemplate < Base # rubocop:disable Metrics/ClassLength
       **Preprocessed template variables:**
 
       ```
-      APP_GVC      - basically GVC or app name
-      APP_LOCATION - default location
-      APP_ORG      - organization
-      APP_IMAGE    - will use latest app image
+      {{APP_ORG}}           - organization name
+      {{APP_NAME}}          - GVC/app name
+      {{APP_LOCATION}}      - location, per YML file, ENV, or command line arg
+      {{APP_LOCATION_LINK}} - full link for location, ready to be used for the value of `staticPlacement.locationLinks` in the templates
+      {{APP_IMAGE}}         - latest app image
+      {{APP_IMAGE_LINK}}    - full link for latest app image, ready to be used for the value of `containers[].image` in the templates
+      {{APP_IDENTITY}}      - default identity
+      {{APP_IDENTITY_LINK}} - full link for identity, ready to be used for the value of `identityLink` in the templates
       ```
     DESC
     EXAMPLES = <<~EX
@@ -36,7 +40,7 @@ class ApplyTemplate < Base # rubocop:disable Metrics/ClassLength
       ```
     EX
 
-    def call # rubocop:disable Metrics/MethodLength, Metrics/PerceivedComplexity
+    def call # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
       ensure_templates!
 
       @created_items = []
@@ -55,6 +59,8 @@ def call # rubocop:disable Metrics/MethodLength, Metrics/PerceivedComplexity
 
       progress.puts if @asked_for_confirmation
 
+      @deprecated_variables = []
+
       pending_templates.each do |template, filename|
         step("Applying template '#{template}'", abort_on_error: false) do
           items = apply_template(filename)
@@ -70,9 +76,13 @@ def call # rubocop:disable Metrics/MethodLength, Metrics/PerceivedComplexity
         end
       end
 
+      warn_deprecated_variables
+
       print_created_items
       print_failed_templates
       print_skipped_templates
+
+      exit(1) if @failed_templates.any?
     end
 
     private
@@ -124,17 +134,55 @@ def confirm_workload(template)
       false
     end
 
-    def apply_template(filename)
+    def apply_template(filename) # rubocop:disable Metrics/MethodLength
       data = File.read(filename)
-                 .gsub("APP_GVC", config.app)
-                 .gsub("APP_LOCATION", config.location)
-                 .gsub("APP_ORG", config.org)
-                 .gsub("APP_IMAGE", latest_image)
+                 .gsub("{{APP_ORG}}", config.org)
+                 .gsub("{{APP_NAME}}", config.app)
+                 .gsub("{{APP_LOCATION}}", config.location)
+                 .gsub("{{APP_LOCATION_LINK}}", app_location_link)
+                 .gsub("{{APP_IMAGE}}", latest_image)
+                 .gsub("{{APP_IMAGE_LINK}}", app_image_link)
+                 .gsub("{{APP_IDENTITY}}", app_identity)
+                 .gsub("{{APP_IDENTITY_LINK}}", app_identity_link)
+                 .gsub("{{APP_SECRETS}}", app_secrets)
+                 .gsub("{{APP_SECRETS_POLICY}}", app_secrets_policy)
+
+      find_deprecated_variables(data)
+
+      # Kept for backwards compatibility
+      data = data
+             .gsub("APP_ORG", config.org)
+             .gsub("APP_GVC", config.app)
+             .gsub("APP_LOCATION", config.location)
+             .gsub("APP_IMAGE", latest_image)
 
       # Don't read in YAML.safe_load as that doesn't handle multiple documents
       cp.apply_template(data)
     end
 
+    def new_variables
+      {
+        "APP_ORG" => "{{APP_ORG}}",
+        "APP_GVC" => "{{APP_NAME}}",
+        "APP_LOCATION" => "{{APP_LOCATION}}",
+        "APP_IMAGE" => "{{APP_IMAGE}}"
+      }
+    end
+
+    def find_deprecated_variables(data)
+      @deprecated_variables.push(*new_variables.keys.select { |old_key| data.include?(old_key) })
+      @deprecated_variables = @deprecated_variables.uniq.sort
+    end
+
+    def warn_deprecated_variables
+      return unless @deprecated_variables.any?
+
+      message = "Please replace these variables in the templates, " \
+                "as support for them will be removed in a future major version bump:"
+      deprecated = @deprecated_variables.map { |old_key| "  - #{old_key} -> #{new_variables[old_key]}" }.join("\n")
+      progress.puts("\n#{Shell.color("DEPRECATED: #{message}", :yellow)}\n#{deprecated}")
+    end
+
     def report_success(item)
       @created_items.push(item)
     end

diff --git a/lib/command/base.rb b/lib/command/base.rb
@@ -249,6 +249,28 @@ def self.clean_on_failure_option(required: false)
       }
     end
 
+    def self.skip_secret_access_binding_option(required: false)
+      {
+        name: :skip_secret_access_binding,
+        params: {
+          desc: "Skips secret access binding",
+          type: :boolean,
+          required: required
+        }
+      }
+    end
+
+    def self.run_release_phase_option(required: false)
+      {
+        name: :run_release_phase,
+        params: {
+          desc: "Runs release phase",
+          type: :boolean,
+          required: required
+        }
+      }
+    end
+
     def self.all_options
       methods.grep(/_option$/).map { |method| send(method.to_s) }
     end
@@ -372,8 +394,32 @@ def cp
       @cp ||= Controlplane.new(config)
     end
 
-    def perform(cmd)
-      system(cmd) || exit(false)
+    def perform!(cmd)
+      system(cmd) || exit(1)
+    end
+
+    def app_location_link
+      "/org/#{config.org}/location/#{config.location}"
+    end
+
+    def app_image_link
+      "/org/#{config.org}/image/#{latest_image}"
+    end
+
+    def app_identity
+      "#{config.app}-identity"
+    end
+
+    def app_identity_link
+      "/org/#{config.org}/gvc/#{config.app}/identity/#{app_identity}"
+    end
+
+    def app_secrets
+      "#{config.app_prefix}-secrets"
+    end
+
+    def app_secrets_policy
+      "#{app_secrets}-policy"
     end
 
     private

diff --git a/lib/command/cleanup_stale_apps.rb b/lib/command/cleanup_stale_apps.rb
@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require "date"
-
 module Command
   class CleanupStaleApps < Base
     NAME = "cleanup-stale-apps"