diff --git a/.gitignore b/.gitignore index 204ed4c..10184f3 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,6 @@ test-results/ /clj/ /.cpcache/ /resources/role.edn +.terraform/ +*.tfstate +*.tfstate.backup diff --git a/CHANGELOG.md b/CHANGELOG.md index 0667818..029bb50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,14 @@ ### Added +## [0.6.0] + + - Added the ability to clone with source and target instances in different VPCs, by automatically falling back to `RestoreDBInstanceFromDBSnapshot`, instead of `CreateDBInstanceReadReplica` + - The restore will be done based on the latest snapshot available, which is usually at most 24h stale. + - Added `--restore-snapshot` to force even same-VPC clones to be done with `RestoreDBInstanceFromDBSnapshot` (it's currently faster) + - Added a terrafrom environment for testing + ## [0.5.0] - Added `--iam-policy` option for generating a IAM policy for a user or role to clone a replica with a minimal set of permissions. - Updated dependencies - diff --git a/README.md b/README.md index 6d1246d..c09c5f5 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ data parity with production in a throw-away environment. ## Process -Suppose for testing or sales purposes it is necessary to maintain an independent application stack with it's own database, which needs to periodically refresh data from the production database. To simplify the illustration, this will focus on the changes to the [AWS RDS](https://aws.amazon.com/rds/) database replication graphs, and omit the application and other services it may depend on. +Suppose for testing or sales purposes it is necessary to maintain an independent application stack with it's own database, which needs to periodically refresh data from the production database. To simplify the illustration, this will focus on the changes to the [AWS RDS](https://aws.amazon.com/rds/) database replication graphs, and omit the application and other services it may depend on. Consider two independent application stacks, production and demo, with primary databases `mitosis-prod` and `mitosis-demo` respectively. Each stack has a replication graph where a primary database is followed by one replica, ie `mitosis-prod` replicates to `mitosis-prod-replica` and `mitosis-demo` replicates to `mitosis-demo-replica`. @@ -34,12 +34,14 @@ Once that is complete, it's safe to rename the `temp-` prefixed clones back to ` ![img](doc/img/rename-2.png) - However, as this is a DNS swap, the application is likely still connected to the original `old-mitosis-demo`. By specifying a restart script, stack-mitosis can force the demo application to restart, and connect to the newly created `mitosis-demo` with fresh data from production. Once it has restarted the application successfully, it deletes the `old-` prefixed database instances from the original demo replication graph. +However, as this is a DNS swap, the application is likely still connected to the original `old-mitosis-demo`. By specifying a restart script, stack-mitosis can force the demo application to restart, and connect to the newly created `mitosis-demo` with fresh data from production. Once it has restarted the application successfully, it deletes the `old-` prefixed database instances from the original demo replication graph. ![img](doc/img/final.png) Note that this replication graph is a simple case, but it supports replacing arbitrarily complex replication graphs on RDS and has been verified with mysql and postgres database engines. The postgres engine on RDS only allows multiple replicas of a single primary, but the Mysql engine on RDS allows cascading replicas of replicas. See the AWS documentation for [working with RDS read replicas](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html) for more information on these limitations. +In the case where source and target instances live in different VPCs, or in case `--restore-snapshot` is used, the first instance (`temp-mitosis-demo` here) is created by restoring the latest available snapshot, instead of using replication. This also skips the promote replica step. All other steps remain the same. + # Install After installing a JDK, follow the [clojure install @@ -79,10 +81,11 @@ Hopefully in the future this can be parsed directly from the `AWS_CONFIG` file. [--credentials resources/role.edn] [--plan] [--iam-policy] + [--restore-snapshot] ## Flight Plan -The `--plan` flag will give a flight plan showing the expected list of API calls it's planning on executing against the Amazon API. +The `--plan` flag will give a flight plan showing the expected list of API calls it's planning on executing against the Amazon API. ``` $ clj -m stack-mitosis.cli --source mitosis-prod --target mitosis-demo --plan @@ -179,7 +182,7 @@ This ensures that a continuous integration or cronjob server like Jenkins can cl Cloudformation and Terraform are wonderful tools focused on declarative architecture transformation from one steady state to another. Stack-mitosis is focused on safely cloning the contents of a database in one environment to another without changing from one steady state to another. As example, for an environment with production and demo environments, they both exist in the correct configuration before running stack-mitosis, and then after running stack-mitosis the configuration remains the same but the demo environment has a fresh copy of the data from production. -I suspect this could also be accomplished using one of these declarative infrastructure tools by transitioning through multiple intervening states, but have not found any examples of anyone doing that. +I suspect this could also be accomplished using one of these declarative infrastructure tools by transitioning through multiple intervening states, but have not found any examples of anyone doing that. # License diff --git a/resources/log4j.properties b/resources/log4j.properties index a2f6e8f..e8cdd4c 100644 --- a/resources/log4j.properties +++ b/resources/log4j.properties @@ -2,3 +2,4 @@ log4j.rootLogger=DEBUG, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} | %-5p | %t | %m%n +log4j.appender.console.Target=System.err diff --git a/src/stack_mitosis/cli.clj b/src/stack_mitosis/cli.clj index a966c5b..2ebd6e6 100644 --- a/src/stack_mitosis/cli.clj +++ b/src/stack_mitosis/cli.clj @@ -4,6 +4,7 @@ [clojure.tools.cli :as cli] [clojure.tools.logging :as log] [stack-mitosis.interpreter :as interpreter] + [stack-mitosis.lookup :as lookup] [stack-mitosis.planner :as plan] [stack-mitosis.policy :as policy] [stack-mitosis.request :as r] @@ -20,6 +21,7 @@ ["-c" "--credentials FILENAME" "Credentials file in edn for iam assume-role"] ["-p" "--plan" "Display expected flightplan for operation."] ["-i" "--iam-policy" "Generate IAM policy for planned actions."] + [nil "--restore-snapshot" "Always clone using snapshot restore."] ["-h" "--help"]]) (defn parse-args [args] @@ -52,26 +54,36 @@ (let [rds (interpreter/client) instances (interpreter/databases rds)] (when (interpreter/verify-databases-exist instances [source target]) - (let [tags (interpreter/list-tags rds instances target) - plan (plan/replace-tree instances source target - :restart restart :tags tags)] - (cond (:plan options) - (do (println (flight-plan (interpreter/check-plan instances plan))) - true) - (:iam-policy options) - (do (json/pprint (policy/from-plan instances plan)) - true) - :else - (let [last-action (interpreter/evaluate-plan rds plan)] - (not (contains? last-action :ErrorResponse)))))))) + (let [same-vpc (lookup/same-vpc? + (lookup/by-id instances source) + (lookup/by-id instances target)) + use-restore-snapshot (or (:restore-snapshot options) (not same-vpc)) + source-snapshot (if use-restore-snapshot + (interpreter/latest-snapshot rds source) + nil)] + + (when (or (not use-restore-snapshot) + (interpreter/verify-snapshot-exists instances [source target] + source-snapshot)) + (let [tags (interpreter/list-tags rds instances target) + plan (plan/replace-tree instances source source-snapshot target + :restart restart :tags tags)] + (cond (:plan options) + (do (println (flight-plan (interpreter/check-plan instances plan))) + true) + (:iam-policy options) + (do (json/pprint (policy/from-plan instances plan)) + true) + :else + (let [last-action (interpreter/evaluate-plan rds plan)] + (not (contains? last-action :ErrorResponse)))))))))) (defn -main [& args] (let [{:keys [ok exit-msg] :as options} (parse-args args)] (when exit-msg (println exit-msg) (System/exit (if ok 0 1))) - (System/exit (if (process options) 0 1)) - )) + (System/exit (if (process options) 0 1)))) (comment (process (parse-args ["--source" "mitosis-prod" "--target" "mitosis-demo" diff --git a/src/stack_mitosis/interpreter.clj b/src/stack_mitosis/interpreter.clj index 6ac7b13..bc506db 100644 --- a/src/stack_mitosis/interpreter.clj +++ b/src/stack_mitosis/interpreter.clj @@ -51,6 +51,22 @@ false) true))) +(defn verify-snapshot-exists + [instances identifiers snapshot] + (let [instances (map (partial lookup/by-id instances) identifiers) + vpcs (map #(get-in % [:DBSubnetGroup :VpcId]) instances) + cross-vpc-mitosis (-> vpcs distinct count (> 1))] + (if (and cross-vpc-mitosis (not snapshot)) + (do + (log/error + (str/join "\n" ["Source database has no snapshots." "" + (str "Source and target databases are in different VPCs." + " When that happens, stack-mitosis uses " + "RestoreDBInstanceFromDBSnapshot to be able to" + " clone the source database to the target VPC.")])) + false) + true))) + (defn list-tags "Mapping of db-id to tags list for each instance in a tree." [rds instances target] @@ -62,6 +78,15 @@ [db-id (:TagList (invoke-logged! rds (op/tags arn)))]))) (into {}))) +(defn latest-snapshot + "Returns the latest snapshot for an instance" + [rds target] + (->> (invoke-logged! rds (op/list-snapshots target)) + (:DBSnapshots) + (sort-by :SnapshotCreateTime) + (last) + (:DBSnapshotIdentifier))) + (defn describe [rds id] (invoke-logged! rds (op/describe id))) @@ -75,7 +100,7 @@ (op/completed? (describe rds new-id)))] [id #(op/completed? (describe rds id))]) started (. System (nanoTime)) - ret (wait/poll-until completed-fn {:delay 60000 :max-attempts 120}) + ret (wait/poll-until completed-fn {:delay 60000 :max-attempts 180}) msecs (/ (double (- (. System (nanoTime)) started)) 1000000.0) status (-> (describe rds result-id) :DBInstances first :DBInstanceStatus) msg (format "Completed after %.2fs with status %s" (/ msecs 1000) status)] @@ -118,7 +143,7 @@ (sudo/sudo-provider (sudo/load-role "resources/role.edn")) (def rds (client)) (-> (predict/state [] (example/create example/template)) - (plan/replace-tree "mitosis-prod" "mitosis-demo")) + (plan/replace-tree "mitosis-prod" "mitosis-demo" nil)) (interpret rds (op/shell-command "echo restart")) (evaluate-plan rds [(op/shell-command "true") (op/shell-command "false") @@ -126,7 +151,7 @@ ;; check plan (let [state (databases rds)] - (check-plan state (plan/replace-tree state "mitosis-prod" "mitosis-demo"))) + (check-plan state (plan/replace-tree state "mitosis-prod" "mitosis-demo" nil))) ;; create a copy of mitosis-prod tree (let [state (databases rds)] diff --git a/src/stack_mitosis/lookup.clj b/src/stack_mitosis/lookup.clj index e2beaea..32a932a 100644 --- a/src/stack_mitosis/lookup.clj +++ b/src/stack_mitosis/lookup.clj @@ -33,6 +33,64 @@ (and (or (seq? v) (vector? v)) (empty? v)))) +(defn same-vpc? [db-a db-b] + (= (get-in db-a [:DBSubnetGroup :VpcId]) (get-in db-b [:DBSubnetGroup :VpcId]))) + +(defn restore-snapshot-attributes + "Creates a list of additional attributes to clone from original instance into + the newly created replica instance. + + https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_RestoreDBInstanceFromDBSnapshot.html + has more information on these attributes." + [original tags] + (let [attributes-to-clone + [:CopyTagsToSnapshot + :PubliclyAccessible + :AutoMinorVersionUpgrade + :DBInstanceClass + ;; :DeletionProtection ; must be false for repeated invocation + ;; :KmsKeyId ;; not supported by restore or modify + ;; but is guaranteed to remain the same + ;; it can only change when we copy a snapshot + ;; :SourceRegion ; not applicable? + :ProcessorFeatures + ;; :UseDefaultProcessorFeatures ; just copy features directly? + :Iops + :StorageType + :MultiAZ] + + translated-attributes + {:Tags tags + :EnableIAMDatabaseAuthentication (:IAMDatabaseAuthenticationEnabled original) + :EnableCloudwatchLogsExports (:EnabledCloudwatchLogsExports original) + :Port (:Port (:Endpoint original)) + :DBSubnetGroupName (:DBSubnetGroupName (:DBSubnetGroup original)) + + ;; all active security groups ids + :VpcSecurityGroupIds + (->> original + :VpcSecurityGroups + (filter (fn [group] (= (:Status group) "active"))) + (map :VpcSecurityGroupId)) + + ;; first synchronized option group name + :OptionGroupName + (->> original + :OptionGroupMemberships + (some (fn [group] + (and (= (:Status group) "in-sync") + (:OptionGroupName group))))) + ;; TODO map for names on original + ;; :DomainMemberships -> :Domain, :DomainIAMRoleName + }] + (-> original + ;; copy as-is with no translation + (select-keys attributes-to-clone) + ;; Attributes requiring custom rules to extract from original and + ;; translate to key for clone-replica request + (merge (into {} (remove (fn [[_ v]] (nil-or-empty? v)) + translated-attributes)))))) + (defn clone-replica-attributes "Creates a list of additional attributes to clone from original instance into the newly created replica instance. @@ -122,3 +180,44 @@ ;; translate to key for modify-db request (merge (into {} (remove (fn [[_ v]] (nil-or-empty? v)) translated-attributes)))))) + +(defn post-restore-snapshot-attributes + "List of additional attributes to apply after creation. + + Some parameters are not available or applicable at time of creation, so they + need to be applied after. + + https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_ModifyDBInstance.html + has more information on these attributes." + [original] + (let [attributes-to-clone ;; attributes not supported by restore-snapshot + [:MonitoringRoleArn + :MonitoringInterval + :PerformanceInsightsKMSKeyId + :PerformanceInsightsRetentionPeriod + :PreferredMaintenanceWindow + :PreferredBackupWindow + ;; TODO ? + ;; :AllocatedStorage ;; Tricky, only supports increase + ;; :MaxAllocatedStorage + ] + + translated-attributes + {:EnablePerformanceInsights (:PerformanceInsightsEnabled original) ;; restore_not_supported + + ;; Triggers "The specified DB instance is already in the target DB subnet group" + ;; probably need to detect if changing? disabling for now + ;; :DBSubnetGroupName (:DBSubnetGroupName (:DBSubnetGroup original)) + ;; first synchronized db parameter group name + :DBParameterGroupName + (->> original + :DBParameterGroups + (some (fn [group] + (and (= (:ParameterApplyStatus group) "in-sync") + (:DBParameterGroupName group)))))}] + (-> original + (select-keys attributes-to-clone) + ;; Attributes requiring custom rules to extract from original and + ;; translate to key for modify-db request + (merge (into {} (remove (fn [[_ v]] (nil-or-empty? v)) + translated-attributes)))))) diff --git a/src/stack_mitosis/operations.clj b/src/stack_mitosis/operations.clj index e533386..c2b1b29 100644 --- a/src/stack_mitosis/operations.clj +++ b/src/stack_mitosis/operations.clj @@ -38,6 +38,20 @@ {:SourceDBInstanceIdentifier source :DBInstanceIdentifier replica})})) +(defn restore-snapshot + ([snapshot-id source target] (restore-snapshot snapshot-id source target {})) + ([snapshot-id source target attributes] + {:op :RestoreDBInstanceFromDBSnapshot + :request (merge attributes + {:DBSnapshotIdentifier snapshot-id + :DBInstanceIdentifier target}) + :meta {:SourceDBInstance source}})) + +(defn list-snapshots + ([target] + {:op :DescribeDBSnapshots + :request {:DBInstanceIdentifier target}})) + (defn promote [id] {:op :PromoteReadReplica @@ -68,7 +82,7 @@ (defn blocking-operation? [action] (contains? #{:CreateDBInstance :CreateDBInstanceReadReplica - :PromoteReadReplica :ModifyDBInstance} (:op action))) + :PromoteReadReplica :ModifyDBInstance :RestoreDBInstanceFromDBSnapshot} (:op action))) (defn transition-to "Maps current rds status to in-progress, failed or done diff --git a/src/stack_mitosis/planner.clj b/src/stack_mitosis/planner.clj index 36ecadc..305a4d3 100644 --- a/src/stack_mitosis/planner.clj +++ b/src/stack_mitosis/planner.clj @@ -28,10 +28,8 @@ (zipmap ids) topological-sort)) -;; postgres does not allow replica of replica, so need to promote before -;; replicating children (defn copy-tree - [instances source target alias-fn & {:keys [tags]}] + [instances source source-snapshot target alias-fn & {:keys [tags]}] (let [alias-tags (->> tags (map (fn [[db-id instance-tags]] [(alias-fn db-id) instance-tags])) @@ -40,11 +38,19 @@ (partial lookup/by-id instances)) (list-tree instances target)) root-id (:DBInstanceIdentifier root) - root-attrs (lookup/clone-replica-attributes root (get alias-tags root-id))] - (into [(op/create-replica source root-id root-attrs) - (op/promote root-id) - ;; postgres only allows backups after promotion - (op/enable-backups root-id (lookup/post-create-replica-attributes root))] + + source-instance (lookup/by-id instances source) + root-attrs (lookup/clone-replica-attributes root (get alias-tags root-id)) + root-restore-attrs (lookup/restore-snapshot-attributes root (get alias-tags root-id))] + (into (if (nil? source-snapshot) + [(op/create-replica source root-id root-attrs) + ;; postgres does not allow replica of replica, so need to promote before + ;; replicating children + (op/promote root-id) + ;; postgres only allows backups after promotion + (op/enable-backups root-id (lookup/post-create-replica-attributes root))] + [(op/restore-snapshot source-snapshot source-instance root-id root-restore-attrs) + (op/enable-backups root-id (lookup/post-restore-snapshot-attributes root))]) (mapcat (fn [instance] [(op/create-replica (:ReadReplicaSourceDBInstanceIdentifier instance) @@ -72,12 +78,12 @@ (map op/delete))) (defn replace-tree - [instances source target & {:keys [restart tags] :or {tags {}}}] + [instances source source-snapshot target & {:keys [restart tags] :or {tags {}}}] ;; actions in copy, rename & delete change the local instances db, so use ;; predict to update that db for calculating next set of operations by ;; applying computation thus far to the initial instances ;; TODO something something sequence monad - (let [copy (copy-tree instances source target + (let [copy (copy-tree instances source source-snapshot target (partial aliased "temp") :tags tags) @@ -113,6 +119,9 @@ (and (= op :CreateDBInstanceReadReplica) (lookup/by-id instances (r/db-id action))) [:skip (duplicate-instance (r/db-id action))] + (and (= op :RestoreDBInstanceFromDBSnapshot) + (lookup/by-id instances (r/db-id action))) + [:skip (duplicate-instance (r/db-id action))] (and (= op :PromoteReadReplica) (not (:ReadReplicaSourceDBInstanceIdentifier (lookup/by-id instances (r/db-id action))))) [:skip (promoted-instance (r/db-id action))] diff --git a/src/stack_mitosis/policy.clj b/src/stack_mitosis/policy.clj index 56d1fdf..87037a6 100644 --- a/src/stack_mitosis/policy.clj +++ b/src/stack_mitosis/policy.clj @@ -36,6 +36,24 @@ [source-arn target-arn])} {:op :AddTagsToResource :arn target-arn}])) +(defmethod permissions :RestoreDBInstanceFromDBSnapshot + [instances action] + ;; For create replica, use the ARN from the source database + (let [source-arn (->> action :meta :SourceDBInstance :DBInstanceArn) + snapshot-arn (-> source-arn (str/split #":db:") first (str ":snapshot:*")) + db-id (r/db-id action) + target-arn (:DBInstanceArn (lookup/by-id (predict/predict instances action) db-id))] + [{:op (:op action) + ;; TODO: can these permissions be more specific instead of wildcard? + ;; a) re-use the base ARN (ie region:account-id) from source + ;; b) generate named ARNs for each subtype used in source? + :arn (into [(make-arn "*" :type "og") + (make-arn "*" :type "pg") + (make-arn "*" :type "subgrp")] + [snapshot-arn target-arn])} + {:op :AddTagsToResource :arn target-arn} + {:op :DescribeDBSnapshots :arn "*"}])) + (defmethod permissions :ModifyDBInstance [instances action] (let [db-id (r/db-id action) diff --git a/src/stack_mitosis/predict.clj b/src/stack_mitosis/predict.clj index db1a50c..22fbf15 100644 --- a/src/stack_mitosis/predict.clj +++ b/src/stack_mitosis/predict.clj @@ -77,6 +77,27 @@ (update (lookup/position instances parent) detach child) (update (lookup/position instances child) promote)))))) +(defmethod predict :RestoreDBInstanceFromDBSnapshot + [instances op] + {:post [(lookup/exists? % (r/db-id op))]} + (let [source-id (->> op :meta :SourceDBInstance :DBInstanceIdentifier) + source-db (lookup/by-id instances source-id) + target-id (r/db-id op) + ;; We use the arn for some policy calculations, and we don't want it to + ;; end up being the source-arn + target-arn (-> source-db :DBInstanceArn (str/split #":db:") first (str ":db:" target-id))] + (as-> op $ + (:request $) + (dissoc $ :DBSnapshotIdentifier) + ;; This is not reeeally what happens, but replicating what happens is + ;; complicated. We're getting a DB in the new subnet, missing a few + ;; params, that are not going to be === the source params, but carry + ;; default values from RDS. + (merge source-db $) + (assoc $ :DBInstanceArn target-arn) + (dissoc $ :ReadReplicaDBInstanceIdentifiers) + (conj instances $)))) + (defmethod predict :ModifyDBInstance [instances op] {:pre [(lookup/exists? instances (r/db-id op))]} diff --git a/terraform/README.md b/terraform/README.md new file mode 100644 index 0000000..c335b22 --- /dev/null +++ b/terraform/README.md @@ -0,0 +1,29 @@ +# Test environment Terraform code + +This terraform code creates a test environment for all invariants of `stack-mitosis`: + +- MySQL + - Same VPC + - Different VPC +- Postgres + - Same VPC + - Different VPC + +## How to use + + $ terraform apply + +Wait some 15 minutes. + +Terraform output: + + Apply complete! Resources: 17 added, 0 changed, 0 destroyed. + + Outputs: + + test_commands = clj -m stack-mitosis.cli --source mitosis-mysql-src --target mitosis-mysql-target-different-vpc + clj -m stack-mitosis.cli --source mitosis-mysql-src --target mitosis-mysql-target-same-vpc + clj -m stack-mitosis.cli --source mitosis-postgres-src --target mitosis-postgres-target-different-vpc + clj -m stack-mitosis.cli --source mitosis-postgres-src --target mitosis-postgres-target-same-vpc + +Run the test commands in the terraform output to validate stack-mitosis against all pairs of source and target instances. \ No newline at end of file diff --git a/terraform/main.tf b/terraform/main.tf new file mode 100644 index 0000000..7d30f06 --- /dev/null +++ b/terraform/main.tf @@ -0,0 +1,142 @@ +terraform { + required_version = "~> 0.13.0" + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 3.0" + } + } +} + +locals { + source = { + port = 5430 + backup_retention_period = 1 + backup_window = "05:30-06:20" + maintenance_window = "tue:07:02-tue:08:00" + } + target = { + port = 5431 + backup_retention_period = 1 + backup_window = "05:20-06:20" + maintenance_window = "mon:07:30-mon:08:00" + } + + mysql = { engine = "mysql" } + postgres = { engine = "postgres" } + vpc_a = { db_subnet_group_name = null } + vpc_b = { db_subnet_group_name = aws_db_subnet_group.main.name } + + sources = { + mitosis-mysql-src = merge(local.source, local.mysql, local.vpc_a), + mitosis-postgres-src = merge(local.source, local.postgres, local.vpc_a), + } + targets = { + mitosis-mysql-target-same-vpc = merge(local.target, local.mysql, local.vpc_a), + mitosis-mysql-target-different-vpc = merge(local.target, local.mysql, local.vpc_b), + mitosis-postgres-target-same-vpc = merge(local.target, local.postgres, local.vpc_a), + mitosis-postgres-target-different-vpc = merge(local.target, local.postgres, local.vpc_b), + } + dbs = merge(local.sources, local.targets) +} + +# Configure the AWS Provider +provider "aws" { + # Or whatever region you want + region = "us-west-2" +} + +# Current region +data "aws_region" "current" {} + +# Create a VPC +resource "aws_vpc" "main" { + cidr_block = "10.0.0.0/16" + tags = { + Name = "stack-mitosis" + Service = "Mitosis" + Env = "test" + } +} + +# Create two subnets so we can create a subnet group +resource "aws_subnet" "a" { + vpc_id = aws_vpc.main.id + cidr_block = "10.0.0.0/24" + availability_zone = "${data.aws_region.current.name}a" + + tags = { + Name = "Mitosis A" + Service = "Mitosis" + Env = "test" + } +} + +resource "aws_subnet" "b" { + vpc_id = aws_vpc.main.id + cidr_block = "10.0.1.0/24" + availability_zone = "${data.aws_region.current.name}b" + + tags = { + Name = "Mitosis B" + Service = "Mitosis" + Env = "test" + } +} + +# Subnet group for our databases +resource "aws_db_subnet_group" "main" { + name = "mitosis-test" + subnet_ids = [aws_subnet.a.id, aws_subnet.b.id] + + tags = { + Name = "Mitosis" + Service = "Mitosis" + Env = "test" + } +} + +# Random password generator for the required field `password` on the DBs +resource "random_password" "password" { + length = 16 + special = false +} + +# Create our databases +resource "aws_db_instance" "main" { + for_each = local.dbs + allocated_storage = 10 + instance_class = "db.t3.micro" + username = "foo" + skip_final_snapshot = true + + password = random_password.password.result + + identifier = each.key + engine = each.value.engine + db_subnet_group_name = each.value.db_subnet_group_name + backup_retention_period = each.value.backup_retention_period + backup_window = each.value.backup_window + maintenance_window = each.value.maintenance_window + port = each.value.port + + tags = { + Service = "Mitosis" + Env = "test" + } +} + +# And their replicas +resource "aws_db_instance" "replica" { + for_each = aws_db_instance.main + identifier = "${each.value.id}-replica" + replicate_source_db = each.value.id + engine = each.value.engine + instance_class = "db.t3.micro" + skip_final_snapshot = true + + tags = { + Service = "Mitosis" + Env = "test" + } +} diff --git a/terraform/outputs.tf b/terraform/outputs.tf new file mode 100644 index 0000000..8d4db45 --- /dev/null +++ b/terraform/outputs.tf @@ -0,0 +1,11 @@ +locals { + commands = flatten([for src in keys(local.sources): + [for target in keys(local.targets): + "clj -m stack-mitosis.cli --source ${src} --target ${target}" + if aws_db_instance.main[src].engine == aws_db_instance.main[target].engine + ] + ]) +} +output "test_commands" { + value = join("\n",local.commands) +} diff --git a/test/stack_mitosis/lookup_test.clj b/test/stack_mitosis/lookup_test.clj index 3be4a14..6a9466a 100644 --- a/test/stack_mitosis/lookup_test.clj +++ b/test/stack_mitosis/lookup_test.clj @@ -55,3 +55,7 @@ ;; :DBSubnetGroupName "subnet-group" } (lookup/post-create-replica-attributes instance))))) + +(deftest same-vpc? + (is (= true (lookup/same-vpc? {:DBSubnetGroup {:VpcId "a"}} {:DBSubnetGroup {:VpcId "a"}}))) + (is (= false (lookup/same-vpc? {:DBSubnetGroup {:VpcId "a"}} {:DBSubnetGroup {:VpcId "b"}})))) \ No newline at end of file diff --git a/test/stack_mitosis/planner_test.clj b/test/stack_mitosis/planner_test.clj index 3dbcfb8..09522dc 100644 --- a/test/stack_mitosis/planner_test.clj +++ b/test/stack_mitosis/planner_test.clj @@ -1,7 +1,8 @@ (ns stack-mitosis.planner-test (:require [clojure.test :refer :all] [stack-mitosis.planner :as plan] - [stack-mitosis.operations :as op])) + [stack-mitosis.operations :as op] + [stack-mitosis.lookup :as lookup])) (deftest list-tree (let [a {:DBInstanceIdentifier :a :ReadReplicaDBInstanceIdentifiers [:b]} @@ -21,6 +22,13 @@ :ReadReplicaSourceDBInstanceIdentifier "target"} {:DBInstanceIdentifier "b" :ReadReplicaSourceDBInstanceIdentifier "target"} {:DBInstanceIdentifier "c" :ReadReplicaSourceDBInstanceIdentifier "b"}] + instances-different-vpc [{:DBInstanceIdentifier "source" :Iops 1000, :DBSubnetGroup {:VpcId "v1"}} + {:DBInstanceIdentifier "target" :ReadReplicaDBInstanceIdentifiers ["a" "b"] :Iops 500} + {:DBInstanceIdentifier "a" :ReadReplicaDBInstanceIdentifiers ["c"] + :ReadReplicaSourceDBInstanceIdentifier "target"} + {:DBInstanceIdentifier "b" :ReadReplicaSourceDBInstanceIdentifier "target"} + {:DBInstanceIdentifier "c" :ReadReplicaSourceDBInstanceIdentifier "b"}] + snapshot-id "rds:source-snapshot-2021-03-17" tags-target [(op/kv "k" "target")] tags-b [(op/kv "k" "b")]] (is (= [(op/create-replica "source" "temp-target" {:Iops 500 :Tags tags-target}) @@ -32,7 +40,21 @@ (op/modify "temp-b" {}) (op/create-replica "temp-b" "temp-c") (op/modify "temp-c" {})] - (plan/copy-tree instances "source" "target" + (plan/copy-tree instances "source" nil "target" + (partial plan/aliased "temp") + :tags {"target" tags-target + "b" tags-b}))) + (is (= [(op/restore-snapshot snapshot-id + (lookup/by-id instances-different-vpc "source") + "temp-target" {:Iops 500 :Tags tags-target}) + (op/enable-backups "temp-target") + (op/create-replica "temp-target" "temp-a") + (op/modify "temp-a" {:BackupRetentionPeriod 1}) + (op/create-replica "temp-target" "temp-b" {:Tags tags-b}) + (op/modify "temp-b" {}) + (op/create-replica "temp-b" "temp-c") + (op/modify "temp-c" {})] + (plan/copy-tree instances-different-vpc "source" snapshot-id "target" (partial plan/aliased "temp") :tags {"target" tags-target "b" tags-b}))))) @@ -66,6 +88,7 @@ [{:DBInstanceIdentifier "production"} {:DBInstanceIdentifier "staging" :ReadReplicaDBInstanceIdentifiers ["staging-replica"]} {:DBInstanceIdentifier "staging-replica" :ReadReplicaSourceDBInstanceIdentifier "staging"}] + snapshot-id "rds:production-2015-10-21" tags [(op/kv "Env" "Staging")]] (is (= [(op/create-replica "production" "temp-staging") (op/promote "temp-staging") @@ -78,7 +101,7 @@ (op/rename "temp-staging" "staging") (op/delete "old-staging-replica") (op/delete "old-staging")] - (plan/replace-tree instances "production" "staging"))) + (plan/replace-tree instances "production" nil "staging"))) (is (= [(op/create-replica "production" "temp-staging") (op/promote "temp-staging") @@ -94,13 +117,43 @@ (op/delete "old-staging-replica") (op/delete "old-staging")] (plan/replace-tree - [{:DBInstanceIdentifier "production" + [{:DBInstanceIdentifier "production" :ReadReplicaDBInstanceIdentifiers ["production-replica"] + :PreferredMaintenanceWindow "tue:01:00-tue:02:00"} + {:DBInstanceIdentifier "production-replica" :ReadReplicaSourceDBInstanceIdentifier "production" :PreferredMaintenanceWindow "tue:01:00-tue:02:00"} {:DBInstanceIdentifier "staging" :ReadReplicaDBInstanceIdentifiers ["staging-replica"] :PreferredMaintenanceWindow "tue:02:00-tue:03:00"} {:DBInstanceIdentifier "staging-replica" :ReadReplicaSourceDBInstanceIdentifier "staging" :PreferredMaintenanceWindow "tue:03:00-tue:04:00"}] - "production" "staging"))) + "production" nil "staging"))) + + (is (= [(op/restore-snapshot snapshot-id {:DBInstanceIdentifier "production" + :DBInstanceArn "arn:db:production" + :PreferredMaintenanceWindow "tue:01:00-tue:02:00" + :ReadReplicaDBInstanceIdentifiers ["production-replica"] + :DBSubnetGroup {:VpcId "v1"}} "temp-staging") + (op/enable-backups "temp-staging" + {:PreferredMaintenanceWindow "tue:02:00-tue:03:00" :MonitoringInterval 60}) + (op/create-replica "temp-staging" "temp-staging-replica") + (op/modify "temp-staging-replica" + {:PreferredMaintenanceWindow "tue:03:00-tue:04:00"}) + (op/rename "staging-replica" "old-staging-replica") + (op/rename "staging" "old-staging") + (op/rename "temp-staging-replica" "staging-replica") + (op/rename "temp-staging" "staging") + (op/delete "old-staging-replica") + (op/delete "old-staging")] + (plan/replace-tree + [{:DBInstanceIdentifier "production" :DBInstanceArn "arn:db:production" + :ReadReplicaDBInstanceIdentifiers ["production-replica"] + :PreferredMaintenanceWindow "tue:01:00-tue:02:00" :DBSubnetGroup {:VpcId "v1"}} + {:DBInstanceIdentifier "production-replica" :ReadReplicaSourceDBInstanceIdentifier "production" + :PreferredMaintenanceWindow "tue:01:00-tue:02:00" :DBSubnetGroup {:VpcId "v1"}} + {:DBInstanceIdentifier "staging" :ReadReplicaDBInstanceIdentifiers ["staging-replica"] + :PreferredMaintenanceWindow "tue:02:00-tue:03:00" :MonitoringInterval 60} + {:DBInstanceIdentifier "staging-replica" :ReadReplicaSourceDBInstanceIdentifier "staging" + :PreferredMaintenanceWindow "tue:03:00-tue:04:00"}] + "production" snapshot-id "staging"))) (is (= [(op/create-replica "production" "temp-staging" {:Tags tags}) (op/promote "temp-staging") @@ -114,7 +167,7 @@ (op/shell-command "./restart.sh") (op/delete "old-staging-replica") (op/delete "old-staging")] - (plan/replace-tree instances "production" "staging" + (plan/replace-tree instances "production" nil "staging" :restart "./restart.sh" :tags {"staging" tags "staging-replica" tags}))))) @@ -128,6 +181,8 @@ (plan/attempt instances (op/create {:DBInstanceIdentifier "a"})))) (is (= [:skip (plan/duplicate-instance "b")] (plan/attempt instances (op/create-replica "a" "b")))) + (is (= [:skip (plan/duplicate-instance "b")] + (plan/attempt instances (op/restore-snapshot "c" "a" "b")))) (is (= [:skip (plan/promoted-instance "a")] (plan/attempt instances (op/promote "a")))) diff --git a/test/stack_mitosis/policy_test.clj b/test/stack_mitosis/policy_test.clj index 819a745..fba49e1 100644 --- a/test/stack_mitosis/policy_test.clj +++ b/test/stack_mitosis/policy_test.clj @@ -7,6 +7,9 @@ (defn fake-arn [name] (str "arn:aws:rds:us-east-1:1234567:db:" name)) +(defn fake-snapshot-arn [name] + (str "arn:aws:rds:us-east-1:1234567:snapshot:" name)) + (defn example-instances [] [{:DBInstanceIdentifier "production" :ReadReplicaDBInstanceIdentifiers ["production-replica"] :DBInstanceArn (fake-arn "production")} @@ -47,6 +50,16 @@ {:op :AddTagsToResource :arn "arn:aws:rds:us-east-1:1234567:db:bar"}] (sut/permissions [instance] (op/create-replica "foo" "bar")))) + (is (= [{:op :RestoreDBInstanceFromDBSnapshot + :arn ["arn:aws:rds:*:*:og:*" + "arn:aws:rds:*:*:pg:*" + "arn:aws:rds:*:*:subgrp:*" + "arn:aws:rds:us-east-1:1234567:snapshot:*" + "arn:aws:rds:us-east-1:1234567:db:bar"]} + {:op :AddTagsToResource + :arn "arn:aws:rds:us-east-1:1234567:db:bar"} + {:op :DescribeDBSnapshots :arn "*"}] + (sut/permissions [instance] (op/restore-snapshot "foo" instance "bar")))) (is (= [{:op :ModifyDBInstance :arn ["arn:aws:rds:*:*:og:*" "arn:aws:rds:*:*:pg:*" @@ -95,4 +108,36 @@ (sut/allow [:DeleteDBInstance] (mapv fake-arn ["old-staging-replica" "old-staging"]))]} (sut/from-plan (example-instances) - (plan/replace-tree (example-instances) "production" "staging"))))) + (plan/replace-tree (example-instances) "production" nil "staging")))) + (is (= {:Version "2012-10-17" + :Statement + [(sut/allow [:DescribeDBInstances :ListTagsForResource] + ["arn:aws:rds:*:*:db:*"]) + (sut/allow [:RestoreDBInstanceFromDBSnapshot] + ["arn:aws:rds:*:*:og:*" + "arn:aws:rds:*:*:pg:*" + "arn:aws:rds:*:*:subgrp:*" + (fake-snapshot-arn "*") + (fake-arn "temp-staging")]) + (sut/allow [:AddTagsToResource] + (mapv fake-arn ["temp-staging" "temp-staging-replica"])) + (sut/allow [:DescribeDBSnapshots] ["*"]) + (sut/allow [:ModifyDBInstance] + (into ["arn:aws:rds:*:*:og:*" + "arn:aws:rds:*:*:pg:*" + "arn:aws:rds:*:*:secgrp:*" + "arn:aws:rds:*:*:subgrp:*"] + (mapv fake-arn ["temp-staging" "temp-staging-replica" + "staging-replica" "old-staging-replica" + "staging" "old-staging"]))) + (sut/allow [:RebootDBInstance] + (mapv fake-arn ["temp-staging" "temp-staging-replica" "staging-replica" "staging"])) + (sut/allow [:CreateDBInstanceReadReplica] + (into ["arn:aws:rds:*:*:og:*" + "arn:aws:rds:*:*:pg:*" + "arn:aws:rds:*:*:subgrp:*"] + (mapv fake-arn ["temp-staging" "temp-staging-replica"]))) + (sut/allow [:DeleteDBInstance] + (mapv fake-arn ["old-staging-replica" "old-staging"]))]} + (sut/from-plan (example-instances) + (plan/replace-tree (example-instances) "production" "snapshot-id" "staging")))))