This documents how to perform a proper upgrade of the Security Monkey instance.
If you are an existing user of Security Monkey (version before 1.0), please read this section as there are breaking changes in the 1.0 version of Security Monkey.
If you are upgrading to 1.0 for the first time, please review the Quickstart and the Autostarting documents as there is a new deployment pattern for Security Monkey.
Security Monkey now has the ability to scale workers horizontally thanks to Celery. This changes the deployment story for Security Monkey, and also changes how the scheduling of tasks works. This is documented in the autostarting file. Please read this file as it explains the 1.0+ architecture.
Security Monkey now has 5 primary components:
- UI (Can have many behind a load balancer)
- Scheduler (EXACTLY ONE)
- Workers (Can have many)
- PostgreSQL Database (for storage)
- Redis (message broker for workers)
Also, (for AWS) please review the IAM documentation as there are new permissions required.
As a general guidance, the deployment of Security Monkey should follow these steps:
- Deploy the UI behind a Load Balancer
- Tear-down the scheduler instance
- Tear-down all worker instances
- At this point, you should perform any database migrations and upgrades as there are no workers that will be affected.
- Deploy the new scheduler instance -- wait for it to come online
- Deploy many worker instances
Performing the steps in this order will ensure:
- That no duplicate schedulers are running (multiple schedulers running causes havoc)
- Proper DB upgrades occur without possibly impacting workers mutating the database
- The schedulers and workers are working properly together
- Prerequisites
- Backup and stop services
- Clone
security_monkeyand update environment - Compile (or download) the web UI
- Tear down scheduler and workers
- Update database and configurations
- Start services (Start UI first, then the scheduler, then the workers)
This doc assumes you already have installed and are running a Security Monkey environment. It especially assumes you have following on your system
- https://github.com/Netflix/security_monkey project files are available under /usr/local/src/security_monkey
- Supervisor configured and running
- Python virtualenv
- Redis
- NGINX
As new features come out, Security Monkey may require new IAM permissions. Always follow the respective IAM doc for the given technology to see if you need to update your Security Monkey permissions. Failure to do this will result in Access Denied errors and items not appearing in Security Monkey.
Backup your /usr/local/src/security_monkey/env-config/config.py and move your existing installation to backup directory
cp /usr/local/src/security_monkey/env-config/config.py ~/
mkdir ~/security_monkey_backup && mv /usr/local/src/security_monkey/ ~/security_monkey_backup/
Stop all Security Monkey services using supervisorctl.
sudo supervisorctl stop securitymonkeyui
sudo supervisorctl stop securitymonkeyscheduler
sudo supervisorctl stop securitymonkeyworkers
Major releases are on the master branch. Please be aware that these releases happen slowly over time. If you require the latest and greatest features (as well as bug fixes), please checkout the develop branch.
git clone https://github.com/Netflix/security_monkey.git into the your Security Monkey location
$ cd /usr/local/src
$ sudo git clone --depth 1 --branch develop https://github.com/Netflix/security_monkey.git
Activate your Python virtualenv and run pip install -e .
cd security_monkey
virtualenv venv
source venv/bin/activate
pip install --upgrade setuptools
pip install --upgrade pip
pip install --upgrade urllib3[secure] # to prevent InsecurePlatformWarning
pip install google-compute-engine # Only required on GCP
pip install oauth2client # Required to retrieve GCP data
pip install google-api-python-client # Required to retrieve GCP data
pip install httplib2 # Required to retrieve GCP data
pip install cloudaux\[gcp\]
pip install cloudaux\[openstack\] # Only required on OpenStack
pip install -e .
pip install -e ."[tests]"
If you're using the stable (master) branch, you have the option of downloading the web UI instead of compiling it. Visit the latest release https://github.com/Netflix/security_monkey/releases/latest and download static.tar.gz.
If you're using the bleeding edge (develop) branch, you will need to compile the web UI by following these instructions. If you have not done this during installation follow this section in quickstart guide
Compile the web-app from the Dart code
cd /usr/local/src/security_monkey/dart
sudo /usr/lib/dart/bin/pub get
sudo /usr/lib/dart/bin/pub build
mkdir -p /usr/local/src/security_monkey/security_monkey/static/
/bin/cp -R /usr/local/src/security_monkey/dart/build/web/* /usr/local/src/security_monkey/security_monkey/static/
chgrp -R www-data /usr/local/src/security_monkey
Replace the config file that we previously backed up.
sudo mv ~/config.py /usr/local/src/security_monkey/env-config/
If your file is named something other than config.py, you will want to set the SECURITY_MONKEY_SETTINGS environment variable to point to your config:
export SECURITY_MONKEY_SETTINGS=/usr/local/src/security_monkey/env-config/config-deploy.py
Security Monkey uses Flask-Migrate (Alembic) to keep database tables up to date. To update the tables, run this command (while
in your virtual environment):
Note: monkey db upgrade is idempotent. You can re-run it without causing any harm.
cd /usr/local/src/security_monkey/
monkey db upgrade
sudo supervisorctl start securitymonkeyui
sudo supervisorctl start securitymonkeyscheduler
sudo supervisorctl start securitymonkeyworkers
Note: Netflix doesn't upgrade/patch Security Monkey systems. Instead simply rebake a new instance with the new version.