Web server for pulling from github and updating static content
Ian's advice: I recommend you never do any more development on the static website udpater. Use something like Bamboo instead. Better to use a tool designed exactly for this purpose instead of this custom code. And our friends in the SPOT team already use bamboo.
The only endpoint is /update_project/{project_name}
e.g. curl -XPOST http://localhost:8001/update_project/hipsci
The process is basically this:
- Queries the rate limiter to find out if the project is already updating
- Queues a new job and exits if a job is already queued
- Uses the GitUpdater module to pull updates from git
- Uses the NPMBuilder module to build the static content from the git repo
- Uses the Rsyncer module to copy the jekyll _site directory to the webserver directories
- Uses the ElasticSitemapIndexer to put the static content into elasticsearch. This enables site search.
- Uses the PubSubHubBub module to announce updates to the rss feeds. This triggers automatic tweets.
- Repeat 2-6 if a new job has been queued since the update began
Steps 2-6 get executed in a forked process because they are blocking.
Errors get emailed to users, set in the config file
./etc/init.d/website_updater install
This installs the following to the ./local directory:
- cpan modules
- Ensembl git tools
- git (because it's not installed on our VMs by default)
Run the installation command from a login node with the same linux distribution as our VM, i.e. from a rh-6 login machine