In this tutorial, we're going to build a simple Shepherd migration spec and apply it to a demo repo.
Note: this tutorial is designed for people who like to learn by doing. If that's not how you prefer to learn, you can check out the docs on the Readme.
ESLint has deprecated extensionless .eslintrc
files - they're now expected to have an extension indicating the format of the contents, like .eslintrc.yml
. Your organization has 100s of JavaScript repositories that were created from the same repository template, and they all use YAML in extensionless .eslintrc
files. You want to get ahead of the game and rename them in preparation for the inevitable breaking change that removes support for the old filename. You could try to convince your coworkers to do this, but they're far too busy disrupting the "Tinder for Uber for cats" industry. You could also make these changes yourself, but that would take forever and you also have better things to do with your time.
Thankfully, you just heard about this tool called Shepherd that can help you automate code changes across infinitely many repositories at once. Let's build a Shepherd migration to do all the hard work for you.
Of course, for this tutorial, you don't have access to hundreds of repositories to try Shepherd out on. So, we'll be running on this migration on only one repository. While it may seem silly to do all this work just to rename a single file in a single repository, keep in mind that once you have a migration working for one repository, it will scale efforlessly to as many repositories as your organization has.
We've set up a simple repository that you can fork for the purpose of this demo: shepherd-demo. You should fork this repository first, but there's no need to clone it to your machine! Shepherd will take care of that for you. Note that there's no actual project in the repository, as that's not relevant to this tutorial.
Create a directory to hold files for this migration somewhere on your machine. We'll be using the directory ~/shepherd-migration
.
mkdir ~/shepherd-migration
cd ~/shepherd-migration
Shepherd migrations are declaratively specified in a file named shepherd.yml
. We call this file a spec.
Create that file in the directory you just made and open it in your favorite editor.
touch shepherd.yml
This tutorial goes somewhat slowly to explain a lot of core concepts. If you're like me and need instant gratification and want to see a fully-automated pull request now, you can scroll all the way to the bottom of this page to see the completed migration spec and the commands needed to apply it. Once that's satiated your need for immediate results, come back here and see how it all works!
Let's bootstrap the file with some contents.
# shepherd.yml
id: 2018.08.15-eslintrc-yml
title: Rename all .eslintrc files to .eslintrc.yml
The id
value will be used to as a unique identifier for this migration, as well as the name of the branch that Shepherd creates. The title
will be used to build a commit message and a title for the pull request that we'll open.
Shepherd isn't tied to any specific version control system; all interactions with repositories and hosted sites like GitHub and Bitbucket happen through a layer of abstraction called an adapter. You can read more about how adapters work here if you're interested.
To tell Shepherd which adapter we want to use, we can add to our shepherd.yml
file:
# shepherd.yml
adapter:
type: github
This tells Shepherd that we should use the GitHub adapter. This will be relevant when finding repositories to operate on, cloning repositories, pushing changes, and opening pull requests.
If we wanted to be naïve, we could check out every repository you own looking for .eslintrc
files. Thankfully, GitHub has the ability to search repos using advanced search qualifiers. You can write a search query to identify repositories that are candidates for migrations. If a repository contains a file matching the search, it will be checked out as a candidate for a migration.
# shepherd.yml
adapter:
type: github
search_query: org:NerdWalletOSS path:/ filename:.eslintrc
This search query will be used to identify any repositories in the NerdWalletOSS
GitHub organization that contain a .eslintrc
file in the repository root.
For the purpose of this demo, we'll change the search query to only match your forked demo repo to avoid modifying any of your own repositories that might actually have a .eslintrc
file in them.
Note: you should change YOURUSERNAME
to your own username if you've cloned our demo repository.
#shepherd.yml
adapter:
type: github
search_query: repo:YOURUSERNAME/shepherd-demo path:/ filename:.eslintrc
Note: you can provide an org
instead of search_query
. GitHub code search can be unreliable. Using org
will return a list of every visible repo in a GitHub organization, allowing you to match more confidently.
#shepherd.yml
adapter:
type: github
org: YOURORGANIZATION
Note: while for purposes of this tutorial, searching codebases for a file is exactly what we want, other use cases may call for different types of searching, which Github supports. In addition to searching code, shepherd also allows you to do a search of repository metadata. For example, if your repo had an associated 'eslint-repo' topic:
#shepherd.yml
adapter:
type: github
search_type: repository
search_query: topic:eslint-repo
GitHub's search is still relatively limited. For instance, it doesn't let you determine which version of a dependency a repository uses. To perform additional checks, Shepherd lets you define commands that Shepherd will run to determine if a checked-out repository should be migrated.
For our demo, let's add a simple sanity check to ensure that the checked out repositories all contain a .eslintrc
file.
# shepherd.yml
hooks:
should_migrate:
- ls .eslintrc
If the file does not exist, ls
will exit with a non-zero exit code, which signals Shepherd that this check failed and that the repository should not be migrated.
This will be largely redundant, since we're already doing this check with the GitHub query. To demonstrate the power of these hooks, we could add a hook to filter out any repositories that haven't been committed to in the current year (2018, at the time of writing). In a large organization, this can help prevent noise on old repositories that might not be actively developed anymore.
# You don't actually need to add this to your shepherd.yml, this is just an example!
hooks:
should_migrate:
- git log -1 --format=%cd | grep 2018 --silent
This prints the date of the last commit and then checks that it contains the string 2018
in it. If it doesn't, grep
will exit with a non-zero exit code, failing the check.
Your shepherd.yml
should now look like this:
# shepherd.yml
id: 2018.08.15-eslintrc-yml
title: Rename all .eslintrc files to .eslintrc.yml
adapter:
type: github
search_query: repo:YOURUSERNAME/shepherd-demo path:/ filename:.eslintrc
hooks:
should_migrate:
- ls .eslintrc
We're finally ready to start checking our the repositories! Make sure you've followed the installation instructions on the README, and then run the following command.
shepherd checkout ~/shepherd-demo
Shepherd will look for a shepherd.yml
file in the provided directory and then check out repositories based on the spec in that file. After this command completes, you should see output similar to the following:
✔ Loaded 1 repos
[nwalters512/shepherd-demo] 1/1
✔ Checked out repo
> Running should_migrate steps
$ ls .eslintrc
.eslintrc
Step "ls .eslintrc" exited with 0
✔ Completed all should_migrate steps successfully
> Running post_checkout steps
✔ Completed all post_checkout steps successfully
Checked out 1 out of 1 repos
Woot! Your repository has been checked out to your machine and passed the should_migrate
checks you defined.
Note that Shepherd doesn't do anything magic with repositories. They're checked out to ~/.shepherd/2018.08.15-eslintrc-yml/repos
, and you can always poke around in the repositories as you're developing migrations to check that they're working, manually run tests, etc.
Now that you've got your repositories checked out, we can define what the migration actually does (rename .eslintrc
to .eslintrc.yml
):
hooks:
apply:
- mv .eslintrc .eslintrc.yml
That's it! Not too bad, right?
Run the following command to apply the migration to all checked out repositories:
shepherd apply ~/shepherd-migration
Run the following command to commit all the changes to their respective repositories:
shepherd commit ~/shepherd-migration
This will commit the changes with a message derived from the migration spec title:
[shepherd] Rename all .eslintrc files to .eslintrc.yml
Run the following command to push all the remote branches up to GitHub:
shepherd push ~/shepherd-migration
You should now see your branch in the repository if you visit GitHub.
We're almost done! Our last step is to open a pull request with these changes. To do that, we'll need to teach Shepherd how to generate a message for the pull request. A special pr_message
hook is available for this purpose. It will execute all the specified commands and concatenate their standard output together to form a message. Add the following to your hooks
:
# shepherd.yml
hooks:
pr_message:
- echo 'Hey! This PR renames `.eslintrc` to `.eslintrc.yml`'
You can now open a pull request for your repos:
shepherd pr ~/shepherd-migration
If you visit your repository, you should see that Shepherd opened a pull request for you!
Congrats! You just applied an automated Shepherd migration to a repo! While this may seem trivial for just a single repo, it scales efforlessly to as many repos as you need. You can also perform significantly more complex tasks than just renaming a file. If you can automate your changes with any tool in any language, you can apply it across all your repos with Shepherd.
Here's our finished migration spec:
# shepherd.yml
id: 2018.08.15-eslintrc-yml
title: Rename all .eslintrc files to .eslintrc.yml
adapter:
type: github
search_query: repo:YOURUSERNAME/shepherd-demo path:/ filename:.eslintrc
hooks:
should_migrate:
- ls .eslintrc
apply:
- mv .eslintrc .eslintrc.yml
pr_message:
- echo 'Hey! This PR renames `.eslintrc` to `.eslintrc.yml`'
And here are all the commands we used to apply this migration:
shepherd checkout ~/shepherd-migration
shepherd apply ~/shepherd-migration
shepherd commit ~/shepherd-migration
shepherd push ~/shepherd-migration
shepherd pr ~/shepherd-migration