lectures/lecture-04-git-and-github/lecture-04-git-and-github.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# git & GitHub: Introduction for Scientists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# git, github, etc... Why do we care?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Replication** and **reproducibility** are two of the cornerstones in the scientific method. With respect to data analysis (and scientific computing in general!), these concepts have the following practical implications:\n",
    "\n",
    "* **Replication**: An author of a scientific paper that involves some data analysis should be able to rerun the analysis code and replicate the results upon request. Other scientists should be able to perform the same analysis and obtain the same results, given the information about the methods used in a publication.\n",
    "\n",
    "* **Reproducibility**: The results obtained by analyzing the data should be reproducible with an independent implementation of the method, or using a different method altogether.\n",
    "\n",
    "\n",
    "In summary: A sound scientific result should be reproducible, and a sound scientific study should be replicable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To achieve these goals, we need to:\n",
    "\n",
    "* Keep and take note of *exactly* which source code and version that was used to produce data and figures in published papers.\n",
    "\n",
    "* Record information of which version of external software that was used. Keep access to the environment that was used.\n",
    "\n",
    "* Make sure that old codes and notes are backed up and kept for future reference.\n",
    "\n",
    "* Ideally, codes should be published online, to make it easier for other scientists interested in the codes to access it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is git?\n",
    "\n",
    "* A tool that efficiently __saves snapshots__ of a set of files (what their contents was at some point in time).\n",
    "* A tool that __tracks relationships__ between those snapshots (which snapshot preceeded which, etc.).\n",
    "* A tool that can efficiently __merge__ different snapshots, even if they have conflicting changes.\n",
    "* A tool that makes it possible to __share__ these snapshots with others, enabling __collaboration__ (and adding a degree of reproducibility)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is GitHub?\n",
    "\n",
    "* It’s a company that will __host copies of your git repositories__.\n",
    "* It’s a [website](github.com) that provides a nice way to __browse, view, and sometimes edit__ the contents of those repositories.\n",
    "* It’s a company that will let you __share__ them with others, and make them __discoverable__.\n",
    "* It’s a website that makes __collaboration__ on software (incl. research, analysis, etc.) projects __easy__. E.g.:\n",
    "  * Report and discuss issues.\n",
    "  * Review, discuss, accept proposed changes.\n",
    "\n",
    "\n",
    "Note: Git and GitHub are two different things -- one is a tool, another one is a service built around it (and there are alternatives). You can use git without GitHub; it's just that GitHub makes some things easier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# git+github: introduction by example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# GitHub Student Developer Pack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>[https://education.github.com/pack](https://education.github.com/pack)</center>\n",
    "<img src=\"images/gh-pack.jpg\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Note\n",
    "\t\n",
    "`git` repositories are also excellent for version controlling manuscripts, figures, thesis files, data files, lab logs, etc. Basically for any digital content that must be preserved and is frequently updated.\n",
    "\n",
    "They are also excellent collaboration tools!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This is how we will practice using git and GitHub\n",
    "\n",
    "### Forking and working on someone elses reposiitory\n",
    "\n",
    "Lets fork the completely useless repository\n",
    "\n",
    "https://github.com/connolly/A302_W19_a_useless_repo\n",
    "\n",
    "```\n",
    "click on \"fork\" at the top right of the page\n",
    "```\n",
    "\n",
    "This creates a version of the repository in your local Github account\n",
    "\n",
    "Now we want to \"clone\" this repository to our own machine\n",
    "\n",
    "```git clone https://github.com/YOURNAME/A302_W19_a_useless_repo.git```\n",
    "\n",
    "We will work on this page using Github and git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What if Andy changes the repository\n",
    "\n",
    "##### Collaboration with GitHub\n",
    "\n",
    "`git` has a very different collaboration philosophy compared to (e.g.) Google\n",
    "Docs.  In Google Docs, you edit in real time, seeing your and everyone\n",
    "else's changes as they happen.  With `git`, you make a copy of the files\n",
    "(a `clone` of the repository), make the edits on the copy, and use `git` \n",
    "(and/or GitHub) to merge them with the original. To signal that a set\n",
    "of changes is ready to be merged into the original, you open a [Pull Request](https://help.github.com/articles/about-pull-requests/).\n",
    "\n",
    "I will add a name to the file \"names.md\"\n",
    "\n",
    "```\n",
    "emacs names.md\n",
    "```\n",
    "\n",
    "We can also check the status of the files in my directory\n",
    "```\n",
    "git status\n",
    "```\n",
    "\n",
    "We can also check on what has happened to the directory over time\n",
    "\n",
    "```\n",
    "git log\n",
    "```\n",
    "\n",
    "Now we want to commit a change to our local repository, push that change to the repository owned by Connolly\n",
    "\n",
    "```\n",
    "git commit names.md -m \"add a name\" # -m adds a comment (required)\n",
    "git pull  # to make sure our version is uptodate\n",
    "git push \n",
    "```\n",
    "\n",
    "On the connolly Github repository it shows an updated \"names.md\" file. On your fork it should show that your version is one \"commit\" behind\n",
    "\n",
    "We can update our github version by issuing a \"pull request\"\n",
    "\n",
    "```\n",
    "connolly:master is up to date with all commits from connolly2:master. Try switching the base for your comparison\n",
    "```\n",
    "This means we havent added anything to the repository so we dont need to push to to the Connolly repository. If we \n",
    "\"switch the base\" we see Connolly is ahead and we can\n",
    "\n",
    "```\n",
    "Create a pull request\n",
    "Merge a pull request \n",
    "Confirm a pull request\n",
    "```\n",
    "Now we are all up to date\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What if I change the repository\n",
    "\n",
    "\n",
    "You should add a name to the file \"names\" (put in the middle of the list)\n",
    "\n",
    "```\n",
    "emacs names.md\n",
    "```\n",
    "\n",
    "We can also check the status of the files in my directory\n",
    "```\n",
    "git status\n",
    "```\n",
    "\n",
    "We can also check on what has happened to the directory over time\n",
    "\n",
    "```\n",
    "git log\n",
    "```\n",
    "\n",
    "Now we want to commit a change to our local repository, push that change to our remote repository and issue a pull request to merge it with the one owned by Connolly\n",
    "\n",
    "```\n",
    "git commit names.md -m \"add my name\" # -m adds a comment (required)\n",
    "git pull  # to make sure our version is uptodate\n",
    "git push \n",
    "```\n",
    "\n",
    "On the our Github repository it shows an updated \"names.md\" file. \n",
    "\n",
    "We can update the Connolly Github version by issuing a \"pull request\" and including a comment if we like\n",
    "\n",
    "Oh no there is a merge conflict!\n",
    "\n",
    "```\n",
    "<<<<<<< master\n",
    "Arthur\n",
    "=======\n",
    "Simon\n",
    ">>>>>>> master\n",
    "```\n",
    "\n",
    "<<<<<<< to ====== is the change we have locally\n",
    "\n",
    "======= to <<<<<< are the lines on the remote repository\n",
    "\n",
    "We need to edit the file and \"resolve\" the conflict\n",
    "\n",
    "We can resolve easy ones in the Github client but lets do it with git\n",
    "\n",
    "\n",
    "First we need to update our local repo to what the remote repo looks like\n",
    "\n",
    "```\n",
    "git pull https://github.com/connolly/A302_W19_a_useless_repo master\n",
    "```\n",
    "\n",
    "pull from the Connolly repo to my ```master```\n",
    "\n",
    "``` \n",
    "error: Your local changes to the following files would be overwritten by merge:\n",
    "\tnames.md\n",
    "```\n",
    "\n",
    "```\n",
    "edit \"names.md\"\n",
    "\n",
    "git commit -a -m \"merge\"\n",
    "```\n",
    "\n",
    "Your pull request has updated and can be merged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's add your own directory and issue a pull request to \n",
    "https://github.com/connolly/A302_2019_Homework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's take a look at homework 2\n",
    "https://github.com/connolly/A302_2019/blob/master/homework/Homework%202.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Follow the instructions to create your own repo on the command line\n",
    "\n",
    "### Creating my own repo\n",
    "\n",
    "Let's create a directory, and a few (two) files in a directory:\n",
    "\n",
    "```\n",
    "mkdir \"astr302-$USER\"\n",
    "cd \"astr302-$USER\"\n",
    "echo \"# My first ASTR 302 git repository\" > README.md\n",
    "echo \"A file I do not wish to track\" > AnotherFile.txt\n",
    "```\n",
    "\n",
    "I'll now create a git _repository_ where git will store snapshot of these files, using the `git init` command:\n",
    "\n",
    "```\n",
    "git init\n",
    "```\n",
    "\n",
    "Now git will be able to \"track\" (store snapshots of) files in your current directory. You have to run `git init` only once for that directory.\n",
    "\n",
    "Running `git init` created a directory named `.git`, where the snapshots will be stored:\n",
    "\n",
    "```\n",
    "ls -l .git\n",
    "```\n",
    "\n",
    "If you want to send someone all of your snapshots, it's as easy as copying this directory (but there's a better way! more later!)\n",
    "\n",
    "Some more bookkeeping before we begin. We need to tell git what our name and preferred e-mail are:\n",
    "\n",
    "```\n",
    "git config --global user.email “your@email”\n",
    "git config --global user.name “your name\"\n",
    "```\n",
    "\n",
    "Now we can begin! Git doesn't automatically snapshot the contents of everything in the directory -- you have to tell it what is the set of files I want it to snapshot. We do this using the `git add` command. For example:\n",
    "\n",
    "```\n",
    "git add README.md\n",
    "```\n",
    "\n",
    "Note that at this point, we didn’t take a snapshot yet. To do that, we use the `git commit` command:\n",
    "\n",
    "```\n",
    "git commit\n",
    "```\n",
    "\n",
    "We've just created a snapshot, which is referred to as a _commit_ in git. Running `git commit` opens an editor in which you can write your _commit message_. This is meant to be a short description of the set of files (or changes to files; we’ll show that in a minute). It should remind you what this snapshot is about (or tell others with whom you share this code) .\n",
    "\n",
    "A convention for commit messages is to start with a short (\\<70 characters) one-line description of the reason for the commit, followed by a blank line, followed by a longer paragraph (if needed). See [here](https://github.com/erlang/otp/wiki/writing-good-commit-messages) for an example of how to write useful commit messages.\n",
    "\n",
    "At this point, git has stored a single commit:\n",
    "\n",
    "```\n",
    "git log\n",
    "```\n",
    "\n",
    "We can also check the status of the files in my directory\n",
    "```\n",
    "git status\n",
    "```\n",
    "\n",
    "Let’s now edit the README.md and check the status again:\n",
    "```\n",
    "echo \"\">> README.md\n",
    "echo \"We’re practicing git here.\" >> README.md\n",
    "\n",
    "git status\n",
    "```\n",
    "\n",
    "Committing all these changes:\n",
    "```\n",
    "git commit -a\n",
    "\n",
    "git log\n",
    "```\n",
    "The `-a` argument to `git commit` tells git to commit all files it's tracking and that have changed. Alternatively, you could've ran `git add` for every file individually, before running `git commit`. Note that `git add` serves two somewhat different purposes: it tells git which files to track, but it also tells git which files to include into the next commit (as an aside: these can be viewed as being one and the same...).\n",
    "\n",
    "How to see the differences to a previous version:\n",
    "```\n",
    "git diff HEAD^\n",
    "```\n",
    "\n",
    "Git keeps a history of all commits. The top of this history is called the “HEAD”. Adding a ^ means “one before head”.\n",
    "\n",
    "Let’s make more changes:\n",
    "```\n",
    "git add AnotherFile.txt\n",
    "git commit\n",
    "```\n",
    "\n",
    "And a prettier log may be shown with:\n",
    "\n",
    "```\n",
    "git log --oneline\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GitHub\n",
    "\n",
    "How do we effectivelly share our repositories? This is where GitHub comes in.\n",
    "\n",
    "First, let’s make sure you all have accounts on GitHub.\n",
    "\n",
    "Then, let us create a new repository (named astr302-$USER$, where $USER$ is\n",
    "your _local_ username) and then:\n",
    "\n",
    "```\n",
    "git remote add origin git@github.com:your-github-account/astr302-USER.git\n",
    "git push --all -u\n",
    "```\n",
    "\n",
    "In the first command, you should replace the URL (the web address) in the first line with the one for your repository. Note that GitHub will give you an address starting with `https://` by default -- unless you've set up [SSH keys](https://help.github.com/articles/connecting-to-github-with-ssh/), use it.\n",
    "The second command _copies_ all the commits stored on your local hard drive repository (the `.git` directory) to GitHub. Now refresh the page on GitHub and explore what GitHub has to offer!\n",
    "\n",
    "You can consider the copy on GitHub as your 'backup copy'. Even if you lose your entire computer, the commits of your files are safe and sound on GitHub.\n",
    "\n",
    "As you make new changes and new commits, you should occasionally push them _upstream_:\n",
    "```\n",
    "git push\n",
    "```\n",
    "This will push the commits from the current branch to the GitHub copy. Make it a habit to do this at least once a day.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Branches\n",
    "\n",
    "Imagine you have your program/analysis working reasonably well. Now you want to make some changes, but are worried you’ll mess up your code if they don’t work out. One way to do it is to just copy the whole directory into a backup folder.\n",
    "\n",
    "But git provides a facility — branches — to do it much more effectively. Creating a branch is like creating a parallel universe: what happens there won’t affect what’s in the “main” universe (called the _master branch_). You're free to experiment with your changes on a different branch, and then merge it back into _master_ once you're happy with the results.\n",
    "\n",
    "Let's create and check out a branch named `new-feature`.\n",
    "\n",
    "```\n",
    "git branch\n",
    "\n",
    "git branch new-feature\n",
    "git branch\n",
    "\n",
    "git checkout new-feature\n",
    "git branch\n",
    "\n",
    "A shorthand for create-and-checkout-a-branch is `git checkout -b new-feature`\n",
    "\n",
    "```\n",
    "\n",
    "Let’s make some edits:\n",
    "```\n",
    "echo \"New edits\" >> README.md\n",
    "git status\n",
    "git commit -a\n",
    "```\n",
    "\n",
    "See the history:\n",
    "```\n",
    "git log --oneline\n",
    "git log --oneline --decorate\n",
    "```\n",
    "Note how individual commits are identified by their _commit id_, a 40-character unique checksum such as 9e130f7b1ff3bfdcf23702ee18da5b7cdc8b6ef4 (a so-called SHA1 hash).\n",
    "\n",
    "We can switch back to the `master` branch\n",
    "```\n",
    "git checkout master\n",
    "cat README.md\n",
    "```\n",
    "\n",
    "Note how changes made on the `new-feature` branch are not visible here (as expected).\n",
    "\n",
    "We can go ahead and also make some changes on master:\n",
    "```\n",
    "echo \"Another tracked file\" > AnotherFile.txt\n",
    "git commit -a\n",
    "```\n",
    "\n",
    "Let's look at what all this looks like:\n",
    "```\n",
    "git log --oneline --decorate\n",
    "git log --oneline --decorate --graph --all\n",
    "```\n",
    "\n",
    "Now we can bring the two sets of changes together: “merge” the two branches.\n",
    "\n",
    "```\n",
    "git merge new-feature\n",
    "git log --oneline --decorate --graph --all\n",
    "```\n",
    "\n",
    "This is “branch & merge” technique is a powerful way to incrementally develop analyses or codes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding out more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * [Google](http://google.com)\n",
    " * [YouTube](http://youtube.com)\n",
    "   * [Scott Chacon on git](https://www.youtube.com/watch?v=ZDR433b0HJY)\n",
    "   \n",
    "   \n",
    " * [gitref.org](http://gitref.org/index.html)\n",
    " * [LSST's page on git](https://confluence.lsstcorp.org/display/LDMDG/Using+Git+for+LSST+Development)\n",
    "\n",
    "\n",
    " * [git website](http://git-scm.com)\n",
    " * [github.com](http://github.com)\n",
    "\n",
    "\n",
    " * [Reproducible Research in Computational Science](http://dx.doi.org/10.1126/science.1213847), Roger D. Peng, Science 334, 1226 (2011).\n",
    " * [Shining Light into Black Boxes](http://dx.doi.org/10.1126/science.1218263), A. Morin et al., Science 336, 159-160 (2012).\n",
    " * [The case for open computer programs](http://dx.doi.org/doi:10.1038/nature10836), D.C. Ince, Nature 482, 485 (2012)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}