index.qmd

---
title: The basics of working on the UNIX command line
date: 2024-10-31
format:
  html:
    theme: cosmo
    css: assets/styles.css
    toc: true
    code-copy: true
    code-block-bg: true
    code-block-border-left: "#31BAE9" 
execute:
  freeze: auto
engine: knitr
ipynb-shell-interactivity: all
code-overflow: wrap
---

<!--
Determine output under the scflocal account on gandalf
-->

# 1 Introduction

## 1.1 This Tutorial


This tutorial covers the basics of navigating in a UNIX-like (e.g., Linux or MacOs) environment. In particular, it covers using the UNIX command line interface, a powerful way to carry out operations on a computer and to automate tasks.  Being familiar with operating on the command line will allow you (with some practice and training) to do things more quickly and in a way that can be reproduced later. That's hard or impossible to do if you are doing point-and-click or drag-and-drop operations in a File Manager or Finder window.

Materials for this tutorial, including the Quarto Markdown file that was used to create this document are [available on GitHub](https://github.com/berkeley-scf/tutorial-unix-basics).

Software Carpentry has a very nice introductory lesson on the [basics of the shell](https://swcarpentry.github.io/shell-novice/). It also has an accompanying [YouTube video](https://www.youtube.com/watch?v=8c1BL5b47kg). Episodes 1-3 (the first 20 minutes) cover the material that is in this tutorial.

:::{.callout-note title="License"}
This tutorial by Christopher Paciorek is licensed under a Creative Commons Attribution 3.0 Unported License.
:::


## 1.2 The shell

Operating on the UNIX command line is also known as "using the terminal" and "using the shell".

The shell is the UNIX program that you interact with when in a terminal window interacting with a UNIX-style operating system (e.g., Linux or MacOS). The shell sits between you and the operating system and provides useful commands and functionality. Basically, the shell is a program that serves to run other commands for you and show you the results. There are actually different shells that you can use, of which `bash` is very common and is the default on many systems. In recent versions of MacOS, `zsh` is the default shell. zsh is an extension of bash, so you should be able to use zsh based on this tutorial.

I've generated this document based on using the bash shell on a computer running the Ubuntu Linux version 22.04 operating system, but you should be able to replicate most of the steps in this tutorial in other UNIX command line environments, ideally using the bash or zsh shells. 

## 1.3 Accessing a UNIX command line interface

Here are some options for accessing a UNIX command line interface:

  - MacOS: If you'd like to work on your own Mac, you'll find the Terminal under
      `Applications -> Utilities -> Terminal`.
  - Windows:
      - If you have a sufficiently new version of Windows 10, you can use the [Windows Subsystem for Linux](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide), which will provide you with an Ubuntu shell running bash on your own machine.
      - If you have access to remote machines running Linux, you can login to them using programs such as MobaXTerm and Putty. Once logged in, you'll find yourself in a Terminal window on the remote machine.
  - JupyterHub: If you have access to a JupyterHub, you will likely be able to start a Terminal session under "New".
  - Cloud-based options: You could also try a cloud service such as Google [Cloud Shell](https://shell.cloud.google.com).

:::{.callout-warning title="Don't use Git Bash for this tutorial"}
You probably shouldn't use `Git Bash` to follow this tutorial as its functionality is limited. 
:::

Once you're in a Terminal window, you'll be interacting with the shell and you can enter commands to get information and work with the system. Commands often have optional arguments (flags) that are specified with a minus in front of them, as we'll see. 

## 1.4 Getting started

Once we are in a terminal, we'll see the "prompt", which indicates that the shell is waiting for us to enter commands. Sometimes the prompt is just `$`:

:::{.cell-output}
$
:::

but often it contains information about the username of the current user and the directory on the filesystem that we are in. For example, here a prompt that shows that the current user is 'scflocal', on the machine named 'gandalf' in the 'tutorial-unix-basics' (sub)directory in the user's home directory (indicated by `~`):

:::{.cell-output}
scflocal@gandalf:~/tutorial-unix-basics>
:::

:::{.callout-note title="Tutorial code formatting"}
In the remainder of this tutorial, you won't see the prompt in front of the commands. All commands will appear in a grey background, with the output (if any) following the code.
:::


:::{.callout-note title="When the shell is waiting for more information"}
Note that if you simply see `>` instead of the usual prompt, that means the shell thinks you haven't finished entering your command (usually that you haven't finished entering a string) and is expecting more input from you. If you see a newline but nothing else, the shell probably expects you to enter some text for it to process.

If you're not sure what to do, type `Ctrl-c` (the control key and 'c' at the same time) to get back to the usual prompt.
:::

Let's start by running a command, `whoami`, that prints out the username of the current user:
```{bash}
whoami
```



# 2 Using `git` for version control

We'll discuss `git` briefly, both because it is an important and useful tool, and because it's the easiest way for us to get a set of files to work with in this tutorial. 

Git is an important tool to become familiar with, at least at the basic level. Git allows you to share files between different computers and different people and for people to collaborate on projects together. In particular, it is a version control tool that allows you to have different versions of your files and to go back to earlier versions of your files. Git stores the files for a project in a *repository*. 

For our purposes here, we'll simply use Git to download materials from GitHub, a website that stores Git repositories in the cloud. 

First we'll download the materials for this tutorial.

To *clone* (i.e., copy) a repository (in this case from GitHub) we do the following. Note that `berkeley-scf` is the *organization* and `tutorial-unix-basics` is the *repository*. Note that everything below that follows the `#` symbol is a comment and not executed.

Here we'll first use the `cd` command (for "change directory") to go to our home directory and then use `git clone` to download materials to a subdirectory (which will be called `tutorial-unix-basics`) within our home directory. 

```{bash}
#| eval: false
cd
git clone https://github.com/berkeley-scf/tutorial-unix-basics
```
::: {.cell-output}
```
Cloning into 'tutorial-unix-basics'...
remote: Enumerating objects: 387, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 387 (delta 37), reused 46 (delta 19), pack-reused 321 (from 1)
Receiving objects: 100% (387/387), 779.53 KiB | 5.27 MiB/s, done.
Resolving deltas: 100% (199/199), done.
```
:::

Now suppose that whoever controls the repository makes some changes to the materials in the repository online and you want an updated copy of the repository on your computer. Simply use `cd` to go into any directory in the repository materials on your computer and run `git pull`.

```{bash}
#| eval: false
cd tutorial-unix-basics
```

```{bash}
git pull
```

In this case, since no changes had been made, git simply reports that things are up-to-date.

We'll discuss how to use `cd` in more detail in the next section.

# 3 Files and directories

## 3.1 Moving around and listing information

We'll start by thinking about the filesystem, which organizes our information/data into files on the computer's disk.

Anytime you are at the UNIX command line, you have a *working directory*, which is your current location in the file system. 

Here's how you can see where you are using the `pwd` ("print working directory") command:

```{bash}
pwd
```

and here's how you use `ls` to list the files (and subdirectories) in the working directory...

```{bash}
ls
```

Now suppose I want to be in a different directory so I can see what is there or do things to the files in that directory.

The command you need is `cd` and an important concept you need to become familiar with is the notion of 'relative' versus 'absolute' *path*. A path is the set of nested directories that specify a location of interest on the filesystem.

First let's go to our home directory, which is generally where our files will be. Simply running `cd` will do that.

```{bash}
cd
pwd
```


Now let's go into a subdirectory. We can use `cd` with the name of the subdirectory. The subdirectory is found 'relative' to our working directory, i.e., found from where we currently are.

```{bash}
#| eval: false
cd tutorial-unix-basics
pwd
```

:::{.cell-output}
/home/scflocal/tutorial-unix-basics
:::

We could also navigate through nested subdirectories. For example, after going back to our home directory, let's go to the `assets` subdirectory of the `tutorial-unix-basics` subdirectory. The `/` is a separate character that distinguishes the nested subdirectories.

```{bash}
cd
cd tutorial-unix-basics/assets
pwd
```


You can access the parent directory of any directory using `..`:

```{bash}
#| eval: false
pwd
cd ..
pwd
```
:::{.cell-output}
/home/scflocal/tutorial-unix-basics/assets
/home/scflocal/tutorial-unix-basics
:::


We can get more complicated in our use of `..` with relative paths. Here we'll go up a directory and then down to a different subdirectory.

```{bash}
cd assets
cd ../_includes 
pwd
```


And here we'll go up two directories and then down to another subdirectory.

```{bash}
#| eval: false
cd ../../Desktop  # go up two directories and down
pwd
```

:::{.cell-output}

/home/scflocal/Desktop
:::

All of the above examples used relative paths to navigate based on your working directory at the moment you ran the command.

We can instead use absolute paths so that it doesn't matter where we are when we run the command. Specifying an absolute path is done by having your path start with `/`, such as `/home/scflocal`. If the path doesn't start with `/` then it is interpreted as being a relative path, relative to your working directory. Here we'll go to the `units` subdirectory again, but this time using an absolute path. 

```{bash}
cd /home/scflocal/tutorial-unix-basics/assets
pwd
```

:::{.callout-warning title="Absolute paths are not robust"}
Note that using absolute paths in scripts is generally a bad idea because the script wouldn't generally work correctly if run on a different machine (which will generally have a different filesystem structure) or as a different user (who will have a different home directory).
:::

## 3.2 The filesystem

The filesystem is basically a upside-down tree.

For example, if we just consider the `tutorial-unix-basics` directory, we can see the tree structure using `tree`:

```{bash}
tree
```

The dot (`.`) means "this directory", so the top of the tree here is the `tutorial-unix-basics` directory itself, within which there are subdirectories, `asset`, `_includes`, `_layouts`, etc. Then within each of these are files and further subdirectories (as seen in the case of `assets`, which has subdirectories named `css` and `fonts`.)

If we consider the entire filesystem, the top, or root of the tree, is the `/` directory. Within `/` there are subdirectories, such as `/home` (which contains users' home directories where all of the files owned by a user are stored) and `/bin` (containing UNIX programs, aka 'binaries'). We'll use `ls` again, this time telling it the directory to operate on:

```{bash}
ls /
```


If there is a user named `scflocal`, everything specific to that user would be stored in the user's *home directory*. Here that is `/home/scflocal`, but the exact location may differ on different systems. The shortcut `~scflocal` refers to the `scflocal` home directory, `/home/scflocal`. If you are the `scflocal` user, you can also refer to your home directory by the shortcut `~`.


```{bash}
ls /home
```

```{bash}
cd /home/scflocal
pwd
```

Go to the home directory of the current user (which happens to be the `scflocal` user):


```{bash}
cd ~
pwd
```


Go to the home directory of the scflocal user explicitly:

```{bash}
cd ~scflocal
pwd
```


Another useful directory is `/tmp`, which is a good place to put temporary files that you only need briefly and don't need to save. These will disappear when a machine is rebooted. 

```{bash}
cd /tmp
ls
```


We can return to the most recent directory we were in like this:

```{bash}
#| eval: false
cd -
pwd
```

:::{.cell-output}
/home/scflocal
:::

# 4 Using commands

## 4.1 Overview

Let's look more at various ways to use commands. We just saw the `ls` command. Here's one way we can modify the behavior of the command by passing a command option. Here the `-F` option (also called a 'flag') shows directories by appending `/` to anything that is a directory (rather than a file) and a `*` to anything that is an executable (i.e., a program).

```{bash}
ls -F
```


Next we'll use multiple options to the `ls` command. `-l` shows extended information about files/directories. `-t` shows files/directories in order of the time at which they were last modified and `-r` shows in reverse order. Before I run `ls`, I'll create an empty file using the `touch` command. Given this, what file do you expect to be displayed last when you do the following?

```{bash}
touch myfile
ls -lrt
```

While each command has its own syntax, there are some rules usually
followed. Generally, executing a command consists of four things:

  - the command
  - command option(s)
  - argument(s) 
  - line acceptance
  
Here's an example:

```{bash}
wc -l example.txt
```

In the above example, `wc` is the command, `-l` is a command option
specifying to count the number of lines, `example.txt` is the argument, and the
line acceptance is indicated by hitting the `Enter` key at the end of
the line.

So that invocation counts the number of lines in the file named `example.txt`.

The spaces are required and distinguish the different parts of the invocation. For this reason,
it's generally a bad idea to have spaces within file names on a UNIX system. But if you do, you can
use quotation marks to distinguish the file name, e.g.,

```{bash}
echo "some text" > "filename with spaces.txt"
ls -l "filename with spaces.txt"
```

Also, capitalization matters. For example `-l` and `-L` are different options.

Note that options, arguments, or both might not be included in some cases. Recall that we've used `ls`
without either options or arguments.

Arguments are usually one or more files or directories.

## 4.2 Options

Often we can specify an option either in short form (as with `-l` here)
or long form (`--lines` here), as seen in the following equivalent invocations:

```{bash}
wc -l example.txt
wc --lines example.txt
```


We can also ask for the number of characters with the `-m` option, which can
be combined with the `-l` option equivalently in two ways:

```{bash}
wc -lm example.txt
wc -l -m example.txt
```


Options will often take values, e.g., if we want to get the first two lines of the file,
the following invocations are equivalent:

```{bash}
head -n 2 example.txt
head --lines=2 example.txt
head --lines 2 example.txt
```


## 4.3 Comments

Anything that follows `#` is a comment and is ignored.

```{bash}
# This is ignored
ls  # Everything after the # is ignored
```

## 4.4 Getting help with UNIX commands

Essentially all UNIX commands have  help information (called a man page), accessed using `man`. We won't show the results here as they are rather long.

```{bash}
#| eval: false
man ls
```

You should try it yourself to practice viewing man pages. Once you are in the man page, you can navigate by hitting the space bar (to scroll down) and the up and down arrows. You can search by typing `/`, typing the string you want to search for and hitting `<Enter>`. You can use `n` and `p` for the next and previous search hits and `q` to quit out of the search.

Unfortunately man pages are often quite long, hard to understand, and without examples. But the information you need is usually there if you take the time to look for it.

Also, UNIX commands as well as other programs run from the command line often provide help information via the `--help` option:

```{bash}
#| eval: false
ls --help
```

Again, we're not showing the output as it is rather long.

## 4.5 Seeing if a command or program is available

You can see if a command or program is installed (and where it is installed) using `type`.

```{bash}
type grep
type R
type python
```

# 5 Working with files

## 5.1 Copying and removing files

You'll often want to make a copy of a file, move it between directories, or remove it. 

```{bash}
cp 
cp example.txt example-new.txt
mv example-new.txt /tmp/.
cd /tmp
ls -lrt
```

When we moved the file, the use of `/.` in `/tmp/.` indicates we want to use the same name as the original file.


```{bash}
cd /tmp
rm example-new.txt
ls -lrt
```

:::{.callout-important title="`rm` is forever"}
I used `rm` above to remove the file. Be very careful about removing files - there is no Trash folder in UNIX - once a file is removed, it's gone for good.
:::

The `mv` command is also used if you want to rename a file. 

```{bash}
cd ~/tutorial-unix-basics
mv example.txt silly_example.txt
ls
```


We can copy and remove entire directories. The `-p` flag preserves the time stamp and other information associated with the files/directories, while the `-r` option copies recursively, such that the directory and all its contents (all child files and directories) are also copied.

```{bash}
cp -pr assets /tmp/.  # Copy the assets directory into /tmp.
cd /tmp
mkdir test
mv assets test     # Move the assets directory into the test directory.
ls -l test/assets
```

```{bash}
rm -rf /tmp/test/assets   # Remove the ps directory and anything contained within it.
ls /tmp/test              # This should be empty now.
```

You can use a variant of `cp` named `scp` to copy files between different UNIX-like machines. Suppose I have access to the machine *radagast.berkeley.edu* and that my user name on that machine is *scf1*. I can copy a file to that machine or from that machine as follows.

(Note that I am not running the code in the process of generating this document.)

```{bash}
#| eval: false
cd ~/tutorial-unix-basics

# FROM the machine you're on TO another machine
# Copy the file to the Desktop subdirectory of the scf1 home directory on the remote machine
scp example.txt username@machinename.berkeley.edu:~/Desktop/.

# FROM another machine TO the machine you're on
# Copy a file from the /tmp directory of the remote machine to a specific directory on this machine
scp username@machinename.berkeley.edu:/tmp/data.txt ~/Downloads/.
```


## 5.2 File names and extensions

The format a file is in is determined by the **actual content** of the file. You can determine the file format using `file`:

```{bash}
file index.qmd
file /usr/local/linux/miniforge-3.12/lib/python3.12/site-packages/numpy/dtypes.py
```


In many cases, files have extensions such as `.csv` (for comma-separated text files), `.pdf` for PDFs, `.jpg` for JPEG files. The extension is a convention that helps us and programs distinguish different kinds of files and therefore know how to manipulate/interpret the files.

:::{.callout-tip title="Filename extensions don't determine the file type" collapse="false"}
The extension is just a convention -- changing the file name doesn't change the file format!

So if make a copy of the `example.txt` file but name it `example.pdf`, we see that it's still just a simple text file even if I give it a name that would suggest it's a PDF.

```{bash}
cp silly_example.txt silly_example.pdf
cat silly_example.pdf
```

```{bash}
file silly_example.pdf
```

However, changing the extension may prevent a program from using the file simply because the program was written to assume that files in a certain format have a certain extension.

:::

# 6 Other useful tools and information

## 6.1 Compressing and uncompressing files

The `zip` utility compresses in a format compatible with zip files for Windows:

```{bash}
zip -r assets.zip assets
```

```{bash}
ls -l assets.zip
```

`gzip` is a standard UNIX compression utility to compress individual files:

```{bash}
cp assets/img/ls_format.png test.png
ls -l test.png
```

Here we see that gzip can't compress the png file much, but it can help
a lot with other formats.

```{bash}
gzip test.png
ls -l test.png.gz   # Not much smaller than the uncompressed file.
```



Finally, the `tar` utility will combine multiple files and directories into a single archive. 

```{bash}
tar -cvf assets.tar assets
```

```{bash}
ls -l assets.tar
```

Adding the `-z` flag also gzips the result. In that case there was more noticeable compression.

```{bash}
tar -cvzf assets.tgz assets
```


```{bash}
ls -l assets.tgz
```

Now let's move that *tarball* (as it is called) to a new directory and unzip and expand it using the -x flag. 

```{bash}
mv assets.tgz /tmp
cd /tmp
tar -xvzf assets.tgz
```


You can see the whole directory structure of what was archived has been recovered in the new location:
```{bash}
ls -l /tmp/assets
```


## 6.2 Disk usage

You can can see how much disk space is being used versus available as follows. The 'Mounted on' column will generally identify the parts of the filesystem in a more user-friendly way than the 'Filesystem' column.

```{bash}
df -h
```


In general, you'll want to look at the '/' line under `Mounted on`, and on standard UNIX machines possibly at '/tmp', '/home', and others.

We can see usage in specific directories like this:

```{bash}
cd assets
du -h
```


Here we see that the total usage is about a bit less than 700 KB, with, for example, about 70 KB of that in the `img` subdirectory.

If we only want a summary of usage for each top-level subdirectory, rather than showing all nested subdirectories:

```{bash}
cd ~/tutorial-unix-basics
du -h -d 1
```


## 6.3 Machine information

Linux machines (but not Macs) have system information provided in a few special files.

`/proc/cpuinfo` shows information on each processor. 

```{bash}
head -n 30 /proc/cpuinfo
```


This indicates there are at least two processors -- numbered 0 and 1 (we'd need to see the whole file to see if there are more). Each is an Intel Core i7. 


This file has information on the memory available:

```{bash}
head -n 10 /proc/meminfo
```


The key line is the *MemTotal* line, indicating 132 GB of RAM.

```{bash}
cat /etc/issue
```

We're running Ubuntu version 22.04.

We can also use commands to get information:

```{bash}
nproc  # how many processors?
```


# 7 The shell

The shell provides a number of useful shortcuts, of which we highlight a couple here.

## 7.1 Tab completion

The shell will try to auto-complete the names of commands/programs or of files when you type part of the name and then hit `<Tab>`. This can save quite a bit of typing, particularly for long file names. 


## 7.2 Keyboard shortcuts

You can navigate within a line using the usual arrows but also:

* `Ctrl-a` moves to the beginning of the line
* `Ctrl-e` moves to the end of the line
* `Ctrl-k` deletes the rest of the line starting at the cursor
* `Ctrl-y` pastes in whatever was deleted previously with `Ctrl-k`
* `Ctrl-r` enables an [interactive history search](http://www.techrepublic.com/article/keyboard-shortcuts-in-bash-and-zsh/)

## 7.3 Command history

The up and down arrow keys will move you through the history of commands you have entered in the terminal. So you can recover something you typed previously and then directly run it again, or edit it and then run the modified version. You run the command by pressing `<Enter>`, which you can do regardless of where your cursor currently is on the line you are editing.

There's also lots more functionality along these lines that we won't go into here.

## 7.4 Saving your code as a shell script

Often (particularly as you learn more sophisticated shell functionality) you will want to save your shell syntax in the form of a code file, called a script, that you could run another time.

For example, suppose you often need to do the following series of steps:
```{bash}
#| eval: false
cd 
tar -cvzf assets.tgz assets
mv assets.tgz /tmp
cd /tmp
tar -xvzf assets.tgz
```

You can put those lines into a file, say, `mv_assets.sh`, which will generally end in .sh.

Then we can run the code in the file as follows. (Results not shown here.)

```{bash}
#| eval: false
chmod ugo+x mv_assets.sh  # Make the script executable by everyone.
./mv_assets.sh            # Run it.
```

The initial `./` is needed because UNIX is not expecting there to be an executable file in this particular directory. 

You'll generally want to have the first line of your shell scripts indicate the shell to be used to execute the script, so you'd want to put `#!/bin/bash` as the first line of `mv_assets.sh`.

# 8 Practice questions

1) Try to run the following command `mkdir ~/projects/drought`. It will fail. Look in the help information on `mkdir` to figure out how to make it work without first creating the `projects` directory.

2) Figure out how to list out the files in a directory in order of decreasing file size, as a way to see easily what the big files are that are taking up the most space. Modify this command to get the result in the ascending order.

3) Use both `zip` and `tar -cvzf` to compress the `tutorial-unix-basics` directory. Is one much smaller than the other?

4) Figure out how to print out free disk space in terms of megabytes. 

5) The `ls` command is itself an executable installed on the system. Where is it located?

6) Where is `gzip` installed on the system? What are some other commands/executables that are installed in the same directory?

7) Practice with moving/removing/copying. Make a copy of the `tutorial-unix-basics` directory (and all its contents) in `/tmp`. Now use `cd` to go into the copied directory. Remove the `/tmp/tutorial-unix-basics/.git` directory. Now run `git status`. Congratulations, you should discover that you've turned a directory that is a Git repository into a directory that is not considered a git repository.

```{bash}
#| include: false
mv silly_example.txt example.txt
rm assets.{tar,zip} silly_example.pdf test.png.gz
```