m02-jupyter_and_bash_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to work in a Jupyter notebook with the Bash language\n",
    "The webpage you are looking at is called a Jupyter notebook. \n",
    "It is a webpage on which you can write text (like this text) and also code. \n",
    "You can execute the code in the webpage, and the output returns to you within the same notebook! \n",
    "This may sound trivial, but it's \"really cool\" to put it in non-scientific terms. \n",
    "Code and text are entered in individual cells, a code cell or a text cell.\n",
    "This cell you are reading now is a text cell.\n",
    "Next, let's look at a code cell and execute it.\n",
    "There are two ways of executing a cell.\n",
    "First you select a cell, either with your mouse or with the up and down keys on your keyboard.\n",
    "Then, you execute it by hitting the 'run' button in the toolbar above, or you hit CTRL+RETURN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Jupyter\n",
    "\n",
    "First we'll learn how to work with Jupyter notebooks, then we'll move on to learn the basics of Bash.\n",
    "\n",
    "Working with Cells in Jupyter is quite straightforward. \n",
    "You learn best by doing, so do all of the things listed below:\n",
    "\n",
    "* You can select a cell with your mouse or the arrow keys on your keyboard. \n",
    "* You can edit a cell by hitting RETURN or by double-clicking it. \n",
    "* A new cell is a code cell by default turn in into a markdown (text) cell by hitting 'm'\n",
    "* A code cell can be executed by hitting CTRL+RETURN.\n",
    "* A markdown cell can be rendered by hitting CTRL+RETURN.\n",
    "* Add an additional cell by hitting the '+' button in the toolbar.\n",
    "* Add an additional cell by clicking between two cells\n",
    "* Add an additional cell by using the keyboard\n",
    " + add a cell below by hitting the 'b' key\n",
    " + and above by hitting the 'a' key\n",
    "* Whenever your notebook turns out to be unresponsive, you may interrupt the underlying programme running the code: the kernel, by\n",
    " + hitting the square stop button in the toolbar\n",
    " + clicking 'restart' or 'interupt' in the 'kernel menu' in your menu bar.\n",
    " + clicking 'close and halt' in the File menu.\n",
    " \n",
    " \n",
    " Try creating a new cell below this cell. Make at least a text cell and a code cell.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jupyter kernels\n",
    "In the metagenomics practical, we will be working mostly in Linux' mother tongue: **BASH**. \n",
    "However, juPYter notebooks work natively in PYthon. \n",
    "Still, we can work in BASH. Often, this works by itself.\n",
    "Sometimes, you will need to type either %%bash at the beginning of a code cell.\n",
    "Or by preceding every bash command with an exclamation mark.\n",
    "Now, let's get to work and learn some Bash!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## bash basics\n",
    "execute the code cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo \"hello world\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the bash language, the first word you type is always the command. So, in this case, that was:\n",
    "> echo\n",
    "\n",
    "This command 'echoes' whatever you give it in the terminal. After you 'call' the command, you give it an argument in this case that is: \n",
    "> \"hello world\"\n",
    "\n",
    "This basic structure of 'command' 'arguments' comes back through the metagenomics practical. \n",
    "\n",
    "**do:** now try to change \"hello world\" to something else in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo \"hello world\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, an argument is a path to a file. \n",
    "We have the `ls` command to see what files we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are learning fast. \n",
    "So we now know what a `command` is, and we know what an `argument` is. \n",
    "Finally, we also need to know what options are. \n",
    "Options are optional extra information that we pass to the `command`.\n",
    "Options are often provided in between the `command` and the `argument`. \n",
    "They look either like this\n",
    "> ls --size --human-readable\n",
    "\n",
    "or in shortened versions like this\n",
    "> ls -sh\n",
    "\n",
    "Note that the above two commands are synonymous. Try out in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls --size --human-readable data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -sh ./data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Commands`, `options` and `arguments` are separated by spaces. \n",
    "Also note that `options` can have their own `arguments`. \n",
    "If this is the case, the manual or help page will specify this. \n",
    "We will get the manual and help pages later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## auto-complete\n",
    "Auto-complete is one of the best features of the `bash` language and your greatest friend during this practical. Let's say we want to list (`ls`) the contents of the `data/` folder but are too lazy to type the whole word 'data/'. Then we can type \n",
    "> ls da\n",
    "\n",
    "and then hit the TAB button on your keyboard. Bash should either automatically complete the path to\n",
    "> ls data/\n",
    "\n",
    "or if there are multiple options to auto-complete, `bash` will give you a little menu with these options.\n",
    "\n",
    "Using autocomplete does not only make your life a lot easier, but it also prevents you from making typos! \n",
    "If bash autocomplete doesn't work, odds are something in your command or argument is wrong. \n",
    "Best to check before you proceed!\n",
    "\n",
    "Try out auto-complete below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls da"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pipes\n",
    "Bash can hand the output of one program to another.\n",
    "This is called piping. If you pipe the output of multiple programs to each other, you made a 'pipeline'.\n",
    "Pipelines look somewhat like this\n",
    "\n",
    "> command1 | command2 | command 3\n",
    "\n",
    "One trick with pipes that we will often use is the `| head` pipe.\n",
    "This pipe shows you only the first ten lines of the output of some command.\n",
    "` | head -n 1` changes this number to 1. \n",
    "See for yourself below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -1 data/reads/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ls -1 data/reads/ | head -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## loops\n",
    "Loops are a useful feature you'll find in most (if not all) programming languages. \n",
    "They are also quite intuitive to use. \n",
    "A loop simply is a series of commands that does the same thing multiple times, but with some small adaptation. \n",
    "Let's make a loop together.\n",
    "First, we need to have two concepts clear\n",
    "\n",
    "* variable\n",
    "* array\n",
    "\n",
    "A **variable** is a specific word that means something else; this something may vary. Hence the name. We can specify a variable like this:\n",
    "> variable1=coffee\n",
    "\n",
    "To refer to the content of a variable, we use a `$` sign. So this looks like so\n",
    "> echo $variable1\n",
    "\n",
    "Now enter the cell below and try for yourself. You can name a variable anything you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "variable1=coffee\n",
    "echo $variable1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are working in a Bash kernel, your variables will be remembered in the entire notebook. \n",
    "If you work with a Python kernel, then your variables are only remembered in one cell.\n",
    "Check what kernel this notebook is running in the top right op this page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An array is a list of variables; it's that simple. To make an array, we type something like this\n",
    ">samples=(L1 L2 L3)\n",
    "\n",
    "To refer back to an array, we type this\n",
    ">echo ${samples[@]}\n",
    "\n",
    "This looks a bit more complicated. \n",
    "The `[@]` part means: 'all contents in the array'. \n",
    "Hence, if you type `echo ${samples[0]}`, you will only get the first variable in the array. \n",
    "Again, try for yourself below in a new cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we get to loops.\n",
    "Let's keep it simple, I will define a loop for you, and you see how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "break=(coffee tea cookies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for   i in ${break[@]}\n",
    "do    echo $i\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you get the loop? \n",
    "Make sure you do. \n",
    "You will make your loops in the following parts of the practical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you completed all exercises above? Then move on to this: \n",
    "\n",
    "## Bash basics extra\n",
    "\n",
    "* Wildcards*\n",
    "* Base filenames.\n",
    "* paths\n",
    "* manual /help pages\n",
    "\n",
    "\n",
    "### wildcards \n",
    "wildcards can be used in the command-line. \n",
    "For example: list every folder/file inside the `./data/ folder`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ./data/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or list every file in `./data/reads` that ends on `.gz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ./data/reads/*.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, list every file in `./data/reads` that starts with `L` and ends in `.gz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### base filenames\n",
    "The base of a file is the part before the extension or extensions; you will need this later in the practical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### paths\n",
    "As we have seen now, you can specify folders with a `/`. You can move from folder to folder. If you ever wonder what folder you are in now, you can 'print work directory' or `pwd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The current folder you are in is denoted as a dot: `.`  \n",
    "Hence, if you type a path like `./data/reads` you tell the computer explicitly to start in the current folder, then move to the data folder, and then move to the reads folder.\n",
    "This dot is not required, `data/reads`  means the exact same. \n",
    "Try in the two cells below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you type `ls /`, then you ask the computer to list the root of the filesystem, the highest level on the hard drive. Somewhat like `C:/` on windows computers.\n",
    "\n",
    "Whenever you see a manual page or a prewritten command with code like this\n",
    "> somecommand /path/to/file\n",
    "\n",
    "Then it is implied that you substitute the `/path/to/file` with a path to a file you want to use or create."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get errors like `file not found`, perhaps check where you are with `pwd` and `ls` to see if you accidentally moved somewhere you did not mean to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### change your working directory.\n",
    "Although you won't need to in this practical, you can change working directories.\n",
    "I designed this practical not to bother you with this, but in the long run it is important to be aware of where you are in a directory structure.\n",
    "You can Change Directory with the command `cd`. \n",
    "For example, if you want to move into a data folder, you may type `cd ./data`. \n",
    "If you want to move back up again, you type `cd ..`. \n",
    "Where `..` means one folder up, you can also use this with `ls` like `ls ..`. \n",
    "If you are completely lost, just type `cd` without any argument; this will take you back to your home directory.\n",
    "If you inadvertently move directories and you need to get back, you now know how to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### help and manual pages\n",
    "\n",
    "Whenever you are asked to use some command or programme, and you don't know exactly how it works, we can ask the computer for help.\n",
    "* type the command without any argument or options\n",
    "* type the command with option `--help`\n",
    "* type the command with option `-h`\n",
    "* get the manual page `man some-command`\n",
    "\n",
    "Not all of these always work for every command, but one or two always do; trial and error.\n",
    "\n",
    "On these webpages, the `man` command doesn't work too well. Better to stick to the `--help` pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite often, you'll find a 'usage' line at the top of the help page. \n",
    "This tells you how to use the command. \n",
    "In the example of `head`, it tells you first to type `head`, then any options, and then any file. \n",
    "Those entries in \\[square brackets\\] are optional. \n",
    "Entries without any brackets, or with <arrows\\> are required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## That's it!\n",
    "You are now ready to work with Bash in Jupyter notebooks! \n",
    "Congratulations. \n",
    "Whenever you get stuck in the subsequent notebooks that use Bash code, maybe come back here for advice."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}