m08-binning-part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binning part 2: create bins with `metabat2`\n",
    "\n",
    "Now, we continue the binning procedure with `metabat2`.\n",
    "In this notebook, we will create the actual bins! \n",
    "We will need:\n",
    "* The scaffolds of the assembly, to bin.\n",
    "* The depth matrix we made with the jgi script in binning part1.\n",
    "* A new folder, to store the newly created bins.\n",
    "\n",
    "First, remember where the first two items listed above are. \n",
    "Use `ls` to confirm in the cell below.\n",
    "\n",
    "**[DO:] Locate the scaffolds file:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Locate the depth matrix:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Make a new directory to store your bins.** \n",
    "Make sure this directory is `data/bins`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Read the help page of** `metabat2`\n",
    "\n",
    "Find out which options you have to use minimally, then make sure you tell MetaBAT to use one thread only!\n",
    "\n",
    "Supply your depth matrix as the --abdFile. Short for AbundanceFile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metabat2 -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Bin your scaffolds with** `metabat2` **:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metabat2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] How many bins were created?** \n",
    "Check the directory where you stored your bins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Q:] Did you get more or less bins than expected from the length/depth plot you made earlier?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## visualisation\n",
    "Now we have our metagenome binned!\n",
    "Congratulations.\n",
    "Let's try to visualise this similarly as done in Binning part 1. \n",
    "We will use some command line tricks to get all data in a similar sheet.\n",
    "In this case, these are given to you already.\n",
    "If you feel up to the challenge, try to reverse engineer the code and understand what is happening.\n",
    "\n",
    "**[DO:] Run the code below to make** `data/binlist.txt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, we make an empty file with a header in which we will make our table. \n",
    "echo -e 'bin\\tcontigName' > binlist\n",
    "# Then we move to the folder in which we made the bins. \n",
    "cd ./data/bins/\n",
    "# Now, we start a loop for each file that ends with `.fa`\n",
    "for f in *.fa\n",
    "do  # For each `.fa` file, we extract the bin number and make a variable that we call `name`\n",
    "    name=$(echo $f | cut -d '.' -f 2)\n",
    "    # Continuing in this iteration of the loop, we filter all fasta headers\n",
    "    grep '>' $f | sed \"s/^>/$name\\t/g\"\n",
    "done | sort -k2 -V >> ../binlist.txt\n",
    "# directly after filtering, replace the fasta header sign '>' with the `name` variable defined earlier.\n",
    "# we end the loop and sort all resulting tables at once on the second column \n",
    "# after sorting, we append our newly made table to the 'binlist' file we defined earlier.\n",
    "cd ../../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Check what the file looks like:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head data/binlist.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] compare these with the names we have in our depth_matrix.txt.**\n",
    "They must be exactly the same to join these two different tables into one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cut -f 1  <<your original depth matrix>> | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use the `join` command to join the two tables.\n",
    "There must be a shared field in both tables.\n",
    "In the first table, this is the second column `-1 2`, and in the second file, this is the first column `-2 1`. \n",
    "\n",
    "We then take both files and give them to the join command.\n",
    "However, since `join` is very picky in how files are sorted, we re-sort them on-the-fly like so ` <(sort -k2d ./somefile.txt)` (second column, sort as dictionary).\n",
    "\n",
    "Lastly, since both files have headers, we supply the `--header` option and save the result as a new file.\n",
    "\n",
    "Notice that I use the `\\` character to spread out this very long `join` commandline over several lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] fill in the path to your depth matrix below and run the code**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "join -1 2 -2 1                                         \\\n",
    "     <(sort -k2d ./data/binlist)                       \\\n",
    "     <(sort -k1d <<...your original depth matrix...>>) \\\n",
    "     --header                                          \\\n",
    "     | tr ' ' \"\\t\"                                     \\\n",
    "     > ./data/binned_depth_matrix.tab "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[DO:] Visualise the binned_depth_matrix.tab in excel**\n",
    "1. Like we did before, download this resulting table `binned_depth_matrix.tab` and open it in excel.\n",
    "2. Sort the file by the bin number. \n",
    "3. Erase columns you do not want to visualise\n",
    "4. Use conditional formatting to visualise depth profiles over the different samples per bin. \n",
    "\n",
    "**[Q:] Does this colour pattern make more sense than it did before?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Q:] Are there any outliers or mistakes you can spot?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bin depth\n",
    "**[Q:]Can you determine the depth of the six bins from this table? Why is this not taking the mean of all depths?**\n",
    "Think about the difference between depth and coverage.\n",
    "You will need to make a pivot table in excel/LibreOffice/googledrive.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Q:] Can you determine the depth of each bin per sample(type)?**\n",
    "In other words, which bin is abundant in the L samples, and which bin is abundant in the P samples.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Q:] The research this practical is based on focusses on microbes inside the leaves (L samples).\n",
    "Which bins would you advise me to study further?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[A:]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: vary binning signals\n",
    "\n",
    "If you'd like, you can try to vary input signals. \n",
    "For each variation, make sure you save the bins in a separate directory with a clear name.\n",
    "* First, you can try to run metabat2 without the depth matrix. How many bins do you get then?\n",
    "* You can edit the depth matrix to only contain samples from one type (E or P)\n",
    "* You can edit the depth matrix to only contain one replicate per sample type, or two replicates?\n",
    "\n",
    "To modify your depth matrix, have a look at the collumns present:\n",
    "```\n",
    "head -n 3 <<your depth matrix>>\n",
    "```\n",
    "\n",
    "You can select certain columns with the cut command.\n",
    "This example shows you how to select only one replicate of one sample type and save this as a separate depth matrix.\n",
    "\n",
    "```\n",
    "cut -f 1-3,10,11 data/depth_matrix > data/depth_matrix_P1\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}