notebooks/03_reading_writing_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 - Reading and writing files  \n",
    "--------------------------------------------------------\n",
    "\n",
    "## Table of Content <a id='toc'></a>\n",
    "\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Introduction**](#1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Where is my file](#1.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[File opening modes](#1.2)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Reading from files**](#3)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading lines manually](#3.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[End-of-line characters](#3.2)  \n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Writing to files**](#4)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Exercises 3.1 and 3.2**](#5)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Additional Material**](#6)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading a file's entire content at once](#6.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Easier reading of .csv formatted file](#6.2)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Opening files without context managers](#6.3)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading files using a while loop](#6.4)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Some new cool syntax for Python >= 3.8](#6.5)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[What does the `flush` argument of `print()` do?](#6.6)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "\n",
    "## Introduction <a id='1'></a>\n",
    "-------------------\n",
    "\n",
    "In many use cases, you will want your python code to **read/write from/to files** stored on your local hard drive.\n",
    "\n",
    "### Code design aspects\n",
    "\n",
    "Here are a few important points to consider when working with files:\n",
    "\n",
    "* Where is my file?\n",
    "* Do I need to read the entire dataset/file into memory?\n",
    "    * Accessing the hard drive is among the slower operations a computer performs. \n",
    "      Reading an entire file when you only need the first few lines will be costly\n",
    "    * If you are reading a very large file, having the entire file in memory at \n",
    "      once may overburden your computer's memory (RAM).\n",
    "* Are there concurrency issues?\n",
    "    * If another software (or even your own code if you have messed up) writes to a file you \n",
    "      are currently reading, you could run into trouble.\n",
    "\n",
    "\n",
    "### Practical aspects\n",
    "\n",
    "* Whether it is for reading or for writing, operations on files occur using **file objects**\n",
    "  (sometimes also referred-to as **file handles**).\n",
    "* In modern python, files are opened with a so-called *context manager* statement, which takes\n",
    "  care of properly opening and closing the file once you are done with it.\n",
    "* This context manager takes the form of a **`with open(...) as ...`** statement. Note that context\n",
    "  managers are code blocks, and their content must therefore be indented:\n",
    "\n",
    "    ```python\n",
    "    # Open a file in a given \"mode\" (e.g. read, write or append).\n",
    "    with open(filename, mode) as file_handle:\n",
    "\n",
    "        # Do something with the file... (note the indentation)\n",
    "        # ...\n",
    "\n",
    "    # When you are outside of the context manager block, the file is automatically closed.\n",
    "\n",
    "    ```\n",
    "\n",
    "<br>\n",
    "\n",
    "### `filename` argument: where is my file located ?  <a id='1.1'></a>\n",
    "\n",
    "The first argument that `open()` takes is the **file name**, which **may include the file path** (relative or absolute) if the file is not located in the current working directory.\n",
    "\n",
    "To pass the correct value to `filename`, you will need to know:\n",
    " 1. where your file is,\n",
    " 2. and where you *code is executing from* (i.e. what is the working directory when you code runs)\n",
    " \n",
    "We leave question 1. to you, but with Jupyter Notebook, the second question has an easy answer: your code is executed where the jupyter notebook is saved (with a more classical python script, the code executes from the directory where you called `python`).\n",
    "\n",
    "Now, we need to make sure that our code can find the file:\n",
    "* **If the file is the same folder as the code**, then you can just use the name of the file,\n",
    "  no need for further modification.\n",
    "* **If the file is elsewhere**, you will have to specify a path to the file.\n",
    "  This can be either a:\n",
    " \n",
    "    * **Absolute path:** from the root of the computer to the file. e.g.,\n",
    "      - `\"C:\\Users\\JohnDoe\\Desktop\\ProjectP\\data\\myFile.txt\"` (Windows)\n",
    "      - `\"/home/JaneDoe/Documents/ProjectP/data/myFile.txt\"` (Linux,Mac)\n",
    "    \n",
    "    * **Relative path:** from your code to the file. e.g.,\n",
    "      - `\"data/myFile.txt\"` - here the file is in a subdirectory named `data`.\n",
    "      - `\"../otherProject/myFile.txt\"` - more complex, the file is in a subdirectory of the parent directory.\n",
    "      - *Note:* `..` indicates the parent directory of the current directory.\n",
    "\n",
    "<br>\n",
    "\n",
    "The last case - `\"../otherProject/myFile.txt\"` - depicts a situation like this:\n",
    "```\n",
    "parentFolder:\n",
    " | \n",
    " ├─ ProjectA\n",
    " |     └─ myCode.ipynb\n",
    " |\n",
    " └─ otherProject\n",
    "       └─ myFile.txt\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "### `mode` argument: the file opening mode  <a id='1.2'></a>\n",
    "\n",
    "When using the `open()` function, the optional **`mode`** argument can be passed to the function. This specifies the type of access you will have on the file. For instance, the `\"r\"` mode will only allow to read the content of a file, and not to write to it (this is useful to avoid accidental writing to the file).\n",
    "\n",
    "There are several possible modes when opening files:\n",
    "* **`\"r\"`**: open file in read-only mode. This is the **default value** of the `mode` argument.\n",
    "* **`\"w\"`**: open file in write-only mode, **overwriting** an existing file with the same name.\n",
    "* **`\"a\"`**: open file in write-only mode, **appending** to an existing file with the same name\n",
    "  (otherwise the file is created).\n",
    "* **`\"rb\"`**, **`\"wb\"`**, **`\"ab\"`**: same as `\"r\"`, `\"w\"` and `\"a\"`, but reading/writing to/from binary files (such as `.zip` or `.bmp` image files). The content is read/written as bytes objects without any decoding.\n",
    "\n",
    "See `help(open)` or the [python online documentation](https://docs.python.org/3/library/functions.html#open) for a full list of modes and details about them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "\n",
    "## Reading from files  <a id='3'></a>\n",
    "--------------------------\n",
    "\n",
    "To start reading a file, one creates a **file object** using `open()` function with `mode='r'`.\n",
    "\n",
    "```python\n",
    "with open(\"path/to/file\", mode=\"r\") as file_handle:\n",
    "    ...\n",
    "```\n",
    "\n",
    "<br>\n",
    "\n",
    "### Reading lines manually <a id='3.1'></a>\n",
    "\n",
    "Now that we have a **file object** (file handle), we can use it to read from the file.  \n",
    "When reading a file with python, you can consider your file object a bit like a cursor which starts at the beginning of your file, and progresses toward the end of the file (it can go backward, but it is often a bit hacky to do so).\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src=\"img/file_pointer2.png\" alt=\"a file pointer at opening\" style=\"height:200px;\" />\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "You can read elements (ie. make the cursor advance) using the following methods:\n",
    " * **`.readline()`**: the most common, reads a single line.\n",
    " * **`.read()`**: reads the remainder of the file (from the current cursor position) in one go.\n",
    " * **`.readlines()`**: reads the remainder of the file (from the current cursor position) in one go,\n",
    "   and returns a `list` of `str` (string), where each element of the list is a line of the file that\n",
    "   was read: `[\"line 1...\", \"line 2...\", \"line 3...\", ...]`.\n",
    "\n",
    "> All these methods return the text they read as `str` (string objects). This means that if you read\n",
    "  a number and want to use it as a `float` (e.g. to do math with it), you will need to convert it\n",
    "  from `str` to `float` yourself (with `float(x)`).\n",
    "\n",
    "<br>\n",
    "\n",
    "Let's focus on **`.readline()`**, the most used method. When we call it, the file pointer progresses to the next line and returns the line that was just read as a `str` object.\n",
    "Note how the returned strings contain the **end of line character `\\n`**.\n",
    "\n",
    "<img src=\"img/file_pointer3.png\" alt=\"a file pointer at opening\" style=\"height:200px;\" />\n",
    "\n",
    "Each time we call `.readline()`, a new line is returned and the pointer progresses to the next line of the file.\n",
    "\n",
    "<img src=\"img/file_pointer4.png\" alt=\"a file pointer at opening\" style=\"height:200px;\" />\n",
    "\n",
    "Once there are no more lines to read, `.readline()` **returns an empty string (`\"\"`)**. This is how we know that the end of the file was reached.\n",
    "\n",
    "<img src=\"img/file_pointer5.png\" alt=\"a file pointer at the end of the file\" style=\"height:200px;\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "**Let's see an example with actual code:**\n",
    "\n",
    "Here we are going to read the file `data/fresh_fruits.txt`, which has the following content:\n",
    "\n",
    "```\n",
    "passionfruit\n",
    "oranges\n",
    "apples\n",
    "grapefruit (whole and segments)\n",
    "pointed sticks\n",
    "```\n",
    "\n",
    "<br>\n",
    "\n",
    "* Open the file, and read a single line from it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\" , mode=\"r\") as reading_handle:\n",
    "    \n",
    "    # Read and print the first line in the file.\n",
    "    line = reading_handle.readline()  # Reads a single line from the file.\n",
    "    print(\"line 1:\", line)            # Print the line to the screen.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "* Open the file, and read all of its content, line by line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\" , mode=\"r\") as reading_handle:\n",
    "    \n",
    "    # Read and print the first line in the file.\n",
    "    line = reading_handle.readline()  # Reads a single line from the file.\n",
    "    print(\"line 1:\", line)            # Print the line to the screen.\n",
    "    \n",
    "    # Keep reading and printing lines.\n",
    "    # Problem: how many time should I do this ?\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 2:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 3:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 4:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 5:\", line)\n",
    "    \n",
    "    # There are only 5 lines in this file.\n",
    "    # Once there are no more lines to read, .readline() returns an empty string.\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 6:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 7:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 8:\", line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "Luckily for us, file objects are **iterable**, and we can thus use a **`for` loop** to read through files:\n",
    "* Each iteration reads 1 line.\n",
    "* The `for` loop ends when the entire file was read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\" , mode=\"r\") as reading_handle:\n",
    "    for line in reading_handle:\n",
    "        print(line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### End-of-line characters  <a id='3.2'></a>\n",
    "As you can see in the example above, there are additional empty lines in between our prints. This is because the lines are read from the file with their **end-of-line** characters, which generally is `\\n` .  \n",
    "\n",
    "To avoid this issue, one typically uses the **`.strip()`** method of string, which removes any whitespace or end-of-line character at the start and end of the string.\n",
    "\n",
    "**Example:** using `.strip()` when reading content from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", 'r') as reading_handle:\n",
    "    \n",
    "    # Reminder: \"enumerate()\" is our friend that automatically enumerates items\n",
    "    # and creates tuples of the form \"(index, element)\".\n",
    "    for i, line in enumerate(reading_handle):\n",
    "        print(\"line\", i, \":\", line.strip())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    \n",
    "### Micro Exercise 1 - reading a file with a for loop\n",
    "\n",
    "* Read the content of `data/titanic_head.csv` and print it. Make sure that no white space is printed between lines.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "## Writing to files  <a id='4'></a>\n",
    "----------------------\n",
    "\n",
    "Writing to a file is achieved in pretty much the same way as reading from it, but the opening mode is now **`\"w\"`**.  \n",
    "And instead of reading lines, we now `print()` them to the file.\n",
    "\n",
    "* To print to a file - instead of standard-output - we will need to use the optional `file` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "help(print)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note*: for people wondering what the `flush` argument does, please see the additional material  \n",
    "> Spoiler alert - it's a fairly minor argument, only useful in some edge cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"shopping_list.txt\", mode=\"w\") as f:\n",
    "    print(\"onion\", file=f)\n",
    "    print(34, \"potato\", file=f)\n",
    "    print(\"shrubbery\", file=f)\n",
    "    print(\"tomato sauce\", file=f)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing the file object (file handle) to the **`file` argument** of the `print()` function, we now print to the file rather than to our terminal.\n",
    "> **Reminder**: the **`\"w\"`** mode **overwrites** the opened file - if you use it on an existing file,\n",
    "> its original content of the file is lost.\n",
    ">\n",
    "> **Pro tip:** you can open more than one file using a single `with` statement by using multiple\n",
    "> context managers in the same code block:\n",
    "> * Either on the same line:\n",
    ">      \n",
    ">  ```python\n",
    ">     with open(\"input.txt\", 'r') as in_file, open(\"output.txt\", 'w') as out_file:\n",
    ">         do_something()\n",
    ">  ```\n",
    ">  \n",
    "> * Or by having one context manager per line, and enclosing them all in parentheses `()`\n",
    ">   as shown below (better for readability) - *note: this only works with python >= 3.10* :\n",
    ">  ```python\n",
    ">    with (\n",
    ">        open(\"input.txt\", 'r') as in_file,\n",
    ">        open(\"output.txt\", 'w') as out_file,\n",
    ">    ):\n",
    ">        do_something()\n",
    ">  ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "#### Additional material\n",
    "\n",
    "You might sometimes see some Python code - especially older one - that uses the **`.write()`** method of the **file object** to write/print content to a file.\n",
    "\n",
    "* There are some differences between the `print()` method and `.write()`; the most important being\n",
    "  that `.write()` does not do any formatting and even the end-of-line (carriage return) characters\n",
    "  need to be manually written.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"shopping_list2.txt\", mode=\"w\") as f:\n",
    "    f.write(\"onion\\n\")\n",
    "    f.write(\"{} potato\\n\".format(34))\n",
    "    f.write(\"shrubbery\\n\")\n",
    "    f.write(\"tomato sauce\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "### Micro Exercise 2 - copy a file's content\n",
    "\n",
    "* Write some code to read, print and copy the content of the `shopping_list.txt` file we just created.  \n",
    "  Specifically, your code should:\n",
    "  * Print the content of `shopping_list.txt` to screen.\n",
    "  * Make a copy of the content in a new file `shopping_list_copy.txt`.\n",
    "  * Make sure that no white space is printed between lines.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "## Exercises 3.1 and 3.2   <a id='5'></a>\n",
    "------------------------------\n",
    "\n",
    "* Exercises are found in a separate Jupyter Notebook.\n",
    "* If you have time, feel free to try the **additional exercises**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "# Additional Material  <a id='6'></a>\n",
    "-------------------------------------\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### `readlines()` - reading a file's entire content at once <a id=\"6.1\"></a>\n",
    "\n",
    "Sometimes it is necessary to load the entire content of a file into memory at once. One way to achieve this is using the `.readlines()` method (note the plural marker \"s\" in the method's name).\n",
    "\n",
    "As its name suggests, `readlines()` reads more than one line at a time (by default, all lines in the file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", mode=\"r\") as f:\n",
    "\n",
    "    entire_file = f.readlines()                      # The whole content of the file is now in memory.\n",
    "    print(f.name, \"has\", len(entire_file), \"lines:\") # We can check how many lines there are\n",
    "                                                     # before we start looping over them.\n",
    "    for i, line in enumerate(entire_file):\n",
    "        print(\"line\", i, \":\", line.strip())\n",
    "\n",
    "print(entire_file)\n",
    "print(\"The file has\", len(entire_file), \"lines.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** while our examples both `readline()` and `readlines()` work equally well, there can be important implications in using one or the other, especially when dealing with large files. Can you think of a drawback of using `readlines()`?\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "**Answer:** <font color='white'>using readlines() will (by default) load the entire file in memory, and this can be problematic when working with large files as is often the case in bioinformatics. Always consider the file sizes you are dealing with when using readlines().</font> (select to reveal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Easier reading of .csv formatted file using modules  <a id='6.2'></a>\n",
    "\n",
    "`csv` (**C**omma **S**eparated **V**alue) is one of the most common file format when it comes to storing tabular data. In this format, each line contain a fixed number of values (columns), separated by a specific character (typically `\",\"`).\n",
    "\n",
    "\n",
    "Classically, when reading these files, we want to create some form of structure which reflects their tabular structure.  \n",
    "For instance, we can create a `list` where each row is a dictionary whose keys are the column name (found in the file's first line):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List where we will store the content of the \".csv\" file.\n",
    "# Each line of the file will be stored as a separate element (dict)\n",
    "# in the list.\n",
    "data = []\n",
    "\n",
    "with open(\"data/titanic_head.csv\") as f:\n",
    "    \n",
    "    line = f.readline()\n",
    "    \n",
    "    # The column names are in the first line\n",
    "    columnNames = line.strip().split(',')  # .split(',') is our best ally here : it cuts a str into a list. \n",
    "    \n",
    "    for line in f:\n",
    "        # Split the line in its different fields.\n",
    "        sl = line.strip().split(',')\n",
    "        \n",
    "        # Now we map the fields onto their constituent columns.\n",
    "        row = {}\n",
    "        for i in range(len(sl)):\n",
    "            row[columnNames[i]] = sl[i]\n",
    "\n",
    "        # Add the dictionary for the current row (line) to the list.\n",
    "        data.append(row)\n",
    "        \n",
    "\n",
    "print(\"Full data:\")\n",
    "for row in data:\n",
    "    print(row)\n",
    "\n",
    "print(\"***\")\n",
    "print(\"Name of passenger 4 : \", data[4][\"Name\"])\n",
    "print(\"Age of passenger 4 : \", data[4][\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sure, this works, but it is also fairly tedious to write.\n",
    "\n",
    "Because csv is such a widespread format, python comes with functions that that can help us out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv   # Imports an external module. Ignore this for now, we'll talk about it in the next notebook\n",
    "\n",
    "data = []\n",
    "\n",
    "with open(\"data/titanic_head.csv\") as f:\n",
    "    \n",
    "    reader = csv.DictReader(f)  # Returns a DictReader object.\n",
    "    for row in reader:\n",
    "        data.append(row)        # Row is a dictionary whose keys correspond to the columns!\n",
    "\n",
    "for row in data:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"full data:\")\n",
    "for row in data:\n",
    "    print(row)\n",
    "\n",
    "print(\"***\")\n",
    "print(\"Name of passenger 4 : \" , data[4][\"Name\"])\n",
    "print(\"Age of passenger 4 : \" , data[4][\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is much simpler.\n",
    "\n",
    "If the file uses **another field delimiter** (eg, `';'`), you can specify it when creating the `DictReader`:\n",
    "\n",
    "   ```python\n",
    "   reader = csv.DictReader( readingHandle , delimiter=';')\n",
    "   ```\n",
    "\n",
    "Additionally, libraries dedicated to data analysis often have functions that read directly from a csv file and create their specific data structure.\n",
    "\n",
    "For instance, for `pandas` (that's a sneak-peak into day3 modules ;-) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  # ignore this, we'll talk about it in the next notebook.\n",
    "\n",
    "# Reading the csv file as a pandas.DataFrame, their custom type for tabular data.\n",
    "df = pd.read_csv(\"data/titanic_head.csv\") \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Opening files without context managers  <a id='6.3'></a>\n",
    "\n",
    "So far we have always used the build-in `open()` function as a *context manager* (i.e. `with open() as ...`).\n",
    "* But `open()` can also be used as a regular function.\n",
    "\n",
    "**Don't forget the close it!** (usually the `with` takes care of that for you)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_handle = open(\"data/fresh_fruits.txt\", \"r\")\n",
    "\n",
    "for i, line in enumerate(file_handle):\n",
    "    print(\"line\", i, \":\", line.strip())\n",
    "        \n",
    "# Don't forget to close the file!\n",
    "file_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Reading files using a while loop  <a id='6.4'></a>\n",
    "\n",
    "Here is an example of file reading where, instead of a `for` loop, we use a `while` loop and `.readline()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reading_handle = open(\"data/fresh_fruits.txt\", \"r\")\n",
    "line = reading_handle.readline()\n",
    "i = 0\n",
    "\n",
    "# When the file has been entirely read, readline() returns an empty string and the while loop will end.\n",
    "# In python a non-empty string evaluates to \"True\", and therefore we can use \"while line\" as a shortcut \n",
    "# for \"while line != '' \".\n",
    "while line:\n",
    "    print(\"line\", i, \":\", line.strip())    # Note: we use the \"strip()\" method of \"str\" to remove the \n",
    "                                           # trailing \"\\n\" (carriage return) of each line.\n",
    "    line = reading_handle.readline()       # Don't forget this or you will have an infinite loop.\n",
    "    i += 1\n",
    "\n",
    "reading_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### The walrus operator: a new syntax for Python >= 3.8  <a id='6.5'></a>\n",
    "\n",
    "Starting with Python 3.8, a new operator **`:=`** (a.k.a, the **walrus operator**) allows to do a variable assignment (`line` in the example below), while at the same time evaluating an expression.\n",
    "\n",
    "This can be used when reading a file to reduce the number of lines of code, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", \"r\") as f:\n",
    "    i = 0\n",
    "    while (line := f.readline()):             # := assigns values to variables as part of a larger expression. \n",
    "        print(\"line\", i, \":\", line.strip())   # It is known as the \"walrus operator” and it works really well\n",
    "        i += 1                                # together with the while-loop\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "\n",
    "### What does the `flush` argument of `print()` do? <a id='6.6'></a>\n",
    "\n",
    "By default (i.e. when `flush=False`), the output of `print()` is buffered until either:\n",
    "* The buffer is full, or\n",
    "* a `\\n` is printed (or some other triggering event occurs).\n",
    "\n",
    "By passing `flush=True` to `print()`, the value passed to `print()` is printed immediately (along with everything that is still in the buffer at that time).\n",
    "\n",
    "<br>\n",
    "\n",
    "**Example:**\n",
    "* To observe the difference between `flush=False` and `flush=True` in practice, you can copy/paste\n",
    "  the following code into a python interpreter.\n",
    "* **Important:** this **will not work as expected in Jupyter Notebook**, as Jupyter Notebook always flushes the buffer.\n",
    "\n",
    "<br>\n",
    "\n",
    "```python\n",
    "import time\n",
    "\n",
    "# Default behavior: the buffer is flushed when a `\\n` gets printed at the end of the loop.\n",
    "# Note that there are no `\\n` printed inside the loop as we manually specify `end=\" \"`.\n",
    "for i in range(5):\n",
    "    print(i, end=\" \")\n",
    "    time.sleep(0.5)\n",
    "print(\"end\")\n",
    "\n",
    "# With `flush=True` the values passed to `print()` are immediately printed.\n",
    "for i in range(5):\n",
    "    print(i, end=\" \", flush=True)\n",
    "    time.sleep(0.5)\n",
    "print(\"end\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}