linear-models.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C9YRk_HtfCzW"
      },
      "source": [
        "## Assignment 1 ##\n",
        "\n",
        "Welcome to your first assignment. This assignment aims to help you understand \n",
        "linear models.\n",
        "\n",
        "Jupyter Notebooks are interactive programming environments embedded in a \n",
        "webpage. In the \"Machine Learning\" course, we use them for the practical part \n",
        "of the course."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiCVAeUgfCzc"
      },
      "source": [
        "---\n",
        "## Part A: NumPy \n",
        "The [**NumPy**](https://numpy.org/) library is the fundamental library for \n",
        "scientific computing in Python. It provides a high-performance object for \n",
        "storing multi-dimensional arrays and tools for processing these arrays.\n",
        "\n",
        "You can refer to the documentation \n",
        "([NumPy Doc](https://numpy.org/doc/1.18/user/quickstart.html)) of these\n",
        "libraries while working on the assignment.\n",
        "\n",
        "By convention, we always load the library as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "-CbAo-vXfCzd"
      },
      "outputs": [],
      "source": [
        "# Run this cell\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A2LkD9xpfCze"
      },
      "source": [
        "**A1**: Define a function that returns a 2-dimensional array with `r` rows\n",
        "and `c` columns containing random values from 0 to 1 using the `np.random.rand` \n",
        "function. Below are tests to verify the correct functionality of the function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "deletable": false,
        "id": "ramFNpdefCze",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "d92c1990a8861a674b06f1d88e60c7db",
          "grade": false,
          "grade_id": "cell-4dd473645ec1eb64",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "def random_2d_array(r, c):\n",
        "    \"\"\"Επιστρέφει διδιάστατο πίνακα με r γραμμές και c στήλες.\"\"\"\n",
        "    return np.array([[np.random.random() for _ in range(c)] for _ in range(r)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "deletable": false,
        "editable": false,
        "id": "rNxBbbhYfCzf",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "58c6d24b66e1df195036c8d5d1df487c",
          "grade": true,
          "grade_id": "random_2d_array_correct_shape",
          "locked": true,
          "points": 5,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "outputId": "9f6a075b-e7e1-455d-cb35-065e0516bb06"
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ επιστροφής πίνακα με τις σωστές διαστάσεις\"\"\"\n",
        "assert np.shape(random_2d_array(3, 5)) == (3, 5)\n",
        "assert np.shape(random_2d_array(5, 2)) == (5, 2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jz9AkM82fCzg"
      },
      "source": [
        "**A2**: Define a function that accepts two 2-dimensional arrays and returns the \n",
        "product of the inverse of the 1st array with the 2nd array using numpy. \n",
        "Below are tests to verify the correct functionality of the function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "deletable": false,
        "id": "RX6QLO1zfCzg",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "4ef0876d963ef6152ae9e8f6e2fdc7a5",
          "grade": false,
          "grade_id": "cell-c697582f03491bb7",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "def multiply_transpose(a, b):\n",
        "    \"\"\"Επιστρέφει το γινόμενο του ανάστροφου του 1ου πίνακα με τον 2ο πίνακα.\"\"\"\n",
        "    return np.dot(a.T, b)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "g6-jZksufCzh",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "c54ef19235f0620cfbf0983e572d5f4e",
          "grade": true,
          "grade_id": "cell-23e086f3e85d8f53",
          "locked": true,
          "points": 5,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ επιστροφής πίνακα με τις σωστές διαστάσεις\"\"\"\n",
        "a = random_2d_array(3, 5)\n",
        "b = random_2d_array(3, 2)\n",
        "assert np.shape(multiply_transpose(a, b)) == (5, 2)\n",
        "assert np.shape(multiply_transpose(b, a)) == (2, 5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t1WfnDaffCzh"
      },
      "source": [
        "## Part B: Linear Models ##\n",
        "\n",
        "In this part of the exercise, we will build a linear regression model \n",
        "from scratch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wtgtAqGNfCzi"
      },
      "source": [
        "**B1:** Implement the sigmoid function $\\sigma(z)= \\dfrac{1}{1 + \\exp(-z)}$ \n",
        "using NumPy. Below are tests to verify the correct functionality of the \n",
        "function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "deletable": false,
        "id": "X2rOxkk3fCzi",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "fc973cf2e7884a507e7eac2cfb3fe1ba",
          "grade": false,
          "grade_id": "cell-c21b32f8218a444e",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "def sigmoid(z):\n",
        "    \"\"\"Υπολογίζει τη σιγμοειδή συνάρτηση.\"\"\"\n",
        "    return 1 / (1 + np.exp(-z))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "drMlPPjufCzj",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "4c6d452aad1fba89422cc05702e86672",
          "grade": true,
          "grade_id": "cell-c522a25040bf7202",
          "locked": true,
          "points": 10,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ ορθού υπολογισμού σιγμοειδούς.\"\"\"\n",
        "assert sigmoid(0) == 0.5\n",
        "assert np.round(sigmoid(7),8) == 0.99908895\n",
        "assert np.round(sigmoid(-1),8) == 0.26894142"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VNErXXl0fCzk"
      },
      "source": [
        "**B2:** Implement a class for the logistic regression algorithm. Initialize the\n",
        "parameters to zero values and store them in the column vector `theta`. \n",
        "The `fit` function should take a matrix `X` with input variable values and a \n",
        "column vector `y` with output variable values. The `predict` function should \n",
        "take a matrix `X` with input variable values. Below are tests to verify the \n",
        "correct functionality of the functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "deletable": false,
        "id": "-TYh2ngOfCzk",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "0875eb2ff3083c0d5a365dfe1d72f3af",
          "grade": false,
          "grade_id": "cell-9df008253a524001",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "class MyLogisticRegression:\n",
        "    def __init__(self, num_iterations=2000, learning_rate=0.004):\n",
        "      self.theta = None\n",
        "      self.num_iterations = num_iterations\n",
        "      self.learning_rate = learning_rate\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        m, n = X.shape\n",
        "\n",
        "        self.theta = np.zeros((n + 1, 1))\n",
        "        X = np.concatenate((np.ones((m, 1)), X), axis=1)\n",
        "\n",
        "        def compute_cost(X, y):\n",
        "          z = np.dot(X, self.theta)\n",
        "          sigma = sigmoid(z)\n",
        "\n",
        "          return (-1/m) * np.sum(y * np.log(sigma) + (1-y) * (np.log(1-sigma)))\n",
        "\n",
        "        def gradient(X, y):\n",
        "          z = np.dot(X, self.theta)\n",
        "          sigma = sigmoid(z)\n",
        "\n",
        "          return (1/m) * multiply_transpose(X, (sigma - y))\n",
        "\n",
        "        def update_parameters(X, y):\n",
        "          for _ in range(self.num_iterations):\n",
        "            self.theta -= self.learning_rate * gradient(X, y)\n",
        "\n",
        "        update_parameters(X, y)\n",
        "\n",
        "    def predict(self, X):\n",
        "        m, n = X.shape\n",
        "\n",
        "        prediction = np.zeros((m,))\n",
        "        X = np.concatenate((np.ones((m, 1)), X), axis=1)\n",
        "\n",
        "        z = np.dot(X, self.theta)\n",
        "        sigma = sigmoid(z)\n",
        "\n",
        "        prediction = np.where(sigma > .5, 1, 0)\n",
        "\n",
        "        return prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "7m4ZfY30dbYS"
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ ορθής λειτουργίας fit\"\"\"\n",
        "from sklearn.datasets import make_blobs\n",
        "\n",
        "# δημιουργία δύο ομάδων από τυχαία σημεία με κέντρο 0,0,0 και 3,3,3 και \n",
        "# τυπική απόκλιση 1\n",
        "centers = [[0.0, 0.0, 0.0], [3.0, 3.0, 3.0]]\n",
        "X, y = make_blobs(n_samples=200, centers=centers,\n",
        "                  random_state=47, shuffle=False)\n",
        "y = np.array(y).reshape(-1,1)\n",
        "\n",
        "lr = MyLogisticRegression()\n",
        "lr.fit(X, y)\n",
        "\n",
        "assert np.allclose(lr.theta, np.array([[-1.1772064],[0.5026043],[0.40811666],\n",
        "                                        [0.55685573]]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "n2blP1qafCzl",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "e81990c6c217da1709ddd0caceeae095",
          "grade": true,
          "grade_id": "cell-e66896ed86d1a7ce",
          "locked": true,
          "points": 25,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ ορθής λειτουργίας predict\"\"\"\n",
        "from sklearn.datasets import make_blobs\n",
        "\n",
        "# δημιουργία δύο ομάδων από τυχαία σημεία με κέντρο 0,0,0 και 3,3,3 και \n",
        "# τυπική απόκλιση 1\n",
        "centers = [[0.0, 0.0, 0.0], [3.0, 3.0, 3.0]]\n",
        "X, y = make_blobs(n_samples=200, centers=centers,\n",
        "                  random_state=47, shuffle=False)\n",
        "y = np.array(y).reshape(-1,1)\n",
        "\n",
        "lr = MyLogisticRegression()\n",
        "lr.fit(X, y)\n",
        "\n",
        "assert np.allclose(lr.predict(X[[0,199],:]), np.array([[0],[1]]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IY4rwNMDfCzm"
      },
      "source": [
        "## Part C: Regularization\n",
        "\n",
        "**C1:** Implement a class for the logistic regression algorithm that supports \n",
        "both L1 and L2 regularization. Initialize the parameters to zero values and \n",
        "store them in the column vector `theta`. The `fit` function should take a \n",
        "matrix `X` with input variable values and a column vector `y` with output \n",
        "variable values. Below are tests to verify the correct functionality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "deletable": false,
        "id": "I3bPK3StfCzm",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "e520ffd85a225409ee0103faedb42018",
          "grade": false,
          "grade_id": "cell-8f651a74aaad362c",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "class MyLogisticRegression:\n",
        "    def __init__(self, num_iterations=2000, learning_rate=0.004, penalty=\"None\", \n",
        "        \t\tl=10):\n",
        "        self.theta = None\n",
        "        self.num_iterations = num_iterations\n",
        "        self.learning_rate = learning_rate\n",
        "        self.penalty = penalty\n",
        "        self.l = l\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        m, n = X.shape\n",
        "\n",
        "        self.theta = np.zeros((n + 1, 1))\n",
        "        X = np.concatenate((np.ones((m, 1)), X), axis=1)\n",
        "\n",
        "        def compute_cost(X, y):\n",
        "          z = np.dot(X, self.theta)\n",
        "          sigma = sigmoid(z)\n",
        "\n",
        "          cost = (-1/m) * np.sum(y * np.log(sigma) + (1-y) * (np.log(1-sigma)))\n",
        "\n",
        "          if self.penalty == \"L1\":\n",
        "            a = self.l / m\n",
        "            r = np.sum(np.abs(self.theta))\n",
        "          elif self.penalty == \"L2\":\n",
        "            a = self.l / (2 * m)\n",
        "            r = np.sum(np.square(self.theta))\n",
        "          \n",
        "          return cost + a * r\n",
        "          \n",
        "        def gradient(X, y):\n",
        "          z = np.dot(X, self.theta)\n",
        "          sigma = sigmoid(z)\n",
        "\n",
        "          if self.penalty == \"L1\":\n",
        "            return np.concatenate(([[np.dot((1/m), np.sum(sigma - y))]], \n",
        "              (1/m) * multiply_transpose(X, (sigma - y))[1:] + \n",
        "              np.dot((self.l/m), np.sign(self.theta[1:]))))\n",
        "          elif self.penalty == \"L2\":\n",
        "            return np.concatenate(([[np.dot((1/m), np.sum(sigma - y))]], \n",
        "              (1/m) * multiply_transpose(X, (sigma - y))[1:] + \n",
        "              np.dot((self.l/m), self.theta[1:])))\n",
        "\n",
        "\n",
        "        def update_parameters(X, y):\n",
        "          for _ in range(self.num_iterations):\n",
        "            self.theta -= self.learning_rate * gradient(X, y)\n",
        "\n",
        "        update_parameters(X, y)\n",
        "\n",
        "    def predict(self, X):\n",
        "        m, n = X.shape\n",
        "\n",
        "        prediction = np.zeros((m,))\n",
        "        X = np.concatenate((np.ones((m, 1)), X), axis=1)\n",
        "\n",
        "        z = np.dot(X, self.theta)\n",
        "        sigma = sigmoid(z)\n",
        "\n",
        "        prediction = np.where(sigma > .5, 1, 0)\n",
        "\n",
        "        return prediction\n",
        "        raise NotImplementedError()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "NcgHEoD1fCzm",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "feea6675fae2d5e1cb3acdb4528cc63f",
          "grade": true,
          "grade_id": "cell-7b4a1e422511b0dd",
          "locked": true,
          "points": 15,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ ορθής λειτουργίας fit L2\"\"\"\n",
        "from sklearn.datasets import make_blobs\n",
        "\n",
        "# δημιουργία δύο ομάδων από τυχαία σημεία με κέντρο 0,0,0 και 3,3,3 και \n",
        "# τυπική απόκλιση 1\n",
        "centers = [[0.0, 0.0, 0.0], [3.0, 3.0, 3.0]]\n",
        "X, y = make_blobs(n_samples=200, centers=centers,\n",
        "                  random_state=47, shuffle=False)\n",
        "y = np.array(y).reshape(-1,1)\n",
        "\n",
        "lr = MyLogisticRegression(penalty=\"L2\", l=10)\n",
        "lr.fit(X, y)\n",
        "assert np.allclose(lr.theta, np.array([[-1.148294],[0.46202407],[0.38788976],\n",
        "\t\t\t\t\t\t\t\t\t\t[0.50990465]]))\n",
        "\n",
        "lr = MyLogisticRegression(penalty=\"L2\", l=100)\n",
        "lr.fit(X, y)\n",
        "assert np.allclose(lr.theta, np.array([[-0.95841014],[0.30961435],[0.29482345],\n",
        "\t\t\t\t\t\t\t\t\t\t[0.33574242]]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "dOsAAacyfCzn",
        "nbgrader": {
          "cell_type": "code",
          "checksum": "335e75a4d82a9d5de16d1940537d3f1b",
          "grade": true,
          "grade_id": "cell-2e2aab6691a11043",
          "locked": true,
          "points": 15,
          "schema_version": 3,
          "solution": false,
          "task": false
        }
      },
      "outputs": [],
      "source": [
        "\"\"\"Τεστ ορθής λειτουργίας fit L1\"\"\"\n",
        "from sklearn.datasets import make_blobs\n",
        "\n",
        "# δημιουργία δύο ομάδων από τυχαία σημεία με κέντρο 0,0,0 και 3,3,3 και \n",
        "# τυπική απόκλιση 1\n",
        "centers = [[0.0, 0.0, 0.0], [3.0, 3.0, 3.0]]\n",
        "X, y = make_blobs(n_samples=200, centers=centers,\n",
        "                  random_state=47, shuffle=False)\n",
        "y = np.array(y).reshape(-1,1)\n",
        "\n",
        "lr = MyLogisticRegression(penalty=\"L1\", l=100)\n",
        "lr.fit(X, y)\n",
        "\n",
        "assert np.allclose(lr.theta, np.array([[-0.39128777],[0.07264989],[0.06445718],\n",
        "\t\t\t\t\t\t\t\t\t\t[0.2047192]]))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}