assignment_59.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "269c671e-021f-4652-9f47-534487c18a2b",
   "metadata": {},
   "source": [
    "Q1. What is Gradient Boosting Regression?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18b10877-4d29-4da6-94bb-e8048c25069f",
   "metadata": {},
   "source": [
    "Ans: Gradient Boosting Regression, often referred to simply as Gradient Boosting or Gradient Boosting Machines (GBM), is a powerful machine learning technique used for regression tasks. It is an ensemble learning method that combines the predictions of multiple weak learners (typically decision trees) to create a strong predictive model.\n",
    "\n",
    "In Gradient Boosting Regression, the model is trained sequentially, with each new weak learner attempting to correct the errors made by the previous learners. The key idea behind Gradient Boosting is to fit a sequence of regression trees to the residuals (or errors) of the previous trees, gradually improving the overall model's performance.\n",
    "\n",
    "Here's how Gradient Boosting Regression works:\n",
    "\n",
    "1. **Initialization**: The process begins with an initial model, usually a simple model like the mean of the target variable. This initial model serves as the starting point for the ensemble.\n",
    "\n",
    "2. **Sequential Training**: Gradient Boosting sequentially trains a series of weak learners (decision trees) on the residuals of the previous models. At each iteration, a new weak learner is trained to predict the residuals (the difference between the actual target values and the predictions of the current ensemble).\n",
    "\n",
    "3. **Gradient Descent**: The predictions of each weak learner are combined with the predictions of the previous models using a weighted sum. The weights are determined using a gradient descent algorithm to minimize a loss function (typically mean squared error for regression tasks).\n",
    "\n",
    "4. **Regularization**: To prevent overfitting, Gradient Boosting Regression often incorporates regularization techniques such as shrinkage (learning rate) and tree depth constraints. These regularization parameters control the contribution of each weak learner to the ensemble and help prevent the model from becoming too complex.\n",
    "\n",
    "5. **Final Prediction**: The final prediction of the Gradient Boosting Regression model is obtained by summing the predictions of all the weak learners in the ensemble. The final prediction is the sum of the initial model's prediction and the predictions of all subsequent weak learners.\n",
    "\n",
    "Gradient Boosting Regression is known for its flexibility, robustness, and ability to handle complex relationships in the data. It is widely used in various regression tasks, including predictive modeling, time series forecasting, and anomaly detection. Popular implementations of Gradient Boosting Regression include scikit-learn's GradientBoostingRegressor, XGBoost, LightGBM, and CatBoost."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f11c80-28b6-4fb2-9910-c3f84f9e11ff",
   "metadata": {},
   "source": [
    "Q2. Implement a simple gradient boosting algorithm from scratch using Python and NumPy. Use a\n",
    "simple regression problem as an example and train the model on a small dataset. Evaluate the model's\n",
    "performance using metrics such as mean squared error and R-squared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d8d968d6-9c0e-41bc-a088-0140641f96a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train MSE: 271.28, Test MSE: 295.97\n",
      "Train R^2: -2.42, Test R^2: -3.34\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "class GradientBoostingRegressor:\n",
    "    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):\n",
    "        self.n_estimators = n_estimators\n",
    "        self.learning_rate = learning_rate\n",
    "        self.max_depth = max_depth\n",
    "        self.models = []\n",
    "        self.residuals = []\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        # Initialize residuals\n",
    "        self.residuals = np.zeros_like(y)\n",
    "\n",
    "        for _ in range(self.n_estimators):\n",
    "            # Fit a weak learner (decision tree) to the residuals\n",
    "            tree = DecisionTreeRegressor(max_depth=self.max_depth)\n",
    "            tree.fit(X, self.residuals)\n",
    "            \n",
    "            # Make predictions with the weak learner\n",
    "            predictions = tree.predict(X)\n",
    "            \n",
    "            # Update residuals\n",
    "            self.residuals -= self.learning_rate * predictions\n",
    "            \n",
    "            # Add the weak learner to the ensemble\n",
    "            self.models.append(tree)\n",
    "\n",
    "    def predict(self, X):\n",
    "        # Initialize predictions\n",
    "        predictions = np.zeros(len(X))\n",
    "        \n",
    "        # Make predictions with each weak learner and update the ensemble prediction\n",
    "        for model in self.models:\n",
    "            predictions += self.learning_rate * model.predict(X)\n",
    "        \n",
    "        return predictions\n",
    "    \n",
    "    def get_params(self, deep=True):\n",
    "        return {\n",
    "            'n_estimators': self.n_estimators,\n",
    "            'learning_rate': self.learning_rate,\n",
    "            'max_depth': self.max_depth\n",
    "        }\n",
    "    def set_params(self, **params):\n",
    "        for param, value in params.items():\n",
    "            setattr(self, param, value)\n",
    "        return self\n",
    "\n",
    "def mean_squared_error(y_true, y_pred):\n",
    "    return np.mean((y_true - y_pred) ** 2)\n",
    "\n",
    "def r_squared(y_true, y_pred):\n",
    "    ss_res = np.sum((y_true - y_pred) ** 2)\n",
    "    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)\n",
    "    return 1 - (ss_res / ss_tot)\n",
    "\n",
    "# Generate synthetic data\n",
    "np.random.seed(42)\n",
    "X = np.random.rand(100, 1) * 10\n",
    "y = 3 * X.squeeze() + np.random.randn(100) * 2  # True relationship + noise\n",
    "\n",
    "# Split data into train and test sets\n",
    "X_train, X_test = X[:80], X[80:]\n",
    "y_train, y_test = y[:80], y[80:]\n",
    "\n",
    "# Train gradient boosting regressor\n",
    "gb_regressor = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3)\n",
    "gb_regressor.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred_train = gb_regressor.predict(X_train)\n",
    "y_pred_test = gb_regressor.predict(X_test)\n",
    "\n",
    "# Evaluate performance\n",
    "train_mse = mean_squared_error(y_train, y_pred_train)\n",
    "test_mse = mean_squared_error(y_test, y_pred_test)\n",
    "train_r2 = r_squared(y_train, y_pred_train)\n",
    "test_r2 = r_squared(y_test, y_pred_test)\n",
    "\n",
    "print(f\"Train MSE: {train_mse:.2f}, Test MSE: {test_mse:.2f}\")\n",
    "print(f\"Train R^2: {train_r2:.2f}, Test R^2: {test_r2:.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2640e8e4-a8c2-48e7-9d05-df81c4b2a974",
   "metadata": {},
   "source": [
    "Q3. Experiment with different hyperparameters such as learning rate, number of trees, and tree depth to\n",
    "optimise the performance of the model. Use grid search or random search to find the best\n",
    "hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "37323091-c841-4ac9-9d23-1a16ebbbe843",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 27 candidates, totalling 135 fits\n",
      "Best Hyperparameters: {'learning_rate': 0.01, 'max_depth': 3, 'n_estimators': 50}\n",
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "Best Hyperparameters: {'n_estimators': 50, 'max_depth': 5, 'learning_rate': 0.01}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "\n",
    "# Define hyperparameters grid for grid search\n",
    "param_grid = {\n",
    "    'learning_rate': [0.01, 0.1, 0.2],\n",
    "    'n_estimators': [50, 100, 150],\n",
    "    'max_depth': [3, 5, 7]\n",
    "}\n",
    "\n",
    "# Create gradient boosting regressor\n",
    "gb_regressor = GradientBoostingRegressor()\n",
    "\n",
    "# Grid search\n",
    "grid_search = GridSearchCV(gb_regressor, param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Best hyperparameters\n",
    "print(\"Best Hyperparameters:\", grid_search.best_params_)\n",
    "\n",
    "# Random search\n",
    "random_search = RandomizedSearchCV(gb_regressor, param_distributions=param_grid, n_iter=10, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "random_search.fit(X_train, y_train)\n",
    "\n",
    "# Best hyperparameters\n",
    "print(\"Best Hyperparameters:\", random_search.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f5f095-977e-4295-9c19-eef8d5dae32f",
   "metadata": {},
   "source": [
    "Q4. What is a weak learner in Gradient Boosting?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3205c7-b020-45a6-80e1-91ab56e4e644",
   "metadata": {},
   "source": [
    "Ans: In Gradient Boosting, a weak learner refers to a simple or base model that is relatively simple and performs only slightly better than random guessing on a given task. Weak learners are typically decision trees with limited depth or \"stumps\" (trees with only one split).\n",
    "\n",
    "The concept of using weak learners in Gradient Boosting is central to the algorithm's effectiveness. Rather than relying on a single complex model to make predictions, Gradient Boosting combines the predictions of many weak learners to create a strong predictive model. Each weak learner focuses on capturing a specific aspect of the data, and their predictions are aggregated to produce the final prediction of the ensemble.\n",
    "\n",
    "The term \"weak\" does not imply that the individual learners are inherently poor or ineffective. Instead, it reflects that they are relatively simple models that perform only slightly better than random guessing. Despite their simplicity, weak learners can contribute valuable insights to the ensemble and collectively lead to significant improvements in predictive performance when combined properly.\n",
    "\n",
    "Some common examples of weak learners used in Gradient Boosting include decision trees with shallow depth, linear models, and even small neural networks. These models are computationally efficient and can capture local patterns in the data, making them suitable for boosting algorithms like Gradient Boosting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f445fd-a830-45c9-b1a4-5228b2758d35",
   "metadata": {},
   "source": [
    "Q5. What is the intuition behind the Gradient Boosting algorithm?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfe492b4-8eb3-47f2-b3b0-4478553eece4",
   "metadata": {},
   "source": [
    "Ans: The intuition behind the Gradient Boosting algorithm can be understood through the following key concepts:\n",
    "\n",
    "1. **Ensemble Learning**: Gradient Boosting is an ensemble learning technique that combines the predictions of multiple weak learners (typically decision trees) to create a strong predictive model. The idea is that by combining the predictions of multiple models, each capturing different aspects of the data, the ensemble can achieve better predictive performance than any individual model.\n",
    "\n",
    "2. **Sequential Training**: Unlike other ensemble methods like bagging, where weak learners are trained independently in parallel, Gradient Boosting trains weak learners sequentially. Each weak learner is trained to correct the errors of the ensemble built by the previous learners. This sequential training process allows the algorithm to focus on the \"hard\" examples that are difficult to predict correctly.\n",
    "\n",
    "3. **Gradient Descent Optimization**: Gradient Boosting optimizes the ensemble by minimizing a loss function (typically mean squared error for regression or cross-entropy for classification) using gradient descent. At each iteration, a new weak learner is added to the ensemble to reduce the error of the previous ensemble. The algorithm updates the ensemble in the direction that minimizes the loss function gradient, hence the name \"Gradient Boosting.\"\n",
    "\n",
    "4. **Residual Fitting**: In each iteration, the new weak learner is trained to fit the residual errors (the difference between the true target values and the predictions of the current ensemble). By focusing on the residuals, Gradient Boosting progressively reduces the errors in the predictions, leading to improved model performance.\n",
    "\n",
    "5. **Shrinkage and Regularization**: Gradient Boosting includes regularization techniques such as shrinkage (learning rate) and tree depth constraints to prevent overfitting. Shrinkage reduces the contribution of each weak learner to the ensemble, while tree depth constraints limit the complexity of the weak learners. These regularization techniques help prevent the model from fitting the noise in the data too closely, improving generalization performance.\n",
    "\n",
    "Overall, the intuition behind Gradient Boosting is to build an ensemble of weak learners that work together to progressively improve the model's predictive performance by focusing on the residuals and optimizing a loss function using gradient descent. Through sequential training and regularization, Gradient Boosting creates a powerful and robust predictive model capable of handling complex datasets and capturing intricate patterns in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e17aad7e-4520-44d5-ae70-64ef9f5c83a0",
   "metadata": {},
   "source": [
    "Q6. How does Gradient Boosting algorithm build an ensemble of weak learners?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "640e84e3-3b4a-4159-b984-10fefba7f953",
   "metadata": {},
   "source": [
    "Ans: The Gradient Boosting algorithm builds an ensemble of weak learners in a sequential manner. Here's how the process typically works:\n",
    "\n",
    "1. **Initialization**: The process starts with an initial model, often a simple one like the mean of the target variable. This initial model serves as the starting point for the ensemble.\n",
    "\n",
    "2. **Sequential Training**: In each iteration, a new weak learner (often a decision tree) is added to the ensemble. The new learner is trained to correct the errors made by the existing ensemble. Specifically, it's trained on the residuals, which are the differences between the actual target values and the predictions of the current ensemble.\n",
    "\n",
    "3. **Gradient Descent Optimization**: The new weak learner is trained to minimize the loss function (e.g., mean squared error for regression tasks) using gradient descent. The gradient indicates the direction of steepest descent in the loss function space, so the weak learner's parameters are adjusted in that direction.\n",
    "\n",
    "4. **Weighted Combination**: After training the new weak learner, its predictions are combined with the predictions of the existing ensemble. Each weak learner's contribution is weighted by a learning rate parameter, which controls the rate at which the algorithm learns. Additionally, the predictions are summed together, producing the updated predictions of the ensemble.\n",
    "\n",
    "5. **Updating Residuals**: Once the new weak learner's predictions are added to the ensemble, the residuals are updated. The residuals are recalculated as the differences between the updated predictions and the actual target values. This step ensures that the next weak learner focuses on the remaining errors not yet captured by the ensemble.\n",
    "\n",
    "6. **Iteration**: Steps 2 to 5 are repeated for a predefined number of iterations or until a stopping criterion is met. Each iteration adds a new weak learner to the ensemble, gradually improving the model's predictive performance.\n",
    "\n",
    "By sequentially adding weak learners and focusing on the residuals, the Gradient Boosting algorithm builds an ensemble of models that collectively capture the underlying patterns in the data. The final ensemble combines the strengths of multiple weak learners, resulting in a powerful predictive model capable of making accurate predictions on complex datasets. Regularization techniques, such as shrinkage and tree depth constraints, are often used to prevent overfitting and improve the generalization performance of the ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f46ae6f-be4c-4982-a75f-15c4983d367f",
   "metadata": {},
   "source": [
    "Q7. What are the steps involved in constructing the mathematical intuition of Gradient Boosting\n",
    "algorithm?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2fa33a-6e6d-4b04-81ee-5c8afc1c4b73",
   "metadata": {},
   "source": [
    "Ans: The mathematical intuition behind the Gradient Boosting algorithm involves several key steps. Here's an overview of the process:\n",
    "\n",
    "1. **Objective Function**: Define an objective function to be optimized. In Gradient Boosting for regression tasks, the objective function is typically the mean squared error (MSE) or another appropriate loss function.\n",
    "\n",
    "2. **Initialize Model**: Start with an initial model, often a simple one like the mean of the target variable. This initial model serves as the starting point for the ensemble.\n",
    "\n",
    "3. **Compute Residuals**: Compute the residuals, which are the differences between the actual target values and the predictions of the current ensemble. These residuals represent the errors made by the current model.\n",
    "\n",
    "4. **Fit Weak Learner to Residuals**: Train a weak learner (e.g., decision tree) on the residuals. The weak learner's task is to capture the patterns in the residuals and learn how to correct the errors made by the current ensemble.\n",
    "\n",
    "5. **Update Ensemble**: Combine the predictions of the weak learner with the predictions of the current ensemble. This combination is often done by adding the weak learner's predictions to the ensemble's predictions, weighted by a learning rate parameter.\n",
    "\n",
    "6. **Update Residuals**: Recalculate the residuals as the differences between the updated predictions and the actual target values. This step ensures that the next weak learner focuses on the remaining errors not yet captured by the ensemble.\n",
    "\n",
    "7. **Iterate**: Repeat steps 4 to 6 for a predefined number of iterations or until a stopping criterion is met. Each iteration adds a new weak learner to the ensemble, gradually improving the model's predictive performance.\n",
    "\n",
    "8. **Regularization**: Apply regularization techniques, such as shrinkage (learning rate) and tree depth constraints, to prevent overfitting and improve the generalization performance of the ensemble.\n",
    "\n",
    "9. **Final Prediction**: Once all iterations are completed, the final prediction of the Gradient Boosting model is obtained by summing the predictions of all weak learners in the ensemble.\n",
    "\n",
    "10. **Optimization**: During the training process, the weak learners are trained to minimize the objective function using gradient descent. The gradient indicates the direction of steepest descent in the loss function space, so the weak learner's parameters are adjusted in that direction.\n",
    "\n",
    "By following these steps, the Gradient Boosting algorithm constructs an ensemble of weak learners that collectively capture the underlying patterns in the data. The final ensemble combines the strengths of multiple weak learners, resulting in a powerful predictive model capable of making accurate predictions on complex datasets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}