Skip to content

Commit

Permalink
Enhance tutorial
Browse files Browse the repository at this point in the history
  • Loading branch information
@darrylong
darrylong committed Nov 4, 2024
1 parent f405ef6 commit b26f38e
Showing 1 changed file with 103 additions and 117 deletions.
220 changes: 103 additions & 117 deletions tutorials/model_ensembling.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1347,46 +1347,42 @@
"metadata": {},
"source": [
"## 2. Simple Model Ensembling\n",
"We'll start by combining predictions from **BPR** and **WMF** models using the **Borda Count** method.\n",
"\n",
"### What is Borda Count?\n",
"Borda Count is a ranking-based ensembling technique that assigns scores to items based on their ranking position in each model. \n",
"For example, an item ranked 1st in a model might get a high score, while a lower-ranked item gets a lower score. \n",
"We then sum these scores across models to get a combined ranking.\n",
"\n",
"**Example**:\n",
"Let's say we have 5 items and 3 models. We rank the items based on the predicted scores from each model.\n",
"\n",
"1. For each model, rank the items from 1 to 5 based on the predicted scores.\n",
"2. Sum the ranks of each item across all models.\n",
"3. Sort the items based on the sum of their ranks.\n",
"4. The top-ranked item is the final recommendation.\n",
"5. Repeat the process for the next user.\n",
"\n",
"Given the below example for a random user **123**:\n",
"\n",
"| Rank | Model 1 | Model 2 | Model 3 | Allocated Points (N - rank) |\n",
"|------|---------|---------|---------|-----------------------------|\n",
"| 1 | A | D | E | 5 - 1 = 4 |\n",
"| 2 | B | C | A | 5 - 2 = 3 |\n",
"| 3 | C | A | B | 5 - 3 = 2 |\n",
"| 4 | D | B | D | 5 - 4 = 1 |\n",
"| 5 | E | E | C | 5 - 5 = 0 |\n",
"In this section, we’ll combine the predictions from the **BPR** and **WMF** models using a method called **Borda Count**.\n",
"\n",
"Based on the allocated points for each of the items, we sum the points up to get our Borda Count.\n",
"\n",
"| Item | Model 1 | Model 2 | Model 3 | Borda Count |\n",
"|------|---------|---------|---------|--------------|\n",
"| A | 4 | 2 | 3 | 9 |\n",
"| B | 3 | 1 | 2 | 6 |\n",
"| C | 2 | 3 | 0 | 5 |\n",
"| D | 1 | 4 | 1 | 6 |\n",
"| E | 0 | 0 | 4 | 4 |\n",
"\n",
"**New ranking: A > B, D > C > E**\n",
"### What is Borda Count?\n",
"\n",
"Borda Count is a simple ranking method that assigns points based on an item’s rank in each model. Higher-ranked items get more points. We then add up the points from all models to get a combined ranking.\n",
"\n",
"Lets implement this below."
"**Example**:\n",
"1. Each model ranks items from 1 to 5.\n",
"2. Items earn points based on their rank (e.g., 1st place gets 4 points, 2nd gets 3 points, etc.).\n",
"3. We sum the points for each item across all models.\n",
"4. The item with the highest total points becomes the top recommendation.\n",
"\n",
"Here’s a sample ranking for a user:\n",
"\n",
"| Rank | Model 1 | Model 2 | Model 3 | Points (5 - rank) |\n",
"|------|---------|---------|---------|-------------------|\n",
"| 1 | A | D | E | 4 |\n",
"| 2 | B | C | A | 3 |\n",
"| 3 | C | A | B | 2 |\n",
"| 4 | D | B | D | 1 |\n",
"| 5 | E | E | C | 0 |\n",
"\n",
"**Borda Count Result**:\n",
"\n",
"| Item | Total Points |\n",
"|------|--------------|\n",
"| A | 9 |\n",
"| B | 6 |\n",
"| C | 5 |\n",
"| D | 6 |\n",
"| E | 4 |\n",
"\n",
"**Final Ranking: A > B, D > C > E**\n",
"\n",
"Now, let’s implement this method!"
]
},
{
Expand Down Expand Up @@ -1681,13 +1677,14 @@
"id": "57994d68",
"metadata": {},
"source": [
"The top recommendation ItemID **313**, was ranked **8** on BPR and **1** on WMF, where the 2nd recommendation ItemID **739** was ranked **7** on BPR and **11** on WMF.\n",
"\n",
"This shows how we can combine the recommendations of multiple models.\n",
"The top recommendation, **ItemID 313**, was ranked **8th** by BPR and **1st** by WMF. Similarly, the second recommendation, **ItemID 739**, was ranked **7th** by BPR and **11th** by WMF.\n",
"\n",
"-------\n",
"This demonstrates how ensembling allows us to leverage the strengths of multiple models to produce a more balanced recommendation.\n",
"\n",
"Next, let's add the recommendations into the genre distribution dataframe to compare its results to the base models."
"---\n",
"\n",
"Next, we’ll incorporate the recommendations into the genre distribution dataframe to compare their performance against the individual base models."
]
},
{
Expand Down Expand Up @@ -1858,22 +1855,20 @@
"metadata": {},
"source": [
"## 3. Further Ensembling\n",
"In this step, we create variations of the **WMF** model and ensemble them using Borda Count. \n",
"Each variation might have slight adjustments (like parameter changes) to capture different aspects of the dataset.\n",
"\n",
"**Analogy**:\n",
"Imagine you're trying to decide on a movie with friends, and each of you has a slightly different preference. \n",
"By considering all opinions, you make a choice that balances everyone's input. \n",
"Similarly, by creating variations of the WMF model, we capture multiple \"opinions\" from the same model type, improving the final ensemble.\n",
"\n",
"With these variations, Borda Count helps balance the predictions, enhancing performance compared to a single model.\n",
"In this step, we enhance our ensemble by creating variations of the **WMF** model and combining their predictions using Borda Count. Each variation introduces slight adjustments, such as changes in parameters, to capture different perspectives of the dataset.\n",
"\n",
"**Approach**:\n",
"An approach is to train a model with different initializations using different random seeds. By adding multiple similar models of different random seeds (`seed=123`), some models could perform better for a set of users, while some models could perform better for others.\n",
"### Analogy:\n",
"Think of choosing a movie with friends, where each person has a slightly different taste. By considering everyone’s preferences, you make a decision that satisfies the group. Similarly, by introducing variations of the WMF model, we gather multiple \"opinions\" from the same model type, leading to a more balanced and robust recommendation.\n",
"\n",
"Another approach is to change the number of latent factors `k`. By changing the number of latent factors, we can capture different aspects of the dataset.\n",
"### Approach:\n",
"1. **Different Random Seeds**: \n",
" Train multiple models with different random seeds (e.g., `seed=123`). This variation captures different nuances, as some models may perform better for certain users than others.\n",
" \n",
"2. **Varying Latent Factors**: \n",
" Adjust the number of latent factors (`k`). By changing `k`, the models can capture diverse aspects of the data, providing a broader view of the underlying patterns.\n",
"\n",
"Let's try adding a few more similar models with different random seed (`seed`) initializations and different latent factors (`k`)."
"Let’s implement this by training several WMF models with different seeds and latent factor values, then ensemble them using Borda Count to improve the overall recommendation performance."
]
},
{
Expand Down Expand Up @@ -2629,35 +2624,29 @@
"metadata": {},
"source": [
"## 4. Ensembling with Regression Models\n",
"Now we explore regression-based ensembling with **linear regression** and **random forest regression**. \n",
"These regression models learn the relationship between each model's predictions and the target outcome, \n",
"producing an ensemble that adapts based on data patterns.\n",
"\n",
"**Why Regression Models?**\n",
"- **Linear Regression**: Simple and interpretable, ideal when the relationship between models' predictions and true values is linear.\n",
"- **Random Forest Regression**: More flexible, capturing non-linear relationships and interactions, which can improve prediction accuracy in complex datasets.\n",
"In this step, we’ll explore ensembling using **linear regression** and **random forest regression**. These methods allow us to model the relationship between the predictions of multiple models and the actual outcomes, resulting in a more adaptive and accurate ensemble.\n",
"\n",
"Using these methods allows us to create an ensemble that adapts to varying data patterns, unlike simple averaging techniques."
]
},
{
"cell_type": "markdown",
"id": "60561dc5",
"metadata": {},
"source": [
"**Approach**:\n",
"### Why Use Regression Models?\n",
"\n",
"- **Linear Regression**: \n",
" A simple and interpretable approach, best suited when the relationship between model predictions and true values is linear.\n",
" \n",
"- **Random Forest Regression**: \n",
" A more flexible method that captures non-linear relationships and complex interactions, making it well-suited for diverse and intricate datasets.\n",
"\n",
"These regression-based methods go beyond basic averaging by adapting to patterns in the data, potentially improving prediction accuracy.\n",
"\n",
"We treat this as a meta-learning problem. Recommendations of each base model are treated as features, which will trains a meta-learner to predict the final recommendation.\n",
"### Approach\n",
"\n",
"Input: \n",
"Predictions of base models (WMF Variations)\n",
"This process can be seen as a meta-learning problem. Here’s how it works:\n",
"\n",
"Output: \n",
"Final Recommendation\n",
"- **Input**: Predictions from base models (e.g., WMF variations).\n",
"- **Output**: Final recommendation.\n",
"\n",
"This gives us the flexibility to use many ML models such as a Linear Regression, Random Forest, Gradient Boosting, or even a Neural Network.\n",
"We use the predictions of the base models as features, and a meta-learner (such as Linear Regression or Random Forest) is trained to make the final prediction. This framework allows flexibility to experiment with various machine learning models, including Linear Regression, Random Forest, Gradient Boosting, or even Neural Networks.\n",
"\n",
"Let's start by training a Linear Regression model to ensemble the predictions of the WMF models."
"Let’s begin by training a **Linear Regression** model to combine the predictions from the WMF variations."
]
},
{
Expand Down Expand Up @@ -3126,23 +3115,23 @@
"id": "048c684f",
"metadata": {},
"source": [
"We have successfully fit into a Linear Regression model using the 9 WMF base models consisting of different seeds and latent factors.\n",
"We have successfully trained a **Linear Regression** model using the predictions from the 9 WMF base models, which included variations with different seeds and latent factors.\n",
"\n",
"As we can tell from the coefficients, the linear regression model has given more weight to the `WMF_k50` model, which has the highest coefficient.\n",
"The coefficients from the model indicate that the `WMF_k50` model received the highest weight, suggesting it plays a significant role in the final predictions.\n",
"\n",
"Let's now fit into a Random Forest Regressor model."
"Next, let's proceed to train a **Random Forest Regressor** model."
]
},
{
"cell_type": "markdown",
"id": "d6bed0bc-f6c1-43fc-9c4b-acb56bcae554",
"metadata": {},
"source": [
"##### 4.3 Fitting Random Forest Model\n",
"##### 4.3 Fitting the Random Forest Model\n",
"\n",
"We reuse the same training data and train a Random Forest Regressor model.\n",
"We will use the same training data to fit a **Random Forest Regressor** model.\n",
"\n",
"While in this example we used a Random forest model, we could also try other models like Gradient Boosting, etc.\n"
"While we are using a Random Forest in this example, we also have the option to experiment with other models, such as Gradient Boosting and others, to see how they perform."
]
},
{
Expand Down Expand Up @@ -3336,13 +3325,13 @@
"id": "7fa25cc1",
"metadata": {},
"source": [
"We have successfully fit into a Random Forest Regressor model using the 9 WMF base models consisting of different seeds and latent factors.\n",
"We have successfully trained a **Random Forest Regressor** model using the predictions from the 9 WMF base models, which included variations with different seeds and latent factors.\n",
"\n",
"From the distribution, we can tell that these different ensemble models utilized the base model predictions differently to come up with the final prediction.\n",
"The distribution of the results indicates that these ensemble models leveraged the base model predictions in different ways to generate the final predictions.\n",
"\n",
"------\n",
"---\n",
"\n",
"In the next section, we will further compare the results of the different models."
"In the next section, we will compare the results of the various models to evaluate their performance."
]
},
{
Expand All @@ -3358,15 +3347,15 @@
"id": "2d02a252",
"metadata": {},
"source": [
"In Section 4, we were only able to view the recommendation distribution and compare them visually based on a **single user**.\n",
"In Section 4, we focused on visualizing the recommendation distribution for a **single user**.\n",
"\n",
"**Issue**:\n",
"What if we want to compare the models based on **multiple users**?\n",
"### Issue:\n",
"How can we effectively compare the models across **multiple users**?\n",
"\n",
"**Approach**:\n",
"We can do so by calculating the score of all users and item combinations, then calculate the **Precision** and **Recall** of the predictions for each model.\n",
"### Approach:\n",
"To address this, we will calculate scores for all combinations of users and items, enabling us to assess the **Precision** and **Recall** of the predictions for each model.\n",
"\n",
"We will also create a dataframe `rank_df` to store the results of the models given **all users and items**."
"Additionally, we will create a dataframe, `rank_df`, to store the results of the models for **all users and items**. This will allow for a comprehensive comparison of model performance."
]
},
{
Expand Down Expand Up @@ -3630,18 +3619,18 @@
"id": "04f63b15",
"metadata": {},
"source": [
"**Hold on!**:\n",
"You may be wondering why we are doing this Borda Count calculation again. This is because we are evaluating the different models based on **Precision** and **Recall**, which requires scores for every user and item combination.\n",
"**Hold on!** \n",
"You might be wondering why we’re recalculating the Borda Count. The reason is that we need to evaluate the different models based on **Precision** and **Recall**, which requires scores for every combination of users and items.\n",
"\n",
"After we have the points calculated. Let's add them up similarly as per the Borda Count formula shown in sections 2 and 3 above.\n",
"Once we have the scores calculated, we will sum them up according to the Borda Count formula outlined in Sections 2 and 3.\n",
"\n",
"**Simple Borda Count**:\n",
"To reiterate, our simple `Borda Count` model consists of the **BPR Model** and **WMF Model**.\n",
"**Simple Borda Count**: \n",
"To clarify, our basic Borda Count model includes the **BPR Model** and the **WMF Model**.\n",
"\n",
"**WMF Borda Count**:\n",
"While the `WMF Borda Count` model consists of multiple models:\n",
"- Different random seed initilization: **wmf_model_123**, **wmf_model_456**, **wmf_model_789**, **wmf_model_888**, **wmf_model_999**, and\n",
"- Different latent factors: **wmf_model_k20**, **wmf_model_k30**, **wmf_model_k40**, **wmf_model_k50**."
"**WMF Borda Count**: \n",
"The `WMF Borda Count` model, on the other hand, consists of multiple variations:\n",
"- Models initialized with different random seeds: **wmf_model_123**, **wmf_model_456**, **wmf_model_789**, **wmf_model_888**, and **wmf_model_999**.\n",
"- Models with different latent factors: **wmf_model_k20**, **wmf_model_k30**, **wmf_model_k40**, and **wmf_model_k50**."
]
},
{
Expand Down Expand Up @@ -4266,33 +4255,30 @@
"source": [
"## 6. Conclusion\n",
"\n",
"Based on the results, we can see varying results for **Precision@50** and **Recall@50**.\n",
"The results demonstrate variations in **Precision@50** and **Recall@50**, highlighting that no single model can be deemed the 'ideal' choice for all datasets and scenarios. Depending on the specific dataset, different models may yield better performance.\n",
"\n",
"However, there is no one model that is the 'ideal' model for all datasets and scenarios.\n",
"Depending on the dataset, different models may perform better than others.\n",
"### 6.1 Key Takeaways\n",
"\n",
"### 6.1 Learning Points\n",
"It is crucial to experiment with various models and ensembling techniques to identify the best approach for a particular dataset and context. Ensemble learning offers a pathway to enhance model performance, and you can continue exploring by:\n",
"\n",
"Therefore, it is important to experiment with different models and ensemble methods to find the best model for a specific dataset and scenario.\n",
"- Testing different base models from Cornac\n",
"- Experimenting with alternative ensembling methods\n",
"- Adjusting parameters and configurations within the base models\n",
"\n",
"Ensemble learning doesn't stop here. You could continue on by: \n",
"- Trying different Cornac base models\n",
"- Trying different ensemble methods\n",
"- Tweaking around base models and different parameters within them\n",
"**Note:** Keep in mind that results can vary based on the dataset and the models employed.\n",
"\n",
"**Note:** The results may vary based on the dataset and the models used.\n",
"### 6.2 Considerations for Model Ensembling\n",
"\n",
"### 6.2 Model Ensembling Considerations\n",
"- While model ensembling theoretically provides superior performance, there could be instances where base models outperforms ensembled models.\n",
"- Model ensembling also requires more computational resources as well. Therefore, we should consider striking a balance between performanace and computational costs.\n",
"- Although ensemble methods often provide improved performance, there may be situations where individual base models outperform the ensemble.\n",
"- Model ensembling typically demands more computational resources, so it's essential to find a balance between performance gains and computational costs.\n",
"\n",
"### 6.3 Food for Thought\n",
"### 6.3 Questions for Further Exploration\n",
"\n",
"- Which base models and configurations are ideal?\n",
"- So what constitutes to a good ensembled model?\n",
"- At what point should we just use base models?\n",
"- Which base models and configurations yield the best results?\n",
"- What defines an effective ensemble model?\n",
"- When should we rely solely on base models instead of ensembling?\n",
"\n",
"These are topics that require further experimentation and discussion."
"These questions highlight areas for further experimentation and discussion as we continue to refine our understanding of model performance in recommendation systems."
]
}
],
Expand Down

0 comments on commit b26f38e

Please sign in to comment.