08-Machine-Learning-with-SQL/08-01-Machine-Learning-Overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Analytics and Machine Learning Overview\n",
    "\n",
    "This notebook offers a basic overview of advanced analytics, some example use cases, and a basic advanced analytics workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Short Primer on Advanced Analytics\n",
    "\n",
    "Advanced analytics refers to a variety of techniques aimed at solving the core problem of deriving insights and making predictions or recommendations based on data. The best organization for machine learning is structured based on the task that you’d like to perform. The most common tasks include:\n",
    "\n",
    "* Supervised learning, including classification and regression, where the goal is to predict a label for each data point based on various features.\n",
    "\n",
    "* Recommendation engines to suggest products to users based on behavior.\n",
    "\n",
    "* Unsupervised learning, including clustering, anomaly detection, and topic modeling, where the goal is to discover structure in the data.\n",
    "\n",
    "* Graph analytics tasks such as searching for patterns in a social network.\n",
    "\n",
    "Let’s review each of these tasks along with some common machine learning and advanced analytics use cases. The following books are great resources for learning more about the individual analytics (and, as a bonus, they are freely available on the web):\n",
    "\n",
    "[An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/) by Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani.\n",
    "\n",
    "[Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/) by Trevor Hastie, Robert Tibshirani, and Jerome Friedman.\n",
    "\n",
    "[Deep Learning](http://www.deeplearningbook.org/) by Ian Goodfellow, Yoshua Bengio, and Aaron Courville."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervised Learning\n",
    "\n",
    "Supervised learning is probably the most common type of machine learning. The goal is simple: using historical data that already has labels (often called the dependent or target variables), train a model to predict the values of those labels based on various features of the data points. One example would be to predict a person’s income (the dependent variable) based on age (a feature). This training process usually proceeds through an iterative optimization algorithm such as gradient descent. The training algorithm starts with a basic model and gradually improves it by adjusting various internal parameters (coefficients) during each training iteration. The result of this process is a trained model that you can use to make predictions on new data. There are a number of different tasks we’ll need to complete as part of the process of training and making predictions, such as measuring the success of trained models before using them in the field, but the fundamental principle is simple: train on historical data, ensure that it generalizes to data we didn’t train on, and then make predictions on new data.\n",
    "\n",
    "We can further organize supervised learning based on the type of variable we’re looking to predict. We’ll get to that next.\n",
    "\n",
    "**CLASSIFICATION**\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-supervised-classification.png?raw=true\" width=\"400\" align=\"center\"/>\n",
    "\n",
    "One common type of supervised learning is classification. Classification is the act of training an algorithm to predict a dependent variable that is categorical (belonging to a discrete, finite set of values). The most common case is binary classification, where our resulting model will make a prediction that a given item belongs to one of two groups. The canonical example is classifying email spam. Using a set of historical emails that are organized into groups of spam emails and not spam emails, we train an algorithm to analyze the words in, and any number of properties of, the historical emails and make predictions about them. Once we are satisfied with the algorithm’s performance, we use that model to make predictions about future emails the model has never seen before.\n",
    "\n",
    "When we classify items into more than just two categories, we call this multiclass classification. For example, we may have four different categories of email (as opposed to the two categories in the previous paragraph): spam, personal, work related, and other. There are many use cases for classification, including:\n",
    "\n",
    "* Predicting disease: A doctor or hospital might have a historical dataset of behavioral and physiological attributes of a set of patients. They could use this dataset to train a model on this historical data (and evaluate its success and ethical implications before applying it) and then leverage it to predict whether or not a patient has heart disease or not. This is an example of binary classification (healthy heart, unhealthy heart) or multiclass classification (healthly heart, or one of several different diseases).\n",
    "\n",
    "* Classifying images: There are a number of applications from companies like Apple, Google, or Facebook that can predict who is in a given photo by running a classification model that has been trained on historical images of people in your past photos. Another common use case is to classify images or label the objects in images.\n",
    "\n",
    "* Predicting customer churn: A more business-oriented use case might be predicting customer churn—that is, which customers are likely to stop using a service. You can do this by training a binary classifier on past customers that have churned (and not churned) and using it to try and predict whether or not current customers will churn.\n",
    "\n",
    "* Buy or won’t buy: Companies often want to predict whether visitors of their website will purchase a given product. They might use information about users’ browsing pattern or attributes such as location in order to drive this prediction.\n",
    "\n",
    "There are many more use cases for classification beyond these examples. We will introduce more use cases in the upcoming notebooks.\n",
    "\n",
    "**REGRESSION**\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-supervised-regression.png?raw=true\" width=\"400\" align=\"center\"/>\n",
    "\n",
    "In classification, our dependent variable is a set of discrete values. In regression, we instead try to predict a continuous variable (a real number). In simplest terms, rather than predicting a category, we want to predict a value on a number line. The rest of the process is largely the same, which is why they’re both forms of supervised learning. We will train on historical data to make predictions about data we have never seen. Here are some typical examples:\n",
    "\n",
    "* Predicting sales: A store may want to predict total product sales on given data using historical sales data. There are a number of potential input variables, but a simple example might be using last week’s sales data to predict the next day’s data.\n",
    "\n",
    "* Predicting height: Based on the heights of two individuals, we might want to predict the heights of their potential children.\n",
    "\n",
    "* Predicting the number of viewers of a show: A media company like Netflix might try to predict how many of their subscribers will watch a particular show."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unsupervised Learning\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-unsupervised.png?raw=true\" width=\"400\" align=\"center\"/>\n",
    "\n",
    "Unsupervised learning is the act of trying to find patterns or discover the underlying structure in a given set of data. This differs from supervised learning because there is no dependent variable (label) to predict.\n",
    "\n",
    "Some example use cases for unsupervised learning include:\n",
    "\n",
    "* Anomaly detection: Given some standard event type often occuring over time, we might want to report when a nonstandard type of event occurs. For example, a security officer might want to receive notifications when a strange object (think vehicle, skater, or bicyclist) is observed on a pathway.\n",
    "\n",
    "* User segmentation: Given a set of user behaviors, we might want to better understand what attributes certain users share with other users. For instance, a gaming company might cluster users based on properties like the number of hours played in a given game. The algorithm might reveal that casual players have very different behavior than hardcore gamers, for example, and allow the company to offer different recommendations or rewards to each player.\n",
    "\n",
    "* Topic modeling: Given a set of documents, we might analyze the different words contained therein to see if there is some underlying relation between them. For example, given a number of web pages on data analytics, a topic modeling algorithm can cluster them into pages about machine learning, SQL, streaming, and so on based on groups of words that are more common in one topic than in others.\n",
    "\n",
    "Intuitively, it is easy to see how segmenting customers could help a platform cater better to each set of users. However, it may be hard to discover whether or not this set of user segments is “correct”. For this reason, it can be difficult to determine whether a particular model is good or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommendation\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-recommendation.png?raw=true\" width=\"400\" align=\"center\"/>\n",
    "\n",
    "Recommendation is one of the most intuitive applications of advanced analytics. By studying people’s explicit preferences (through ratings) or implicit ones (through observed behavior) for various products or items, an algorithm can make recommendations on what a user may like by drawing similarities between the users or items. By looking at these similarities, the algorithm makes recommendations to users based on what similar users liked, or what other products resemble the ones the user already purchased. Here are some example use cases:\n",
    "\n",
    "* Movie recommendations: Netflix uses recommender engines to make large-scale movie recommendations to its users. It does this by studying what movies users watch and do not watch in the Netflix application. In addition, Netflix likely takes into consideration how similar a given user’s ratings are to other users’.\n",
    "\n",
    "* Product recommendations: Amazon uses product recommendations as one of its main tools to increase sales. For instance, based on the items in our shopping cart, Amazon may recommend other items that were added to similar shopping carts in the past. Likewise, on every product page, Amazon shows similar products purchased by other users.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph Analytics\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-graph-analytics.png?raw=true\" width=\"400\" align=\"center\"/>\n",
    "\n",
    "While less common than classification and regression, graph analytics is a powerful tool. Fundamentally, graph analytics is the study of structures in which we specify vertices (which are objects) and edges (which represent the relationships between those objects). For example, the vertices might represent people and products, and edges might represent a purchase. By looking at the properties of vertices and edges, we can better understand the connections between them and the overall structure of the graph. Since graphs are all about relationships, anything that specifies a relationship is a great use case for graph analytics. Some examples include:\n",
    "\n",
    "* Fraud prediction: Capital One uses graph analytics to better understand fraud networks. By using historical fraudulent information (like phone numbers, addresses, or names) they discover fraudulent credit requests or transactions. For instance, any user accounts within two hops of a fraudulent phone number might be considered suspicious.\n",
    "\n",
    "* Anomaly detection: By looking at how networks of individuals connect with one another, outliers and anomalies can be flagged for manual analysis. For instance, if typically in our data each vertex has ten edges associated with it and a given vertex only has one edge, that might be worth investigating as something strange.\n",
    "\n",
    "* Classification: Given some facts about certain vertices in a network, you can classify other vertices according to their connection to the original node. For instance, if a certain individual is labeled as an influencer in a social network, we could classify other individuals with similar network structures as influencers.\n",
    "\n",
    "* Recommendation: Google’s original web recommendation algorithm, PageRank, is a graph algorithm that analyzes website relationships in order to rank the importance of web pages. For example, a web page that has a lot of links to it is ranked as more important than one with no links to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Advanced Analytics Process\n",
    "\n",
    "You should have a firm grasp of some fundamental use cases for machine learning and advanced analytics. However, finding a use case is only a small part of the actual advanced analytics process. There is a lot of work in preparing your data for analysis, testing different ways of modeling it, and evaluating these models. This section will provide structure to the overall anaytics process and the steps we have to take to not just perform one of the tasks just outlined, but actually evaluate success objectively in order to understand whether or not we should apply our model to the real world.\n",
    "\n",
    "<img src=\"https://github.com/soltaniehha/Business-Analytics/blob/master/figs/10-01-machine-learning-workflow.png?raw=true\" width=\"800\" align=\"center\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall process involves, the following steps (with some variation):\n",
    "\n",
    "1. Gathering and collecting the relevant data for your task.\n",
    "\n",
    "2. Cleaning and inspecting the data to better understand it.\n",
    "\n",
    "3. Performing feature engineering to allow the algorithm to leverage the data in a suitable form (e.g., converting the data to numerical vectors).\n",
    "\n",
    "4. Using a portion of this data as a training set to train one or more algorithms to generate some candidate models.\n",
    "\n",
    "5. Evaluating and comparing models against your success criteria by objectively measuring results on a subset of the same data that was not used for training. This allows you to better understand how your model may perform in the wild.\n",
    "\n",
    "6. Leveraging the insights from the above process and/or using the model to make predictions, detect anomalies, or solve more general business challenges.\n",
    "\n",
    "These steps won’t be the same for every advanced analytics task. However, this workflow does serve as a general framework for what you’re going to need to be successful with advanced analytics. Let’s break down the process to better understand the overall objective of each step.\n",
    "\n",
    "**DATA COLLECTION**\n",
    "\n",
    "Naturally it’s hard to create a training set without first collecting data. Typically this means at least gathering the datasets you’ll want to leverage to train your algorithm. \n",
    "\n",
    "**DATA CLEANING**\n",
    "\n",
    "After you’ve gathered the proper data, you’re going to need to clean and inspect it. This is typically done as part of the exploratory data analysis process, or EDA. EDA generally means using interactive queries and visualization methods in order to better understand distributions, correlations, and other details in your data. During this process you may notice you need to remove some values that may have been misrecorded upstream or that other values may be missing. Whatever the case, it’s always good to know what is in your data to avoid mistakes down the road. \n",
    "\n",
    "**FEATURE ENGINEERING**\n",
    "\n",
    "Now that you collected and cleaned your dataset, it’s time to convert it to a form suitable for machine learning algorithms, which generally means numerical features. Proper feature engineering can often make or break a machine learning application, so this is one task you’ll want to do carefully. The process of feature engineering includes a variety of tasks, such as normalizing data, adding variables to represent the interactions of other variables, manipulating categorical variables, and converting them to the proper format to be input into our machine learning model. \n",
    "\n",
    "**TRAINING MODELS**\n",
    "\n",
    "At this point in the process we have a dataset of historical information (e.g., spam or not spam emails) and a task we would like to complete (e.g., classifying spam emails). Next, we will want to train a model to predict the correct output, given some input. During the training process, the parameters inside of the model will change according to how well the model performed on the input data. For instance, to classify spam emails, our algorithm will likely find that certain words are better predictors of spam than others and therefore weight the parameters associated with those words higher. In the end, the trained model will find that certain words should have more influence (because of their consistent association with spam emails) than others. The output of the training process is what we call a model. Models can then be used to gain insights or to make future predictions. To make predictions, you will give the model an input and it will produce an output based on a mathematical manipulation of these inputs. Using the classification example, given the properties of an email, it will predict whether that email is spam or not by comparing to the historical spam and not spam emails that it was trained on.\n",
    "\n",
    "However, just training a model isn’t the objective—we want to leverage our model to produce insights. Thus, we must answer the question: how do we know our model is any good at what it’s supposed to do? That’s where model tuning and evaluation come in.\n",
    "\n",
    "**MODEL TUNING AND EVALUATION**\n",
    "\n",
    "You likely noticed earlier that we mentioned that you should split your data into multiple portions and use only one for training. This is an essential step in the machine learning process because when you build an advanced analytics model you want that model to generalize to data it has not seen before. Splitting our dataset into multiple portions allows us to objectively test the effectiveness of the trained model against a set of data that it has never seen before. The objective is to see if your model understands something fundamental about this data process or whether or not it just noticed the things particular to only the training set (sometimes called overfitting). That’s why it is called a test set. In the process of training models, we also might take another, separate subset of data and treat that as another type of test set, called a validation set, in order to try out different hyperparameters (parameters that affect the training process) and compare different variations of the same model without overfitting to the test set.\n",
    "\n",
    "Note: Following proper training, validation, and test set best practices is essential to successfully using machine learning. It’s easy to end up overfitting (training a model that does not generalize well to new data) if we do not properly isolate these sets of data.\n",
    "\n",
    "**LEVERAGING THE MODEL AND/OR INSIGHTS**\n",
    "\n",
    "After running the model through the training process and ending up with a well-performing model, you are now ready to use it! Taking your model to production can be a significant challenge in and of itself. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}