Fanalytics: Find Your Bandwagon

NBA league makes >$1 billion from merchandise annually. Meanwhile, there is an online sports forum with 2.7 million NBA fans. In order to capitalize on this large fan group, NBA could push ads for jerseys and gear to those users. On this forum, half of the users have already tagged themselves as fans of a certain team. So, it is convenient to push team-specific ads to those users. However, for the other half of the users, how do we identify their favorite teams?

I took a Natural Language Processing approach to classify fans' teams by their comments and visiting history on other forums. By applying different feature extraction techniques and an ensemble model, the prediction reached a classification accuracy of 64%, which is 20x to the chance level and 2.2x to my MVP model.

With my model, NBA could make about $648k from merchandise. My model could also be applied to other online sports forums.

Webapp: NBA Fan Identifier

Type in your comment then we can tell which NBA team you like.

Presentation on the project

Insight Project

Dependencies

• Data Scraping:
  • praw 6.3.1
• NLP:
  • spaCy 2.1.8
  • gensim 3.7.3
• Machine Learning:
  • scikit-learn 0.21.2
  • imblearn 0.5.0

Repository folders

• docker: docker configure files
• figures: figures used in README
• notebooks: jupyter notebooks
• scripts: python functions
• webapp: code to deploy the webapp

Usage

• Scrap data

  1. Scrap titles/ids of all comment threads:
     python ./scripts/scraping/pushshift_scrap.py (Also see ./notebooks/pushshift_scrap_submission_reg18.ipynb)
  2. Scrap all comments
     python ./scripts/scraping/praw_scrap.py (Also see ./notebooks/praw_scrap_comments_reg18.ipynb)
  3. Scrap users' visiting history
     python ./scripts/scraping/praw_scrap_history.py (Also see ./notebooks/scrap_user_history.ipynb)

• Run models
  ./notebooks/Wk3_Final_Model.ipynb

Docker image

I made a Docker image with scrapped data and the final model (Wk3_Final_Model.ipynb). Ask me if you want a copy of the image.

The Docker image was made by running:
sudo docker build -t insight . -f ./docker/Dockerfile

The Docker image can be run by:
sudo docker run --name insight -p 8888:8888 insight

Then go to: localhost:8888/tree
Ask me for the password!

The notebook can be found at:
/home/insight/notebooks/

To clean the container, run:
sudo docker rm insight

Overview of notebooks

• Week 1: Pilot data scraping and EDA
  ./notebooks/Wk1_pilot_EDA.ipynb
• Week 2: Building a MVP model
  ./notebooks/Wk2_EDA_MVP.ipynb
• Week 3: Optimizing the model and creating an ensemble model
  ./notebooks/Wk3_Final_Model.ipynb

List of other scripts

Run Named Entity Recognition with spaCy
./scripts/ner.py

Run TextRank keywords extraction with gensim
./scripts/textrank.py

Load all data
./scripts/load_data.py

Preprocess texts, remove emoji, special charactors, etc.
./scripts/preprocess.py

Standerdize all team names in training data into 3 letters (e.g. TOR)
./scripts/teamname_stdize.py

Calculate AUC for a model
./scripts/cal_auc.py

A simple unit test for cal_auc
./scripts/test_cal_auc.py

Calculate and plot confusion matrix
./scripts/plot_confusion_matrix.py