haterz_classification.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:08eed76edef1b3fea1477b58a5f395349824217f92b19e3ad172c37fbd3057bd"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "![](http://www.blogcdn.com/www.urlesque.com/media/2010/05/haterafrican.jpg)\n",
      "\n",
      "# Hater News\n",
      "\n",
      "Haterz gonna hate. But now you know who the haterz are.\n",
      "\n",
      "###My Goal:\n",
      "\n",
      "Be able to look at any user from [hacker news](https://news.ycombinator.com/) and be able to tell if they are a \"hater\" or not based on a score I give them. \n",
      "\n",
      "###Feel free to look under the hood.\n",
      "\n",
      "[The Classifier Data I'm Using.](https://www.kaggle.com/c/detecting-insults-in-social-commentary/data)\n",
      "\n",
      "[Hacker News API I'm Using.](https://github.com/HackerNews/API)\n",
      "\n",
      "*Eventually I would love to turn this into a [Chrome App](https://developer.chrome.com/extensions/getstarted) that will just real-time analize any user on a page when you visit hacker news and put a score right next to them so the world can see if they hate or love.*"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Lets Get Started! \n",
      "We will begin by importing all of the required items we need from things like [scikit learn](http://scikit-learn.org/stable/), [pandas](http://pandas.pydata.org/pandas-docs/stable/10min.html), [numpy](http://www.numpy.org/), and even play around with things like [nlk](http://www.nltk.org/) and [re](https://docs.python.org/2/library/re.html).\n",
      "\n",
      "The basic idea will be to split out each comment, create features based on things like if a word shows up in a comment or not, if insulting words appear, and other types of features. Once I have that then I will add in a classifier and if it scores well we are on our way to finding haters!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Importing my standard stuff\n",
      "import pandas as pd\n",
      "import numpy as np\n",
      "\n",
      "# Importing different Vectorizers to see what makes my soul sing the loudest.\n",
      "from sklearn.feature_extraction.text import TfidfVectorizer\n",
      "from sklearn.feature_extraction.text import CountVectorizer\n",
      "\n",
      "# Importing various Classifiers to see what wins out. There can only be one! \n",
      "from sklearn.linear_model import Perceptron\n",
      "from sklearn.svm import LinearSVC\n",
      "from sklearn.svm import SVC\n",
      "from sklearn.linear_model import LogisticRegression\n",
      "\n",
      "# Snagging the validation stuff to see how well things do.\n",
      "from sklearn.pipeline import Pipeline\n",
      "from sklearn.cross_validation import cross_val_score\n",
      "\n",
      "\n",
      "# Getting Stopwords to potentially improve accuracy.\n",
      "from nltk.corpus import stopwords\n",
      "\n",
      "# Messing with tokenization via NLTK\n",
      "from nltk import word_tokenize, pos_tag\n",
      "# Messing with Stemming\n",
      "from nltk.stem import PorterStemmer\n",
      "# Messing with Lemmatizing\n",
      "from nltk.stem.wordnet import WordNetLemmatizer\n",
      "\n",
      "# Messing with some other string splitting techniques. You can use the following thing like this:\n",
      "# re.split('; |, ',str)\n",
      "import re"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Exploring Different Ways To Achieve My Goal.\n",
      "\n",
      "Here I am importing my data using **pandas**, pulling out the right features and assigning them to my X & y to put into my models. You'll notice that I'm actually using both my test & training data in a single variable because of how [cross_val_score](http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.cross_val_score.html) works later on."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Setting up my english stopwords, yo.\n",
      "stopwords = stopwords.words('english')\n",
      "\n",
      "# Here I combined my training & test data. I will used this as main total \"training\" data in production & for cross_val stuff.\n",
      "corpus = pd.read_csv('data/total_data.csv')\n",
      "\n",
      "# Setting Up my X & y\n",
      "X_raw = corpus.Comment\n",
      "y_raw = corpus.Insult\n",
      "    \n",
      "# Setting Up My Different Pipelines. I tried out several different combos to see where to go next.\n",
      "pipeline = Pipeline([\n",
      "                     ('vect', TfidfVectorizer(binary=True, use_idf=True, stop_words='english')),\n",
      "                     ('clf', LogisticRegression())\n",
      "])\n",
      "\n",
      "other_pipeline = Pipeline([\n",
      "                     ('vect', CountVectorizer(binary=True)),\n",
      "                     ('clf', LogisticRegression())\n",
      "])\n",
      "\n",
      "\n",
      "LSVC_pipeline = Pipeline([\n",
      "                     ('vect',  TfidfVectorizer(binary=True, use_idf=True, stop_words='english')),\n",
      "                     ('clf', LinearSVC())\n",
      "])\n",
      "\n",
      "Perceptron_pipeline = Pipeline([\n",
      "                     ('vect',  TfidfVectorizer(binary=True, use_idf=True, stop_words='english')),\n",
      "                     ('clf', Perceptron())\n",
      "])\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# EVALUATION TIME!\n",
      "Here I am looking at **Accuracy**, **Precision**, & **Recall** ([learn more](http://en.wikipedia.org/wiki/Precision_and_recall)) and using the **cross_val_score** method which splits up my data, takes the **pipelines** I made, and then evaluates it over and over, in different combinations against itself to get a reliable score.\n",
      "\n",
      "I want a good overall score but I would also like to improve my Recall as much as possible. Recall gives me a measurement of false negatives, meaning a score based on if I miss-classify something by saying a comment is NOT insulting when in fact it actually is. I am interested in a high score using this measurement to ensure [no hater is left behind](http://www.quickmeme.com/img/a3/a3e69566093ab74d0625a7c5892cade8c2adeecd86112356875ff2b05fcd020d.jpg). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "\n",
      "# THE INITIAL 'STATNDARD' MODEL I STARTED WITH\n",
      "accuracy_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5)\n",
      "precision_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5, scoring='precision')\n",
      "recall_scores = cross_val_score(pipeline, X_raw, y_raw, cv=5, scoring='recall')\n",
      "\n",
      "print \"***************OUR RESULTS***************\"\n",
      "print\n",
      "print \n",
      "\n",
      "print \"TfidfVect + Logistic Regression\"\n",
      "print \"-------------------------------\"\n",
      "print \"Accuracy:\", accuracy_scores, \"Mean:\", np.mean(accuracy_scores)\n",
      "print \"Precision:\", precision_scores, \"Mean:\", np.mean(precision_scores)\n",
      "print \"Recall:\", recall_scores, \"Mean:\", np.mean(recall_scores)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "# VS NORMAL COUNT VECTORIZER **NOTE** Out of all of the models without extra features, this performed the best for my goals.\n",
      "\n",
      "other_accuracy_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5)\n",
      "other_precision_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5, scoring='precision')\n",
      "other_recall_scores = cross_val_score(other_pipeline, X_raw, y_raw, cv=5, scoring='recall')\n",
      "print\n",
      "print\n",
      "print \"CountVect + Logistic Regression\"\n",
      "print \"-------------------------------\"\n",
      "print \"Accuracy:\", other_accuracy_scores, \"Mean:\", np.mean(other_accuracy_scores)\n",
      "print \"Precision:\", other_precision_scores, \"Mean:\", np.mean(other_precision_scores)\n",
      "print \"Recall:\", other_recall_scores, \"Mean:\", np.mean(other_recall_scores)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "# VS NORMAL COUNT VECTORIZER\n",
      "\n",
      "LSVC_accuracy_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5)\n",
      "LSVC_precision_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5, scoring='precision')\n",
      "LSVC_recall_scores = cross_val_score(LSVC_pipeline, X_raw, y_raw, cv=5, scoring='recall')\n",
      "print\n",
      "print\n",
      "print \"TfidfVect + LinearSVC\"\n",
      "print \"-------------------------------\"\n",
      "print \"Accuracy:\", LSVC_accuracy_scores, \"Mean:\", np.mean(LSVC_accuracy_scores)\n",
      "print \"Precision:\", LSVC_precision_scores, \"Mean:\", np.mean(LSVC_precision_scores)\n",
      "print \"Recall:\", LSVC_recall_scores, \"Mean:\", np.mean(LSVC_recall_scores)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "# VS NORMAL COUNT VECTORIZER\n",
      "\n",
      "Perceptron_accuracy_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5)\n",
      "Perceptron_precision_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5, scoring='precision')\n",
      "Perceptron_recall_scores = cross_val_score(Perceptron_pipeline, X_raw, y_raw, cv=5, scoring='recall')\n",
      "print\n",
      "print\n",
      "print \"TfidfVect + Percptron\"\n",
      "print \"-------------------------------\"\n",
      "print \"Accuracy:\", Perceptron_accuracy_scores, \"Mean:\", np.mean(Perceptron_accuracy_scores)\n",
      "print \"Precision:\", Perceptron_precision_scores, \"Mean:\", np.mean(Perceptron_precision_scores)\n",
      "print \"Recall:\", Perceptron_recall_scores, \"Mean:\", np.mean(Perceptron_recall_scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "***************OUR RESULTS***************\n",
        "\n",
        "\n",
        "TfidfVect + Logistic Regression\n",
        "-------------------------------\n",
        "Accuracy: [ 0.81969697  0.81439394  0.81790592  0.81942337  0.81866464] Mean: 0.818016967858\n",
        "Precision: [ 0.85806452  0.825       0.84615385  0.8313253   0.84713376] Mean: 0.84153548429\n",
        "Recall: [ 0.38108883  0.3782235   0.37931034  0.39655172  0.38218391] Mean: 0.383471659586\n",
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "CountVect + Logistic Regression\n",
        "-------------------------------\n",
        "Accuracy: [ 0.83939394  0.84848485  0.84218513  0.84825493  0.84066768] Mean: 0.843797305375\n",
        "Precision: [ 0.75464684  0.76895307  0.74137931  0.75694444  0.73793103] Mean: 0.751970939603\n",
        "Recall: [ 0.58166189  0.61031519  0.61781609  0.62643678  0.61494253] Mean: 0.610234495933\n",
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "TfidfVect + LinearSVC\n",
        "-------------------------------\n",
        "Accuracy: [ 0.83181818  0.82878788  0.8323217   0.84066768  0.82776935] Mean: 0.832272957189\n",
        "Precision: [ 0.74329502  0.743083    0.73605948  0.75367647  0.7122807 ] Mean: 0.737678935001\n",
        "Recall: [ 0.55587393  0.53868195  0.56896552  0.58908046  0.58333333] Mean: 0.567187036854\n",
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "TfidfVect + Percptron\n",
        "-------------------------------\n",
        "Accuracy: [ 0.8         0.80606061  0.79742033  0.79817906  0.79362671] Mean: 0.799057341242\n",
        "Precision: [ 0.64214047  0.63716814  0.6231003   0.61452514  0.60795455] Mean: 0.624977719778\n",
        "Recall: [ 0.55014327  0.61891117  0.58908046  0.63218391  0.61494253] Mean: 0.601052267562\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "\n",
      "#Now That I Have A Baseline, Time to Pick A Model And Build It Out Further. \n",
      "\n",
      "I ended it up picking **CountVectorizer + Logistic Regression** which you will see further down. In order to improve the score I started to play with extra features adding them manually and \"hand making\" them. These are almost meta data about each document that I was hoping would help give extra information to my classifier. They are things like How many \"bad words\" are used in a specific comment or the ratio of bad words used vs how many words total were in a comment. I even tried to look at things like if someone spoke in all CAPS or not to see if that could help predict if the comment is insulting or not.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Setting up our actually train and test data sets\n",
      "train_corpus = pd.read_csv('data/train.csv')\n",
      "test_corpus = pd.read_csv('data/test_with_solutions.csv')\n",
      "\n",
      "# Getting our list of \"badwords\"\n",
      "badwords = set(pd.read_csv('data/my_badwords.csv').words)\n",
      "\n",
      "# If you would like to see what some of these words look like, play with this:\n",
      "print \"Some Badwords We Will Check:\", list(badwords)[0:9]\n",
      "print \n",
      "\n",
      "# Setting Up my X & y\n",
      "# NOTE: If you want to use all the data (including the test data), uncomment out the other X & y trains below.\n",
      "X_train = train_corpus.Comment\n",
      "y_train = train_corpus.Insult\n",
      "X_train = corpus.Comment\n",
      "y_train = corpus.Insult\n",
      "\n",
      "X_test = test_corpus.Comment\n",
      "y_test = test_corpus.Insult\n",
      "\n",
      "# Just giving feedback so we know if we are using all of the data to train the model or not.\n",
      "print \"X_train's number of instances:\",X_train.shape[0]\n",
      "print\n",
      "\n",
      "if X_train.shape[0] > train_corpus.Comment.shape[0]:\n",
      "    print \"*************************************************************************************************************\"\n",
      "    print \"***JUST AN FYI, YOU'RE USING BOTH THE TRAINING & TESTING DATA FOR TRAINING. SHOULD BE FOR PRODUCTION ONLY.***\"\n",
      "    print \"*************************************************************************************************************\"\n",
      "else:\n",
      "    print \"----------------------------------------------------------------------\"\n",
      "    print \"You are using just the training data for training. Makes sense, right?\"\n",
      "    print \"----------------------------------------------------------------------\"\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Some Badwords We Will Check: ['bugger', 'jizz', 'l3i+ch', 'f u c k e r', 's hit', 'ma5terbate', 'bum', 'bimbo', 'pimpis']\n",
        "\n",
        "X_train's number of instances: 6594\n",
        "\n",
        "*************************************************************************************************************\n",
        "***JUST AN FYI, YOU'RE USING BOTH THE TRAINING & TESTING DATA FOR TRAINING. SHOULD BE FOR PRODUCTION ONLY.***\n",
        "*************************************************************************************************************\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Adding In Extra Features.\n",
      "\n",
      "Here I add in various extra features to help my classifier out. These are features that I had to manually create and add into my model so I didn't use cross_val_score and had to evaluate them manually. \n",
      "\n",
      "###Features I added to improve the model are the following:\n",
      "* **badwords_count - **A count of bad words used in each comment.\n",
      "* **n_words - **A count of words used in each comment.\n",
      "* **allcaps - **A count of capital letters in each comment.\n",
      "* **allcaps_ratio - **A count of capital letters in each comment / the total words used in each comment.\n",
      "* **bad_ratio - **A count of bad words used in each comment / the total words used in each comment.\n",
      "* **exclamation - **A count of \"!\" used in each comment.\n",
      "* **addressing - **A count of \"@\" symbols used in each comment.\n",
      "* **spaces - **A count of spaces used in each comment.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Since I was unhappy with aspects of my score, I added addtional features my classifier can use\n",
      "\n",
      "# This is just a count of how many bad word\n",
      "train_badwords_count = []\n",
      "test_badwords_count = []\n",
      "\n",
      "for el in X_train:\n",
      "    tokens = el.split(' ')\n",
      "    train_badwords_count.append(len([i for i in tokens if i.lower() in badwords]))\n",
      "    \n",
      "for el in X_test:\n",
      "    tokens = el.split(' ')\n",
      "    test_badwords_count.append(len([i for i in tokens if i.lower() in badwords]))\n",
      "    \n",
      "# **SHOUT OUT**\n",
      "# I was messing with stuff from Andreas Mueller for these next features. Thanks man! :) \n",
      "# His Blog Post on this: http://blog.kaggle.com/2012/09/26/impermium-andreas-blog/ \n",
      "# **SHOUT OUT**\n",
      "    \n",
      "train_n_words = [len(c.split()) for c in X_train]\n",
      "test_n_words = [len(c.split()) for c in X_test]\n",
      "\n",
      "train_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in X_train]\n",
      "test_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in X_test]\n",
      "\n",
      "train_allcaps_ratio = np.array(train_allcaps) / np.array(train_n_words, dtype=np.float)\n",
      "test_allcaps_ratio = np.array(test_allcaps) / np.array(test_n_words, dtype=np.float)\n",
      "\n",
      "train_bad_ratio = np.array(train_badwords_count) / np.array(train_n_words, dtype=np.float)\n",
      "test_bad_ratio = np.array(test_badwords_count) / np.array(test_n_words, dtype=np.float)\n",
      "\n",
      "train_exclamation = [c.count(\"!\") for c in X_train]\n",
      "test_exclamation = [c.count(\"!\") for c in X_test]\n",
      "\n",
      "train_addressing = [c.count(\"@\") for c in X_train]\n",
      "test_addressing = [c.count(\"@\") for c in X_test]\n",
      "\n",
      "train_spaces = [c.count(\" \") for c in X_train]\n",
      "test_spaces = [c.count(\" \") for c in X_test]\n",
      "\n",
      "\n",
      "print \"train_badwords count:\", len(train_badwords_count), \"test_badwords count:\", len(test_badwords_count),\n",
      "print \"train_allcaps count:\", len(train_allcaps), \"test_allcaps count:\", len(test_allcaps)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "train_badwords count: 6594 test_badwords count: 2647 train_allcaps count: 6594 test_allcaps count: 2647\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Time To Pick Our Model & Add In Our New Features.\n",
      "Now that we have our new features we need to pick a vectorizer, smash the new features inside of our vectorized old features, and evaluated everything with our classfier."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# SIDE NOTE: TfidfVectorizer sucks for this type of problem. Well, you could use it and not use idf and it does get better but not too much...\n",
      "# vect = TfidfVectorizer(binary=True, use_idf=True, stop_words='english')\n",
      "vect = CountVectorizer(binary=True)\n",
      "clf = LogisticRegression()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###*Side Note: This is the \"smashing\" process I was talking about.\n",
      "I'm basically fiting and transforming our features via our vectorizer and then useing [.reshape( )](http://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html) & [np.hstack( )](http://docs.scipy.org/doc/numpy/reference/generated/numpy.hstack.html) to manually add in our extra features."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X_train_transform = vect.fit_transform(X_train)\n",
      "X_test_transform = vect.transform(X_test)\n",
      "\n",
      "train_reshaped_badwords = np.array(train_badwords_count).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_badwords = np.array(test_badwords_count).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_n_words = np.array(train_n_words).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_n_words = np.array(test_n_words).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_allcaps = np.array(train_allcaps).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_allcaps = np.array(test_allcaps).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_allcaps_ratio = np.array(train_allcaps_ratio).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_allcaps_ratio = np.array(test_allcaps_ratio).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_bad_ratio = np.array(train_bad_ratio).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_bad_ratio = np.array(test_bad_ratio).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_exclamation = np.array(train_exclamation).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_exclamation = np.array(test_exclamation).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_addressing = np.array(train_addressing).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_addressing = np.array(test_addressing).reshape((len(test_badwords_count),1))\n",
      "\n",
      "train_reshaped_spaces = np.array(train_spaces).reshape((len(train_badwords_count),1))\n",
      "test_reshaped_spaces = np.array(test_spaces).reshape((len(test_badwords_count),1))\n",
      "\n",
      "\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform.todense(), train_reshaped_badwords))\n",
      "X_test_transform = np.hstack((X_test_transform.todense(), test_reshaped_badwords))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_n_words))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_n_words))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_allcaps))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_allcaps))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_allcaps_ratio))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_allcaps_ratio))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_bad_ratio))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_bad_ratio))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_exclamation))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_exclamation))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_addressing))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_addressing))\n",
      "\n",
      "X_train_transform = np.hstack((X_train_transform, train_reshaped_spaces))\n",
      "X_test_transform = np.hstack((X_test_transform, test_reshaped_spaces))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Scoring Our New Model.\n",
      "Finally We get to see our new score. It seems to be doing a bit better but to start to look at recall and other measures we can set our predictions and then see how many items we missed and in what ways we missed them. Execute the final cells to see the scores."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf.fit(X_train_transform,y_train)\n",
      "predictions = clf.predict(X_test_transform)\n",
      "clf.score(X_test_transform, y_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "0.97468832640725345"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Run this cell to get a print out of each of our wrong predcitions, and how they were predicted incorrectly.\n",
      "wrong_predictions_number = 0\n",
      "false_negatives = 0\n",
      "false_postitives = 0\n",
      "for i,p in enumerate(predictions):\n",
      "    if p != y_test[i]:\n",
      "        wrong_predictions_number += 1\n",
      "        if p > y_test[i]:\n",
      "            false_postitives += 1\n",
      "            print\n",
      "            print \"++FALSE POSITIVE++\"\n",
      "        if p < y_test[i]:\n",
      "            false_negatives += 1\n",
      "            print\n",
      "            print \"--FALSE NEGATIVE--\"\n",
      "        print \"predicted:\", p, \"actual\", y_test[i]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "++FALSE POSITIVE++\n",
        "predicted: 1 actual 0\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n",
        "\n",
        "--FALSE NEGATIVE--\n",
        "predicted: 0 actual 1\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Execute this cell to see a final scoring of our new model\n",
      "print \"total number of instances:\", len(predictions), \n",
      "print \"| total number wrong:\", wrong_predictions_number, \n",
      "print \"| total false negatives:\", false_negatives, \n",
      "print \"| total false positives:\", false_postitives\n",
      "print\n",
      "print\n",
      "print \"            THE FINAL STATS\"\n",
      "print \"----------------------------------------\"\n",
      "print \"Percent Right:\", 100 - (float(wrong_predictions_number) / float(len(predictions))) * 100,\"%\"\n",
      "print \"Percent Wrong:\", (float(wrong_predictions_number) / float(len(predictions))) * 100,\"%\"\n",
      "print \"Percent False Negatives:\", (float(false_negatives) / float(len(predictions))) * 100,\"%\"\n",
      "print \"Percent False Positives:\", (float(false_postitives) / float(len(predictions))) * 100,\"%\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "total number of instances: 2647 | total number wrong: 67 | total false negatives: 54 | total false positives: 13\n",
        "\n",
        "\n",
        "            THE FINAL STATS\n",
        "----------------------------------------\n",
        "Percent Right: 97.4688326407 %\n",
        "Percent Wrong: 2.53116735927 %\n",
        "Percent False Negatives: 2.04004533434 %\n",
        "Percent False Positives: 0.491122024934 %\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Uncomment this and execute to get a list of insults to look through for more potential patterns\n",
      "# corpus.Comment[corpus.Insult == 1].to_csv(\"insults.csv\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Now Time To Pull In Data From Hacker News.\n",
      "Thanks [Hacker News](https://news.ycombinator.com/) for your great [API](https://github.com/HackerNews/API)! \n",
      "Here I will be using urllib2 and json to pull in and organize the data I get back from HN."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Setting up a connection to the internetz\n",
      "from urllib2 import urlopen\n",
      "# Getting things to come back from json to a dict\n",
      "import json\n",
      "# Making that json pretty if needed\n",
      "from pprint import pprint\n",
      "# Dealing with stupid Unicode issues\n",
      "from django.utils.encoding import smart_str\n",
      "\n",
      "# Let's make sure things work by getting a magical kitten from the interwebs\n",
      "kittens = urlopen('http://placekitten.com/')\n",
      "response = kittens.read()\n",
      "kitty = response[559:1000]\n",
      "print \"MY PRETTY KITTY. HIS NAME IS FRANK.\"\n",
      "print\n",
      "print kitty\n",
      "print\n",
      "\n",
      "# Now time to try with Hacker News\n",
      "username = \"testhater\"\n",
      "strt_item = \"8478757\"\n",
      "url_usr_strt = \"https://hacker-news.firebaseio.com/v0/user/\"\n",
      "url_itm_strt = \"https://hacker-news.firebaseio.com/v0/item/\"\n",
      "url_end = \".json\"\n",
      "\n",
      "kevin = urlopen( url_usr_strt+username+url_end )\n",
      "post = urlopen( url_itm_strt+strt_item+url_end )\n",
      "\n",
      "kevin = json.loads(kevin.read())\n",
      "post = json.loads(post.read())\n",
      "\n",
      "print \"MY HACKER NEWS ACCOUNT:\"\n",
      "print \"-----------------------\"\n",
      "pprint(kevin)\n",
      "\n",
      "print\n",
      "print\n",
      "\n",
      "print \"MY HACKER NEWS POST:\"\n",
      "print \"-----------------------\"\n",
      "pprint(post)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "MY PRETTY KITTY. HIS NAME IS FRANK.\n",
        "\n",
        "                      __     __,\n",
        "                      \\,`~\"~` /\n",
        "      .-=-.           /    . .\\\n",
        "     / .-. \\          {  =    Y}=\n",
        "    (_/   \\ \\          \\      / \n",
        "           \\ \\        _/`'`'`b\n",
        "            \\ `.__.-'`        \\-._\n",
        "             |            '.__ `'-;_\n",
        "             |            _.' `'-.__)\n",
        "              \\    ;_..-`'/     //  \\\n",
        "              |   /  /   |     //    |\n",
        "              \\  \\ \\__)   \\   //    /\n",
        "               \\__)  \n",
        "\n",
        "MY HACKER NEWS ACCOUNT:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "-----------------------\n",
        "{u'created': 1413737356,\n",
        " u'delay': 0,\n",
        " u'id': u'testhater',\n",
        " u'karma': 1,\n",
        " u'submitted': [8478765, 8478763, 8478761, 8478759, 8478757]}\n",
        "\n",
        "\n",
        "MY HACKER NEWS POST:\n",
        "-----------------------\n",
        "{u'by': u'testhater',\n",
        " u'id': 8478757,\n",
        " u'parent': 8478079,\n",
        " u'text': u'nope. everything is terrible.',\n",
        " u'time': 1413737378,\n",
        " u'type': u'comment'}\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Building Out Some Functions to Ping The Hacker News API and Return Back Some Hate.\n",
      "The three main ones I use are to get the comment ID's accociated with a username, Get all the text from each of the comments and feed them into my model to get predictions, and finally run them through a function that gives me my \"hater score\"."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "# Get all of a user's comments\n",
      "def get_user_comments(username):\n",
      "    comments = []\n",
      "    \n",
      "    url_usr_strt = \"https://hacker-news.firebaseio.com/v0/user/\"\n",
      "    url_itm_strt = \"https://hacker-news.firebaseio.com/v0/item/\"\n",
      "    url_end = \".json\"\n",
      "    user = urlopen( url_usr_strt+username+url_end )\n",
      "    user = json.loads( user.read() )\n",
      "    if len(user['submitted']) > 100:\n",
      "        for c in user['submitted'][:101]:\n",
      "            item = urlopen( url_itm_strt+str(c)+url_end )\n",
      "            json_item = json.loads( item.read() )\n",
      "            if 'text' in json_item:\n",
      "                comments.append(smart_str(json_item['text']))\n",
      "    return comments\n",
      "\n",
      "# Get all of an item's comments\n",
      "def get_item_comments(itemid):\n",
      "    comments = []\n",
      "    url_itm_strt = \"https://hacker-news.firebaseio.com/v0/item/\"\n",
      "    url_end = \".json\"\n",
      "    item = urlopen( url_itm_strt+itemid+url_end )\n",
      "    item = json.loads( item.read() )\n",
      "    if 'kids' in item:\n",
      "        for c in item['kids']:\n",
      "            comment = urlopen( url_itm_strt+smart_str(c)+url_end )\n",
      "            json_comment = json.loads( comment.read() )\n",
      "            if 'text' in json_comment:\n",
      "                comments.append(json_comment['text'])\n",
      "            if 'kids' in json_comment:\n",
      "                for c in json_comment['kids']:\n",
      "                    kcomment = urlopen( url_itm_strt+smart_str(c)+url_end )\n",
      "                    kjson_comment = json.loads( kcomment.read() )\n",
      "                    if 'text' in kjson_comment:\n",
      "                        comments.append(kjson_comment['text'])\n",
      "                    if 'kids' in kjson_comment:\n",
      "                        for c in kjson_comment['kids']:\n",
      "                            k2comment = urlopen( url_itm_strt+smart_str(c)+url_end )\n",
      "                            k2json_comment = json.loads( k2comment.read() )\n",
      "                            if 'text' in k2json_comment:\n",
      "                                comments.append(k2json_comment['text'])\n",
      "                            if 'kids' in k2json_comment:\n",
      "                                for c in k2json_comment['kids']:\n",
      "                                    k3comment = urlopen( url_itm_strt+smart_str(c)+url_end )\n",
      "                                    k3json_comment = json.loads( k3comment.read() )\n",
      "                                    if 'text' in k2json_comment:\n",
      "                                        comments.append(k3json_comment['text'])\n",
      "\n",
      "    return comments\n",
      "\n",
      "# Get a final Hater Score. 100 is the worst, 0 is the best. \n",
      "def calculate_score(predictions):\n",
      "    total_score = []\n",
      "    for s in predictions:\n",
      "        total_score.append(s[1])\n",
      "    \n",
      "    return np.mean(total_score) * 100\n",
      "\n",
      "# Get all a users comments and run them through my model\n",
      "def user_score(username, my_vect, clf):\n",
      "    comments = filter(None, get_user_comments(username))\n",
      "    badwords = set(pd.read_csv('data/my_badwords.csv').words)\n",
      "    badwords_count = []\n",
      "\n",
      "    for el in comments:\n",
      "        tokens = el.split(' ')\n",
      "        badwords_count.append(len([i for i in tokens if i.lower() in badwords]))\n",
      "    \n",
      "    n_words = [len(c.split()) for c in comments]\n",
      "    allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in comments]\n",
      "    allcaps_ratio = np.array(allcaps) / np.array(n_words, dtype=np.float)\n",
      "    bad_ratio = np.array(badwords_count) / np.array(n_words, dtype=np.float)\n",
      "    exclamation = [c.count(\"!\") for c in comments]\n",
      "    addressing = [c.count(\"@\") for c in comments]\n",
      "    spaces = [c.count(\" \") for c in comments]\n",
      "\n",
      "    re_badwords = np.array(badwords_count).reshape((len(badwords_count),1))\n",
      "    re_n_words = np.array(n_words).reshape((len(badwords_count),1))\n",
      "    re_allcaps = np.array(allcaps).reshape((len(badwords_count),1))\n",
      "    re_allcaps_ratio = np.array(allcaps_ratio).reshape((len(badwords_count),1))\n",
      "    re_bad_ratio = np.array(bad_ratio).reshape((len(badwords_count),1))\n",
      "    re_exclamation = np.array(exclamation).reshape((len(badwords_count),1))\n",
      "    re_addressing = np.array(addressing).reshape((len(badwords_count),1))\n",
      "    re_spaces = np.array(spaces).reshape((len(badwords_count),1))\n",
      "    \n",
      "    vect = my_vect.transform(comments)\n",
      "    features = np.hstack((vect.todense(), re_badwords))\n",
      "    features = np.hstack((features, re_n_words))\n",
      "    features = np.hstack((features, re_allcaps))\n",
      "    features = np.hstack((features, re_allcaps_ratio))\n",
      "    features = np.hstack((features, re_bad_ratio))\n",
      "    features = np.hstack((features, re_exclamation))\n",
      "    features = np.hstack((features, re_addressing))\n",
      "    features = np.hstack((features, re_spaces))\n",
      "    predictions = clf.predict_proba(features)\n",
      "    \n",
      "    return calculate_score(predictions)\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Testing out my function. Put in a User Name here (example: 'pg', 'vegabook' or 'KevinMcAlear')\n",
      "username = 'CmonDev'\n",
      "# Here is there hater score! 100% is a pure hater.\n",
      "print \"The Hater Score of\",username,\":\",user_score(username, vect, clf),\"%\"\n",
      "print \n",
      "print \n",
      "\n",
      "# Here is there most reacent comments maxed out at 100.\n",
      "user_comments = get_user_comments(username)\n",
      "for c in user_comments:\n",
      "    print c\n",
      "    print \"---------------------\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The Hater Score of CmonDev : "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "4.53430404659 %\n",
        "\n",
        "\n",
        "The damn thing hasn&#x27;t even been released yet."
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "---------------------\n",
        "&quot;Unity brings the cutting edge graphics&quot; - huh?<p>Great initiative though.\n",
        "---------------------\n",
        "&quot;I&#x27;ve received nothing but positive feedback...&quot;<p>People are generally nice because it is considered polite.\n",
        "---------------------\n",
        "More like: &quot;What the fuck is wrong with this guy. He poses a complex ethical dilemma. It makes me think and thinking hurts my brain. I will better pretend he is some sort of creep.&quot;\n",
        "---------------------\n",
        "And why would an intelligent being with a free will still hang around?\n",
        "---------------------\n",
        "As calculated as the Nexus-muting &quot;leak&quot;.\n",
        "---------------------\n",
        "We are at the high of the HTML5&#x2F;JS hype cycle.\n",
        "---------------------\n",
        "If it&#x27;s just an expression than maybe &quot;forking&quot; is also just an expression with a random meaning? Either that or he never was a BDFL.\n",
        "---------------------\n",
        "\n",
        "---------------------\n",
        "Well, at least not something weak&#x2F;dynamic, so why not.\n",
        "---------------------\n",
        "The Visual Studio 2020 might be completely in-browser unfortunately :(.\n",
        "---------------------\n",
        "What is the out-of-box solution for adorner decorators, then?\n",
        "---------------------\n",
        "That simple pattern is already implemented:<p><a href=\"http://en.wikipedia.org/wiki/Reactor_pattern#Java\" rel=\"nofollow\">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Reactor_pattern#Java</a>\n",
        "---------------------\n",
        "&quot;useless organ&quot; - yet revolution did not remove it. Did modern science finally decide whether caffeine or fat are ultimately good&#x2F;bad? Because they seem to be changing their opinion every year.\n",
        "---------------------\n",
        "&gt; <i>So Atwood&#x27;s Law again.</i><p>Not really. There is a big difference between writing manually and transpiling into a dialect.\n",
        "---------------------\n",
        "Did you try the Voxatron one?\n",
        "---------------------\n",
        "Just share your games with us and you will get a plenty of useful feedback. It must be one of those multi-million titles - so exciting!\n",
        "---------------------\n",
        "Only a small comment: where are you going to put your views? Are you going to use separate folders for views and template views (similar to WPF data templates)?\n",
        "---------------------\n",
        "&gt; <i>&quot;I have even come to see the beauty in the ugly that is PHP.&quot;</i><p>I recommend stopping immediately and going through a 6 month of therapeutic Scala&#x2F;F#-only development.\n",
        "---------------------\n",
        "After reading and based on my experience: &quot;<i>Web</i> programming is terrible; so learn to avoid it. Focus on a nice server-side language (not PHP).&quot;\n",
        "---------------------\n",
        "Did you choose Node.js because you expect it to address the problem it is meant to address or for other reasons (fashion, learning, reduce hiring costs)?<p>Sorry, cannot comment on code too much as I don&#x27;t have TS experience. Probably not enough code to spot any obvious problems.\n",
        "---------------------\n",
        "\n",
        "---------------------\n",
        "&quot;... of Anglo-american ...&quot;\n",
        "---------------------\n",
        "&gt; <i>Yeah, those assembly-writing badasses making $500k at HFT funds are really hurting.</i><p>Yeah, all ~100 of them.<p>Better normal Java than it&#x27;s scripting counterpart though (the assembly of the web).\n",
        "---------------------\n",
        "At least don&#x27;t write the code in an assembly of your times :).\n",
        "---------------------\n",
        "Isn&#x27;t it better to solve tough niche problems while being on a salary or already rich and experienced instead of adding that to an already hard problem of starting up a business?\n",
        "---------------------\n",
        "They are pretty happy with Apple&#x27;s empire. You don&#x27;t need Microsoft&#x27;s hardware to build stuff for Windows though.\n",
        "---------------------\n",
        "But those guys were vesting not just resting. Salary is not same.\n",
        "---------------------\n",
        "They should assign them to work on an ultra-violent mini shooter with an unreasonable number of different weapons and monsters.\n",
        "---------------------\n",
        "Yes, I hate the ceremony. Oh the sacred ancient language that is so standard we can never touch it because Open Web!\n",
        "---------------------\n",
        "It&#x27;s a legacy language now. Time to let go.\n",
        "---------------------\n",
        "No, I think they dropped this since Vista. There is a pinch of old-school, you are right.\n",
        "---------------------\n",
        "<i>&quot;Most senior board positions in the corporate world are held by men&quot;</i><p>Why focus on gender? I think every board should have first-generation immigrants, people of different faiths, different races and genders and most importantly no more than 50% should come from rich backgrounds and prestigious universities.\n",
        "---------------------\n",
        "This is so sad. Why not say this:<p>&quot;I believe all <i>intelligent beings</i> should get equal pay for equal work.&quot;<p>This includes races, genders, being offshore&#x2F;onshore, being a shareholder or an employee etc.\n",
        "---------------------\n",
        "&quot;with my greatest creation&quot; - maybe he doesn&#x27;t want this to be his greatest creation? Hi is only 35.\n",
        "---------------------\n",
        "Did you at least like the design language they borrowed from Microsoft?\n",
        "---------------------\n",
        "That is thousands of proper native apps, that are available today.\n",
        "---------------------\n",
        "Unfortunately it is part of their push to lock down the web. Lock it down to legacy languages - JavaScript and HTML. Transpilation is of course not an answer, because it is not nearly same development experience. I hope mainstream platform providers and democratic solution providers such as Xamarin will keep on their good work in letting us use the languages we love, be it Scala, F#, C# on any other as beautiful.\n",
        "---------------------\n",
        "But it seems people capable of modern way of thinking start to emerge, e.g. Tim Sweeney of Epic Games:<p>&quot;\u2013 Will gladly sacrifice 10% of our performance\n",
        "for 10% higher productivity&quot;<p><a href=\"http://www.cs.princeton.edu/~dpw/popl/06/Tim-POPL.ppt\" rel=\"nofollow\">http:&#x2F;&#x2F;www.cs.princeton.edu&#x2F;~dpw&#x2F;popl&#x2F;06&#x2F;Tim-POPL.ppt</a>\n",
        "---------------------\n",
        "This is so true! When this type of developers starts talking about design of the architecture they start discussing optimization on second minute.<p>Here is a good example (it&#x27;s a good blog and people are talented though):<p><a href=\"http://bitsquid.blogspot.co.uk/2014/08/building-data-oriented-entity-system.html\" rel=\"nofollow\">http:&#x2F;&#x2F;bitsquid.blogspot.co.uk&#x2F;2014&#x2F;08&#x2F;building-data-oriente...</a>\n",
        "---------------------\n",
        "WP is an uncool but useful friend of MS now :(.\n",
        "---------------------\n",
        "There is a whole business of &#x27;integrating&#x27; the software products that are kind of ready (much more so than OSS) - ERP (Dynamics, SAP etc.).\n",
        "---------------------\n",
        "&quot;...if only people were not starving in the streets...&quot;<p>You have a bigger problem with IT then investment banking and international politics?<p>I am personally very happy that once in a while engineers are getting a <i>kind of</i> fair deal. Of course let&#x27;s not forget that salaries in Silicon Valley were artificially suppressed for a while in an illegal agreement between the likes of Google and Apple. So being underpaid is what we should be really complaining about.\n",
        "---------------------\n",
        "Yep, not so relevant now given the announcements since then.\n",
        "---------------------\n",
        "&quot;to the best developer tools&quot; - a bit far-fetched for very many of those.\n",
        "---------------------\n",
        "When I said &quot;managed&quot; I actually meant &quot;first-class CLR&#x2F;JVM&#x2F;Mono support&quot;.\n",
        "---------------------\n",
        "Yes, when thinking about a modern managed multi-paradigm, multi-typed, multi-platform language the choice is pretty much limited to Scala, C# and F#.\n",
        "---------------------\n",
        "If you mention that Angular was built by Google devs, then please mention that MVVM was invented by Microsoft devs (WPF):<p><a href=\"http://en.wikipedia.org/wiki/Model_View_ViewModel\" rel=\"nofollow\">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Model_View_ViewModel</a>\n",
        "---------------------\n",
        "Forgotten? This is the fifth time this year alone I am reading about both (Hopper is a bit overrated and part of military propaganda). Besides how could someone who knows about Babbage wouldn&#x27;t know about Lovelace?\n",
        "---------------------\n",
        "Such beauty: <a href=\"http://wtfjs.com\" rel=\"nofollow\">http:&#x2F;&#x2F;wtfjs.com</a>\n",
        "---------------------\n",
        "Well, both releases are nice, thanks for posting :)\n",
        "---------------------\n",
        "\n",
        "---------------------\n",
        "Kind of based on an assumption that using JavaScript at all is a good thing. For me the idea is to avoid using it where possible. Can we stop the web monopolisation&#x2F;locking&#x2F;despotism and drop specific languages from titles and ideas? How about &quot;Isomorphic Applications that give developers freedom and a choice of language&quot;?\n",
        "---------------------\n",
        "The only thing sadder would be programming a strong AI or brain-computer interfaces in javascript. Hopefully the disaster will be stopped way before.\n",
        "---------------------\n",
        "In a way any Python program is probabilistic.\n",
        "---------------------\n",
        "Oh, it&#x27;s JavaScript...\n",
        "---------------------\n",
        "I&#x27;d like tabs to go around the browser - not just the top edge.\n",
        "---------------------\n",
        "Wow, speech recognition in a Turing-complete language. Amazing.\n",
        "---------------------\n",
        "Well, as long as I don&#x27;t have to use HTML and JavaScript.\n",
        "---------------------\n",
        "Unfortunately, this is not a proper replacement for the Axum project that they killed for some lame reason:<p><a href=\"http://en.wikipedia.org/wiki/Axum_(programming_language)\" rel=\"nofollow\">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Axum_(programming_language)</a><p><a href=\"http://download.microsoft.com/download/B/D/5/BD51FFB2-C777-43B0-AC24-BDE3C88E231F/Axum%20Programmers%20Guide.pdf\" rel=\"nofollow\">http:&#x2F;&#x2F;download.microsoft.com&#x2F;download&#x2F;B&#x2F;D&#x2F;5&#x2F;BD51FFB2-C777-4...</a><p>Now people will be choosing Akka.NET instead.\n",
        "---------------------\n",
        "&quot;Light Table will continue to go on strong.&quot;<p>Sure.\n",
        "---------------------\n",
        "Good article, not too much anti-russian speech.<p>Also see the space laser handgun: <a href=\"http://englishrussia.com/2013/10/05/laser-gun-for-a-soviet-cosmonaut\" rel=\"nofollow\">http:&#x2F;&#x2F;englishrussia.com&#x2F;2013&#x2F;10&#x2F;05&#x2F;laser-gun-for-a-soviet-c...</a>\n",
        "---------------------\n",
        "Rust is meant to be practical, Haskell is more for academia (horrible ML syntax etc.).\n",
        "---------------------\n",
        "E.g. iOS lets me use: Objective C, Swift, C#, F#, Java, Scala. All compiling to lower-level code that is either native to platform or native to the language.<p>Web lets me use: only JavaScript (that is written manually or  transpiled).<p>Yes I will have to pay extra for some solutions, but I don&#x27;t mind doing this as opposed to using language that I hate.<p><a href=\"http://en.wikipedia.org/wiki/Ad_hominem\" rel=\"nofollow\">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Ad_hominem</a>\n",
        "---------------------\n",
        "It&#x27;s a choice between restricted standards that hinder technology innovation and restricted markets that hinder business.<p>As a technologist I hope that de-facto monopolic legacy poorly-designed technologies (JS and HTML) will give way to new better technologies.<p>I am very happy that at least mobile still gives developers many options - mostly without lame transpilation.\n",
        "---------------------\n",
        "Some dude said they are bad?\n",
        "---------------------\n",
        "Also two very well supported versions of VM with distinct advantages:<p>1) Mono - deploy to every major mobile platform using a well-defined single-vendor pipeline.<p>2) .NET - unmatched performance on Windows.\n",
        "---------------------\n",
        "JVM runs faster on some minor consumer platforms such as Linux, but that is pretty much it.\n",
        "---------------------\n",
        "Not sure what is so difficult: <a href=\"http://downloadvisualstudio.com\" rel=\"nofollow\">http:&#x2F;&#x2F;downloadvisualstudio.com</a>\n",
        "---------------------\n",
        "&quot;you&#x27;re not free to booby trap the path to your front door&quot;<p>In UK you can&#x27;t even booby trap <i>inside</i> of your house. Even killing someone who is trying to kill you is not recommended.\n",
        "---------------------\n",
        "Suggested name: Virtual Keyboards Club.\n",
        "---------------------\n",
        "In some way they are quite honest. You can see the offer very clearly: work with brogrammers for a low salary.\n",
        "---------------------\n",
        "TL;DR: because you need professionals for modern CG and programming, professionals deserve good pay and you need a lot of them, also considerable marketing is required for non-innovative games.\n",
        "---------------------\n",
        "I hope one day speed of trade execution will depend on beating cancer. Those guys will solve it in a month.\n",
        "---------------------\n",
        "Thanks for generalizing. I assume everyone in USA owns a shotgun and a trailer?\n",
        "---------------------\n",
        "&quot;humanizes the people of USSR&quot; - so you are saying they weren&#x27;t actual humans?\n",
        "---------------------\n",
        "&quot;monstrosity&quot; is not an explanation, CLR is a state of the art technology. What would you suggest instead of this? A dynamic language monstrosity? A JVM monstrosity? A legacy language (C++&#x2F;JS) monstrosity? An unestablished language monstrosity? Some other kind of monstrosity?\n",
        "---------------------\n",
        "&quot;...Minecraft is exactly the kind of game he&#x27;s talking about...&quot;<p>I guess having a flexible system of optional plots is better than no plot whatsoever.\n",
        "---------------------\n",
        "Making an issue<p>&quot;...in a market that moves very fast, saturated with product...&quot;<p>even worse<p>&quot;...there has been a counterpoint: the wave of DRM-free indies...&quot;<p>Look at Steam Greenlight.\n",
        "---------------------\n",
        "It has to do with wetware (biological computers).\n",
        "---------------------\n",
        "Since Microsoft doesn&#x27;t care about WPF anymore people started pretending they like HTML5 and JS, kinda like &quot;hey, dirt doesn&#x27;t taste <i>that</i> bad if you give it some time!&quot;.\n",
        "---------------------\n",
        "TL;DR version: <a href=\"https://roslyn.codeplex.com/wikipage?title=Language%20Feature%20Status&amp;referringTitle=Documentation\" rel=\"nofollow\">https:&#x2F;&#x2F;roslyn.codeplex.com&#x2F;wikipage?title=Language%20Featur...</a>\n",
        "---------------------\n",
        "The whole OOP vs FP argument is like food vs drinks argument. I am empowered and capable of using both. What is the problem?\n",
        "---------------------\n",
        "Staged demo proves a staged demo?\n",
        "---------------------\n",
        "Buy a tank. Travel the world. Meet interesting people and ... them.\n",
        "---------------------\n",
        "Very true, at least the rest of people were smart enough to keep evolving their concepts instead of going to mainstream same ol&#x27;.\n",
        "---------------------\n",
        "Now we just need to establish what &quot;adult&quot; means. Which country&#x27;s interpretation at which point of time in history do you like the best?\n",
        "---------------------\n",
        "The ad is an idea that someone knows what you need better than yourself. I prefer need-based research.\n",
        "---------------------\n",
        "It must be about some ideal world, where people become managers to do better good (as opposed to e.g. maximizing bonus or doing more talking than working).\n",
        "---------------------\n",
        "Random sub-second pauses on Nexus 7 2013.\n",
        "---------------------\n",
        "I guess learning to utilize all the available cores is more important.\n",
        "---------------------\n",
        "People are willing to pay for HTML5 games?!\n",
        "---------------------\n",
        "Practical answer: having to support one more country flag, one more ISO country code and maybe one more english flavour locale or a currency :).\n",
        "---------------------\n",
        "That&#x27;s the biggest problem with pure FRP: people like to talk about it and make libraries for it, but non-trivial game implementations are rare. I am still not sure what is supposed to be a unit of re-usability (like components&#x2F;systems in ECS) in FRP?\n",
        "---------------------\n",
        "Try finding the designers behind Microsoft&#x27;s Metro. Based on the amount of copying by iOS and Android since then, they must be one of the best.\n",
        "---------------------\n"
       ]
      }
     ],
     "prompt_number": 78
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Doing Everything Manually To One Time To Be Safe.\n",
      "Everything seemed to work great with my nice functions! But to be sure let's manually do what my functions just did to be sure."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Starting the transforming of my comments\n",
      "test = vect.transform(user_comments)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Running through my feature manipulation manually \n",
      "#  *NOTE: This time I printed the features out each transformation I do to them so you can kind of see how they change.\n",
      "\n",
      "p_badwords_count = []\n",
      "\n",
      "for el in user_comments:\n",
      "    tokens = el.split(' ')\n",
      "    p_badwords_count.append(len([i for i in tokens if i.lower() in badwords]))\n",
      "    \n",
      "p_n_words = [len(c.split()) for c in user_comments]\n",
      "p_allcaps = [np.sum([w.isupper() for w in comment.split()]) for comment in user_comments]\n",
      "p_allcaps_ratio = np.array(p_allcaps) / np.array(p_n_words, dtype=np.float)\n",
      "p_bad_ratio = np.array(p_badwords_count) / np.array(p_n_words, dtype=np.float)\n",
      "p_exclamation = [c.count(\"!\") for c in user_comments]\n",
      "p_addressing = [c.count(\"@\") for c in user_comments]\n",
      "p_spaces = [c.count(\" \") for c in user_comments]\n",
      "\n",
      "p_re_badwords = np.array(p_badwords_count).reshape((len(p_badwords_count),1))\n",
      "p_re_n_words = np.array(p_n_words).reshape((len(p_badwords_count),1))\n",
      "p_re_allcaps = np.array(p_allcaps).reshape((len(p_badwords_count),1))\n",
      "p_re_allcaps_ratio = np.array(p_allcaps_ratio).reshape((len(p_badwords_count),1))\n",
      "p_re_bad_ratio = np.array(p_bad_ratio).reshape((len(p_badwords_count),1))\n",
      "p_re_exclamation = np.array(p_exclamation).reshape((len(p_badwords_count),1))\n",
      "p_re_addressing = np.array(p_addressing).reshape((len(p_badwords_count),1))\n",
      "p_re_spaces = np.array(p_spaces).reshape((len(p_badwords_count),1))\n",
      "\n",
      "X_p_transform = np.hstack((test.todense(), p_re_badwords))\n",
      "print \"Adding In badwords\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_n_words))\n",
      "print \"Adding In n_words\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_allcaps))\n",
      "print \"Adding In allcaps\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_allcaps_ratio))\n",
      "print \"Adding In allcaps_ratio\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_bad_ratio))\n",
      "print \"Adding In bad_ratio\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_exclamation))\n",
      "print \"Adding In exclamation\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_addressing))\n",
      "print \"Adding In addressing\"\n",
      "print \"-----------------\"\n",
      "print X_test_transform\n",
      "print\n",
      "print\n",
      "X_p_transform = np.hstack((X_p_transform, p_re_spaces))\n",
      "print \"Adding In spaces\"\n",
      "print \"-----------------\"\n",
      "print X_p_transform\n",
      "print\n",
      "print"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Adding In badwords\n",
        "-----------------\n",
        "[[0 0 0 ..., 0 0 0]\n",
        " [0 0 0 ..., 0 0 0]\n",
        " [0 0 0 ..., 0 0 0]\n",
        " ..., \n",
        " [0 0 0 ..., 0 0 0]\n",
        " [0 0 0 ..., 0 0 0]\n",
        " [0 0 0 ..., 0 0 0]]\n",
        "\n",
        "\n",
        "Adding In n_words\n",
        "-----------------\n",
        "[[  0   0   0 ...,   0   0   8]\n",
        " [  0   0   0 ...,   0   0  21]\n",
        " [  0   0   0 ...,   0   0   3]\n",
        " ..., \n",
        " [  0   0   0 ...,   0   0  16]\n",
        " [  0   0   0 ...,   0   0 100]\n",
        " [  0   0   0 ...,   0   0  30]]\n",
        "\n",
        "\n",
        "Adding In allcaps\n",
        "-----------------\n",
        "[[   0.    0.    0. ...,    0.    8.    0.]\n",
        " [   0.    0.    0. ...,    0.   21.    3.]\n",
        " [   0.    0.    0. ...,    0.    3.    0.]\n",
        " ..., \n",
        " [   0.    0.    0. ...,    0.   16.    0.]\n",
        " [   0.    0.    0. ...,    0.  100.    1.]\n",
        " [   0.    0.    0. ...,    0.   30.    0.]]\n",
        "\n",
        "\n",
        "Adding In allcaps_ratio\n",
        "-----------------\n",
        "[[  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   8.00000000e+00\n",
        "    0.00000000e+00   0.00000000e+00]\n",
        " [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   2.10000000e+01\n",
        "    3.00000000e+00   1.42857143e-01]\n",
        " [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   3.00000000e+00\n",
        "    0.00000000e+00   0.00000000e+00]\n",
        " ..., \n",
        " [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   1.60000000e+01\n",
        "    0.00000000e+00   0.00000000e+00]\n",
        " [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   1.00000000e+02\n",
        "    1.00000000e+00   1.00000000e-02]\n",
        " [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   3.00000000e+01\n",
        "    0.00000000e+00   0.00000000e+00]]\n",
        "\n",
        "\n",
        "Adding In bad_ratio\n",
        "-----------------\n",
        "[[ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  3.          0.14285714  0.        ]\n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " ..., \n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  1.          0.01        0.        ]\n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]]\n",
        "\n",
        "\n",
        "Adding In exclamation\n",
        "-----------------\n",
        "[[ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  0.14285714  0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " ..., \n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  0.01        0.          0.        ]\n",
        " [ 0.          0.          0.         ...,  0.          0.          0.        ]]\n",
        "\n",
        "\n",
        "Adding In addressing"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "-----------------\n",
        "[[   0.    0.    0. ...,    0.    0.  335.]\n",
        " [   0.    0.    0. ...,    0.    1.   25.]\n",
        " [   0.    0.    0. ...,    0.    0.   21.]\n",
        " ..., \n",
        " [   0.    0.    0. ...,    0.    0.    7.]\n",
        " [   0.    0.    0. ...,    2.    0.   51.]\n",
        " [   0.    0.    0. ...,    0.    0.  187.]]\n",
        "\n",
        "\n",
        "Adding In spaces\n",
        "-----------------\n",
        "[[   0.    0.    0. ...,    0.    0.    7.]\n",
        " [   0.    0.    0. ...,    0.    0.   21.]\n",
        " [   0.    0.    0. ...,    0.    0.    2.]\n",
        " ..., \n",
        " [   0.    0.    0. ...,    0.    0.   16.]\n",
        " [   0.    0.    0. ...,    0.    0.  102.]\n",
        " [   0.    0.    0. ...,    0.    0.   29.]]\n",
        "\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 74
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Predicting the comments insult probaility\n",
      "test_predictions = clf.predict_proba( np.nan_to_num(X_p_transform)  )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 75
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Seeing what some predictions look like\n",
      "test_predictions[:5]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 76,
       "text": [
        "array([[ 0.93827726,  0.06172274],\n",
        "       [ 0.99678827,  0.00321173],\n",
        "       [ 0.93641375,  0.06358625],\n",
        "       [ 0.94203673,  0.05796327],\n",
        "       [ 0.99731428,  0.00268572]])"
       ]
      }
     ],
     "prompt_number": 76
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Finally Evaulating the hater score manually.\n",
      "print \"The Probability of\",username,\"being a hater is:\", calculate_score(test_predictions),\"%\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The Probability of pg being a hater is: 2.76497951931 %\n"
       ]
      }
     ],
     "prompt_number": 77
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Some Final Thoughts.\n",
      "* Sarcasm is hard to detect.\n",
      "* Using Pipelines when building my own meta-features is difficult.\n",
      "* Most people on Hacker News are pretty nice people.\n",
      "* I want to build in more training data from comments in Hacker News using the web app.\n",
      "* I want to build a Chrome app that adds in Twitter, Facebook, Reddit, etc. and overlay a post and a user \"hater score\" into the page your actually on."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}