2_Capstone Project - The Battle of Neighborhoods (Week 1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Capstone Project - The Battle of Neighborhoods (Week 1)\n",
    "### Applied Data Science Capstone by IBM/Coursera\n",
    "*Author: Henrique M. L. Pereira*\n",
    "\n",
    "This is the final part of Coursera Capstone Project. In this notebook I'll describe the problem at hand, \n",
    "explain why it is important, and explain how and where i'll retrieve the data needed to accomplish my objective.\n",
    "I am asked to use location data to explore geographical locations (New York, Toronto or another city of my choice)\n",
    "using Foursquare location data, to be creative and find a possible problem that can be solved with this approach."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Table of Contents\n",
    "* [Introduction](#introduction)\n",
    "* [Data and Methodology](#data)\n",
    "* [Data Exploration](#exploration)\n",
    "* [Results and Discussion](#discussion)\n",
    "* [Conclusion](#conclusion)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1. Introduction <a name=\"introduction\"></a>\n",
    "Everyone has faced or will face, some time of their life, a difficult decision to make: i.e. get or not get married, move\n",
    "away from your parent's house to your own place, have ou not have kids, enroll or not in that awesome course, etc...\n",
    "\n",
    "The situation I present might as well be a one of particular difficulty, mostly due to the challenge it may present. Let's\n",
    "imagine that you, as an entrepreneur, creative professional, and keen on producing innovative products, decide (along with\n",
    "some good and daring colleagues) to start a StartUp company. You are offered some help to establish an office in only 5 cities,\n",
    "all of them away from your hometown and also your colleagues' hometown.\n",
    "\n",
    "Let's define further our problem."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 1.1 Problem Definition <a name=\"problem\"></a>\n",
    "The 5 cities are major hubs in Portugal, diverse in all the ways, financial, economic, artistic, academic, full of opportunities, etc...\n",
    "The 5 cities were selected using data from the Nomad List, and are classified as good cities to establish a startup. These are 'LISBOA', 'PORTO', 'COIMBRA', 'BRAGA', 'AVEIRO'.\n",
    "\n",
    "Your StartUp, and it's work, may be defined by the following keywords:\n",
    "* Innovative;\n",
    "* Culturally impacting;\n",
    "* Environment friendly;\n",
    "* Information Technology;\n",
    "* Game changing;\n",
    "* Daring designs and approaches;\n",
    "\n",
    "You and your colleagues have the following concerns and needs (with no particular order):\n",
    "* Affordability of office and living;\n",
    "* Multicultural environment;\n",
    "* Near essential commodities;\n",
    "* Near places to relax and meet people\n",
    "* Always wanted to live in a buzzing city but without stressing situations;"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 1.2 Objective <a name=\"Objective\"></a>\n",
    "My objective is, with Data Science and Machine Learning, to advise the best city and neighborhood to set up said StartUp\n",
    "company based on the factors above specified, as well all the data from these cities that will be retrieved dinamically."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 2. Data <a name=\"data\"></a>\n",
    "Every data used in the project will be either stored in the resources folder of the GitHub repository associated with this project,\n",
    "or be available in the internet.\n",
    "\n",
    "I'll use the data related to the cities of Portugal from official Portuguese Government open source information. \n",
    "I'll also use data from the \"Nomad List\" ( https://nomadlist.com/ ) to rank the cities so I can put similarities and dissimilarities in perspective when comparing selected neighborhoods."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# import of relevant packages\n",
    "import pandas as pd\n",
    "import urllib.request\n",
    "import json\n",
    "from pandas.io.json import json_normalize\n",
    "import re\n",
    "from geopy.geocoders import Nominatim\n",
    "import geocoder\n",
    "import requests\n",
    "from bs4 import BeautifulSoup as bs\n",
    "import time\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "import folium\n",
    "from sklearn.cluster import KMeans\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.colors as colors\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)\n",
    "print('Libraries imported...')\n",
    "%matplotlib inline"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.1 Scrapping of data from Nomad List\n",
    "This data is intended to further help on selecting the best place for the startup. I'm expecting to have matches between\n",
    "neighborhoods of all cities, so this is the best objective way to select an appropriate neighborhood or at least the\n",
    "top best ones."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Define the list of Cities\n",
    "list_cities = ['lisbon', 'porto', 'coimbra', 'braga', 'aveiro']\n",
    "content = []\n",
    "col_data = []\n",
    "# Scrapping data\n",
    "try:\n",
    "    for i, c in enumerate(list_cities):\n",
    "        time.sleep(random.choice(np.linspace(1, 5))) # a random delayer to avoid blocking\n",
    "        req_parsed = bs(requests.get('https://nomadlist.com/{}'.format(c)).text, 'html.parser')\n",
    "        #Find relevant information and store it\n",
    "        table = req_parsed.find('table',{'class':'details'})\n",
    "        fullrows = []\n",
    "        for row in table.find_all('tr'):\n",
    "            col = row.find_all('td')\n",
    "            fullrows.append([row.text for row in col])\n",
    "            fullrows[-1][0] = re.sub('\\W', '', fullrows[-1][0])\n",
    "            fullrows[-1][1] = re.sub('\\W', ' ', fullrows[-1][1])\n",
    "        fullrows.append(['City', list_cities[i]])\n",
    "        row_data = np.array(fullrows).T.tolist()[1]\n",
    "        col_data = np.array(fullrows).T.tolist()[0]\n",
    "        fulldata = pd.DataFrame(row_data).T\n",
    "        fulldata.columns = col_data\n",
    "        content.append(fulldata)\n",
    "    print('Done!')\n",
    "except:\n",
    "    print('Something went wrong...')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# constructing final df\n",
    "NL_cities = pd.concat(content, axis=0, sort=False).reset_index(drop=True)\n",
    "NL_cities"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# now some data clean up\n",
    "for col in NL_cities.columns:\n",
    "    NL_cities[col] = NL_cities[col].fillna(NL_cities[col].dropna(axis=0).mode()[0])\n",
    "\n",
    "corresp = {'Great':5, 'Good':4, 'Okay':3, 'Bad':1}\n",
    "\n",
    "NL_cities['NomadScore'] = NL_cities['NomadScore'].astype('str').\\\n",
    "    replace(r'\\s\\d+\\s\\w+$', '', regex=True).\\\n",
    "    replace(r'\\s', '.', regex=True).astype('float')\n",
    "\n",
    "NL_cities['Cost'] = NL_cities['Cost'].astype('str').\\\n",
    "    replace(r'[A-z]\\s*', '', regex=True).replace(r'\\s', '', regex=True).astype('int')\n",
    "\n",
    "NL_cities['Internet'] = NL_cities['Internet'].astype('str').\\\n",
    "    replace(r'[A-z]\\s*', '', regex=True).astype('int')\n",
    "\n",
    "NL_cities['Temperaturenow'] = NL_cities['Temperaturenow'].astype('str').\\\n",
    "    replace(r'[^0-9]', '', regex=True).\\\n",
    "    replace(r'[0-9]{2}$', '', regex=True).astype('int')\n",
    "\n",
    "NL_cities['Humiditynow'] = NL_cities['Humiditynow'].astype('str').\\\n",
    "    replace(r'[^0-9]', '', regex=True).\\\n",
    "    replace(r'[0-9]{1}$', '', regex=True).astype('int')\n",
    "     \n",
    "NL_cities['Airqualitynow'] = NL_cities['Airqualitynow'].astype('str').\\\n",
    "    replace(r'[^0-9]{2}.', '', regex=True).astype('float')\n",
    "\n",
    "list_cols = ['Fun','Safety', 'Qualityoflife', 'Walkability', \n",
    "           'Peace', 'Trafficsafety', 'Hospitals', \n",
    "           'Happiness', 'Nightlife', 'FreeWiFiincity', \n",
    "           'Placestoworkfrom', 'ACorheating', \n",
    "           'Friendlytoforeigners', 'Englishspeaking', \n",
    "           'Freedomofspeech', 'Racialtolerance', \n",
    "           'Femalefriendly', 'LGBTfriendly', 'StartupScore']\n",
    "for col in list_cols:\n",
    "    NL_cities[col] = NL_cities[col].replace(corresp).astype('int')\n",
    "\n",
    "NL_cities['Peopledensity'] = NL_cities['Peopledensity'].astype('str').\\\n",
    "    replace(r'k.*', '000', regex=True).replace(r'\\w*\\s+', '', regex=True).astype('int')\n",
    "\n",
    "NL_cities = NL_cities.rename(columns={'Cost':'CostPerMonth',\n",
    "                                      'Internet':'Internet_Mbps_avg',\n",
    "                                      'Weather':'WeatherCelsiusNow',\n",
    "                                      'Airqualitynow':'Airqualitynow_mcg_cubicm',\n",
    "                                      'Peopledensity':'Peopledensity_sqrKm'})\n",
    "\n",
    "cols = NL_cities.columns.tolist()\n",
    "cols = cols[-1:] + cols[:-1]\n",
    "NL_cities = NL_cities[cols]\n",
    "NL_cities['City'] = NL_cities['City'].str.upper().replace('LISBON', 'LISBOA') "
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally I have a clean and final Dataframe about all the cities."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "NL_cities.to_csv('resources_battle_n/NL_cities.csv')\n",
    "NL_cities"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's scale data between 1 and 0 to use it further to rank these cities."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "NL_cities_norm = pd.DataFrame(scaler.fit(NL_cities.set_index('City')).transform(NL_cities.set_index('City')), columns=NL_cities.set_index('City').columns)\n",
    "NL_cities_norm['City'] = NL_cities['City']\n",
    "NL_cities_norm = NL_cities_norm.set_index('City').reset_index()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "NL_cities_norm"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "I'll remove now the variables that are not interesting to our startup:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "columns_to_remove = ['Internet_Mbps_avg','Nightlife','FreeWiFiincity','Placestoworkfrom','ACorheating','Temperaturenow','Humiditynow',\n",
    "                     'Airqualitynow_mcg_cubicm','Friendlytoforeigners','Englishspeaking','Freedomofspeech','Racialtolerance','Femalefriendly','LGBTfriendly',]\n",
    "NL_cities = NL_cities.drop(columns_to_remove, axis=1)\n",
    "NL_cities_norm = NL_cities_norm.drop(columns_to_remove, axis=1)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.2 Loading of Portuguese cities neighborhood's and location data\n",
    "This data was retrieved from official Portuguese government urls (available at https://dados.gov.pt/). The precise url's are presented in the code below"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "column_names = ['City', 'Neighborhood', 'Latitude', 'Longitude']\n",
    "\n",
    "# We'll start by getting the data from all the cities of portugal\n",
    "pt_nbh_url = 'https://dados.gov.pt/pt/datasets/r/2266425a-18ca-44a8-8655-9c39624c0ccb'\n",
    "\n",
    "with urllib.request.urlopen(pt_nbh_url) as url:\n",
    "    pt_nbh_url = json.load(url)\n",
    "\n",
    "# Check the needed columns and relevant data\n",
    "json_normalize(pt_nbh_url['d']).info()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# let's save data from 'entidade' and 'codigopostal' to a Dataframe\n",
    "# I will want only the cities named in the introduction\n",
    "m_target = ['LISBOA', 'PORTO', 'COIMBRA', 'BRAGA', 'AVEIRO']\n",
    "PT_neighborhoods = pd.DataFrame(columns=['City', 'Neighborhood', 'PC'])\n",
    "\n",
    "for data in pt_nbh_url['d']:\n",
    "    city = re.sub('^(.)*(?=\\s\\(.)\\s\\(', '', # regex to clean the name\n",
    "                     data[u'entidade']).strip(')')\n",
    "    neighborhood_name = re.sub('\\s\\((.*)', '', # regex to clean the name of neighborhoods\n",
    "                               data[u'entidade'])\n",
    "    postalcode = data[u'codigopostal']\n",
    "\n",
    "    PT_neighborhoods = PT_neighborhoods.append({'City': city,\n",
    "                                          'Neighborhood': neighborhood_name,\n",
    "                                          'PC': postalcode}, ignore_index=True)\n",
    "\n",
    "PT_neighborhoods = PT_neighborhoods[PT_neighborhoods['City'].isin(m_target)].reset_index().iloc[:,1:]\n",
    "\n",
    "# Check of a sample from the resulting DataFrame\n",
    "PT_neighborhoods.sample(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Now I build a function to extract and store coordinates in a DataFrame and by using the geocoder package to search for coordinates based on portuguese postal code\n",
    "def find_coordinates_pt(place, nbh, postal_code, df):\n",
    "    coordinates_df = pd.DataFrame(columns=('PC','Latitude', 'Longitude'))\n",
    "    print('Retrieving coordinates: ', end='')\n",
    "    \n",
    "    for p, n, post in zip(df[place], df[nbh], df[postal_code]): # For every line\n",
    "        try:\n",
    "            l = geocoder.arcgis(n + ', ' + p + ', ' + post + ', PT').latlng \n",
    "            coordinates_df = coordinates_df.append({\n",
    "                'PC':post,\n",
    "                'Latitude':l[0],\n",
    "                'Longitude':l[1]\n",
    "            }, ignore_index=True)\n",
    "            print(' .', end='')\n",
    "        except:\n",
    "            l = geocoder.arcgis(post + ', PT').latlng #\n",
    "            coordinates_df = coordinates_df.append({\n",
    "                'PC':post,\n",
    "                'Latitude':l[0],\n",
    "                'Longitude':l[1]\n",
    "            }, ignore_index=True)\n",
    "            print(' o', end='')\n",
    "    \n",
    "    print('End of function!')\n",
    "    \n",
    "    return pd.merge(df, coordinates_df, on=postal_code, how='inner') # merge of the 2 dataframes on the key 'postal code'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Let's create our final dataset for the locations\n",
    "Neighborhoods = find_coordinates_pt('City', 'Neighborhood', 'PC', PT_neighborhoods).drop('PC', axis=1)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now our dataframe is built with the cities we want, their neighborhoods and coordinates, so let's take a look at it:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "Neighborhoods.sample(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Check if every location was returned a value of latitude and longitude\n",
    "print('Found', sum(Neighborhoods.Latitude.isna()), 'Latitude values missing from Dataframe, and',\n",
    "      sum(Neighborhoods.Longitude.isna()), 'Longitude values missing from Dataframe.')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Save the dataframe as a csv file in the resources folder\n",
    "Neighborhoods.to_csv('resources_battle_n/Neighborhoods.csv')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.3 Retrieval of Venue Data from FourSquare\n",
    "I'll retrieve the data of the Venues near each coordinate using the FourSquare API, and concatenate this information into another\n",
    "Dataframe.\n",
    "\n",
    "Now I'll define the required Foursquare Credentials and Version to use its request API url"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "CLIENT_ID = '4QFEINHYB35HFDKHZQYHUSKX5UWGPTCFUCMXSIRHS5CQJJ51'\n",
    "CLIENT_SECRET = 'XUIM0KSEAAPNBMC0XXDK3QMS3XDYEHGRUG1VABVMKJMVTSCI'\n",
    "VERSION = '20190618'\n",
    "LIMIT = 100 # number of maximum venues to be returned for every neighborhood\n",
    "RADIUS = 500 # radius in meters to search for venues"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "I'll create a function to repeat the process of pulling all venues for all the neighborhoods"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "def get_nearby_venues(cities, names, latitudes, longitudes, radius=RADIUS, limit=LIMIT):\n",
    "    venues_list=[]\n",
    "    print('Retrieving venues: ', end='')\n",
    "    for city, name, lat, lng in zip(cities, names, latitudes, longitudes):\n",
    "        # build of the API request URL\n",
    "        url = 'https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(\n",
    "            CLIENT_ID, \n",
    "            CLIENT_SECRET, \n",
    "            VERSION, \n",
    "            lat, \n",
    "            lng, \n",
    "            radius, \n",
    "            limit)\n",
    "        # make the GET request\n",
    "        try:\n",
    "            results = requests.get(url).json()[\"response\"]['groups'][0]['items']\n",
    "            # return only relevant information for each nearby venue\n",
    "            venues_list.append([(\n",
    "                city,\n",
    "                name, \n",
    "                lat, \n",
    "                lng, \n",
    "                v['venue']['id'],\n",
    "                v['venue']['name'], \n",
    "                v['venue']['location']['lat'], \n",
    "                v['venue']['location']['lng'],  \n",
    "                v['venue']['categories'][0]['id'],  \n",
    "                v['venue']['categories'][0]['name']) for v in results])\n",
    "            print(' .', end='')\n",
    "        except KeyError:\n",
    "            print('KeyError... The API is not responding or max requests per day maxed out...')\n",
    "            break\n",
    "        \n",
    "    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])\n",
    "    nearby_venues.columns = ['City',\n",
    "                  'Neighborhood', \n",
    "                  'Neighborhood Latitude', \n",
    "                  'Neighborhood Longitude',\n",
    "                  'VenueID',\n",
    "                  'Venue', \n",
    "                  'Venue Latitude', \n",
    "                  'Venue Longitude',\n",
    "                  'Venue CategoryID',\n",
    "                  'Venue Category']\n",
    "    \n",
    "    \n",
    "    # this will return the parent categories for each venue child category\n",
    "    url = 'https://api.foursquare.com/v2/venues/categories?&client_id={}&client_secret={}&v={}'.format(\n",
    "            CLIENT_ID, \n",
    "            CLIENT_SECRET, \n",
    "            VERSION,)\n",
    "\n",
    "    categories_results = requests.get(url).json()['response']['categories']\n",
    "    \n",
    "    fs_categories = json_normalize(data=categories_results, \n",
    "                               record_path='categories', \n",
    "                               record_prefix='child_', \n",
    "                               meta=['name'])[['name', 'child_name', 'child_id']]\n",
    "    \n",
    "    nearby_venues = pd.merge(nearby_venues, fs_categories, left_on='Venue CategoryID', right_on='child_id')\\\n",
    "        .drop(['Venue CategoryID', 'child_name', 'child_id'], axis=1)\\\n",
    "        .rename(columns={'name':'ParentCategory'})\n",
    "    \n",
    "    print('End of function!')\n",
    "    \n",
    "    return nearby_venues"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now I'll retrieve, with the above function, every venue for each neighborhood and \n",
    "create a new dataframe with this information."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_venues = get_nearby_venues(\n",
    "                cities = Neighborhoods['City'],\n",
    "                names = Neighborhoods['Neighborhood'],\n",
    "                latitudes = Neighborhoods['Latitude'],\n",
    "                longitudes = Neighborhoods['Longitude'])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The resulting Dataframe was created and now I'll check its shape and information about the columns retrieved"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_venues.to_csv('resources_battle_n/all_venues_retrieved.csv')\n",
    "all_venues.shape"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_venues.info()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_venues = pd.read_csv('resources_battle_n/all_venues_retrieved.csv').iloc[:,1:]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "I want to check how many venues were retrived by city"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_venues.groupby(by='City').count()['Venue']"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.4 Methodology\n",
    "For this problem I will first make a data exploratory analisys in order to find hidden patterns among the data, and also check if all selected cities have venues that in some way are very different one from another.\n",
    "\n",
    "Next I'll apply 2 machine learning techniques (unsupervised), firstly K-Means, and OPTICS, in this order. My intention is to validade clusters found by k-means with the capability of finding outliers provided by OPTICS (in a similar fashion as DBSCAN)\n",
    "\n",
    "After selecting the best suited neighborhoods with the appropriate venues for our startup, I´ll use the data from Nomad List to rank these a little bit further to narrow down the eligible neighborhoods so that the clients can have an easy task selecting the best one to establish their startup.\n",
    "\n",
    "In all steps I'll try to make this notebook as dynamical and automated as possible, so the results can be repeated and even reflect changes in the future."
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}