assignment4.1_spark2.3_python3.6 (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 4\n",
    "\n",
    "Welcome to Assignment 4. This will be the most fun. Now we will prepare data for plotting.\n",
    "\n",
    "Just make sure you hit the play button on each cell from top to down. There are three functions you have to implement. Please also make sure than on each change on a function you hit the play button again on the corresponding cell to make it available to the rest of this notebook.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n",
    "\n",
    "In case you are facing issues, please read the following two documents first:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ\n",
    "\n",
    "Then, please feel free to ask:\n",
    "\n",
    "https://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n",
    "\n",
    "Please make sure to follow the guidelines before asking a question:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n",
    "\n",
    "\n",
    "If running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if ('sc' in locals() or 'sc' in globals()):\n",
    "    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==3.5.0 -q\n",
    "!pip install findspark -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from pyspark import SparkContext, SparkConf\n",
    "    from pyspark.sql import SparkSession\n",
    "except ImportError as e:\n",
    "    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n",
    "\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling is one of the most important things when it comes to visualization because often the data set gets so huge that you simply\n",
    "\n",
    "- can't copy all data to a local Spark driver (Watson Studio is using a \"local\" Spark driver)\n",
    "- can't throw all data at the plotting library\n",
    "\n",
    "Please implement a function which returns a 10% sample of a given data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getSample():\n",
    "    #TODO Please enter your code here, you are not required to use the template code below\n",
    "    sample10 = spark.sql(\"SELECT * FROM washing TABLESAMPLE (10 PERCENT)\")\n",
    "    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n",
    "    #https://spark.apache.org/docs/latest/api/sql/\n",
    "    return df10.show() #YOUR CODE GOES HERE(False,#YOUR CODE GOES HERE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to create a histogram and boxplot. Please ignore the sampling for now and return a python list containing all temperature values from the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getListForHistogramAndBoxPlot():\n",
    "    #TODO Please enter your code here, you are not required to use the template code below\n",
    "    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n",
    "    #https://spark.apache.org/docs/latest/api/sql/\n",
    "    my_list = spark.sql(\"select temperature from washing where temperature is not null\")\n",
    "        # SELECT #YOUR CODE GOES HERE from washing where temperature is not null\n",
    "    my_list.rdd.map(lambda row: row.temperature).sample(False,0.1).collect()\n",
    "    if not type(my_list)==list:\n",
    "        raise Exception('return type not a list')\n",
    "    return my_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we want to create a run chart. Please return two lists (encapsulated in a python tuple object) containing temperature and timestamp (ts) ordered by timestamp. Please refer to the following link to learn more about tuples in python: https://www.tutorialspoint.com/python/python_tuples.htm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#should return a tuple containing the two lists for timestamp and temperature\n",
    "#please make sure you take only 10% of the data by sampling\n",
    "#please also ensure that you sample in a way that the timestamp samples and temperature samples correspond (=> call sample on an object still containing both dimensions)\n",
    "def getListsForRunChart():\n",
    "    #TODO Please enter your code here, you are not required to use the template code below\n",
    "    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n",
    "    #https://spark.apache.org/docs/latest/api/sql/\n",
    "    result_rdd = spark.sql(\"select temperature,ts from washing where temperature is not null order by ts asc\")\n",
    "       # select #YOUR CODE GOES HERE,#YOUR CODE GOES HERE from washing where temperature is not null order by ts asc\n",
    "    double_tuple_rdd = result_rdd.rdd.sample(False,0.1).rdd.map(lambda row : (row.ts,row.temperature))\n",
    "    result_array_ts = double_tuple_rdd.map(lambda ts_temperature: ts_temperature[0]).collect()\n",
    "    result_array_temperature = double_tuple_rdd.map(lambda ts_temperature: ts_temperature[1]).collect()\n",
    "    return (result_array_ts,result_array_temperature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to grab a PARQUET file and create a dataframe out of it. Using SparkSQL you can handle it like a database. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "--2023-10-14 07:00:58--  https://github.com/IBM/coursera/blob/master/coursera_ds/washing.parquet?raw=true\n",
      "Resolving github.com (github.com)... 140.82.113.4\n",
      "Connecting to github.com (github.com)|140.82.113.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: https://github.com/claimed-framework/component-library/blob/master/coursera_ds/washing.parquet?raw=true [following]\n",
      "--2023-10-14 07:00:58--  https://github.com/claimed-framework/component-library/blob/master/coursera_ds/washing.parquet?raw=true\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github.com/claimed-framework/component-library/raw/master/coursera_ds/washing.parquet [following]\n",
      "--2023-10-14 07:00:58--  https://github.com/claimed-framework/component-library/raw/master/coursera_ds/washing.parquet\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ds/washing.parquet [following]\n",
      "--2023-10-14 07:00:58--  https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ds/washing.parquet\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8000::154, 2606:50c0:8003::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 112048 (109K) [application/octet-stream]\n",
      "Saving to: 'washing.parquet@raw=true.7'\n",
      "\n",
      "     0K .......... .......... .......... .......... .......... 45% 5.73M 0s\n",
      "    50K .......... .......... .......... .......... .......... 91% 42.4M 0s\n",
      "   100K .........                                             100%  180G=0.01s\n",
      "\n",
      "2023-10-14 07:00:58 (11.0 MB/s) - 'washing.parquet@raw=true.7' saved [112048/112048]\n",
      "\n",
      "'mv' is not recognized as an internal or external command,\n",
      "operable program or batch file.\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/blob/master/coursera_ds/washing.parquet?raw=true\n",
    "!mv washing.parquet?raw=true washing.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\n",
      "|                 _id|                _rev|count|flowrate|fluidlevel|frequency|hardness|speed|temperature|           ts|voltage|\n",
      "+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\n",
      "|0d86485d0f88d1f9d...|1-57940679fb8a713...|    4|      11|acceptable|     NULL|      77| NULL|        100|1547808723923|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-15ff3a0b304d789...|    2|    NULL|      NULL|     NULL|    NULL| 1046|       NULL|1547808729917|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-97c2742b68c7b07...|    4|    NULL|      NULL|       71|    NULL| NULL|       NULL|1547808731918|    236|\n",
      "|0d86485d0f88d1f9d...|1-eefb903dbe45746...|   19|      11|acceptable|     NULL|      75| NULL|         86|1547808738999|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-5f68b4c72813c25...|    7|    NULL|      NULL|       75|    NULL| NULL|       NULL|1547808740927|    235|\n",
      "|0d86485d0f88d1f9d...|1-cd4b6c57ddbe77e...|    5|    NULL|      NULL|     NULL|    NULL| 1014|       NULL|1547808744923|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-a35b25b5bf43aaf...|   32|      11|acceptable|     NULL|      73| NULL|         84|1547808752028|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-b717f7289a8476d...|   48|      11|acceptable|     NULL|      79| NULL|         84|1547808768065|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-c2f1f8fcf178b2f...|   18|    NULL|      NULL|       73|    NULL| NULL|       NULL|1547808773944|    228|\n",
      "|0d86485d0f88d1f9d...|1-15033dd9eebb4a8...|   59|      11|acceptable|     NULL|      72| NULL|         96|1547808779093|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-753dae825f9a6c2...|   62|      11|acceptable|     NULL|      73| NULL|         88|1547808782113|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-b168089f44f03f0...|   13|    NULL|      NULL|     NULL|    NULL| 1097|       NULL|1547808784940|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-403b687c6be0dea...|   23|    NULL|      NULL|       80|    NULL| NULL|       NULL|1547808788955|    236|\n",
      "|0d86485d0f88d1f9d...|1-195551e0455a24b...|   72|      11|acceptable|     NULL|      77| NULL|         87|1547808792134|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-060a39fc6c2ddee...|   26|    NULL|      NULL|       62|    NULL| NULL|       NULL|1547808797959|    233|\n",
      "|0d86485d0f88d1f9d...|1-2234514bffee465...|   27|    NULL|      NULL|       61|    NULL| NULL|       NULL|1547808800960|    226|\n",
      "|0d86485d0f88d1f9d...|1-4265898bb401db0...|   82|      11|acceptable|     NULL|      79| NULL|         96|1547808802154|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-2fbf7ca9a0425a0...|   94|      11|acceptable|     NULL|      73| NULL|         90|1547808814186|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-203c0ee6d7fbd21...|   97|      11|acceptable|     NULL|      77| NULL|         88|1547808817190|   NULL|\n",
      "|0d86485d0f88d1f9d...|1-47e1965db94fcab...|  104|      11|acceptable|     NULL|      75| NULL|         80|1547808824198|   NULL|\n",
      "+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.read.parquet('washing.parquet')\n",
    "df.createOrReplaceTempView('washing')\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we gonna test the functions you've completed and visualize the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "ename": "Py4JJavaError",
     "evalue": "An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 6) (DESKTOP-O3FMA23 executor driver): java.io.IOException: Cannot run program \"python3\": CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:328)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)\r\n\tat org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:141)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\r\nCaused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessImpl.create(Native Method)\r\n\tat java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)\r\n\tat java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)\r\n\t... 19 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1242)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1242)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1242)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3048)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:984)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\r\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:407)\r\n\tat org.apache.spark.rdd.RDD.collect(RDD.scala:1045)\r\n\tat org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)\r\n\tat org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)\r\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)\r\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)\r\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\r\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)\r\n\tat py4j.Gateway.invoke(Gateway.java:282)\r\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\r\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\r\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)\r\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:106)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\r\nCaused by: java.io.IOException: Cannot run program \"python3\": CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:328)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)\r\n\tat org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:141)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\t... 1 more\r\nCaused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessImpl.create(Native Method)\r\n\tat java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)\r\n\tat java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)\r\n\t... 19 more\r\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[22], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m plt\u001b[38;5;241m.\u001b[39mhist(\u001b[43mgetListForHistogramAndBoxPlot\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[0;32m      2\u001b[0m plt\u001b[38;5;241m.\u001b[39mshow()\n",
      "Cell \u001b[1;32mIn[9], line 7\u001b[0m, in \u001b[0;36mgetListForHistogramAndBoxPlot\u001b[1;34m()\u001b[0m\n\u001b[0;32m      5\u001b[0m my_list \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39msql(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mselect temperature from washing where temperature is not null\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m      6\u001b[0m     \u001b[38;5;66;03m# SELECT #YOUR CODE GOES HERE from washing where temperature is not null\u001b[39;00m\n\u001b[1;32m----> 7\u001b[0m \u001b[43mmy_list\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrdd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmap\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mlambda\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mrow\u001b[49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mrow\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtemperature\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msample\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[38;5;241;43m0.1\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcollect\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m      8\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mtype\u001b[39m(my_list)\u001b[38;5;241m==\u001b[39m\u001b[38;5;28mlist\u001b[39m:\n\u001b[0;32m      9\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mreturn type not a list\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "File \u001b[1;32mE:\\Program Files\\Python311\\Lib\\site-packages\\pyspark\\rdd.py:1833\u001b[0m, in \u001b[0;36mRDD.collect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1831\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m SCCallSiteSync(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontext):\n\u001b[0;32m   1832\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mctx\u001b[38;5;241m.\u001b[39m_jvm \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m-> 1833\u001b[0m     sock_info \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_jvm\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mPythonRDD\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcollectAndServe\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_jrdd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrdd\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1834\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mlist\u001b[39m(_load_from_socket(sock_info, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jrdd_deserializer))\n",
      "File \u001b[1;32mE:\\Program Files\\Python311\\Lib\\site-packages\\py4j\\java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[1;34m(self, *args)\u001b[0m\n\u001b[0;32m   1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[0;32m   1317\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[0;32m   1318\u001b[0m     args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[0;32m   1319\u001b[0m     proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[0;32m   1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[1;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m \u001b[43mget_return_value\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   1323\u001b[0m \u001b[43m    \u001b[49m\u001b[43manswer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgateway_client\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtarget_id\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[0;32m   1326\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
      "File \u001b[1;32mE:\\Program Files\\Python311\\Lib\\site-packages\\pyspark\\errors\\exceptions\\captured.py:179\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[1;34m(*a, **kw)\u001b[0m\n\u001b[0;32m    177\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mdeco\u001b[39m(\u001b[38;5;241m*\u001b[39ma: Any, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkw: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m    178\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 179\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mf\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43ma\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkw\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    180\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m Py4JJavaError \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m    181\u001b[0m         converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n",
      "File \u001b[1;32mE:\\Program Files\\Python311\\Lib\\site-packages\\py4j\\protocol.py:326\u001b[0m, in \u001b[0;36mget_return_value\u001b[1;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[0;32m    324\u001b[0m value \u001b[38;5;241m=\u001b[39m OUTPUT_CONVERTER[\u001b[38;5;28mtype\u001b[39m](answer[\u001b[38;5;241m2\u001b[39m:], gateway_client)\n\u001b[0;32m    325\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m answer[\u001b[38;5;241m1\u001b[39m] \u001b[38;5;241m==\u001b[39m REFERENCE_TYPE:\n\u001b[1;32m--> 326\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m Py4JJavaError(\n\u001b[0;32m    327\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAn error occurred while calling \u001b[39m\u001b[38;5;132;01m{0}\u001b[39;00m\u001b[38;5;132;01m{1}\u001b[39;00m\u001b[38;5;132;01m{2}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39m\n\u001b[0;32m    328\u001b[0m         \u001b[38;5;28mformat\u001b[39m(target_id, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m, name), value)\n\u001b[0;32m    329\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    330\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m Py4JError(\n\u001b[0;32m    331\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAn error occurred while calling \u001b[39m\u001b[38;5;132;01m{0}\u001b[39;00m\u001b[38;5;132;01m{1}\u001b[39;00m\u001b[38;5;132;01m{2}\u001b[39;00m\u001b[38;5;124m. Trace:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{3}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39m\n\u001b[0;32m    332\u001b[0m         \u001b[38;5;28mformat\u001b[39m(target_id, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m, name, value))\n",
      "\u001b[1;31mPy4JJavaError\u001b[0m: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 6) (DESKTOP-O3FMA23 executor driver): java.io.IOException: Cannot run program \"python3\": CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:328)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)\r\n\tat org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:141)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\r\nCaused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessImpl.create(Native Method)\r\n\tat java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)\r\n\tat java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)\r\n\t... 19 more\r\n\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779)\r\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1242)\r\n\tat org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1242)\r\n\tat scala.Option.foreach(Option.scala:407)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1242)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3048)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:984)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)\r\n\tat org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\r\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:407)\r\n\tat org.apache.spark.rdd.RDD.collect(RDD.scala:1045)\r\n\tat org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)\r\n\tat org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)\r\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)\r\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)\r\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\r\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)\r\n\tat py4j.Gateway.invoke(Gateway.java:282)\r\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\r\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\r\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)\r\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:106)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\r\nCaused by: java.io.IOException: Cannot run program \"python3\": CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:181)\r\n\tat org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:109)\r\n\tat org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)\r\n\tat org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:328)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)\r\n\tat org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:141)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)\r\n\tat org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\t... 1 more\r\nCaused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified\r\n\tat java.base/java.lang.ProcessImpl.create(Native Method)\r\n\tat java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)\r\n\tat java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)\r\n\tat java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)\r\n\t... 19 more\r\n"
     ]
    }
   ],
   "source": [
    "plt.hist(getListForHistogramAndBoxPlot())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.boxplot(getListForHistogramAndBoxPlot())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'result_rdd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[16], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m lists \u001b[38;5;241m=\u001b[39m \u001b[43mgetListsForRunChart\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "Cell \u001b[1;32mIn[10], line 10\u001b[0m, in \u001b[0;36mgetListsForRunChart\u001b[1;34m()\u001b[0m\n\u001b[0;32m      8\u001b[0m double_tuple_rdd \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39msql(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mselect temperature,ts from washing where temperature is not null order by ts asc\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m      9\u001b[0m    \u001b[38;5;66;03m# select #YOUR CODE GOES HERE,#YOUR CODE GOES HERE from washing where temperature is not null order by ts asc\u001b[39;00m\n\u001b[1;32m---> 10\u001b[0m double_tuple_rdd \u001b[38;5;241m=\u001b[39m \u001b[43mresult_rdd\u001b[49m\u001b[38;5;241m.\u001b[39mrdd\u001b[38;5;241m.\u001b[39msample(\u001b[38;5;28;01mFalse\u001b[39;00m,\u001b[38;5;241m0.1\u001b[39m)\u001b[38;5;241m.\u001b[39mrdd\u001b[38;5;241m.\u001b[39mmap(\u001b[38;5;28;01mlambda\u001b[39;00m row : (row\u001b[38;5;241m.\u001b[39mts,row\u001b[38;5;241m.\u001b[39mtemperature))\n\u001b[0;32m     11\u001b[0m result_array_ts \u001b[38;5;241m=\u001b[39m double_tuple_rdd\u001b[38;5;241m.\u001b[39mmap(\u001b[38;5;28;01mlambda\u001b[39;00m ts_temperature: ts_temperature[\u001b[38;5;241m0\u001b[39m])\u001b[38;5;241m.\u001b[39mcollect()\n\u001b[0;32m     12\u001b[0m result_array_temperature \u001b[38;5;241m=\u001b[39m double_tuple_rdd\u001b[38;5;241m.\u001b[39mmap(\u001b[38;5;28;01mlambda\u001b[39;00m ts_temperature: ts_temperature[\u001b[38;5;241m1\u001b[39m])\u001b[38;5;241m.\u001b[39mcollect()\n",
      "\u001b[1;31mNameError\u001b[0m: name 'result_rdd' is not defined"
     ]
    }
   ],
   "source": [
    "lists = getListsForRunChart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'lists' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[17], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m plt\u001b[38;5;241m.\u001b[39mplot(\u001b[43mlists\u001b[49m[\u001b[38;5;241m0\u001b[39m],lists[\u001b[38;5;241m1\u001b[39m])\n\u001b[0;32m      2\u001b[0m plt\u001b[38;5;241m.\u001b[39mxlabel(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtime\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m      3\u001b[0m plt\u001b[38;5;241m.\u001b[39mylabel(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtemperature\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[1;31mNameError\u001b[0m: name 'lists' is not defined"
     ]
    }
   ],
   "source": [
    "plt.plot(lists[0],lists[1])\n",
    "plt.xlabel(\"time\")\n",
    "plt.ylabel(\"temperature\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, you are done! The following code submits your solution to the grader. Again, please update your token from the grader's submission page on Coursera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rklib import submitAll\n",
    "import json\n",
    "\n",
    "key = \"S5PNoSHNEeisnA6YLL5C0g\"\n",
    "email = ###_YOUR_CODE_GOES_HERE_###\n",
    "token = ###_YOUR_CODE_GOES_HERE_### #you can obtain it from the grader page on Coursera (have a look here if you need more information on how to obtain the token https://youtu.be/GcDo0Rwe06U?t=276)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parts_data = {}\n",
    "parts_data[\"iLdHs\"] = json.dumps(str(type(getListForHistogramAndBoxPlot())))\n",
    "parts_data[\"xucEM\"] = json.dumps(len(getListForHistogramAndBoxPlot()))\n",
    "parts_data[\"IyH7U\"] = json.dumps(str(type(getListsForRunChart())))\n",
    "parts_data[\"MsMHO\"] = json.dumps(len(getListsForRunChart()[0]))\n",
    "\n",
    "submitAll(email, token, key, parts_data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}