3.1CurationFlagURL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 3.1 - Curation-flag-URL\n",
    "\n",
    "This notebook analyses the URL-based fields of the SSH Open Marketplace and writes back to the system via two dedicated curation properties: `curation-flag-url` and `curation-detail` properties.\n",
    "\n",
    "This notebook flags Marketplace items that have errors in their URL-based fields, helping Moderators identify curation priorities to improve data quality. \n",
    "\n",
    "This notebook is part of a series of 4 notebooks that inform the curation properties used in the SSH Open Marketplace Editorial Dashboard.\n",
    "\n",
    "It is composed of 4 sections:\n",
    "\n",
    "0. Requirements to run the notebook\n",
    "1. Check & flag error values in `accessibleAt`\n",
    "2. Check & flag error values in URL-based properties\n",
    "3. Check & flag error values in URL-based properties for a given source - example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 Requirements to run this notebook\n",
    "\n",
    "This section gives all the relevant information to \"interact\" with the MP data.\n",
    "\n",
    "### 0.1 libraries\n",
    "*There are a number of external libraries needed to run the notebook* \n",
    "\n",
    "*Furthermore, a dedicated SSH Open Marketplace library - sshmarketplacelib - with customised functions has been created and can be imported using the python import commands.* \n",
    "\n",
    "*Below the libraries import needed to run this notebook*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "#import the MarketPlace Library \n",
    "from sshmarketplacelib import MPData as mpd\n",
    "from sshmarketplacelib import  eval as eva, helper as hel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.2 Get the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n"
     ]
    }
   ],
   "source": [
    "mpdata = mpd()\n",
    "df_tool_flat =mpdata.getMPItems (\"toolsandservices\", True)\n",
    "df_publication_flat =mpdata.getMPItems (\"publications\", True)\n",
    "df_trainingmaterials_flat =mpdata.getMPItems (\"trainingmaterials\", True)\n",
    "df_workflows_flat =mpdata.getMPItems (\"workflows\", True)\n",
    "df_datasets_flat =mpdata.getMPItems (\"datasets\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.3 A look at the data\n",
    "\n",
    "df_all_items.head() will show the first 5 rows of the dataframe\n",
    "\n",
    "df_all_items.tail() will show the 5 last rows of the dataframe\n",
    "\n",
    "df_all_items.shape will give the dataframe shape (number of rows and columns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "      <th>persistentId</th>\n",
       "      <th>lastInfoUpdate</th>\n",
       "      <th>status</th>\n",
       "      <th>description</th>\n",
       "      <th>contributors</th>\n",
       "      <th>properties</th>\n",
       "      <th>externalIds</th>\n",
       "      <th>...</th>\n",
       "      <th>thumbnail.info.mediaId</th>\n",
       "      <th>thumbnail.info.category</th>\n",
       "      <th>thumbnail.info.filename</th>\n",
       "      <th>thumbnail.info.mimeType</th>\n",
       "      <th>thumbnail.info.hasThumbnail</th>\n",
       "      <th>thumbnail.info.location.sourceUrl</th>\n",
       "      <th>thumbnail.caption</th>\n",
       "      <th>dateCreated</th>\n",
       "      <th>dateLastUpdated</th>\n",
       "      <th>composedOf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28230</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>140kit</td>\n",
       "      <td>SIU1nO</td>\n",
       "      <td>2021-11-23T17:24:25+0000</td>\n",
       "      <td>approved</td>\n",
       "      <td>140kit provides a management layer for tweet c...</td>\n",
       "      <td>[{'actor': {'id': 2224, 'name': 'Ian Pearce, D...</td>\n",
       "      <td>[{'type': {'code': 'mode-of-use', 'label': 'Mo...</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>36324</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DF Zephyr - photogrammetry software - 3d mode...</td>\n",
       "      <td>4gDAHv</td>\n",
       "      <td>2022-01-13T11:49:02+0000</td>\n",
       "      <td>approved</td>\n",
       "      <td>3DF Zephyr\\[1\\]\\[2\\] is a commercial photogram...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'type': {'code': 'language', 'label': 'Langu...</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36552</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DHOP</td>\n",
       "      <td>UcxOmD</td>\n",
       "      <td>2022-01-13T11:50:31+0000</td>\n",
       "      <td>approved</td>\n",
       "      <td>3DHOP (3D Heritage Online Presenter) is an ope...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'type': {'code': 'language', 'label': 'Langu...</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36555</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DHOP: 3D Heritage Online Presenter</td>\n",
       "      <td>uFIMPQ</td>\n",
       "      <td>2022-01-13T11:50:32+0000</td>\n",
       "      <td>approved</td>\n",
       "      <td>No description provided.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>36189</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DReshaper \\| 3DReshaper</td>\n",
       "      <td>kAkzuz</td>\n",
       "      <td>2022-01-13T11:47:44+0000</td>\n",
       "      <td>approved</td>\n",
       "      <td>No description provided.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'type': {'code': 'language', 'label': 'Langu...</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      id         category                                              label  \\\n",
       "0  28230  tool-or-service                                             140kit   \n",
       "1  36324  tool-or-service  3DF Zephyr - photogrammetry software - 3d mode...   \n",
       "2  36552  tool-or-service                                              3DHOP   \n",
       "3  36555  tool-or-service                3DHOP: 3D Heritage Online Presenter   \n",
       "4  36189  tool-or-service                           3DReshaper \\| 3DReshaper   \n",
       "\n",
       "  persistentId            lastInfoUpdate    status  \\\n",
       "0       SIU1nO  2021-11-23T17:24:25+0000  approved   \n",
       "1       4gDAHv  2022-01-13T11:49:02+0000  approved   \n",
       "2       UcxOmD  2022-01-13T11:50:31+0000  approved   \n",
       "3       uFIMPQ  2022-01-13T11:50:32+0000  approved   \n",
       "4       kAkzuz  2022-01-13T11:47:44+0000  approved   \n",
       "\n",
       "                                         description  \\\n",
       "0  140kit provides a management layer for tweet c...   \n",
       "1  3DF Zephyr\\[1\\]\\[2\\] is a commercial photogram...   \n",
       "2  3DHOP (3D Heritage Online Presenter) is an ope...   \n",
       "3                           No description provided.   \n",
       "4                           No description provided.   \n",
       "\n",
       "                                        contributors  \\\n",
       "0  [{'actor': {'id': 2224, 'name': 'Ian Pearce, D...   \n",
       "1                                                 []   \n",
       "2                                                 []   \n",
       "3                                                 []   \n",
       "4                                                 []   \n",
       "\n",
       "                                          properties externalIds  ...  \\\n",
       "0  [{'type': {'code': 'mode-of-use', 'label': 'Mo...          []  ...   \n",
       "1  [{'type': {'code': 'language', 'label': 'Langu...          []  ...   \n",
       "2  [{'type': {'code': 'language', 'label': 'Langu...          []  ...   \n",
       "3                                                 []          []  ...   \n",
       "4  [{'type': {'code': 'language', 'label': 'Langu...          []  ...   \n",
       "\n",
       "  thumbnail.info.mediaId thumbnail.info.category thumbnail.info.filename  \\\n",
       "0                    NaN                     NaN                     NaN   \n",
       "1                    NaN                     NaN                     NaN   \n",
       "2                    NaN                     NaN                     NaN   \n",
       "3                    NaN                     NaN                     NaN   \n",
       "4                    NaN                     NaN                     NaN   \n",
       "\n",
       "  thumbnail.info.mimeType  thumbnail.info.hasThumbnail  \\\n",
       "0                     NaN                          NaN   \n",
       "1                     NaN                          NaN   \n",
       "2                     NaN                          NaN   \n",
       "3                     NaN                          NaN   \n",
       "4                     NaN                          NaN   \n",
       "\n",
       "  thumbnail.info.location.sourceUrl thumbnail.caption dateCreated  \\\n",
       "0                               NaN               NaN         NaN   \n",
       "1                               NaN               NaN         NaN   \n",
       "2                               NaN               NaN         NaN   \n",
       "3                               NaN               NaN         NaN   \n",
       "4                               NaN               NaN         NaN   \n",
       "\n",
       "  dateLastUpdated composedOf  \n",
       "0             NaN        NaN  \n",
       "1             NaN        NaN  \n",
       "2             NaN        NaN  \n",
       "3             NaN        NaN  \n",
       "4             NaN        NaN  \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all_items=pd.concat([df_tool_flat, df_publication_flat, df_trainingmaterials_flat, df_workflows_flat, df_datasets_flat])\n",
    "df_all_items.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df_all_items_work` selects the columns/attributes of interest "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>persistentId</th>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "      <th>contributors</th>\n",
       "      <th>accessibleAt</th>\n",
       "      <th>source.label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>12634</td>\n",
       "      <td>l8gLBb</td>\n",
       "      <td>dataset</td>\n",
       "      <td>Yelp Academic Challenge Dataset</td>\n",
       "      <td>[{'actor': {'id': 1752, 'name': 'Eva Bacas', '...</td>\n",
       "      <td>[https://www.yelp.com/dataset]</td>\n",
       "      <td>Humanities Data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>12631</td>\n",
       "      <td>xvYQQ4</td>\n",
       "      <td>dataset</td>\n",
       "      <td>YelpCHI</td>\n",
       "      <td>[{'actor': {'id': 1752, 'name': 'Eva Bacas', '...</td>\n",
       "      <td>[http://odds.cs.stonybrook.edu/yelpchi-dataset/]</td>\n",
       "      <td>Humanities Data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305</th>\n",
       "      <td>12632</td>\n",
       "      <td>IdZGtV</td>\n",
       "      <td>dataset</td>\n",
       "      <td>YelpNYC</td>\n",
       "      <td>[{'actor': {'id': 1752, 'name': 'Eva Bacas', '...</td>\n",
       "      <td>[http://odds.cs.stonybrook.edu/yelpnyc-dataset/]</td>\n",
       "      <td>Humanities Data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>12633</td>\n",
       "      <td>OMny6U</td>\n",
       "      <td>dataset</td>\n",
       "      <td>YelpZIP</td>\n",
       "      <td>[{'actor': {'id': 1752, 'name': 'Eva Bacas', '...</td>\n",
       "      <td>[http://odds.cs.stonybrook.edu/yelpzip-dataset/]</td>\n",
       "      <td>Humanities Data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>12589</td>\n",
       "      <td>YnEaU0</td>\n",
       "      <td>dataset</td>\n",
       "      <td>\"You Are Where You Tweet: A Content-Based Appr...</td>\n",
       "      <td>[{'actor': {'id': 1752, 'name': 'Eva Bacas', '...</td>\n",
       "      <td>[https://archive.org/details/twitter_cikm_2010]</td>\n",
       "      <td>Humanities Data</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id persistentId category  \\\n",
       "303  12634       l8gLBb  dataset   \n",
       "304  12631       xvYQQ4  dataset   \n",
       "305  12632       IdZGtV  dataset   \n",
       "306  12633       OMny6U  dataset   \n",
       "307  12589       YnEaU0  dataset   \n",
       "\n",
       "                                                 label  \\\n",
       "303                    Yelp Academic Challenge Dataset   \n",
       "304                                            YelpCHI   \n",
       "305                                            YelpNYC   \n",
       "306                                            YelpZIP   \n",
       "307  \"You Are Where You Tweet: A Content-Based Appr...   \n",
       "\n",
       "                                          contributors  \\\n",
       "303  [{'actor': {'id': 1752, 'name': 'Eva Bacas', '...   \n",
       "304  [{'actor': {'id': 1752, 'name': 'Eva Bacas', '...   \n",
       "305  [{'actor': {'id': 1752, 'name': 'Eva Bacas', '...   \n",
       "306  [{'actor': {'id': 1752, 'name': 'Eva Bacas', '...   \n",
       "307  [{'actor': {'id': 1752, 'name': 'Eva Bacas', '...   \n",
       "\n",
       "                                         accessibleAt     source.label  \n",
       "303                    [https://www.yelp.com/dataset]  Humanities Data  \n",
       "304  [http://odds.cs.stonybrook.edu/yelpchi-dataset/]  Humanities Data  \n",
       "305  [http://odds.cs.stonybrook.edu/yelpnyc-dataset/]  Humanities Data  \n",
       "306  [http://odds.cs.stonybrook.edu/yelpzip-dataset/]  Humanities Data  \n",
       "307   [https://archive.org/details/twitter_cikm_2010]  Humanities Data  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all_items_work=df_all_items[['id', 'persistentId', 'category', 'label', 'contributors', 'accessibleAt', 'source.label']]\n",
    "df_all_items_work.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Check & flag values in `accessibleAt`\n",
    "\n",
    "`accessibleAt` is the main URL field of MP items.\n",
    "\n",
    "The following cell checks if there are empty values in `accessibleAt` for all items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " There are 526 items without accessibleAt URLs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_all_items_work_emptyurls=df_all_items_work[df_all_items_work['accessibleAt'].str.len()==0]\n",
    "\n",
    "emptyurldescriptionsn=df_all_items_work_emptyurls.count()[0]\n",
    "\n",
    "print(f'\\n There are {emptyurldescriptionsn} items without accessibleAt URLs\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Check the validity of URLs in the accessibleAt property using the HTTP Result Status code\n",
    "\n",
    "The code below explicitly execute an http call for every URL, waits for the [Result Status Code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) of the call and then registers the code.\n",
    "Depending on connections and server answer times it may take several minutes to process all URLs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inspecting accessibleAt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MPUrl</th>\n",
       "      <th>persistentId</th>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "      <th>property</th>\n",
       "      <th>url</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tool-or-service/SIU1nO</td>\n",
       "      <td>SIU1nO</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>140kit</td>\n",
       "      <td>accessibleAt</td>\n",
       "      <td>https://github.com/WebEcologyProject/140kit</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tool-or-service/4gDAHv</td>\n",
       "      <td>4gDAHv</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DF Zephyr - photogrammetry software - 3d mode...</td>\n",
       "      <td>accessibleAt</td>\n",
       "      <td>https://www.3dflow.net/3df-zephyr-pro-3d-model...</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tool-or-service/UcxOmD</td>\n",
       "      <td>UcxOmD</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DHOP</td>\n",
       "      <td>accessibleAt</td>\n",
       "      <td>http://vcg.isti.cnr.it/3dhop/</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tool-or-service/uFIMPQ</td>\n",
       "      <td>uFIMPQ</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DHOP: 3D Heritage Online Presenter</td>\n",
       "      <td>accessibleAt</td>\n",
       "      <td>https://github.com/cnr-isti-vclab/3DHOP</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tool-or-service/kAkzuz</td>\n",
       "      <td>kAkzuz</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>3DReshaper \\| 3DReshaper</td>\n",
       "      <td>accessibleAt</td>\n",
       "      <td>https://www.3dreshaper.com/en/</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    MPUrl persistentId         category  \\\n",
       "0  tool-or-service/SIU1nO       SIU1nO  tool-or-service   \n",
       "1  tool-or-service/4gDAHv       4gDAHv  tool-or-service   \n",
       "2  tool-or-service/UcxOmD       UcxOmD  tool-or-service   \n",
       "3  tool-or-service/uFIMPQ       uFIMPQ  tool-or-service   \n",
       "4  tool-or-service/kAkzuz       kAkzuz  tool-or-service   \n",
       "\n",
       "                                               label      property  \\\n",
       "0                                             140kit  accessibleAt   \n",
       "1  3DF Zephyr - photogrammetry software - 3d mode...  accessibleAt   \n",
       "2                                              3DHOP  accessibleAt   \n",
       "3                3DHOP: 3D Heritage Online Presenter  accessibleAt   \n",
       "4                           3DReshaper \\| 3DReshaper  accessibleAt   \n",
       "\n",
       "                                                 url status  \n",
       "0        https://github.com/WebEcologyProject/140kit    200  \n",
       "1  https://www.3dflow.net/3df-zephyr-pro-3d-model...    200  \n",
       "2                      http://vcg.isti.cnr.it/3dhop/    200  \n",
       "3            https://github.com/cnr-isti-vclab/3DHOP    200  \n",
       "4                     https://www.3dreshaper.com/en/    200  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#The list of categories is defined in the following statement\n",
    "\n",
    "categories=\"toolsandservices\"#, publications, trainingmaterials, workflows, datasets\"\n",
    "\n",
    "check=eva.URLCheck()\n",
    "df_urls=check.checkURLValues(categories, 'accessibleAt')\n",
    "df_urls.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1878, 7)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_urls.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_urls.drop_duplicates(keep='first', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1220, 7)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_urls.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_3168acec_8419_11ec_8c9a_acde48001122\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >MPUrl</th>        <th class=\"col_heading level0 col1\" >persistentId</th>        <th class=\"col_heading level0 col2\" >category</th>        <th class=\"col_heading level0 col3\" >label</th>        <th class=\"col_heading level0 col4\" >property</th>        <th class=\"col_heading level0 col5\" >url</th>        <th class=\"col_heading level0 col6\" >status</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row0\" class=\"row_heading level0 row0\" >1380</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col0\" class=\"data row0 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/25grr1\">tool-or-service/25grr1</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col1\" class=\"data row0 col1\" >25grr1</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col2\" class=\"data row0 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col3\" class=\"data row0 col3\" >VLE - Viennese Lexicographic Editor</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col4\" class=\"data row0 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col5\" class=\"data row0 col5\" >https://www.oeaw.ac.at/en/acdh/tools/vle/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row0_col6\" class=\"data row0 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row1\" class=\"row_heading level0 row1\" >375</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col0\" class=\"data row1 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/3ZpHvD\">tool-or-service/3ZpHvD</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col1\" class=\"data row1 col1\" >3ZpHvD</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col2\" class=\"data row1 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col3\" class=\"data row1 col3\" >Finding People or Characters from A Text (Named-Entity Recognition)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col4\" class=\"data row1 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col5\" class=\"data row1 col5\" >https://github.com/TAPoR-3-Tools/Tapor-Coding-Tools/tree/master/tapor_coding_tools/natural%20language%20processing/Finging%20people%20or%20characters%20with%20NER</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row1_col6\" class=\"data row1 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row2\" class=\"row_heading level0 row2\" >1217</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col0\" class=\"data row2 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/6xmMnK\">tool-or-service/6xmMnK</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col1\" class=\"data row2 col1\" >6xmMnK</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col2\" class=\"data row2 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col3\" class=\"data row2 col3\" >Tagger - Other (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col4\" class=\"data row2 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col5\" class=\"data row2 col5\" >http://taporware.ualberta.ca/~taporware/otherTools/tagger.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row2_col6\" class=\"data row2 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row3\" class=\"row_heading level0 row3\" >1843</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col0\" class=\"data row3 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/Aezoqo\">tool-or-service/Aezoqo</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col1\" class=\"data row3 col1\" >Aezoqo</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col2\" class=\"data row3 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col3\" class=\"data row3 col3\" >Wordle</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col4\" class=\"data row3 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col5\" class=\"data row3 col5\" >http://www.wordle.net/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row3_col6\" class=\"data row3 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row4\" class=\"row_heading level0 row4\" >363</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col0\" class=\"data row4 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/C80AK6\">tool-or-service/C80AK6</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col1\" class=\"data row4 col1\" >C80AK6</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col2\" class=\"data row4 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col3\" class=\"data row4 col3\" >Extract Text From HTML - Beta (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col4\" class=\"data row4 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col5\" class=\"data row4 col5\" >http://taporware.ualberta.ca/~taporware/betaTools/textextractor.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row4_col6\" class=\"data row4 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row5\" class=\"row_heading level0 row5\" >379</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col0\" class=\"data row5 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/CzSzkq\">tool-or-service/CzSzkq</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col1\" class=\"data row5 col1\" >CzSzkq</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col2\" class=\"data row5 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col3\" class=\"data row5 col3\" >Fixed Phrase - XML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col4\" class=\"data row5 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col5\" class=\"data row5 col5\" >http://taporware.ualberta.ca/~taporware/xmlTools/fixedphrase-xml.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row5_col6\" class=\"data row5 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row6\" class=\"row_heading level0 row6\" >725</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col0\" class=\"data row6 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/EcLbxP\">tool-or-service/EcLbxP</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col1\" class=\"data row6 col1\" >EcLbxP</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col2\" class=\"data row6 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col3\" class=\"data row6 col3\" >Integrated Authority File (GND)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col4\" class=\"data row6 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col5\" class=\"data row6 col5\" >http://www.dnb.de/EN/Standardisierung/GND/gnd_node.html</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row6_col6\" class=\"data row6 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row7\" class=\"row_heading level0 row7\" >377</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col0\" class=\"data row7 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/F4hxi5\">tool-or-service/F4hxi5</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col1\" class=\"data row7 col1\" >F4hxi5</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col2\" class=\"data row7 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col3\" class=\"data row7 col3\" >Fixed Phrase - HTML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col4\" class=\"data row7 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col5\" class=\"data row7 col5\" >http://taporware.ualberta.ca/~taporware/htmlTools/fixedphrase-html.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row7_col6\" class=\"data row7 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row8\" class=\"row_heading level0 row8\" >114</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col0\" class=\"data row8 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/FOhmdk\">tool-or-service/FOhmdk</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col1\" class=\"data row8 col1\" >FOhmdk</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col2\" class=\"data row8 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col3\" class=\"data row8 col3\" >Bibliopedia</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col4\" class=\"data row8 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col5\" class=\"data row8 col5\" >http://sul-cidr.github.io/Bibliopedia/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row8_col6\" class=\"data row8 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row9\" class=\"row_heading level0 row9\" >221</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col0\" class=\"data row9 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/G4iyld\">tool-or-service/G4iyld</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col1\" class=\"data row9 col1\" >G4iyld</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col2\" class=\"data row9 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col3\" class=\"data row9 col3\" >Concraft -> DependencyParser</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col4\" class=\"data row9 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col5\" class=\"data row9 col5\" >http://zil.ipipan.waw.pl/DependencyParser</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row9_col6\" class=\"data row9 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row10\" class=\"row_heading level0 row10\" >179</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col0\" class=\"data row10 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/GwWY2h\">tool-or-service/GwWY2h</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col1\" class=\"data row10 col1\" >GwWY2h</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col2\" class=\"data row10 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col3\" class=\"data row10 col3\" >CMU-Cambridge Statistical Language Modelling Toolkit</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col4\" class=\"data row10 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col5\" class=\"data row10 col5\" >http://svr-www.eng.cam.ac.uk/~prc14/toolkit.html</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row10_col6\" class=\"data row10 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row11\" class=\"row_heading level0 row11\" >238</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col0\" class=\"data row11 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/HLnGEm\">tool-or-service/HLnGEm</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col1\" class=\"data row11 col1\" >HLnGEm</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col2\" class=\"data row11 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col3\" class=\"data row11 col3\" >Co-Occurrence - XML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col4\" class=\"data row11 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col5\" class=\"data row11 col5\" >http://taporware.ualberta.ca/~taporware/xmlTools/cooccur.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row11_col6\" class=\"data row11 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row12\" class=\"row_heading level0 row12\" >138</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col0\" class=\"data row12 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/I2egYT\">tool-or-service/I2egYT</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col1\" class=\"data row12 col1\" >I2egYT</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col2\" class=\"data row12 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col3\" class=\"data row12 col3\" >Capture Fox</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col4\" class=\"data row12 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col5\" class=\"data row12 col5\" >https://addons.mozilla.org/en-US/firefox/addon/capture-fox/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row12_col6\" class=\"data row12 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row13\" class=\"row_heading level0 row13\" >427</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col0\" class=\"data row13 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/KNVuNk\">tool-or-service/KNVuNk</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col1\" class=\"data row13 col1\" >KNVuNk</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col2\" class=\"data row13 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col3\" class=\"data row13 col3\" >Google Correlate</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col4\" class=\"data row13 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col5\" class=\"data row13 col5\" >https://www.google.com/trends/correlate</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row13_col6\" class=\"data row13 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row14\" class=\"row_heading level0 row14\" >295</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col0\" class=\"data row14 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/OKoqgh\">tool-or-service/OKoqgh</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col1\" class=\"data row14 col1\" >OKoqgh</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col2\" class=\"data row14 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col3\" class=\"data row14 col3\" >DH Press</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col4\" class=\"data row14 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col5\" class=\"data row14 col5\" >https://digitalhumanities.duke.edu/tools/dh-press</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row14_col6\" class=\"data row14 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row15\" class=\"row_heading level0 row15\" >401</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col0\" class=\"data row15 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/S2Bt9F\">tool-or-service/S2Bt9F</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col1\" class=\"data row15 col1\" >S2Bt9F</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col2\" class=\"data row15 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col3\" class=\"data row15 col3\" >Generating Concordances</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col4\" class=\"data row15 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col5\" class=\"data row15 col5\" >https://github.com/TAPoR-3-Tools/Tapor-Coding-Tools/blob/master/Tapor_Coding_Tools/Methodica/Concordance.ipynb</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row15_col6\" class=\"data row15 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row16\" class=\"row_heading level0 row16\" >1136</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col0\" class=\"data row16 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/UdB9FI\">tool-or-service/UdB9FI</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col1\" class=\"data row16 col1\" >UdB9FI</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col2\" class=\"data row16 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col3\" class=\"data row16 col3\" >SketchUp</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col4\" class=\"data row16 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col5\" class=\"data row16 col5\" >https://www.sketchup.com/home</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row16_col6\" class=\"data row16 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row17\" class=\"row_heading level0 row17\" >1249</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col0\" class=\"data row17 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/UlRiAy\">tool-or-service/UlRiAy</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col1\" class=\"data row17 col1\" >UlRiAy</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col2\" class=\"data row17 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col3\" class=\"data row17 col3\" >Text Classification Using SVM</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col4\" class=\"data row17 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col5\" class=\"data row17 col5\" >https://github.com/TAPoR-3-Tools/Tapor-Coding-Tools/blob/master/tapor_coding_tools/data%20manipulation/Classifying%20text%20using%20SVM/Classifying%20text%20using%20SVM.ipynb</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row17_col6\" class=\"data row17 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row18\" class=\"row_heading level0 row18\" >309</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col0\" class=\"data row18 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/V6aWAu\">tool-or-service/V6aWAu</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col1\" class=\"data row18 col1\" >V6aWAu</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col2\" class=\"data row18 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col3\" class=\"data row18 col3\" >Discursis</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col4\" class=\"data row18 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col5\" class=\"data row18 col5\" >http://www.discursis.com/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row18_col6\" class=\"data row18 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row19\" class=\"row_heading level0 row19\" >945</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col0\" class=\"data row19 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/Vus9vX\">tool-or-service/Vus9vX</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col1\" class=\"data row19 col1\" >Vus9vX</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col2\" class=\"data row19 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col3\" class=\"data row19 col3\" >Open Calais Issue Discovery</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col4\" class=\"data row19 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col5\" class=\"data row19 col5\" >https://tools.digitalmethods.net/beta/opencalais/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row19_col6\" class=\"data row19 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row20\" class=\"row_heading level0 row20\" >256</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col0\" class=\"data row20 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/WCXCgY\">tool-or-service/WCXCgY</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col1\" class=\"data row20 col1\" >WCXCgY</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col2\" class=\"data row20 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col3\" class=\"data row20 col3\" >CSV Sort</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col4\" class=\"data row20 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col5\" class=\"data row20 col5\" >https://bitbucket.org/richardpenman/csvsort</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row20_col6\" class=\"data row20 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row21\" class=\"row_heading level0 row21\" >1163</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col0\" class=\"data row21 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/X2UbFI\">tool-or-service/X2UbFI</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col1\" class=\"data row21 col1\" >X2UbFI</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col2\" class=\"data row21 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col3\" class=\"data row21 col3\" >Speech Tagger - Plain Text (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col4\" class=\"data row21 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col5\" class=\"data row21 col5\" >http://taporware.ualberta.ca/~taporware/textTools/speechtagger.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row21_col6\" class=\"data row21 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row22\" class=\"row_heading level0 row22\" >805</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col0\" class=\"data row22 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/Y6plQG\">tool-or-service/Y6plQG</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col1\" class=\"data row22 col1\" >Y6plQG</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col2\" class=\"data row22 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col3\" class=\"data row22 col3\" >List Tags - HTML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col4\" class=\"data row22 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col5\" class=\"data row22 col5\" >http://taporware.ualberta.ca/~taporware/htmlTools/listtags.shtml?</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row22_col6\" class=\"data row22 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row23\" class=\"row_heading level0 row23\" >480</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col0\" class=\"data row23 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/aY4ryv\">tool-or-service/aY4ryv</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col1\" class=\"data row23 col1\" >aY4ryv</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col2\" class=\"data row23 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col3\" class=\"data row23 col3\" >IBeA Raman Database of Inorganic Artists Material</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col4\" class=\"data row23 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col5\" class=\"data row23 col5\" >http://www.ehu.eus/udps/database/index1.html</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row23_col6\" class=\"data row23 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row24\" class=\"row_heading level0 row24\" >315</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col0\" class=\"data row24 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/ak3Xvt\">tool-or-service/ak3Xvt</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col1\" class=\"data row24 col1\" >ak3Xvt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col2\" class=\"data row24 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col3\" class=\"data row24 col3\" >DocScanner</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col4\" class=\"data row24 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col5\" class=\"data row24 col5\" >https://play.google.com/store/apps/details?id=com.document.cam.scanner.book.pdf.docscanner&hl=en_US</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row24_col6\" class=\"data row24 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row25\" class=\"row_heading level0 row25\" >1301</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col0\" class=\"data row25 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/bvr6CG\">tool-or-service/bvr6CG</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col1\" class=\"data row25 col1\" >bvr6CG</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col2\" class=\"data row25 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col3\" class=\"data row25 col3\" >Tokenize - Plain Text (TAPoR)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col4\" class=\"data row25 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col5\" class=\"data row25 col5\" >http://taporware.ualberta.ca/~taporware/textTools/tokenize.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row25_col6\" class=\"data row25 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row26\" class=\"row_heading level0 row26\" >304</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col0\" class=\"data row26 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/gfF0V9\">tool-or-service/gfF0V9</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col1\" class=\"data row26 col1\" >gfF0V9</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col2\" class=\"data row26 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col3\" class=\"data row26 col3\" >Digitate</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col4\" class=\"data row26 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col5\" class=\"data row26 col5\" >https://itunes.apple.com/us/app/digitate/id610952790?mt=8</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row26_col6\" class=\"data row26 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row27\" class=\"row_heading level0 row27\" >150</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col0\" class=\"data row27 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/i3v5q0\">tool-or-service/i3v5q0</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col1\" class=\"data row27 col1\" >i3v5q0</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col2\" class=\"data row27 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col3\" class=\"data row27 col3\" >CERL Portal</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col4\" class=\"data row27 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col5\" class=\"data row27 col5\" >https://www.cerl.org/resources/cerl_portal</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row27_col6\" class=\"data row27 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row28\" class=\"row_heading level0 row28\" >1837</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col0\" class=\"data row28 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/kRIrWC\">tool-or-service/kRIrWC</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col1\" class=\"data row28 col1\" >kRIrWC</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col2\" class=\"data row28 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col3\" class=\"data row28 col3\" >Word Brush - Beta (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col4\" class=\"data row28 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col5\" class=\"data row28 col5\" >http://taporware.ualberta.ca/~taporware/betaTools/wordbrush.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row28_col6\" class=\"data row28 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row29\" class=\"row_heading level0 row29\" >1302</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col0\" class=\"data row29 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/knCfk3\">tool-or-service/knCfk3</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col1\" class=\"data row29 col1\" >knCfk3</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col2\" class=\"data row29 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col3\" class=\"data row29 col3\" >Tokenize - XML (TAPoR)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col4\" class=\"data row29 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col5\" class=\"data row29 col5\" >http://taporware.ualberta.ca/~taporware/xmlTools/tokenize.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row29_col6\" class=\"data row29 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row30\" class=\"row_heading level0 row30\" >1828</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col0\" class=\"data row30 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/mfmPPR\">tool-or-service/mfmPPR</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col1\" class=\"data row30 col1\" >mfmPPR</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col2\" class=\"data row30 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col3\" class=\"data row30 col3\" >WinBrill</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col4\" class=\"data row30 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col5\" class=\"data row30 col5\" >http://www.atilf.fr/spip.php?article899</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row30_col6\" class=\"data row30 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row31\" class=\"row_heading level0 row31\" >237</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col0\" class=\"data row31 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/mrFG7W\">tool-or-service/mrFG7W</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col1\" class=\"data row31 col1\" >mrFG7W</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col2\" class=\"data row31 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col3\" class=\"data row31 col3\" >Co-Occurrence - Plain Text (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col4\" class=\"data row31 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col5\" class=\"data row31 col5\" >http://taporware.ualberta.ca/~taporware/textTools/cooccur.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row31_col6\" class=\"data row31 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row32\" class=\"row_heading level0 row32\" >236</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col0\" class=\"data row32 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/oVSGJ5\">tool-or-service/oVSGJ5</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col1\" class=\"data row32 col1\" >oVSGJ5</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col2\" class=\"data row32 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col3\" class=\"data row32 col3\" >Co-Occurrence - HTML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col4\" class=\"data row32 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col5\" class=\"data row32 col5\" >http://taporware.ualberta.ca/~taporware/htmlTools/cooccur.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row32_col6\" class=\"data row32 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row33\" class=\"row_heading level0 row33\" >1300</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col0\" class=\"data row33 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/t4Tzag\">tool-or-service/t4Tzag</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col1\" class=\"data row33 col1\" >t4Tzag</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col2\" class=\"data row33 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col3\" class=\"data row33 col3\" >Tokenize - HTML (TAPoR)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col4\" class=\"data row33 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col5\" class=\"data row33 col5\" >http://taporware.ualberta.ca/~taporware/htmlTools/tokenize.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row33_col6\" class=\"data row33 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row34\" class=\"row_heading level0 row34\" >378</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col0\" class=\"data row34 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/thoAhw\">tool-or-service/thoAhw</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col1\" class=\"data row34 col1\" >thoAhw</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col2\" class=\"data row34 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col3\" class=\"data row34 col3\" >Fixed Phrase - Plain Text (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col4\" class=\"data row34 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col5\" class=\"data row34 col5\" >http://taporware.ualberta.ca/~taporware/textTools/fixedphrase-text.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row34_col6\" class=\"data row34 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row35\" class=\"row_heading level0 row35\" >336</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col0\" class=\"data row35 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/tjoXDf\">tool-or-service/tjoXDf</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col1\" class=\"data row35 col1\" >tjoXDf</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col2\" class=\"data row35 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col3\" class=\"data row35 col3\" >EgoWeb 2.0</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col4\" class=\"data row35 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col5\" class=\"data row35 col5\" >http://www.rand.org/methods/egoweb.html</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row35_col6\" class=\"data row35 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row36\" class=\"row_heading level0 row36\" >937</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col0\" class=\"data row36 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/veS7n4\">tool-or-service/veS7n4</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col1\" class=\"data row36 col1\" >veS7n4</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col2\" class=\"data row36 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col3\" class=\"data row36 col3\" >OLiA ontologies</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col4\" class=\"data row36 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col5\" class=\"data row36 col5\" >http://www.acoli.informatik.uni-frankfurt.de/resources/olia/#olia</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row36_col6\" class=\"data row36 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row37\" class=\"row_heading level0 row37\" >801</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col0\" class=\"data row37 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/vi3pn2\">tool-or-service/vi3pn2</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col1\" class=\"data row37 col1\" >vi3pn2</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col2\" class=\"data row37 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col3\" class=\"data row37 col3\" >Link Extractor - HTML (TAPoRware)</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col4\" class=\"data row37 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col5\" class=\"data row37 col5\" >http://taporware.ualberta.ca/~taporware/htmlTools/linkextractor.shtml</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row37_col6\" class=\"data row37 col6\" >404</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_3168acec_8419_11ec_8c9a_acde48001122level0_row38\" class=\"row_heading level0 row38\" >1096</th>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col0\" class=\"data row38 col0\" ><a target=\"_blank\" href=\"https://marketplace-api.sshopencloud.eu/tool-or-service/zwJRaY\">tool-or-service/zwJRaY</a></td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col1\" class=\"data row38 col1\" >zwJRaY</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col2\" class=\"data row38 col2\" >tool-or-service</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col3\" class=\"data row38 col3\" >Screenr</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col4\" class=\"data row38 col4\" >accessibleAt</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col5\" class=\"data row38 col5\" >https://www.screenr.co/</td>\n",
       "                        <td id=\"T_3168acec_8419_11ec_8c9a_acde48001122row38_col6\" class=\"data row38 col6\" >404</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f900ca7e5e0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils=hel.Util()\n",
    "df_http_status_nf_err=df_urls[df_urls['status'] == 404].sort_values('persistentId').drop_duplicates(keep='first', inplace=False)\n",
    "df_http_status_serv_err=df_urls[df_urls['status'] == 200].sort_values('persistentId').drop_duplicates(keep='first', inplace=False)\n",
    "df_http_status_err=pd.concat([df_http_status_nf_err, df_http_status_serv_err])\n",
    "df_http_status_err=df_urls.sort_values('persistentId').drop_duplicates(keep='first', inplace=False)\n",
    "#df_http_status_err.to_pickle('data/urlstatus.pickle')\n",
    "#df_http_status_nf_err.to_pickle('data/urlstatus404.pickle')\n",
    "myclickable_table = df_http_status_nf_err.style.format({'MPUrl': utils.make_clickable})\n",
    "myclickable_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(39, 7)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_http_status_nf_err.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Flag items with wrong accessibleAt URLs in the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_flag_property={\"code\": \"curation-flag-url\"}\n",
    "curation_detail_property={\"code\": \"curation-detail\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: FOhmdk has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: I2egYT has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: i3v5q0 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "Appending curation_detail_flag {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "\n",
      "\n",
      "\n",
      "Running in debug mode, Marketplace dataset not updated.\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: GwWY2h has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: G4iyld has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: oVSGJ5 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: mrFG7W has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: HLnGEm has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: WCXCgY has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {'actor.website': '404'},  {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: OKoqgh has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: gfF0V9 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: V6aWAu has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {'actor.website': '404'},  {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: ak3Xvt has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: tjoXDf has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: C80AK6 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: 3ZpHvD has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: F4hxi5 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: thoAhw has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: CzSzkq has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: S2Bt9F has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: KNVuNk has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: aY4ryv has a 404 HTTP status for the property accessibleAt, (False)\n",
      "Appending curation_detail_flag {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "\n",
      "\n",
      "\n",
      "Running in debug mode, Marketplace dataset not updated.\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: EcLbxP has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: vi3pn2 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: Y6plQG has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: veS7n4 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "Appending curation_detail_flag {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "\n",
      "\n",
      "\n",
      "Running in debug mode, Marketplace dataset not updated.\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: Vus9vX has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: zwJRaY has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: UdB9FI has a 404 HTTP status for the property accessibleAt, (False)\n",
      "Appending curation_detail_flag {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "\n",
      "\n",
      "\n",
      "Running in debug mode, Marketplace dataset not updated.\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: X2UbFI has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: 6xmMnK has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: UlRiAy has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: t4Tzag has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: bvr6CG has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: knCfk3 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: 25grr1 has a 404 HTTP status for the property accessibleAt, (False)\n",
      "Appending curation_detail_flag {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "\n",
      "\n",
      "\n",
      "Running in debug mode, Marketplace dataset not updated.\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: mfmPPR has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: kRIrWC has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n",
      "accessibleAt, tools \n",
      "\n",
      "The item with PID: Aezoqo has a 404 HTTP status for the property accessibleAt, (False)\n",
      "flag property exists, value:  {'url': [ {\"accessibleAt\": \"404\"}]}\n"
     ]
    }
   ],
   "source": [
    "res=mpdata.setHTTPStatusFlags(df_http_status_nf_err, curation_flag_property, curation_detail_property)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Check & flag error values in URL-based properties\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_properties=utils.getProperties()\n",
    "df_properties.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_properties[\"type.type\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_properties_url=df_properties[df_properties[\"type.type\"]==\"url\"]\n",
    "df_properties_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_properties_url[\"type.code\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_df_properties=check.checkURLValuesInDataset(df_all_items, 'terms-of-use-url, user-manual-url, privacy-policy-url, access-policy-url, service-level-url, see-also, helpdesk-url')\n",
    "urls_df_properties.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check error values in URL-based properties for items whose source is: *EOSC Catalogue*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe with all the items having the EOSC Catalogue source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ec_items=df_all_items[df_all_items['source.label']=='EOSC Catalogue']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ec_items.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the URL properties by invoking the function **checkURLValuesInDataset(dataset, props)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_df_hd=check.checkURLValuesInDataset(df_ec_items, 'terms-of-use-url, user-manual-url, privacy-policy-url, access-policy-url, service-level-url, see-also, helpdesk-url')\n",
    "urls_df_hd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_df_hd_status_nf_err=urls_df_hd[urls_df_hd['status'] == 404].sort_values('persistentId').drop_duplicates(keep='first', inplace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myclickable_table = urls_df_hd_status_nf_err.style.format({'MPUrl': utils.make_clickable})\n",
    "myclickable_table"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}