Example_Challenge2C.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "94bd8285",
   "metadata": {},
   "source": [
    "# Challenge 2C - Reuse the SSH Open Marketplace data using the Application Programming Interface (API)\n",
    "\n",
    "This notebook provide examples of the Python API used to access the SSH Open Marketplace dataset. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e6eda23",
   "metadata": {},
   "source": [
    "#### Libraries needed to run the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a9798a4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd #to manage dataframes\n",
    "from sshmarketplacelib import MPData as mpd\n",
    "from sshmarketplacelib import  eval as eva, helper as hel\n",
    "mpdata = mpd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdfb90c6",
   "metadata": {},
   "source": [
    "#### The function *getMPItems* is used to download the data from the MP dataset. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc321bf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method getMPItems in module sshmarketplacelib.mpdata:\n",
      "\n",
      "getMPItems(itemscategory, localrepository=False, pages=0) method of sshmarketplacelib.mpdata.MPData instance\n",
      "    Loads data from MP dataset. This method creates a dataframe, stores it in a local repository and returns it to the caller.\n",
      "    \n",
      "    Parameters:\n",
      "    -----------\n",
      "    itemscategory : str\n",
      "        The category of items, possible values are: \"toolsandservices\", \"publications\", \"trainingmaterials\", \"workflows\", \"dataset\"\n",
      "    localrepository : boolean, optional\n",
      "    pages : int, optional\n",
      "        The number of pages, default is all\n",
      "    \n",
      "    Returns:\n",
      "    --------\n",
      "    DataFrame: Returning value\n",
      "    \n",
      "    If the localrepository parameter is 'FALSE' or is not specified, the items are downloaded from the MP dataset, \n",
      "    if the localrepository parameter is 'TRUE' the items are first searched in the local repository and \n",
      "    if they are not present they are downloaded from the remote MP dataset. \n",
      "    When the items are downloaded from the remote MP dataset they are stored in locally.\n",
      "    If the pages parameter is not provided all descriptions are returned, otherwise 20*pages items are returned.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(mpdata.getMPItems)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eec8c8e",
   "metadata": {},
   "source": [
    "#### Download the *tools or services* items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "55ebfcfb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n",
      "getting data from local repository...\n"
     ]
    }
   ],
   "source": [
    "df_tool_flat =mpdata.getMPItems (\"toolsandservices\", True)\n",
    "df_publication_flat =mpdata.getMPItems (\"publications\", True)\n",
    "df_trainingmaterials_flat =mpdata.getMPItems (\"trainingmaterials\", True)\n",
    "df_workflows_flat =mpdata.getMPItems (\"workflows\", True)\n",
    "df_datasets_flat =mpdata.getMPItems (\"datasets\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06730feb",
   "metadata": {},
   "source": [
    "#### Show data frame info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ae1900fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1738 entries, 0 to 1737\n",
      "Data columns (total 33 columns):\n",
      " #   Column                                   Non-Null Count  Dtype  \n",
      "---  ------                                   --------------  -----  \n",
      " 0   id                                       1738 non-null   int64  \n",
      " 1   category                                 1738 non-null   object \n",
      " 2   label                                    1738 non-null   object \n",
      " 3   persistentId                             1738 non-null   object \n",
      " 4   lastInfoUpdate                           1738 non-null   object \n",
      " 5   status                                   1738 non-null   object \n",
      " 6   description                              1738 non-null   object \n",
      " 7   contributors                             1738 non-null   object \n",
      " 8   properties                               1738 non-null   object \n",
      " 9   externalIds                              1738 non-null   object \n",
      " 10  accessibleAt                             1738 non-null   object \n",
      " 11  sourceItemId                             1620 non-null   object \n",
      " 12  relatedItems                             1738 non-null   object \n",
      " 13  media                                    1738 non-null   object \n",
      " 14  informationContributor.id                1738 non-null   int64  \n",
      " 15  informationContributor.username          1738 non-null   object \n",
      " 16  informationContributor.displayName       1738 non-null   object \n",
      " 17  informationContributor.status            1738 non-null   object \n",
      " 18  informationContributor.registrationDate  1738 non-null   object \n",
      " 19  informationContributor.role              1738 non-null   object \n",
      " 20  informationContributor.config            1738 non-null   bool   \n",
      " 21  source.id                                1620 non-null   float64\n",
      " 22  source.label                             1620 non-null   object \n",
      " 23  source.url                               1620 non-null   object \n",
      " 24  source.urlTemplate                       1620 non-null   object \n",
      " 25  version                                  62 non-null     object \n",
      " 26  thumbnail.info.mediaId                   110 non-null    object \n",
      " 27  thumbnail.info.category                  110 non-null    object \n",
      " 28  thumbnail.info.filename                  40 non-null     object \n",
      " 29  thumbnail.info.mimeType                  110 non-null    object \n",
      " 30  thumbnail.info.hasThumbnail              110 non-null    object \n",
      " 31  thumbnail.info.location.sourceUrl        70 non-null     object \n",
      " 32  thumbnail.caption                        61 non-null     object \n",
      "dtypes: bool(1), float64(1), int64(2), object(29)\n",
      "memory usage: 436.3+ KB\n"
     ]
    }
   ],
   "source": [
    "df_tool_flat.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "670f0c6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>category</th>\n",
       "      <th>description</th>\n",
       "      <th>accessibleAt</th>\n",
       "      <th>source.label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1733</th>\n",
       "      <td>Zoomify</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>A suite of products developed for web designer...</td>\n",
       "      <td>[]</td>\n",
       "      <td>TAPoR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1734</th>\n",
       "      <td>Zotero</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotero is a free tool that collects, manages a...</td>\n",
       "      <td>[https://www.zotero.org/]</td>\n",
       "      <td>TAPoR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1735</th>\n",
       "      <td>ZotFile</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotero add-on for Firefox to manage your attac...</td>\n",
       "      <td>[http://zotfile.com/]</td>\n",
       "      <td>TAPoR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1736</th>\n",
       "      <td>Zotpress</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotpress is a WordPress plugin that enables us...</td>\n",
       "      <td>[https://wordpress.org/plugins/zotpress/]</td>\n",
       "      <td>TAPoR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1737</th>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>This tool simply takes HTML and returns it wit...</td>\n",
       "      <td>[http://www.zubrag.com/tools/html-tags-strippe...</td>\n",
       "      <td>TAPoR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          label         category  \\\n",
       "1733                    Zoomify  tool-or-service   \n",
       "1734                     Zotero  tool-or-service   \n",
       "1735                    ZotFile  tool-or-service   \n",
       "1736                   Zotpress  tool-or-service   \n",
       "1737  Zubrag HTML Tags Stripper  tool-or-service   \n",
       "\n",
       "                                            description  \\\n",
       "1733  A suite of products developed for web designer...   \n",
       "1734  Zotero is a free tool that collects, manages a...   \n",
       "1735  Zotero add-on for Firefox to manage your attac...   \n",
       "1736  Zotpress is a WordPress plugin that enables us...   \n",
       "1737  This tool simply takes HTML and returns it wit...   \n",
       "\n",
       "                                           accessibleAt source.label  \n",
       "1733                                                 []        TAPoR  \n",
       "1734                          [https://www.zotero.org/]        TAPoR  \n",
       "1735                              [http://zotfile.com/]        TAPoR  \n",
       "1736          [https://wordpress.org/plugins/zotpress/]        TAPoR  \n",
       "1737  [http://www.zubrag.com/tools/html-tags-strippe...        TAPoR  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tool_flat[['label', 'category', 'description', 'accessibleAt', 'source.label']].tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4578ef4f",
   "metadata": {},
   "source": [
    "#### Get properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cd7a48ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "utils=hel.Util()\n",
    "df_tool_properties=utils.getProperties(df_tool_flat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "818eb87b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 16408 entries, 0 to 16407\n",
      "Data columns (total 27 columns):\n",
      " #   Column                        Non-Null Count  Dtype \n",
      "---  ------                        --------------  ----- \n",
      " 0   id                            16408 non-null  int64 \n",
      " 1   category                      16408 non-null  object\n",
      " 2   label                         16408 non-null  object\n",
      " 3   persistentId                  16408 non-null  object\n",
      " 4   accessibleAt                  16408 non-null  object\n",
      " 5   description                   16408 non-null  object\n",
      " 6   relatedItems                  16408 non-null  object\n",
      " 7   media                         16408 non-null  object\n",
      " 8   source.label                  15583 non-null  object\n",
      " 9   source.url                    15583 non-null  object\n",
      " 10  type.code                     16408 non-null  object\n",
      " 11  type.label                    16408 non-null  object\n",
      " 12  type.type                     16408 non-null  object\n",
      " 13  type.groupName                16408 non-null  object\n",
      " 14  type.allowedVocabularies      16408 non-null  object\n",
      " 15  concept.code                  14249 non-null  object\n",
      " 16  concept.vocabulary.code       14249 non-null  object\n",
      " 17  concept.vocabulary.scheme     14249 non-null  object\n",
      " 18  concept.vocabulary.namespace  14249 non-null  object\n",
      " 19  concept.vocabulary.label      14249 non-null  object\n",
      " 20  concept.vocabulary.closed     14249 non-null  object\n",
      " 21  concept.label                 14249 non-null  object\n",
      " 22  concept.notation              14249 non-null  object\n",
      " 23  concept.uri                   14249 non-null  object\n",
      " 24  concept.candidate             14249 non-null  object\n",
      " 25  value                         2159 non-null   object\n",
      " 26  concept.definition            298 non-null    object\n",
      "dtypes: int64(1), object(26)\n",
      "memory usage: 3.5+ MB\n"
     ]
    }
   ],
   "source": [
    "df_tool_properties.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b5a6a4dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>label</th>\n",
       "      <th>source.label</th>\n",
       "      <th>type.label</th>\n",
       "      <th>concept.label</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16398</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotpress</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Activity</td>\n",
       "      <td>Disseminating</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16399</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotpress</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Keyword</td>\n",
       "      <td>web publishing</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16400</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotpress</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Keyword</td>\n",
       "      <td>citations</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16401</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zotpress</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Keyword</td>\n",
       "      <td>bibliography</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16402</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Terms Of Use</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16403</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Mode of use</td>\n",
       "      <td>Web application</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16404</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Activity</td>\n",
       "      <td>Enriching</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16405</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Keyword</td>\n",
       "      <td>Enrichment</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16406</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Activity</td>\n",
       "      <td>Data Cleansing</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16407</th>\n",
       "      <td>tool-or-service</td>\n",
       "      <td>Zubrag HTML Tags Stripper</td>\n",
       "      <td>TAPoR</td>\n",
       "      <td>Deprecated at Source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TRUE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              category                      label source.label  \\\n",
       "16398  tool-or-service                   Zotpress        TAPoR   \n",
       "16399  tool-or-service                   Zotpress        TAPoR   \n",
       "16400  tool-or-service                   Zotpress        TAPoR   \n",
       "16401  tool-or-service                   Zotpress        TAPoR   \n",
       "16402  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "16403  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "16404  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "16405  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "16406  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "16407  tool-or-service  Zubrag HTML Tags Stripper        TAPoR   \n",
       "\n",
       "                 type.label    concept.label value  \n",
       "16398              Activity    Disseminating   NaN  \n",
       "16399               Keyword   web publishing   NaN  \n",
       "16400               Keyword        citations   NaN  \n",
       "16401               Keyword     bibliography   NaN  \n",
       "16402          Terms Of Use              NaN  Free  \n",
       "16403           Mode of use  Web application   NaN  \n",
       "16404              Activity        Enriching   NaN  \n",
       "16405               Keyword       Enrichment   NaN  \n",
       "16406              Activity   Data Cleansing   NaN  \n",
       "16407  Deprecated at Source              NaN  TRUE  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tool_properties[['category', 'label', 'source.label', 'type.label', 'concept.label', 'value']].tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad0bd24d",
   "metadata": {},
   "source": [
    "#### _Helper_ functions: get the number of items provided by every source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ce2c68b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dblp: DH Publications             2810\n",
       "TAPoR                             1337\n",
       "SSK Zotero Resources               355\n",
       "Humanities Data                    302\n",
       "The Programming Historian          169\n",
       "NA                                 137\n",
       "DARIAH-CAMPUS                       80\n",
       "Language Resource Switchboard       51\n",
       "DARIAH contribution tool            35\n",
       "EOSC Catalogue                      29\n",
       "Standardization Survival Kit        25\n",
       "SSHopencloud Service Catalogue      20\n",
       "Name: source.label, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utils=hel.Util()\n",
    "df_items_by_sources=utils.getAllItemsBySources()\n",
    "df_items_by_sources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e7eb12a",
   "metadata": {},
   "source": [
    "#### Helper functions: get the number of items per category provided by every source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a3ea17f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>source.label</th>\n",
       "      <th>DARIAH contribution tool</th>\n",
       "      <th>DARIAH-CAMPUS</th>\n",
       "      <th>EOSC Catalogue</th>\n",
       "      <th>Humanities Data</th>\n",
       "      <th>Language Resource Switchboard</th>\n",
       "      <th>SSHopencloud Service Catalogue</th>\n",
       "      <th>SSK Zotero Resources</th>\n",
       "      <th>Standardization Survival Kit</th>\n",
       "      <th>TAPoR</th>\n",
       "      <th>The Programming Historian</th>\n",
       "      <th>dblp: DH Publications</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Categories</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dataset</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>302.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>publication</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2810.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tool-or-service</th>\n",
       "      <td>31.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>156.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1337.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>training-material</th>\n",
       "      <td>1.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workflow</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "source.label       DARIAH contribution tool  DARIAH-CAMPUS  EOSC Catalogue  \\\n",
       "Categories                                                                   \n",
       "dataset                                 3.0            0.0             0.0   \n",
       "publication                             0.0            0.0             0.0   \n",
       "tool-or-service                        31.0            0.0            29.0   \n",
       "training-material                       1.0           80.0             0.0   \n",
       "workflow                                0.0            0.0             0.0   \n",
       "\n",
       "source.label       Humanities Data  Language Resource Switchboard  \\\n",
       "Categories                                                          \n",
       "dataset                      302.0                            0.0   \n",
       "publication                    0.0                            0.0   \n",
       "tool-or-service                0.0                           51.0   \n",
       "training-material              0.0                            0.0   \n",
       "workflow                       0.0                            0.0   \n",
       "\n",
       "source.label       SSHopencloud Service Catalogue  SSK Zotero Resources  \\\n",
       "Categories                                                                \n",
       "dataset                                       2.0                   0.0   \n",
       "publication                                   0.0                 136.0   \n",
       "tool-or-service                              16.0                 156.0   \n",
       "training-material                             2.0                  63.0   \n",
       "workflow                                      0.0                   0.0   \n",
       "\n",
       "source.label       Standardization Survival Kit   TAPoR  \\\n",
       "Categories                                                \n",
       "dataset                                     0.0     0.0   \n",
       "publication                                 0.0     0.0   \n",
       "tool-or-service                             0.0  1337.0   \n",
       "training-material                           0.0     0.0   \n",
       "workflow                                   25.0     0.0   \n",
       "\n",
       "source.label       The Programming Historian  dblp: DH Publications  \n",
       "Categories                                                           \n",
       "dataset                                  0.0                    0.0  \n",
       "publication                              0.0                 2810.0  \n",
       "tool-or-service                          0.0                    0.0  \n",
       "training-material                      169.0                    0.0  \n",
       "workflow                                 0.0                    0.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_categories_bysources=utils.getCategoriesBySources()\n",
    "df_categories_bysources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f3253c",
   "metadata": {},
   "source": [
    "#### Example: how many items do have the string 'digital humanities' in the label property"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "665b552e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "searched_string='digital humanities'\n",
    "#create a dataframe with all the items\n",
    "df_all_items=pd.concat([df_tool_flat, df_publication_flat, df_trainingmaterials_flat, df_workflows_flat, df_datasets_flat])\n",
    "#count the number of occurrences of the searched String\n",
    "df_all_items.label.str.contains(searched_string).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9525a703",
   "metadata": {},
   "source": [
    "#### Example: what are the  categories of items that has the string 'digital humanities' in the label property"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "06cc20f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "publication          18\n",
       "training-material     1\n",
       "Name: category, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all_items=df_all_items[df_all_items.label.str.contains('digital humanities')]\n",
    "df_all_items.category.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8ae4d43",
   "metadata": {},
   "source": [
    "#### More examples  [here](https://gitlab.gwdg.de/sshoc/marketplace-curation/-/blob/main/LibTest.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c74cfb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}