diff --git a/dev-requirements.txt b/dev-requirements.txt index f7b08a7..3930480 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -1,2 +1 @@ -flake8 # for the travis build -mkdocs +flake8 diff --git a/docs/about.md b/docs/about.md deleted file mode 100644 index 0e3d634..0000000 --- a/docs/about.md +++ /dev/null @@ -1,25 +0,0 @@ -# About - -## What is B2FIND? -Please find out more about our interdisciplinary discovery portal B2FIND in the [EUDAT Service Catalogue](https://sp.eudat.eu/catalog/resources/33bc21d5-f53d-4eed-9a15-56f98f5c7f69). - -## Contact -If you would like to join B2FIND as a provider, need support or have any other questions for us, please contact us via the [EUDAT request tracker](https://www.eudat.eu/support-request) or write an email to: b2find@dkrz.de - -## Publications -* Searching for Research Data across Disciplines - heiBOOKS - 2022 | Journal article - DOI: 10.11588/HEIBOOKS.979.C13729 - -* Discovery and re-use of Nordic community specific data in EOSC - 2021-02-25 | Report - DOI: 10.5281/zenodo.4607188 - -* Interdisciplinary discovery in practice - Zenodo - 2020-11-17 | Online resource - DOI: 10.5281/zenodo.4277667 - - - diff --git a/docs/communities.md b/docs/communities.md deleted file mode 100644 index 92ad448..0000000 --- a/docs/communities.md +++ /dev/null @@ -1 +0,0 @@ -# Communities \ No newline at end of file diff --git a/docs/datacatalogue.md b/docs/datacatalogue.md deleted file mode 100644 index b9b9d66..0000000 --- a/docs/datacatalogue.md +++ /dev/null @@ -1 +0,0 @@ -# Data Catalogue \ No newline at end of file diff --git a/docs/forproviders.md b/docs/forproviders.md deleted file mode 100644 index 10330b8..0000000 --- a/docs/forproviders.md +++ /dev/null @@ -1,87 +0,0 @@ -# For Providers - -## Guidelines -Welcome to the guidelines of EUDAT’s metadata service B2FIND for data providers. B2FIND gathers diverse metadata on the research output of many heterogeneous sources with the aim of providing a unified discovery portal. The portal then allows widespread and cross-disciplinary search and access to the underlying data collections. These guidelines are intended to provide information about the requirements for successful metadata integration into B2FIND. - -### Providing Metadata -In order to join B2FIND, the data provider must meet a few requirements: - -* The data provider must agree with the licensing principles of B2FIND, see the [Terms of Use](https://www.eudat.eu/eudat-cdi-aup) of EUDAT Services and the [Legal Notice](https://www.dkrz.de/en/about-en/contact/impressum?set_language=en) of DKRZ. -* In particular, the provider must consent to the provided metadata being made publicly available and openly accessible under CC-BY International v.4.0 or subsequent without any restrictions on reuse in original and derivative forms. - ***Note: This open access licence only applies to the metadata records published and visible in the B2FIND portal, not to the underlying data collections referred to and described by the B2FIND datasets.*** -* The data provider agrees to the metadata being made available for free in B2FIND and also for it to be harvested by and re-distributed to other metadata aggregators. No confidential metadata should be provided (although the described research data sets themselves may have access limitations). -* Copyright-protected metadata can only be published if there is a licence agreement between the data provider and EUDAT that meets the B2FIND requirements. -* An interface to retrieve metadata must be available, accessible and usable. - -The Metadata provided should be stable and 'good enough'. Some of the central issues include: -* Metadata records should be as complete as possible. -* Metadata shall not be encrypted or obfuscated. -* Metadata must use Unicode with UTF-8 encoding. If the metadata is in a non-Latin script such as Chinese, a version transliterated or transcribed to Latin characters should be provided as well. -* In addition to the metadata, the data provider must provide documentation needed for successful loading of the records, such as descriptions of the structure, syntax and semantics of the metadata records. - -### Harvesting Channels -Harvesting is the process of automatically fetching remote metadata. While OAI-PMH still is the de facto standard for metadata harvesting, other channels become more important. B2FIND supports several methods, as described below. Once one of these transfer methods have been successfully implemented, B2FIND first takes up a few test samples to analyse their content. As soon as the harvesting and mapping has been consolidated and the data provider gives their consent, the metadata are published on the B2FIND discovery portal, and an operational and stable ingestion process is established. - -#### OAI-PMH -OAI-PMH [Open Archive Initiative - Protocol for Metadata Harvesting](https://www.openarchives.org/pmh/) is B2FIND’s preferred metadata harvesting protocol. It can be used to fetch metadata directly from the data providers within research communities. The simplicity of the protocol allows a controlled and easy-to-manage transfer of metadata and it also allows to fetch update information from a defined periode. Very little information must be provided to enable B2FIND to perform the harvesting process using this protocol: - -* OAI endpoint: This is the URL of the OAI provider server on data provider site, which must be open for OAI-PMH read requests -* OAI mdprefix: This is the OAI acronym for the metadata schema in which the provided XML records are coded in (default is DublinCore, Datacite is supported) -* OAI sets (optional): It is recommended to group your records in subsets, because this simplifies the controlled harvesting - -#### JSON-API -Some data providers offer their metadata encoded as JSON records, which can be retrieved, queried and browsed via a REST API. The API is generally RESTful and returns results in JSON, as the API follows the JSONAPI specification. - -#### CSW -Catalog Service for the Web (CSW) is a standard for exposing a catalogue of geospatial records in XML on the Internet (over HTTP). The catalogue is made up of records that describe geospatial data and services. B2FIND uses a CSW implementation to harvest XML records from so-called GeoNetwork portals. - -#### REST API -Communities or data provider that do not expose their metadata via the channels mentioned above may be integrated in B2FIND as long as they offer any REST API. However, developing a new harvesting method requires resources and effort, thus the integration process will take time. - - -### Supported Metadata Standards for Mapping -B2FIND supports several generic as well as thematic metadata standards, such as: - -* Datacite -* DublinCore -* OpenAire -* ISO 10115/19139 (INSPIRE) -* DDI 2.5 - -Community specific metadata schemas are supported as well, while the concrete mapping is done in close cooperation with the communities and/or data provider. B2FIND continously develops new 'reader' for the ingestion of records with other metadata schemas or standards, such as DCAT or CERIF. However, this requires resources and effort and will take time. - -As B2FIND follows a low-barrier approach, only four metadata elements are mandatory: - -1. the `Title` of the data set, i.e. a unique and unambiguous name or heading by which the referred resource is known -2. at least one `Identifier`, which has two roles: to identify the described resource, and to facilitate a persistent link to the research data set itself, which should be available on the web. If the identifier is not persistent (actionable/resolvable), an HTTP URI of the described resource must be provided as well. The URI should be as persistent as reasonably achievable. -3. a `Publisher` and -4. a `PublicationYear` - - -## EUDAT Core Metadata Schema -The EUDAT Core Metadata Schema defines and describes metadata for research output in order to transfer metadata information through different EUDAT CDI services. It originated from the need to define a common schema that allows to harmonise metadata elements used for storage, publication and discovery of digital research objects across EUDAT partners and beyond. You can access the EUDAT Core Metadata Schema XSD file [here](https://gitlab.eudat.eu/eudat-metadata/eudat-core-schema/-/blob/master/eudat-core.xsd) on GitLab. - -B2FIND uses the EUDAT Core Metadata Schema for its ingestion process. This generic metadata schema is based on Datacite with some additional elements: - -* `Temporal Coverage` which relates to the temporal coverage of the data -* `Discipline` which adheres to the scientific discipline(s) the data can be categorized in -* `Instrument` which allows to describe instruments that are used for producing the data -* `Contact` which is always useful - -A documentation of the EUDAT Core Metadata Schema can be found [here](https://eudat-b2find.github.io/schema-doc/introduction.html). - -Even though the EUDAT Core Metadata Schema was developed for streamlining metadata transfer across EUDAT services, of course it may be used by everyone to exchange metadata. For that, we strongly recommend using standardised vocabularies wherever possible. However, in order to make as much research output as possible searchable and findable, only few elements are mandatory and only few elements require specific attributes. - -Please note that the EUDAT Core Metadata Schema is maintained by the EUDAT Metadata Working Group, which means it may be subject of changes in further development. - - -## How to be integrated? -All (meta)data provider must agree with the licensing principles described above. Apart from that we need - -* a harvesting endpoint (where and how do you expose your metadata? Using a standardised protocol like OAI-PMH or CSW? Or via RestAPI? If RestAPI, please specify the metadata retrieval commands) -* a metadata schema (a generic standard like DublinCore, Datacite or OpenAire? Or thematic like ISO19115/19139 or DDI? Or your own? If so, please specify where we can retrieve the `Identifier`) -* Ideally you use our [template](https://b2drop.eudat.eu/s/KZJXroDeB24HEgi/download) for metadata integration in B2FIND. - -As soon as we have sufficient information we will do a testingestion on one of our testmachines, (if needed) discuss the results and (if needed) improve the mapping. After all issues are solved we will do the metadata ingestion on the productive B2FIND and document the integration process. - -If you want to join B2FIND as a data provider, please use the [EUDAT Request Tracker](https://eudat.eu/contact-support-request) or contact the B2FIND team directly via email: b2find@dkrz.de. diff --git a/docs/forusers.md b/docs/forusers.md deleted file mode 100644 index affcf28..0000000 --- a/docs/forusers.md +++ /dev/null @@ -1,97 +0,0 @@ -# For Users - -## Search Guide - -### The B2FIND Portal -You can access the EUDAT metadata discovery portal B2FIND at b2find.eudat.eu. It can be utilized in two ways: - -* The discovery web portal supports user-friendly navigation and filtering features. Powerful search functionalities are provided, that include: - * Free text search over the full text bodies of all datasets indexed in the B2FIND catalogue - * Geospatial and temporal search for all datasets, that cover a chosen region or, respectively, a chosen time period. - * Other faceted search, i.e. selecting values from certain metadata fields - -In the following we describe the usage of B2FIND step by step. - -### Welcome -The B2FIND homepage is your starting point. - -![B2FIND Homepage](img/for_users/searchguide/Welcome_B2FIND.png) - -Clicking on 'Data Catalogue' will lead you directly to the metadata catalogue with its faceted search options. By clicking 'Communities' you get an overview about all communities that provide metadata to B2FIND. -The 'Repositories' tab shows you a list of all underlying repositories that B2FIND harvest from. One community can comprise several repositories, like GFZ Data Services for example. The 'For Providers' and 'For Users' tabs lead to targeted information for either metadata providers or searchers. And in the 'About' section you find the official EUDAT service description as well as ways to contact the B2FIND team. - -### Free Text Search -There are two ways to start the search process and to get taken to the search result page with all available datasets listed on the right side and the different filter options or 'facets' on the left: - -1. By clicking 'Data Catalogue' and then using the free text search field on top of the page. -2. By pressing the magnifying glass in the free text field of the homepage. Here you can already enter a search query. - -Either way, simply type in your search terms and hit return. The Solr search engine will then search in all metadata fields for your search terms. For example, if you are interested in documents or datasets on 'climate' and you know that somebody with the name 'Scott' did something with it you would type "climate Scott" and get this result: - -![Search result](img/for_users/searchguide/climate_scott_searchresult.png) - -### Faceted Search -The faceted search interface provides you with options to filter your search by choosing 'facets'. This tool may help you to narrow down the search results for your specific search demands. - -B2FIND provides the opportunity to filter out datasets that have a given extent in space or in time. This is implemented by the following three graphical interfaces : - -* 'Filter by location' searches for all datasets which have an intersection with a region chosen from the world map. -* 'Filter by time' searches for all datasets which cover a chosen time period. -* 'Publication Year' searches for all datasets which have been published (or will be made publicly available) within a given period of years. - -Furthermore, you can filter by all facets provided by the [EUDAT Core metadata schema](https://eudat-b2find.github.io/schema-doc/introduction.html) which B2FIND uses. - -#### Filter by location - -In the world map widget in the left upper corner you can select a geographical region your search query is about. This triggers a search for all datasets whose spatial extension has an intersection with the selected region. - -For Windows: Hold STRG+left mouse button pressed and move over the map to create a bounding box over the region you are interested in. Releasing STRG+mouse button then triggers a search request (give it up to 5 seconds please), the results will appear on the right result page. -For Mac Os: hold Command+left mouse button. - -![map](img/for_users/searchguide/map_bb.png) - -If you draw a bounding box over Africa, for example, all datasets whose spatial extension has an overlap with the selected region are listed in the right panel.The dataset search result page will then look like this: - -![map search result](img/for_users/searchguide/map_bb_result.png) - -Note the dark grey tag announcing the filter in use. To delete the filter, just click on the 'x' on its right. - -#### Temporal Coverage -With the search widget 'Temporal Coverage' you can select a time period the research data are related to as described in the following. - -1. Expanding the facet 'Temporal Coverage' in the navigation panel opens the time line chart. Here you can already see, what time period most datasets (for example combined with the keyword search 'climate change') deal with. The temporal coverage depicted here strictly refers to the temporal period the datasets are about, **not** the publication year (that will be covered by the next facet). - -![temporal coverage search result](img/for_users/searchguide/tempcoverage.png) - -2. To select a certain time period, draw a box over the period in question by mouse-over while keeping the left mouse button pressed. Releasing your mouse will trigger the search and the result page -will change accordingly (again, give it 5 seconds please). - -![temporal coverage selection](img/for_users/searchguide/tempcoverage_bb -.png) - -3. Repeat as you wish for zooming in further. The search results on the right will narrow down accordingly. -To delete the temporal coverage filter, press on the 'x' on the filter button above the result page. - -#### Filter by Publication Year -With the search widget 'Publication Year' you can search for datasets that have been published (or in case of an embargo period: will be made publicly available) within a certain period of time. This gives a nice overview on when a lot of datasets catalogued in B2FIND were published on your search topic, i.e.psychoanalysis. - -This facet works exactly like the 'Temporal Coverage' one above: -1. Drag your mouse over the selected publication time period while holding the left mouse button pressed. -2. Release to trigger search. -3. Repeat to zoom in more. -4. If execution of the query is finished, all datasets that are published within the chosen years are listed in the right panel. To delete the filder, press the 'x' on the dark grey filter tag above the search results. - -### Metadata Display - -By clicking the dataset title the dataset view opens. The textual metadata are shown on the right side; on top the title, the description and the keywords as clickable buttons and underneath the other B2FIND metadata fields with their values in tabulated form. If the dataset contains spatial information, the spatial extent of the dataset is shown on the map in the upper left corner. - -![map](img/for_users/searchguide/map_bb.png) - -### Data Access -Among these fields are links provided that enable access on related data. - -* The data resources the metadata are related to. -* The original metadata - as harvested from the provider by OAI-PMH and formatted in XML. - -### Metadata as Harvested -If you want to examine the 'raw' metadata as originally harvested by B2FIND from the data provider you can click on the field 'MetadataAccess' and the associated XML record is listed in the browser. diff --git a/docs/img/B2FIND_faceted_search.jpg b/docs/img/B2FIND_faceted_search.jpg deleted file mode 100644 index 436672a..0000000 Binary files a/docs/img/B2FIND_faceted_search.jpg and /dev/null differ diff --git a/docs/img/favicon.ico b/docs/img/favicon.ico deleted file mode 100644 index 9450a50..0000000 Binary files a/docs/img/favicon.ico and /dev/null differ diff --git a/docs/img/for_users/searchguide/Welcome_B2FIND.png b/docs/img/for_users/searchguide/Welcome_B2FIND.png deleted file mode 100644 index fc51f2f..0000000 Binary files a/docs/img/for_users/searchguide/Welcome_B2FIND.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/climate_scott_searchresult.png b/docs/img/for_users/searchguide/climate_scott_searchresult.png deleted file mode 100644 index 1b1696b..0000000 Binary files a/docs/img/for_users/searchguide/climate_scott_searchresult.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/map_bb.png b/docs/img/for_users/searchguide/map_bb.png deleted file mode 100644 index f2f8ff2..0000000 Binary files a/docs/img/for_users/searchguide/map_bb.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/map_bb_result.png b/docs/img/for_users/searchguide/map_bb_result.png deleted file mode 100644 index 648d649..0000000 Binary files a/docs/img/for_users/searchguide/map_bb_result.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/singleresultpage.png b/docs/img/for_users/searchguide/singleresultpage.png deleted file mode 100644 index 535448f..0000000 Binary files a/docs/img/for_users/searchguide/singleresultpage.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/tempcoverage.png b/docs/img/for_users/searchguide/tempcoverage.png deleted file mode 100644 index 3b6b836..0000000 Binary files a/docs/img/for_users/searchguide/tempcoverage.png and /dev/null differ diff --git a/docs/img/for_users/searchguide/tempcoverage_bb.png b/docs/img/for_users/searchguide/tempcoverage_bb.png deleted file mode 100644 index fad4198..0000000 Binary files a/docs/img/for_users/searchguide/tempcoverage_bb.png and /dev/null differ diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index cde1150..0000000 --- a/docs/index.md +++ /dev/null @@ -1,3 +0,0 @@ -# Welcome to B2FIND - -*This site is under construction* diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index cd2b84d..0000000 --- a/mkdocs.yml +++ /dev/null @@ -1,26 +0,0 @@ -site_name: B2Find -site_url: http://b2find.eudat.eu/ -site_dir: site -nav: - - Home: index.md - - For Providers: forproviders.md - - For Users: forusers.md - - About: about.md - -markdown_extensions: - - toc: - permalink: True - -theme: - name: readthedocs - # name: windmill - highlightjs: true - hljs_languages: - - yaml - - rust - - xml - - python - features: - - toc.integrate - - toc.follow - diff --git a/requirements.txt b/requirements.txt index 9983e69..4d85cc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,2 @@ bokeh -pandas -mkdocs \ No newline at end of file +pandas \ No newline at end of file