diff --git a/docs/requirements.txt b/docs/requirements.txt
index 43d13278..41487bbe 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,6 +1,6 @@
# with version specifier
sphinx>=7.2.6
-pydata-sphinx-theme>=0.14.3
+pydata-sphinx-theme>=0.14.4
docutils>=0.20.1
# without version specifier
trafilatura
diff --git a/docs/tutorial-epsilla.rst b/docs/tutorial-epsilla.rst
index 1c491f79..ae0dcc58 100644
--- a/docs/tutorial-epsilla.rst
+++ b/docs/tutorial-epsilla.rst
@@ -1,5 +1,5 @@
-Text embedding
-===============
+Tutorial: Text embedding
+========================
.. meta::
:description lang=en:
@@ -28,7 +28,11 @@ In this tutorial, we will show you how to perform text embedding on results from
Setup Epsilla
------------------------------------------------
-In this tutorial, we will run an Epsilla databse server. You can start one locally with a `Docker `_ image.
+In this tutorial, we will need an Epsilla database server. There are two ways to get one: use the free cloud version or start one locally.
+
+Epsilla has a `cloud version `_ with a free tier. You can sign up and get a server running in a few steps.
+
+Alternatively, you can start one locally with a `Docker `_ image.
.. code-block:: bash
@@ -37,6 +41,8 @@ In this tutorial, we will run an Epsilla databse server. You can start one local
See `Epsilla documentation `_ for a full quick start guide.
+The rest of this guide assumes you are running a local Epsilla server on port 8888. If you are using the cloud version, replace the host and port with the cloud server address.
+
We need to install the database client. You can do this with pip:
.. code-block:: bash
@@ -145,5 +151,7 @@ We can now perform a vector search to find the most relevant project based on a
You will see the returned response is React! That is the correct answer. React is a modern frontend library, but PyTorch and Tensorflow are not.
+.. image:: https://static.scarf.sh/a.png?x-pxid=51f549d1-aabf-473c-b971-f8d9c3ac8ac5
+ :alt:
diff --git a/docs/used-by.rst b/docs/used-by.rst
index 0b514bac..c6ca6b39 100644
--- a/docs/used-by.rst
+++ b/docs/used-by.rst
@@ -18,11 +18,12 @@ Notable projects using this software
Known institutional users
^^^^^^^^^^^^^^^^^^^^^^^^^
-- `Data against Feminicide `_
-- `Kagi search engine `_ (notably Teclis component)
-- `Media Cloud platform `_ for media analysis
-- The Internet Archive's `sandcrawler `_ which crawls and processes the scholarly web for the `Fatcat catalog `_ of research publications
+- Falcon LLM (TII UAE) and its underlying `RefinedWeb Dataset `_
+- `FinGPT `_ (Finland)
+- `Media Cloud platform `_ for media analysis, e.g. `Data against Feminicide `_
- `SciencesPo médialab `_ through its `Minet `_ webmining package
+- `Teclis component `_ of the Kagi search engine
+- The Internet Archive's `sandcrawler `_ which crawls and processes the scholarly web for the `Fatcat catalog `_ of research publications
Various software repositories
@@ -32,6 +33,7 @@ Various software repositories
- `CommonCrawl downloader `_, to derive massive amounts of language data
- `GLAM Workbench `_ for cultural heritage (web archives section)
- `llama-hub `_, a library of data loaders for large language models
+- `LlamaIndex `_, a data framework for LLM applications
- `Obsei `_, a text collection and analysis tool
- `Vulristics `_, a framework for analyzing publicly available information about vulnerabilities
- `Website-to-Chatbot `_, a personalized chatbot
@@ -114,6 +116,7 @@ Publications citing Trafilatura
- Brandon, C., Doherty, A. J., Kelly, D., Leddin, D., & Margaria, T. (2023). HIPPP: Health Information Portal for Patients and Public. Applied Sciences, 13(16), 9453.
- Braun, D. (2021). "Automated Semantic Analysis, Legal Assessment, and Summarization of Standard Form Contracts", PhD Thesis, Technische Universität München.
- Chen, X., Zeynali, A., Camargo, C., Flöck, F., Gaffney, D., Grabowicz, P., ... & Samory, M. (2022). SemEval-2022 Task 8: Multilingual news article similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) (pp. 1094-1106).
+- De Cesare, A. M. (2023). Assessing the quality of ChatGPT’s generated output in light of human-written texts: A corpus study based on textual parameters. CHIMERA: Revista de Corpus de Lenguas Romances y Estudios Lingüísticos, 10, 179-210.
- Di Giovanni, M., Tasca, T., & Brambilla, M. (2022). DataScience-Polimi at SemEval-2022 Task 8: Stacking Language Models to Predict News Article Similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) (pp. 1229-1234).
- Dumitru, V., Iorga, D., Ruseti, S., & Dascalu, M. (2023). Garbage in, garbage out: An analysis of HTML text extractors and their impact on NLP performance. In 2023 24th International Conference on Control Systems and Computer Science (CSCS) (pp. 403-410). IEEE.
- Fröbe, M., Hagen, M., Bevendorff, J., Völske, M., Stein, B., Schröder, C., ... & Potthast, M. (2021). "The Impact of Main Content Extraction on Near-Duplicate Detection". arXiv preprint arXiv:2111.10864.
@@ -131,6 +134,7 @@ Publications citing Trafilatura
- Kuehn, P., Schmidt, M., & Reuter, C. (2023). ThreatCrawl: A BERT-based Focused Crawler for the Cybersecurity Domain. arXiv preprint arXiv:2304.11960.
- Laippala, V., Rönnqvist, S., Hellström, S., Luotolahti, J., Repo, L., Salmela, A., ... & Pyysalo, S. (2020). "From Web Crawl to Clean Register-Annotated Corpora", Proceedings of the 12th Web as Corpus Workshop (pp. 14-22).
- Laippala, V., Salmela, A., Rönnqvist, S., Aji, A. F., Chang, L. H., Dhifallah, A., ... & Pyysalo, S. (2022). Towards better structured and less noisy Web data: Oscar with Register annotations. In Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022) (pp. 215-221).
+- Luukkonen, R., Komulainen, V., Luoma, J., Eskelinen, A., Kanerva, J., Kupari, H. M., ... & Pyysalo, S. (2023). FinGPT: Large Generative Models for a Small Language. arXiv preprint arXiv:2311.05640.
- Madrid-Morales, D. (2021). "Who Set the Narrative? Assessing the Influence of Chinese Media in News Coverage of COVID-19 in 30 African Countries", Global Media and China, 6(2), 129-151.
- Meier-Vieracker, S. (2022). "Fußballwortschatz digital–Korpuslinguistische Ressourcen für den Sprachunterricht." Korpora Deutsch als Fremdsprache (KorDaF), 2022/01 (pre-print).
- Meng, K. (2021). "An End-to-End Computational System for Monitoring and Verifying Factual Claims" (pre-print).
@@ -140,6 +144,7 @@ Publications citing Trafilatura
- Öhman, J., Verlinden, S., Ekgren, A., Gyllensten, A. C., Isbister, T., Gogoulou, E., ... & Sahlgren, M. (2023). The Nordic Pile: A 1.2 TB Nordic Dataset for Language Modeling. arXiv preprint arXiv:2303.17183.
- Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Pannier, B., ... & Launay, J. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.
- Piskorski, J., Stefanovitch, N., Da San Martino, G., & Nakov, P. (2023). Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023) (pp. 2343-2361).
+- Pohlmann, J., Barbaresi, A., & Leinen, P. (2023). Platform regulation and “overblocking”–The NetzDG discourse in Germany. Communications, 48(3), 395-419.
- Robertson, F., Lagus, J., & Kajava, K. (2021). "A COVID-19 news coverage mood map of Europe", Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation (pp. 110-115).
- Salmela, A. (2022). "Distinguishing Noise and Main Text Content from Web-Sourced Plain Text Documents Using Sequential Neural Networks", Master's thesis, University of Turku.
- Sawczyn, A., Binkowski, J., Janiak, D., Augustyniak, Ł., & Kajdanowicz, T. (2021). "Fact-checking: relevance assessment of references in the Polish political domain", Procedia Computer Science, 192, 1285-1293.
diff --git a/tests/metadata_tests.py b/tests/metadata_tests.py
index 770e2cbd..551e9e3d 100644
--- a/tests/metadata_tests.py
+++ b/tests/metadata_tests.py
@@ -192,6 +192,7 @@ def test_dates():
mystring = 'Veröffentlicht am 1.9.17
'
metadata = extract_metadata(mystring, fastmode=False)
assert metadata.date == '2017-09-01'
+ # behavior for fastmode=True changed in htmldate==1.6.0. On 1.5.2 and earlier, result was None
metadata = extract_metadata(mystring, fastmode=True)
assert metadata.date == '2017-09-01'