docs: update including used-by and deprecation info (#745)

* docs: general update

* update changelog and corresponding doc pages

HISTORY.md

@@ -4,9 +4,39 @@
 ## future v2.0.0
 
 Breaking changes:
-- `bare_extraction()`: the function now returns an instance of the Document class by default
-- `bare_extraction()`: `as_dict` deprecation warning → use `.as_dict()` method on return value
-- `bare_extraction()` and `extract()`: `no_fallback` deprecation warning → use `fast` instead
+- Python 3.6 and 3.7 deprecated (#709)
+- `bare_extraction()`:
+   - now returns an instance of the `Document` class by default
+   - `as_dict` deprecation warning → use `.as_dict()` method on return value (#730)
+- `bare_extraction()` and `extract()`: `no_fallback` deprecation warning → use `fast` instead (#730)
+- downloads: remove `decode` argument in `fetch_url()` → use `fetch_response` instead (#724)
+- deprecated graphical user interface now removed (#713)
+- extraction: move `max_tree_size` parameter to `settings.cfg` (#742)
+- see [Python](https://trafilatura.readthedocs.io/en/latest/usage-python.html#deprecations) and [CLI](https://trafilatura.readthedocs.io/en/latest/usage-cli.html#deprecations) deprecations in the docs
+
+Fixes:
+- set `options.source` before raising error on empty doc tree by @dmoklaf (#707)
+- robust encoding in `options.source` (#717)
+- more robust mapping for conversion to HTML (#721)
+- CLI downloads: use all information in settings file (#734)
+- downloads: cleaner urllib3 code (#736)
+- CLI: print URLs early for feeds and sitemaps with `--list` with @gremid (#744)
+
+Metadata:
+- more robust URL extraction (#710)
+
+Maintenance:
+- remove already deprecated functions and args (#716)
+- add type hints (#723, #728)
+- setup: use `pyproject.toml` file (#715)
+- simplify code (#708, #709, #727)
+- better debug messages in `main_extractor` (#714)
+- evaluation: review data, update packages, add magic_html (#731)
+- setup: explicit exports through `__all__` (#740)
+
+Documentation:
+- fix link in `docs/index.html` by @nzw0301 (#711)
+- remove docs from published packages (#743)
 
 
 ## 1.12.2

docs/index.rst

@@ -127,6 +127,12 @@ Contributions of all kinds are welcome. Visit the `Contributing page <https://gi
 Many thanks to the `contributors <https://github.com/adbar/trafilatura/graphs/contributors>`_ who extended the docs or submitted bug reports, features and bugfixes!
 
 
+Changes
+-------
+
+For version history and changes see the `changelog <https://github.com/adbar/trafilatura/blob/master/HISTORY.md>`_.
+
+
 Context
 -------
 

docs/installation.rst

@@ -72,6 +72,7 @@ In case this does not happen automatically, specify the version number:
 
 ``pip install trafilatura==number``
 
+- Last version for Python 3.6 and 3.7: ``1.12.2``
 - Last version for Python 3.5: ``0.9.3``
 - Last version for Python 3.4: ``0.8.2``
 
@@ -123,6 +124,8 @@ htmldate[all] / htmldate[speed]
 py3langid
     Language detection on extracted main text
 pycurl
-    Faster downloads, possibly less robust though
+    Faster downloads, useful where urllib3 fails
+urllib3[socks]
+    Downloads through SOCKS proxy with urllib3
 zstandard
     Additional compression algorithm for downloads

docs/requirements.txt

@@ -1,7 +1,8 @@
 # with version specifier
-sphinx>=7.3.7
-pydata-sphinx-theme>=0.15.4
+sphinx>=8.1.3
+pydata-sphinx-theme>=0.16.0
 docutils>=0.21.2
+sphinx-sitemap>=2.6.0
+
 # without version specifier
 trafilatura
-sphinx-sitemap

docs/usage-cli.rst

@@ -289,10 +289,10 @@ For all usage instructions see ``trafilatura -h``:
                    [--parallel PARALLEL] [-b BLACKLIST] [--list]
                    [-o OUTPUTDIR] [--backup-dir BACKUP_DIR] [--keep-dirs]
                    [--feed [FEED] | --sitemap [SITEMAP] | --crawl [CRAWL] |
-                   --explore [EXPLORE]] [--archived]
+                   --explore [EXPLORE] | --probe [PROBE]] [--archived]
                    [--url-filter URL_FILTER [URL_FILTER ...]] [-f]
                    [--formatting] [--links] [--images] [--no-comments]
-                   [--no-tables] [--only-with-metadata]
+                   [--no-tables] [--only-with-metadata] [--with-metadata]
                    [--target-language TARGET_LANGUAGE] [--deduplicate]
                    [--config-file CONFIG_FILE] [--precision] [--recall]
                    [--output-format {csv,json,html,markdown,txt,xml,xmltei} | 

docs/usage-gui.rst

@@ -8,6 +8,10 @@ Note that the GUI was a feature in Trafilatura until version 1.8.1, but it is cu
 If installation fails, usage on the command-line is recommended.
 
 
+.. hint::
+    This interface is removed until further notice starting from Trafilatura version 2, mostly due to issues with cross-platform tests and maintenance.
+
+
 Installation
 ~~~~~~~~~~~~
 

docs/usage-python.rst

@@ -481,7 +481,12 @@ Deprecations
 
 The following functions and arguments are deprecated:
 
-- extraction: ``process_record()`` function → use ``extract()`` instead
+- extraction:
+   - ``process_record()`` function → use ``extract()`` instead
+   - ``csv_output``, ``json_output``, ``tei_output``, ``xml_output`` → use ``output_format`` parameter instead
+   - ``bare_extraction(as_dict=True)`` → the function returns a ``Document`` object, use ``.as_dict()`` method on it
+   - ``bare_extraction()`` and ``extract()``: ``no_fallback`` → use ``fast`` instead
+   - ``max_tree_size`` parameter moved to ``settings.cfg`` file
+- downloads: ``decode`` argument in ``fetch_url()`` → use ``fetch_response`` instead
 - utils: ``decode_response()`` function → use ``decode_file()`` instead
-- extraction: ``csv_output``, ``json_output``, ``tei_output``, ``xml_output`` → use ``output_format`` parameter instead
 - metadata: ``with_metadata`` (include metadata) had once the effect of today's ``only_with_metadata`` (only documents with necessary metadata)