Add markdown linting to pre-commit (#721)

* add markdown lint pre-commit

* remove lint from markdown

* add missing newline

.github/ISSUE_TEMPLATE/bug.md

@@ -24,9 +24,9 @@ Provide a description of your system and the software versions.
 
 - Machine
 - OS
-* Software versions: 
-    - Python user: `pip list`
-    - R user: `sessionInfo()`
+- Software versions:
+  - Python user: `pip list`
+  - R user: `sessionInfo()`
 
 ## Additional context
 

.github/ISSUE_TEMPLATE/feature-request.md

@@ -24,6 +24,6 @@ _e.g. I found and ad-hoc solution but incorporating it into the code base adds s
 
 Provide any alternate or temporary solutions you have considered or used.
 
-## Ideal behavior 
+## Ideal behavior
 
 Provide a brief description of the behavior you'd expect.

.markdownlint.yaml

@@ -0,0 +1,23 @@
+# Configuration for markdownlint-cli, used in pre-commit
+
+# Default state for all rules
+default: true
+
+# MD013/line-length - Line length increased to a large number
+MD013:
+  line_length: 1000
+
+# MD024/no-duplicate-heading/no-duplicate-header - Multiple headings with the same content
+MD024:
+  allow_different_nesting: true
+
+# MD026/no-trailing-punctuation - Trailing punctuation in heading
+MD026:
+  # Punctuation characters
+  punctuation: ".,;。，；：！"
+
+# MD033/no-inline-html - disable, allowing HTML
+MD033: false
+
+# MD041/first-line-heading/first-line-h1 - First line in a file should be a top-level heading
+MD041: false

.markdownlintignore

@@ -0,0 +1 @@
+SECURITY.md

.pre-commit-config.yaml

@@ -72,3 +72,8 @@ repos:
     hooks:
       - id: nbqa-black
         files: ^api/python/notebooks
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.35.0
+    hooks:
+    - id: markdownlint

README.md

@@ -2,14 +2,13 @@
 
 # CZ CELLxGENE Discover Census
 
-
 The Census of [CZ CELLxGENE Discover](https://cellxgene.cziscience.com/) is a free-to-use service (API + Data) that allows for querying its single-cell data corpus at low-latency directly into Python or R.
 
 To learn more and start using the Census please go to the main [**Census site**](https://chanzuckerberg.github.io/cellxgene-census/).
 
 ## Issues
 
-- Bugs: please submit a [github issue](https://github.com/chanzuckerberg/cellxgene-census/issues). 
+- Bugs: please submit a [github issue](https://github.com/chanzuckerberg/cellxgene-census/issues).
 - Security issues: if you believe you have found a security issue, in lieu of filing an issue please responsibly disclose it by contacting <[email protected]>.
 
 ## Reuse
@@ -19,4 +18,3 @@ The contents of this Github repository are freely available for reuse under the
 ## Code of Conduct
 
 This project adheres to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to <[email protected]>.
-

api/python/cellxgene_census/README.md

@@ -1,9 +1,9 @@
 # CZ CELLxGENE Discover Census
 
-The `cellxgene_census` package provides an API to facilitate the use of the CZ CELLxGENE Discover Census. For more information about the API and the project visit the [chanzuckerberg/cellxgene-census GitHub repo](https://github.com/chanzuckerberg/cellxgene-census/). 
-
+The `cellxgene_census` package provides an API to facilitate the use of the CZ CELLxGENE Discover Census. For more information about the API and the project visit the [chanzuckerberg/cellxgene-census GitHub repo](https://github.com/chanzuckerberg/cellxgene-census/).
 
 ## For More Help
+
 For more help, please file a issue on the repo, or contact us at <[email protected]>.
 
 If you believe you have found a security issue, we would appreciate notification. Please send email to <[email protected]>.

api/python/cellxgene_census/release_process.md

@@ -13,9 +13,9 @@ The following approach is used to manage releases of the Python cellxgene_census
 While not strictly required, this process assumes you have met the following prerequisites:
 
 - You have write access to the `chanzuckerberg/cellxgene-census` repo
-- You have an account on pypi.org and test.pypi.org, both with access to the cellxgene_census project. You will need to have created an API token on each account so that you can authenticate to test.pypi.org and pypi.org accounts when using `twine`. Usually this means adding these tokens to your `~/.pypirc` file. See https://pypi.org/help/#apitoken for more information.
+- You have an account on pypi.org and test.pypi.org, both with access to the cellxgene_census project. You will need to have created an API token on each account so that you can authenticate to test.pypi.org and pypi.org accounts when using `twine`. Usually this means adding these tokens to your `~/.pypirc` file. See <https://pypi.org/help/#apitoken> for more information.
 - You have the Github CLI tool (`gh`) installed. See [documentation](https://cli.github.com/).
-- You have the `pipx` CLI tool installed. See [documentation](https://pypa.github.io/pipx/). 
+- You have the `pipx` CLI tool installed. See [documentation](https://pypa.github.io/pipx/).
 
 ## Step 1: Building the package assets
 
@@ -28,14 +28,16 @@ Unless you are revising and testing the build process itself, there is no need t
 Any pre-built asset on Github can be installed and tested from the Github URL. For example:
 
 1. Identify the GH workflow run ID that contains the asset you wish to test. A simple way to do this is:
+
    ```shell
-   $ gh run list
+   gh run list
    ```
+
    Alternatively, you can use the "Actions" tag in the GitHub web UI.
 2. Download the build artifact.zip from GitHub, using the GH Action run ID associated with the `build` action for your commit OR utilizing the web UI:
 
    ```shell
-   $ gh run download <ID>
+   gh run download <ID>
    ```
 
    If you download using the browser, unzip into a temp directory, e.g.,
@@ -48,9 +50,10 @@ Any pre-built asset on Github can be installed and tested from the Github URL. F
    ```
 
 3. Install and test the downloaded build, e.g.,
+
    ```shell
-   $ pip uninstall cellxgene-census
-   $ pip install ./artifact/cellxgene_census-*-any.whl
+   pip uninstall cellxgene-census
+   pip install ./artifact/cellxgene_census-*-any.whl
    ```
 
 To test a release candidate:
@@ -71,46 +74,58 @@ To create a release, perform the following:
 
 1. Identify both the (tested & validated) commit and semver for the release.
 2. Tag the commit with the release version (_including_ a `v` prefix) and push the tag to origin. **Important**: use an annotated tag, e.g., `git tag -a v1.9.4 -m 'Release 1.9.4`. For example (please replace <SEMVER> with your version, _including_ a `v`, e.g. `v1.9.4`:
+
    ```shell
-   $ git tag -a <SEMVER> -m 'Release <SEMVER>'
-   $ git push origin <SEMVER>
+   git tag -a <SEMVER> -m 'Release <SEMVER>'
+   git push origin <SEMVER>
    ```
+
 3. Trigger a build for this tag by manually triggering the `py-build.yml` workflow. For example:
+
    ```shell
-   $ gh workflow run py-build.yml --ref <SEMVER>
+   gh workflow run py-build.yml --ref <SEMVER>
    ```
+
 4. When the workflow completes, make note of the run ID (e.g., using `gh run list`).
 5. Optional, _but recommended_: download the asset from the build workflow and validate it.
-6. Create and publish a GitHub Release [here](https://github.com/chanzuckerberg/cellxgene-census/releases/new). Set the release title to the `<SEMVER>`. Select `Set as the latest release`. Use the `Generate Release Notes` button to auto-populate the summary with a changelog. It is reasonable to remove any R-specific or builder-specific entries. Add a prelude to the summary, noting any major new features or API changes. 
+6. Create and publish a GitHub Release [here](https://github.com/chanzuckerberg/cellxgene-census/releases/new). Set the release title to the `<SEMVER>`. Select `Set as the latest release`. Use the `Generate Release Notes` button to auto-populate the summary with a changelog. It is reasonable to remove any R-specific or builder-specific entries. Add a prelude to the summary, noting any major new features or API changes.
 
 ## Step 4: Publish assets to PyPi
 
 To publish built release assets to PyPi (_note_: this will require your pypi/testpypi login):
 
 1. Delete any existing release builds you may have accumulated in the past: `rm ./artifact/*`.
 2. Download the assets built for your release commit, using the same method as step 2 above, e.g.,
+
    ```shell
-   $ gh run download <ID>
+   gh run download <ID>
    ```
+
 3. Optional: upload to TestPyPi (this assumes the downloaded assets are in ./artifact/).
 
    ```shell
    pipx run twine upload --repository testpypi ./artifact/*
    ```
 
-   Following the upload, confirm correct presentation on the project page and ability to download install from TestPyPi. 
+   Following the upload, confirm correct presentation on the project page and ability to download install from TestPyPi.
 4. To test installation from TestPyPi:
+
    ```shell
    pip install --no-cache-dir -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple cellxgene-census
    python -c "import cellxgene_census; print(cellxgene_census.__version__)"
    ```
+
    Note that the `--extra-index-url` option ensures that any transitive package dependencies that are _not_ available on
    `test.pypi.org` can be satisfied by installing them from the production `pypi.org`.  You can find more information
    [here](https://packaging.python.org/en/latest/guides/using-testpypi/).
 5. Use twine to upload to PyPi (this assumes the downloaded assets are in ./artifact/), e.g.,
+
    ```shell
    pipx run twine upload ./artifact/*
+   ```
+
 6. Test the installation from PyPi, as a final sanity check. Note that it may take a minute for the new release to be visible on pypi.org:
+
    ```shell
    pip install --no-cache-dir cellxgene-census
    python -c "import cellxgene_census; print(cellxgene_census.__version__)"

api/python/cellxgene_census/tests/README.md

@@ -36,7 +36,7 @@ You can also combine them, e.g.,
 
 > pytest -m 'not live_corpus' --expensive --experimental
 
-# Acceptance (expensive) tests
+## Acceptance (expensive) tests
 
 These tests are periodically run, and are not part of CI due to their overhead.
 
@@ -52,13 +52,13 @@ When run, please record the results in this file (below) and commit the change t
 - any run notes
 - full output of: `pytest -v --durations=0 --expensive ./api/python/cellxgene_census/tests/`
 
-## 2023-07-26
+### 2023-07-26
 
 - Host: EC2 instance type: `r6id.32xlarge`, all nvme mounted as swap.
 - Uname: Linux 5.19.0-1028-aws #29~22.04.1-Ubuntu SMP Tue Jun 20 19:12:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 - Python & census versions:
 
-```
+```python
 >>> import cellxgene_census, tiledbsoma
 >>> tiledbsoma.show_package_versions()
 tiledbsoma.__version__        1.2.7
@@ -71,7 +71,7 @@ OS version                    Linux 5.19.0-1028-aws
 
 **Pytest output:**
 
-```
+```text
 ============================= test session starts ==============================
 platform linux -- Python 3.10.6, pytest-7.1.3, pluggy-1.0.0 -- /home/ubuntu/venv/bin/python
 cachedir: .pytest_cache
@@ -237,20 +237,21 @@ api/python/cellxgene_census/tests/experimental/pp/test_stats.py::test_mean_varia
 =============== 72 passed, 202 deselected in 14584.03s (4:03:04) ===============
 ```
 
-## 2023-06-23
+### 2023-06-23
 
 - Host: EC2 instance type: `r6id.32xlarge`, all nvme mounted as swap.
 - Uname: Linux 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 - Python & census versions:
-```
+
+```python
 >>> import cellxgene_census, tiledbsoma
 >>> tiledbsoma.show_package_versions()
 tiledbsoma.__version__        1.2.5
 TileDB-Py tiledb.version()    (0, 21, 5)
 TileDB core version           2.15.4
 libtiledbsoma version()       libtiledb=2.15.2
 python version                3.10.6.final.0
-OS version  
+OS version
 >>> cellxgene_census.__version__
 '1.2.1'
 >>> cellxgene_census.get_census_version_description('latest')
@@ -259,7 +260,7 @@ OS version
 
 **Pytest output:**
 
-```
+```text
 ============================= test session starts ==============================
 platform linux -- Python 3.10.6, pytest-7.4.0, pluggy-1.2.0 -- /home/ubuntu/venv/bin/python3
 cachedir: .pytest_cache
@@ -420,12 +421,13 @@ test_util.py::test_uri_join PASSED                                       [100%]
 =============== 69 passed, 111 deselected in 15040.59s (4:10:40) ===============
 ```
 
-## 2023-05-16
+### 2023-05-16
 
 - Host: EC2 instance type: `r6id.32xlarge`, all nvme mounted as swap.
 - Uname: Linux 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 - Python & census versions:
-```
+
+```python
 >>> import cellxgene_census, tiledbsoma
 >>> tiledbsoma.show_package_versions()
 tiledbsoma.__version__        1.2.3
@@ -442,7 +444,7 @@ OS version                    Linux 5.19.0-1022-aws
 
 **Pytest output:**
 
-```
+```text
 ============================= test session starts ==============================
 platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0 -- /home/ubuntu/venv-cellxgene-census/bin/python3
 cachedir: .pytest_cache
@@ -565,15 +567,15 @@ tests/test_util.py::test_uri_join PASSED                                 [100%]
 ======================= 51 passed in 10696.20s (2:58:16) =======================
 ```
 
-## 2023-03-29
+### 2023-03-29
 
-**Config**
+**Config:**
 
 - Host: EC2 instance type: `r6id.32xlarge`, all nvme mounted as swap.
 - Uname: Linux bruce.aegea 5.15.0-1033-aws #37~20.04.1-Ubuntu SMP Fri Mar 17 11:39:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 - Python & census versions:
 
-```
+```python
 In [1]: import cell_census, tiledbsoma
 
 In [2]: tiledbsoma.show_package_versions()
@@ -585,7 +587,7 @@ python version                3.9.16.final.0
 OS version                    Linux 5.15.0-1033-aws
 
 In [3]: cell_census.get_census_version_description('latest')
-Out[3]: 
+Out[3]:
 {'release_date': None,
  'release_build': '2023-03-16',
  'soma': {'uri': 's3://cellxgene-data-public/cell-census/2023-03-16/soma/',
@@ -603,14 +605,14 @@ The test `test_acceptance.py::test_get_anndata[None-homo_sapiens]` manifest a la
 
 **Pytest output:**
 
-```
+```text
 $ pytest -v --durations=0 --expensive ./api/python/cell_census/tests/
 ==================================================== test session starts =====================================================
 platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0 -- /home/bruce/cell-census/venv/bin/python
 cachedir: .pytest_cache
 rootdir: /home/bruce/cell-census/api/python/cell_census, configfile: pyproject.toml
 plugins: requests-mock-1.10.0, anyio-3.6.2
-collected 45 items                                                                                                           
+collected 45 items
 
 api/python/cell_census/tests/test_acceptance.py::test_load_axes[homo_sapiens] PASSED                                   [  2%]
 api/python/cell_census/tests/test_acceptance.py::test_load_axes[mus_musculus] PASSED                                   [  4%]

api/python/notebooks/README.md

@@ -8,6 +8,7 @@ Demonstration notebooks for the CZ CELLxGENE Discover Census. There are two kind
 ## Dependencies
 
 You must be on a Linux or MacOS system, with the following installed:
+
 * Python 3.8 to 3.11
 * Jupyter or some other means of running notebooks (e.g., vscode)
 
@@ -19,25 +20,32 @@ not been tested).
 I also recommend you use a `d` instance type, and mount all of the NVME drives as swap,
 as it will keep you from running out of RAM.
 
-
 ## Set up Python environment
+
 1. (optional, but highly recommended) In your working directory, make and activate a virtual environment. For example:
-```shell
-  $ python -m venv ./venv
-  $ source ./venv/bin/activate
-```
+
+    ```shell
+      python -m venv ./venv
+      source ./venv/bin/activate
+    ```
+
 2. Install the required dependencies:
-```shell
-  $ pip install -U -r cellxgene-census/api/python/notebooks/requirements.txt
-```
+
+    ```shell
+      pip install -U -r cellxgene-census/api/python/notebooks/requirements.txt
+    ```
+
 ## Verify your installation
+
 Check that your installation works - this make take a few seconds, as it loads metadata from S3:
+
 ```shell
 $ python -c 'import cellxgene_census; print(cellxgene_census.open_soma().soma_type)'
 SOMACollection
 ```
 
 ## Run notebooks
+
 Run notebooks, which you can find in the `cellxgene-census/api/python/notebooks` directory.
 
 ## For more help