Add development guidelines

docs/source/guidelines/index.md

@@ -0,0 +1,103 @@
+# Development guidelines
+
+All our software is built as a team, and this means we must agree on some development conventions. This page 
+covers our internal development guidelines, which we hope are also useful to external contributors.
+
+
+## General guidelines
+
+* All projects will use [Git](https://git-scm.com/) for version control and [GitHub](https://github.com/) for hosting.
+
+* By default we will use the [neuroinformatics-unit](https://github.com/neuroinformatics-unit) organisation for new projects. 
+Otherwise we may use other organisations, e.g. the [SWC organisation](https://github.com/SainsburyWellcomeCentre) when 
+working with SWC researchers, or when releasing software to the community. Often we will use a project-specific organisation (e.g. [BrainGlobe](https://github.com/brainglobe)).
+
+* Unless there is a specific reason not to, we will develop all software in the open (i.e. public). 
+This facilitates collaboration and encourages good software development practices.
+
+* All changes to a codebase must be via a 
+[Pull Request (PR)](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) approved by another member of the team. 
+How detailed the review should be will depend depends on the changes made, and the current status of the project. E.g. 
+a brand new, exploratory project may not need every line of code checked in detail, but a heavily used community project may do.
+
+* All software is developed in branches. When developing a new feature, or planning to make substantive changes 
+(e.g. not a quick bug fix which could go straight to a ready to review PR), make a new branch as well as a draft 
+PR detailing the purpose of the new code. Then, new commits can be easily tracked and commented on in-progress, 
+and extended periods of coding in isolation avoided. When the code is ready to be merged, the PR can be changed to 
+"ready to review" for full review. All code must be reviewed by at least one other team member 
+(or collaborator) before merging.
+
+* Code should be reviewed often, in as small a block as is practical. There should be no "magic unveiling" of code 
+developed in secret over a long time. Commits should be pushed to GitHub as often as practical, and feedback from 
+others (code review, and informal conversation) solicited as often as possible.
+
+* We will use the best tool (language, framework etc.) for the job. This depends on many factors such as the 
+computation itself, relevant other packages and collaborators. We will always prefer to work in open-source languages. 
+Python is likely to often be the most appropriate language due to the level of adoption in the community (we want 
+researchers to contribute to the code). 
+Guidelines specific to each language or framework can be found in 
+[Development guidelines for specific languages and frameworks](./languages_frameworks).
+
+* We aim that software can be used across operating systems, by novice users and using anything from a standard laptop 
+to HPC.
+
+* All software will be fully documented. Our understanding of documentation is guided by the systematic 
+[diataxis](https://diataxis.fr/) framework. We will strive to standardise the documentation structure across projects 
+by [gradually adopting the diataxis approach](https://diataxis.fr/how-to-use-diataxis/#).
+
+* All software will have (close to) 100% test coverage. This applies at all stages, i.e. tests shouldn't be added 
+"later" (later often doesn't come). **All code should be fully tested (and passing those tests) before submitting for 
+review**. If the code cannot be fully tested, or is failing the tests, the PR should be marked as "draft" to enable discussion.
+
+* All tests should run automatically on GitHub actions on macOS, Windows and Linux where appropriate. Other testing 
+permutations will depend on the language, e.g. for Python, all supported Python versions should be tested on at least 
+one operating system.
+
+* As far as possible, we should follow [Test Driven Development](https://en.wikipedia.org/wiki/Test-driven_development) practices.
+
+* We will use [semantic versioning](https://semver.org/) for all projects.
+
+## Pull requests
+- Please submit _draft_ pull requests as early as possible (you can still push to the branch once submitted) to
+  allow for discussion.
+- One approval of a PR is enough for it to be merged.
+- Unless someone approves the PR with optional comments, the PR can be immediately merged by the approving reviewer.
+- Please merge via "Squash and Merge" on GitHub to maintain a clean commit history.
+- Ask for a review from someone specific if you think they would be a particularly suited reviewer (possibly noting
+  why they are suited on the PR description)
+
+
+## Starting a new project
+When beginning a new project, the above guidelines can be relaxed until the basic structure of the code is established. 
+In particular, rather than waiting for a series of small PRs to be merged, one model could be:
+
+* Start the project in a `dev` branch
+* Open a draft PR, and assign someone as a reviewer (for another set of eyes, not necessarily a proper review)
+* Keep committing to `dev` until a good initial project structure is defined
+* Merge `dev` into `main`
+* Move to feature branch/PR workflow (branching from `dev` if necessary while waiting for `dev` to be merged)
+
+
+## Issue tags
+
+All new repositories in the NIU organisation will have the following tags. These can be added to existing projects to 
+help standardise how we categorise issues. As a general rule, `critical` issues should be tackled ASAP and 
+`priority` issues should be tackled before any others.
+
+| Label                 | Description                             |
+|-----------------------|-----------------------------------------|
+| critical              | To be addressed first                  |
+| priority              | More important than other issues       |
+| documentation         | Improvements or additions to documentation  |
+| bug                   | Something isn't working                |
+| enhancement           | New feature or request                 |
+| duplicate             | This issue or pull request already exists |
+| good first issue      | Good for newcomers                     |
+| question              | Further information is requested       |
+| wontfix               | This will not be worked on             |
+
+## Further details
+```{toctree}
+:maxdepth: 2
+languages_frameworks
+```

docs/source/guidelines/languages_frameworks.md

@@ -0,0 +1,83 @@
+# Development guidelines for specific languages and frameworks
+
+## Python
+
+### Setting up a project
+* To set up a new Python project, use the [NIU cookiecutter](https://github.com/neuroinformatics-unit/python-cookiecutter) 
+repository. This ensures that the required tooling is in place (e.g. CI) and also that we are all using a familiar 
+project structure. Part of using this cookiecutter is keeping it updated (e.g. Python versions, best practices) and 
+communicating those changes to all users.
+
+* Whenever possible, we will use `pyproject.toml` as the main configuration file for the project. Support for 
+`setup.cfg` and `setup.py` will be deprecated in future python versions. This will also be the default option for the SWC cookiecutter.
+
+### Virtual environments
+* We will use [conda](https://docs.conda.io/en/latest/) environments for all Python projects.
+* Package installations will be handled by either `conda` or `pip`.
+
+### Testing and coverage
+* All Python software should have automated tests written using [pytest](https://docs.pytest.org) that run quickly on 
+all operating systems. Tests that take many minutes on low-powered systems (e.g. GitHub actions) are often ignored.
+* We will use [codecov](https://about.codecov.io/) as the coverage reporting tool. This is free for open-source 
+projects and integrates with GitHub actions.
+
+### Formatting and QC
+* We use [Black](https://black.readthedocs.io/en/stable/), [ruff](https://beta.ruff.rs/docs/), and [mypy](https://mypy.readthedocs.io/en/stable/) to ensure a consistent
+code style.
+* Currently, the above tools are automated in the SWC cookiecutter using [pre-commit](https://pre-commit.com/). Running 
+`pre-commit install` once locally will set up the pre-commit hooks to be executed automatically before each commit.
+
+### Continuous integration
+* All pushes and pull requests will be built by [GitHub actions](https://docs.github.com/en/actions). This will usually include linting, testing and deployment.
+* As a rule, actions will run on all operating systems (Linux, macOS, Windows) and on all Python versions that are supported by the project.
+* GitHub actions workflows should be contributed to the [NIU actions repository](https://github.com/neuroinformatics-unit/actions) to aid re-use.
+
+### Automated versioning
+We use [`setuptools_scm`](https://github.com/pypa/setuptools_scm) to automatically version packages. It should be 
+configured in the `pyproject.toml` file (as per the [cookiecutter template](https://github.com/neuroinformatics-unit/python-cookiecutter)). 
+`setuptools_scm` will automatically infer the version using git. 
+To manually set a new semantic version, create a tag and make sure the tag is pushed to GitHub. Make sure you commit 
+any changes you wish to be included in this version. E.g. to bump the version to `1.0.0`:
+
+```bash
+git add .
+git commit -m "Add new changes"
+git tag -a v1.0.0 -m "Bump to version 1.0.0"
+git push --follow-tags
+```
+
+### Dependency support
+Packages have to choose which versions of dependencies they officially support, with minimum supported versions of each 
+dependency used in continuous integration testing. All NIU projects should follow 
+[NEP 29 — Recommend Python and NumPy version support as a community policy standard](https://numpy.org/neps/nep-0029-deprecation_policy.html) 
+to determine the **minimum** set of supported package versions:
+
+- The last 42 months of Python releases
+- The last 24 months of NumPy releases
+
+In addition to this, the last 24 months of other dependencies should also be supported.
+
+### Licensing
+* Unless the software has commercial potential or depends upon other software with a restrictive license, we will default to using
+[The 3-Clause BSD License](https://opensource.org/licenses/BSD-3-Clause) (BSD-3-Clause).
+* A `LICENSE` file should be included in the root of the repository (already present in the SWC cookiecutter).
+* To learn more about our licensing policies, see [our licensing guide](https://howto.neuroinformatics.dev/open_science/Licensing.html).
+
+### Documentation
+* We will use [Sphinx](https://www.sphinx-doc.org/en/master/) to generate documentation for Python projects. 
+We will build the documentation websites with the [PyData Sphinx Theme](https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html).
+* Docstrings should be written in [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) format.
+* The documentation structure should be informed by the [diataxis](https://diataxis.fr) framework.
+
+### Logging
+All Python software should use the inbuilt Python logging module, rather than e.g. print statements. To standardise 
+this, we will use [fancylog](https://github.com/neuroinformatics-unit/fancylog). This tool should be updated 
+regularly to reflect best practices and ensure it is suitable for use with all projects.
+
+### Configuration files
+User-editable configuration files should use the YAML format. As far as possible, machine-readable outputs from software should also use this format.
+
+### Release
+* All Python software should be released on PyPI to enable a simple `pip install`. At a minimum, each project should 
+* have two owners - [Adam (adamltyson)](https://github.com/adamltyson) and the lead developer. Others can be added as owners/maintainers as appropriate.
+* We will also aim to release projects on [conda-forge](https://conda-forge.org/), especially for packages with non-Python dependencies.

docs/source/index.md

@@ -58,4 +58,5 @@ people
 projects
 blog/index
 resources
+guidelines/index
 ```

docs/source/resources.md

@@ -15,3 +15,4 @@ The HowTo site hosts resources for SWC & GCNU researchers (e.g. troubleshooting
 ::::
 
 
+