diff --git a/docs/source/guidelines/index.md b/docs/source/guidelines/index.md new file mode 100644 index 0000000..e6e8550 --- /dev/null +++ b/docs/source/guidelines/index.md @@ -0,0 +1,103 @@ +# Development guidelines + +All our software is built as a team, and this means we must agree on some development conventions. This page +covers our internal development guidelines, which we hope are also useful to external contributors. + + +## General guidelines + +* All projects will use [Git](https://git-scm.com/) for version control and [GitHub](https://github.com/) for hosting. + +* By default we will use the [neuroinformatics-unit](https://github.com/neuroinformatics-unit) organisation for new projects. +Otherwise we may use other organisations, e.g. the [SWC organisation](https://github.com/SainsburyWellcomeCentre) when +working with SWC researchers, or when releasing software to the community. Often we will use a project-specific organisation (e.g. [BrainGlobe](https://github.com/brainglobe)). + +* Unless there is a specific reason not to, we will develop all software in the open (i.e. public). +This facilitates collaboration and encourages good software development practices. + +* All changes to a codebase must be via a +[Pull Request (PR)](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) approved by another member of the team. +How detailed the review should be will depend depends on the changes made, and the current status of the project. E.g. +a brand new, exploratory project may not need every line of code checked in detail, but a heavily used community project may do. + +* All software is developed in branches. When developing a new feature, or planning to make substantive changes +(e.g. not a quick bug fix which could go straight to a ready to review PR), make a new branch as well as a draft +PR detailing the purpose of the new code. Then, new commits can be easily tracked and commented on in-progress, +and extended periods of coding in isolation avoided. When the code is ready to be merged, the PR can be changed to +"ready to review" for full review. All code must be reviewed by at least one other team member +(or collaborator) before merging. + +* Code should be reviewed often, in as small a block as is practical. There should be no "magic unveiling" of code +developed in secret over a long time. Commits should be pushed to GitHub as often as practical, and feedback from +others (code review, and informal conversation) solicited as often as possible. + +* We will use the best tool (language, framework etc.) for the job. This depends on many factors such as the +computation itself, relevant other packages and collaborators. We will always prefer to work in open-source languages. +Python is likely to often be the most appropriate language due to the level of adoption in the community (we want +researchers to contribute to the code). +Guidelines specific to each language or framework can be found in +[Development guidelines for specific languages and frameworks](./languages_frameworks). + +* We aim that software can be used across operating systems, by novice users and using anything from a standard laptop +to HPC. + +* All software will be fully documented. Our understanding of documentation is guided by the systematic +[diataxis](https://diataxis.fr/) framework. We will strive to standardise the documentation structure across projects +by [gradually adopting the diataxis approach](https://diataxis.fr/how-to-use-diataxis/#). + +* All software will have (close to) 100% test coverage. This applies at all stages, i.e. tests shouldn't be added +"later" (later often doesn't come). **All code should be fully tested (and passing those tests) before submitting for +review**. If the code cannot be fully tested, or is failing the tests, the PR should be marked as "draft" to enable discussion. + +* All tests should run automatically on GitHub actions on macOS, Windows and Linux where appropriate. Other testing +permutations will depend on the language, e.g. for Python, all supported Python versions should be tested on at least +one operating system. + +* As far as possible, we should follow [Test Driven Development](https://en.wikipedia.org/wiki/Test-driven_development) practices. + +* We will use [semantic versioning](https://semver.org/) for all projects. + +## Pull requests +- Please submit _draft_ pull requests as early as possible (you can still push to the branch once submitted) to + allow for discussion. +- One approval of a PR is enough for it to be merged. +- Unless someone approves the PR with optional comments, the PR can be immediately merged by the approving reviewer. +- Please merge via "Squash and Merge" on GitHub to maintain a clean commit history. +- Ask for a review from someone specific if you think they would be a particularly suited reviewer (possibly noting + why they are suited on the PR description) + + +## Starting a new project +When beginning a new project, the above guidelines can be relaxed until the basic structure of the code is established. +In particular, rather than waiting for a series of small PRs to be merged, one model could be: + +* Start the project in a `dev` branch +* Open a draft PR, and assign someone as a reviewer (for another set of eyes, not necessarily a proper review) +* Keep committing to `dev` until a good initial project structure is defined +* Merge `dev` into `main` +* Move to feature branch/PR workflow (branching from `dev` if necessary while waiting for `dev` to be merged) + + +## Issue tags + +All new repositories in the NIU organisation will have the following tags. These can be added to existing projects to +help standardise how we categorise issues. As a general rule, `critical` issues should be tackled ASAP and +`priority` issues should be tackled before any others. + +| Label | Description | +|-----------------------|-----------------------------------------| +| critical | To be addressed first | +| priority | More important than other issues | +| documentation | Improvements or additions to documentation | +| bug | Something isn't working | +| enhancement | New feature or request | +| duplicate | This issue or pull request already exists | +| good first issue | Good for newcomers | +| question | Further information is requested | +| wontfix | This will not be worked on | + +## Further details +```{toctree} +:maxdepth: 2 +languages_frameworks +``` diff --git a/docs/source/guidelines/languages_frameworks.md b/docs/source/guidelines/languages_frameworks.md new file mode 100644 index 0000000..1552096 --- /dev/null +++ b/docs/source/guidelines/languages_frameworks.md @@ -0,0 +1,83 @@ +# Development guidelines for specific languages and frameworks + +## Python + +### Setting up a project +* To set up a new Python project, use the [NIU cookiecutter](https://github.com/neuroinformatics-unit/python-cookiecutter) +repository. This ensures that the required tooling is in place (e.g. CI) and also that we are all using a familiar +project structure. Part of using this cookiecutter is keeping it updated (e.g. Python versions, best practices) and +communicating those changes to all users. + +* Whenever possible, we will use `pyproject.toml` as the main configuration file for the project. Support for +`setup.cfg` and `setup.py` will be deprecated in future python versions. This will also be the default option for the SWC cookiecutter. + +### Virtual environments +* We will use [conda](https://docs.conda.io/en/latest/) environments for all Python projects. +* Package installations will be handled by either `conda` or `pip`. + +### Testing and coverage +* All Python software should have automated tests written using [pytest](https://docs.pytest.org) that run quickly on +all operating systems. Tests that take many minutes on low-powered systems (e.g. GitHub actions) are often ignored. +* We will use [codecov](https://about.codecov.io/) as the coverage reporting tool. This is free for open-source +projects and integrates with GitHub actions. + +### Formatting and QC +* We use [Black](https://black.readthedocs.io/en/stable/), [ruff](https://beta.ruff.rs/docs/), and [mypy](https://mypy.readthedocs.io/en/stable/) to ensure a consistent +code style. +* Currently, the above tools are automated in the SWC cookiecutter using [pre-commit](https://pre-commit.com/). Running +`pre-commit install` once locally will set up the pre-commit hooks to be executed automatically before each commit. + +### Continuous integration +* All pushes and pull requests will be built by [GitHub actions](https://docs.github.com/en/actions). This will usually include linting, testing and deployment. +* As a rule, actions will run on all operating systems (Linux, macOS, Windows) and on all Python versions that are supported by the project. +* GitHub actions workflows should be contributed to the [NIU actions repository](https://github.com/neuroinformatics-unit/actions) to aid re-use. + +### Automated versioning +We use [`setuptools_scm`](https://github.com/pypa/setuptools_scm) to automatically version packages. It should be +configured in the `pyproject.toml` file (as per the [cookiecutter template](https://github.com/neuroinformatics-unit/python-cookiecutter)). +`setuptools_scm` will automatically infer the version using git. +To manually set a new semantic version, create a tag and make sure the tag is pushed to GitHub. Make sure you commit +any changes you wish to be included in this version. E.g. to bump the version to `1.0.0`: + +```bash +git add . +git commit -m "Add new changes" +git tag -a v1.0.0 -m "Bump to version 1.0.0" +git push --follow-tags +``` + +### Dependency support +Packages have to choose which versions of dependencies they officially support, with minimum supported versions of each +dependency used in continuous integration testing. All NIU projects should follow +[NEP 29 — Recommend Python and NumPy version support as a community policy standard](https://numpy.org/neps/nep-0029-deprecation_policy.html) +to determine the **minimum** set of supported package versions: + +- The last 42 months of Python releases +- The last 24 months of NumPy releases + +In addition to this, the last 24 months of other dependencies should also be supported. + +### Licensing +* Unless the software has commercial potential or depends upon other software with a restrictive license, we will default to using +[The 3-Clause BSD License](https://opensource.org/licenses/BSD-3-Clause) (BSD-3-Clause). +* A `LICENSE` file should be included in the root of the repository (already present in the SWC cookiecutter). +* To learn more about our licensing policies, see [our licensing guide](https://howto.neuroinformatics.dev/open_science/Licensing.html). + +### Documentation +* We will use [Sphinx](https://www.sphinx-doc.org/en/master/) to generate documentation for Python projects. +We will build the documentation websites with the [PyData Sphinx Theme](https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html). +* Docstrings should be written in [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) format. +* The documentation structure should be informed by the [diataxis](https://diataxis.fr) framework. + +### Logging +All Python software should use the inbuilt Python logging module, rather than e.g. print statements. To standardise +this, we will use [fancylog](https://github.com/neuroinformatics-unit/fancylog). This tool should be updated +regularly to reflect best practices and ensure it is suitable for use with all projects. + +### Configuration files +User-editable configuration files should use the YAML format. As far as possible, machine-readable outputs from software should also use this format. + +### Release +* All Python software should be released on PyPI to enable a simple `pip install`. At a minimum, each project should +* have two owners - [Adam (adamltyson)](https://github.com/adamltyson) and the lead developer. Others can be added as owners/maintainers as appropriate. +* We will also aim to release projects on [conda-forge](https://conda-forge.org/), especially for packages with non-Python dependencies. diff --git a/docs/source/index.md b/docs/source/index.md index 882c5df..b4d0b4d 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -58,4 +58,5 @@ people projects blog/index resources +guidelines/index ``` diff --git a/docs/source/resources.md b/docs/source/resources.md index 61dd3f9..635eb5e 100644 --- a/docs/source/resources.md +++ b/docs/source/resources.md @@ -15,3 +15,4 @@ The HowTo site hosts resources for SWC & GCNU researchers (e.g. troubleshooting :::: +