diff --git a/.gitignore b/.gitignore index b91f91736aa..c17c6f21067 100644 --- a/.gitignore +++ b/.gitignore @@ -93,3 +93,6 @@ package.json artifacts deposit-* cabal.project.freeze + +# Emacs +*~ diff --git a/docs/site/src/SUMMARY.md b/docs/site/src/SUMMARY.md index 73d0df33a80..aa1f3f865d7 100644 --- a/docs/site/src/SUMMARY.md +++ b/docs/site/src/SUMMARY.md @@ -54,8 +54,14 @@ - [Testing](contributor/how/testing.md) - [Continuous Integration](contributor/how/continuous-integration.md) - [Release Process](contributor/how/release-process.md) - - [Release checklist](contributor/how/release-checklist.md) + - [Release Checklist template](contributor/how/release-checklist.md) - [Code Review Guidelines](contributor/how/code-review-guidelines.md) - [Notes](contributor/notes.md) - [Updating Dependencies](contributor/notes/updating-dependencies.md) - [Notes from upgrading GHC version](contributor/notes/notes-from-upgrading-ghc-version.md) + - [Decisions Record](contributor/decisions.md) + - [2024-12-03 - Store documents alongside code](contributor/decisions/2024-12-03-document-with-code.md) + - [2024-03-23 - Release Process](contributor/decisions/2024-03-13-release-process.md) + - [2023-07-28 - Team Workflow](contributor/decisions/2023-07-28-workflow-review.md) + - [2023-01-27 - Continuous Integration](contributor/decisions/2023-01-27-continuous-integration.md) + - [2022-10-04 - Document Storage](contributor/decisions/2022-10-04-document-storage.md) diff --git a/docs/site/src/contributor/decisions.md b/docs/site/src/contributor/decisions.md new file mode 100644 index 00000000000..993939adf57 --- /dev/null +++ b/docs/site/src/contributor/decisions.md @@ -0,0 +1,3 @@ +# Decisions Record + +We keep in these documents a timestamped record of process related decisions. diff --git a/docs/site/src/contributor/decisions/2022-10-04-document-storage.md b/docs/site/src/contributor/decisions/2022-10-04-document-storage.md new file mode 100644 index 00000000000..b15b8e54c56 --- /dev/null +++ b/docs/site/src/contributor/decisions/2022-10-04-document-storage.md @@ -0,0 +1,42 @@ +# **Document Storage :: AP** + +| | | +|---------|------------| +| Started | 2022-09-30 | +| Decided | 2022-10-14 | + +## **Why** + +We need to store our documents — such as decision records, design documents — somewhere, but there are a plethora of options: Google Docs, Confluence, Github, Custom wiki, … with various pros and cons. This decision records our current choice and its rationale. + +## **Decision** + +The entry point to our documentation is the Adrestia (Haskell) Team Dashboard on Google Docs. All other documents are hyperlinked from this dashboard. + +For documents that are not public, such as meeting minutes or Advice Process records, we use Google Docs. + +For documents that may eventually be public, such as design documentation or release checklists, we use Rodney’s HedgeDoc instance at [https://md.adrestia.iohkdev.io/](https://md.adrestia.iohkdev.io/) . (As Rodney is no longer with us, I’m actually not happy with that state of affairs. Moritz Angerman has set up another instance at [https://hd.devx.iog.io/](https://hd.devx.iog.io/) ). + +## **Details** + +See Section “Decision”. + +## **Rationale** + +**Making** a **decision** — Document storage systems come with various trade-offs, and none seems to be perfect. The act of simply picking one system is at least as important as making sure that the pick is reasonable — we can’t even record our choice unless we have already made it\! Thus, I (the decision maker) have decided to favor a speedy decision over an optimal one. + +What do we want out of a document storage system? Here is a list of features, in order of importance, and some thoughts as to how these are satisfied with my current decision. + +**Sharing** and **Access Control** — We need to be able to share documents with team members, and maybe external parties. Also, we need to keep some types of documents internal to the company or team. + +**Readability** — Documents should be pleasant on the eyes. For me, this means that they should have a rendered view, but they do not need to be WSIWYG. + +**Commenting** and **Tracking Changes** — It should be pleasant to comment on documents and track changes that different contributors made. For me, this means that it should be possible to comment on the *rendered* view. For example, markdown documents in a Github repository do not satisfy this property, as I can only see changes in the source code, and comments are also attached to the source code rather than the rendering. + +**Writing format** — Documents should be easy to export to Markdown, so that we can more easily move them between different storage solutions, e.g. from HedgeDoc to our Github repositories. Unfortunately, Google Docs does not allow easy export to Markdown; this is particularly painful for documents with a lot of source code — hence we use HedgeDoc for more technical documents. + +**Navigation** — Documents should be easy to **search** (“I know what I’m looking for, but I cannot find it.”) and to **discover** (“I don’t know what I’m looking for, but I find it anyway”). + +From my experience, good discoverability comes from good curation. In my view, Hyperlinks are the best tool for curating document listings. For example, Confluence has a hierarchical view for each space, but I still find it hard to navigate if the sections are not curated properly. The [Haskell wikibook](https://en.wikibooks.org/wiki/Haskell) uses ToC templates and manually curated tables of hyperlinks, I think it works very well. + +For searching a document, curation helps, too, but a search box is usually more effective. diff --git a/docs/site/src/contributor/decisions/2023-01-27-continuous-integration.md b/docs/site/src/contributor/decisions/2023-01-27-continuous-integration.md new file mode 100644 index 00000000000..b62d00b669f --- /dev/null +++ b/docs/site/src/contributor/decisions/2023-01-27-continuous-integration.md @@ -0,0 +1,156 @@ +# **Continuous Integration** + +| | | +|--------------|------------| +| Started | 2022-12-06 | +| Decided | 2023-01-27 | +| Last amended | 2023-04-04 | + +## **Why** + +The sudden decommission of Hydra forces us to revisit our Continuous Integration (CI) setup. + +## **Decision** + +We predominantly rely on [Buildkite](https://buildkite.com) as CI system. + +We build **artifacts** and run **checks** on them. Artifacts include compiled executables, but also the source code itself. Checks include unit and integration tests, but also source code linters. We specify our artifacts in \`flake.nix\`, and most of our checks, too. + +We perform these builds and checks with different **granularity** — in order to keep our computing resources within reasonable limits, we don’t automatically check everything on every commit. Instead, the granularities are: + +* `post-commit` \= (at most once) after each commit; for a quick sanity check after a push +* `pre-merge` \= before merging each pull request; for a reasonably complete, automated check that master will satisfy functional requirements +* `post-merge` \= after merging each pull request to master; for an exhaustive check that master satisfies functional requirements on all platforms +* `nightly` \= every night; for an exhaustive, automated check of functional and non-functional requirements + +The following **table** lists our artifacts, the checks performed on them (\`.\` for build), the granularity at which the check is performed, and the CI system used for doing that: + +| Artifact | Check | Granularity | CI System | (Status) | +| :---- | :---- | :---- | :---- | ----- | +| Source code | Code formatting style | post-commit | Buildkite | 🔵 | +| Documentation | . | post-commit | Github Action | 🔵 | +| Pull request (PR) | Mergeable to master concurrently with other PRs | pre-merge | Bors | 🔵 | +| Compiled modules | . | post-commit | Buildkite | 🔵 | +| | Unit tests (linux) | post-commit | Buildkite | 🔵 | +| | Unit tests (macos) | post-merge | Buildkite | 🔵 | +| | Unit tests (windows) | nightly | Github Action | 🔵 | +| Executables / Release archive | . (linux) | pre-merge | Buildkite | 🟡[ADP-2502](https://cardanofoundation.atlassian.net/browse/ADP-2502) | +| | . (macos) | post-merge | Buildkite | 🔵 | +| | . (windows, cross-compiled) | pre-merge | Buildkite | 🔵 | +| Executables | Integration tests (linux) | pre-merge | Buildkite | 🔵 | +| | ~~Integration tests (macos)~~ | | Buildkite | 🔴[ADP-2522](https://cardanofoundation.atlassian.net/browse/ADP-2522) | +| | ~~Integration tests (windows)~~ | | Github Action | 🔴[ADP-2517](https://cardanofoundation.atlassian.net/browse/ADP-2517) | +| | Benchmarks | nightly | Buildkite | 🔵 | +| Release archive | E2E tests | nightly | Github Action | 🔵 | +| Docker image | . | post-commit | Buildkite | 🔵 | + +Legend: Status 🔵\= working; 🟡= needs work; 🔴\= not working + +## Details + +### **Granularity** + +* Granularity refers to **automatic** actions taken by the CI system. It should be possible to trigger a build or check manually at any time. +* The purpose of granularity is to **conserve** computing **resources** — in a world with infinite resources, the system would perform every build and check on every commit. +* The name of the granularity “**post-commit**” was chosen for brevity — the action is performed automatically on the **latest** commit after a \`git push\`, not on the git commits in between. In other words, the action is performed at most once per commit. +* We use the “**post-merge**” granularity for actions that + * consume scarce resources and have a high chance of failing, e.g. builds and checks on macOS +* We use the “**nightly**” granularity for actions that + * consume many resources, e.g. benchmarks + +### **CI System** + +As a general rule, we choose + +* Github Actions for actions that + * are very simple and do not require a nix store / environment + * run on Windows +* Buildkite otherwise + * especially for actions that require a nix store + +We have a **tension** where we have to set up some checks (e.g. unit tests) in two different environments due to different availability of operating systems: + +* Linux, macOS — in Buildkite +* Windows — in Github Actions + +We hope to address this tension by requesting a **Windows machine** for use with **Buildkite**. + +### **Platform macOS** + +At the time of writing, we have two mac-mini machines that act as Buildkite agents. Unfortunately, they are frequently overloaded and fail the builds or checks. Hence, we only use granularity “post-merge” or “nightly” for them. + +### **Company Processes** + +For developing and maintaining our CI, we may use DevX/SRE expertise from IOG. + +* Our **tribe** is responsible for choosing our CI tooling +* Our **tribe** should have a process for getting DevX/SRE support +* Our tribes’ DevX/SRE resources can help teach us how to debug problems that arise +* Link to the [SRE Chapter of IOG](https://input-output.atlassian.net/wiki/spaces/CI/pages/3528785931/SRE+Chapter) + +## **Rationale** + +### **Artifacts and checks** + +The two main concerns of a CI pipeline are: building **artifacts** and running **checks**. + +The purpose of building an artifact is to produce, say, an executable or HTML. The purpose of running a check is to check that the artifact satisfies certain properties, e.g. all unit tests pass. + +Different CI systems, like Hydra, Cicero, Buildkite or Github Actions, have a different focus regarding these concerns. + +* The world view of Hydra is that everything is about building artifacts. Hydra was surprisingly successful as a CI tool, because this world view can be used for running checks, too — they can be expressed as trivial artifacts, where success of the check is equivalent to success of building \`()\`, and failure of the check is equivalent to failure of the artifact build. +* The world view of Github Actions, Buildkite or Cicero is that everything is about running checks. The drawback is that building artifacts is more difficult and we have problems managing the build cache. + +For us, the main takeaway is that we should try to separate these concerns clearly. + +Our **artifacts** include: **source code** and **compiled** **executables**. We have different **checks** on these: Linters and style checkers on the source code, unit and integration tests on the executables. + +As we are coming from Hydra, compiling executables is easiest to do through a **cached nix store**. At the moment, it looks like only Buildkite has good support for that; hence we choose Buildkite. + +### **Our options for CI system** + +Buildkite + +* Pro — Good at artifacts, working nix cache +* Pro — Good documentation, easy to write +* Con — Dependency on machine (currently provided by SRE / [Samuel Leathers](mailto:samuel.leathers@iohk.io)) +* Con — no Windows machine +* Con — Dependency on permissions (currently only SRE / [Samuel Leathers](mailto:samuel.leathers@iohk.io) has write permission) + +In a pinch, the dependencies can be solved by forking the repository and providing our own machines. + +Github Action + +* Neutral — Good at small actions, but problems at scale +* Neutral — Good documentation, but a bit cumbersome to write +* Pro — No dependency on machine +* Pro — Windows machine +* Pro — No dependency on permissions + +Cicero + +* Con — Poor at artifacts, nix cache currently not working properly +* Con — Poor documentation +* Neutral — Dependency on machine (provided by SRE, but they have long-time commitment) +* Con — no Windows machine +* Pro — No dependency on permission + +## **References** + +\[1\] G Kim, K Behr, G Spafford; [The Phoenix Project](https://www.goodreads.com/book/show/17255186-the-phoenix-project); IT Revolution Press (2013). A business novel about the DevOps movement: make the flow of work visible and automate it, to an extreme of, say, 30 releases per day. + +\[2\] [Cicero on Github](https://github.com/input-output-hk/cicero#readme) + +# **Scratchbook** + +## **Random Findings** + +Installing Nix with the \`cachix/install-nix-action\` Github Action: [https://github.com/input-output-hk/cardano-node/blob/db396b163af615aa89286aa985583ef8843cfcde/.github/workflows/check-mainnet-config.yml\#L16-L23](https://github.com/input-output-hk/cardano-node/blob/db396b163af615aa89286aa985583ef8843cfcde/.github/workflows/check-mainnet-config.yml#L16-L23) + +## **Documentation Findings** + +### **Cicero** + +[Cicero](https://github.com/input-output-hk/cicero#readme) \= An *engine* for executing actions. An “action” is an arbitrary program (Bash, Python, Nix, …) that is run in the Nomad execution environment. + +[Tullia](https://github.com/input-output-hk/tullia#readme) \= A domain specific language, embedded in the Nix language, for expressing actions to be run with Cicero. This is useful when writing Cicero actions that mainly build stuff with Nix. diff --git a/docs/site/src/contributor/decisions/2023-07-28-workflow-review.md b/docs/site/src/contributor/decisions/2023-07-28-workflow-review.md new file mode 100644 index 00000000000..8cc3983ce8a --- /dev/null +++ b/docs/site/src/contributor/decisions/2023-07-28-workflow-review.md @@ -0,0 +1,34 @@ +# **Workflow Review ** + +| | | +|---------|------------| +| Started | 2023-07-07 | +| Decided | 2023-07-28 | + + +## **Why** + +As we have now moved to the Cardano Foundation, it’s an appropriate time to review how we operate as a team and see what improvements we can implement. + +## **Decision** + +1. Due to the distributed nature of the team, where possible and practical we will continue to favor asynchronous working. +2. We will adopt a Kanban-esque approach to managing our workflow, therefore: + 1. Work items will be continuously prioritized in line with agreed goals and roadmap. + 2. Weekly planning & replenishment meetings will be held. + 3. We will no longer plan work into sprints or iterations. + 4. Updated meetings and ceremonies: + +| Meeting / Activity | Frequency | How | +|:------------------------------|:----------------------|:----------------------------------------------------------------------------| +| Daily Updates | Daily / as applicable | Provided in slack \#hal-daily | +| Planning & Replenishment | Weekly | Friday meeting (½ hr) | +| Development Meeting | Weekly | Wednesday (1hr) | +| Backlog Refinement & Grooming | Continuously | Asynchronously, supported by breakout sessions and ad hoc team meetings. | +| Demos | Ad hoc | Scheduled upon feature completion. Will demo as part of a feature showcase. | +| Retro | Quarterly / ad hoc | Deep Dive meeting (3 hrs) | + +3. To determine how best to implement an internal team “No Meeting Day”, this will be progressed via its own advice process. +4. Public team channel (\#hal-public) created to facilitate communications and collaboration within the Cardano Foundation and IOG. +5. Current ticketing system JIRA will continue to be used, subject to future Advice Process. +6. Cardano Wallet [public forum on Github](https://github.com/cardano-foundation/cardano-wallet/discussions) to be utilized for engaging with the community and for also announcing new releases. diff --git a/docs/site/src/contributor/decisions/2024-03-13-release-process.md b/docs/site/src/contributor/decisions/2024-03-13-release-process.md new file mode 100644 index 00000000000..1acff6730e3 --- /dev/null +++ b/docs/site/src/contributor/decisions/2024-03-13-release-process.md @@ -0,0 +1,24 @@ +# Update Release Process + +| | | +|---------|------------| +| Started | 2023-07-19 | +| Decided | 2024-03-23 | + +## Why + +The release v2023-07-18 of cardano-wallet highlighted the need to further automate the release process, and to clarify consistency between artifacts and tests. The release v2023-12-18 highlighted further gaps in test and dependency maintenance. + +## Decision + +* We continue to use **trunk-based development**, where all code is merged into the master branch frequently, such that this branch is (almost) ready to be released at any point in time. +* A **release** is a collection of artifacts that have been tested together and will be published. + * We **create** a release from a specific Git commit, marked by a **Git tag**. + * We create and **test artifacts** without human involvement using **automation**. + * We publish human-readable release notes, specifically a **changelog** and a list of **known issues**. + * We **publish** a release by clicking a button on [Github Releases](https://docs.github.com/en/repositories/releasing-projects-on-github). Artifacts are automatically pushed to other platforms. +* The release is made under **human supervision** using a **release checklist**. + +## Details + +see [Release process](../how/release-process.md) diff --git a/docs/site/src/contributor/decisions/2024-12-03-document-with-code.md b/docs/site/src/contributor/decisions/2024-12-03-document-with-code.md new file mode 100644 index 00000000000..70ed9aab54e --- /dev/null +++ b/docs/site/src/contributor/decisions/2024-12-03-document-with-code.md @@ -0,0 +1,48 @@ +# **Store Documents alongside Code** + +| | | +|---------|-------------| +| Started | 13 Nov 2024 | +| Decided | 3 Dec 2024 | + +## **Why** + +When developing open-source projects, it helps if all the core “artifacts” related to the project are easily accessible along with the code: + +- Links between various artifacts, between code and documents, or documents and code, are easier to add, verify, and maintain over time +- Versioning everything in the same way increases the likelihood of maintaining consistency over time +- It reduces “friction” as one only needs to check-out the project and look in some folder to find the needed information +- Documents can be used as build resources to produce e.g. websites or PDFs, and conversely up-to-date code can be included in documents if needed +- Wannabe contributors have all the needed context to contribute more effectively + +## **Decision** + +Store all documents related to the team's *product* (roadmap, design documents, specifications) and *process* (Advice Process decisions, Logbook, Meeting minutes and recordings, etc.) to the relevant repository (ie. usually https://github.com/cardano-foundation/cardano-wallet). + +## **Details** + +* This Advice Process supersedes [Document Storage](2022-10-04-document-storage.md) +* The documents shall be moved manually and converted to markdown as needed, i.e. we won't waste time migrating everything all at once and might keep some links to Google docs active for some time +* New documents shall be created in the relevant repository +* Folder structure is not fixed in this advice, and will be defined and refined as needed. Tentative structure could be: + * docs/ : toplevel documentation directory + * docs/process : advice process directory + * docs/process/README.md: contains the Advice Process logbook and summary + * docs/specifications: specification documents +* The team maintains several code repositories but for the time being, we still consider cardano-wallet to be our main focus, therefore more general documents related to process or products should go there. +* Private documents will be kept in Google Docs +* Public documents that require some collaborative writing or editing process might live for a while in Google Docs or [https://hackmd.io](https://hackmd.io). The latter should be preferred as it ensures we stick to markdown format. + +## **Rationale** + +The “Why” section already states a few benefits of keeping documents in the same place as the code the team is responsible for. The main rationale behind this proposal is to increase the visibility of the open source Cardano community on the work the HAL team is doing, in order to maximise the likelihood this work will be deemed relevant and important, increase or lower the barrier of entry for potential contribution, and more generally increase the level of transparency of our work. + +The original Advice Process’ rationale stated some requirements for documents storage which were valid and this proposal tries to stick with those: + +- **Sharing** and **Access Control**: There are very few documents we need to keep private and by default, everything related to the product and process should be public, so no need for specific access control policies +- **Readability**: GH is able to render sophisticated document, and if needed it's possible to enrich raw markdown to produce web sites or PDFs +- **Commenting** and **Tracking Changes**: it's possible and easy to comment through change proposals. While it's not possible to comment directly on rendered document, I don't think this is a showstopper as the documents we manage are not so sophisticated the rendering is completely decorrelated from the source +- **Writing format**: markdown is very versatile and easy to work with. Should we need a WYSIWYG collaborative editor, it's always possible to work in a [https://hackmd.io](https://hackmd.io) document, even in private, and commit the result +- **Navigation** + - **Search**: repository is indexed and easily searchable through GH interface or directly on the filesystem + - **Discover**: ditto diff --git a/docs/site/src/contributor/how/continuous-integration.md b/docs/site/src/contributor/how/continuous-integration.md index ffa337c8ad4..3358d567142 100644 --- a/docs/site/src/contributor/how/continuous-integration.md +++ b/docs/site/src/contributor/how/continuous-integration.md @@ -1,3 +1,3 @@ # Continuous Integration -TODO: Information about our continuous integration pipeline. +See [Buildkite workflow descriptors](https://github.com/cardano-foundation/cardano-wallet/blob/master/.buildkite/README.md#L1) for details about the CI process currently in use. diff --git a/docs/site/src/contributor/how/release-checklist.md b/docs/site/src/contributor/how/release-checklist.md new file mode 100644 index 00000000000..5266405d17c --- /dev/null +++ b/docs/site/src/contributor/how/release-checklist.md @@ -0,0 +1,48 @@ +> This is the latest release checklist template that one needs to go through in order to check quality of the release + +## Prepare the release process + +- [ ] Made a copy of the latest release + [Release-vYYYY-MM-DD]() + documented as a new page on cardano-wallet's wiki called [Release-vYYYY-MM-DD']() + and substituted the version number in the title and the links. +- [ ] Picked up the latest green commgit from the release pipeline and prove it's green by linking the successful buildkite and github actions builds here. + - [ ] [release pipeline build](https://buildkite.com/cardano-foundation/cardano-wallet-release/) + - [ ] check for performance regressions on the `release commit` artifacts: + - [ ] check that the [restoration benchmarks](https://buildkite.com/cardano-foundation/cardano-wallet-restoration-benchmarks) ran +- [ ] Unblock the release blocked on [release pipeline build](https://buildkite.com/cardano-foundation/cardano-wallet-release/) + +## Update the release page + +- [ ] Fix the release note on the draft release automatically created by the release pipeline + https://github.com/cardano-foundation/cardano-wallet/releases + +## Check out the docker image is on the docker hub + +- [ ] link: https://hub.docker.com/r/cardanofoundation/cardano-wallet/tags +- [ ] Report the link in the release notes + [link to docker hub](https://hub.docker.com/layers/cardanofoundation/cardano-wallet/) + +## Check sensitive fields in the API + +- [ ] Verify that sensitive fields listed in + [Cardano/Wallet/Api/Server](https://github.com/cardano-foundation/cardano-wallet/blob/89faf170f388f9b475974896c349ad7676f0f44c/lib/exe/lib/Cardano/Wallet/Application/Server.hs#L128) + are still accurate and aren't missing any new ones. + ``` + sensitive = + [ "passphrase" + , "old_passphrase" + , "new_passphrase" + , "mnemonic_sentence" + , "mnemonic_second_factor" + ] + ``` +- [ ] Check out the documentation has been published by the [github action](https://github.com/cardano-foundation/cardano-wallet/actions/) + +## Publication + +- [ ] Once two engineers have signed off, publish the release draft. + +- [ ] Merge the (administrative) commits created for the release tag back into the `master` branch. + +- [ ] Remember to leave this checklist in an up-to-date status for the next releaser diff --git a/docs/site/src/contributor/how/release-process.md b/docs/site/src/contributor/how/release-process.md index 5ff969cc082..071d500f5de 100644 --- a/docs/site/src/contributor/how/release-process.md +++ b/docs/site/src/contributor/how/release-process.md @@ -1,9 +1,121 @@ # Release Process +## **Trunk-based development** + +We use [Trunk-based Development](https://martinfowler.com/articles/branching-patterns.html#Trunk-basedDevelopment): We merge features and improvements into the `master` branch very frequently. If a large feature takes more than one pull request to develop, and is therefore incomplete, we hide it behind a feature flag. The `master` branch should be kept in a state where all tests pass, so that we can **release** it **at any time**. + +Even though the act of making a release frequently reveals **problems** in our **testing** and **dependency** infrastructure, solving these problems is **not in scope** for the present decision — they need their own decisions. + +## **Release candidate branches** + +Unfortunately, releasing directly from the `master` branch is not quite practical for two reasons: a) secondary CI and b) administrative commits. + +Our [Continuous Integration](../decisions/2023-01-27-continuous-integration.md) can be grouped into two categories: + +* Primary CI — gatekeeper for accepting pull requests to the `master` branch. +* Secondary CI — gatekeeper for making releases. + +**Secondary CI** denotes tests that are impractical to run before every merge into the `master` branch, such as manual tests or automated tests that take a long time to run, e.g. benchmarks. The core teaching of Continuous Integration is to minimize secondary CI as much as possible, but it is a fact of life that we still have it. + +**Administrative commits** are commits that change metadata in the repository, such as version numbers, release dates, or artifact URLs. These commits need to be checked by the secondary CI as well, but they are made just before a release, so if we were to include them in `master`, we would incur a time delay between our intention to release and the act of releasing where we wait for secondary CI to finish. + +In order to accommodate these two facts of life, we use **automated release candidate branches**. The branches relevant for the release process are: + +* `master` branch — development can progress even while preparing a release. +* `release-candidate/vYYYY-MM-DD` branch — at the granularity of the secondary CI, one release candidate branch is created automatically, i.e. daily. This branch forks from a commit on `master`, and adds a single administrative commit, i.e. the version number bump. Secondary CI is run on that branch. Then, we can release at any time from the last release candidate branch where the secondary CI succeeded. (As a technical synchronization aid, the primary CI creates a Git tag `rc-latest` on the release candidate branch when it runs successfully; this tag signals to the secondary CI that it may proceed on this branch.) + +Since the release candidate branches are created frequently, we need to **garbage collect** them. For the sake of being specific, we keep the last **31** release candidate branches where secondary CI succeeded and remove the rest. + +## **Release Artifacts and their Publication** + +A **release** is a collection of artifacts that have been tested together and will be published. We use automation to create and test these artifacts automatically from the source code in the repository. + +All artifacts in a release are created from the same **Git commit**, also called the **release commit**. This Git commit is marked by a **Git tag** of the form `vVersionNumber`, e.g. `v2024-03-01`. This release commit is the head of one of the **release candidate branches** described above. + +We keep a **Changelog** which records all user-facing changes between the previous release and the current release. We use **pull requests** (PR) as the smallest unit for user-facing changes. In order to ensure that PRs are indeed the smallest unit, we require that the Git commit of `master` from which release candidate branch is forked is the merge commit of a pull request. + +The **release artifacts** and their places of **publication** are: + +| Artifact | Artifact Details | Place of Publication | +| :---- | :---- | :---- | +| Release notes (metadata) | Version of compatible cardano-node Changes to previous release [Changelog](http://keepachangelog.com) HTTP API changes Known issues | [Github Releases (link)](https://github.com/cardano-foundation/cardano-wallet/releases/latest) | +| Self-contained executables on all platforms | Compressed archive containing cardano-wallet cardano-node shared library dependencies (static linking preferred) Platforms Linux, macOS, Windows Testing includes E2E tests with all components | [Github Releases (link)](https://github.com/cardano-foundation/cardano-wallet/releases/latest) | +| Documentation | Command-line interface help OpenAPI specification of HTTP API | [Github Pages (link)](https://cardano-foundation.github.io/cardano-wallet/) | +| Docker image | | [Docker Hub (link)](https://hub.docker.com/r/cardanofoundation/cardano-wallet) | +| Haskell packages | | [CHaP](https://github.com/input-output-hk/cardano-haskell-packages) [Hackage](https://hackage.haskell.org/) | + +We use [Github Releases](https://github.com/cardano-foundation/cardano-wallet/releases/latest) as the **source of truth** for publication: A release is considered to be published once it has been published there. + +## **Changelog** + +We publish a **changelog** as part of the release notes. + +We adhere to the format and rationale described at [https://keepachangelog.com/en/1.1.0/](https://keepachangelog.com/en/1.1.0/) . Importantly, a changelog focuses on being **human-readable**. + +The changelog is distilled from a sequence of pull requests as follows: + +* Keep only those pull requests which represent **user-facing** changes. + * A user-facing change is one where the user may have to adapt. This is typically the case for changes that + * change the API + * or accommodate a changed API of a run-time dependency +* **Group** the pull requests numbers by their user-facing changes. + * Sometimes, the list of PRs associated with a user-facing change can be large. In such cases, mention only the PRs that we would need to revert in order to hide the change.. For example, when an experimental feature is built behind a feature flag, the PR that merges the flag into the core functionality is to be mentioned here; whereas pull requests that do the implementation work need not be mentioned. +* Amend the **description** of the change to give more context. + +## **Release checklist** + +We use a **release checklist** in order to ensure that we do not miss creating, testing, or publishing any artifact. The focus of the checklist is for a human to **verify** that nothing has been missed — creating and testing the artifacts should be automated as much as possible by [Continuous Integration](../decisions/2023-01-27-continuous-integration.md). + +(The work required to perform the release checklist, resulting in the publication of the release artifacts, is tracked in a ticketing system decided in our [HAL Workflow Review](../decisions/2023-07-28-workflow-review.md).) + +The release checklist is subject to updates as the environment changes. Here, we only specify the **general outline** of the release checklist: + +1. Choose a Git commit from which we want to release. + 1. This commit must be the head of a release candidate branch (Trunk-based development). + 2. All packages in this commit must have the version numbers to be released. + 3. Secondary CI must have succeeded on this commit, including artifact creation. +2. Mark the commit with a Git Tag specifying the version number, i.e. `vYYYY-MM-DD`. +3. Write the release notes, specifically the human-readable Changelog section. +4. When the commit is ready to be released, press the “Publish release” button in [Github Releases](https://docs.github.com/en/repositories/releasing-projects-on-github) . + 1. The commit is ready to be released if all of the following are satisfied: + 1. All automated tests pass (both primary and secondary CI). + 2. A human has inspected the test logs and feels confident that the automated tests have run correctly, i.e. the “pass” mark is not the result of a bug in the test pipeline. + 3. Release notes have been written. + 2. We use automation to publish the release artifacts on other platforms, such as [Docker Hub](https://hub.docker.com) or [Read the Docs](https://readthedocs.org) as desired. + +The release checklist is kept in the [project's wiki](https://github.com/cardano-foundation/cardano-wallet/wiki). To create checklist for a new version: +1. Copy the latest version, eg. [v2025-01-09](https://github.com/cardano-foundation/cardano-wallet/wiki/Release-v2025-01-09) as a new page named `Release-vYYYY-MM-DD.md` +2. Uncheck all the boxes and remove all the links there +3. Go through all the steps, checking box and updating links as needed + +## **Automation** + +We use the following **tools** to help automate the release process: + +| Type | Name | Task | +|:------------|:--------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Script | updateVersionNumbers `release-candidate.sh` | Updates the version numbers of all packages in the repository. Useful for preparing the administrative commit. | +| Script | draftReleaseNotes `make_changelog.sh` | Prepares draft release notes by collecting a list of merged pull requests and known issues. To support automation, Github labels can be utilized. To support human-readability, PRs should be connected to JIRA tickets. | +| CI pipeline | | Pipeline that automatically creates release artifacts and runs tests on them, used for the release candidate branches. It’s important that the created executables are subject to test, not a surrogate. | + +# **Rationale** + +## **Trunk-based development** + +Experience, both by us ([Continuous Integration](../decisions/2023-01-27-continuous-integration.md)) and by others _Phoenix2013_[^2], _Google2020_[^3], indicates that trunk-based development is the way to go — the ability to release at any time ensures the ability to release in the first place. Even though problems with testing and dependency management frequently resurface, they do not grow so bad that they become intractable. + +## **Bundling dependent executables** + +We bundle both the `cardano-wallet` executable, but also dependent executables such as `cardano-node` or `carano-address` in our compressed archive that is released. I believe that this is worth doing for several reasons: + +* Bundling specifies precisely which executables have been tested together — not just by the version number, but down to the **precise executable file**. Even though a version number might suffice, this prevents obscure, platform specific bugs or missing shared libraries. Through bundling, the user can rely on the fact that at least this specific combination has been tested to work. That said, bundling a **hash** of the executable file as opposed to the file would satisfy the same requirement. +* Bundling makes it convenient to test the integration of the different executables, by running them from the archive. We have to test their integration on **each platform**, as the communication between executables can be highly platform-specific (IPC, named sockets, file paths on Windows…) and subject to obscure failures, such as delayed file deletion on Windows. Again, a hash would suffice. +* Bundling is convenient for users, including Daedalus, as they get ready-to-use executables with a single download. Users do not have to collect dependencies separately (download from other repositories), or build them themselves (takes time, may fail). + - Create new page on cardano-wallet's [wiki](https://github.com/cardano-foundation/cardano-wallet/wiki/_new) called `Release vYYYY-MM-DD` by copying contents of the previous releas. - Follow the `Release checklist`. Update progress or report obstacles on the thread. -## Releasing packages to CHaP +# Releasing packages to CHaP [CHaP](https://github.com/IntersectMBO/cardano-haskell-packages) or the _Cardano Haskell Packages_ repository is useful to publish _Haskell_ packages relevant to the Cardano ecosystem as this allows standard build tools like `Cabal` to work out-of-the-box with only some configuration to point to the repository in addition to https://hackage.haskell.org @@ -16,3 +128,8 @@ To publish packages on CHaP: $ scripts/add-from-github.sh https://github.com/cardano-foundation/cardano-wallet 9eb5f59c328163ca061a20f47519686b6f118d74 lib/coin-selection lib/primitive lib/test-utils lib/delta-types lib/crypto-primitives lib/launcher lib/numeric lib/text-class ``` 3. Open a Pull Request on upstream repository (ie. https://github.com/IntersectMBO/cardano-haskell-packages/) + + +[^3]: T. Winters, T. Manshreck, and H. Wright, “[Software Engineering at Google: Lessons Learned from Programming over Time](https://abseil.io/resources/swe-book)”, O'Reilly (2020). + +[^2]: G Kim, K Behr, G Spafford; [The Phoenix Project](https://www.goodreads.com/book/show/17255186-the-phoenix-project); IT Revolution Press (2013).