From a519e96c03dc9f8aecfad390ce57cfd14a2582d9 Mon Sep 17 00:00:00 2001 From: Julian Geiger Date: Fri, 13 Dec 2024 12:00:35 +0100 Subject: [PATCH] Most plugins there --- docs/news/posts/2024-12-13.md | 115 +++++++++++++++++++++++++++++++--- 1 file changed, 108 insertions(+), 7 deletions(-) diff --git a/docs/news/posts/2024-12-13.md b/docs/news/posts/2024-12-13.md index 9fc1db6..0317440 100644 --- a/docs/news/posts/2024-12-13.md +++ b/docs/news/posts/2024-12-13.md @@ -12,11 +12,39 @@ Dear users, we're happy to announce that we have recently crossed the 100-plugin the [AiiDA plugin registry](https://aiidateam.github.io/aiida-registry/). Among those are classics like [`aiida-quantumespresso`](https://github.com/aiidateam/aiida-quantumespresso/) and [`aiida-vasp`](https://github.com/aiidateam/aiida-vasp/), to drive the electronic structure codes we all know and love. -However, did you know that there are various non research-domain specific plugins that core developers have created +However, did you know that there are various non research-domain specific plugins that AiiDA developers have created over the years, which can extend AiiDA's core functionality? In today's post, we would therefore like to give you a short introduction to each of these plugins, and show you -how and when you can incorporate them into your work, to make sure you'll get the most out of your AiiDA journey![^1] +when you can incorporate them into your work, to make sure you'll get the most out of your AiiDA journey![^1] + +## Simplifying code execution and workflow creation + +### [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph/) + +We've been making quite a bit of fuzz about this one in recent weeks already, as we're targeting a first, stable release +for early next year (fingers crossed :crossed_fingers:). The `aiida-workgraph` provides a new, alternative way of construction workflows, +in addition to the classical `WorkChain`. +It comes with all the typical AiiDA features, like provenance tracking, +checkpointing, caching, and remote execution, but provides a simpler syntax to construct workflows. This can be achieved +by adding nodes (or `Tasks`, as we call them) and links, in equivalence to how one would typically build-up a graph. In +this way, `aiida-workgraph` makes it +simpler to quickly chain existing AiiDA components (`CalcJobs`, `WorkChains`, or other `WorkGraphs`) together, and even allows +visualizing the flow of data via the different processes before the workflow is executed (through an interactive web GUI). +As we're currently stabilizing the API, any feedback is valuable and helps us to polish the tool as best as we can until +the release! +You can read more about it on its [RTD page](https://aiida-workgraph.readthedocs.io/en/latest/), and find example `WorkGraphs` in [this repository](https://github.com/superstar54/workgraph-collections). + +### [`aiida-shell`](https://github.com/sphuber/aiida-shell/) + +Probably we don't have to talk too much about this one, as we covered it in depth in one of our recent +[blog posts](https://aiida.net/news/posts/2024-11-01-aiida-shell.html). +In a nutshell, `aiida-shell` allows you to run any executable (e.g., a shell script, python script, compiled binary, +etc.) without the need for writing an entire plugin interface for it. Just plug it into the `launch_shell_job` function, +and +you're ready to rumble! If you eventually _do_ want to have a more sophisticated interface for your code or script, and +you can't find one on the [AiiDA plugin registry](https://aiidateam.github.io/aiida-registry/), +[this later section](#aiida-plugin-cutter) of the post might be interesting for you ## Enhancing high-throughput @@ -28,17 +56,90 @@ about it [here](https://nccr-marvel.ch/highlights/AiiDA-hero-run-Alps)) can be o ### [`aiida-hyperqueue`](https://github.com/aiidateam/aiida-hyperqueue/) If you've ever run simulations on a high-performance computing (HPC) infrastructure, you're likely familiar with -scheduling systems, such as - -## Simplifying code execution and workflow creation +scheduling systems such as SLURM, SGE and PBSPro (for all of which AiiDA provides an interface, btw :wink:). These are +all mature projects that work well, but now imagine you find yourself in the situation (maybe you've already found +yourself in it) that you have to run many, say, a couple thousand, small jobs. Utilizing one node for each job, these +might finish quickly, but you have to queue for every single one of them. +What the +[HyperQueue scheduler](https://github.com/It4innovations/hyperqueue) allows you to do instead, is to submit a whole +array of such jobs (say, 128 if you're working with a 128-core node), where each job runs on sub-node resources (1 CPU +core per job in the given example). While, the jobs then take a bit longer (maybe a couple hours rather than a couple +minutes), if they are small enough, that is probably still acceptable. However, as the whole array is submitted as one job +through the scheduling system provided by the HPC, you only have to queue once, rather than 128 times! +So if you ever need to run a lot of shorter simulations, make sure to give it a shot![^2] ## AiiDA setup +### [`aiida-project`](https://github.com/aiidateam/aiida-project/) + +As we outlined in +[another one of our recent blog posts](https://aiida.net/news/posts/2024-09-20-simpler-installation.html), since v2.0, +we have significantly simplified the creation of an AiiDA profile. Still, if you find yourself juggling many different +profiles on your machine, each with its own configuration, Python environment, and directory of setup files (e.g., for +`Computer` and `Code` instances), it can become easy to mess things up.[^3] +To help out with this, `aiida-project` encapsulates and automates the steps to create and switch between different +projects using AiiDA, basically functioning as an _AiiDA project manager_. It creates the necessary directories and Python environments for you +(via `pip` or `conda`) in standardized locations, and enables easy switching between projects. +Installation is simple via `pipx`, and +`uv` support is [coming soon](https://github.com/aiidateam/aiida-project/pull/28)! + +### [`aiida-code-registry`](https://github.com/aiidateam/aiida-code-registry/) + +In the previous paragraph, we mentioned storing setup files for `Computer` and `Code` instances in specific directories. +Let's elaborate on this further: +If you have interactively created such entities via `verdi computer setup` and `verdi code create` by filling +in the prompts you are being asked on the command line (which is the route we usually mention in the tutorials), it might be worthwhile to +export their configurations from the AiiDA database into YAML files.[^4] +From these files, new instances can then be created by passing them to the above two commands via the `--config` flag. +This allows you to re-use the configurations, make and track changes, and follow a Configuration-as-Code (CaC) approach. +To facilitate this process, we maintain the `aiida-code-registry`[^5] as a collection of such configuration files, which +can thus serve as a great way to share existing and start setting up new resources! If you don't find the national HPC +your are using in the repository, and your configuration is somewhat transferable, feel free to make a PR — you +might help out others! + ## Miscellaneous +### [`aiida-plugin-cutter`](https://github.com/aiidateam/aiida-plugin-cutter/) +abc + +### [`aiida-pythonjob`](https://github.com/aiidateam/aiida-pythonjob/) + +### [`aiida-firecrest`](https://github.com/aiidateam/aiida-firecrest/) + +Initially part of th + +*** + +As most of the plugins discussed in this post have been in use by many developers over the years, we now feel +comfortable to promote them to a larger audience. Therefore, we added an +[overview of the plugins](https://aiida.readthedocs.io/projects/aiida-core/en/latest/reference/core_plugins.html) to the +RTD page of AiiDA, which is also linked to from a new panel of the +[landing page](https://aiida.readthedocs.io/projects/aiida-core/en/latest/). + +*** + + [^1]: Please note that, while core developers have been using these plugins for a while to enhance their own work with AiiDA, many of them are still in a development, pre-stable release phase. Therefore, use them at your own risk, and - please do provide us with your feedback so we can improve and stabilize these tools. As always, issues and PRs very - welcome! + please do provide us with your feedback so we can stabilize and improve them. As always, issues and PRs more than welcome! + +[^2]: + Bonus points because the tool is written in Rust! :crab: + +[^3]: + Forgetting to set the `$AIIDA_PATH` environment variable when switching profiles — we've all been there. + +[^4]: + Which can be easily achieved via `verdi computer export ` and + `verdi code export `. + +[^5]: + As written in the `README` of the + [`aiida-code-registry`](https://github.com/aiidateam/aiida-code-registry), + migration is eventually planned to the [`aiida-resource-registry`](https://github.com/aiidateam/aiida-resource-registry), which contains configuration + files for [AiiDAlab](https://github.com/aiidalab/). Currently, the migration is still hindered, as we use `jinja2` + fields for AiiDAlab, which are not yet supported for the creation of `Computer` and `Code` instances in `aiida-core` + (see open issue [here](https://github.com/aiidateam/aiida-core/issues/4680)). For now, it is good to keep an eye + open on both repositories, but we are planning to fix this soon!