New NBFT initramfs module #2620

tbzatek · 2024-12-13T13:54:56Z

Cc: @mwilck
-- Please check the included README.md first

This started as an experiment from an idea coming from our inhouse dracut and systemd developers. I kinda like the result even though there are many loose ends - read on. There's an unpublished internal plan to slim dracut down in the mid-term and while I can't speak of the details one major argument was something like "dracut was born before systemd" and "systemd nowadays can take over many tasks dracut is doing". Add embedded, automotive and otherwise resource-restricted environments and the push for minimization grows further.

This experiment was about creating specially crafted systemd units injected at the right place with as little dracut involvement as possible. I admit all this work may speak itself a little of a NIH syndrome, still I hope it becomes useful.

There are some loose ends here:

This is all based around NetworkManager with the hope to spark an interest of other distributors adding support for more network management frameworks.
Need to find a way how to play nice with the existing 95nvmf dracut module. Haven't approached dracut upstream yet.
Interface renaming is the thing. Originally planned to be part of the NetworkManager NBFT patch but NM folks wanted to keep their initramfs clean and minimal (for a good reason). So this landed as an extra nbft plugin command, at least for now. We can move this somewhere else as the only entry point is the systemd service here. Also the commandline arguments and help strings may need some tweaks. It was strongly suggested to use systemd link files instead of udev rules for interface renaming.
I have proposed slightly different nbft interface naming, geared towards predictable naming scheme.
Nothing is set in stone yet - the design, file and function naming can change.
As NetworkManager is a new consumer of the libnvme nbft API, we need to be more careful with future API additions and changes.

Needs #2614 (already merged).
This work also gives answer for #2179 and includes the files we've kept downstream so far.

And some more wild ideas:

There is a growing interest to split the libnvme NBFT parser in a separate, independent library (just like the -mi one) that would be small and suitable for initramfs, or statically linked. One suggestion was to have native udev support for nbft interface naming within builtin-net_id.c however the parser dependencies were a problem. The ACPI NBFT table is not a fixed-size struct and is not so trivial to parse if somebody wanted to reimplement the parser just to extract the HFI descriptors.

mwilck · 2024-12-13T16:51:58Z

Very interesting. The timing is kind of unfortunate for me. I'll not be able to do an in-depth review until the beginning of next year.

Some quick questions / remarks. I've only glanced about this quickly, so I hope I'm not getting it all wrong.

Why do you submit this here rather than for dracut itself?
Am I assuming correctly that this module can't coexist with the existing dracut 95nvmf module?
Given that this is nvme-cli upstream, I find it questionable to include code that's solely focused on NetworkManager. The current dracut module is agnostic of the network management tool. I would prefer an implementation-agnostic approach here in nvme-cli, too. Simply putting the responsibility for that on "other distributions" isn't enough. Perhaps you could think of some sort of API that can be served by the nm nbft plugin, but alternatively by other plugins, too. Note the "wicked" already has an nbft extension, which is a simple shell script that prints interface configuration in an XML format that wicked understands.
I'm not too happy about the new interface naming scheme, given that we already introduced nbft$N. I'm not sure how important it is to have predictability. As you said yourself in the README, NBFT variables can change between reboots, by design. What matters is that the NVMe targets are found, and that it's obvious that the related interfaces have NBFT (aka firmware) origin, not necessarily which HFI they correspond to. There should at least be an option to fall back to the previous convention.
About "interface renaming is the thing" – was anything wrong with the way this was done in the existing dracut module?

mwilck · 2024-12-13T16:59:29Z

In my mind, the current "API" of nvme-cli wrt NBFT is the JSON format. I am aware that that's subobtimal if you want to build a minimal initramfs, as it requires either a JSON library or something like jq, which is rather big. Still, discuss carefully if we want replace this by something else.

igaw · 2024-12-16T07:56:23Z

One thing I'd like to bring up, is that I think we should reconsider the command-all command. The implementation is very complex, hard to maintain and easy to break. And I have the impression the use case which it suppose to cover are contradicting themself.

Almost all other commands by nvme-cli are just simple commands which do one thing. I think it should be possible build any complex logic ontop of it, e.g. using scripts.

My goal would be to move the missing parts of discover and the connect logic into the library, and
nvme-cli does just the command line parsing etc.

This is obviously all v3 material.

tbzatek · 2025-01-07T17:03:25Z

Thanks for your first comments, Martin. There's a little more background for this work. I'll try to explain but take it with a grain
of salt as nothing has been decided officially yet - it's just a plan that's being discussed widely inhouse.

There's a desire (Fedora, RHEL) to gradually slim dracut down mid-term or even phase it out in the long-term. There's an ambition to use projects like mkosi to minimize differences between initramfs and OS/rootfs environment.

This pull request is a first step towards this goal - minimizing dracut involvement in the boot process. Dracut would stay just as a tool to build the initramfs image, copying required files in place and not much more. Then it would be mostly a matter of changing systemd units and targets dependencies when porting to a different initramfs framework. This should answer some of your questions about "why not dracut".

Related to that, there's a tendency to move the particular initramfs functionality towards their native packages and transfer maintenance responsibility along with it. Updating a specific dracut module in a distro and the associated QA cost is often much higher than keeping the module at the edge, i.e. together with the required binaries and libraries.

Perhaps this work should be called an initramfs module instead.

mwilck · 2025-01-16T16:48:37Z

There's a desire (Fedora, RHEL) to gradually slim dracut down mid-term

Understood. Similar here.

Even now, dracut has no monopoly on building initramfs images. mkosi is relatively new on the stage, but Debian's update-initramfs has existed for years. We've actually discussed about it in the Timberland group, but there didn't seem to be anyone from the Debian/Ubuntu realm interested in NVMeoF-boot. That's why only dracut is currently supported.

Dracut would stay just as a tool to build the initramfs image

Hm, that's exactly what it is today, no?

there's a tendency to move the particular initramfs functionality towards their native packages

That makes some sense.

The nvme-cli maintainers will need to decide whether they want to take the responsibility of maintaining the dracut module. It's tricky because the module interfaces intimately with both nvme-cli and dracut.

But I should look at the code first...

mwilck · 2025-01-16T16:56:04Z

nvmf-autoconnect/NetworkManager/99-nvme-nbft-no-ignore-carrier.conf.in

+# /usr/lib/NetworkManager/dispatcher.d/99-nvme-nbft-connect.sh hook
+# will respawn nvmf-connect-nbft.service on such occasion.
+
+[device-nbft-no-ignore-carrier]


I'm confused about this config section. Will you implement support for this section, and the directives it contains, in NetworkManager? If not, how is this supposed to work?

No need to implement anything, this already works. Added a comment clarifying this override.

mwilck

The code quality is eccelent, AFAICS. I also like the design. For me, the open question is mostly which part of this code actually belongs into nvme-cli.

I'd prefer if we could separate out the NetworkManager-specific part more. So that we'd have obvious ways to extend this to other network management tools like wicked.

I believe that libnvme/nvme-cli shouldn't come with a preference for one specific network management tool, even if NM is the "de facto standard" these days.

Likewise, it would be nice not to depend on dracut directly and to just include the dracut code as a PoC, showing how the systemd services need to be complemented in the initramfs. It would be even nicer if we had a mkosi PoC as well.

Wrt dracut specifically, it's up to the nmve-cli / libnvme maintainers to tell whether they are willing to maintain a dracut module.

nvmf-autoconnect/dracut-95nbft/README.md

mwilck · 2025-01-16T17:33:03Z

nvmf-autoconnect/dracut-95nbft/README.md

+  dracut to activate networking
+* dracut runs `nm-initrd-generator` and starts the NetworkManager daemon
+* `systemd-udev-trigger.service` renames the network interfaces
+* `nm-wait-online-initrd.service` finishes, indicating networking is up and ready


See my remark below about nbft-boot-connect.service. I am somewhat confused about when exactly network-online.target is considered to be reached.

nvmf-autoconnect/dracut-95nbft/README.md

mwilck · 2025-01-16T17:50:26Z

nvmf-autoconnect/dracut-95nbft/README.md

+## nm-initrd-generator NBFT support
+
+https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/2077
+


This seems to be the most important difference wrt 95nvmf. You create a tool that directly transforms libnvme NBFT data structures into the preferrred format of your Network configuration tool (NM), whereas 95nvmf converts NBFT to JSON first, then JSON to dracut-style command line arguments, which would then further processed by the conventional nm-initrd-generator.

Your approach is of course much more efficient, but at the cost of being compatible only with (a future version of) NM.

A similar approach could be taken by wicked or other network management tools.

But I wonder if there might be some middle ground, perhaps we can provide the HFI data in some format that any network management tool can easily convert?

The "dracut command line" format is obviously very clumsy and simplistic.
I'd love to see dracut develop a more capable generic format for describing networking parameters. But I doubt that's going to happen. After all, dracut's format was designed with the kernel command line in mind.

So we could keep 95nvmf for those network management tools that have no native NBFT support, and take your approach for others. I'd need to discuss with the wicked people if it makes sense for us to write a similar plugin.

The original intention was to keep nm-initrd-generator stateless, needing no input or prior configuration as it would parse the NBFT table itself. Then the whole dracut network stack can be eventually replaced with a simple systemd unit file, executing ExecStartPre=nm-initrd-generator + ExecStart=/usr/sbin/NetworkManager. This was planned as a next step later in the dracut minimization effort once other modules depending on network would take the same road. Obviously this was all originally planned as a downstream change...

But I wonder if there might be some middle ground, perhaps we can provide the HFI data in some format that any network management tool can easily convert?

This would require a new code somewhere in the middle. I don't see much benefit compared to the libnvme nbft parser, except of the required dependency on libnvme. The HFI structure itself is rather simple.

To be fair, there is currently one place where this could happen, however I was hoping to get rid of that one in the near term: interface renaming. We haven't agreed with the NetworkManager developers what component should be responsible for that. Renaming needs to be done through udev rules or through udev network link files. Udev then needs to be trigerred for the rename to actually happen, that's currently ensured through unit dependencies.

A natural place for this that was suggested to me was udev's builtin-net_id.c (see the note my the original PR post). Then everything could be even more simple.

So yes, the proposed call to nvme nbft gen-udev-link-files can be used for an intermediate NBFT data representation, however that would mean a new data format, a generator at one place and separate new parsers in each network management tools. That sounds like an overkill in this little initramfs module.

So we could keep 95nvmf for those network management tools that have no native NBFT support, and take your approach for others. I'd need to discuss with the wicked people if it makes sense for us to write a similar plugin.

Yes, it all falls down to the requirement of implementing the NBFT HFI parser elsewhere.

mwilck · 2025-01-16T17:58:44Z

nvmf-autoconnect/dracut-95nbft/README.md

+attempts are not driven by network link up events but have fixed respawn
+interval. This may potentially help the cases where the NIC is slow to
+initialize, reports link up yet it takes another 5+ seconds before it's fully
+able to send/receive packets. We've seen this issue with some 25Gb NICs.


Good point.

Ideally we wouldn't react on "link up" events but on events that indicate an L3 connection. But I'm not sure if such events exist... (see below)

Note that 95nvmf also has a timeout action. But repeating this in regular intervals is actually a nice idea.

mwilck · 2025-01-16T18:20:08Z

plugins/nbft/nbft-plugin.c

@@ -571,3 +571,108 @@ int show_nbft(int argc, char **argv, struct command *cmd, struct plugin *plugin)
 	}
 	return ret;
 }


So here you add support for one specific network configuration file format, udev's .link files. It would be possible to derive the .link files from the JSON output, but this is obviously much faster and more light-weight, especially compared to using jq for the conversion in the current code.

Yet it's not immediately obvious why we'd implement this in nvme-cli, while the conversion of HFI data to .nmconnection files is done in the Networkmanager plugin. Instead of adding this functionality to libnvme, you might as well have written a separate tool that links to libnvme and creates the .link files. Why do this here and that there?

Yes, this is one of the loose stones. See also my earlier comments. I hate to parse the NBFT table at two different places, it's a recipe for inconsistency.

My original intention was to have this in the NetworkManager plugin, but was not liked by the NM upstream. I still believe this would be a good fit to nm-initrd-generator because, it is just a initrd tool after all.

On the other hand this thing might come useful for other network management frameworks. Subject to further evaluation.

I wonder if it makes sense to have this code live in systemd-network-generator instead which already generates udev link files based on kernel command line arguments. I don't immediately see an issue with having it optionally link against libnvme in which case it can parse the NBFT ACPI tables and generates udev link files based on that information.

I wonder if it makes sense to have this code live in systemd-network-generator

I'm afraid systemd-networkd is unsupported on RHEL and has been intentionally removed in favour of NetworkManager. Good point though.

@tbzatek At least in Fedora systemd-network-generator now lives in the systemd-udevd package as it's separate from systemd-networkd and can be used independently from it to generate link files for udev.

Same in openSUSE, this is shipped in the udev package.

Ah, found it in the master systemd.rpm in RHEL 9 and RHEL 10:
https://kojihub.stream.centos.org/koji/fileinfo?rpmID=2004778&filename=/usr/lib/systemd/systemd-network-generator
https://kojihub.stream.centos.org/koji/fileinfo?rpmID=2023136&filename=/usr/lib/systemd/systemd-network-generator

Let me have a look at the sources...

I wonder if it makes sense to have this code live in systemd-network-generator instead which already generates udev link files based on kernel command line arguments. I don't immediately see an issue with having it optionally link against libnvme in which case it can parse the NBFT ACPI tables and generates udev link files based on that information.

I've checked the sources, this looks positive.

I have opened systemd/systemd#36443 to separate this discussion.

mwilck · 2025-01-16T20:07:59Z

nvmf-autoconnect/dracut-95nbft/README.md

+The boot process looks roughly as follows:
+* `nbft-boot-pre.service` is run, creates udev network link files and tells
+  dracut to activate networking
+* dracut runs `nm-initrd-generator` and starts the NetworkManager daemon


dracut runs nm-initrd-generator

I fail to see where this happens. Is it an ExecStartPre= in NetworkManager.service?

Ah, this is done by the dracut 35network-manager module. Clarified in the docs.

dracut runs nm-initrd-generator

I fail to see where this happens. Is it an ExecStartPre= in NetworkManager.service?

It's within the NM's cmdline hook, nm_generate_connections function: https://github.com/dracutdevs/dracut/blob/master/modules.d/35network-manager/nm-config.sh#L35

nvmf-autoconnect/NetworkManager/99-nvme-nbft-no-ignore-carrier.conf.in

mwilck · 2025-01-16T20:22:07Z

nvmf-autoconnect/dracut-95nbft/README.md

+the required block device appears, the wait cycle is ended and the system
+continues booting, stopping any queued `nbft-boot-connect.service` respawns
+seamlessly.
+


I see below that nbft-boot-connect.service contains After=network-online.target. Thus no NBFT target will be connected before this target is reached. Maybe I misunderstand when you consider "online" state to be reached. Is it already reached when just one iBFT interface is configured?

This is unfortunately a way too vague definition of "network online" state as various network management frameworks tend to treat such state differently. The NetworkManager-wait-online.service docs describes various conditions, there are corresponding config options for a connection (that roughly equals to an interface setup) that can be tweaked by the NM NBFT plugin.

This After=network-online.target dependency is again (not yet) set in stone, I'm curious to see how e.g. wicked works with this.

See my remarks above. IMO this can't be handled with a generic target like network-online. It makes much more sense to react on online events of specific interfaces.

Wonder if we could have something like [email protected] or [email protected]_24.target (IOW, translate network events into systemd unit state changes)...

Oh that would be lovely, such kind of fine-grained instantiated network targets.

FWIW, neither the NetworkManager's dispatcher service is available in the initramfs, it's just too heavy, while the main daemon is present.

mwilck · 2025-01-16T21:17:19Z

nvmf-autoconnect/dracut-95nbft/README.md

+* `nm-wait-online-initrd.service` finishes, indicating networking is up and ready
+* `nbft-boot-connect.service` initiates actual NVMe connections
+* the dracut initqueue is waiting for specific block devices (rootfs) to appear
+


So this, and the starting of NM above, are the only 2 things that dracut needs to perform, and therefore it's relatively easy to plug this scheme into mkosi or another initramfs generator, as long as that generator is based on systemd. Nice.

Am I understanding correctly?

Exactly. The idea is to have this (somewhat) easily replaceable with something else, like the mkosi counterparts. I.e. keeping the dracut involvement to a required minimum.

It's mostly a matter of the right systemd unit dependencies. Particular boot phases may need better abstraction. During my experiments I first tried to use generic systemd targets and units but found it unreliable and unpredictable. Thus the unit file dependencies are mostly dracut-specific, though I wish to find a better solution in the future.

Anyway, clarified the two-unit boot flow in the docs.

igaw · 2025-01-17T07:40:43Z

The nvme-cli maintainers will need to decide whether they want to take the responsibility of maintaining the dracut module. It's tricky because the module interfaces intimately with both nvme-cli and dracut.

I don't have any problems if this is added BUT I'd really want to see some sort of CI testing for this. This stuff will bit rot immediately after merging if we don't have regular upstream testing.

mwilck · 2025-01-17T13:58:08Z

I don't have any problems if this is added BUT I'd really want to see some sort of CI testing for this. This stuff will bit rot immediately after merging if we don't have regular upstream testing.

That's the point, it's hard to write meaningful tests for this. It requires simulating at least a client/server setup with appropriately crafted BIOS settings. While there are client/server tests in the dracut test suite, I'm not aware of a test for NVMeoF/NBFT.

igaw · 2025-01-17T14:18:35Z

We have a bunch of exising mock tests. Could we turn this into something more?

aafeijoo-suse · 2025-02-12T08:23:34Z

Likewise, it would be nice not to depend on dracut directly ...

Definitely. Please, try not to create new systemd services exclusively depending on dracut, the trend nowadays is removing custom logic from dracut, not adding more. These 2 units can be outside of this new dracut module, they are generic enough to be used by other initrd generators, order nbft-boot-pre.service before systemd-udevd.service, dracut can just have a dropin with the Before=dracut-cmdline.service and the rd.neednet stuff.

tbzatek · 2025-02-12T14:24:39Z

One thing I'd like to bring up, is that I think we should reconsider the command-all command. The implementation is very complex, hard to maintain and easy to break. And I have the impression the use case which it suppose to cover are contradicting themself.

Almost all other commands by nvme-cli are just simple commands which do one thing. I think it should be possible build any complex logic ontop of it, e.g. using scripts.

My goal would be to move the missing parts of discover and the connect logic into the library, and nvme-cli does just the command line parsing etc.

This is obviously all v3 material.

The NBFT connection code could use some changes too. Most of the connect-all --nbft code path is equal except of the connection part itself where the NBFT code places some restrictions and workarounds. Probably better to separate this in a new ticket.

The nvme-cli maintainers will need to decide whether they want to take the responsibility of maintaining the dracut module. It's tricky because the module interfaces intimately with both nvme-cli and dracut.

There's more to consider: this NBFT module still depends on certain dracut mechanisms, notably on block device signature scanning, setting up LVM, etc. and matching a desired filesystems to mount. In case any dracut minimization actually happens, it will likely be a gradual process. I heard rumors about slowly getting rid of dracut hooks and the initiqueue at the end. There are no clear plans how this effort look like nor I suppose it has been discussed publicly. This will present a challenge of possibly maintaining several versions of this NBFT module. Although uncertain, this is still something to consider when discussing maintenance burden and the CI.

Well, this looks almost like a separate project to maintain.

I don't have any problems if this is added BUT I'd really want to see some sort of CI testing for this. This stuff will bit rot immediately after merging if we don't have regular upstream testing.

That's the point, it's hard to write meaningful tests for this. It requires simulating at least a client/server setup with appropriately crafted BIOS settings. While there are client/server tests in the dracut test suite, I'm not aware of a test for NVMeoF/NBFT.

I think mocking a /sys/firmware/acpi/tables/NBFT file very early in the boot could be enough. Might need to write a tool to generate a NBFT table. Might also need a VM due to networking setup.

LaszloGombos · 2025-02-12T15:35:03Z

nor I suppose it has been discussed publicly

Just in case it helps with the discussion on this PR, here is on relevant upstream dracut PR - dracut-ng/dracut-ng#1194

tbzatek · 2025-02-12T16:39:09Z

Just in case it helps with the discussion on this PR, here is on relevant upstream dracut PR - dracut-ng/dracut-ng#1194

Thanks for bringing this to my attention. It roughly corresponds to what I've been hearing from dracut maintainers inhouse, nice to see it formalized and actually happening.

keszybz · 2025-02-13T09:38:54Z

nvmf-autoconnect/dracut-95nbft/README.md

+While this module is currently built around dracut, the amount of dracut
+involvement in this module is kept to a required minimum with the intention
+of supporting more initramfs frameworks (like `mkosi`) in the future.


igaw · 2025-02-19T09:29:51Z

nvmf-autoconnect/systemd/nbft-boot-pre.service.in

+
+[Service]
+Type=oneshot
+ExecStartPre=-/usr/bin/mkdir -p /etc/systemd/network


The paths shouldn't be hard coded, use @SYSCONFDIR@ & co.

Good point, fixed.

(ideally this code will be removed once agreed in systemd/systemd#36443)

Intended to create udev network link files (see systemd.link(5)) from ACPI NBFT tables present in the system, typically called during early boot process. A semi-predictable interface naming is proposed to accurately identify HFI that the interface is set up for. The syntax is as follows: nbftXhY where X is the NBFT table index (filename, defaults to 0) and Y is the HFI record index. Signed-off-by: Tomas Bzatek <[email protected]>

This is a minimal dracut module providing Boot from NVMe/TCP functionality. Consisting of two simple systemd units to set up networking and perform actual connections afterwards. Requires the NetworkManager nm-initrd-generator NBFT support. Signed-off-by: Tomas Bzatek <[email protected]>

In case the `nvme connect-all --nbft` call fails, respawn the service after 10 seconds and try again. Depends on nvme-cli to return non-zero status in case of any of the SSNS records that are not marked as 'unavailable' fails to connect. The respawn cycle is broken by stopping the unit once rootfs is mounted and system switches the root. Signed-off-by: Tomas Bzatek <[email protected]>

tbzatek force-pushed the nbft-boot-refactor-1 branch from 5d23b62 to 3fdb4b9 Compare December 13, 2024 16:34

tbzatek changed the title ~~New dracut 95nbft module~~ New initramfs module Jan 7, 2025

tbzatek changed the title ~~New initramfs module~~ New NBFT initramfs module Jan 7, 2025

mwilck reviewed Jan 16, 2025

View reviewed changes

tbzatek force-pushed the nbft-boot-refactor-1 branch from 3fdb4b9 to 2473008 Compare February 12, 2025 14:29

keszybz reviewed Feb 13, 2025

View reviewed changes

tbzatek force-pushed the nbft-boot-refactor-1 branch 2 times, most recently from fb13046 to 4333526 Compare February 18, 2025 16:57

igaw reviewed Feb 19, 2025

View reviewed changes

tbzatek mentioned this pull request Feb 19, 2025

[RFE] network-generator: Add NBFT support systemd/systemd#36443

Open

tbzatek added 5 commits February 26, 2025 15:39

(squash) Move dracut-specific parts into dropins

344f237

(squash) Use meson-defined path prefixes

06c0ee5

tbzatek force-pushed the nbft-boot-refactor-1 branch from 4333526 to 06c0ee5 Compare February 26, 2025 14:40

		## nm-initrd-generator NBFT support

		https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/2077

New NBFT initramfs module #2620

Are you sure you want to change the base?

New NBFT initramfs module #2620

Conversation

tbzatek commented Dec 13, 2024 • edited Loading

mwilck commented Dec 13, 2024

mwilck commented Dec 13, 2024

igaw commented Dec 16, 2024

tbzatek commented Jan 7, 2025

mwilck commented Jan 16, 2025

mwilck Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwilck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

igaw commented Jan 17, 2025

mwilck commented Jan 17, 2025

igaw commented Jan 17, 2025 • edited Loading

aafeijoo-suse commented Feb 12, 2025

tbzatek commented Feb 12, 2025

LaszloGombos commented Feb 12, 2025

tbzatek commented Feb 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tbzatek commented Dec 13, 2024 •

edited

Loading

mwilck Jan 16, 2025 •

edited

Loading

igaw commented Jan 17, 2025 •

edited

Loading