ISO SR : allow scan of subdirectories

[imakiro]

During our tests of the solution, it seems impossible to add an ISO storage repository and be able to retrieve ISOs located in the subfolders of this share.
It seems that this constraint exists since forever on xenserver...
What would it take to enable this functionality? is it possible? Was this requested and closed (if so, sorry, but I didn't find it using the "iso" keyword)

[olivierlambert]

I suppose this is the case to avoid very long SR rescan (going deep in the folder tree and filter all ISOs files inside).

I'm not 100% sure where the ISO filter code is, but it's very likely in Python so easy to hack and test. Would you like to make those experiments @imakiro ?

[imakiro]

Sure @olivierlambert , I'm all up for it and testing stuff is the least I can do to help right now.
Or maybe you could point me to the appropriate code segment?

[olivierlambert]

I suppose it's there: https://github.com/xapi-project/sm/blob/master/drivers/ISOSR.py

Current code parse all files and return only ISO and img files. So it would require extra code to inspect content of folders, probably with a depth of 1 folder to start.

[imakiro]

Ok, I'm having a look, but a quick question : under wich version of python is the project developped against?

[olivierlambert]

2.7 IIRC. @stormi ?

[stormi]

Indeed

[imakiro]

Ah, I've been hacking at the project with 3.8, I usually don't deal with obsolete versions of python and learned only Python 3 syntax.
It'll probably take a bit to setup a new environment with 2.7, I'll see what I can do.
update 1 : Ok, I need to also totally switch to a new distro : fnctl and syslog are totally linux dependant. Contributing to this project can't really be done easily...

[imakiro]

For this functionality, setting a depth parameter forbids the use of "os.walk", also is there a reason a non purely pythonic solution that runs an external command to list files / directories is used ?

[stormi]

We're not the original authors of that code so we don't know. Maybe the commit history of the repository can give a clue.

[imakiro]

succes !
Ok, so in util.py replacing
def listdir(path, quiet = False): cmd = ["ls", path, "-1", "--color=never"] try: text = pread2(cmd, quiet = quiet)[:-1] if len(text) == 0: return [] return text.split('\n') except CommandException, inst: if inst.code == errno.ENOENT: raise CommandException(errno.EIO, inst.cmd, inst.reason) else: raise CommandException(inst.code, inst.cmd, inst.reason)

with

def listdir(path, quiet = False, depth = 10): cmd = ["find", path, "-maxdepth", str(depth)] try: text = pread2(cmd, quiet = quiet)[:-1] if len(text) == 0: return [] return text.split('\n') except CommandException, inst: if inst.code == errno.ENOENT: raise CommandException(errno.EIO, inst.cmd, inst.reason) else: raise CommandException(inst.code, inst.cmd, inst.reason)

Does the trick !

[imakiro]

After having a whole lot of trouble with the project's signoff system, here's the pull request : xapi-project/sm#469

[TheFou]

Hi,
I see this has been open for a long time, without any implementation.
I'm not a not a developer unfortunately, but I really wonder how such a simple thing can stay stale for so long.

Also, one thing I don't understand : Why the need to scan all the isos in subfolders ?
Why not present the ISO SR "as is" with folders, maybe with a filter on some relevant file extensions, but nothing more ?
This way we can browse into subfolders as needed... especially since, if there are subfolders, it is probably for a proper management and sorting reason !
The current behaviour makes no sense.

Honestly, I find there are many little friction points like that, that make it really hard to go from VMware to XCP-NG.
I'm trying, but the itch to go back to VMware is really there.

[olivierlambert]

Hi,

I'm sorry you feel this way, but I’d like to give some context on why this feature hasn’t been implemented yet. Our backlog contains at least 500 main items, and at any given time, we can actively work on only about 10% of them. We prioritize based on urgency and impact: removing the 2TiB limit, for example, has been a higher priority. While Vates has been growing rapidly (tripling the team size in 2 years), we still have to make tough choices about where to focus our efforts.

Regarding your technical question: detecting new ISOs in deeply nested folders isn't trivial. The current storage API requires a full rescan to detect changes, and without a major architectural overhaul, that’s the only viable method. Introducing real-time filesystem event detection would be a deep and complex change, potentially introducing regressions and instability. XCP-ng’s stability is one of its key strengths: we’ve never pushed an update that broke anything, even with monthly security updates. So any modification to core behaviors needs careful consideration.

That said, I completely understand that certain differences from VMware can feel like friction points. But here’s the beauty of XCP-ng: it’s fully open source, meaning you have the power to influence its direction, even if you're not a developer.

Constructive discussions and positive engagement make developers more inclined to prioritize a request, especially when they’re doing this work for free. Additionally, customers with professional support contracts help shape priorities since they directly contribute to funding development. And if a particular feature is critical for you, sponsoring its implementation is always an option.

I hope this gives you more insight into how the project operates and why things are prioritized the way they are

[nagilum99]

I'm not a not a developer unfortunately, but I really wonder how such a simple thing can stay stale for so long.

That's a reason for the problem: What looks just like an easy thing, under the hood might not be that way.

I'm not mainly a developer, but I did that aside for a few years and some very simple looking things took a lot to code and check