CHIA-1697: Add new flag to support recursively scanning and following links

[emlowe]

Fixes #16268

Adds new harvester option follow_links that only works when used with recursive_scan and uses glob.glob instead of Path.rglob in order to follow links when scanning recursively for plots.

[emlowe]

It seems the hardlink test was skipped on all the OSs.

A little more research tells me hard links don't work for directories at all on any of the main OS (macos, linux, windows)

[coveralls-official]

Pull Request Test Coverage Report for Build 11632202557

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 37 of 37 (100.0%) changed or added relevant lines in 2 files are covered.
• 31 unchanged lines in 5 files lost coverage.
• Overall coverage increased (+0.03%) to 90.893%

---

Files with Coverage Reduction	New Missed Lines	%
chia/full_node/full_node_api.py	1	82.72%
chia/server/node_discovery.py	1	81.83%
chia/rpc/rpc_server.py	3	88.0%
chia/wallet/wallet_node.py	4	88.26%
chia/_tests/core/util/test_lockfile.py	22	77.31%

Totals
Change from base Build 11618130161:	0.03%
Covered Lines:	102788
Relevant Lines:	112866

---

💛 - Coveralls

[arvidn]

this looks good. However, the bug report you reference is talking about hard links, not symbolic links. I suppose it may be a bit tricker to test as not all filesystems may support them. But did you test this manually with hard links too?

[altendky]

I see the term hard-links there. But, is that much of a thing? I thought hard links didn't even show as being a link since they are just a regular whatever like the first whatever. They also presumably couldn't exist across volumes like are shown there? I know I'm extrapolating a bit here.

lrwxrwxrwx+ 1 root root 16 Sep 29 2021 plots01 -> /volume2/plots01
lrwxrwxrwx+ 1 root root 16 Sep 29 2021 plots02 -> /volume3/plots02
lrwxrwxrwx+ 1 root root 16 Sep 29 2021 plots03 -> /volume4/plots03

Do you know of a filesystem that supports directory hard links?

[arvidn]

oh yeah, those do look like soft links

[emlowe]

Yea, I think the reporter was just incorrect but I didn't want to correct the report. My research indicates hard links to directories aren't something that any OS supports at least in the normal filesystems.