Update the fsfinder to use concurrency

[kehoecj]

The current implementation of the file system finder is suitable for most use-cases but can get slow when recursing through a very large directory tree. It would improve performance to update the fsfinder to launch a new goroutine for each subdirectory. Determine a benchmark and determine if the worker pool needs to be limited to maximize performance as well as measuring the performance increase using goroutines

[AshwinKul28]

I can pick this up! @kehoecj

[kehoecj]

It's yours!

[kehoecj]

@AshwinKul28 Are you still interested in working on this issue?

[Axecalever]

@kehoecj, can I take this ?

[kehoecj]

@kehoecj, can I take this ?

Absolutely! I assigned it to you

[Axecalever]

@kehoecj is there any specific dataset of config files that I could benchmark against?

[kehoecj]

@kehoecj is there any specific dataset of config files that I could benchmark against?

No, we don't have a current dataset to benchmark against. It would be less about a dataset and more about a large tree of subdirectories, for instance running the validator against /etc, /, etc

[kehoecj]

@Axecalever Did you still want to work this issue?

[nikonhub]

Hello.

What do you think about using an external module like https://github.com/iafan/cwalk
I've tried it, and it seems 2x times faster.

[kehoecj]

@nikonhub You're welcome to put up a draft MR with the implementation