implement ephemeral port reservation for k3d

[chrisseto]

Prior to this commit we discovered that CI can flake when standing up multiple k3d clusters due to port conflicts.

This commit attempts to implement "port reservation" such that pkg/k3d can always create k3d clusters without conflicts without coordination across different processes.

It's unclear if this will actually work as engineering an intentional failure is not feasible.

[RafalKorepta]

I like this improvement even if #296 fixed other issues.

[chrisseto]

So, fun story with this. I can't actually prove/showcase that this will work. The implementation on linux may be different from macos/darwin/bsd but: When testing on macos, it seems that port conflicts do still get generated after running this test with -count=50 though the sockets/ports do get put into the TIME_WAIT state.

I've not been able to track down any information about how ephemeral port assignment works in macos. It seems that ports in TIME_WAIT will be left alone until ephemeral ports near exhaustion. After which they get reassigned.

[birdayz]

Thanks, this is an interesting approach!

Copying this approach to cloud's internal repo as well.
Whenever possible, we try to allocate with address ":", so the OS assigns a free port to our application, and then we pass along this address.
However, sometimes this is hard, eg if third party deps don't support this approach. So it's great to have a working fallback solution; so far we did only randomly allocate, and hope no other test randomly picks the same random port.

[chrisseto]

Let me know if this actually solves any issues in cloud. On macOS at least, this doesn't seem to do much 😓