Fix workload socket denied

[huguesBouvier]

Fix for #5672, #5505 , #5693

Restarting iotedge many time lead to the following permission issues:

This doesn't seem to happen in 1.1.6. I tried many times but could not get it to fail however some customer experienced the same symptoms so this is most likely an issue across version.
The fact that it seems to fail more on some setup is something we could not explain.

Tests:

1. Tried all of those. A few times manually (3-5 times), a cycle of hundreds time each with a script (Ubuntu + Centos).
   For each test we check that all module are up and running, that permissions and user are correct and that there is a listener on the socket:
   1.1 sudo iotedge system restart
   1.2 sudo iotedge system stop + delete all containers + sudo iotedge system restart
   1.3 sudo iotedge system stop + delete /var/lib/aziot/edged/mnt/ folder + sudo iotedge system restart
   1.4 sudo iotedge system stop + delete all container + delete /var/lib/aziot/edged/mnt/ folder + sudo iotedge system restart
   1.5 sudo iotedge system stop + delete workloads inside /var/lib/aziot/edged/mnt + create a sudo dir inside with the name of the sockets + sudo iotedge system restart

Logs
ubuntu18.txt
centos.txt

scripts code:

while :
do
  echo "Test 1"
  sudo iotedge system restart
  sleep 310s
  sudo iotedge list
  ls -l /var/lib/aziot/edged/mnt/
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeAgent.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeHub.sock http://127.0.0.1


  echo "Test 2"
  sudo iotedge system stop
  sudo docker rm -f edgeAgent
  sudo docker rm -f edgeHub
  sudo docker rm -f SimulatedTemperatureSensor
  sudo iotedge system restart
  sleep 120s
  sudo iotedge list
  ls -l /var/lib/aziot/edged/mnt/
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeAgent.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeHub.sock http://127.0.0.1

  echo "Test 3"
  sudo iotedge system stop
  sudo rm -r /var/lib/aziot/edged/mnt
  sudo iotedge system restart
  sleep 310s
  sudo iotedge list
  ls -l /var/lib/aziot/edged/mnt/
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeAgent.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeHub.sock http://127.0.0.1

  echo "Test 4"
  sudo iotedge system stop
  sudo docker rm -f edgeAgent
  sudo docker rm -f edgeHub
  sudo docker rm -f SimulatedTemperatureSensor
  sudo rm -r /var/lib/aziot/edged/mnt
  sudo iotedge system restart
  sleep 120s
  sudo iotedge list
  ls -l /var/lib/aziot/edged/mnt/
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeAgent.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeHub.sock http://127.0.0.1

  echo "Test 5"
  sudo iotedge system stop
  sudo rm /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock
  sudo rm /var/lib/aziot/edged/mnt/edgeAgent.sock
  sudo rm /var/lib/aziot/edged/mnt/edgeHub.sock
  sudo mkdir /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock
  sudo mkdir /var/lib/aziot/edged/mnt/edgeAgent.sock
  sudo mkdir /var/lib/aziot/edged/mnt/edgeHub.sock
  sudo iotedge system restart
  sleep 310s
  sudo iotedge list
  ls -l /var/lib/aziot/edged/mnt/
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/SimulatedTemperatureSensor.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeAgent.sock http://127.0.0.1
  curl curl --unix-socket  /var/lib/aziot/edged/mnt/edgeHub.sock http://127.0.0.1

done

2. Pipeline run:
   Package: 47895915
   Run https://dev.azure.com/msazure/One/_build/results?buildId=47896816&view=results

3. Test the protection against 2 listeners: Crashed edgeAgent to have edged restart it. EdgeAgent doesn't get stopped before starting like other module:
   Listener EdgeAgent already started, removing old listener.
   Confirmed that 2 listeners is not a problem.

Azure IoT Edge PR checklist:

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines and Best Practices

• I have read the contribution guidelines.
• Title of the pull request is clear and informative.
• Description of the pull request includes a concise summary of the enhancement or bug fix.

Testing Guidelines

• Pull request includes test coverage for the included changes.
• Description of the pull request includes
  • concise summary of tests added/modified
  • local testing done.

Draft PRs

• Open the PR in Draft mode if it is:
  • Work in progress or not intended to be merged.
  • Encountering multiple pipeline failures and working on fixes.

Note: We use the kodiakhq bot to merge PRs once the necessary checks and approvals are in place. When it merges a PR, kodiakhq converts the PR title to the commit title, PR description to the commit description, and squashes all the commits in the PR to a single commit. The net effect is that entire PR becomes a single commit. Please follow the best practices mentioned here for the PR title and description

[damonbarry]

I'd like to see the end-to-end tests passing with this change, Windows and Linux, before we check it in.

[huguesBouvier]

I'd like to see the end-to-end tests passing with this change, Windows and Linux, before we check it in.

Agreed. This is 1.2, for some unknown reason, I could not reproduce it in 1.1

[damonbarry]

Agreed. This is 1.2, for some unknown reason, I could not reproduce it in 1.1
Alright, thanks.