Create salt-troubleshooting.md

user/troubleshooting/salt-troubleshooting.md

@@ -0,0 +1,55 @@
+---
+lang: en
+layout: doc
+permalink: /doc/salt-troubleshooting/
+redirect_from:
+- /doc/salt/
+- /en/doc/salt/
+ref: 1000000000000
+title: Salt troubleshooting
+---
+
+For ease of Qubes Os managament and reproductible deployment, [Salt](/doc/salt/) allows to control states on `dom0` and other vms from the `dom0`.
+
+Behind the scenes
+-----------------
+
+Except for `dom0` where the host is controlled locally.
+Each vm (named `minion-vm` for instance) is controlled by a disposable master vm based on `disposable-mgmt-vm`, named `disp-mgmt-minion-vm` and created only for the duration of `qubesctl` execution.
+
+The required files are copied from `dom0` to `disp-mgmt-minion-vm` via `qubes.Filecopy`, then `qubes.SaltLinuxVM` and expect two lines on stdin :
+```
+minion-vm
+salt-command
+```
+
+Usually `salt-command` is `state.apply` with the provided arguments like `test=True`.
+
+Then, a fake `ssh` command wrapper included in `qubes-mgmt-salt-vm-connector` allow to run the command on the target (`minion-vm`) via `qubes.VMShell` or `qubes.VMRootShell`.
+On the management vm `disp-mgmt-minion-vm`, salt firstly creates `/var/cache/salt/master/thin/thin.tgz` and transfers it to the `minion-vm` to ensure destination host has the required python files.
+
+How to debug the ephemeral disposable management vm
+---------------------------------------------------
+
+First, the transfered content from `dom0` to the disposable management vm needs to be retrieved. To do so, it is suggested to:
+1. Call from `dom0`, `qubesctl` with requested command like `qubesctl --show-output --targets minion-vm --skip-dom0 state.apply`,
+2. Freeze the previous command with `Ctrl+Z` as soon as you see `minion-vm is starting`.
+3. Get the console on the disposable management vm with `qvm-console-dispvm disp-mgmt-minion-vm` on the dom0
+4. Type `root` to log as root on the console
+5. Edit in `disp-mgmt-minion-vm`, `/etc/qubes-rpc/qubes.SaltLinuxVM` and add after the line `eval "dir=~$user/QubesIncoming/dom0/srv"`, the line `qvm-copy $dir`.
+6. On the `dom0` resume the freezed process with `fg`
+7. Copy the content to another qubes vm (`side-vm` for instance)
+
+Second, a debugable disposable management vm is setup. To do so, it is suggested to:
+1. Call from `dom0`, `qubesctl` with requested command like `qubesctl --show-output --targets minion-vm --skip-dom0 state.apply`,
+2. Freeze the previous command with `Ctrl+Z` as soon as you see `minion-vm is starting`.
+3. Copy the retrieved content from `side-vm` to the disposable management vm `disp-mgmt-minion-vm` (with `qvm-copy`)
+4. Get the console on the disposable management vm with `qvm-console-dispvm disp-mgmt-minion-vm` on the dom0
+5. Type `root` to log as root on the console. All following commands are done inside the console.
+6. Move the copied content to emulate a content coming from `dom0`: `cd /home/user/QubesIncoming; mv * dom0`. `dom0` directory should contain a directory `srv`.
+7. Emulate a call to `qubes.SaltLinuxVM` with `bash /etc/qubes-rpc/qubes.SaltLinuxVM`
+8. Emulate stdin. Type the destination vm on the first line (`minion-vm`), the salt command on the second line (`state.apply` for instance) then `Ctrl+D`.
+9. A first execution is launched
+10. Get wrappers in the `PATH` with `export PATH="/usr/lib/qubes-vm/connector/ssh-wrapper:$PATH" (the line is available in `/etc/qubes-rpc/qubes.SaltLinuxVM`
+
+Third, launch as many times as needed the following command to emulate a new call of master to the minion `rm -r /var/cache/salt /var/tmp/.root*; salt-ssh -w minion-vm salt-command` in the console.