-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
jordan
committed
Dec 4, 2020
0 parents
commit 61f803c
Showing
42 changed files
with
5,126 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| *~ | ||
| /config.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,353 @@ | ||
| <h1>QEMU Supported Raspberry PI Farming</h1> | ||
|
|
||
| <p>There is a set of <em>Bourne</em>-ish shell scripts supporting software development on | ||
| a farm of interconnected Raspberry PI entities. These entities can be guests of | ||
| some <em>QEMU virtual machine</em>, or physical <em>Raspberry PI 3 B+</em> hardware boxes. | ||
| All entities run on a copy of the same disk image.</p> | ||
|
|
||
| <p>The Raspberry PI entites are called <em>instances</em> when managed by the tools here. | ||
| When seen as part of the IP network, they are called | ||
| <em><a href="#na" title="Main Chapter">nodes</a></em>.</p> | ||
|
|
||
| <h1>Contents</h1> | ||
|
|
||
| <ul> | ||
| <li><a href="#qs" title="Main Chapter">Quick Start</a></li> | ||
| <li><a href="#ra" title="Main Chapter">Rationale</a> | ||
| <ul> | ||
| <li><a href="#dc" title="Chapter Reference">Development Cycle</a></li> | ||
| <li><a href="#mw" title="Chapter Reference">Motivation to write this toolbox</a></li> | ||
| </ul></li> | ||
| <li><a href="#so" title="Main Chapter">Set of Shell Scripts</a> | ||
| <ul> | ||
| <li><a href="#pt" title="Chapter Reference">Provided Tools</a></li> | ||
| <li><a href="#tc" title="Chapter Reference">Tweaking, Customising</a></li> | ||
| </ul></li> | ||
| <li><a href="#na" title="Main Chapter">Node Architecture</a> | ||
| <ul> | ||
| <li><a href="#hn" title="Chapter Reference">Hardware Node</a></li> | ||
| <li><a href="#vq" title="Chapter Reference">Virtual QEMU Guest Node</a></li> | ||
| </ul></li> | ||
| <li><a href="#li" title="Main Chapter">Licence</a></li> | ||
| </ul> | ||
|
|
||
| <h1><a name="qs"></a>Quick Start</h1> | ||
|
|
||
| <p>In order to see something work, copy a RaspiOS disk image and name it | ||
| <em>raspios.img</em>. Prepare the image for QEMU with</p> | ||
|
|
||
| <pre><code> sh pimage.sh --provide-qboot --disk-image=raspios.img --expand-image | ||
| </code></pre> | ||
|
|
||
| <p>and then start the virtual guest with</p> | ||
|
|
||
| <pre><code> sh pibox.sh --run | ||
| </code></pre> | ||
|
|
||
| <p>Make the virtual guest accessible for remote administration via</p> | ||
|
|
||
| <pre><code> sh pijack.sh --enrol | ||
| </code></pre> | ||
|
|
||
| <p>Finally shut down the runing application and therminate QEMU with</p> | ||
|
|
||
| <pre><code> sh pijack.sh --shutdown | ||
| </code></pre> | ||
|
|
||
| <p>For any of the tools, documentation can be printed with the combined options | ||
| <em>--help --verbose</em> (or <em>-hv</em> for short) as in</p> | ||
|
|
||
| <pre><code> sh pibox.sh --help --verbose | ||
| </code></pre> | ||
|
|
||
| <p>Instructions explain how to start from a pristine <em>RaspiOS</em> disk image | ||
| downloaded from an internet repository.</p> | ||
|
|
||
| <h1><a name="ra"></a>Rationale</h1> | ||
|
|
||
| <p>In 2019, I run a Perl application which was based on a replicated data | ||
| base implemented on a set of Raspberry PI nodes. This application was exposed | ||
| to rough conditions such as power cuts, missing Internet access, etc. Having | ||
| run successfully a proof of concept at the | ||
| <a href="http://anthroposfestival.org" title="Festival Website">Anthropos Festival</a>, | ||
| the software has been developed further into a layered architecture.</p> | ||
|
|
||
| <h2><a name="dc"></a>Development Cycle</h2> | ||
|
|
||
| <p>My intened software development cycle re the virtual evironment looks something like</p> | ||
|
|
||
| <pre><code> +-----------------------------------------------------------------+ | ||
| | | | ||
| | running a virtual test, tweak | | ||
| | +---> clone <1> of the ---> and test, ----+ | ||
| | | primary instance <0> more tests | | ||
| | | | | ||
| | | | | ||
| | | running a virtual test, tweak | | ||
| | +---> clone <2> of the ---> and test, ----+ | ||
| v | primary instance <0> more tests | | ||
| | | | ||
| enrolment, OS upgrade, | | | ||
| stock disk image installation of | running a virtual test, tweak | | ||
| downloaded from ----> additional software on ----+---> clone <3> of the ---> and test, ----+ | ||
| PI repository primary instance <0> primary instance <0> more tests | ||
|
|
||
| | ... ... | ||
| | | ||
| v | ||
|
|
||
| flash instance <0> | ||
| image copies to SD | ||
| cards and run them on | ||
| Raspberry PIs | ||
| </code></pre> | ||
|
|
||
| <p>This was sort of how I developed my proof of concept software for the festival | ||
| events, albeit without virtual support. It involved duplicating quite a few | ||
| flash cards.</p> | ||
|
|
||
| <h2><a name="mw"></a>Motivation for scripting</h2> | ||
|
|
||
| <p>I got tired of repeatedly typing variations of commands like</p> | ||
|
|
||
| <pre><code> sudo qemu-system-aarch64 -M raspi3 -m 1G -smp 4 -usb \ | ||
| -append 'rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootdelay=1' \ | ||
| -drive 'media=disk,if=sd,format=raw,file=/qemu/raspios-armhf.img' \ | ||
| -dtb '/qemu/bcm2710-rpi-3-b-plus.dtb' \ | ||
| -kernel '/qemu/kernel8.img' \ | ||
| -serial 'stdio' \ | ||
| -netdev 'user,id=user0,ipv6=off,net=100.64.63.0/24,hostfwd=tcp::5700-:22' \ | ||
| -netdev 'tap,id=tap0,ifname=q0tap0,script=no,downscript=no' \ | ||
| -netdev 'tap,id=tap1,ifname=q1tap0,script=no,downscript=no' \ | ||
| -device 'usb-mouse' \ | ||
| -device 'usb-kbd' \ | ||
| -device 'usb-net,netdev=user0' \ | ||
| -device 'usb-net,netdev=tap0' \ | ||
| -device 'usb-net,netdev=tap1' | ||
| </code></pre> | ||
|
|
||
| <p>Tools like <em>Vagrant</em> and <em>Ansible</em> come to mind for help, but I decided | ||
| against any of them because their of generality. I aimed for a more | ||
| lightweight solution tuned towards my particular development | ||
| <a href="#dc" title="Chapter Reference">cycle</a> and <a href="#na" title="Main Chapter">node</a> architecture.</p> | ||
|
|
||
| <h1><a name="so"></a>Set of Shell Scripts</h1> | ||
|
|
||
| <p>The tools are written in near <em>Bourne</em> shell syntax. Where it diverges from | ||
| the pure syntax is the use of the <em>local</em> keyword. This is used to declare | ||
| and initialise function variables with local scope. Nevertheless, the <em>local</em> | ||
| keyword feature should be supported by the following popular shells:</p> | ||
|
|
||
| <pre><code> ash, bash, bosh, busybox, dash, yash, zsh | ||
| </code></pre> | ||
|
|
||
| <p>This <em>local</em> keyword feature as used here is not <em>ksh</em> (i.e. <em>Korn</em> shell) | ||
| compatible.</p> | ||
|
|
||
| <h2><a name="pt"></a>Provided Tools</h2> | ||
|
|
||
| <ul> | ||
| <li><p><em>pimage -- disk image maintenance tool</em><br> | ||
| The tool prepares a RaspiOS disk image for use with QEMU. It will extract a | ||
| copy of RaspiOS boot files needed by QEMU to start on the guest disk image. | ||
| It registers the location of the argument disk image file path to be used by | ||
| subsequent invocations. The tool works with <em>raw</em> images only, e.g. | ||
| extracted with <em>dd if=/dev/sda of=disk.img</em> (as opposed to the <em>qcow2</em> | ||
| virtual machine format.)</p></li> | ||
| <li><p><em>pibox -- virtual machine host manager</em><br> | ||
| The tool starts and stops the virtual QEMU machine for guest disk images. | ||
| Guest disk images are referred to as <em>instances</em> and identified by a number. | ||
| Default is the <em>zero</em> instance <em><0></em> running on the primary <em>raw</em> | ||
| RaspiOS disk image. All other instances are <em>qcow2</em> formatted copies of the | ||
| <em><0></em> instance.<br> | ||
| <br> | ||
| By extension, a virtual QEMU machine for instance <em><N></em> is | ||
| called <em>the</em> instance <em><N></em> if the meaning is otherwise clear.</p></li> | ||
| <li><p><em>pijack -- virtual guest manager</em><br> | ||
| The tool manages a running virtual QEMU guest instance via IP network | ||
| connections for <em>console</em> or <em>SSH</em>. Administrative commands are typically | ||
| sent to the guest instance via <em>SSH</em>, authenticated by the command user's | ||
| public key. The tool can hijack the guest instance via <em>console</em> login and | ||
| prepare it for <em>SSH</em> with public key authentication.</p></li> | ||
| <li><p><em>pibridge -- virtual bridge setup</em><br> | ||
| This tool scans the network interface table on the host system for | ||
| active virtual QEMU guest <em><lan></em> interfaces. Then the tool adds these | ||
| interfaces to a bridge. Seen from a virtual guest, all active interfaces | ||
| are connected.<br> | ||
| <br> | ||
| Additional non-virtual interfaces can be added to the bridge. If this is | ||
| an outbound interface connected to <em><lan></em> interfaces of physical | ||
| Raspberry PIs, these systems join the virtual network.</p></li> | ||
| </ul> | ||
|
|
||
| <p>All commands support the options combination --dry-run --verbose which have | ||
| the tools print out the shell commands that <em>would</em> be run rather than really | ||
| executing them. Apart from auditing privilege escalation via <em>sudo</em>, this | ||
| feature is intended to be used when building bespoke scripts for particular | ||
| QEMU related tasks.</p> | ||
|
|
||
| <p>The following tools do <em>not</em> use <em>sudo</em> privilege escalation</p> | ||
|
|
||
| <ul> | ||
| <li><em>pijacks</em> -- never (but will call <em>pibox</em> for terminating QEMU)</li> | ||
| <li><em>pimage</em> -- never if <em>guestmount</em> is available</li> | ||
| </ul> | ||
|
|
||
| <p>The other tools will do need <em>sudo</em>, see the command help pages (--help | ||
| --verbose) for details.</p> | ||
|
|
||
| <h2><a name="tc"></a>Tweaking, Customising</h2> | ||
|
|
||
| <p>Provide a file <em>config.sh</em> in the same folder where the <em>pibox.sh</em> and the | ||
| other tools reside. See the comments in the file <em>lib/variables.sh</em> for | ||
| possible settings. There is an advanced example configuration script | ||
| <em>examples/config.sh</em>.</p> | ||
|
|
||
| <p>The tools will complain if any of the following commands is unavailable</p> | ||
|
|
||
| <ul> | ||
| <li><em>fatcat</em> -- tool for extracting files from DOS partition images</li> | ||
| <li><em>remote-viewer</em> -- a VNC gui and reote desktop viewer</li> | ||
| <li><em>guestmount</em> -- user mode mounting tool for disk partition images</li> | ||
| </ul> | ||
|
|
||
| <p>Complaints can be stopped by disabling any of these tools setting variables | ||
| <em>NOFATCAT</em>, <em>NOREMOTE_VIEWER</em>, or <em>NOGUESTMOUNT</em> in the <em>config.sh</em> | ||
| configuration file. For example. setting</p> | ||
|
|
||
| <pre><code> NOGUESTMOUNT=set | ||
| </code></pre> | ||
|
|
||
| <p>will tell the tools to ignore the particular program completely in which | ||
| case functionality is degraded unless there is a work-around (e.g. <em>sudo</em> | ||
| losetup+mount.)</p> | ||
|
|
||
| <h1><a name="na"></a>Node Architecture</h1> | ||
|
|
||
| <p>Raspberry PI entites are called <em>nodes</em> when part of an IP network. <em>Nodes</em> | ||
| are identified by a non-negative number called the node ID (which coincides | ||
| with the <em>instance ID</em>.)</p> | ||
|
|
||
| <p>So a <em>node</em> can be associated with</p> | ||
|
|
||
| <ul> | ||
| <li>a copy of a particular disk image, and</li> | ||
| <li>either | ||
| <ul> | ||
| <li>a physical <em>Raspberry PI 3 B+</em> hardware box, or</li> | ||
| <li>a guest of a <em>QEMU virtual machine</em></li> | ||
| </ul></li> | ||
| </ul> | ||
|
|
||
| <p>Nodes share the same disk image, still they need unique <em>node IDs</em>. For | ||
| <em>Raspberry PI 3 B+</em> hardware boxes, they differ by the MAC address of their | ||
| built-in NIC which is exploited to map the <em>node ID</em>. For the virtual guest | ||
| case, differentiation is handled by the way the QEMU machine is started.</p> | ||
|
|
||
| <p>Each node has three interfaces of type <em>WAN</em>, <em>LAN</em>, and <em>WLAN</em>. The <em>LAN</em> | ||
| interface is used for peer-to-peer communication between nodes and the <em>WLAN</em> | ||
| interface for providing outbound services. The <em>WAN</em> interface is considered | ||
| administrative, only.</p> | ||
|
|
||
| <h2><a name="hn"></a>Hardware Node</h2> | ||
|
|
||
| <p>For the Raspberry PI 3 B+ hardware, a diagram for a node <id> looks like</p> | ||
|
|
||
| <pre><code> Raspberry PI 3 B+ : External Access | ||
| -------------------------+--------------------------------- | ||
| ,---------------. : | ||
| | | : | ||
| | <wan> o----------------o USB/pluggable adaptor | ||
| | | : | ||
| | lan<id> o----------------o internal NIC | ||
| | | : | ||
| | wlan0 o----------------o internal WiFi access point | ||
| | | : | ||
| `---------------' : | ||
| </code></pre> | ||
|
|
||
| <p>where <id> is some positive number. There is no <em>zero</em> node for the | ||
| <em>Raspberry PI 3 B+</em> hardware box. The hardware node interfaces have the | ||
| following properties:</p> | ||
|
|
||
| <ul> | ||
| <li><p><wan></p> | ||
|
|
||
| <ul> | ||
| <li>the <wan> interface name depends on the USB/pluggable adaptor</li> | ||
| <li>the interface need not be available, neither activated</li> | ||
| <li>when activated, the default route points to that interface</li> | ||
| <li>unpredictable IP address</li> | ||
| </ul></li> | ||
| <li><p>lan<id> <em>(implies node ID)</em></p> | ||
|
|
||
| <ul> | ||
| <li>the <id> is implied by the MAC address of the built-in NIC</li> | ||
| <li>the interface is activated</li> | ||
| <li>predictable IP address dependent on the node ID</li> | ||
| </ul></li> | ||
| <li><p>wlan0</p> | ||
|
|
||
| <ul> | ||
| <li>interface is activated and configured as WiFi access point</li> | ||
| <li>same IP address for all nodes</li> | ||
| </ul></li> | ||
| </ul> | ||
|
|
||
| <p>Apparently, as interface names are dependent on MAC addresses, they need to | ||
| be per-configured in the <em>/etc/iftab</em> file. The node ID mapping information is | ||
| held in the <em>/etc/host</em> file. If properly configured, the <wan> | ||
| interface name reads <em>wan<n></em> for some positive number <em>n</em>.</p> | ||
|
|
||
| <h2><a name="vq"></a>Virtual QEMU Guest Node</h2> | ||
|
|
||
| <pre><code> Virtual Guest : Virtual Host | ||
| -------------------------+--------------------------------- | ||
| ,---------------. : | ||
| | | : | ||
| | wan0 o----------------o QEMU user socket | ||
| | | : | ||
| | lan0 o----------------o TAP interface q0tap<id> | ||
| | | : | ||
| | wlan0 o----------------o TAP interface q1tap<id> | ||
| | | : | ||
| `---------------' : | ||
| </code></pre> | ||
|
|
||
| <p>where the <id> stand for non-negative numbers (i.e. it can be zero). The | ||
| interfaces have the following properties:</p> | ||
|
|
||
| <ul> | ||
| <li><p>wan0 <em>(implies node ID)</em></p> | ||
|
|
||
| <ul> | ||
| <li>interface is activated</li> | ||
| <li>the default route points to that interface</li> | ||
| <li>IP address is provided via DHCP by the QEMU virtual machine</li> | ||
| <li><id> is implied by the IP address</li> | ||
| <li>accessible from virtual host via ssh://localhost:<5700+id></li> | ||
| </ul></li> | ||
| <li><p>lan0</p> | ||
|
|
||
| <ul> | ||
| <li>interface is activated</li> | ||
| <li>predictable IP address dependent on the node ID</li> | ||
| </ul></li> | ||
| <li><p>wlan0</p> | ||
|
|
||
| <ul> | ||
| <li>interface is activated and configured as WiFi access point</li> | ||
| <li>same IP address for all nodes (the same as in the hardware case)</li> | ||
| </ul></li> | ||
| </ul> | ||
|
|
||
| <p>Here, node ID configuration is superimposed by the <em>QEMU virtual machine</em> when | ||
| configuring the WAN interface via DHCP. Nevertheless, LAN and WLAN interfaces | ||
| follow the same logic as in the hardware case.</p> | ||
|
|
||
| <h1><a name="li"></a>Licence</h1> | ||
|
|
||
| <p>This is free and unencumbered software released into the public domain.</p> | ||
|
|
||
| <p>See <a href="UNLICENSE" title="Local Copy of Unlicense">UNLICENSE</a> or | ||
| <a href="http://unlicense.org/" title="Web Location">Unlicense project</a> for details.</p> |
Oops, something went wrong.