From 2cd6fa7990fd58a583194de998a82c31c260bf6e Mon Sep 17 00:00:00 2001 From: Julian Squires Date: Thu, 23 Nov 2023 14:28:41 -0330 Subject: [PATCH] Add some ancient text updates These mostly date back to 2017 but just sat in the worktree uncommitted. Fixes #6 from the looks of it. --- README.md | 22 +++++++++++++++++++++- next-steps.md | 2 ++ stage_1.md | 11 ++++++++++- stage_2.md | 14 +++++++++++++- stage_4.md | 3 ++- 5 files changed, 48 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index cb209c4..e439b0f 100644 --- a/README.md +++ b/README.md @@ -120,6 +120,14 @@ of different approaches. # Supplementary Material +## Tools + + - The [shtepper] is a great resource for understanding shell + execution. You can input an expression and see in excruciating + detail how it should be evaluated. + +[shtepper](http://shell.cs.pomona.edu/shtepper) + ## Documents - [Advanced Programming in the Unix Environment] by Stevens covers @@ -158,6 +166,8 @@ you're having trouble with a stage they cover: its utilities in reasonable detail. - there are [POSIX conformance test suites] but they don't seem to be available in convenient, non-restricted forms. + - [yash's posix-shell-tests] are only runnable with yash, but the + tests themselves are full of useful ideas. [A Unix Shell in Ruby]: http://www.jstorimer.com/blogs/workingwithcode/7766107-a-unix-shell-in-ruby [Advanced Programming in the Unix Environment]: http://www.apuebook.com/ @@ -173,6 +183,7 @@ you're having trouble with a stage they cover: [Unix system programming in OCaml]: https://ocaml.github.io/ocamlunix/ [Write a Shell in C]: https://brennan.io/2015/01/16/write-a-shell-in-c/ [POSIX conformance test suites]: https://www.opengroup.org/testing/testsuites/vscpcts2003.htm +[yash's posix-shell-tests]: https://github.com/posix-shell-tests/posix-shell-tests ## Shells to Examine @@ -189,6 +200,7 @@ you're having trouble with a stage they cover: - [oil]: Python and C++; has an extensive test suite. - [xonsh](http://xon.sh/): Python. - [oh](https://github.com/michaelmacinnis/oh): Go. + - [yash-rs](https://github.com/magicant/yash-rs): Rust. [busybox]: https://git.busybox.net/busybox/tree/shell [cash]: https://github.com/ShamoX/cash @@ -234,7 +246,7 @@ There's also [sb-posix] in `sbcl` for the daring. - use [the unix package](https://hackage.haskell.org/package/unix) - [Hell] might be a starting point -[Hell]: https://github.com/chrisdone/hell/tree/master/src +[Hell]: https://github.com/chrisdone/hell/ ### Java / JVM-based languages @@ -247,8 +259,10 @@ There's also [jtux](http://basepath.com/aup/jtux/index.htm). ### Lua There are a variety of approaches, but [ljsyscall] looks promising. +[luaposix] might be sufficient. [ljsyscall]: https://github.com/justincormack/ljsyscall +[luaposix]: https://github.com/luaposix/luaposix ### OCaml @@ -269,8 +283,14 @@ Although Python provides higher-level abstractions like [`subprocess`], for the purposes of this workshop you probably want to use the functions in [`os`]. +Please note an important gotcha for stage 2! Since [Python 3.4], fds +have defaulted to non-inheritable, which means you'll need to +explicitly `os.set_inheritable(fd, True)` any file descriptor you +intend to pass down to a child. + [`os`]: https://docs.python.org/3/library/os.html [`subprocess`]: https://docs.python.org/3/library/subprocess.html +[Python 3.4]: https://peps.python.org/pep-0446/ ### Racket diff --git a/next-steps.md b/next-steps.md index 6d23d62..9e82203 100644 --- a/next-steps.md +++ b/next-steps.md @@ -61,6 +61,8 @@ sh`? - GNU parallel is a very popular tool; is it possible to provide its major features directly in a shell? +See also [`dgsh`](https://www2.dmst.aueb.gr/dds/sw/dgsh/). + ## alternative uses of the shell - it's often been noted that `make` and `cron`, maybe with a dash of diff --git a/stage_1.md b/stage_1.md index ffb45c4..387f9c9 100644 --- a/stage_1.md +++ b/stage_1.md @@ -164,7 +164,7 @@ lists*, respectively. See section 2.9.3, [Lists]. We'll look at (If you want to add support for compound lists surrounded by braces, go ahead, but I didn't consider them important enough for an -interactive shell to bother testing them. +interactive shell to bother testing them.) [Lists]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_03 @@ -213,6 +213,12 @@ See [Token Recognition] in the standard. Writing a tokenizer that follows this, ignoring the parts we aren't doing yet, will make writing your parser easier. I'll expand this section more, shortly. +You're faced with a tricky choice, here: I wrote this originally +trying to put off quoting and other difficulties until later, just +doing the simplest operations to get things going. But after writing +a few shells, I feel that getting proper parsing working is worth the +hassle. It's painful, but it makes everything else much easier. + [Token Recognition]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_03 ## Notes @@ -261,6 +267,9 @@ See also https://ewontfix.com/7/ Stevens, APUE, chapter 8. Kerrisk, LPI, chapters 24 through 28. +A fork() in the road: https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/ +Introducing `io_uring_spawn`: https://lwn.net/Articles/908268/ + ["A much faster popen and system implementation for Linux"]: https://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/ [FreeBSD's posix_spawn]: https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/posix_spawn.c [glibc's generic implementation of posix_spawn]: https://github.com/bminor/glibc/blob/master/sysdeps/posix/spawni.c diff --git a/stage_2.md b/stage_2.md index 6577f2b..95b31ee 100644 --- a/stage_2.md +++ b/stage_2.md @@ -102,6 +102,10 @@ The following table summarizes the fd redirections: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07 +Did you know about `<>`? That's something I only learned when I wrote +this workshop. Note there's also `>|` which interacts with the +shell's `noclobber` option, which I haven't bothered to implement. + Note that `!`, which you added in stage 1, may appear only at the start of a pipeline, and affects the return value of the whole pipeline. @@ -130,7 +134,7 @@ but it's really convenient. You might have a bunch of files open in your shell, for example, files for maintaining history or configuration. Your children shouldn't have to care about these files; file descriptors are a limited -resource and most peolple wouldn't appreciate having that limit +resource and most people wouldn't appreciate having that limit unnecessarily decreased just because you were lazy. How can you prevent your children from inheriting fds you don't want @@ -145,6 +149,13 @@ about, particularly when writing a library that might open fds. Recently, Linux [added an `fdmap` syscall] that could be used for this. +Since I first wrote this, several languages have made `CLOEXEC` the +default, which means you'll have to disable it on the file descriptors +you actually want passed to the child. For example [PEP-446], adopted +in Python 3.4, makes `CLOEXEC` the default for all these reasons, so +you'll need to `os.set_inheritable(fd, True)` explicitly on the file +descriptors you actually want passed to the child. + See Chris Siebenmann's [fork() and closing file descriptors] and CERT's [FIO22-C] (close files before spawning processes). @@ -155,6 +166,7 @@ your shell, with various redirections, and see what happens. [fork() and closing file descriptors]: https://utcc.utoronto.ca/~cks/space/blog/unix/ForkFDsAndRaces [FIO22-C]: https://www.securecoding.cert.org/confluence/display/c/FIO22-C.+Close+files+before+spawning+processes [added an `fdmap` syscall]: https://lwn.net/Articles/734709/ +[PEP-446]: https://peps.python.org/pep-0446/ ### Builtins diff --git a/stage_4.md b/stage_4.md index f6cffeb..a3b652b 100644 --- a/stage_4.md +++ b/stage_4.md @@ -127,9 +127,10 @@ pipe. There can actually be multiple heredocs on a line, but if you only want to implement one for now, I think that's ok. -See [Heredocs]. +See [Heredocs], [How to Parse Here Documents]. [Heredocs]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 +[How to Parse Here Documents]: http://www.oilshell.org/blog/2016/10/17.html ### Globbing