From 0854cb7705d93ba8a950092cc7aab229f0786580 Mon Sep 17 00:00:00 2001 From: Dave Abrahams Date: Tue, 23 Feb 2021 11:00:02 -0800 Subject: [PATCH 1/3] New README --- IMPLEMENTATION.md | 574 ++++++++++++++++++++++++++++++++++++++++++ README.md | 628 ++++++++-------------------------------------- 2 files changed, 680 insertions(+), 522 deletions(-) create mode 100644 IMPLEMENTATION.md diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md new file mode 100644 index 0000000..15aaffe --- /dev/null +++ b/IMPLEMENTATION.md @@ -0,0 +1,574 @@ +# lmt - literate markdown tangle + +This README describes a tangle program for a literate programming style +where the code is weaved into markdown code blocks. There is no corresponding +weave, because the markdown itself can already be read as the documentation, +either through a text editor or through an online system that already renders +markdown such as GitHub. + +## Why? + +[Literate programming](https://en.wikipedia.org/wiki/Literate_programming) is a +style of programming where, instead of directly writing source code, the programmer +writes their reasoning in human prose, and intersperses fragments of code which +can be extracted into the compilable source code with one tool (called "tangle"), +and conversely can be converted to a human readable document explaining the code +with another (called "weave"). + +[Markdown](http://daringfireball.net/projects/markdown/syntax) is a plaintextish +format popular with programmers. It's simple, easy and already has support +for embedding code blocks using triple backticks (```), mostly for the purposes +of syntax highlighting in documentation. + +The existing literate programming for markdown tools seem too heavyweight for me, +and too much like learning a new domain specific language which defeats the +purpose of using markdown. + +I started tangling [the shell](https://github.com/driusan/dsh) that I was writing +to experiment with literate programming using copy and paste. It works, but is +cumbersome. This is a tool to automate that process. + +It's written in Go, because the Go tooling (notably `go fmt`) lends itself well +to writing in this paradigm. + +## Syntax + +To be useful for literate programming code blocks need a few features that don't +exist in standard markdown: + +1. The ability to embed macros, which will get expanded upon tangle. +2. The ability to denote code blocks as the macro to be expanded when referenced. +3. The ability to either append to or replace code blocks/macros, so that we can + expand on our train of thought incrementally. +4. The ability to redirect a code block into a file (while expanding macros.) + +Since markdown codeblocks will already let you specify the language of the block +for syntax highlighting purposes by naming the language after the three backticks, +my first thought was to put the file/codeblock name on the same line, after the +language name. + +For a convention, we'll say that a string with quotations denotes the name of a +code block, and a string without quotations denotes a filename to put the code +block into. If a code block header ends in `+=` it'll mean "append to the named +code block", otherwise it'll mean "create or replace the existing code block." +We'll use a line inside of a code block containing nothing but a title inside +`<<<` and `>>>` (with optional whitespace) as a macro to expand, because it's a +convention that's unlikely to be used otherwise inside of source code in any +language. + +### Implementation/Example. + +The above paragraph fully defines our spec. So, an example of a file code block +might look like this: + +```go main.go +package main + +import ( + <<>> +) + +<<>> + +<<>> + +func main() { + <<
>> +} +``` + +For our implementation, we'll need to parse the markdown file (which file? We'll +use the arguments from the command line) one line at a time, starting from the +top to ensure we replace code blocks in the right order. If there are multiple +files, we'll process them in the order they were passed on the command line. + +For now, we don't need to process any command line arguments, we'll just assume +everything passed is a file. + +So an example of a named code block is like this: + +```go "main implementation" +files := os.Args +for _, file := range files { + <<>> +} +``` + +How do we process a file? We'll need to keep 2 maps: one for named macros, and +one for file output content. We won't do any expansion until all the files have +been processed, because a block might refer to another block that either hasn't +been defined yet, or later has its definition changed. Let's define our maps, +define a stub of a `process file` function, and redefine our `main implementation` +to take that into account. + +Our maps, with some types defined for good measure: + +```go "global variables" +<<>> +``` + +```go "global block variables" +type File string +type CodeBlock string +type BlockName string + +var blocks map[BlockName]CodeBlock +var files map[File]CodeBlock +``` + +Our ProcessFile function: + +```go "other functions" +<<>> +``` + +```go "ProcessFile Declaration" +// Updates the blocks and files map for the markdown read from r. +func ProcessFile(r io.Reader) error { + <<>> +} +``` + +And our new main: + +```go "main implementation" +<<>> + +// os.Args[0] is the command name, "lmt". We don't want to process it. +for _, file := range os.Args[1:] { + <<>> + +} +<<>> +``` + +We used a few packages, so let's import them before declaring the blocks we +just used. + +```go "main.go imports" +"fmt" +"os" +"io" +``` + +Initializing the maps is pretty straight forward: + +```go "Initialize" +// Initialize the maps +blocks = make(map[BlockName]CodeBlock) +files = make(map[File]CodeBlock) +``` + +As is opening the files, since we already declared the ProcessFile function and +we just need to open the file to turn it into an `io.Reader`: + +```go "Open and process file" +f, err := os.Open(file) +if err != nil { + fmt.Fprintln(os.Stderr, "error: ", err) + continue +} + +if err := ProcessFile(f); err != nil { + fmt.Fprintln(os.Stderr, "error: ", err) +} +// Don't defer since we're in a loop, we don't want to wait until the function +// exits. +f.Close() +``` + +### Processing Files + +Now that we've got the obvious overhead out of the way, we need to begin +implementing the code which parses a file. + +We'll start by scanning each line. The Go `bufio` package has a Reader which +has a `ReadString` method that will stop at a delimiter (in our case, '\n') + +We can do use this bufio Reader to iterate through lines like so: + +```go "process file implementation" +scanner := bufio.NewReader(r) +var err error +var line string +for { + line, err = scanner.ReadString('\n') + switch err { + case io.EOF: + return nil + case nil: + // Nothing special + default: + return err + } + <<>> + +} +``` + +We'll need to import the `bufio` package which we just used too: + +```go "main.go imports" += +"bufio" +``` + +How do we handle a line? We'll need to keep track of a little state: + +1. Are we in a code block? +2. If so, what name or file is it for? +3. Are we ending a code block? If so, update the map (either replace or append.) + +So let's add a little state to our implementation: + +```go "process file implementation" +scanner := bufio.NewReader(r) +var err error +var line string + +var inBlock, appending bool +var bname BlockName +var fname File +var block CodeBlock + +for { + line, err = scanner.ReadString('\n') + switch err { + case io.EOF: + return nil + case nil: + // Nothing special + default: + return err + } + <<>> +} +``` + +We'll replace all of the variables with their zero value when we're not in a +block. + +The flow of handling a line will be something like: + +```go "Handle file line" +if inBlock { + if line == "```\n" { + <<>> + continue + } else { + <<>> + } +} else { + <<>> +} +``` + +Handling a code block line is easy, we just add it to the `block` if it's not +a block ending, and update the map/reset all the variables if it is. + +```go "Handle block line" +block += CodeBlock(line) +``` + +```go "Handle block ending" +// Update the files map if it's a file. +if fname != "" { + if appending { + files[fname] += block + } else { + files[fname] = block + } +} + +// Update the named block map if it's a named block. +if bname != "" { + if appending { + blocks[bname] += block + } else { + blocks[bname] = block + } +} + +<<>> +``` + +```go "Reset block flags" +inBlock = false +appending = false +bname = "" +fname = "" +block = "" +``` + +#### Processing Non-Block lines + +Processing non-block lines is easy, and we don't have to do anything since we +are only concerned with code blocks. +we don't need to care and can just reset the flags. +Otherwise, for triple backticks, we can just check the first three characters +of the line (we don't care if there's a language specified or not). + +```go "Handle nonblock line" +if line == "" { + continue +} + +switch line[0] { +case '`': + <<>> +default: + <<>> +} +``` + +When a code block is reached we will need to reset the flags and parse the line +for the following information: + + - a filename + - a block name/label + - an append flag + +```go "Check block start" +if len(line) >= 3 && line[0:3] == "```" { + inBlock = true + <<>> +} +``` + +#### Parsing Headers With a Regex + +Parsing headers is a little more difficult, but shouldn't be too hard with +a regular expression. There's four potential components: + + 1. 3 or more '`' characters. We don't care how many there are. + 2. 0 or more non-whitespace characters, which will may be the language type. + 3. 0 or more alphanumeric characters, which can be a file name. + 4. 0 or 1 string enclosed in quotation marks. + 5. It may or may not end in `+=`. + +So the regex will look something like ```/^(`+)([a-zA-Z0-9\.]*)("[.*]"){0,1}(+=){0,1}$/``` +(there are more characters that might be in a file name, but to keep the regex simple +we'll just assume letters, numbers, and dots.) + +That regex is already starting to look hairy, so instead let's split it up into +two: one for checking if it's a named block, and if that fails one for checking +if it's a file name. It means we can't have a block which is *both* a named +block and *also* goes into a filename, but that's probably not a very useful +case and can always be done with two blocks (one named, and a file which only +contains a macro expanding to the named block.) + +In fact, we'll put the whole thing into a function to make it easier to debug +and write tests if we want to. + +```go "Check block header" +fname, bname, appending = parseHeader(line) +// We're outside of a block, so just blindly reset it. +block = "" +``` + +Then we need to define our parseHeader function: + +```go "other functions" += +<<>> +``` + +```go "ParseHeader Declaration" +func parseHeader(line string) (File, BlockName, bool) { + line = strings.TrimSpace(line) + <<>> +} +``` + +Our implementation is going to use a regex for a namedBlock, and compare the +line against it, so let's start by importing the regex package. + +```go "main.go imports" += +"regexp" +``` + +```go "parseHeader implementation" +namedBlockRe := regexp.MustCompile("^([`]+\\s?)[\\w\\+]*[\\s]*\"(.+)\"[\\s]*([+][=])?$") +matches := namedBlockRe.FindStringSubmatch(line) +if matches != nil { + return "", BlockName(matches[2]), (matches[3] == "+=") +} +<<>> +return "", "", false +``` + +There's no reason to constantly be re-compiling the namedBlockRe, we can just +make it global and compile it once on initialization. + +```go "global variables" += +var namedBlockRe *regexp.Regexp +``` + +```go "Initialize" += +<<>> +``` + +```go "Namedblock Regex" +namedBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+\"(.+)\"[\\s]*([+][=])?$") +``` + +Then our parse implementation without the MustCompile is: + +```go "parseHeader implementation" +matches := namedBlockRe.FindStringSubmatch(line) +if matches != nil { + return "", BlockName(matches[2]), (matches[3] == "+=") +} +<<>> +return "", "", false +``` + +Checking a filename header is fairly simple: just make sure there's alphanumeric +characters or dots and no spaces. If it's neither, we can just return the zero +value, since the header must immediately preceed the code block according to our +specification. + +This time, we'll just go straight to declaring the regex as a global. + +```go "global variables" += +var fileBlockRe *regexp.Regexp +``` + +```go "Initialize" += +<<>> +``` + +```go "Fileblock Regex" +fileBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+([\\w\\.\\-\\/]+)[\\s]*([+][=])?$") +``` + +```go "Check filename header" +matches = fileBlockRe.FindStringSubmatch(line) +if matches != nil { + return File(matches[2]), "", (matches[3] == "+=") +} +``` + +### Outputting The Files + +Now, we've finally finished processing the file, all that remains is going through +the output files that were declared, expanding the macros, and writing them to +disk. Since our files is a `map[File]CodeBlock`, we can define methods on +`CodeBlock` as needed for things like expanding the macros. + +Let's start by just ranging through our files map, and assuming there's a method +on code block which does the replacing. + +```go "Output files" +for filename, codeblock := range files { + f, err := os.Create(string(filename)) + if err != nil { + fmt.Fprintf(os.Stderr, "%v\n", err) + continue + } + fmt.Fprintf(f, "%s", codeblock.Replace()) + // We don't defer this so that it'll get closed before the loop finishes. + f.Close() + +} +``` + +Now, we'll have to declare the Replace() method that we just used. The Replace() +will take a codeblock, go through it line by line, check if the current line is +a macro, and if so replace the content (recursively). We can use another regex +to determine if it's a macro line, and we can use a scanner similar to our +markdown line scanner to our previous one, + +```go "other functions" += +<<>> +``` + +```go "Replace Declaration" +// Replace expands all macros in a CodeBlock and returns a CodeBlock with no +// references to macros. +func (c CodeBlock) Replace() (ret CodeBlock) { + <<>> +} +``` + +```go "Replace codeblock implementation" +scanner := bufio.NewReader(strings.NewReader(string(c))) + +for { + line, err := scanner.ReadString('\n') + // ReadString will eventually return io.EOF and this will return. + if err != nil { + return + } + <<>> +} +return +``` + +We'll have to import the strings package we just used to convert our CodeBlock +into an io.Reader: + +```go "main.go imports" += +"strings" +``` + +Now, our replacement regex should be fairly simple: + +```go "global variables" += +var replaceRe *regexp.Regexp +``` + +```go "Initialize" += +<<>> +``` + +```go "Replace Regex" +replaceRe = regexp.MustCompile(`^[\s]*<<<(.+)>>>[\s]*$`) +``` + +Okay, so let's do the actual line handling. If it doesn't match, add it to `ret` +and go on to the next line. If it matches, look up the part that matched in +blocks and include the replaced CodeBlock from there. (If it doesn't exist, +we'll add the line unexpanded and print a warning.) + +```go "Handle replace line" +matches := replaceRe.FindStringSubmatch(line) +if matches == nil { + ret += CodeBlock(line) + continue +} +<<>> +``` + +Looking up a replacement is fairly straight forward, since we have a map by the +time this is called. + +```go "Lookup replacement and add to ret" +bname := BlockName(matches[1]) +if val, ok := blocks[bname]; ok { + ret += val.Replace() +} else { + fmt.Fprintf(os.Stderr, "Warning: Block named %s referenced but not defined.\n", bname) + ret += CodeBlock(line) +} +``` + +## Fin + +And now, our tool is finally done! We've finally implemented our `lmt` tool tangle +tool, and can use it to write other literate markdown style programs with the +same syntax. + +The output of running it on itself (included [patches](#patches) and then running `go fmt`) +is in this repo to make it a go-gettable executable for bootstrapping purposes. + +To use it after installing it just run, for example + +```shell +lmt README.md WhitespacePreservation.md SubdirectoryFiles.md LineNumbers.md IndentedBlocks.md +``` + +## Patches + + 1. [Whitespace Preservation](WhitespacePreservation.md) + 2. [Subdirectory Files](SubdirectoryFiles.md) + 3. [Line Numbers](LineNumbers.md) + 3. [Indented Blocks](IndentedBlocks.md) diff --git a/README.md b/README.md index 15aaffe..b76ab0f 100644 --- a/README.md +++ b/README.md @@ -1,574 +1,158 @@ -# lmt - literate markdown tangle +# Literate Markdown For C++ Programmers -This README describes a tangle program for a literate programming style -where the code is weaved into markdown code blocks. There is no corresponding -weave, because the markdown itself can already be read as the documentation, -either through a text editor or through an online system that already renders -markdown such as GitHub. +[lmt](https://github.com/driusan/lmt) is a tool for extracting text from the +code blocks in markdown files. This file demonstrates all the lmt features in a +C++-centric way. -## Why? +## Installing lmt -[Literate programming](https://en.wikipedia.org/wiki/Literate_programming) is a -style of programming where, instead of directly writing source code, the programmer -writes their reasoning in human prose, and intersperses fragments of code which -can be extracted into the compilable source code with one tool (called "tangle"), -and conversely can be converted to a human readable document explaining the code -with another (called "weave"). +First, install the go language if you don't have it (homebrew: `brew install +go`). -[Markdown](http://daringfireball.net/projects/markdown/syntax) is a plaintextish -format popular with programmers. It's simple, easy and already has support -for embedding code blocks using triple backticks (```), mostly for the purposes -of syntax highlighting in documentation. +Then build the tool. The following assumes you have a `~/bin` directory in your +`PATH`: -The existing literate programming for markdown tools seem too heavyweight for me, -and too much like learning a new domain specific language which defeats the -purpose of using markdown. - -I started tangling [the shell](https://github.com/driusan/dsh) that I was writing -to experiment with literate programming using copy and paste. It works, but is -cumbersome. This is a tool to automate that process. - -It's written in Go, because the Go tooling (notably `go fmt`) lends itself well -to writing in this paradigm. - -## Syntax - -To be useful for literate programming code blocks need a few features that don't -exist in standard markdown: - -1. The ability to embed macros, which will get expanded upon tangle. -2. The ability to denote code blocks as the macro to be expanded when referenced. -3. The ability to either append to or replace code blocks/macros, so that we can - expand on our train of thought incrementally. -4. The ability to redirect a code block into a file (while expanding macros.) - -Since markdown codeblocks will already let you specify the language of the block -for syntax highlighting purposes by naming the language after the three backticks, -my first thought was to put the file/codeblock name on the same line, after the -language name. - -For a convention, we'll say that a string with quotations denotes the name of a -code block, and a string without quotations denotes a filename to put the code -block into. If a code block header ends in `+=` it'll mean "append to the named -code block", otherwise it'll mean "create or replace the existing code block." -We'll use a line inside of a code block containing nothing but a title inside -`<<<` and `>>>` (with optional whitespace) as a macro to expand, because it's a -convention that's unlikely to be used otherwise inside of source code in any -language. - -### Implementation/Example. - -The above paragraph fully defines our spec. So, an example of a file code block -might look like this: - -```go main.go -package main - -import ( - <<>> -) - -<<>> - -<<>> - -func main() { - <<
>> -} +```bash +git clone https://github.com/driusan/lmt +cd lmt +go build -o ~/bin ``` -For our implementation, we'll need to parse the markdown file (which file? We'll -use the arguments from the command line) one line at a time, starting from the -top to ensure we replace code blocks in the right order. If there are multiple -files, we'll process them in the order they were passed on the command line. - -For now, we don't need to process any command line arguments, we'll just assume -everything passed is a file. - -So an example of a named code block is like this: - -```go "main implementation" -files := os.Args -for _, file := range files { - <<>> -} -``` - -How do we process a file? We'll need to keep 2 maps: one for named macros, and -one for file output content. We won't do any expansion until all the files have -been processed, because a block might refer to another block that either hasn't -been defined yet, or later has its definition changed. Let's define our maps, -define a stub of a `process file` function, and redefine our `main implementation` -to take that into account. - -Our maps, with some types defined for good measure: - -```go "global variables" -<<>> -``` - -```go "global block variables" -type File string -type CodeBlock string -type BlockName string - -var blocks map[BlockName]CodeBlock -var files map[File]CodeBlock -``` - -Our ProcessFile function: - -```go "other functions" -<<>> -``` - -```go "ProcessFile Declaration" -// Updates the blocks and files map for the markdown read from r. -func ProcessFile(r io.Reader) error { - <<>> -} -``` +## Demo -And our new main: +To observe `lmt` at work, put this file in an empty directory, cd to that +directory, and `lmt Literate.md`. Now look in the directory and you'll see +extracted files extracted from the code blocks alongside this markdown file. In +literate programming lingo, this extraction is (somewhat counterintuitively) +called “tangling.” -```go "main implementation" -<<>> +### Tangling into a file. -// os.Args[0] is the command name, "lmt". We don't want to process it. -for _, file := range os.Args[1:] { - <<>> +The markup for the code block below starts with `​```cpp hello.cpp +=`: + +```cpp hello.cpp += +<<>> +<<>> +int main() { + <<>> } -<<>> ``` -We used a few packages, so let's import them before declaring the blocks we -just used. +The header says 3 things: -```go "main.go imports" -"fmt" -"os" -"io" -``` +1. `cpp`: the code block is written in C++. In the rendered markdown output, that + affects syntax highlighting, to lmt it means that language-appropriate + `#line` directives will be added so that when debugging the extracted code, + your debugger will show you the line in the original source markdown file. + (If you don't want this effect, just use an unrecognized language name like + `cxx`). +2. `hello.cpp`: The code block will be written to the file `hello.cpp`. +3. `+=`: The code block will be appended to that file, rather than overwriting + its content. Since we haven't written anything to `hello.cpp` yet, the + effect is the same, but since overwriting the code you've already extracted + is kind of a nice case, to enable developing examples like those in `lmt`'s + own source, you might want to use `+=` by default. + -Initializing the maps is pretty straight forward: +### Macro References -```go "Initialize" -// Initialize the maps -blocks = make(map[BlockName]CodeBlock) -files = make(map[File]CodeBlock) -``` +The `<<<`*string*`>>>` sequences in the body of the code block are called +“macro references.” An LMT “macro” is just a variable whose value can be extracted +from one or more code blocks, and will be substituted wherever its name appears +in triple angle brackets. -As is opening the files, since we already declared the ProcessFile function and -we just need to open the file to turn it into an `io.Reader`: +### Macro Content -```go "Open and process file" -f, err := os.Open(file) -if err != nil { - fmt.Fprintln(os.Stderr, "error: ", err) - continue -} +The markup for the code block below starts with `​```cpp "body of main"` -if err := ProcessFile(f); err != nil { - fmt.Fprintln(os.Stderr, "error: ", err) -} -// Don't defer since we're in a loop, we don't want to wait until the function -// exits. -f.Close() +```cpp "body of main" +std::cout << "Hello, werld!" << std::endl; ``` -### Processing Files - -Now that we've got the obvious overhead out of the way, we need to begin -implementing the code which parses a file. +The double quotes around `body of main` mean that the code block will be +extracted into a macro of that name. You can see where its value will be +injected into hello.cpp via `<<>>`, +[above](#tangling-into-a-file). Since there's no `+=` at the end of the block's +first line of markup, this code block overwrites any existing value the macro +might already have (but since it has no existing value, it's a wash). -We'll start by scanning each line. The Go `bufio` package has a Reader which -has a `ReadString` method that will stop at a delimiter (in our case, '\n') +### Appending To A Macro -We can do use this bufio Reader to iterate through lines like so: +We can use `#include`s to demonstrate `+=` on macros. There are two includes in +this program. The markup for the following block starts with `​```cpp +"includes"`, which causes the (empty) value of the `includes` macro to be +overwritten. -```go "process file implementation" -scanner := bufio.NewReader(r) -var err error -var line string -for { - line, err = scanner.ReadString('\n') - switch err { - case io.EOF: - return nil - case nil: - // Nothing special - default: - return err - } - <<>> - -} +```cpp "includes" +#include ``` -We'll need to import the `bufio` package which we just used too: +The markup for the next code block, however, starts with `​```cpp "includes" +=`, +which causes the block to be appended to the `includes` macro. -```go "main.go imports" += -"bufio" +```cpp "includes" += +#include ``` -How do we handle a line? We'll need to keep track of a little state: - -1. Are we in a code block? -2. If so, what name or file is it for? -3. Are we ending a code block? If so, update the map (either replace or append.) - -So let's add a little state to our implementation: - -```go "process file implementation" -scanner := bufio.NewReader(r) -var err error -var line string - -var inBlock, appending bool -var bname BlockName -var fname File -var block CodeBlock - -for { - line, err = scanner.ReadString('\n') - switch err { - case io.EOF: - return nil - case nil: - // Nothing special - default: - return err - } - <<>> -} -``` +Its value is now: -We'll replace all of the variables with their zero value when we're not in a -block. - -The flow of handling a line will be something like: - -```go "Handle file line" -if inBlock { - if line == "```\n" { - <<>> - continue - } else { - <<>> - } -} else { - <<>> -} +```cpp +#include +#include ``` -Handling a code block line is easy, we just add it to the `block` if it's not -a block ending, and update the map/reset all the variables if it is. +(the code block above is not being tangled). -```go "Handle block line" -block += CodeBlock(line) -``` +### Hidden content. -```go "Handle block ending" -// Update the files map if it's a file. -if fname != "" { - if appending { - files[fname] += block - } else { - files[fname] = block - } -} +The raw markdown in this file contains a comment containing a code block with a +copyright notice. It looks a bit like this one: -// Update the named block map if it's a named block. -if bname != "" { - if appending { - blocks[bname] += block - } else { - blocks[bname] = block - } -} + -<<>> -``` +If you're reading the rendered markdown in your browser, you can't see the +*actual* comment, but it still gets tangled into the `copyright` macro, which is +substituted into hello.cpp by the `<<>>` macro reference. This +technique lets you tangle content that you don't want showing up in the +documentation. -```go "Reset block flags" -inBlock = false -appending = false -bname = "" -fname = "" -block = "" + -#### Processing Non-Block lines +### What Tangles and What Doesn't. -Processing non-block lines is easy, and we don't have to do anything since we -are only concerned with code blocks. -we don't need to care and can just reset the flags. -Otherwise, for triple backticks, we can just check the first three characters -of the line (we don't care if there's a language specified or not). +We can tangle into a random data file (`​```csv data.csv`) -```go "Handle nonblock line" -if line == "" { - continue -} -switch line[0] { -case '`': - <<>> -default: - <<>> -} +```csv data.csv +foo, bar, baz, +qix, qux, quux, ``` -When a code block is reached we will need to reset the flags and parse the line -for the following information: - - - a filename - - a block name/label - - an append flag +You need to specify both a file type and a destination (macro or file)if you +want the code block tangled: -```go "Check block start" -if len(line) >= 3 && line[0:3] == "```" { - inBlock = true - <<>> -} -``` - -#### Parsing Headers With a Regex - -Parsing headers is a little more difficult, but shouldn't be too hard with -a regular expression. There's four potential components: - - 1. 3 or more '`' characters. We don't care how many there are. - 2. 0 or more non-whitespace characters, which will may be the language type. - 3. 0 or more alphanumeric characters, which can be a file name. - 4. 0 or 1 string enclosed in quotation marks. - 5. It may or may not end in `+=`. - -So the regex will look something like ```/^(`+)([a-zA-Z0-9\.]*)("[.*]"){0,1}(+=){0,1}$/``` -(there are more characters that might be in a file name, but to keep the regex simple -we'll just assume letters, numbers, and dots.) - -That regex is already starting to look hairy, so instead let's split it up into -two: one for checking if it's a named block, and if that fails one for checking -if it's a file name. It means we can't have a block which is *both* a named -block and *also* goes into a filename, but that's probably not a very useful -case and can always be done with two blocks (one named, and a file which only -contains a macro expanding to the named block.) - -In fact, we'll put the whole thing into a function to make it easier to debug -and write tests if we want to. - -```go "Check block header" -fname, bname, appending = parseHeader(line) -// We're outside of a block, so just blindly reset it. -block = "" -``` - -Then we need to define our parseHeader function: - -```go "other functions" += -<<>> +No file type (`​``` bar.txt`—note the space): +``` bar.txt +This doesn't get tangled anywhere ``` -```go "ParseHeader Declaration" -func parseHeader(line string) (File, BlockName, bool) { - line = strings.TrimSpace(line) - <<>> -} +No destination (`​```cpp`): +```cpp +auto x = "nor does this"; ``` -Our implementation is going to use a regex for a namedBlock, and compare the -line against it, so let's start by importing the regex package. - -```go "main.go imports" += -"regexp" -``` +But any file type string and filename (`​```arbitrary foo.txt`) will do -```go "parseHeader implementation" -namedBlockRe := regexp.MustCompile("^([`]+\\s?)[\\w\\+]*[\\s]*\"(.+)\"[\\s]*([+][=])?$") -matches := namedBlockRe.FindStringSubmatch(line) -if matches != nil { - return "", BlockName(matches[2]), (matches[3] == "+=") -} -<<>> -return "", "", false +```arbitrary foo.txt +This gets tangled +into foo.txt. ``` - -There's no reason to constantly be re-compiling the namedBlockRe, we can just -make it global and compile it once on initialization. - -```go "global variables" += -var namedBlockRe *regexp.Regexp -``` - -```go "Initialize" += -<<>> -``` - -```go "Namedblock Regex" -namedBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+\"(.+)\"[\\s]*([+][=])?$") -``` - -Then our parse implementation without the MustCompile is: - -```go "parseHeader implementation" -matches := namedBlockRe.FindStringSubmatch(line) -if matches != nil { - return "", BlockName(matches[2]), (matches[3] == "+=") -} -<<>> -return "", "", false -``` - -Checking a filename header is fairly simple: just make sure there's alphanumeric -characters or dots and no spaces. If it's neither, we can just return the zero -value, since the header must immediately preceed the code block according to our -specification. - -This time, we'll just go straight to declaring the regex as a global. - -```go "global variables" += -var fileBlockRe *regexp.Regexp -``` - -```go "Initialize" += -<<>> -``` - -```go "Fileblock Regex" -fileBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+([\\w\\.\\-\\/]+)[\\s]*([+][=])?$") -``` - -```go "Check filename header" -matches = fileBlockRe.FindStringSubmatch(line) -if matches != nil { - return File(matches[2]), "", (matches[3] == "+=") -} -``` - -### Outputting The Files - -Now, we've finally finished processing the file, all that remains is going through -the output files that were declared, expanding the macros, and writing them to -disk. Since our files is a `map[File]CodeBlock`, we can define methods on -`CodeBlock` as needed for things like expanding the macros. - -Let's start by just ranging through our files map, and assuming there's a method -on code block which does the replacing. - -```go "Output files" -for filename, codeblock := range files { - f, err := os.Create(string(filename)) - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - continue - } - fmt.Fprintf(f, "%s", codeblock.Replace()) - // We don't defer this so that it'll get closed before the loop finishes. - f.Close() - -} -``` - -Now, we'll have to declare the Replace() method that we just used. The Replace() -will take a codeblock, go through it line by line, check if the current line is -a macro, and if so replace the content (recursively). We can use another regex -to determine if it's a macro line, and we can use a scanner similar to our -markdown line scanner to our previous one, - -```go "other functions" += -<<>> -``` - -```go "Replace Declaration" -// Replace expands all macros in a CodeBlock and returns a CodeBlock with no -// references to macros. -func (c CodeBlock) Replace() (ret CodeBlock) { - <<>> -} -``` - -```go "Replace codeblock implementation" -scanner := bufio.NewReader(strings.NewReader(string(c))) - -for { - line, err := scanner.ReadString('\n') - // ReadString will eventually return io.EOF and this will return. - if err != nil { - return - } - <<>> -} -return -``` - -We'll have to import the strings package we just used to convert our CodeBlock -into an io.Reader: - -```go "main.go imports" += -"strings" -``` - -Now, our replacement regex should be fairly simple: - -```go "global variables" += -var replaceRe *regexp.Regexp -``` - -```go "Initialize" += -<<>> -``` - -```go "Replace Regex" -replaceRe = regexp.MustCompile(`^[\s]*<<<(.+)>>>[\s]*$`) -``` - -Okay, so let's do the actual line handling. If it doesn't match, add it to `ret` -and go on to the next line. If it matches, look up the part that matched in -blocks and include the replaced CodeBlock from there. (If it doesn't exist, -we'll add the line unexpanded and print a warning.) - -```go "Handle replace line" -matches := replaceRe.FindStringSubmatch(line) -if matches == nil { - ret += CodeBlock(line) - continue -} -<<>> -``` - -Looking up a replacement is fairly straight forward, since we have a map by the -time this is called. - -```go "Lookup replacement and add to ret" -bname := BlockName(matches[1]) -if val, ok := blocks[bname]; ok { - ret += val.Replace() -} else { - fmt.Fprintf(os.Stderr, "Warning: Block named %s referenced but not defined.\n", bname) - ret += CodeBlock(line) -} -``` - -## Fin - -And now, our tool is finally done! We've finally implemented our `lmt` tool tangle -tool, and can use it to write other literate markdown style programs with the -same syntax. - -The output of running it on itself (included [patches](#patches) and then running `go fmt`) -is in this repo to make it a go-gettable executable for bootstrapping purposes. - -To use it after installing it just run, for example - -```shell -lmt README.md WhitespacePreservation.md SubdirectoryFiles.md LineNumbers.md IndentedBlocks.md -``` - -## Patches - - 1. [Whitespace Preservation](WhitespacePreservation.md) - 2. [Subdirectory Files](SubdirectoryFiles.md) - 3. [Line Numbers](LineNumbers.md) - 3. [Indented Blocks](IndentedBlocks.md) From a8d3bebf01dbfec87b06cefbf4e8b86d57f84fd8 Mon Sep 17 00:00:00 2001 From: Dave MacFarlane Date: Thu, 4 Mar 2021 14:08:19 -0500 Subject: [PATCH 2/3] Edit README.md and Implementation --- IMPLEMENTATION.md => Implementation.md | 125 +++++++++++++++---------- README.md | 123 ++++++++++++++++++------ 2 files changed, 170 insertions(+), 78 deletions(-) rename IMPLEMENTATION.md => Implementation.md (78%) diff --git a/IMPLEMENTATION.md b/Implementation.md similarity index 78% rename from IMPLEMENTATION.md rename to Implementation.md index 15aaffe..6831ac4 100644 --- a/IMPLEMENTATION.md +++ b/Implementation.md @@ -1,6 +1,6 @@ # lmt - literate markdown tangle -This README describes a tangle program for a literate programming style +This file implements a tangle program for a literate programming style where the code is weaved into markdown code blocks. There is no corresponding weave, because the markdown itself can already be read as the documentation, either through a text editor or through an online system that already renders @@ -16,7 +16,7 @@ and conversely can be converted to a human readable document explaining the code with another (called "weave"). [Markdown](http://daringfireball.net/projects/markdown/syntax) is a plaintextish -format popular with programmers. It's simple, easy and already has support +format popular with programmers. It's simple, easy to write and already has support for embedding code blocks using triple backticks (```), mostly for the purposes of syntax highlighting in documentation. @@ -24,11 +24,7 @@ The existing literate programming for markdown tools seem too heavyweight for me and too much like learning a new domain specific language which defeats the purpose of using markdown. -I started tangling [the shell](https://github.com/driusan/dsh) that I was writing -to experiment with literate programming using copy and paste. It works, but is -cumbersome. This is a tool to automate that process. - -It's written in Go, because the Go tooling (notably `go fmt`) lends itself well +The tool is written in Go, because the Go tooling (notably `go fmt`) lends itself well to writing in this paradigm. ## Syntax @@ -43,9 +39,8 @@ exist in standard markdown: 4. The ability to redirect a code block into a file (while expanding macros.) Since markdown codeblocks will already let you specify the language of the block -for syntax highlighting purposes by naming the language after the three backticks, -my first thought was to put the file/codeblock name on the same line, after the -language name. +for syntax highlighting purposes by naming the language after the three backticks, we can extend that by adding the file/codeblock name on the same line, after +the language name. For a convention, we'll say that a string with quotations denotes the name of a code block, and a string without quotations denotes a filename to put the code @@ -77,10 +72,11 @@ func main() { } ``` -For our implementation, we'll need to parse the markdown file (which file? We'll -use the arguments from the command line) one line at a time, starting from the -top to ensure we replace code blocks in the right order. If there are multiple -files, we'll process them in the order they were passed on the command line. +For our implementation, we'll need to parse the markdown file one line at a +time, starting from the top to ensure we replace code blocks in the right +order (which file? We'll use the arguments from the command line). If there +are multiple files, we'll process them in the order they were passed on the +command line. For now, we don't need to process any command line arguments, we'll just assume everything passed is a file. @@ -98,15 +94,20 @@ How do we process a file? We'll need to keep 2 maps: one for named macros, and one for file output content. We won't do any expansion until all the files have been processed, because a block might refer to another block that either hasn't been defined yet, or later has its definition changed. Let's define our maps, -define a stub of a `process file` function, and redefine our `main implementation` -to take that into account. +define a stub of a `process file` function, and redefine our +`main implementation` to take that into account. + +Let's define our maps, with some types defined for good measure: -Our maps, with some types defined for good measure: +(Note: we start by adding a new macro reference to our "global variables" macro so that it can be redefined in further patches without overwriting other +content that was appended.) ```go "global variables" <<>> ``` +Let's really define the `"global block variables"` types and maps, now: + ```go "global block variables" type File string type CodeBlock string @@ -116,12 +117,17 @@ var blocks map[BlockName]CodeBlock var files map[File]CodeBlock ``` -Our ProcessFile function: +We'll similarly add a `"ProcessFile Declaration"` macro to our +`"other functions"` macro, so that it can be redefined in later +patches. ```go "other functions" <<>> ``` +And then define the function prototype, leaving the implementation +to a macro for now. + ```go "ProcessFile Declaration" // Updates the blocks and files map for the markdown read from r. func ProcessFile(r io.Reader) error { @@ -129,7 +135,9 @@ func ProcessFile(r io.Reader) error { } ``` -And our new main: +Our main function, recall, is going to initialize the program, +process each command line argument in order, and then output +files. ```go "main implementation" <<>> @@ -142,8 +150,8 @@ for _, file := range os.Args[1:] { <<>> ``` -We used a few packages, so let's import them before declaring the blocks we -just used. +We used a few standard library packages, so let's import them before +declaring the blocks we just used. ```go "main.go imports" "fmt" @@ -151,7 +159,8 @@ just used. "io" ``` -Initializing the maps is pretty straight forward: +Initializing the maps is pretty straight forward (note: the source names the +following block `"Initialize"`): ```go "Initialize" // Initialize the maps @@ -159,8 +168,9 @@ blocks = make(map[BlockName]CodeBlock) files = make(map[File]CodeBlock) ``` -As is opening the files, since we already declared the ProcessFile function and -we just need to open the file to turn it into an `io.Reader`: +Opening and processing files is fairly straight forward as well, since we +already declared the ProcessFile function and we just need to open the +file to turn it into an `io.Reader`: ```go "Open and process file" f, err := os.Open(file) @@ -185,7 +195,7 @@ implementing the code which parses a file. We'll start by scanning each line. The Go `bufio` package has a Reader which has a `ReadString` method that will stop at a delimiter (in our case, '\n') -We can do use this bufio Reader to iterate through lines like so: +We can use this bufio Reader to iterate through lines like so: ```go "process file implementation" scanner := bufio.NewReader(r) @@ -206,7 +216,8 @@ for { } ``` -We'll need to import the `bufio` package which we just used too: +We'll need to import the `bufio` package which we just used too, by +appending it to `"main.go imports"`: ```go "main.go imports" += "bufio" @@ -265,10 +276,12 @@ if inBlock { Handling a code block line is easy, we just add it to the `block` if it's not a block ending, and update the map/reset all the variables if it is. +`"Handle Block line`": ```go "Handle block line" block += CodeBlock(line) ``` +`"Handle block ending"`: ```go "Handle block ending" // Update the files map if it's a file. if fname != "" { @@ -291,6 +304,10 @@ if bname != "" { <<>> ``` +Since we've used a `"Reset block flags"` macro, we need to define +it. We said we were going to reset our state variables to their zero +value. + ```go "Reset block flags" inBlock = false appending = false @@ -303,7 +320,8 @@ block = "" Processing non-block lines is easy, and we don't have to do anything since we are only concerned with code blocks. -we don't need to care and can just reset the flags. + +We don't need to care and can just reset the flags. Otherwise, for triple backticks, we can just check the first three characters of the line (we don't care if there's a language specified or not). @@ -327,6 +345,7 @@ for the following information: - a block name/label - an append flag +`"Check block start"`: ```go "Check block start" if len(line) >= 3 && line[0:3] == "```" { inBlock = true @@ -340,10 +359,10 @@ Parsing headers is a little more difficult, but shouldn't be too hard with a regular expression. There's four potential components: 1. 3 or more '`' characters. We don't care how many there are. - 2. 0 or more non-whitespace characters, which will may be the language type. + 2. 0 or more non-whitespace characters, which will be the language type. 3. 0 or more alphanumeric characters, which can be a file name. 4. 0 or 1 string enclosed in quotation marks. - 5. It may or may not end in `+=`. + 5. It may or may not end in the string literal `+=`. So the regex will look something like ```/^(`+)([a-zA-Z0-9\.]*)("[.*]"){0,1}(+=){0,1}$/``` (there are more characters that might be in a file name, but to keep the regex simple @@ -367,10 +386,12 @@ block = "" Then we need to define our parseHeader function: +`"other functions"`: ```go "other functions" += <<>> ``` +`"ParseHeader Declaration"`: ```go "ParseHeader Declaration" func parseHeader(line string) (File, BlockName, bool) { line = strings.TrimSpace(line) @@ -398,20 +419,24 @@ return "", "", false There's no reason to constantly be re-compiling the namedBlockRe, we can just make it global and compile it once on initialization. +`"global variables" +=`: ```go "global variables" += var namedBlockRe *regexp.Regexp ``` +`"Initialize" +=`: ```go "Initialize" += <<>> ``` +`"Named block Regex"`: ```go "Namedblock Regex" namedBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+\"(.+)\"[\\s]*([+][=])?$") ``` Then our parse implementation without the MustCompile is: +`"parseHeader implementation"`: ```go "parseHeader implementation" matches := namedBlockRe.FindStringSubmatch(line) if matches != nil { @@ -428,18 +453,22 @@ specification. This time, we'll just go straight to declaring the regex as a global. +`"global variables" +=`: ```go "global variables" += var fileBlockRe *regexp.Regexp ``` +`"Initialize" +=`: ```go "Initialize" += <<>> ``` +`"File block Regex"`: ```go "Fileblock Regex" fileBlockRe = regexp.MustCompile("^([`]+\\s?)[\\w\\+]+[\\s]+([\\w\\.\\-\\/]+)[\\s]*([+][=])?$") ``` +`"Check filename header"`: ```go "Check filename header" matches = fileBlockRe.FindStringSubmatch(line) if matches != nil { @@ -455,7 +484,7 @@ disk. Since our files is a `map[File]CodeBlock`, we can define methods on `CodeBlock` as needed for things like expanding the macros. Let's start by just ranging through our files map, and assuming there's a method -on code block which does the replacing. +on code block which does the replacing for `"Output files"`. ```go "Output files" for filename, codeblock := range files { @@ -472,15 +501,17 @@ for filename, codeblock := range files { ``` Now, we'll have to declare the Replace() method that we just used. The Replace() -will take a codeblock, go through it line by line, check if the current line is -a macro, and if so replace the content (recursively). We can use another regex -to determine if it's a macro line, and we can use a scanner similar to our -markdown line scanner to our previous one, +will operate on a codeblock, go through it line by line, check if the current +line is a macro, and if so replace the content (recursively). We can use +another regex to determine if it's a macro line, and we can use a scanner +similar to our markdown line scanner to our previous one, +`"other functions" +=`: ```go "other functions" += <<>> ``` +`"Replace Declaration"`: ```go "Replace Declaration" // Replace expands all macros in a CodeBlock and returns a CodeBlock with no // references to macros. @@ -489,6 +520,7 @@ func (c CodeBlock) Replace() (ret CodeBlock) { } ``` +`"Replace codeblock implementation"`, as described above: ```go "Replace codeblock implementation" scanner := bufio.NewReader(strings.NewReader(string(c))) @@ -512,21 +544,24 @@ into an io.Reader: Now, our replacement regex should be fairly simple: +`"global variables" +=`: ```go "global variables" += var replaceRe *regexp.Regexp ``` +`"Initialize" +=`: ```go "Initialize" += <<>> ``` +`"Replace Regex"`: ```go "Replace Regex" replaceRe = regexp.MustCompile(`^[\s]*<<<(.+)>>>[\s]*$`) ``` Okay, so let's do the actual line handling. If it doesn't match, add it to `ret` and go on to the next line. If it matches, look up the part that matched in -blocks and include the replaced CodeBlock from there. (If it doesn't exist, +`blocks` and include the replaced CodeBlock from there. (If it doesn't exist, we'll add the line unexpanded and print a warning.) ```go "Handle replace line" @@ -553,22 +588,12 @@ if val, ok := blocks[bname]; ok { ## Fin -And now, our tool is finally done! We've finally implemented our `lmt` tool tangle -tool, and can use it to write other literate markdown style programs with the +And now, our tool is finally done! We've implemented our `lmt` tangle tool, +and can use it to write other literate markdown style programs with the same syntax. -The output of running it on itself (included [patches](#patches) and then running `go fmt`) +The output of running it on itself (including patches) and then running `go fmt`) is in this repo to make it a go-gettable executable for bootstrapping purposes. -To use it after installing it just run, for example - -```shell -lmt README.md WhitespacePreservation.md SubdirectoryFiles.md LineNumbers.md IndentedBlocks.md -``` - -## Patches - - 1. [Whitespace Preservation](WhitespacePreservation.md) - 2. [Subdirectory Files](SubdirectoryFiles.md) - 3. [Line Numbers](LineNumbers.md) - 3. [Indented Blocks](IndentedBlocks.md) +If you're not familiar with `go get`, see the README for installation +instructions and a simple non-Go example. diff --git a/README.md b/README.md index b76ab0f..b2b1638 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,41 @@ -# Literate Markdown For C++ Programmers +# Literate Markdown Tangle [lmt](https://github.com/driusan/lmt) is a tool for extracting text from the -code blocks in markdown files. This file demonstrates all the lmt features in a -C++-centric way. +code blocks in markdown files. It allows programmers to write in a [literate programming](https://en.wikipedia.org/wiki/Literate_programming) style using +markdown as the source language. ## Installing lmt -First, install the go language if you don't have it (homebrew: `brew install -go`). +lmt is a self-contained Go program written using the LP paradigm. The source is committed alongside the markdown source of this repository for bootstrapping +purposes. -Then build the tool. The following assumes you have a `~/bin` directory in your -`PATH`: +You require the [Go language](https://golang.org/) if you don't already have it. + +To build the tool: ```bash git clone https://github.com/driusan/lmt cd lmt -go build -o ~/bin +go build ``` +This will build the binary named `lmt` for your platform in the current +directory. You can use the `-o $path` argument to `go build` to build +the binary in a different location. (i.e. `go build -o ~/bin/` to put the +binary in `~/bin/`.) + ## Demo To observe `lmt` at work, put this file in an empty directory, cd to that -directory, and `lmt Literate.md`. Now look in the directory and you'll see -extracted files extracted from the code blocks alongside this markdown file. In +directory, and run `lmt README.md`. Now look in the directory and you'll see +files extracted from the code blocks alongside this markdown file. In literate programming lingo, this extraction is (somewhat counterintuitively) -called “tangling.” +called "tangling." Generating documentation from the source is called +"weaving", and `lmt` leaves that to existing markdown renderers (such as +the GitHub frontend.) + +`lmt` is language agnostic. The below demonstration of features is written +in (very trivial) C++ to demonstrate using other languages. ### Tangling into a file. @@ -43,24 +54,37 @@ The header says 3 things: 1. `cpp`: the code block is written in C++. In the rendered markdown output, that affects syntax highlighting, to lmt it means that language-appropriate - `#line` directives will be added so that when debugging the extracted code, - your debugger will show you the line in the original source markdown file. - (If you don't want this effect, just use an unrecognized language name like - `cxx`). + pragma directives will be added so that when debugging the extracted code, + your debugger will show you the line in the original markdown source file. + (If you don't want this effect, you can just use an unrecognized language + name like `cxx`). 2. `hello.cpp`: The code block will be written to the file `hello.cpp`. -3. `+=`: The code block will be appended to that file, rather than overwriting - its content. Since we haven't written anything to `hello.cpp` yet, the - effect is the same, but since overwriting the code you've already extracted - is kind of a nice case, to enable developing examples like those in `lmt`'s - own source, you might want to use `+=` by default. +3. `+=`: The code block will be appended to the most recent code block + defining that file, rather than overwriting its content. Since we haven't + written anything to `hello.cpp` yet, the effect is the same, but this + demonstrates the ability to use it. ### Macro References The `<<<`*string*`>>>` sequences in the body of the code block are called -“macro references.” An LMT “macro” is just a variable whose value can be extracted -from one or more code blocks, and will be substituted wherever its name appears -in triple angle brackets. +"macro references." An LMT "macro" is just a variable whose value can be +extracted from one or more code blocks, and will be substituted wherever +its name appears in triple angle brackets on a line. There are no arguments +to `lmt` macros. + +If we were to run lmt on this file at this point, we would get the warnings: + +``` +Warning: Block named copyright referenced but not defined. +Warning: Block named includes referenced but not defined. +Warning: Block named body of main referenced but not defined. +``` + +This allows us to stub in a macro reference whenever we want in our code, +and only later define them in whatever order best fits our prose. When there +are no more warnings, the `hello.cpp` file should build (assuming we didn't +include any syntax or other compiler errors.) ### Macro Content @@ -77,6 +101,21 @@ injected into hello.cpp via `<<>>`, first line of markup, this code block overwrites any existing value the macro might already have (but since it has no existing value, it's a wash). +`lmt` uses quotation marks to differentiate between macros and file +destinations. If a name is encased in quotes, it's a macro, if not, it's +a file. + +We can later re-define a macro to overwrite it (`​```cpp "body of main"`, +again) + +```cpp "body of main" +std::cout << "Hello, world!" << std::endl; +``` + +`lmt` parses each file passed on the command line in order. The last +definition of a macro will be used for all references to that macro in +other code blocks (including blocks which preceeded it in the source.) + ### Appending To A Macro We can use `#include`s to demonstrate `+=` on macros. There are two includes in @@ -137,22 +176,50 @@ foo, bar, baz, qix, qux, quux, ``` -You need to specify both a file type and a destination (macro or file)if you -want the code block tangled: +You need to specify both a language and a destination (macro or file) if +you want the code block tangled: + +No language (`​``` bar.txt`—note the space): -No file type (`​``` bar.txt`—note the space): ``` bar.txt This doesn't get tangled anywhere ``` -No destination (`​```cpp`): +No destination, but includes syntax highlighting (`​```cpp`) + ```cpp auto x = "nor does this"; ``` -But any file type string and filename (`​```arbitrary foo.txt`) will do +But any language string and filename (`​```arbitrary foo.txt`) will do ```arbitrary foo.txt This gets tangled into foo.txt. ``` + +Running `lmt` on this file at this point should generate the files `data.csv`, +`foo.txt`, and `hello.cpp` with the expected contents and produce no warnings. + +## Building lmt from source + +While the tangled source of `lmt` is included for bootstrapping purposes, +the markdown is considered the canonoical version. The Go source can +be re-extracted with: + +```shell +lmt Implementation.md WhitespacePreservation.md SubdirectoryFiles.md LineNumbers.md IndentedBlocks.md +``` + +If you'd like to read the source, the order of the files and patches were +written is the same as passed on the command line. + + 1. [Basic Implementation](Implementation.md) + 2. [Whitespace Preservation](WhitespacePreservation.md) + 3. [Subdirectory Files](SubdirectoryFiles.md) + 4. [Line Numbers](LineNumbers.md) + 5. [Indented Blocks](IndentedBlocks.md) + +Small bug fixes can be contributed by modifying the prose and code in +the existing files. Larger features can be included as a patch in a +new file. From 6570a8deba69a96adc274857d5f77ebc1a68f978 Mon Sep 17 00:00:00 2001 From: Dave MacFarlane Date: Mon, 8 Mar 2021 16:41:32 -0500 Subject: [PATCH 3/3] Add Credits to README --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index b2b1638..23b967f 100644 --- a/README.md +++ b/README.md @@ -223,3 +223,13 @@ written is the same as passed on the command line. Small bug fixes can be contributed by modifying the prose and code in the existing files. Larger features can be included as a patch in a new file. + +## Credits + +`lmt` is primarily authored by Dave MacFarlane ([@driusan](https://github.com/driusan/)). Bryan Allred ([@bmallred](https://github.com/bmallred/)) improved the +parsing code to include the metadata in the code block header rather than a +rendered markdown header. [@mek-apelsin](https://github.com/mek-apelsin/) +wrote the patch to include pragmas for line numbers, and Dave Abrahams +([@dabrahams](https://github.com/dabrahams/)) wrote the demo of features in +this README, making it more user-focused (it previously dove straight into +implementation.)