WIP try to use non-index tag for code indexes

[newfangle.git] / fangle.txt

fangle

 Sam Liddicott

sam@liddicott.com



August 2009





Introduction 
Fangle is a tool for fangled literate programming. Newfangled is defined as New and often needlessly novel by TheFreeDictionary.com.
In this case, fangled means yet another not-so-new1. but improved.  ^1 method for literate programming.
Literate Programming has a long history starting with the great Donald Knuth himself, whose literate programming tools seem to make use of as many escape sequences for semantic markup as TeX (also by Donald Knuth).
Norman Ramsey wrote the Noweb set of tools (notangle, noweave and noroots) and helpfully reduced the amount of magic character sequences to pretty much just <<, >> and @, and in doing so brought the wonders of literate programming within my reach.
While using the L Y X editor for LaTeX editing I had various troubles with the noweb tools, some of which were my fault, some of which were noweb's fault and some of which were L Y X's fault.
Noweb generally brought literate programming to the masses through removing some of the complexity of the original literate programming, but this would be of no advantage to me if the L Y X / LaTeX combination brought more complications in their place.
Fangle was thus born (originally called Newfangle) as an awk replacement for notangle, adding some important features, like better integration with L Y X and LaTeX (and later TeXmacs), multiple output format conversions, and fixing notangle bugs like indentation when using -L for line numbers.
Significantly, fangle is just one program which replaces various programs in Noweb. Noweave is done away with and implemented directly as LaTeX macros, and noroots is implemented as a function of the untangler fangle.
Fangle is written in awk for portability reasons, awk being available for most platforms. A Python version2. hasn't anyone implemented awk in python yet?  ^2 was considered for the benefit of L Y X but a scheme version for TeXmacs will probably materialise first; as TeXmacs macro capabilities help make edit-time and format-time rendering of fangle chunks simple enough for my weak brain.
As an extension to many literate-programming styles, Fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Name parameters (or even local variables in the callers scope) are anticipated, as parameterized chunks — useful though they are — are hard to comprehend in the literate document.
License 
Fangle is licensed under the GPL 3 (or later).
This doesn't mean that sources generated by fangle must be licensed under the GPL 3.
This doesn't mean that you can't use or distribute fangle with sources of an incompatible license, but it means you must make the source of fangle available too.
As fangle is currently written in awk, an interpreted language, this should not be too hard.

4a <gpl3-copyright[1](\v), lang=text> ≡ 
      ________________________________________________________________________
  1  | fangle - fully featured notangle replacement in awk
  2  | 
  3  | Copyright (C) 2009-2010 Sam Liddicott <sam@liddicott.com>
  4  | 
  5  | This program is free software: you can redistribute it and/or modify
  6  | it under the terms of the GNU General Public License as published by
  7  | the Free Software Foundation, either version 3 of the License, or
  8  | (at your option) any later version.
  9  | 
  10  | This program is distributed in the hope that it will be useful,
  11  | but WITHOUT ANY WARRANTY; without even the implied warranty of
  12  | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  13  | GNU General Public License for more details.
  14  | 
  15  | You should have received a copy of the GNU General Public License
  16  | along with this program.  If not, see <http://www.gnu.org/licenses/>.
     |________________________________________________________________________


Table of contents
Introduction   3
License   4
I Using Fangle   9
1 Introduction to Literate Programming   11
2 Running Fangle   13
2.1 Listing roots   13
2.2 Extracting roots   13
2.3 Formatting the document   13
3 Using Fangle with L^ A T_ E X   15
4 Using Fangle with L Y X   17
4.1 Installing the L Y X module   17
4.2 Obtaining a decent mono font   17
4.2.1 txfonts   17
4.2.2 ams pmb   17
4.2.3 Luximono   17
4.3 Formatting your Lyx document   18
4.3.1 Customising the listing appearance   18
4.3.2 Global customisations   18
4.4 Configuring the build script   19
4.4.1 ...   19
5 Using Fangle with T_ E X_( M A CS)   21
6 Fangle with Makefiles   23
6.1 A word about makefiles formats   23
6.2 Extracting Sources   23
6.2.1 Converting from L Y X to L^ A T_ E X   24
6.2.2 Converting from T_ E X_( M A CS)   24
6.3 Extracting Program Source   25
6.4 Extracting Source Files   25
6.5 Extracting Documentation   28
6.5.1 Formatting T_ E X   28
6.5.1.1 Running pdflatex   28
6.5.2 Formatting T_ E X_( M A CS)   28
6.5.3 Building the Documentation as a Whole   28
6.6 Other helpers   29
6.7 Boot-strapping the extraction   29
6.8 Incorporating Makefile.inc into existing projects   30
Example   30
II Source Code   33
7 Fangle Makefile   35
8 Fangle awk source code   37
8.1 AWK tricks   37
8.2 Catching errors   38
9 T_ E X_( M A CS) args   39
10 L^ A T_ E X and lstlistings   41
10.1 Additional lstlstings parameters   41
10.2 Parsing chunk arguments   43
10.3 Expanding parameters in the text   44
11 Language Modes & Quoting   47
11.1 Modes explanation   47
11.2 Modes affect included chunks   47
11.3 Language Mode Definitions   48
11.3.1 Backslash   49
11.3.2 Strings   49
11.3.3 Parentheses, Braces and Brackets   50
11.3.4 Customizing Standard Modes   51
11.3.5 Comments   51
11.3.6 Regex   52
11.3.7 Perl   53
11.3.8 sh   53
11.3.9 Make   53
11.4 Quoting scenarios   56
11.4.1 Direct quoting   56
11.5 Some tests   57
11.6 A non-recursive mode tracker   58
11.6.1 Constructor   58
11.6.2 Management   58
11.6.3 Tracker   60
11.6.3.1 One happy chunk   62
11.6.3.2 Tests   62
11.7 Escaping and Quoting   63
12 Recognizing Chunks   65
12.1 Chunk start   65
12.1.1 T_ E X_( M A CS)   65
12.1.2 lstlistings   66
12.2 Chunk Body   67
12.2.1 T_ E X_( M A CS)   67
12.2.2 Noweb   68
12.3 Chunk end   68
12.3.1 lstlistings   68
12.3.2 noweb   69
12.4 Chunk contents   69
12.4.1 lstlistings   70
13 Processing Options   73
14 Generating the Output   75
14.1 Assembling the Chunks   76
14.1.1 Chunk Parts   76
15 Storing Chunks   81
16 getopt   83
17 Fangle LaTeX source code   87
17.1 fangle module   87
17.1.1 The Chunk style   87
17.1.2 The chunkref style   88
17.2 Latex Macros   88
17.2.1 The chunk command   89
17.2.1.1 Chunk parameters   90
17.2.2 The noweb styled caption   90
17.2.3 The chunk counter   90
17.2.4 Cross references   93
17.2.5 The end   94
18 Extracting fangle   95
18.1 Extracting from Lyx   95
18.2 Extracting documentation   95
18.3 Extracting from the command line   96
III Tests   97
19 Tests   99
20 Chunk Parameters   101
20.1 L Y X   101
20.2 T_ E X_( M A CS)   101
21 Compile-log-lyx   103
Part I  Using Fangle 
Chapter 1Introduction to Literate Programming
Todo: Should really follow on from a part-0 explanation of what literate programming is.
Chapter 2Running Fangle
Fangle is a replacement for noweb, which consists of notangle, noroots and noweave.
Like notangle and noroots, fangle can read multiple named files, or from stdin.
2.1 Listing roots 
The -r option causes fangle to behave like noroots.
fangle -r filename.tex
will print out the fangle roots of a tex file. 
Unlike the noroots command, the printed roots are not enclosed in angle brackets e.g. <<name>>, unless at least one of the roots is defined using the notangle notation <<name>>=.
Also, unlike noroots, it prints out all roots --- not just those that are not used elsewhere. I find that a root not being used doesn't make it particularly top level — and so-called top level roots could also be included in another root as well. 
My convention is that top level roots to be extracted begin with ./ and have the form of a filename.
Makefile.inc, discussed in 6, can automatically extract all such sources prefixed with ./
2.2 Extracting roots 
notangle's -R and -L options are supported.
If you are using L Y X or LaTeX, the standard way to extract a file would be:
fangle -R./Makefile.inc fangle.tex > ./Makefile.inc
If you are using TeXmacs, the standard way to extract a file would similarly be:
fangle -R./Makefile.inc fangle.txt > ./Makefile.inc
TeXmacs users would obtain the text file with a verbatim export from TeXmacs which can be done on the command line with texmacs -s -c fangle.tm fangle.txt -q
Unlike the noroots command, the -L option to generate C pre-preocessor #file style line-number directives,does not break indenting of the generated file..
Also, thanks to mode tracking (described in 11) the -L option does not interrupt (and break) multi-line C macros either.
This does mean that sometimes the compiler might calculate the source line wrongly when generating error messages in such cases, but there isn't any other way around if multi-line macros include other chunks.
Future releases will include a mapping file so that line/character references from the C compiler can be converted to the correct part of the source document.
2.3 Formatting the document 
The noweave replacement built into the editing and formatting environment for TeXmacs, L Y X (which uses LaTeX), and even for raw LaTeX.
Use of fangle with TeXmacs, L Y X and LaTeX are explained the the next few chapters.
Chapter 3Using Fangle with LaTeX
Because the noweave replacement is impemented in LaTeX, there is no processing stage required before running the LaTeX command. Of course, LaTeX may need running two or more times, so that the code chunk references can be fully calculated.
The formatting is managed by a set of macros shown in 17, and can be included with:
\usepackage{fangle.sty}
Norman Ramsay's origial noweb.sty package is currently required as it is used for formatting the code chunk captions.
The listings.sty package is required, and is used for formatting the code chunks and syntax highlighting.
The xargs.sty package is also required, and makes writing LaTeX macro so much more pleasant.
To do: Add examples of use of Macros

Chapter 4Using Fangle with L Y X
L Y X uses the same LaTeX macros shown in 17 as part of a L Y X module file fangle.module, which automatically includes the macros in the document pre-amble provided that the fangle L Y X module is used in the document.
4.1 Installing the L Y X module 
Copy fangle.module to your L Y X layouts directory, which for unix users will be ~/.lyx/layouts
In order to make the new literate styles availalble, you will need to reconfigure L Y X by clicking Tools->Reconfigure, and then re-start L Y X.
4.2 Obtaining a decent mono font 
The syntax high-lighting features of lstlistings makes use of bold; however a mono-space tt font is used to typeset the listings. Obtaining a bold tt font can be impossibly difficult and amazingly easy. I spent many hours at it, following complicated instructions from those who had spend many hours over it, and was finally delivered the simple solution on the lyx mailing list.
4.2.1 txfonts 
The simple way was to add this to my preamble:
\usepackage{txfonts}
\renewcommand{\ttdefault}{txtt}

4.2.2 ams pmb 
The next simplest way was to use ams poor-mans-bold, by adding this to the pre-amble:
\usepackage{amsbsy}
%\renewcommand{\ttdefault}{txtt}
%somehow make \pmb be the command for bold, forgot how, sorry, above line not work
It works, but looks wretched on the dvi viewer.
4.2.3 Luximono 
The lstlistings documention suggests using Luximono.
Luximono was installed according to the instructions in Ubuntu Forums thread 11591811. http://ubuntuforums.org/showthread.php?t=1159181  ^1 with tips from miknight2. http://miknight.blogspot.com/2005/11/how-to-install-luxi-mono-font-in.html  ^2 stating that sudo updmap --enable MixedMap ul9.map is required. It looks fine in PDF and PS view but still looks rotten in dvi view.
4.3 Formatting your Lyx document 
It is not necessary to base your literate document on any of the original L Y X literate classes; so select a regular class for your document type.
Add the new module Fangle Literate Listings and also Logical Markup which is very useful.
In the drop-down style listbox you should notice a new style defined, called Chunk.
When you wish to insert a literate chunk, you enter it's plain name in the Chunk style, instead of the old noweb method that uses <<name>>= type tags. In the line (or paragraph) following the chunk name, you insert a listing with: Insert->Program Listing.
Inside the white listing box you can type (or paste using shift+ctrl+V) your listing. There is no need to use ctrl+enter at the end of lines as with some older L Y X literate techniques --- just press enter as normal.
4.3.1 Customising the listing appearance 
The code is formatted using the lstlistings package. The chunk style doesn't just define the chunk name, but can also define any other chunk options supported by the lstlistings package \lstset command. In fact, what you type in the chunk style is raw latex. If you want to set the chunk language without having to right-click the listing, just add ,lanuage=C after the chunk name. (Currently the language will affect all subsequent listings, so you may need to specify ,language= quite a lot).
To do: so fix the bug

Of course you can do this by editing the listings box advanced properties by right-clicking on the listings box, but that takes longer, and you can't see at-a-glance what the advanced settings are while editing the document; also advanced settings apply only to that box --- the chunk settings apply through the rest of the document3. It ought to apply only to subsequent chunks of the same name. I'll fix that later  ^3.
To do: So make sure they only apply to chunks of that name

4.3.2 Global customisations 
As lstlistings is used to set the code chunks, it's \lstset command can be used in the pre-amble to set some document wide settings.
If your source has many words with long sequences of capital letters, then columns=fullflexible may be a good idea, or the capital letters will get crowded. (I think lstlistings ought to use a slightly smaller font for captial letters so that they still fit).
The font family \ttfamily looks more normal for code, but has no bold (an alternate typewriter font is used). 
With \ttfamily, I must also specify columns=fullflexible or the wrong letter spacing is used.
In my LaTeX pre-amble I usually specialise my code format with:

19a <document-preamble[1](\v), lang=tex> ≡ 
      ________________________________________________________________________
  1  | \lstset{
  2  | numbers=left, stepnumber=1, numbersep=5pt,
  3  | breaklines=false,
  4  | basicstyle=\footnotesize\ttfamily,
  5  | numberstyle=\tiny,
  6  | language=C,
  7  | columns=fullflexible,
  8  | numberfirstline=true
  9  | }
     |________________________________________________________________________



4.4 Configuring the build script 
You can invoke code extraction and building from the L Y X menu option Document->Build Program.
First, make sure you don't have a conversion defined for Lyx->Program
From the menu Tools->Preferences, add a conversion from Latex(Plain)->Program as:
set -x ; fangle -Rlyx-build $$i | 
  env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
(But don't cut-n-paste it from this document or you may be be pasting a multi-line string which will break your lyx preferences file). 
I hope that one day, L Y X will set these into the environment when calling the build script.
You may also want to consider adding options to this conversion...
parselog=/usr/share/lyx/scripts/listerrors
...but if you do you will lose your stderr4. There is some bash plumbing to get a copy of stderr but this footnote is too small  ^4.
Now, a shell script chunk called lyx-build will be extracted and run whenever you choose the Document->Build Program menu item.
This document was originally managed using L Y X and lyx-build script for this document is shown here for historical reference. 
lyx -e latex fangle.lyx && \
  fangle fangle.lyx > ./autoboot
This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
4.4.1 ... 
When the lyx-build chunk is executed, the current directory will be a temporary directory, and LYX_SOURCE will refer to the tex file in this temporary directory. This is unfortunate as our makefile wants to run from the project directory where the Lyx file is kept.
We can extract the project directory from $$r, and derive the probable Lyx filename from the noweb file that Lyx generated.

19b <lyx-build-helper[1](\v), lang=sh> ≡  95b⊳
      ________________________________________________________________________
  1  | PROJECT_DIR="$LYX_r"
  2  | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
  3  | TEX_DIR="$LYX_p"
  4  | TEX_SRC="$TEX_DIR/$LYX_i"
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
And then we can define a lyx-build fragment similar to the autoboot fragment

20a <lyx-build[1](\v), lang=sh> ≡  95a⊳
      ________________________________________________________________________
  1  | #! /bin/sh
  2  | «lyx-build-helper 19b»
  3  | cd $PROJECT_DIR || exit 1
  4  | 
  5  | #/usr/bin/fangle -filter ./notanglefix-filter \
  6  | #  -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx" \
  7  | #  | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/' \
  8  | #  > ./Makefile.inc
  9  | #
  10  | #make -f ./Makefile.inc fangle_sources
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

Chapter 5Using Fangle with TeXmacs
To do: Write this chapter

Chapter 6Fangle with Makefiles
Here we describe a Makefile.inc that you can include in your own Makefiles, or glue as a recursive make to other projects.
Makefile.inc will cope with extracting all the other source files from this or any specified literate document and keeping them up to date. 
It may also be included by a Makefile or Makefile.am defined in a literate document to automatically deal with the extraction of source files and documents during normal builds.
Thus, if Makefile.inc is included into a main project makefile it add rules for the source files, capable of extracting the source files from the literate document.
6.1 A word about makefiles formats 
Whitespace formatting is very important in a Makefile. The first character of each action line must be a TAB. 
target: pre-requisite
↦action
↦action
This requires that the literate programming environment have the ability to represent a TAB character in a way that fangle will generate an actual TAB character.
We also adopt a convention that code chunks whose names beginning with ./ should always be automatically extracted from the document. Code chunks whose names do not begin with ./ are for internal reference. Such chunks may be extracted directly, but will not be automatically extracted by this Makefile.
6.2 Extracting Sources 
Our makefile has two parts; variables must be defined before the targets that use them.
As we progress through this chapter, explaining concepts, we will be adding lines to <Makefile.inc-vars 23b> and <Makefile.inc-targets 24c> which are included in <./Makefile.inc 23a> below.

23a <./Makefile.inc[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | «Makefile.inc-vars 23b»
  2  | «Makefile.inc-default-targets 28a»
  3  | «Makefile.inc-targets 24c»
     |________________________________________________________________________


We first define a placeholder for the tool fangle in case it cannot be found in the path.

23b <Makefile.inc-vars[1](\v), lang=make> ≡  24a⊳
      ________________________________________________________________________
  1  | FANGLE=fangle
  2  | AWK=awk
  3  | RUN_FANGLE=$(AWK) -f $(FANGLE)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also define a placeholder for LITERATE_SOURCE to hold the name of this document. This will normally be passed on the command line or set by the including makefile.

24a <Makefile.inc-vars[2](\v) ⇑23b, lang=> +≡ ⊲23b 24b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | #LITERATE_SOURCE=
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Fangle cannot process L Y X or TeXmacs documents directly, so the first stage is to convert these to more suitable text based formats1. L Y X and TeXmacs formats are text-based, but not suitable for fangle  ^1.
6.2.1 Converting from L Y X to LaTeX 
The first stage will always be to convert the L Y X file to a LaTeX file. Fangle must run on a TeX file because the L Y X command server-goto-file-line2. The Lyx command server-goto-file-line is used to position the Lyx cursor at the compiler errors.  ^2 requries that the line number provided be a line of the TeX file and always maps this the line in the L Y X docment. We use server-goto-file-line when moving the cursor to error lines during compile failures.
The command lyx -e literate fangle.lyx will produce fangle.tex, a TeX file; so we define a make target to be the same as the L Y X file but with the .tex extension.
The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.

24b <Makefile.inc-vars[3](\v) ⇑23b, lang=> +≡ ▵24a 24d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | LYX_SOURCE=$(LITERATE_SOURCE) # but only the .lyx files
  6  | TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
  7  | EXTRA_DIST+=$(TEX_SOURCE)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then specify that the TeX source is to be generated from the L Y X source.

24c <Makefile.inc-targets[1](\v), lang=make> ≡  25a⊳
      ________________________________________________________________________
  1  | .SUFFIXES: .tex .lyx
  2  | .lyx.tex:
  3  | ↦lyx -e latex $<
  4  | clean_tex:
  5  | ↦rm -f -- $(TEX_SOURCE)
  6  | clean: clean_tex
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.2.2 Converting from TeXmacs 
Fangle cannot process TeXmacs files directly3. but this is planned when TeXmacs uses xml as it's native format  ^3, but must first convert them to text files.
The command texmacs -c fangle.tm fangle.txt -q will produce fangle.txt, a text file; so we define a make target to be the same as the TeXmacs file but with the .txt extension.
The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.

24d <Makefile.inc-vars[4](\v) ⇑23b, lang=> +≡ ▵24b 25b⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  8  | TEXMACS_SOURCE=$(LITERATE_SOURCE) # but only the .tm files
  9  | TXT_SOURCE=$(LITERATE_SOURCE:.tm=.txt)
  10  | EXTRA_DIST+=$(TXT_SOURCE)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
To do: Add loop around each $< so multiple targets can be specified


25a <Makefile.inc-targets[2](\v) ⇑24c, lang=> +≡ ⊲24c 25d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  7  | .SUFFIXES: .txt .tm
  8  | .tm.txt:
  9  | ↦texmacs -s -c $< $@ -q
  10  | .PHONEY: clean_txt
  11  | clean_txt:
  12  | ↦rm -f -- $(TXT_SOURCE)
  13  | clean: clean_txt
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.3 Extracting Program Source 
The program source is extracted using fangle, which is designed to operate on text or a LaTeX documents4. LaTeX documents are just slightly special text documents  ^4.

25b <Makefile.inc-vars[5](\v) ⇑23b, lang=> +≡ ⊲24d 25c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  11  | FANGLE_SOURCE=$(TXT_SOURCE)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The literate document can result in any number of source files, but not all of these will be changed each time the document is updated. We certainly don't want to update the timestamps of these files and cause the whole source tree to be recompiled just because the literate explanation was revised. We use CPIF from the Noweb tools to avoid updating the file if the content has not changed, but should probably write our own.
However, if a source file is not updated, then the fangle file will always have a newer time-stamp and the makefile would always re-attempt to extact a newer source file which would be a waste of time.
Because of this, we use a stamp file which is always updated each time the sources are fully extracted from the LaTeX document. If the stamp file is newer than the document, then we can avoid an attempt to re-extract any of the sources. Because this stamp file is only updated when extraction is complete, it is safe for the user to interrupt the build-process mid-extraction.
We use echo rather than touch to update the stamp file beause the touch command does not work very well over an sshfs mount  that I was using.

25c <Makefile.inc-vars[6](\v) ⇑23b, lang=> +≡ ▵25b 26a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  12  | FANGLE_SOURCE_STAMP=$(FANGLE_SOURCE).stamp
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

25d <Makefile.inc-targets[3](\v) ⇑24c, lang=> +≡ ▵25a 26b⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  14  | $(FANGLE_SOURCE_STAMP): $(FANGLE_SOURCE) \
  15  | ↦                $(FANGLE_SOURCES) ; \
  16  | ↦echo -n > $(FANGLE_SOURCE_STAMP)
  17  | clean_stamp:
  18  | ↦rm -f $(FANGLE_SOURCE_STAMP)
  19  | clean: clean_stamp
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.4 Extracting Source Files 
We compute FANGLE_SOURCES to hold the names of all the source files defined in the document. We compute this only once, by means of := in assignent. The sed deletes the any << and >> which may surround the roots names (for compatibility with Noweb's noroots command).
As we use chunk names beginning with ./ to denote top level fragments that should be extracted, we filter out all fragments that do not begin with ./
Note 1. FANGLE_PREFIX is set to ./ by default, but whatever it may be overridden to, the prefix is replaced by a literal ./ before extraction so that files will be extracted in the current directory whatever the prefix. This helps namespace or sub-project prefixes like documents: for chunks like documents:docbook/intro.xml 
To do: This doesn't work though, because it loses the full name and doesn't know what to extact!


26a <Makefile.inc-vars[7](\v) ⇑23b, lang=> +≡ ⊲25c 26e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  13  | FANGLE_PREFIX:=\.\/
  14  | FANGLE_SOURCES:=$(shell \
  15  |   $(RUN_FANGLE) -r $(FANGLE_SOURCE) |\
  16  |   sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(FANGLE_PREFIX)/!d' \
  17  |       -e 's/^$(FANGLE_PREFIX)/\.\//' )
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The target below, echo_fangle_sources is a helpful debugging target and shows the names of the files that would be extracted.

26b <Makefile.inc-targets[4](\v) ⇑24c, lang=> +≡ ⊲25d 26c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  20  | .PHONY: echo_fangle_sources
  21  | echo_fangle_sources: ; @echo $(FANGLE_SOURCES)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We define a convenient target called fangle_sources so that make -f fangle_sources will re-extract the source if the literate document has been updated. 

26c <Makefile.inc-targets[5](\v) ⇑24c, lang=> +≡ ▵26b 26d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  22  | .PHONY: fangle_sources
  23  | fangle_sources: $(FANGLE_SOURCE_STAMP)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
And also a convenient target to remove extracted sources.

26d <Makefile.inc-targets[6](\v) ⇑24c, lang=> +≡ ▵26c 27e⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  24  | .PHONY: clean_fangle_sources
  25  | clean_fangle_sources: ; \
  26  |         rm -f -- $(FANGLE_SOURCE_STAMP) $(FANGLE_SOURCES)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We now look at the extraction of the source files.
This makefile macro if_extension takes 4 arguments: the filename $(1), some extensions to match $(2) and a shell command to return if the filename does match the exensions $(3), and a shell command to return if it does not match the extensions $(4).

26e <Makefile.inc-vars[8](\v) ⇑23b, lang=> +≡ ▵26a 26f⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  18  | if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
For some source files like C files, we want to output the line number and filename of the original LaTeX document from which the source came5. I plan to replace this option with a separate mapping file so as not to pollute the generated source, and also to allow a code pretty-printing reformatter like indent be able to re-format the file and adjust for changes through comparing the character streams.  ^5.
To make this easier we define the file extensions for which we want to do this.

27a <Makefile.inc-vars[9](\v) ⇑23b, lang=> +≡ ⊲26e 27a▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  19  | C_EXTENSIONS=.c .h
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We can then use the if_extensions macro to define a macro which expands out to the -L option if fangle is being invoked in a C source file, so that C compile errors will refer to the line number in the TeX document. 

27b <Makefile.inc-vars[10](\v) ⇑23b, lang=> +≡ ▵26f 27b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  20  | TABS=8
  21  | nf_line=-L -T$(TABS)
  22  | fangle=$(RUN_FANGLE) $(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line)) -R"$(2)" $(1)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We can use a similar trick to define an indent macro which takes just the filename as an argument and can return a pipeline stage calling the indent command. Indent can be turned off with make fangle_sources indent=

27c <Makefile.inc-vars[11](\v) ⇑23b, lang=> +≡ ▵27a 27c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  23  | indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
  24  | indent=$(call if_extension,$(1),$(C_EXTENSIONS), | indent $(indent_options))
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We now define the pattern for extracting a file. The files are written using noweb's cpif so that the file timestamp will not be touched if the contents haven't changed. This avoids the need to rebuild the entire project because of a typographical change in the documentation, or if none or a few C source files have changed.

27d <Makefile.inc-vars[12](\v) ⇑23b, lang=> +≡ ▵27b 27d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  25  | fangle_extract=@mkdir -p $(dir $(1)) && \
  26  |   $(call fangle,$(2),$(1)) > "$(1).tmp" && \
  27  |   cat "$(1).tmp" $(indent) | cpif "$(1)" \
  28  |   && rm -f -- "$(1).tmp" || \
  29  |   (echo error fangling $(1) from $(2) ; exit 1)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We define a target which will extract or update all sources. To do this we first defined a makefile template that can do this for any source file in the LaTeX document.

27e <Makefile.inc-vars[13](\v) ⇑23b, lang=> +≡ ▵27c 28b⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  30  | define FANGLE_template
  31  |   $(1): $(2)
  32  | ↦$$(call fangle_extract,$(1),$(2))
  33  |   FANGLE_TARGETS+=$(1)
  34  | endef
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then enumerate the discovered FANGLE_SOURCES to generate a makefile rule for each one using the makefile template we defined above.

27f <Makefile.inc-targets[7](\v) ⇑24c, lang=> +≡ ⊲26d 27f▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  27  | $(foreach source,$(FANGLE_SOURCES),\
  28  |   $(eval $(call FANGLE_template,$(source),$(FANGLE_SOURCE))) \
  29  | )
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
These will all be built with FANGLE_SOURCE_STAMP.
We also remove the generated sources on a make distclean.

27g <Makefile.inc-targets[8](\v) ⇑24c, lang=> +≡ ▵27e 28c⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  30  | _distclean: clean_fangle_sources
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.5 Extracting Documentation 
We then identify the intermediate stages of the documentation and their build and clean targets.

28a <Makefile.inc-default-targets[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | .PHONEY : clean_pdf
     |________________________________________________________________________


6.5.1 Formatting TeX 
6.5.1.1 Running pdflatex 
We produce a pdf file from the tex file.

28b <Makefile.inc-vars[14](\v) ⇑23b, lang=> +≡ ⊲27d 28d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  35  | FANGLE_PDF+=$(TEX_SOURCE:.tex=.pdf)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We run pdflatex twice to be sure that the contents and aux files are up to date. We certainly are required to run pdflatex at least twice if these files do not exist.

28c <Makefile.inc-targets[9](\v) ⇑24c, lang=> +≡ ⊲27f 28e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  31  | .SUFFIXES: .tex .pdf
  32  | .tex.pdf:
  33  | ↦pdflatex $< && pdflatex $<
  34  | 
  35  | clean_pdf_tex:
  36  | ↦rm -f -- $(FANGLE_PDF) $(TEX_SOURCE:.tex=.toc) \
  37  | ↦  $(TEX_SOURCE:.tex=.log) $(TEX_SOURCE:.tex=.aux)
  38  | clean_pdf: clean_pdf_tex
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.5.2 Formatting TeXmacs 
TeXmacs can produce a PDF file directly.

28d <Makefile.inc-vars[15](\v) ⇑23b, lang=> +≡ ▵28b 28f⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  36  | FANGLE_PDF+=$(LITERATE_SOURCE:.tm=.pdf)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
To do: Outputting the PDF may not be enough to update the links and page references. I think
we need to update twice, generate a pdf, update twice mode and generate a new PDF.
Basically the PDF export of TeXmacs is pretty rotten and doesn't work properly from the CLI


28e <Makefile.inc-targets[10](\v) ⇑24c, lang=> +≡ ▵28c 29a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  39  | .SUFFIXES: .tm .pdf
  40  | .tm.pdf:
  41  | ↦texmacs -s -c $< $@ -q
  42  | 
  43  | clean_pdf_texmacs:
  44  | ↦rm -f -- $(FANGLE_PDF)
  45  | clean_pdf: clean_pdf_texmacs
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
6.5.3 Building the Documentation as a Whole 
Currently we only build pdf as a final format, but FANGLE_DOCS may later hold other output formats.

29a <Makefile.inc-vars[16](\v) ⇑23b, lang=> +≡ ⊲28d
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  37  | FANGLE_DOCS=$(FANGLE_PDF)
     |________________________________________________________________________


We also define fangle_docs as a convenient phony target.

29b <Makefile.inc-targets[11](\v) ⇑24c, lang=> +≡ ⊲28e 29b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  46  | .PHONY: fangle_docs
  47  | fangle_docs: $(FANGLE_DOCS)
  48  | docs: fangle_docs
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
And define a convenient clean_fangle_docs which we add to the regular clean target

29c <Makefile.inc-targets[12](\v) ⇑24c, lang=> +≡ ▵29a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  49  | .PHONEY: clean_fangle_docs
  50  | clean_fangle_docs: clean_tex clean_pdf
  51  | clean: clean_fangle_docs
  52  | 
  53  | distclean_fangle_docs: clean_tex clean_fangle_docs
  54  | distclean: clean distclean_fangle_docs
     |________________________________________________________________________


6.6 Other helpers 
If Makefile.inc is included into Makefile, then extracted files can be updated with this command:
make fangle_sources
otherwise, with:
make -f Makefile.inc fangle_sources
6.7 Boot-strapping the extraction 
As well as having the makefile extract or update the source files as part of it's operation, it also seems convenient to have the makefile re-extracted itself from this document.
It would also be convenient to have the code that extracts the makefile from this document to also be part of this document, however we have to start somewhere and this unfortunately requires us to type at least a few words by hand to start things off.
Therefore we will have a minimal root fragment, which, when extracted, can cope with extracting the rest of the source. This shell script fragment can do that. It's name is * — out of regard for Noweb, but when extracted might better be called autoupdate.
To do: De-lyxify


29d <*[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | #! /bin/sh
  2  | 
  3  | MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
  4  | MAKE_SRC=‘dirname "$MAKE_SRC"‘/‘basename "$MAKE_SRC" .lyx‘
  5  | NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
  6  | lyx -e latex $MAKE_SRC
  7  | 
  8  | fangle -R./Makefile.inc ${MAKE_SRC}.tex \
  9  |   | sed "/FANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$FANGLE_SRC" \
  10  |   | cpif ./Makefile.inc
  11  | 
  12  | make -f ./Makefile.inc fangle_sources
     |________________________________________________________________________


The general Makefile can be invoked with ./autoboot and can also be included into any automake file to automatically re-generate the source files.
The autoboot can be extracted with this command:
lyx -e latex fangle.lyx && \
  fangle fangle.lyx > ./autoboot
This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
On a unix system this will extract fangle.module and the fangle awk script, and run some basic tests. 
To do: cross-ref to test chapter when it is a chapter all on its own

6.8 Incorporating Makefile.inc into existing projects 
If you are writing a literate module of an existing non-literate program you may find it easier to use a slight recursive make instead of directly including Makefile.inc in the projects makefile. 
This way there is less chance of definitions in Makefile.inc interfering with definitions in the main makefile, or with definitions in other Makefile.inc from other literate modules of the same project.
To do this we add some glue to the project makefile that invokes Makefile.inc in the right way. The glue works by adding a .PHONY target to call the recursive make, and adding this target as an additional pre-requisite to the existing targets.
Example Sub-module of existing system
In this example, we are building module.so as a literate module of a larger project.
We will show the sort glue that can be inserted into the projects Makefile — or more likely — a regular Makefile included in or invoked by the projects Makefile.

30a <makefile-glue[1](\v), lang=> ≡  30b▿
      ________________________________________________________________________
  1  | module_srcdir=modules/module
  2  | MODULE_SOURCE=module.tm
  3  | MODULE_STAMP=$(MODULE_SOURCE).stamp
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The existing build system may already have a build target for module.o, but we just add another pre-requisite to that. In this case we use module.tm.stamp as a pre-requisite, the stamp file's modified time indicating when all sources were extracted6. If the projects build system does not know how to build the module from the extracted sources, then just add build actions here as normal.  ^6.

30b <makefile-glue[2](\v) ⇑30a, lang=make> +≡ ▵30a 30c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | $(module_srcdir)/module.o: $(module_srcdir)/$(MODULE_STAMP)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The target for this new pre-requisite will be generated by a recursive make using Makefile.inc which will make sure that the source is up to date, before it is built by the main projects makefile.

30c <makefile-glue[3](\v) ⇑30a, lang=> +≡ ▵30b 31a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | $(module_srcdir)/$(MODULE_STAMP): $(module_srcdir)/$(MODULE_SOURCE)
  6  | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc fangle_sources LITERATE_SOURCE=$(MODULE_SOURCE)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We can do similar glue for the docs, clean and distclean targets. In this example the main prject was using a double colon for these targets, so we must use the same in our glue.

31a <makefile-glue[4](\v) ⇑30a, lang=> +≡ ⊲30c
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  7  | docs:: docs_module
  8  | .PHONY: docs_module
  9  | docs_module:
  10  | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc docs LITERATE_SOURCE=$(MODULE_SOURCE)
  11  | 
  12  | clean:: clean_module
  13  | .PHONEY: clean_module
  14  | clean_module:
  15  | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc clean LITERATE_SOURCE=$(MODULE_SOURCE)
  16  | 
  17  | distclean:: distclean_module
  18  | .PHONY: distclean_module
  19  | distclean_module:
  20  | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc distclean LITERATE_SOURCE=$(MODULE_SOURCE)
     |________________________________________________________________________


We could do similarly for install targets to install the generated docs.
Part II  Source Code 
Chapter 7Fangle Makefile
We use the copyright notice from chapter 2, and the Makefile.inc from chapter 6

35a <./Makefile[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | # «gpl3-copyright 4a»
  2  | 
  3  | «make-fix-make-shell 55c»
  4  | 
  5  | LITERATE_SOURCE=fangle.tm
  6  | 
  7  | all: fangle_sources
  8  | include Makefile.inc
  9  | 
  10  | fangle: test
  11  | ./fangle: test
  12  | 
  13  | .PHONEY: test
  14  | test: fangle.txt
  15  | ↦$(RUN_FANGLE) -R"test:*" fangle.txt > test.sh
  16  | ↦bash test.sh ; echo pass $$?
     |________________________________________________________________________


Chapter 8Fangle awk source code
We use the copyright notice from chapter 2.

37a <./fangle[1](\v), lang=awk> ≡  37b▿
      ________________________________________________________________________
  1  | #! /usr/bin/awk -f
  2  | # «gpl3-copyright 4a»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also use code from Arnold Robbins public domain getopt (1993 revision) defined in 85a, and naturally want to attribute this appropriately.

37b <./fangle[2](\v) ⇑37a, lang=> +≡ ▵37a 37c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  3  | # NOTE: Arnold Robbins public domain getopt for awk is also used:
  4  | «getopt.awk-header 83a»
  5  | «getopt.awk-getopt() 83c»
  6  | 
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
And include the following chunks (which are explained further on) to make up the program:

37c <./fangle[3](\v) ⇑37a, lang=> +≡ ▵37b 42a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  7  | «helper-functions 38d»
  8  | «mode-tracker 62b»
  9  | «parse_chunk_args 44a»
  10  | «chunk-storage-functions 81b»
  11  | «output_chunk_names() 75d»
  12  | «output_chunks() 75e»
  13  | «write_chunk() 76a»
  14  | «expand_chunk_args() 44b»
  15  | 
  16  | «begin 73d»
  17  | «recognize-chunk 65a»
  18  | «end 75c»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
8.1 AWK tricks 
The portable way to erase an array in awk is to split the empty string, so we define a fangle macro that can split an array, like this:

37d <awk-delete-array[1](ARRAY\v\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | split("", ${ARRAY});
     |________________________________________________________________________


For debugging it is sometimes convenient to be able to dump the contents of an array to stderr, and so this macro is also useful.

37e <dump-array[1](ARRAY\v\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | print "\nDump: ${ARRAY}\n--------\n" > "/dev/stderr";
  2  | for (_x in ${ARRAY}) {
  3  |   print _x "=" ${ARRAY}[_x] "\n" > "/dev/stderr";
  4  | }
  5  | print "========\n" > "/dev/stderr";
     |________________________________________________________________________


8.2 Catching errors 
Fatal errors are issued with the error function:

38a <error()[1](\v), lang=awk> ≡  38b▿
      ________________________________________________________________________
  1  | function error(message)
  2  | {
  3  |   print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
  4  |   exit 1;
  5  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
and likewise for non-fatal warnings:

38b <error()[2](\v) ⇑38a, lang=awk> +≡ ▵38a 38c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  6  | function warning(message)
  7  | {
  8  |   print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
  9  |   warnings++;
  10  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
and debug output too:

38c <error()[3](\v) ⇑38a, lang=awk> +≡ ▵38b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  11  | function debug_log(message)
  12  | {
  13  |   print "DEBUG: " FILENAME ":" FNR " " message > "/dev/stderr";
  14  | }
     |________________________________________________________________________


To do: append=helper-functions


38d <helper-functions[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «error() 38a»
     |________________________________________________________________________


Chapter 9TeXmacs args
TeXmacs functions with arguments1. or function declarations with parameters  ^1 appear like this:
blah((I came, I saw, I conquered)<wide-overbrace>^(argument 1)(^K, )<wide-overbrace>^(sep.)(and then went home asd)<wide-overbrace>^(argument 3)(^K))<wide-overbrace>^(term.)_arguments
Arguments commence after the opening parenthesis. The first argument runs up till the next ^K. 
If the following character is a , then another argument follows. If the next character after the , is a space character, then it is also eaten. The fangle stylesheet emits ^K,Space as separators, but the fangle untangler will forgive a missing space.
If the following character is ) then this is a terminator and there are no more arguments.

39a <constants[1](\v), lang=> ≡  81a⊳
      ________________________________________________________________________
  1  | ARG_SEPARATOR=sprintf("%c", 11);
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
To process the text in this fashion, we split the string on ^K


39b <get_chunk_args[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function get_texmacs_chunk_args(text, args,   a, done) {
  2  |   split(text, args, ARG_SEPARATOR);
  3  | 
  4  |   done=0
  5  |   for (a=1; (a in args); a++) if (a>1) {
  6  |     if (args[a] == "" || substr(args[a], 1, 1) == ")") done=1;
  7  |     if (done) {
  8  |       delete args[a];
  9  |       break;
  10  |     }
  11  | 
  12  |     if (substr(args[a], 1, 2) == ", ") args[a]=substr(args[a], 3);
  13  |     else if (substr(args[a], 1, 1) == ",") args[a]=substr(args[a], 2);  
  14  |   }
  15  | }
     |________________________________________________________________________


Chapter 10LaTeX and lstlistings
To do: Split LyX and TeXmacs parts

For L Y X and LaTeX, the lstlistings package is used to format the lines of code chunks. You may recal from chapter XXX that arguments to a chunk definition are pure LaTeX code. This means that fangle needs to be able to parse LaTeX a little.
LaTeX arguments to lstlistings macros are a comma seperated list of key-value pairs, and values containing commas are enclosed in { braces } (which is to be expected for LaTeX).
A sample expressions is:
name=thomas, params={a, b}, something, something-else
but we see that this is just a simpler form of this expression:
name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
We may consider that we need a function that can parse such LaTeX expressions and assign the values to an AWK associated array, perhaps using a recursive parser into a multi-dimensional hash1. as AWK doesn't have nested-hash support  ^1, resulting in:
key                 value
a[name]             freddie
a[foo, bar]         baz
a[foo, quux, quirk] 
a[foo, quux, a]     fleeg
a[etc]              

Yet, also, on reflection it seems that sometimes such nesting is not desirable, as the braces are also used to delimit values that contain commas --- we may consider that
name={williamson, freddie}
should assign williamson, freddie to name.
In fact we are not so interested in the detail so as to be bothered by this, which turns out to be a good thing for two reasons. Firstly TeX has a malleable parser with no strict syntax, and secondly whether or not williamson and freddie should count as two items will be context dependant anyway.
We need to parse this latex for only one reason; which is that we are extending lstlistings to add some additional arguments which will be used to express chunk parameters and other chunk options.
10.1 Additional lstlstings parameters 
Further on we define a \Chunk LaTeX macro whose arguments will consist of a the chunk name, optionally followed by a comma and then a comma separated list of arguments. In fact we will just need to prefix name= to the arguments to in order to create valid lstlistings arguments. 
There will be other arguments supported too; 
params.As an extension to many literate-programming styles, fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Chunk parameters are declared with a chunk argument called params, which holds a semi-colon separated list of parameters, like this:
achunk,language=C,params=name;address
addto.a named chunk that this chunk is to be included into. This saves the effort of having to declare another listing of the named chunk merely to include this one. 
Function get_chunk_args() will accept two paramters, text being the text to parse, and values being an array to receive the parsed values as described above. The optional parameter path is used during recursion to build up the multi-dimensional array path.

42a <./fangle[4](\v) ⇑37a, lang=> +≡ ⊲37c
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  19  | «get_chunk_args() 42b»
     |________________________________________________________________________



42b <get_chunk_args()[1](\v), lang=> ≡  42c▿
      ________________________________________________________________________
  1  | function get_tex_chunk_args(text, values,
  2  |   # optional parameters
  3  |   path, # hierarchical precursors
  4  |   # local vars
  5  |   a, name)
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The strategy is to parse the name, and then look for a value. If the value begins with a brace {, then we recurse and consume as much of the text as necessary, returning the remaining text when we encounter a leading close-brace }. This being the strategy --- and executed in a loop --- we realise that we must first look for the closing brace (perhaps preceded by white space) in order to terminate the recursion, and returning remaining text.

42c <get_chunk_args()[2](\v) ⇑42b, lang=> +≡ ▵42b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  6  | {
  7  |   split("", values);
  8  |   while(length(text)) {
  9  |     if (match(text, "^ *}(.*)", a)) {
  10  |       return a[1];
  11  |     }
  12  |     «parse-chunk-args 42d»
  13  |   }
  14  |   return text;
  15  | }
     |________________________________________________________________________


We can see that the text could be inspected with this regex:

42d <parse-chunk-args[1](\v), lang=> ≡  43a⊳
      ________________________________________________________________________
  1  | if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a)) {
  2  |   return text;
  3  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
and that a will have the following values:
a[n] assigned text
1    freddie
2    =freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
3    =
4    freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
5    freddie
6    , foo={bar=baz, quux={quirk, a=fleeg}}, etc

a[3] will be either = or , and signify whether the option named in a[1] has a value or not (respectively).
If the option does have a value, then if the expression substr(a[4],1,1) returns a brace { it will signify that we need to recurse:

43a <parse-chunk-args[2](\v) ⇑42d, lang=> +≡ ⊲42d
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | name=a[1];
  5  | if (a[3] == "=") {
  6  |   if (substr(a[4],1,1) == "{") {
  7  |     text = get_tex_chunk_args(substr(a[4],2), values, path name SUBSEP);
  8  |   } else {
  9  |     values[path name]=a[5];
  10  |     text = a[6];
  11  |   }
  12  | } else {
  13  |   values[path name]="";
  14  |   text = a[2];
  15  | }
     |________________________________________________________________________


We can test this function like this:

43b <gca-test.awk[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «get_chunk_args() 42b»
  2  | BEGIN {
  3  |   SUBSEP=".";
  4  | 
  5  |   print get_tex_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc", a);
  6  |   for (b in a) {
  7  |     print "a[" b "] => " a[b];
  8  |   }
  9  | }
     |________________________________________________________________________


which should give this output:

43c <gca-test.awk-results[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | a[foo.quux.quirk] => 
  2  | a[foo.quux.a] => fleeg
  3  | a[foo.bar] => baz
  4  | a[etc] => 
  5  | a[name] => freddie
     |________________________________________________________________________


10.2 Parsing chunk arguments 
Arguments to paramterized chunks are expressed in round brackets as a comma separated list of optional arguments. For example, a chunk that is defined with:
\Chunk{achunk, params=name ; address}
could be invoked as:
\chunkref{achunk}(John Jones, jones@example.com)
An argument list may be as simple as in \chunkref{pull}(thing, otherthing) or as complex as:
\chunkref{pull}(things[x, y], get_other_things(a, "(all)"))
--- which for all it's commas and quotes and parenthesis represents only two parameters: things[x, y] and get_other_things(a, "(all)").
If we simply split parameter list on commas, then the comma in things[x,y] would split into two seperate arguments: things[x and y]--- neither of which make sense on their own.
One way to prevent this would be by refusing to split text between matching delimiters, such as [, ], (, ), {, } and most likely also ", " and ', '. Of course this also makes it impossible to pass such mis-matched code fragments as parameters, but I think that it would be hard for readers to cope with authors who would pass such code unbalanced fragments as chunk parameters2. I know that I couldn't cope with users doing such things, and although the GPL3 license prevents me from actually forbidding anyone from trying, if they want it to work they'll have to write the code themselves and not expect any support from me.  ^2.
Unfortunately, the full set of matching delimiters may vary from language to language. In certain C++ template contexts, < and > would count as delimiters, and yet in other contexts they would not.
This puts me in the unfortunate position of having to parse-somewhat all programming languages without knowing what they are!
However, if this universal mode-tracking is possible, then parsing the arguments would be trivial. Such a mode tracker is described in chapter 11 and used here with simplicity.

44a <parse_chunk_args[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function parse_chunk_args(language, text, values, mode,
  2  |   # local vars
  3  |   c, context, rest)
  4  | {
  5  |   «new-mode-tracker\v(context\v, language\v, mode\v) 58b»
  6  |   rest = mode_tracker(context, text, values);
  7  |   # extract values
  8  |   for(c=1; c <= context[0, "values"]; c++) {
  9  |     values[c] = context[0, "values", c];
  10  |   }
  11  |   return rest;
  12  | }
     |________________________________________________________________________


10.3 Expanding parameters in the text 
Within the body of the chunk, the parameters are referred to with: ${name} and ${address}. There is a strong case that a LaTeX style notation should be used, like \param{name} which would be expressed in the listing as =<\param{name}> and be rendered as ${name}. Such notation would make me go blind, but I do intend to adopt it.
We therefore need a function expand_chunk_args which will take a block of text, a list of permitted parameters, and the arguments which must substitute for the parameters. 
Here we split the text on ${ which means that all parts except the first will begin with a parameter name which will be terminated by }. The split function will consume the literal ${ in each case.

44b <expand_chunk_args()[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function expand_chunk_args(text, params, args,  
  2  |   p, text_array, next_text, v, t, l)
  3  | {
  4  |   if (split(text, text_array, "\\${")) {
  5  |     «substitute-chunk-args 45a»
  6  |   }
  7  | 
  8  |   return text;
  9  | }
     |________________________________________________________________________


First, we produce an associative array of substitution values indexed by parameter names. This will serve as a cache, allowing us to look up the replacement values as we extract each name.

45a <substitute-chunk-args[1](\v), lang=> ≡  45b▿
      ________________________________________________________________________
  1  | for(p in params) {
  2  |   v[params[p]]=args[p];
  3  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We accumulate substituted text in the variable text. As the first part of the split function is the part before the delimiter --- which is ${ in our case --- this part will never contain a parameter reference, so we assign this directly to the result kept in $text.

45b <substitute-chunk-args[2](\v) ⇑45a, lang=> +≡ ▵45a 45c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | text=text_array[1];
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then iterate over the remaining values in the array, and substitute each reference for it's argument.

45c <substitute-chunk-args[3](\v) ⇑45a, lang=> +≡ ▵45b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | for(t=2; t in text_array; t++) {
  6  |   «substitute-chunk-arg 45d»
  7  | }
     |________________________________________________________________________


After the split on ${ a valid parameter reference will consist of valid parameter name terminated by a close-brace }. A valid character name begins with the underscore or a letter, and may contain letters, digits or underscores.
A valid looking reference that is not actually the name of a parameter will be and not substituted. This is good because there is nothing to substitute anyway, and it avoids clashes when writing code for languages where ${...} is a valid construct --- such constructs will not be interfered with unless the parameter name also matches.

45d <substitute-chunk-arg[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
  2  |     l[1] in v) 
  3  | {
  4  |   text = text v[l[1]] substr(text_array[t], length(l[1])+2);
  5  | } else {
  6  |   text = text "${" text_array[t];
  7  | }
     |________________________________________________________________________


Chapter 11Language Modes & Quoting
lstlistings and fangle both recognize source languages, and perform some basic parsing and syntax highlighting in the rendered document1. although lstlisting supports many more languages  ^1. lstlistings can detect strings and comments within a language definition and perform suitable rendering, such as italics for comments, and visible-spaces within strings.
Fangle similarly can recognize strings, and comments, etc, within a language, so that any chunks included with \chunkref{a-chunk} or <a-chunk ?> can be suitably escape or quoted.
11.1 Modes explanation 
As an example, the C language has a few parse modes, which affect the interpretation of characters.
One parse mode is the string mode. The string mode is commenced by an un-escaped quotation mark " and terminated by the same. Within the string mode, only one additional mode can be commenced, it is the backslash mode \, which is always terminated by the following character.
Another mode is [ which is terminated by a ] (unless it occurs in a string).
Consider this fragment of C code:
do_something((things([x, y])<wide-overbrace>^(2. [ mode), get_other_things((a, "(all)"_(4. " mode)))<wide-overbrace>^(3. ( mode)))<wide-overbrace>^(1. ( mode)

Mode nesting prevents the close parenthesis in the quoted string (part 4) from terminating the parenthesis mode (part 3).
Each language has a set of modes, the default mode being the null mode. Each mode can lead to other modes.
11.2 Modes affect included chunks 
For instance, consider this chunk with language=perl:

47a <test:example-perl[1](\v), lang=perl> ≡ 
      ________________________________________________________________________
  1  | print "hello world $0\n";
     |________________________________________________________________________


If it were included in a chunk with language=sh, like this:

47b <test:example-sh[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | perl -e "«test:example-perl 47a»"
     |________________________________________________________________________


we might want fangle would to generate output like this:

48a <test:example-sh.result[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | perl -e "print \"hello world \$0\\n\";"
     |________________________________________________________________________


See that the double quote ", back-slash \ and $ have been quoted with a back-slash to protect them from shell interpretation.
If that were then included in a chunk with language=make, like this:

48b <test:example-makefile[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | target: pre-req
  2  | ↦«test:example-sh 47b»
     |________________________________________________________________________


We would need the output to look like this --- note the $$ as the single $ has been makefile-quoted with another $.

48c <test:example-makefile.result[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | target: pre-req
  2  | ↦perl -e "print \"hello world \$$0\\n\";"
     |________________________________________________________________________


11.3 Language Mode Definitions 
In order to make this work, we must define a mode-tracker supporting each language, that can detect the various quoting modes, and provide a transformation that may be applied to any included text so that included text will be interpreted correctly after any interpolation that it may be subject to at run-time.
For example, the sed transformation for text to be inserted into shell double-quoted strings would be something like:
s/\\/\\\\/g;s/$/\\$/g;s/"/\\"/g;
which would protect \ $ "
All modes definitions are stored in a single multi-dimensional hash called modes:
modes[language, mode, properties]
The first index is the language, and the second index is the mode. The third indexes hold properties such as terminators, possible submodes, transformations, and so forth.

48d <xmode:set-terminators[1](language\v, mode\v, terminators\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | modes["${language}", "${mode}", "terminators"]="${terminators}";
     |________________________________________________________________________



48e <xmode:set-submodes[1](language\v, mode\v, submodes\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | modes["${language}", "${mode}",  "submodes"]="${submodes}";
     |________________________________________________________________________


A useful set of mode definitions for a nameless general C-type language is shown here.
Don't be confused by the double backslash escaping needed in awk. One set of escaping is for the string, and the second set of escaping is for the regex.
To do: TODO: Add =<\mode{}> command which will allow us to signify that a string is
 regex and thus fangle will quote it for us.

Sub-modes are identified by a backslash, a double or single quote, various bracket styles or a /* comment; specifically: \ " ' { ( [ /*
For each of these sub-modes modes we must also identify at a mode terminator, and any sub-modes or delimiters that may be entered2. Because we are using the sub-mode characters as the mode identifier it means we can't currently have a mode character dependant on it's context; i.e. { can't behave differently when it is inside [.  ^2.

49a <common-mode-definitions[1](language\v\v), lang=> ≡  49b▿
      ________________________________________________________________________
  1  | modes[${language}, "",  "submodes"]="\\\\|\"|'|{|\\(|\\[";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
In the default mode, a comma surrounded by un-important white space is a delimiter of language items3. whatever a language item might be  ^3. Delimiters are used so that fangle can parse and recognise arguments individually.

49b <common-mode-definitions[2](language\v\v) ⇑49a, lang=> +≡ ▵49a 49d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  2  | modes[${language}, "",  "delimiters"]=" *, *";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
and should pass this test:
To do: Why do the tests run in ?(? mode and not ?? mode


49c <test:mode-definitions[1](\v), lang=> ≡  50g⊳
      ________________________________________________________________________
  1  | parse_chunk_args("c-like", "1,2,3", a, "");
  2  | if (a[1] != "1") e++;
  3  | if (a[2] != "2") e++;
  4  | if (a[3] != "3") e++;
  5  | if (length(a) != 3) e++;
  6  | «pca-test.awk:summary 62d»
  7  | 
  8  | parse_chunk_args("c-like", "joe, red", a, "");
  9  | if (a[1] != "joe") e++;
  10  | if (a[2] != "red") e++;
  11  | if (length(a) != 2) e++;
  12  | «pca-test.awk:summary 62d»
  13  | 
  14  | parse_chunk_args("c-like", "${colour}", a, "");
  15  | if (a[1] != "${colour}") e++;
  16  | if (length(a) != 1) e++;
  17  | «pca-test.awk:summary 62d»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
11.3.1 Backslash 
The backslash mode has no submodes or delimiters, and is terminated by any character. Note that we are not so much interested in evaluating or interpolating content as we are in delineating content. It is no matter that a double backslash (\\) may represent a single backslash while a backslash-newline may represent white space, but it does matter that the newline in a backslash newline should not be able to terminate a C pre-processor statement; and so the newline will be consumed by the backslash terminator however it may uultimately be interpreted.

49d <common-mode-definitions[3](language\v\v) ⇑49a, lang=> +≡ ▵49b 50f⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  3  | modes[${language}, "\\", "terminators"]=".";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
11.3.2 Strings 
Common languages support two kinds of strings quoting, double quotes and single quotes.
In a string we have one special mode, which is the backslash. This may escape an embedded quote and prevent us thinking that it should terminate the string.

50a <mode:common-string[1](language\v, quote\v\v), lang=> ≡  50b▿
      ________________________________________________________________________
  1  | modes[${language}, ${quote}, "submodes"]="\\\\";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Otherwise, the string will be terminated by the same character that commenced it.

50b <mode:common-string[2](language\v, quote\v\v) ⇑50a, lang=> +≡ ▵50a 50c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  2  | modes[${language}, ${quote}, "terminators"]=${quote};
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
In C type languages, certain escape sequences exist in strings. We need to define mechanism to enclode any chunks included in this mode using those escape sequences. These are expressed in two parts, s meaning search, and r meaning replace.
The first substitution is to replace a backslash with a double backslash. We do this first as other substitutions may introduce a backslash which we would not then want to escape again here.
Note: Backslashes need double-escaping in the search pattern but not in the replacement string, hence we are replacing a literal \ with a literal \\.

50c <mode:common-string[3](language\v, quote\v\v) ⇑50a, lang=> +≡ ▵50b 50d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  3  | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\\\";
  4  | escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="\\\\";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If the quote character occurs in the text, it should be preceded by a backslash, otherwise it would terminate the string unexpectedly.

50d <mode:common-string[4](language\v, quote\v\v) ⇑50a, lang=> +≡ ▵50c 50e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]=${quote};
  6  | escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="\\" ${quote};
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Any newlines in the string, must be replaced by \n.

50e <mode:common-string[5](language\v, quote\v\v) ⇑50a, lang=> +≡ ▵50d
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  7  | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\n";
  8  | escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="\\n";
     |________________________________________________________________________


For the common modes, we define this string handling for double and single quotes.

50f <common-mode-definitions[4](language\v\v) ⇑49a, lang=> +≡ ⊲49d 51b⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | «mode:common-string\v(${language}\v, "\""\v) 50a»
  5  | «mode:common-string\v(${language}\v, "'"\v) 50a»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Working strings should pass this test:

50g <test:mode-definitions[2](\v) ⇑49c, lang=> +≡ ⊲49c 57c⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  18  | parse_chunk_args("c-like", "say \"I said, \\\"Hello, how are you\\\".\", for me", a, "");
  19  | if (a[1] != "say \"I said, \\\"Hello, how are you\\\".\"") e++;
  20  | if (a[2] != "for me") e++;
  21  | if (length(a) != 2) e++;
  22  | «pca-test.awk:summary 62d»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
11.3.3 Parentheses, Braces and Brackets 
Where quotes are closed by the same character, parentheses, brackets and braces are closed by an alternate character.

51a <mode:common-brackets[1](language\v, open\v, close\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | modes[${language}, ${open},  "submodes" ]="\\\\|\"|{|\\(|\\[|'|/\\*";
  2  | modes[${language}, ${open},  "delimiters"]=" *, *";
  3  | modes[${language}, ${open},  "terminators"]=${close};
     |________________________________________________________________________


Note that the open is NOT a regex but the close token IS. 
To do: When we can quote regex we won't have to put the slashes in here


51b <common-mode-definitions[5](language\v\v) ⇑49a, lang=> +≡ ⊲50f
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  6  | «mode:common-brackets\v(${language}\v, "{"\v, "}"\v) 51a»
  7  | «mode:common-brackets\v(${language}\v, "["\v, "\\]"\v) 51a»
  8  | «mode:common-brackets\v(${language}\v, "("\v, "\\)"\v) 51a»
     |________________________________________________________________________


11.3.4 Customizing Standard Modes 

51c <mode:add-submode[1](language\v, mode\v, submode\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | modes[${language}, ${mode}, "submodes"] = modes[${language}, ${mode}, "submodes"] "|" ${submode};
     |________________________________________________________________________



51d <mode:add-escapes[1](language\v, mode\v, search\v, replace\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "s"]=${search};
  2  | escapes[${language}, ${mode},   escapes[${language}, ${mode}], "r"]=${replace};
     |________________________________________________________________________



11.3.5 Comments 
We can define /* comment */ style comments and //comment style comments to be added to any language:

51e <mode:multi-line-comments[1](language\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «mode:add-submode\v(${language}\v, ""\v, "/\\*"\v) 51c»
  2  | modes[${language}, "/*", "terminators"]="\\*/";
     |________________________________________________________________________



51f <mode:single-line-slash-comments[1](language\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «mode:add-submode\v(${language}\v, ""\v, "//"\v) 51c»
  2  | modes[${language}, "//", "terminators"]="\n";
  3  | «mode:add-escapes\v(${language}\v, "//"\v, "\n"\v, "\n//"\v) 51d»
     |________________________________________________________________________


We can also define # comment style comments (as used in awk and shell scripts) in a similar manner.
To do: I'm having to use # for hash and ¯extbackslash{} for  and have hacky work-arounds in the parser for now


51g <mode:add-hash-comments[1](language\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «mode:add-submode\v(${language}\v, ""\v, "#"\v) 51c»
  2  | modes[${language}, "#", "terminators"]="\n";
  3  | «mode:add-escapes\v(${language}\v, "#"\v, "\n"\v, "\n#"\v) 51d»
     |________________________________________________________________________


In C, the # denotes pre-processor directives which can be multi-line

51h <mode:add-hash-defines[1](language\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «mode:add-submode\v(${language}\v, ""\v, "#"\v) 51c»
  2  | modes[${language}, "#", "submodes" ]="\\\\";
  3  | modes[${language}, "#", "terminators"]="\n";
  4  | «mode:add-escapes\v(${language}\v, "#"\v, "\n"\v, "\\\\\n"\v) 51d»
     |________________________________________________________________________



52a <mode:quote-dollar-escape[1](language\v, quote\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\$";
  2  | escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="\\$";
     |________________________________________________________________________


We can add these definitions to various languages

52b <mode-definitions[1](\v), lang=> ≡  53a⊳
      ________________________________________________________________________
  1  | «common-mode-definitions\v("c-like"\v) 49a»
  2  | 
  3  | «common-mode-definitions\v("c"\v) 49a»
  4  | «mode:multi-line-comments\v("c"\v) 51e»
  5  | «mode:single-line-slash-comments\v("c"\v) 51f»
  6  | «mode:add-hash-defines\v("c"\v) 51h»
  7  | 
  8  | «common-mode-definitions\v("awk"\v) 49a»
  9  | «mode:add-hash-comments\v("awk"\v) 51g»
  10  | «mode:add-naked-regex\v("awk"\v) 52g»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The awk definitions should allow a comment block like this:

52c <test:comment-quote[1](\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | # Comment: «test:comment-text 52d»
     |________________________________________________________________________



52d <test:comment-text[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | Now is the time for
  2  | the quick brown fox to bring lemonade
  3  | to the party
     |________________________________________________________________________


to come out like this:

52e <test:comment-quote:result[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # Comment: Now is the time for
  2  | #the quick brown fox to bring lemonade
  3  | #to the party
     |________________________________________________________________________


The C definition for such a block should have it come out like this:

52f <test:comment-quote:C-result[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # Comment: Now is the time for\
  2  | the quick brown fox to bring lemonade\
  3  | to the party
     |________________________________________________________________________


11.3.6 Regex 
This pattern is incomplete, but meant to detect naked regular expressions in awk and perl; e.g. /.*$/, however required capabilities are not present.
Current it only detects regexes anchored with ^ as used in fangle.
For full regex support, modes need to be named not after their starting character, but some other more fully qualified name.

52g <mode:add-naked-regex[1](language\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «mode:add-submode\v(${language}\v, ""\v, "/\\^"\v) 51c»
  2  | modes[${language}, "/^", "terminators"]="/";
     |________________________________________________________________________


11.3.7 Perl 

53a <mode-definitions[2](\v) ⇑52b, lang=> +≡ ⊲52b 53b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  11  | «common-mode-definitions\v("perl"\v) 49a»
  12  | «mode:multi-line-comments\v("perl"\v) 51e»
  13  | «mode:add-hash-comments\v("perl"\v) 51g»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Still need to add add s/, submode /, terminate both with //. This is likely to be impossible as perl regexes can contain perl.
11.3.8 sh 
Shell single-quote strings are different to other strings and have no escape characters. The only special character is the single quote ' which always closes the string. Therefore we cannot use <common-mode-definitions\v("sh"\v) 49a> but we will invoke most of it's definition apart from single-quote strings. 

53b <mode-definitions[3](\v) ⇑52b, lang=awk> +≡ ▵53a 54a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  14  | modes["sh", "",  "submodes"]="\\\\|\"|'|{|\\(|\\[|\\$\\(";
  15  | modes["sh", "\\", "terminators"]=".";
  16  | 
  17  | modes["sh", "\"", "submodes"]="\\\\|\\$\\(";
  18  | modes["sh", "\"", "terminators"]="\"";
  19  | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\\\\";
  20  | escapes["sh", "\"",   escapes["sh", "\""], "r"]="\\\\";
  21  | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\"";
  22  | escapes["sh", "\"",   escapes["sh", "\""], "r"]="\\" "\"";
  23  | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\n";
  24  | escapes["sh", "\"",   escapes["sh", "\""], "r"]="\\n";
  25  | 
  26  | modes["sh", "'", "terminators"]="'";
  27  | escapes["sh", "'", ++escapes["sh", "'"], "s"]="'";
  28  | escapes["sh", "'",   escapes["sh", "'"], "r"]="'\\'" "'";
  29  | «mode:common-brackets\v("sh"\v, "$("\v, "\\)"\v) 51a»
  30  | «mode:add-tunnel\v("sh"\v, "$("\v, ""\v) 53c»
  31  | «mode:common-brackets\v("sh"\v, "{"\v, "}"\v) 51a»
  32  | «mode:common-brackets\v("sh"\v, "["\v, "\\]"\v) 51a»
  33  | «mode:common-brackets\v("sh"\v, "("\v, "\\)"\v) 51a»
  34  | «mode:add-hash-comments\v("sh"\v) 51g»
  35  | «mode:quote-dollar-escape\v("sh"\v, "\""\v) 52a»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The definition of add-tunnel is:

53c <mode:add-tunnel[1](language\v, mode\v, tunnel\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "tunnel"]=${tunnel};
     |________________________________________________________________________


11.3.9 Make 
BUGS: makefile tab mode is terminated by newline, but chunks never end in a newline! So tab mode is never closed unless there is a trailing blank line!
For makefiles, we currently recognize 2 modes: the null mode and ↦ mode, which is tabbed mode and contains the makefile recipie. 


54a <mode-definitions[4](\v) ⇑52b, lang=awk> +≡ ⊲53b 54b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  36  | modes["make", "",  "submodes"]="↦";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
In the null mode the only escape is $ which must be converted to $$, and hash-style comments. POSIX requires that line-continuations extend hash-style comments and so fangle-style transformations to replicate the hash at the start of each line is not strictly required, however it is harmless, easier to read, and required by some implementations of make which do not implement POSIX requirements correctly.

54b <mode-definitions[5](\v) ⇑52b, lang=awk> +≡ ▵54a 56a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  37  | escapes["make", "", ++escapes["make", ""], "s"]="\\$";
  38  | escapes["make", "", escapes["make", ""], "r"]="$$";
  39  | «mode:add-hash-comments\v("make"\v) 51g»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Tabbed mode is harder to manage, as the GNU Make Manual says in the section on splitting lines4. http://www.gnu.org/s/hello/manual/make/Splitting-Lines.html  ^4. There is no obvious way to escape a multi-line text that occurs as part of a makefile recipe.
Traditionally, if the newline's in the shell script all occur at points of top-level shell syntax, then we could replace them with  ;\n↦and largely get the right effect.

54c <test:make:1[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦«test:make:1-inc\v($@) 54d»
     |________________________________________________________________________





54d <test:make:1-inc[1](target\v\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | if test "${target}" = "all"
  2  | then echo yes, all
  3  | else echo "${target}" | sed -e '/^\//{
  4  |                                  p;s/^/../
  5  |                        }'
  6  | fi
     |________________________________________________________________________


The two chunks above could reasonably produce something like this:

54e <test:make:1.result.bad[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦if test "$@" = "all" ;\
  4  | ↦then echo yes, all ;\
  5  | ↦else echo "$@" | sed -e '/^\//{ ;\
  6  | ↦                           p;s/^/../ ;\
  7  | ↦                         }' ;\
  8  | ↦fi
     |________________________________________________________________________


However ;\ is not a proper continuation inside a multi-line sed script. There is no simple continuation that fangle could use — and in any case it would depend on what type of quote marks were used in the bash that contained the sed. 
We would prefer to use a more intuitive single backslash at the end of the line, giving these results.

54f <test:make:1.result[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦if test "$$@" = "all"\
  4  | ↦ then echo yes, all\
  5  | ↦ else echo "$$@" | sed -e '/^\//{\
  6  | ↦                                  p;s/^/../\
  7  | ↦                        }'\
  8  | ↦ fi
     |________________________________________________________________________


The difficulty lies in the way that make handles the recipe. Each line of the recipe is invoked as a separate shell command (using $(SHELL) -c) unless the last character of the line was a backslash. In such a case, the backslash and the newline and the nextline are handed to the shell (although the tab character that prefixes the next line is stripped).
This behaviour makes it impossible to hand a newline character to the shell unless it is prefixed by a backslash. If an included shell fragment contained strings with literal newline characters then there would be no easy way to escape these and preserve the value of the string.
A different style of makefile construction might be used — the recipe could be stored in a target specific variable5. http://www.gnu.org/s/hello/manual/make/Target_002dspecific.html  ^5 which contains the recipe with a more normal escape mechanism.
A better solution is to use a shell helper that strips the back-slash which precedes the newline character and then passes the arguments to the normal shell.
Because this is a simple operation and because bash is so flexible, this can be managed in a single line within the makefile itself.
As a newline will only exist when preceded by the backslash, and as the purpose of the backash is to protect th newline, that is needed is to remove any backslash that is followed by a newline.
Bash is capable of doing this with its pattern substitution. If A=123:=456:=789 then ${A//:=/=} will be 123=456=789. We don't want to just perform the substitution in a single variable but in fact in all of $@'', however bash will repeat substitution over all members of an array, so this is done automatically.
In bash, $'\012' represents the newline character (expressed as an octal escape sequence), so this expression will replace backslash-newline with a single newline.

55a <fix-requote-newline[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | "${@//\\$'\012'/$'\012'}"
     |________________________________________________________________________


We use this as part of a larger statement which will invoke such a transformed command ine using any particular shell. The trailing -- prevents any options in the command line from being interpreted as options to our bash command — instead they will be transformed and passed to the inner shell which is invoked with exec so that our fixup-shell does not hang around longer than is needed.

55b <fix-make-shell[1](shell\v\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | bash -c 'exec ${shell} «fix-requote-newline 55a»' --
     |________________________________________________________________________


We can then cinlude a line like this in our makefiles. We should rather pass $(SHELL) as the chunk argument than bash, but currently fangle will not track which nested-inclusion level the argument comes from and will quote the $ in $(SHELL) in the same way it quotes a $ that may occur in the bash script, so this would come out as $$(SHELL) and have the wrong effect.

55c <make-fix-make-shell[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | SHELL:=«fix-make-shell\v(bash\v) 55b»
     |________________________________________________________________________


The full escaped and quoted text with $(SHELL) and suitale for inclusion in a Makefile is:
SHELL:=bash -c 'exec $(SHELL) "$${@//\\$$'\''\012'\''/$$'\''\012'\''}"' --
Based on this, we just need to escape newlines (in tabbed mode) with a regular backslash:
Note that terminators applies to literal, not included text, escapes apply to included, not literal text; also that the tab character is hard-wired into the pattern, and that the make variable .RECIPEPREFIX might change this to something else.

56a <mode-definitions[6](\v) ⇑52b, lang=awk> +≡ ⊲54b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  40  | modes["make", "↦", "terminators"]="\\n";
  41  | escapes["make", "↦", ++escapes["make", "↦"], "s"]="\\n";
  42  | escapes["make", "↦",   escapes["make", "↦"], "r"]="\\\n↦";
     |________________________________________________________________________


With this improved quoting, the test on 54c will actually produce this:

56b <test:make:1.result-actual[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦if test "$$@" = "all"\
  4  | ↦ then echo yes, all\
  5  | ↦ else echo not all\
  6  | ↦ fi
     |________________________________________________________________________


The chunk argument $@ has been quoted (which would have been fine if we were passing the name of a shell variable), and the other shell lines are (harmlessly) indented by 1 space as part of fangle indent-matching which should have taken into account the expanded tab size, and should generally take into account the expanded prefix of the line whose indent it is trying to match, but which in this case we want to have no effect at all!
To do: The $@ was passed from a make fragment. In what cases should it be converted to $$@?
Do we need to track the language of sources of arguments?

A more ugly work-around until this problem can be solved would be to use this notation:

56c <test:make:2[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦ARG="$@"; «test:make:1-inc\v($ARG) 54d»
     |________________________________________________________________________


which produces this output which is more useful (because it works):

56d <test:make:2.result[1](\v), lang=make> ≡ 
      ________________________________________________________________________
  1  | all:
  2  | ↦echo making
  3  | ↦ARG="$@"; if test "$$ARG" = "all"\
  4  | ↦           then echo yes, all\
  5  | ↦           else echo "$$ARG" | sed -e '/^\//{\
  6  | ↦                                            p;s/^/../\
  7  | ↦                                  }'\
  8  | ↦           fi
     |________________________________________________________________________


11.4 Quoting scenarios 
11.4.1 Direct quoting 
He we give examples of various quoting scenarios and discuss what the expected outcome might be and how this could be obtained.

56e <test:q:1[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | echo "$(«test:q:1-inc 57a»)"
     |________________________________________________________________________



57a <test:q:1-inc[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | echo "hello"
     |________________________________________________________________________


Should this examples produce echo "$(echo "hello")" or echo "$(echo \"hello\")" ?
This depends on what the author intended, but we must provde a way to express that intent.
We might argue that as both chunks have lang=sh the intent must have been to quote the included chunk — but consider that this might be shell script that writes shell script.
If <test:q:1-inc 57a> had lang=text then it certainly would have been right to quote it, which leads us to ask: in what ways can we reduce quoting if lang of the included chunk is compatible with the lang of the including chunk?
If we take a completely nested approach then even though $( mode might do no quoting of it's own, " mode will still do it's own quoting. We need a model where the nested $( mode will prevent " from quoting.
This leads rise to the tunneling feature. In bash, the $( gives rise to a new top-level parsing scenario, so we need to enter the null mode, and also ignore any quoting and then undo-this when the $( mode is terminated by the corresponding close ).
We shall say that tunneling is when a mode in a language ignores other modes in the same language and arrives back at an earlier null mode of the same language.
In example <test:q:1 56e> above, the nesting of modes is: null, ", $(
When mode $( is commenced, the stack of nest modes will be traversed. If the null mode can be found in the same language, without the language varying, then a tunnel will be established so that the intervening modes, " in this case, can be skipped when the modes are enumerated to quote the texted being emitted.
In such a case, the correct result would be:

57b <test:q:1.result[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | echo "$(echo "hello")"
     |________________________________________________________________________


11.5 Some tests 
Also, the parser must return any spare text at the end that has not been processed due to a mode terminator being found.

57c <test:mode-definitions[3](\v) ⇑49c, lang=> +≡ ⊲50g 57d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  23  | rest = parse_chunk_args("c-like", "1, 2, 3) spare", a, "(");
  24  | if (a[1] != 1) e++;
  25  | if (a[2] != 2) e++;
  26  | if (a[3] != 3) e++;
  27  | if (length(a) != 3) e++;
  28  | if (rest != " spare") e++;
  29  | «pca-test.awk:summary 62d»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We must also be able to parse the example given earlier.

57d <test:mode-definitions[4](\v) ⇑49c, lang=> +≡ ▵57c
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  30  | parse_chunk_args("c-like", "things[x, y], get_other_things(a, \"(all)\"), 99", a, "(");
  31  | if (a[1] != "things[x, y]") e++;
  32  | if (a[2] != "get_other_things(a, \"(all)\")") e++;
  33  | if (a[3] != "99") e++;
  34  | if (length(a) != 3) e++;
  35  | «pca-test.awk:summary 62d»
     |________________________________________________________________________


11.6 A non-recursive mode tracker 
As each chunk is output a new mode tracker for that language is initialized in it's normal state. As text is output for that chunk the output mode is tracked. When a new chunk is included, a transformation appropriate to that mode is selected and pushed onto a stack of transformations. Any text to be output is passed through this stack of transformations.
It remains to consider if the chunk-include function should return it's generated text so that the caller can apply any transformations (and formatting), or if it should apply the stack of transformations itself.
Note that the transformed included text should have the property of not being able to change the mode in the current chunk.
To do: Note chunk parameters should probably also be transformed

11.6.1 Constructor 
The mode tracker holds its state in a stack based on a numerically indexed hash. This function, when passed an empty hash, will intialize it.

58a <new_mode_tracker()[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function new_mode_tracker(context, language, mode) {
  2  |   context[""] = 0;
  3  |   context[0, "language"] = language;
  4  |   context[0, "mode"] = mode;
  5  | }
     |________________________________________________________________________


Awk functions cannot return an array, but arrays are passed by reference. Because of this we must create the array first and pass it in, so we have a fangle macro to do this:

58b <new-mode-tracker[1](context\v, language\v, mode\v\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | «awk-delete-array\v(${context}\v) 37d»
  2  | new_mode_tracker(${context}, ${language}, ${mode});
     |________________________________________________________________________


11.6.2 Management 
And for tracking modes, we dispatch to a mode-tracker action based on the current language

58c <mode_tracker[1](\v), lang=awk> ≡  59a⊳
      ________________________________________________________________________
  1  | function push_mode_tracker(context, language, mode,
  2  |   # local vars
  3  |   top)
  4  | {
  5  |   if (! ("" in context)) {
  6  |     «new-mode-tracker\v(context\v, language\v, mode\v) 58b»
  7  |     return;
  8  |   } else {
  9  |     top = context[""];
  10  | #    if (context[top, "language"] == language && mode=="") mode = context[top, "mode"];
  11  |     if (context[top, "language"] == language && context[top, "mode"] == mode) return top - 1;
  12  |     old_top = top;
  13  |     top++;
  14  |     context[top, "language"] = language;
  15  |     context[top, "mode"] = mode;
  16  |     context[""] = top;
  17  |   }
  18  |   return old_top;
  19  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

59a <mode_tracker[2](\v) ⇑58c, lang=> +≡ ⊲58c 59b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  20  | function dump_mode_tracker(context,  
  21  |   c, d)
  22  | {
  23  |   for(c=0; c <= context[""]; c++) {
  24  |     printf(" %2d   %s:%s\n", c, context[c, "language"], context[c, "mode"]) > "/dev/stderr";
  25  | #    for(d=1; ( (c, "values", d) in context); d++) {
  26  | #      printf("   %2d %s\n", d, context[c, "values", d]) > "/dev/stderr";
  27  | #    }
  28  |   }
  29  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

59b <mode_tracker[3](\v) ⇑58c, lang=> +≡ ▵59a 63b⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  30  | function pop_mode_tracker(context, context_origin)
  31  | {
  32  |   if ( (context_origin) && ("" in context) && context[""] != (1+context_origin) && context[""] != context_origin) {
  33  |     print "Context level: " context[""] ", origin: " context_origin "\n" > "/dev/stderr"
  34  |     return 0;
  35  |   }
  36  |   context[""] = context_origin;
  37  |   return 1;
  38  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
This implies that any chunk must be syntactically whole; for instance, this is fine:

59c <test:whole-chunk[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (1) {
  2  |   «test:say-hello 59d»
  3  | }
     |________________________________________________________________________



59d <test:say-hello[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | print "hello";
     |________________________________________________________________________


But this is not fine; the chunk <test:hidden-else 59f> is not properly cromulent.

59e <test:partial-chunk[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (1) {
  2  |   «test:hidden-else 59f»
  3  | }
     |________________________________________________________________________



59f <test:hidden-else[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  |   print "I'm fine";
  2  | } else {
  3  |   print "I'm not";
     |________________________________________________________________________


These tests will check for correct behaviour:

59g <test:cromulence[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | echo Cromulence test
  2  | passtest $FANGLE -Rtest:whole-chunk $TXT_SRC &>/dev/null || ( echo "Whole chunk failed" && exit 1 )
  3  | failtest $FANGLE -Rtest:partial-chunk $TXT_SRC &>/dev/null || ( echo "Partial chunk failed" && exit 1 )
     |________________________________________________________________________


11.6.3 Tracker 
We must avoid recursion as a language construct because we intend to employ mode-tracking to track language mode of emitted code, and the code is emitted from a function which is itself recursive, so instead we implement psuedo-recursion using our own stack based on a hash.

60a <mode_tracker()[1](\v), lang=awk> ≡  60b▿
      ________________________________________________________________________
  1  | function mode_tracker(context, text, values, 
  2  |   # optional parameters
  3  |   # local vars
  4  |   mode, submodes, language,
  5  |   cindex, c, a, part, item, name, result, new_values, new_mode, 
  6  |   delimiters, terminators)
  7  | {
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We could be re-commencing with a valid context, so we need to setup the state according to the last context.

60b <mode_tracker()[2](\v) ⇑60a, lang=> +≡ ▵60a 60e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  8  |   cindex = context[""] + 0;
  9  |   mode = context[cindex, "mode"];
  10  |   language = context[cindex, "language" ];
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
First we construct a single large regex combining the possible sub-modes for the current mode along with the terminators for the current mode.

60c <parse_chunk_args-reset-modes[1](\v), lang=> ≡  60d▿
      ________________________________________________________________________
  1  |   submodes=modes[language, mode, "submodes"];
  2  | 
  3  |   if ((language, mode, "delimiters") in modes) {
  4  |     delimiters = modes[language, mode, "delimiters"];
  5  |     if (length(submodes)>0) submodes = submodes "|";
  6  |     submodes=submodes delimiters;
  7  |   } else delimiters="";
  8  |   if ((language, mode, "terminators") in modes) {
  9  |     terminators = modes[language, mode, "terminators"];
  10  |     if (length(submodes)>0) submodes = submodes "|";
  11  |     submodes=submodes terminators;
  12  |   } else terminators="";
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If we don't find anything to match on --- probably because the language is not supported --- then we return the entire text without matching anything.

60d <parse_chunk_args-reset-modes[2](\v) ⇑60c, lang=> +≡ ▵60c
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  13  |  if (! length(submodes)) return text;
     |________________________________________________________________________



60e <mode_tracker()[3](\v) ⇑60a, lang=> +≡ ▵60b 60f▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  11  | «parse_chunk_args-reset-modes 60c»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then iterate the text (until there is none left) looking for sub-modes or terminators in the regex.

60f <mode_tracker()[4](\v) ⇑60a, lang=> +≡ ▵60e 61a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  12  |   while((cindex >= 0) && length(text)) {
  13  |     if (match(text, "(" submodes ")", a)) {
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
A bug that creeps in regularly during development is bad regexes of zero length which result in an infinite loop (as no text is consumed), so I catch that right away with this test.

61a <mode_tracker()[5](\v) ⇑60a, lang=> +≡ ⊲60f 61b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  14  |       if (RLENGTH<1) {
  15  |         error(sprintf("Internal error, matched zero length submode, should be impossible - likely regex computation error\n" \
  16  |                 "Language=%s\nmode=%s\nmatch=%s\n", language, mode, submodes));
  17  |       }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
part is defined as the text up to the sub-mode or terminator, and this is appended to item --- which is the current text being gathered. If a mode has a delimiter, then item is reset each time a delimiter is found.
("hello_item, there_item")<wide-overbrace>^item,  (he said.)<wide-overbrace>^item

61b <mode_tracker()[6](\v) ⇑60a, lang=> +≡ ▵61a 61c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  18  |       part = substr(text, 1, RSTART -1);
  19  |       item = item part;
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We must now determine what was matched. If it was a terminator, then we must restore the previous mode.

61c <mode_tracker()[7](\v) ⇑60a, lang=> +≡ ▵61b 61d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  20  |       if (match(a[1], "^" terminators "$")) {
  21  | #printf("%2d EXIT  MODE [%s] by [%s] [%s]\n", cindex, mode, a[1], text) > "/dev/stderr"
  22  |         context[cindex, "values", ++context[cindex, "values"]] = item;
  23  |         delete context[cindex];
  24  |         context[""] = --cindex;
  25  |         if (cindex>=0) {
  26  |           mode = context[cindex, "mode"];
  27  |           language = context[cindex, "language"];
  28  |           «parse_chunk_args-reset-modes 60c»
  29  |         }
  30  |         item = item a[1];
  31  |         text = substr(text, 1 + length(part) + length(a[1]));
  32  |       }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If a delimiter was matched, then we must store the current item in the parsed values array, and reset the item.

61d <mode_tracker()[8](\v) ⇑60a, lang=> +≡ ▵61c 61e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  33  |       else if (match(a[1], "^" delimiters "$")) {
  34  |         if (cindex==0) {
  35  |           context[cindex, "values", ++context[cindex, "values"]] = item;
  36  |           item = "";
  37  |         } else {
  38  |           item = item a[1];
  39  |         }
  40  |         text = substr(text, 1 + length(part) + length(a[1]));
  41  |       }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
otherwise, if a new submode is detected (all submodes have terminators), we must create a nested parse context until we find the terminator for this mode.

61e <mode_tracker()[9](\v) ⇑60a, lang=> +≡ ▵61d 62a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  42  |  else if ((language, a[1], "terminators") in modes) {
  43  |         #check if new_mode is defined
  44  |         item = item a[1];
  45  | #printf("%2d ENTER MODE [%s] in [%s]\n", cindex, a[1], text) > "/dev/stderr"
  46  |         text = substr(text, 1 + length(part) + length(a[1]));
  47  |         context[""] = ++cindex;
  48  |         context[cindex, "mode"] = a[1];
  49  |         context[cindex, "language"] = language;
  50  |         mode = a[1];
  51  |         «parse_chunk_args-reset-modes 60c»
  52  |       } else {
  53  |         error(sprintf("Submode '%s' set unknown mode in text: %s\nLanguage %s Mode %s\n", a[1], text, language, mode));
  54  |         text = substr(text, 1 + length(part) + length(a[1]));
  55  |       }
  56  |     }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
In the final case, we parsed to the end of the string. If the string was entire, then we should have no nested mode context, but if the string was just a fragment we may have a mode context which must be preserved for the next fragment. Todo: Consideration ought to be given if sub-mode strings are split over two fragments.

62a <mode_tracker()[10](\v) ⇑60a, lang=> +≡ ⊲61e
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  57  | else {
  58  |       context[cindex, "values", ++context[cindex, "values"]] = item text;
  59  |       text = "";
  60  |       item = "";
  61  |     }
  62  |   }
  63  | 
  64  |   context["item"] = item;
  65  | 
  66  |   if (length(item)) context[cindex, "values", ++context[cindex, "values"]] = item;
  67  |   return text;
  68  | }
     |________________________________________________________________________


11.6.3.1 One happy chunk 
All the mode tracker chunks are referred to here:

62b <mode-tracker[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «new_mode_tracker() 58a»
  2  | «mode_tracker() 60a»
     |________________________________________________________________________


11.6.3.2 Tests 
We can test this function like this:

62c <pca-test.awk[1](\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | «error() 38a»
  2  | «mode-tracker 62b»
  3  | «parse_chunk_args() ?»
  4  | BEGIN {
  5  |   SUBSEP=".";
  6  |   «mode-definitions 52b»
  7  | 
  8  |   «test:mode-definitions 49c»
  9  | }
     |________________________________________________________________________



62d <pca-test.awk:summary[1](\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | if (e) {
  2  |   printf "Failed " e
  3  |   for (b in a) {
  4  |     print "a[" b "] => " a[b];
  5  |   }
  6  | } else {
  7  |   print "Passed"
  8  | }
  9  | split("", a);
  10  | e=0;
     |________________________________________________________________________


which should give this output:

63a <pca-test.awk-results[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | a[foo.quux.quirk] => 
  2  | a[foo.quux.a] => fleeg
  3  | a[foo.bar] => baz
  4  | a[etc] => 
  5  | a[name] => freddie
     |________________________________________________________________________


11.7 Escaping and Quoting 
For the time being and to get around TeXmacs inability to export a TAB character, the right arrow ↦ whose UTF-8 sequence is ...
To do: complete

Another special character is used, the left-arrow ↤ with UTF-8 sequence 0xE2 0x86 0xA4 is used to strip any preceding white space as a way of un-tabbing and removing indent that has been applied — this is important for bash here documents, and the like. It's a filthy hack.
To do: remove the hack


63b <mode_tracker[4](\v) ⇑58c, lang=> +≡ ⊲59b 63c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

  39  | function untab(text) {
  40  |   gsub("[[:space:]]*\xE2\x86\xA4","", text);
  41  |   return text;
  42  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Each nested mode can optionally define a set of transforms to be applied to any text that is included from another language.
This code can perform transforms from index c downwards.

63c <mode_tracker[5](\v) ⇑58c, lang=awk> +≡ ▵63b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  43  | function transform_escape(context, text, top,
  44  |   c, cp, cpl, s, r)
  45  | {
  46  |   for(c = top; c >= 0; c--) {
  47  |     if ( (context[c, "language"], context[c, "mode"]) in escapes) {
  48  |       cpl = escapes[context[c, "language"], context[c, "mode"]];
  49  |       for (cp = 1; cp <= cpl; cp ++) {
  50  |         s = escapes[context[c, "language"], context[c, "mode"], cp, "s"];
  51  |         r = escapes[context[c, "language"], context[c, "mode"], cp, "r"];
  52  |         if (length(s)) {
  53  |           gsub(s, r, text);
  54  |         }
  55  |         if ( (context[c, "language"], context[c, "mode"], cp, "t") in escapes ) {
  56  |           quotes[src, "t"] = escapes[context[c, "language"], context[c, "mode"], cp, "t"];
  57  |         }
  58  |       }
  59  |     }
  60  |   }
  61  |   return text;
  62  | }
  63  | function dump_escaper(quotes, r, cc) {
  64  |   for(cc=1; cc<=c; cc++) {
  65  |     printf("%2d s[%s] r[%s]\n", cc, quotes[cc, "s"], quotes[cc, "r"]) > "/dev/stderr"
  66  |   }
  67  | }
     |________________________________________________________________________



64a <test:escapes[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | echo escapes test
  2  | passtest $FANGLE -Rtest:comment-quote $TXT_SRC &>/dev/null || ( echo "Comment-quote failed" && exit 1 )
     |________________________________________________________________________


Chapter 12Recognizing Chunks
Fangle recognizes noweb chunks, but as we also want better LaTeX integration we will recognize any of these:
•  notangle chunks matching the pattern ^<<.*?>>=
•  chunks beginning with \begin{lstlistings}, possibly with \Chunk{...} on the previous line
•  an older form I have used, beginning with \begin{Chunk}[options] --- also more suitable for plain LaTeX users1. Is there such a thing as plain LaTeX?  ^1. 
12.1 Chunk start 
The variable chunking is used to signify that we are processing a code chunk and not document. In such a state, input lines will be assigned to the current chunk; otherwise they are ignored.
12.1.1 TeXmacs 
We don't handle TeXmacs files natively yet, but rather instead emit unicode character sequences to mark up the text-export file which we do process.
These hacks detect the unicode character sequences and retro-fit in the old TeX parsing.
We convert ↦ into a tab character.

65a <recognize-chunk[1](\v), lang=> ≡  65b▿
      ________________________________________________________________________

  1  | #/\n/ {
  2  | #  gsub("\n*$","");
  3  | #  gsub("\n", " ");
  4  | #}
  5  | #===
  6  | /\xE2\x86\xA6/ {
  7  |   gsub("\\xE2\\x86\\xA6", "\x09");
  8  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
TeXmacs back-tick handling is obscure, and a cut-n-paste back-tick from a shell window comes out as a unicode sequence2. that won't export to html, except as a NULL character (literal 0x00)  ^2 that is fixed-up here.

65b <recognize-chunk[2](\v) ⇑65a, lang=> +≡ ▵65a 66a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  9  | 
  10  | /\xE2\x80\x98/ {
  11  |   gsub("\\xE2\\x80\\x98", "‘");
  12  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
In the TeXmacs output, the start of a chunk will appear like this:
  5b<example-chunk^K[1](arg1,^K arg2^K^K), lang=C> ≡
We detect the the start of a TeXmacs chunk by detecting the ≡ symbol which occurs near the end of the line. We obtain the chunk name, the chunk parameters, and the chunk language.

66a <recognize-chunk[3](\v) ⇑65a, lang=> +≡ ⊲65b 66b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  13  | 
  14  | /\xE2\x89\xA1/ {
  15  |   if (match($0, "^ *([^[ ]* |)<([^[ ]*)\\[[0-9]*\\][(](.*)[)].*, lang=([^ ]*)>", line)) {
  16  |     next_chunk_name=line[2];
  17  |     get_texmacs_chunk_args(line[3], next_chunk_params);
  18  |     gsub(ARG_SEPARATOR ",? ?", ";", line[3]);
  19  |     params = "params=" line[3];
  20  |     if ((line[4])) {
  21  |       params = params ",language=" line[4]
  22  |     }
  23  |     get_tex_chunk_args(params, next_chunk_opts);
  24  |     new_chunk(next_chunk_name, next_chunk_opts, next_chunk_params);
  25  |     texmacs_chunking = 1;
  26  |   } else {
  27  |     # warning(sprintf("Unexpected chunk match: %s\n", $_))
  28  |   }
  29  |   next;
  30  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.1.2 lstlistings 
Our current scheme is to recognize the new lstlisting chunks, but these may be preceded by a \Chunk command which in L Y X is a more convenient way to pass the chunk name to the \begin{lstlistings} command, and a more visible way to specify other lstset settings.
The arguments to the \Chunk command are a name, and then a comma-seperated list of key-value pairs after the manner of \lstset. (In fact within the LaTeX \Chunk macro (section 17.2.1) the text name= is prefixed to the argument which is then literally passed to \lstset).

66b <recognize-chunk[4](\v) ⇑65a, lang=awk> +≡ ▵66a 66c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  31  | /^\\Chunk{/ {
  32  |   if (match($0, "^\\\\Chunk{ *([^ ,}]*),?(.*)}", line)) {
  33  |     next_chunk_name = line[1];
  34  |     get_tex_chunk_args(line[2], next_chunk_opts);
  35  |   }
  36  |   next;
  37  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also make a basic attempt to parse the name out of the \lstlistings[name=chunk-name] text, otherwise we fall back to the name found in the previous chunk command. This attempt is very basic and doesn't support commas or spaces or square brackets as part of the chunkname. We also recognize \begin{Chunk} which is convenient for some users3. but not yet supported in the LaTeX macros  ^3.

66c <recognize-chunk[5](\v) ⇑65a, lang=> +≡ ▵66b 67a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  38  | /^\\begin{lstlisting}|^\\begin{Chunk}/ {
  39  |   if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
  40  |     new_chunk(line[1]);
  41  |   } else {
  42  |     new_chunk(next_chunk_name, next_chunk_opts);
  43  |   }
  44  |   chunking=1;
  45  |   next;
  46  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.2 Chunk Body 
12.2.1 TeXmacs 
A chunk body in TeXmacs ends with |________... if it is the final chunklet of a chunk, or if there are further chunklets it ends with |\/\/\/... which is a depiction of a jagged line of torn paper.

67a <recognize-chunk[6](\v) ⇑65a, lang=> +≡ ⊲66c 67b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  47  | /^ *\|____________*/ && texmacs_chunking {
  48  |   active_chunk="";
  49  |   texmacs_chunking=0;
  50  |   chunking=0;
  51  | }
  52  | /^ *\|\/\\/ && texmacs_chunking {
  53  |   texmacs_chunking=0;
  54  |   chunking=0;
  55  |   active_chunk="";
  56  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
It has been observed that not every line of output when a TeXmacs chunk is active is a line of chunk. This may no longer be true, but we set a variable texmacs_chunk if the current line is a chunk line.
Initially we set this to zero...

67b <recognize-chunk[7](\v) ⇑65a, lang=> +≡ ▵67a 67c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  57  | texmacs_chunk=0;
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
...and then we look to see if the current line is a chunk line.
TeXmacs lines look like this:   3 | main() { so we detect the lines by leading white space, digits, more whiter space and a vertical bar followed by at least once space.
If we find such a line, we remove this line-header and set texmacs_chunk=1 as well as chunking=1

67c <recognize-chunk[8](\v) ⇑65a, lang=> +≡ ▵67b 67d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  58  | /^ *[1-9][0-9]* *\| / {
  59  |   if (texmacs_chunking) {
  60  |     chunking=1;
  61  |     texmacs_chunk=1;
  62  |     gsub("^ *[1-9][0-9]* *\\| ", "")
  63  |   }
  64  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
When TeXmacs chunking, lines that commence with \/ or __ are not chunk content but visual framing, and are skipped.

67d <recognize-chunk[9](\v) ⇑65a, lang=> +≡ ▵67c 68a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  65  | /^ *\.\/\\/ && texmacs_chunking {
  66  |   next;
  67  | }
  68  | /^ *__*$/ && texmacs_chunking {
  69  |   next;
  70  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Any other line when TeXmacs chunking is considered to be a line-wrapped line.

68a <recognize-chunk[10](\v) ⇑65a, lang=> +≡ ⊲67d 68b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  71  | texmacs_chunking {
  72  |   if (! texmacs_chunk) {
  73  |     # must be a texmacs continued line
  74  |     chunking=1;
  75  |     texmacs_chunk=1;
  76  |   }
  77  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
This final chunklet seems bogus and probably stops L Y X working.

68b <recognize-chunk[11](\v) ⇑65a, lang=> +≡ ▵68a 68c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  78  | ! texmacs_chunk {
  79  | #  texmacs_chunking=0;
  80  |   chunking=0;
  81  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.2.2 Noweb 
We recognize notangle style chunks too:

68c <recognize-chunk[12](\v) ⇑65a, lang=awk> +≡ ▵68b 68d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  82  | /^[<]<.*[>]>=/ {
  83  |   if (match($0, "^[<]<(.*)[>]>= *$", line)) {
  84  |     chunking=1;
  85  |     notangle_mode=1;
  86  |     new_chunk(line[1]);
  87  |     next;
  88  |   }
  89  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.3 Chunk end 
Likewise, we need to recognize when a chunk ends.
12.3.1 lstlistings 
The e in [e]nd{lislisting} is surrounded by square brackets so that when this document is processed, this chunk doesn't terminate early when the lstlistings package recognizes it's own end-string!4. This doesn't make sense as the regex is anchored with ^, which this line does not begin with!  ^4

68d <recognize-chunk[13](\v) ⇑65a, lang=> +≡ ▵68c 69a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  90  | /^\\[e]nd{lstlisting}|^\\[e]nd{Chunk}/ {
  91  |   chunking=0;
  92  |   active_chunk="";
  93  |   next;
  94  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.3.2 noweb 

69a <recognize-chunk[14](\v) ⇑65a, lang=> +≡ ⊲68d 69b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  95  | /^@ *$/ {
  96  |   chunking=0;
  97  |   active_chunk="";
  98  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
All other recognizers are only of effect if we are chunking; there's no point in looking at lines if they aren't part of a chunk, so we just ignore them as efficiently as we can.

69b <recognize-chunk[15](\v) ⇑65a, lang=> +≡ ▵69a 69c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  99  | ! chunking { next; }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
12.4 Chunk contents 
Chunk contents are any lines read while chunking is true. Some chunk contents are special in that they refer to other chunks, and will be replaced by the contents of these chunks when the file is generated.
We add the output record separator ORS to the line now, because we will set ORS to the empty string when we generate the output5. So that we can partial print lines using print instead of printf. 
To do: This does't make sense
^5.

69c <recognize-chunk[16](\v) ⇑65a, lang=> +≡ ▵69b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  100  | length(active_chunk) {
  101  |   «process-chunk-tabs 69e»
  102  |   «process-chunk 70b»
  103  | }
     |________________________________________________________________________


If a chunk just consisted of plain text, we could handle the chunk like this:

69d <process-chunk-simple[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | chunk_line(active_chunk, $0 ORS);
     |________________________________________________________________________


but in fact a chunk can include references to other chunks. Chunk includes are traditionally written as <<chunk-name>> but we support other variations, some of which are more suitable for particular editing systems.
However, we also process tabs at this point. A tab at input can be replaced by a number of spaces defined by the tabs variable, set by the -T option. Of course this is poor tab behaviour, we should probably have the option to use proper counted tab-stops and process this on output.

69e <process-chunk-tabs[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (length(tabs)) {
  2  |   gsub("\t", tabs);
  3  | }
     |________________________________________________________________________


12.4.1 lstlistings 
If \lstset{escapeinside={=<}{>}} is set, then we can use <chunk-name ?> in listings. The sequence =< was chosen because:
1.it is a better mnemonic than <<chunk-name>> in that the = sign signifies equivalence or substitutability.
2.and because =< is not valid in C or any language I can think of.
3.and also because lstlistings doesn't like >> as an end delimiter for the texcl escape, so we must make do with a single > which is better complemented by =< than by <<. 
Unfortunately the =<...> that we use re-enters a LaTeX parsing mode in which some characters are special, e.g. # \ and so these cause trouble if used in arguments to \chunkref. At some point I must fix the LaTeX command \chunkref so that it can accept these literally, but until then, when writing chunkref argumemts that need these characters, I must use the forms \textbackslash{} and \#; so I also define a hacky chunk delatex to be used further on whose purpose it is to remove these from any arguments parsed by fangle.

70a <delatex[1](text\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # FILTHY HACK
  2  | gsub("\\\\#", "#", ${text});
  3  | gsub("\\\\textbackslash{}", "\\", ${text});
  4  | gsub("\\\\\\^", "^", ${text});
     |________________________________________________________________________


As each chunk line may contain more than one chunk include, we will split out chunk includes in an iterative fashion6. Contrary to our use of split when substituting parameters in chapter ?  ^6.
First, as long as the chunk contains a \chunkref command we take as much as we can up to the first \chunkref command.
TeXmacs text output uses ⟨...⟩ which comes out as unicode sequences 0xC2 0xAB ... 0xC2 0xBB. Modern awk will interpret [^\xC2\xBB] as a single unicode character if LANG is set correctly to the sub-type UTF-8, e.g. LANG=en_GB.UTF-8, otherwise [^\xC2\xBB] will be treated as a two character negated match — but this should not interfere with the function.

70b <process-chunk[1](\v), lang=> ≡  70c▿
      ________________________________________________________________________
  1  | chunk = $0;
  2  | indent = 0;
  3  | while(match(chunk,"(\xC2\xAB)([^\xC2\xBB]*) [^\xC2\xBB]*\xC2\xBB", line) ||
  4  |       match(chunk, 
  5  |             "([=]<\\\\chunkref{([^}>]*)}(\\(.*\\)|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)", 
  6  |             line)\
  7  | ) {
  8  |   chunklet = substr(chunk, 1, RSTART - 1);
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We keep track of the indent count, by counting the number of literal characters found. We can then preserve this indent on each output line when multi-line chunks are expanded.
We then process this first part literal text, and set the chunk which is still to be processed to be the text after the \chunkref command, which we will process next as we continue around the loop.

70c <process-chunk[2](\v) ⇑70b, lang=> +≡ ▵70b 71a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  9  |   indent += length(chunklet);
  10  |   chunk_line(active_chunk, chunklet);
  11  |   chunk = substr(chunk, RSTART + RLENGTH);
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then consider the type of chunk command we have found, whether it is the fangle style command beginning with =< the older notangle style beginning with <<.
Fangle chunks may have parameters contained within square brackets. These will be matched in line[3] and are considered at this stage of processing to be part of the name of the chunk to be included.

71a <process-chunk[3](\v) ⇑70b, lang=> +≡ ⊲70c 71b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  12  |   if (substr(line[1], 1, 1) == "=") {
  13  |     # chunk name up to }
  14  |         «delatex\v(line[3]\v) 70a»
  15  |     chunk_include(active_chunk, line[2] line[3], indent);
  16  |   } else if (substr(line[1], 1, 1) == "<") {
  17  |     chunk_include(active_chunk, line[4], indent);
  18  |   } else if (line[1] == "\xC2\xAB") {
  19  |     chunk_include(active_chunk, line[2], indent);
  20  |   } else {
  21  |     error("Unknown chunk fragment: " line[1]);
  22  |   }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The loop will continue until there are no more chunkref statements in the text, at which point we process the final part of the chunk.

71b <process-chunk[4](\v) ⇑70b, lang=> +≡ ▵71a 71c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  23  | }
  24  | chunk_line(active_chunk, chunk);
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We add the newline character as a chunklet on it's own, to make it easier to detect new lines and thus manage indentation when processing the output.

71c <process-chunk[5](\v) ⇑70b, lang=> +≡ ▵71b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  25  | chunk_line(active_chunk, "\n");
     |________________________________________________________________________


We will also permit a chunk-part number to follow in square brackets, so that <chunk-name[1] ?> will refer to the first part only. This can make it easy to include a C function prototype in a header file, if the first part of the chunk is just the function prototype without the trailing semi-colon. The header file would include the prototype with the trailing semi-colon, like this:
<chunk-name[1] ?>
This is handled in section 14.1.1
We should perhaps introduce a notion of language specific chunk options; so that perhaps we could specify:
=<\chunkref{chunk-name[function-declaration]}
which applies a transform function-declaration to the chunk --- which in this case would extract a function prototype from a function. 
To do: Do it

Chapter 13Processing Options
At the start, first we set the default options.

73a <default-options[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | debug=0;
  2  | linenos=0;
  3  | notangle_mode=0;
  4  | root="*";
  5  | tabs = "";
     |________________________________________________________________________


Then we use getopt the standard way, and null out ARGV afterwards in the normal AWK fashion.

73b <read-options[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | Optind = 1    # skip ARGV[0]
  2  | while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
  3  |   «handle-options 73c»
  4  | }
  5  | for (i=1; i<Optind; i++) { ARGV[i]=""; }
     |________________________________________________________________________


This is how we handle our options:

73c <handle-options[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (Optopt == "R") root = Optarg;
  2  | else if (Optopt == "r") root="";
  3  | else if (Optopt == "L") linenos = 1;
  4  | else if (Optopt == "d") debug = 1;
  5  | else if (Optopt == "T") tabs = indent_string(Optarg+0);
  6  | else if (Optopt == "h") help();
  7  | else if (Optopt == "?") help();
     |________________________________________________________________________


We do all of this at the beginning of the program

73d <begin[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | BEGIN {
  2  |   «constants 39a»
  3  |   «mode-definitions 52b»
  4  |   «default-options 73a»
  5  | 
  6  |   «read-options 73b»
  7  | }
     |________________________________________________________________________


And have a simple help function

73e <help()[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function help() {
  2  |   print "Usage:"
  3  |   print "  fangle [-L] -R<rootname> [source.tex ...]"
  4  |   print "  fangle -r [source.tex ...]"
  5  |   print "  If the filename, source.tex is not specified then stdin is used"
  6  |   print
  7  |   print "-L causes the C statement: #line <lineno> \"filename\"" to be issued"
  8  |   print "-R causes the named root to be written to stdout"
  9  |   print "-r lists all roots in the file (even those used elsewhere)"
  10  |   exit 1;
  11  | }
     |________________________________________________________________________


Chapter 14Generating the Output
We generate output by calling output_chunk, or listing the chunk names.

75a <generate-output[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (length(root)) output_chunk(root);
  2  | else output_chunk_names();
     |________________________________________________________________________


We also have some other output debugging:

75b <debug-output[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (debug) {
  2  |   print "------ chunk names "
  3  |   output_chunk_names();
  4  |   print "====== chunks"
  5  |   output_chunks();
  6  |   print "++++++ debug"
  7  |   for (a in chunks) {
  8  |     print a "=" chunks[a];
  9  |   }
  10  | }
     |________________________________________________________________________


We do both of these at the end. We also set ORS="" because each chunklet is not necessarily a complete line, and we already added ORS to each input line in section 12.4.

75c <end[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | END {
  2  |   «debug-output 75b»
  3  |   ORS="";
  4  |   «generate-output 75a»
  5  | }
     |________________________________________________________________________


We write chunk names like this. If we seem to be running in notangle compatibility mode, then we enclose the name like this <<name>> the same way notangle does:

75d <output_chunk_names()[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function output_chunk_names(   c, prefix, suffix) 
  2  | {
  3  |   if (notangle_mode) {
  4  |     prefix="<<";
  5  |     suffix=">>";
  6  |   }
  7  |   for (c in chunk_names) {
  8  |     print prefix c suffix "\n";
  9  |   }
  10  | }
     |________________________________________________________________________


This function would write out all chunks

75e <output_chunks()[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | function output_chunks(  a) 
  2  | {
  3  |   for (a in chunk_names) {
  4  |     output_chunk(a);
  5  |   }
  6  | }
  7  | 
  8  | function output_chunk(chunk) {
  9  |   newline = 1;
  10  |   lineno_needed = linenos;
  11  | 
  12  |   write_chunk(chunk);
  13  | }
  14  | 
     |________________________________________________________________________


14.1 Assembling the Chunks 
chunk_path holds a string consisting of the names of all the chunks that resulted in this chunk being output. It should probably also contain the source line numbers at which each inclusion also occured.
We first initialize the mode tracker for this chunk.

76a <write_chunk()[1](\v), lang=awk> ≡  76b▿
      ________________________________________________________________________
  1  | function write_chunk(chunk_name) {
  2  |   «awk-delete-array\v(context\v) 37d»
  3  |   return write_chunk_r(chunk_name, context);
  4  | }
  5  | 
  6  | function write_chunk_r(chunk_name, context, indent, tail,
  7  |   # optional vars
  8  |   chunk_path, chunk_args, 
  9  |   # local vars
  10  |   context_origin,
  11  |   chunk_params, part, max_part, part_line, frag, max_frag, text, 
  12  |   chunklet, only_part, call_chunk_args, new_context)
  13  | {
  14  |   if (debug) debug_log("write_chunk_r(" chunk_name ")");
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
14.1.1 Chunk Parts 
As mentioned in section ?, a chunk name may contain a part specifier in square brackets, limiting the parts that should be emitted.

76b <write_chunk()[2](\v) ⇑76a, lang=> +≡ ▵76a 76c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  15  |   if (match(chunk_name, "^(.*)\\[([0-9]*)\\]$", chunk_name_parts)) {
  16  |     chunk_name = chunk_name_parts[1];
  17  |     only_part = chunk_name_parts[2];
  18  |   }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then create a mode tracker

76c <write_chunk()[3](\v) ⇑76a, lang=> +≡ ▵76b 77a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  19  |   context_origin = context[""];
  20  |   new_context = push_mode_tracker(context, chunks[chunk_name, "language"], "");
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We extract into chunk_params the names of the parameters that this chunk accepts, whose values were (optionally) passed in chunk_args.

77a <write_chunk()[4](\v) ⇑76a, lang=> +≡ ⊲76c 77b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  21  |   split(chunks[chunk_name, "params"], chunk_params, " *; *");
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
To assemble a chunk, we write out each part.

77b <write_chunk()[5](\v) ⇑76a, lang=> +≡ ▵77a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  22  |   if (! (chunk_name in chunk_names)) {
  23  |     error(sprintf(_"The root module <<%s>> was not defined.\nUsed by: %s",\
  24  |                   chunk_name, chunk_path));
  25  |   }
  26  | 
  27  |   max_part = chunks[chunk_name, "part"];
  28  |   for(part = 1; part <= max_part; part++) {
  29  |     if (! only_part || part == only_part) {
  30  |       «write-part 77c»
  31  |     }
  32  |   }
  33  |   if (! pop_mode_tracker(context, context_origin)) {
  34  |     dump_mode_tracker(context);
  35  |     error(sprintf(_"Module %s did not close context properly.\nUsed by: %s\n", chunk_name, chunk_path));
  36  |   }
  37  | }
     |________________________________________________________________________


A part can either be a chunklet of lines, or an include of another chunk.
Chunks may also have parameters, specified in LaTeX style with braces after the chunk name --- looking like this in the document: chunkname{param1, param2}. Arguments are passed in square brackets: \chunkref{chunkname}[arg1, arg2].
Before we process each part, we check that the source position hasn't changed unexpectedly, so that we can know if we need to output a new file-line directive.

77c <write-part[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «check-source-jump 79d»
  2  | 
  3  | chunklet = chunks[chunk_name, "part", part];
  4  | if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
  5  |   «write-included-chunk 77d»
  6  | } else if (chunklet SUBSEP "line" in chunks) {
  7  |   «write-chunklets 78a»
  8  | } else {
  9  |   # empty last chunklet
  10  | }
     |________________________________________________________________________


To write an included chunk, we must detect any optional chunk arguments in parenthesis. Then we recurse calling write_chunk().

77d <write-included-chunk[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (match(chunklet, "^([^\\[\\(]*)\\((.*)\\)$", chunklet_parts)) {
  2  |   chunklet = chunklet_parts[1];
  3  | # hack
  4  | gsub(sprintf("%c",11), "", chunklet);
  5  | gsub(sprintf("%c",11), "", chunklet_parts[2]);
  6  |   parse_chunk_args("c-like", chunklet_parts[2], call_chunk_args, "(");
  7  |   for (c in call_chunk_args) {
  8  |     call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params, chunk_args);
  9  |   }
  10  | } else {
  11  |   split("", call_chunk_args);
  12  | }
  13  | 
  14  | write_chunk_r(chunklet, context,
  15  |             chunks[chunk_name, "part", part, "indent"] indent,
  16  |             chunks[chunk_name, "part", part, "tail"],
  17  |             chunk_path "\n         " chunk_name,
  18  |             call_chunk_args);
     |________________________________________________________________________


Before we output a chunklet of lines, we first emit the file and line number if we have one, and if it is safe to do so.
Chunklets are generally broken up by includes, so the start of a chunklet is a good place to do this. Then we output each line of the chunklet.
When it is not safe, such as in the middle of a multi-line macro definition, lineno_suppressed is set to true, and in such a case we note that we want to emit the line statement when it is next safe.

78a <write-chunklets[1](\v), lang=> ≡  78b▿
      ________________________________________________________________________
  1  | max_frag = chunks[chunklet, "line"];
  2  | for(frag = 1; frag <= max_frag; frag++) {
  3  |   «write-file-line 79c»
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then extract the chunklet text and expand any arguments.

78b <write-chunklets[2](\v) ⇑78a, lang=> +≡ ▵78a 78c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  4  | 
  5  |   text = chunks[chunklet, frag];
  6  |  
  7  |   /* check params */
  8  |   text = expand_chunk_args(text, chunk_params, chunk_args);
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If the text is a single newline (which we keep separate - see 6) then we increment the line number. In the case where this is the last line of a chunk and it is not a top-level chunk we replace the newline with an empty string --- because the chunk that included this chunk will have the newline at the end of the line that included this chunk.
We also note by newline = 1 that we have started a new line, so that indentation can be managed with the following piece of text.

78c <write-chunklets[3](\v) ⇑78a, lang=> +≡ ▵78b 78d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  9  | 
  10  |  if (text == "\n") {
  11  |     lineno++;
  12  |     if (part == max_part && frag == max_frag && length(chunk_path)) {
  13  |       text = "";
  14  |       break;
  15  |     } else {
  16  |       newline = 1;
  17  |     }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If this text does not represent a newline, but we see that we are the first piece of text on a newline, then we prefix our text with the current indent. 
Note 1. newline is a global output-state variable, but the indent is not. 

78d <write-chunklets[4](\v) ⇑78a, lang=> +≡ ▵78c 79a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  18  |   } else if (length(text) || length(tail)) {
  19  |     if (newline) text = indent text;
  20  |     newline = 0;
  21  |   }
  22  | 
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Tail will soon no longer be relevant once mode-detection is in place.

79a <write-chunklets[5](\v) ⇑78a, lang=> +≡ ⊲78d 79b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  23  |   text = text tail;
  24  |   mode_tracker(context, text);
  25  |   print untab(transform_escape(context, text, new_context));
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
If a line ends in a backslash --- suggesting continuation --- then we supress outputting file-line as it would probably break the continued lines.

79b <write-chunklets[6](\v) ⇑78a, lang=> +≡ ▵79a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  26  |   if (linenos) {
  27  |     lineno_suppressed = substr(lastline, length(lastline)) == "\\";
  28  |   }
  29  | }
     |________________________________________________________________________


Of course there is no point in actually outputting the source filename and line number (file-line) if they don't say anything new! We only need to emit them if they aren't what is expected, or if we we not able to emit one when they had changed.

79c <write-file-line[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (newline && lineno_needed && ! lineno_suppressed) {
  2  |   filename = a_filename;
  3  |   lineno = a_lineno;
  4  |   print "#line " lineno " \"" filename "\"\n"
  5  |   lineno_needed = 0;
  6  | }
     |________________________________________________________________________


We check if a new file-line is needed by checking if the source line matches what we (or a compiler) would expect.

79d <check-source-jump[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in chunks)) {
  2  |   a_filename = chunks[chunk_name, "part", part, "FILENAME"];
  3  |   a_lineno = chunks[chunk_name, "part", part, "LINENO"];
  4  |   if (a_filename != filename || a_lineno != lineno) {
  5  |     lineno_needed++;
  6  |   }
  7  | }
     |________________________________________________________________________


Chapter 15Storing Chunks
Awk has pretty limited data structures, so we will use two main hashes. Uninterrupted sequences of a chunk will be stored in chunklets and the chunklets used in a chunk will be stored in chunks.

81a <constants[2](\v) ⇑39a, lang=> +≡ ⊲39a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  2  | part_type_chunk=1;
  3  | SUBSEP=",";
     |________________________________________________________________________


The params mentioned are not chunk parameters for parameterized chunks, as mentioned in 10.2, but the lstlistings style parameters used in the \Chunk command1. The params parameter is used to hold the parameters for parameterized chunks  ^1.

81b <chunk-storage-functions[1](\v), lang=> ≡  81c▿
      ________________________________________________________________________
  1  | function new_chunk(chunk_name, opts, args,
  2  |   # local vars
  3  |   p, append )
  4  | {
  5  |   # HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
  6  |   gsub("\\(\\)$", "", chunk_name);
  7  |   if (! (chunk_name in chunk_names)) {
  8  |     if (debug) print "New chunk " chunk_name;
  9  |     chunk_names[chunk_name];
  10  |     for (p in opts) {
  11  |       chunks[chunk_name, p] = opts[p];
  12  |       if (debug) print "chunks[" chunk_name "," p "] = " opts[p];
  13  |     }
  14  |     for (p in args) {
  15  |       chunks[chunk_name, "params", p] = args[p];
  16  |     }
  17  |     if ("append" in opts) {
  18  |       append=opts["append"];
  19  |       if (! (append in chunk_names)) {
  20  |         warning("Chunk " chunk_name " is appended to chunk " append " which is not defined yet");
  21  |         new_chunk(append);
  22  |       }
  23  |       chunk_include(append, chunk_name);
  24  |       chunk_line(append, ORS);
  25  |     }
  26  |   }
  27  |   active_chunk = chunk_name;
  28  |   prime_chunk(chunk_name);
  29  | }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

81c <chunk-storage-functions[2](\v) ⇑81b, lang=> +≡ ▵81b 82a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  30  | 
  31  | function prime_chunk(chunk_name)
  32  | {
  33  |   chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = \
  34  |          chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"];
  35  |   chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
  36  |   chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR + 1;
  37  | }
  38  | 
  39  | function chunk_line(chunk_name, line){
  40  |   chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
  41  |          ++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"], "line"]  ] = line;
  42  | }
  43  | 
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Chunk include represents a chunkref statement, and stores the requirement to include another chunk. The parameter indent represents the quanity of literal text characters that preceded this chunkref statement and therefore by how much additional lines of the included chunk should be indented.

82a <chunk-storage-functions[3](\v) ⇑81b, lang=> +≡ ⊲81c 82b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  44  | function chunk_include(chunk_name, chunk_ref, indent, tail)
  45  | {
  46  |   chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
  47  |   chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_chunk;
  48  |   chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_string(indent);
  49  |   chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
  50  |   prime_chunk(chunk_name);
  51  | }
  52  | 
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The indent is calculated by indent_string, which may in future convert some spaces into tab characters. This function works by generating a printf padded format string, like %22s for an indent of 22, and then printing an empty string using that format.

82b <chunk-storage-functions[4](\v) ⇑81b, lang=> +≡ ▵82a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  53  | function indent_string(indent) {
  54  |   return sprintf("%" indent "s", "");
  55  | }
     |________________________________________________________________________


Chapter 16getopt
I use Arnold Robbins public domain getopt (1993 revision). This is probably the same one that is covered in chapter 12 of âĂIJEdition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU AwkâĂİ but as that is licensed under the GNU Free Documentation License, Version 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompanying explanations), so I do my best to explain how it works here.
The getopt.awk header is:

83a <getopt.awk-header[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # getopt.awk --- do C library getopt(3) function in awk
  2  | #
  3  | # Arnold Robbins, arnold@skeeve.com, Public Domain
  4  | #
  5  | # Initial version: March, 1991
  6  | # Revised: May, 1993
  7  | 
     |________________________________________________________________________


The provided explanation is:

83b <getopt.awk-notes[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # External variables:
  2  | #    Optind -- index in ARGV of first nonoption argument
  3  | #    Optarg -- string value of argument to current option
  4  | #    Opterr -- if nonzero, print our own diagnostic
  5  | #    Optopt -- current option letter
  6  | 
  7  | # Returns:
  8  | #    -1     at end of options
  9  | #    ?      for unrecognized option
  10  | #    <c>    a character representing the current option
  11  | 
  12  | # Private Data:
  13  | #    _opti  -- index in multi-flag option, e.g., -abc
  14  | 
     |________________________________________________________________________


The function follows. The final two parameters, thisopt and i are local variables and not parameters --- as indicated by the multiple spaces preceding them. Awk doesn't care, the multiple spaces are a convention to help us humans.

83c <getopt.awk-getopt()[1](\v), lang=> ≡  84a⊳
      ________________________________________________________________________
  1  | function getopt(argc, argv, options,    thisopt, i)
  2  | {
  3  |     if (length(options) == 0)    # no options given
  4  |         return -1
  5  |     if (argv[Optind] == "--") {  # all done
  6  |         Optind++
  7  |         _opti = 0
  8  |         return -1
  9  |     } else if (argv[Optind] !~ /^-[^: \t\n\f\r\v\b]/) {
  10  |         _opti = 0
  11  |         return -1
  12  |     }
  13  |     if (_opti == 0)
  14  |         _opti = 2
  15  |     thisopt = substr(argv[Optind], _opti, 1)
  16  |     Optopt = thisopt
  17  |     i = index(options, thisopt)
  18  |     if (i == 0) {
  19  |         if (Opterr)
  20  |             printf("%c -- invalid option\n",
  21  |                                   thisopt) > "/dev/stderr"
  22  |         if (_opti >= length(argv[Optind])) {
  23  |             Optind++
  24  |             _opti = 0
  25  |         } else
  26  |             _opti++
  27  |         return "?"
  28  |     }
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
At this point, the option has been found and we need to know if it takes any arguments.

84a <getopt.awk-getopt()[2](\v) ⇑83c, lang=> +≡ ⊲83c
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  29  |     if (substr(options, i + 1, 1) == ":") {
  30  |         # get option argument
  31  |         if (length(substr(argv[Optind], _opti + 1)) > 0)
  32  |             Optarg = substr(argv[Optind], _opti + 1)
  33  |         else
  34  |             Optarg = argv[++Optind]
  35  |         _opti = 0
  36  |     } else
  37  |         Optarg = ""
  38  |     if (_opti == 0 || _opti >= length(argv[Optind])) {
  39  |         Optind++
  40  |         _opti = 0
  41  |     } else
  42  |         _opti++
  43  |     return thisopt
  44  | }
     |________________________________________________________________________


A test program is built in, too

84b <getopt.awk-begin[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | BEGIN {
  2  |     Opterr = 1    # default is to diagnose
  3  |     Optind = 1    # skip ARGV[0]
  4  |     # test program
  5  |     if (_getopt_test) {
  6  |         while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
  7  |             printf("c = <%c>, optarg = <%s>\n",
  8  |                                        _go_c, Optarg)
  9  |         printf("non-option arguments:\n")
  10  |         for (; Optind < ARGC; Optind++)
  11  |             printf("\tARGV[%d] = <%s>\n",
  12  |                                     Optind, ARGV[Optind])
  13  |     }
  14  | }
     |________________________________________________________________________


The entire getopt.awk is made out of these chunks in order

84c <getopt.awk[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | «getopt.awk-header 83a»
  2  | 
  3  | «getopt.awk-notes 83b»
  4  | «getopt.awk-getopt() 83c»
  5  | «getopt.awk-begin 84b»
     |________________________________________________________________________


Although we only want the header and function:

85a <getopt[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | # try: locate getopt.awk for the full original file
  2  | # as part of your standard awk installation
  3  | «getopt.awk-header 83a»
  4  | 
  5  | «getopt.awk-getopt() 83c»
     |________________________________________________________________________


Chapter 17Fangle LaTeX source code
17.1 fangle module 
Here we define a L Y X .module file that makes it convenient to use L Y X for writing such literate programs.
This file ./fangle.module can be installed in your personal .lyx/layouts folder. You will need to Tools Reconfigure so that L Y X notices it. It adds a new format Chunk, which should precede every listing and contain the chunk name.

87a <./fangle.module[1](\v), lang=lyx-module> ≡ 
      ________________________________________________________________________
  1  | #\DeclareLyXModule{Fangle Literate Listings}
  2  | #DescriptionBegin
  3  | #  Fangle literate listings allow one to write
  4  | #   literate programs after the fashion of noweb, but without having
  5  | #   to use noweave to generate the documentation. Instead the listings
  6  | #   package is extended in conjunction with the noweb package to implement
  7  | #   to code formating directly as latex.
  8  | #  The fangle awk script
  9  | #DescriptionEnd
  10  | 
  11  | «gpl3-copyright.hashed 87b»
  12  | 
  13  | Format 11
  14  | 
  15  | AddToPreamble
  16  | «./fangle.sty 88d»
  17  | EndPreamble
  18  | 
  19  | «chunkstyle 88a»
  20  | 
  21  | «chunkref 88c»
     |________________________________________________________________________


Because L Y X modules are not yet a language supported by fangle or lstlistings, we resort to this fake awk chunk below in order to have each line of the GPL3 license commence with a #

87b <gpl3-copyright.hashed[1](\v), lang=awk> ≡ 
      ________________________________________________________________________
  1  | #«gpl3-copyright 4a»
  2  | 
     |________________________________________________________________________


17.1.1 The Chunk style 
The purpose of the chunk style is to make it easier for L Y X users to provide the name to lstlistings. Normally this requires right-clicking on the listing, choosing settings, advanced, and then typing name=chunk-name. This has the further disadvantage that the name (and other options) are not generally visible during document editing.
The chunk style is defined as a LaTeX command, so that all text on the same line is passed to the LaTeX command Chunk. This makes it easy to parse using fangle, and easy to pass these options on to the listings package. The first word in a chunk section should be the chunk name, and will have name= prepended to it. Any other words are accepted arguments to lstset.
We set PassThru to 1 because the user is actually entering raw latex.

88a <chunkstyle[1](\v), lang=> ≡  88b▿
      ________________________________________________________________________
  1  | Style Chunk
  2  |   LatexType             Command
  3  |   LatexName             Chunk
  4  |   Margin                First_Dynamic
  5  |   LeftMargin            Chunk:xxx
  6  |   LabelSep              xx
  7  |   LabelType             Static
  8  |   LabelString           "Chunk:"
  9  |   Align                 Left
  10  |   PassThru              1
  11  | 
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
To make the label very visible we choose a larger font coloured red.

88b <chunkstyle[2](\v) ⇑88a, lang=> +≡ ▵88a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  12  |   LabelFont
  13  |     Family              Sans
  14  |     Size                Large
  15  |     Series              Bold
  16  |     Shape               Italic
  17  |     Color               red
  18  |   EndFont
  19  | End
     |________________________________________________________________________


17.1.2 The chunkref style 
We also define the Chunkref style which can be used to express cross references to chunks.

88c <chunkref[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | InsetLayout Chunkref
  2  |   LyxType               charstyle
  3  |   LatexType             Command
  4  |   LatexName             chunkref
  5  |   PassThru              1
  6  |   LabelFont             
  7  |     Shape               Italic
  8  |     Color               red
  9  |   EndFont
  10  | End
     |________________________________________________________________________


17.2 Latex Macros 
We require the listings, noweb and xargs packages. As noweb defines it's own \code environment, we re-define the one that L Y X logical markup module expects here.

88d <./fangle.sty[1](\v), lang=tex> ≡  89a⊳
      ________________________________________________________________________
  1  | \usepackage{listings}%
  2  | \usepackage{noweb}%
  3  | \usepackage{xargs}%
  4  | \renewcommand{\code}[1]{\texttt{#1}}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also define a CChunk macro, for use as: \begin{CChunk} which will need renaming to \begin{Chunk} when I can do this without clashing with \Chunk.

89a <./fangle.sty[2](\v) ⇑88d, lang=> +≡ ⊲88d 89b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | \lstnewenvironment{Chunk}{\relax}{\relax}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also define a suitable \lstset of parameters that suit the literate programming style after the fashion of noweave.

89b <./fangle.sty[3](\v) ⇑88d, lang=> +≡ ▵89a 89c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  6  | \lstset{numbers=left, stepnumber=5, numbersep=5pt,
  7  |         breaklines=false,basicstyle=\ttfamily,
  8  |         numberstyle=\tiny, language=C}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also define a notangle-like mechanism for escaping to LaTeX from the listing, and by which we can refer to other listings. We declare the =<...> sequence to contain LaTeX code, and include another like this chunk: <chunkname ?>. However, because =<...> is already defined to contain LaTeX code for this document --- this is a fangle document after all --- the code fragment below effectively contains the LaTeX code: }{. To avoid problems with document generation, I had to declare an lstlistings property: escapeinside={} for this listing only; which in L Y X was done by right-clicking the listings inset, choosing settings->advanced. Therefore =< isn't interpreted literally here, in a listing when the escape sequence is already defined as shown... we need to somehow escape this representation...

89c <./fangle.sty[4](\v) ⇑88d, lang=> +≡ ▵89b 89d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  9  | \lstset{escapeinside={=<}{>}}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Although our macros will contain the @ symbol, they will be included in a \makeatletter section by L Y X; however we keep the commented out \makeatletter as a reminder. The listings package likes to centre the titles, but noweb titles are specially formatted and must be left aligned. The simplest way to do this turned out to be by removing the definition of \lst@maketitle. This may interact badly if other listings want a regular title or caption. We remember the old maketitle in case we need it.

89d <./fangle.sty[5](\v) ⇑88d, lang=> +≡ ▵89c 89e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  10  | %\makeatletter
  11  | %somehow re-defining maketitle gives us a left-aligned title
  12  | %which is extactly what our specially formatted title needs!
  13  | \global\let\fangle@lst@maketitle\lst@maketitle%
  14  | \global\def\lst@maketitle{}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
17.2.1 The chunk command 
Our chunk command accepts one argument, and calls \ltset. Although \ltset will note the name, this is erased when the next \lstlisting starts, so we make a note of this in \lst@chunkname and restore in in lstlistings Init hook.

89e <./fangle.sty[6](\v) ⇑88d, lang=> +≡ ▵89d 90a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  15  | \def\Chunk#1{%
  16  |   \lstset{title={\fanglecaption},name=#1}%
  17  |   \global\edef\lst@chunkname{\lst@intname}%
  18  | }%
  19  | \def\lst@chunkname{\empty}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
17.2.1.1 Chunk parameters 
Fangle permits parameterized chunks, and requires the paramters to be specified as listings options. The fangle script uses this, and although we don't do anything with these in the LaTeX code right now, we need to stop the listings package complaining.

90a <./fangle.sty[7](\v) ⇑88d, lang=> +≡ ⊲89e 90b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  20  | \lst@Key{params}\relax{\def\fangle@chunk@params{#1}}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
As it is common to define a chunk which then needs appending to another chunk, and annoying to have to declare a single line chunk to manage the include, we support an append= option.

90b <./fangle.sty[8](\v) ⇑88d, lang=> +≡ ▵90a 90c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  21  | \lst@Key{append}\relax{\def\fangle@chunk@append{#1}}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
17.2.2 The noweb styled caption 
We define a public macro \fanglecaption which can be set as a regular title. By means of \protect, It expands to \fangle@caption at the appopriate time when the caption is emitted.

90c <./fangle.sty[9](\v) ⇑88d, lang=> +≡ ▵90b 90d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\def\fanglecaption{\protect\fangle@caption}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
22c ⟨some-chunk 19b⟩≡+   ⊲22b 24d⊳

In this example, the current chunk is 22c, and therefore the third chunk on page 22.
It's name is some-chunk. 
The first chunk with this name (19b) occurs as the second chunk on page 19.
The previous chunk (22d) with the same name is the second chunk on page 22.
The next chunk (24d) is the fourth chunk on page 24.

Figure 1. Noweb Heading

The general noweb output format compactly identifies the current chunk, and references to the first chunk, and the previous and next chunks that have the same name.
This means that we need to keep a counter for each chunk-name, that we use to count chunks of the same name.
17.2.3 The chunk counter 
It would be natural to have a counter for each chunk name, but TeX would soon run out of counters1. ...soon did run out of counters and so I had to re-write the LaTeX macros to share a counter as described here.  ^1, so we have one counter which we save at the end of a chunk and restore at the beginning of a chunk.

90d <./fangle.sty[10](\v) ⇑88d, lang=> +≡ ▵90c 91c⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  22  | \newcounter{fangle@chunkcounter}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We construct the name of this variable to store the counter to be the text lst-chunk- prefixed onto the chunks own name, and store it in \chunkcount. 
We save the counter like this:

91a <save-counter[1](\v), lang=> ≡ 
      ________________________________________________________________________
\global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
     |________________________________________________________________________


and restore the counter like this:

91b <restore-counter[1](\v), lang=> ≡ 
      ________________________________________________________________________
\setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
     |________________________________________________________________________


If there does not already exist a variable whose name is stored in \chunkcount, then we know we are the first chunk with this name, and then define a counter. 
Although chunks of the same name share a common counter, they must still be distinguished. We use is the internal name of the listing, suffixed by the counter value. So the first chunk might be something-1 and the second chunk be something-2, etc.
We also calculate the name of the previous chunk if we can (before we increment the chunk counter). If this is the first chunk of that name, then \prevchunkname is set to \relax which the noweb package will interpret as not existing.

91c <./fangle.sty[11](\v) ⇑88d, lang=> +≡ ⊲90d 91d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  23  | \def\fangle@caption{%
  24  |   \edef\chunkcount{lst-chunk-\lst@intname}%
  25  |   \@ifundefined{\chunkcount}{%
  26  |     \expandafter\gdef\csname \chunkcount\endcsname{0}%
  27  |     \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
  28  |     \let\prevchunkname\relax%
  29  |   }{%
  30  |     \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
  31  |     \edef\prevchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
  32  |   }%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
After incrementing the chunk counter, we then define the name of this chunk, as well as the name of the first chunk.

91d <./fangle.sty[12](\v) ⇑88d, lang=> +≡ ▵91c 91e▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  33  |   \addtocounter{fangle@chunkcounter}{1}%
  34  |   \global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
  35  |   \edef\chunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
  36  |   \edef\firstchunkname{\lst@intname-1}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We now need to calculate the name of the next chunk. We do this by temporarily skipping the counter on by one; however there may not actually be another chunk with this name! We detect this by also defining a label for each chunk based on the chunkname. If there is a next chunkname then it will define a label with that name. As labels are persistent, we can at least tell the second time LaTeX is run. If we don't find such a defined label then we define \nextchunkname to \relax.

91e <./fangle.sty[13](\v) ⇑88d, lang=> +≡ ▵91d 92a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  37  |   \addtocounter{fangle@chunkcounter}{1}%
  38  |   \edef\nextchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
  39  |   \@ifundefined{r@label-\nextchunkname}{\let\nextchunkname\relax}{}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
The noweb package requires that we define a \sublabel for every chunk, with a unique name, which is then used to print out it's navigation hints.
We also define a regular label for this chunk, as was mentioned above when we calculated \nextchunkname. This requires LaTeX to be run at least twice after new chunk sections are added --- but noweb requried that anyway.

92a <./fangle.sty[14](\v) ⇑88d, lang=> +≡ ⊲91e 92b▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  40  |   \sublabel{\chunkname}%
  41  | % define this label for every chunk instance, so we
  42  | % can tell when we are the last chunk of this name
  43  |   \label{label-\chunkname}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We also try and add the chunk to the list of listings, but I'm afraid we don't do very well. We want each chunk name listing once, with all of it's references.

92b <./fangle.sty[15](\v) ⇑88d, lang=> +≡ ▵92a 92c▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  44  |   \addcontentsline{lol}{lstlisting}{\lst@name~[\protect\subpageref{\chunkname}]}%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
We then call the noweb output macros in the same way that noweave generates them, except that we don't need to call \nwstartdeflinemarkup or \nwenddeflinemarkup — and if we do, it messes up the output somewhat.

92c <./fangle.sty[16](\v) ⇑88d, lang=> +≡ ▵92b 92d▿
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  45  |   \nwmargintag{%
  46  |     {%
  47  |       \nwtagstyle{}%
  48  |       \subpageref{\chunkname}%
  49  |     }%
  50  |   }%
  51  | %
  52  |   \moddef{%
  53  |     {\lst@name}%
  54  |     {%
  55  |       \nwtagstyle{}\/%
  56  |       \@ifundefined{fangle@chunk@params}{}{%
  57  |         (\fangle@chunk@params)%
  58  |       }%
  59  |       [\csname \chunkcount\endcsname]~%
  60  |       \subpageref{\firstchunkname}%
  61  |     }%
  62  |     \@ifundefined{fangle@chunk@append}{}{%
  63  |     \ifx{}\fangle@chunk@append{x}\else%
  64  |         ,~add~to~\fangle@chunk@append%
  65  |     \fi%
  66  |     }%
  67  | \global\def\fangle@chunk@append{}%
  68  | \lstset{append=x}%
  69  |   }%
  70  | %
  71  |   \ifx\relax\prevchunkname\endmoddef\else\plusendmoddef\fi%
  72  | %  \nwstartdeflinemarkup%
  73  |   \nwprevnextdefs{\prevchunkname}{\nextchunkname}%
  74  | %  \nwenddeflinemarkup%
  75  | }%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
Originally this was developed as a listings aspect, in the Init hook, but it was found easier to affect the title without using a hook — \lst@AddToHookExe{PreSet} is still required to set the listings name to the name passed to the \Chunk command, though.

92d <./fangle.sty[17](\v) ⇑88d, lang=> +≡ ▵92c 93a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  76  | %\lst@BeginAspect{fangle}
  77  | %\lst@Key{fangle}{true}[t]{\lstKV@SetIf{#1}{true}}
  78  | \lst@AddToHookExe{PreSet}{\global\let\lst@intname\lst@chunkname}
  79  | \lst@AddToHook{Init}{}%\fangle@caption}
  80  | %\lst@EndAspect
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
17.2.4 Cross references 
We define the \chunkref command which makes it easy to generate visual references to different code chunks, e.g.
Macro                           Appearance
\chunkref{preamble}             
\chunkref[3]{preamble}          
\chunkref{preamble}[arg1, arg2] 

Chunkref can also be used within a code chunk to include another code chunk. The third optional parameter to chunkref is a comma sepatarated list of arguments, which will replace defined parameters in the chunkref.
Note 1. Darn it, if I have: =<\chunkref{new-mode-tracker}[{chunks[chunk_name, "language"]},{mode}]> the inner braces (inside [ ]) cause _ to signify subscript even though we have lst@ReplaceIn 

93a <./fangle.sty[18](\v) ⇑88d, lang=> +≡ ⊲92d 94a⊳
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  81  | \def\chunkref@args#1,{%
  82  |   \def\arg{#1}%
  83  |   \lst@ReplaceIn\arg\lst@filenamerpl%
  84  |   \arg%
  85  |   \@ifnextchar){\relax}{, \chunkref@args}%
  86  | }%
  87  | \newcommand\chunkref[2][0]{%
  88  |   \@ifnextchar({\chunkref@i{#1}{#2}}{\chunkref@i{#1}{#2}()}%
  89  | }%
  90  | \def\chunkref@i#1#2(#3){%
  91  |   \def\zero{0}%
  92  |   \def\chunk{#2}%
  93  |   \def\chunkno{#1}%
  94  |   \def\chunkargs{#3}%
  95  |   \ifx\chunkno\zero%
  96  |     \def\chunkname{#2-1}%
  97  |   \else%
  98  |     \def\chunkname{#2-\chunkno}%
  99  |   \fi%
  100  |   \let\lst@arg\chunk%
  101  |   \lst@ReplaceIn\chunk\lst@filenamerpl%
  102  |   \LA{%\moddef{%
  103  |     {\chunk}%
  104  |     {%
  105  |       \nwtagstyle{}\/%
  106  |       \ifx\chunkno\zero%
  107  |       \else%
  108  |       [\chunkno]%
  109  |       \fi%
  110  |       \ifx\chunkargs\empty%
  111  |       \else%
  112  |         (\chunkref@args #3,)%
  113  |       \fi%
  114  |       ~\subpageref{\chunkname}%
  115  |     }%
  116  |   }%
  117  |   \RA%\endmoddef%
  118  | }%
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
17.2.5 The end 

94a <./fangle.sty[19](\v) ⇑88d, lang=> +≡ ⊲93a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  119  | %
  120  | %\makeatother
     |________________________________________________________________________


Chapter 18Extracting fangle
18.1 Extracting from Lyx 
To extract from L Y X, you will need to configure L Y X as explained in section ?.
And this lyx-build scrap will extract fangle for me.

95a <lyx-build[2](\v) ⇑20a, lang=sh> +≡ ⊲20a
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  11  | #! /bin/sh
  12  | set -x
  13  | 
  14  | «lyx-build-helper 19b»
  15  | cd $PROJECT_DIR || exit 1
  16  | 
  17  | /usr/local/bin/fangle -R./fangle $TEX_SRC > ./fangle
  18  | /usr/local/bin/fangle -R./fangle.module $TEX_SRC > ./fangle.module
  19  | 
  20  | export FANGLE=./fangle
  21  | export TMP=${TMP:-/tmp}
  22  | «test:* 99a»
     |________________________________________________________________________


With a lyx-build-helper

95b <lyx-build-helper[2](\v) ⇑19b, lang=sh> +≡ ⊲19b
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  5  | PROJECT_DIR="$LYX_r"
  6  | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
  7  | TEX_DIR="$LYX_p"
  8  | TEX_SRC="$TEX_DIR/$LYX_i"
  9  | TXT_SRC="$TEX_SRC"
     |________________________________________________________________________


18.2 Extracting documentation 

95c <./gen-www[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | #python -m elyxer --css lyx.css $LYX_SRC | \
  2  | #  iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT | \
  3  | #  sed 's/UTF-8"\(.\)>/ISO-8859-1"\1>/' > www/docs/fangle.html
  4  | 
  5  | python -m elyxer --css lyx.css --iso885915 --html --destdirectory www/docs/fangle.e \
  6  |        fangle.lyx > www/docs/fangle.e/fangle.html
  7  | 
  8  | ( mkdir -p www/docs/fangle && cd www/docs/fangle && \
  9  |   lyx -e latex ../../../fangle.lyx && \
  10  |   htlatex ../../../fangle.tex "xhtml,fn-in" && \
  11  |   sed -i -e 's/<!--l\. [0-9][0-9]* *-->//g' fangle.html
  12  | )
  13  | 
  14  | ( mkdir -p www/docs/literate && cd www/docs/literate && \
  15  |   lyx -e latex ../../../literate.lyx && \
  16  |   htlatex ../../../literate.tex "xhtml,fn-in" && \
  17  |   sed -i -e 's/<!--l\. [0-9][0-9]* *-->$//g' literate.html
  18  | )
     |________________________________________________________________________


18.3 Extracting from the command line 
First you will need the tex output, then you can extract:

96a <lyx-build-manual[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | lyx -e latex fangle.lyx
  2  | fangle -R./fangle fangle.tex > ./fangle
  3  | fangle -R./fangle.module fangle.tex > ./fangle.module
     |________________________________________________________________________



Part III  Tests 
Chapter 19Tests

99a <test:*[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | #! /bin/bash
  2  | 
  3  | export SRC="${SRC:-./fangle.tm}"
  4  | export FANGLE="${FANGLE:-./fangle}"
  5  | export TMP="${TMP:-/tmp}"
  6  | export TESTDIR="$TMP/$USER/fangle.tests"
  7  | export TXT_SRC="${TXT_SRC:-$TESTDIR/fangle.txt}"
  8  | export AWK="${AWK:-awk}"
  9  | export RUN_FANGLE="${RUN_FANGLE:-$AWK -f}"
  10  | 
  11  | fangle() {
  12  |   ${AWK} -f ${FANGLE} "$@"
  13  | }
  14  | 
  15  | mkdir -p "$TESTDIR"
  16  | 
  17  | tm -s -c "$SRC" "$TXT_SRC" -q
  18  | 
  19  | «test:helpers 100a»
  20  | run_tests() {
  21  |   «test:run-tests 99b»
  22  | }
  23  | 
  24  | # test current fangle
  25  | echo Testing current fangle
  26  | run_tests
  27  | 
  28  | # extract new fangle
  29  | echo testing new fangle
  30  | fangle -R./fangle "$TXT_SRC" > "$TESTDIR/fangle"
  31  | export FANGLE="$TESTDIR/fangle"
  32  | run_tests
  33  | 
  34  | # Now check that it can extract a fangle that also passes the tests!
  35  | echo testing if new fangle can generate itself
  36  | fangle -R./fangle "$TXT_SRC" > "$TESTDIR/fangle.new"
  37  | passtest diff -bwu "$FANGLE" "$TESTDIR/fangle.new"
  38  | export FANGLE="$TESTDIR/fangle.new"
  39  | run_tests
     |________________________________________________________________________



99b <test:run-tests[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | # run tests
  2  | fangle -Rpca-test.awk $TXT_SRC | awk -f - || exit 1
  3  | «test:cromulence 59g»
  4  | «test:escapes 64a»
  5  | «test:test-chunk\v(test:example-sh\v) 100b»
  6  | «test:test-chunk\v(test:example-makefile\v) 100b»
  7  | «test:test-chunk\v(test:q:1\v) 100b»
  8  | «test:test-chunk\v(test:make:1\v) 100b»
  9  | «test:test-chunk\v(test:make:2\v) 100b»
  10  | «test:chunk-params 101e»
     |________________________________________________________________________



100a <test:helpers[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | passtest() {
  2  |   if "$@"
  3  |   then echo "Passed $TEST"
  4  |   else echo "Failed $TEST"
  5  |        return 1
  6  |   fi
  7  | }
  8  | 
  9  | failtest() {
  10  |   if ! "$@"
  11  |   then echo "Passed $TEST"
  12  |   else echo "Failed $TEST"
  13  |        return 1
  14  |   fi
  15  | }
     |________________________________________________________________________


This chunk will render a named chunk and compare it to another rendered nameed chunk

100b <test:test-chunk[1](chunk\v\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | «test:test-chunk-result\v(${chunk}\v, ${chunk}.result\v) 100c»
     |________________________________________________________________________



100c <test:test-chunk-result[1](chunk\v, result\v\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | TEST="${result}" passtest diff -u --label "EXPECTED: ${result}" <( fangle -R${result} $TXT_SRC ) \
  2  |                                  --label "ACTUAL: ${chunk}" <( fangle -R${chunk} $TXT_SRC )
     |________________________________________________________________________


Chapter 20Chunk Parameters
20.1 L Y X 

101a <test:lyx:chunk-params:sub[1](THING\v, colour\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | I see a ${THING},
  2  | a ${THING} of colour ${colour}, 
  3  | and looking closer =<\chunkref{test:lyx:chunk-params:sub:sub}(${colour})>
     |________________________________________________________________________



101b <test:lyx:chunk-params:sub:sub[1](colour\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | a funny shade of ${colour}
     |________________________________________________________________________



101c <test:lyx:chunk-params:text[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | What do you see? "=<\chunkref{test:lyx:chunk-params:sub}(joe, red)>"
  2  | Well, fancy!
     |________________________________________________________________________


Should generate output:

101d <test:lyx:chunk-params:result[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | What do you see? "I see a joe,
  2  |                   a joe of colour red, 
  3  |                   and looking closer a funny shade of red"
  4  | Well, fancy!
     |________________________________________________________________________


And this chunk will perform the test:

101e <test:chunk-params[1](\v), lang=> ≡  102b⊳
      ________________________________________________________________________
  1  | «test:test-chunk-result\v(test:lyx:chunk-params:text\v, test:lyx:chunk-params:result\v) 100c» || exit 1
     |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 
20.2 TeXmacs 

101f <test:chunk-params:sub[1](THING\v, colour\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | I see a ${THING},
  2  | a ${THING} of colour ${colour}, 
  3  | and looking closer «test:chunk-params:sub:sub\v(${colour}\v) 101g»
     |________________________________________________________________________



101g <test:chunk-params:sub:sub[1](colour\v\v), lang=> ≡ 
      ________________________________________________________________________
  1  | a funny shade of ${colour}
     |________________________________________________________________________



101h <test:chunk-params:text[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | What do you see? "«test:chunk-params:sub\v(joe\v, red\v) 101f»"
  2  | Well, fancy!
     |________________________________________________________________________


Should generate output:

102a <test:chunk-params:result[1](\v), lang=> ≡ 
      ________________________________________________________________________
  1  | What do you see? "I see a joe,
  2  |                   a joe of colour red, 
  3  |                   and looking closer a funny shade of red"
  4  | Well, fancy!
     |________________________________________________________________________


And this chunk will perform the test:

102b <test:chunk-params[2](\v) ⇑101e, lang=> +≡ ⊲101e
     ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
  2  | «test:test-chunk-result\v(test:chunk-params:text\v, test:chunk-params:result\v) 100c» || exit 1
     |________________________________________________________________________


Chapter 21Compile-log-lyx

103a <Chunk:./compile-log-lyx[1](\v), lang=sh> ≡ 
      ________________________________________________________________________
  1  | #! /bin/sh
  2  | # can't use gtkdialog -i, cos it uses the "source" command which ubuntu sh doesn't have
  3  | 
  4  | main() {
  5  |   errors="/tmp/compile.log.$$"
  6  | #  if grep '^[^ ]*:\( In \|[0-9][0-9]*: [^ ]*:\)' > $errors
  7  | if grep '^[^ ]*(\([0-9][0-9]*\)) *: *\(error\|warning\)' > $errors
  8  |   then
  9  |     sed -i -e 's/^[^ ]*[/\\]\([^/\\]*\)(\([ 0-9][ 0-9]*\)) *: */\1:\2|\2|/' $errors
  10  |     COMPILE_DIALOG='
  11  |  <vbox>
  12  |   <text>
  13  |     <label>Compiler errors:</label>
  14  |   </text>
  15  |   <tree exported_column="0">
  16  |     <variable>LINE</variable>
  17  |     <height>400</height><width>800</width>
  18  |     <label>File | Line | Message</label>
  19  |     <action>'". $SELF ; "'lyxgoto $LINE</action>
  20  |     <input>'"cat $errors"'</input>
  21  |   </tree>
  22  |   <hbox>
  23  |    <button><label>Build</label>
  24  |      <action>lyxclient -c "LYXCMD:build-program" &</action>
  25  |    </button>
  26  |    <button ok></button>
  27  |   </hbox>
  28  |  </vbox>
  29  | '
  30  |     export COMPILE_DIALOG
  31  |     ( gtkdialog --program=COMPILE_DIALOG ; rm $errors ) &
  32  |   else
  33  |     rm $errors
  34  |   fi
  35  | }
  36  | 
  37  | lyxgoto() {
  38  |   file="${LINE%:*}"
  39  |   line="${LINE##*:}"
  40  |   extraline=‘cat $file | head -n $line | tac | sed '/^\\\\begin{lstlisting}/q' | wc -l‘
  41  |   extraline=‘expr $extraline - 1‘
  42  |   lyxclient -c "LYXCMD:command-sequence server-goto-file-row $file $line ; char-forward ; repeat $extraline paragraph-down ; paragraph-up-select"
  43  | }
  44  | 
  45  | SELF="$0"
  46  | if test -z "$COMPILE_DIALOG"
  47  | then main "$@" 
  48  | fi
     |________________________________________________________________________