Fix newfangle.module, fix generation of non-defined languages

[newfangle.git] / newfangle.lyx

#LyX 1.6.4 created this file. For more info see http://www.lyx.org/
\lyxformat 345
\begin_document
\begin_header
\textclass book
\begin_preamble
%\usepackage{xcolor}
%\definecolor{darkgreen}{rgb}{0,0.5,0}
\lstset{numbers=left, stepnumber=1, numbersep=5pt, breaklines=false,
basicstyle=\footnotesize\ttfamily,
%keywordstyle=\color{darkgreen},
numberstyle=\tiny,language=C,columns=fullflexible,
numberfirstline=true}
\end_preamble
\use_default_options true
\begin_modules
logicalmkup
newfangle
\end_modules
\language english
\inputencoding auto
\font_roman default
\font_sans default
\font_typewriter default
\font_default_family default
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100

\graphics default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_title "Newfangle"
\pdf_author "Sam Liddicott"
\pdf_subject "Literate Programing"
\pdf_keywords "notangle noweb noweave literate programming cweb"
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks false
\pdf_pdfborder false
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry false
\use_amsmath 1
\use_esint 1
\cite_engine basic
\use_bibtopic false
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation skip
\defskip medskip
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\author "" 
\author "" 
\end_header

\begin_body

\begin_layout Title
newfangle
\end_layout

\begin_layout Author
Sam Liddicott
\end_layout

\begin_layout Date
August 2009
\end_layout

\begin_layout Chapter*
Introduction
\end_layout

\begin_layout Standard

\noun on
Newfangle
\noun default
 is a tool for newfangled literate programming.
 Newfangled is defined as 
\emph on
New and often needlessly novel
\emph default
 by 
\noun on
TheFreeDictionary.com
\noun default
.
\end_layout

\begin_layout Standard
In this case, newfangled means yet another new and improved method for literate
 programming.
\end_layout

\begin_layout Standard

\noun on
Literate Programming
\noun default
 has a long history starting with the great 
\noun on
Donald Knuth
\noun default
 himself, whose literate programming tools seem to make use of as many escaped
 abbreviations for semantic markup as TeX itself.
\end_layout

\begin_layout Standard

\noun on
Norman Ramsey
\noun default
 wrote the 
\noun on
noweb
\noun default
 set of tools (notangle, noweave and noroots) and helpfully reduced the
 amount of magic character sequences to just 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
@
\end_layout

\end_inset

, and in doing so brought the wonders of literate programming within my
 reach.
\end_layout

\begin_layout Standard
While using the LyX editor for LaTeX editing I had various troubles with
 the noweb tools, some of which were my fault, some of which were noweb's
 fault and some of which were LyX's fault.
\end_layout

\begin_layout Standard

\noun on
Noweb
\noun default
 generally brought literate programming to the masses through removing some
 of the complexity of the original literate programming, but this would
 be of no advantage to me if the LyX / LaTeX combination brought more complicati
ons in their place.
\end_layout

\begin_layout Standard

\noun on
Newfangle
\noun default
 was thus born --- as an awk replacement for notangle, adding some important
 features, like better integration with LyX and LaTeX, multiple output format
 conversions, and fixing notangle bugs like indentation when using -L for
 line numbers.
\end_layout

\begin_layout Standard
Significantly, newfangle is just one program which replaces various programs
 in Noweb.
 Specifically noweave is done away with and implemented directly as LaTeX
 macros, and noroots is implemented as a function of the untangler 
\noun on
newfangle
\noun default
.
\end_layout

\begin_layout Standard
Newfangle is written in awk for portability reasons, awk being available
 for most platforms.
 A python conversion will probably be attempted for the benefit of LyX.
 (Hasn't anyone implemented awk in python yet?)
\end_layout

\begin_layout Standard
As an extension to many literate-programming styles, Newfangle permits code
 chunks to take parameters and thus operate somewhat like C pre-processor
 macros, or like C++ templates.
\end_layout

\begin_layout Section*
Todo
\end_layout

\begin_layout Enumerate
^^ is always going to be a problem, see texbytopic 1.2.2 (Work out what I
 meant by this).
\end_layout

\begin_layout Enumerate
copy over up to date Makefile guide from noweb-lyx document
\end_layout

\begin_layout Enumerate
Make chunk-name settings only apply to chunks with that name
\end_layout

\begin_layout Enumerate
indent of multi-line chunks may be mode dependant (i.e.
 not in string literals)
\end_layout

\begin_layout Enumerate
support chunk-param usage =<
\backslash
param{name}>
\end_layout

\begin_layout Enumerate
trim spaces from param
\end_layout

\begin_layout Enumerate
add support for other commands in =<...>, starting with 
\backslash
label which takes the line-number within the chunk, and maybe should also
 take the chunk name/page
\end_layout

\begin_layout Enumerate
cant have listing inside a ruled box
\end_layout

\begin_layout Enumerate
when a parameterized chunk is included as well as the #line emission, say
 what the paremeters were for that invocation.
\end_layout

\begin_layout Enumerate
with 2 macro expansions on one line ${} ${} the first is too greedy and
 looks to the final }
\end_layout

\begin_layout Enumerate

\backslash
chunkref[3]{preamble} to include a certain chunk needs to work in newfangle.awk
 instead of failing to be recognized at all
\end_layout

\begin_layout Enumerate
make in-listins labels track the chunk ref too, and make 
\backslash
chunref{[2],thing}> resolve to 41c (or d, or whatever chunk the 2nd chunk
 of thing is
\end_layout

\begin_layout Enumerate

\backslash
chunkref in text needs a trailing space maybe, it keeps butting up to the
 next word
\end_layout

\begin_layout Enumerate
because the white-space indent is output by the parent chunk, the #line
 is that of the parent chunk.
 White space indents must be passed to the child chunk
\end_layout

\begin_layout Chapter*
License
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "cha:License"

\end_inset

Newfangle is licensed under the GPL 3
\begin_inset CommandInset citation
LatexCommand cite
key "gpl-licenses"

\end_inset

 (or later).
 This doesn't mean that you can't use or distribute newfangle with sources
 of an incompatible license, but it means you must make the source of newfangle
 available too.
\end_layout

\begin_layout Chunk
gpl3-copyright,language=
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

newfangle - fully featured notangle replacement in awk
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

Copyright (C) Sam Liddicott 2009
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

This program is free software: you can redistribute it and/or modify
\end_layout

\begin_layout Plain Layout

it under the terms of the GNU General Public License as published by
\end_layout

\begin_layout Plain Layout

the Free Software Foundation, either version 3 of the License, or
\end_layout

\begin_layout Plain Layout

(at your option) any later version.
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

This program is distributed in the hope that it will be useful,
\end_layout

\begin_layout Plain Layout

but WITHOUT ANY WARRANTY; without even the implied warranty of
\end_layout

\begin_layout Plain Layout

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  See the
\end_layout

\begin_layout Plain Layout

GNU General Public License for more details.
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

You should have received a copy of the GNU General Public License
\end_layout

\begin_layout Plain Layout

along with this program.
  If not, see <http://www.gnu.org/licenses/>.
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset CommandInset toc
LatexCommand tableofcontents

\end_inset


\end_layout

\begin_layout Part
Using Newfangle
\end_layout

\begin_layout Chapter
Running Newfangle
\end_layout

\begin_layout Standard
Newfangle is a replacement for noweb, which consists of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
notangle
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noroots
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noweave
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Like 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
notangle
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noroots
\end_layout

\end_inset

 it can read multiple named files, or from stdin.
\end_layout

\begin_layout Section
Listing roots
\end_layout

\begin_layout Standard
The -r option causes newfangle to behave like noroots.
\end_layout

\begin_layout LyX-Code
newfangle -r filename.tex
\end_layout

\begin_layout Standard
will print out the newfangle roots of a tex file.
 
\end_layout

\begin_layout Standard
Unlike the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noroots
\end_layout

\end_inset

 command, the roots are not enclosed in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<name>>
\end_layout

\end_inset

, unless at least one of the roots is defined using the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
notangle
\end_layout

\end_inset

 style 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<name>>=
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Also, unlike noroots, it prints out all roots --- not just those that are
 not used elsewhere.
 I find that a root not being used, doesn't make it particularly top level.
 My convention is that top level roots to be extracted begin with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
./
\end_layout

\end_inset

 and have the form of a filename.
\end_layout

\begin_layout Section
Extracting roots
\end_layout

\begin_layout Standard
notangle's 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-R
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-L
\end_layout

\end_inset

 options are supported.
\end_layout

\begin_layout Standard
The standard way to extract a file would be:
\end_layout

\begin_layout LyX-Code
newfangle -R./Makefile.inc newfangle.tex > ./Makefile.inc
\end_layout

\begin_layout Standard
Unlike the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noroots
\end_layout

\end_inset

 command, the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-L
\end_layout

\end_inset

 option does not break indenting; also the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-L
\end_layout

\end_inset

 option does not interrupt (and break) multi-line C macros --- or indeed
 any line ending with a backslash.
 This does mean that sometimes the compiler might calculate the source line
 wrongly when generating error messages in such cases, but there isn't any
 other way around if multi-line macros include other chunks.
\end_layout

\begin_layout Section
Formatting source in LaTeX
\end_layout

\begin_layout Standard
The noweave replacement is a set of LaTeX macros dependant upon 
\emph on
noweb.sty
\emph default
, and which can be included with:
\end_layout

\begin_layout LyX-Code

\backslash
usepackage{newfangle.sty}
\end_layout

\begin_layout Standard
The LaTeX macros are shown in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Latex-Macros"

\end_inset

, and are part of a LyX module file 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle.module
\end_layout

\end_inset

, which automatically includes the macros in the document pre-amble when
 the newfangle LyX module is used.
\end_layout

\begin_layout Standard
Because the noweave replacement is impemented in LaTeX, there is no processing
 stage required before running the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
latex
\end_layout

\end_inset

 command.
 LaTeX may need running two or more times, so that the code chunk references
 can be fully calculated.
\end_layout

\begin_layout Standard
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
noweb.sty
\end_layout

\end_inset

 package is required as it is used for formatting the code chunk captions
\end_layout

\begin_layout Standard
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
listings.sty
\end_layout

\end_inset

 package is also required, as it is used for formatting the code chunks
 themselves.
\end_layout

\begin_layout Standard
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
xargs.sty
\end_layout

\end_inset

 package is also required.
\end_layout

\begin_layout Chapter
Literate Programming with Newfangle
\end_layout

\begin_layout Standard
Todo.
 Should really follow on from a part-0 explanation of what literate programming
 is.
\end_layout

\begin_layout Chapter
Using Newfangle with LyX
\end_layout

\begin_layout Section
Setting up Lyx
\end_layout

\begin_layout Subsection
Installing the LyX module
\end_layout

\begin_layout Standard
Copy 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle.module
\end_layout

\end_inset

 to your LyX layouts directory, which for unix users will be 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
~/.lyx/layouts
\end_layout

\end_inset


\end_layout

\begin_layout Standard
You will need to reconfigure LyX by clicking Tools\SpecialChar \menuseparator
Reconfigure, and then
 re-start LyX.
\end_layout

\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sub:Configuring-the-build"

\end_inset

Configuring the build script
\end_layout

\begin_layout Standard
Make sure you don't have a conversion defined for Lyx → Program
\end_layout

\begin_layout Standard
From the menu Tools\SpecialChar \menuseparator
Preferences, add a conversion from Latex(Plain) → Program
 as:
\end_layout

\begin_layout LyX-Code
set -x ; newfangle -Rlyx-build $$i | 
\end_layout

\begin_layout LyX-Code
  env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
\end_layout

\begin_layout Standard
(But don't cut-n-paste it from this document or you'll be pasting a multi-line
 string which will break your lyx preferences file).
 
\end_layout

\begin_layout Standard
I hope that one day, LyX will set these into the environment when calling
 the build script.
\end_layout

\begin_layout Standard
You may also want to consider adding options to this conversion\SpecialChar \ldots{}

\end_layout

\begin_layout LyX-Code
parselog=/usr/share/lyx/scripts/listerrors
\end_layout

\begin_layout Standard
\SpecialChar \ldots{}
but if you do you will lose your stderr
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
There is some bash plumbing to get a copy of stderr but this footnote is
 too small
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Now, a shell script chunk called 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
lyx-build
\end_layout

\end_inset

 will be extracted and run whenever you choose the Document\SpecialChar \menuseparator
Build Program
 menu item.
\end_layout

\begin_layout Standard
The lyx-build script for this document is in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "lyx-build-script"

\end_inset

 and on a unix system will extract 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle.module
\end_layout

\end_inset

 and the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle
\end_layout

\end_inset

 awk script.
\end_layout

\begin_layout Subsection
Preparing your Lyx document
\end_layout

\begin_layout Standard
It is not necessary to base your literate document on any of the original
 LyX literate classes; so select a regular class for your document type.
\end_layout

\begin_layout Standard
Add the new module 
\emph on
Newfangle Literate
\emph default
 
\emph on
Listings
\emph default
 and possibly also 
\emph on
Logical Markup
\emph default
.
\end_layout

\begin_layout Standard
In the drop-down style listbox you should notice a new style defined, called
 
\emph on
Chunk
\emph default
.
\end_layout

\begin_layout Standard
When you wish to insert a literate chunk, you enter it's plain name in the
 Chunk style, instead of the older method that used 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<name>>=
\end_layout

\end_inset

 type tags.
 Following the chunk name, you insert a listing with: Insert\SpecialChar \menuseparator
Program Listing.
\end_layout

\begin_layout Standard
Inside the white listing box you can type (or paste using shift+ctrl+V)
 your listing.
 There is not need to use ctrl+enter at the end of lines as with some older
 LyX literate techniques --- just press enter as normal.
\end_layout

\begin_layout Subsubsection
Customising the listing appearance
\end_layout

\begin_layout Standard
In the final document, the code is formatted using the 
\noun on
lstlistings
\noun default
 package.
 The chunk style doesn't just define the chunk name, but can also define
 any other chunk options supported by the lstlistings package 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

 command.
 In fact, what you type in the chunk style is raw latex.
 If you want to set the chunk language without having to right-click the
 listing, just add 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
,lanuage=C
\end_layout

\end_inset

 after the chunk name.
\end_layout

\begin_layout Standard
Of course you can do this by editing the listings box advanced properties
 by right-clicking on the listings box, but that takes longer, and you can't
 see at-a-glance what the advanced settings are while editing the document;
 also advanced settings apply only to that box --- the chunk settings apply
 through the rest of the document
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
It ought to apply only to subsequent chunks of the same name.
 I'll fix that later
\end_layout

\end_inset


\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
So make sure they only apply to chunks of that name
\end_layout

\end_inset

.
\end_layout

\begin_layout Subsubsection
Global customisations
\end_layout

\begin_layout Standard
As 
\emph on
lstlistings
\emph default
 is used to set the code chunks, it's 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

 command can be used in the pre-amble to set some document wide settings.
\end_layout

\begin_layout Standard
If your source has many words with long sequences of capital letters, then
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
columns=fullflexible
\end_layout

\end_inset

 may be a good idea, or the capital letters will get crowded.
 (I think lstlistings ought to use a slightly smaller font for captial letters
 so that they still fit).
\end_layout

\begin_layout Standard
The font family 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
ttfamily
\end_layout

\end_inset

 looks more normal for code, but has no bold (unless luximono is used, but
 it doesn't work for me); so I use 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
color{darkgreen}
\end_layout

\end_inset

 for my keywords.
 With 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
ttfamily
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
columns=fullflexible
\end_layout

\end_inset

 is used or the wrong letter spacing is used.
\end_layout

\begin_layout Standard
In my LeTeX pre-amble I usually specialise my code format with:
\end_layout

\begin_layout Chunk
document-preamble,language=tex
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
usepackage{xcolor}
\end_layout

\begin_layout Plain Layout


\backslash
definecolor{darkgreen}{rgb}{0,0.5,0}
\end_layout

\begin_layout Plain Layout


\backslash
lstset{numbers=left, stepnumber=5, numbersep=5pt, breaklines=false,
\end_layout

\begin_layout Plain Layout

  basicstyle=
\backslash
footnotesize
\backslash
ttfamily,
\end_layout

\begin_layout Plain Layout

  keywordstyle=
\backslash
color{darkgreen},
\end_layout

\begin_layout Plain Layout

  numberstyle=
\backslash
tiny,language=C,columns=fullflexible,
\end_layout

\begin_layout Plain Layout

  numberfirstline=true
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Newfangle with Makefiles
\end_layout

\begin_layout Standard
\begin_inset Note Note
status open

\begin_layout Plain Layout
This chapter needs revising
\end_layout

\end_inset


\begin_inset Note Greyedout
status open

\begin_layout Plain Layout
This chapter needs revising
\end_layout

\end_inset

Here we describe a Makefile.inc that you can include in your own Makefiles,
 or glue as a recursive make to other projects.
\end_layout

\begin_layout Standard
The Makefile.inc described here was put together for a Samba4 vfs module,
 but can be used in any Make project, including automake projects.
\end_layout

\begin_layout Section
A word about makefiles formats
\end_layout

\begin_layout Standard
Whitespace formatting is very important in a Makefile.
 The first character of each command line must be a TAB.
\end_layout

\begin_layout LyX-Code
target: pre-requisite
\begin_inset Newline newline
\end_inset

   →    action
\begin_inset Newline newline
\end_inset

   →    action
\end_layout

\begin_layout Standard
But a TAB is pretty hard to enter into most of the Lyx formats and insets
 I've been using.
 An alternative is to use a semi-colon after the pre-requisite, and a backslash
 at the end of each line (except the last).
 Then any whitespace (or none) can prefix each action.
\end_layout

\begin_layout LyX-Code
target: pre-requisite ; 
\backslash

\begin_inset Newline newline
\end_inset

␣␣action 
\backslash

\begin_inset Newline newline
\end_inset

␣␣action
\end_layout

\begin_layout Standard
This is the style that we use and it works pretty well for GNU make at least.
\end_layout

\begin_layout Standard
We also adopt a convention that code chunks whose names beginning with ./
 should always be automatically extracted from the document.
 Code chunks whose names do not begin with ./ are for internal reference.
 (This doesn't prevent such chunks from being extracted directly).
\end_layout

\begin_layout Section
Boot-strapping the extraction
\end_layout

\begin_layout Subsection
Using a Makefile
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "sub:Bootstrap-Using-a-Makefile"

\end_inset

It seems convenient to have the makefile extract or update the C source
 files as part of it's operation.
 It also seems convenient to have the makefile itself extracted from this
 document.
\end_layout

\begin_layout Standard
It would also be convenient to have the code to extract the makefile from
 this document to also be part of this document, however we have to start
 somewhere and this unfortunately requires us to type at least a few words
 by hand to start things off.
\end_layout

\begin_layout Standard
Therefore we will have a minimal root fragment, which, when extracted, can
 cope with extracting the rest of the source.
 perhaps with this shell script, which could be called 
\emph on
autoboot
\emph default
.
\begin_inset Note Note
status open

\begin_layout Plain Layout
FIX THIS CHUNK AND TEST IT
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
*
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

#! /bin/sh
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
\end_layout

\begin_layout Plain Layout

MAKE_SRC=`dirname "$MAKE_SRC"`/`basename "$MAKE_SRC" .lyx`
\end_layout

\begin_layout Plain Layout

NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
\end_layout

\begin_layout Plain Layout

lyx -e latex $MAKE_SRC
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

newfangle -R./Makefile.inc ${MAKE_SRC}.tex 
\backslash

\end_layout

\begin_layout Plain Layout

  | sed "/NEWFANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$NEWFANGLE_SRC" 
\backslash

\end_layout

\begin_layout Plain Layout

  | cpif ./Makefile.inc
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

make -f ./Makefile.inc newfangle_sources
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The general Makefile can be invoked with 
\emph on
./autoboot
\emph default
 and can also be included into any automake file to automatically re-generate
 the source files.
\end_layout

\begin_layout Standard
The 
\emph on
autoboot
\emph default
 can be extracted with this command:
\end_layout

\begin_layout LyX-Code
lyx -e latex newfangle.lyx && 
\backslash

\end_layout

\begin_layout LyX-Code
newfangle newfangle.lyx > ./autoboot
\end_layout

\begin_layout Standard
This looks simple enough, but as mentioned, newfangle has to be had from
 somewhere before it can be extracted.
\end_layout

\begin_layout Subsection
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
MERGE THIS WITH THE SECTIONS OF THIS DOCUMENT
\end_layout

\end_inset

\SpecialChar \ldots{}

\end_layout

\begin_layout Standard
When the lyx-build chunk is executed, the current directory will be a temporary
 directory, and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
LYX_SOURCE
\end_layout

\end_inset

 will refer to the tex file in this temporary directory.
 This is unfortunate as our makefile wants to run from the project directory
 where the Lyx file is kept.
\end_layout

\begin_layout Standard
We can extract the project directory from $$r, and derive the probable Lyx
 filename from the noweb file that Lyx generated.
\end_layout

\begin_layout Chunk
lyx-build-helper
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

PROJECT_DIR="$LYX_r"
\end_layout

\begin_layout Plain Layout

LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
\end_layout

\begin_layout Plain Layout

TEX_DIR="$LYX_p"
\end_layout

\begin_layout Plain Layout

TEX_SRC="$TEX_DIR/$LYX_i"
\end_layout

\end_inset


\end_layout

\begin_layout Standard
And then we can define a lyx-build fragment similar to the autoboot fragment
\end_layout

\begin_layout Chunk
lyx-build
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

#! /bin/sh
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{lyx-build-helper}>
\end_layout

\begin_layout Plain Layout

cd $PROJECT_DIR || exit 1
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

#/usr/bin/newfangle -filter ./notanglefix-filter 
\backslash

\end_layout

\begin_layout Plain Layout

#  -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx" 
\backslash

\end_layout

\begin_layout Plain Layout

#  | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/' 
\backslash

\end_layout

\begin_layout Plain Layout

#  > ./Makefile.inc
\end_layout

\begin_layout Plain Layout

#
\end_layout

\begin_layout Plain Layout

#make -f ./Makefile.inc newfangle_sources
\end_layout

\end_inset


\end_layout

\begin_layout Section
Extracting Sources
\end_layout

\begin_layout Subsection
Including Makefile.inc
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "sub:Keeping-extracted-files"

\end_inset

Makefile.inc will cope with extracting all the other source files from this
 document and keeping them up to date.
 
\end_layout

\begin_layout Standard
It may also be included by a Makefile or Makefile.am defined in a Lyx document
 to automatically deal with the extraction of source files and documents.
\end_layout

\begin_layout Standard
A makefile has two parts; variables must be defined before the targets that
 use them.
\end_layout

\begin_layout Chunk
./Makefile.inc
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{Makefile.inc-vars}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{Makefile.inc-targets}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We first define 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
NOWEB_SOURCE
\end_layout

\end_inset

 to hold the name of this Lyx file.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

LYX_SOURCE=
\end_layout

\begin_layout Plain Layout

LITERATE_SOURCE=$(LYX_SOURCE)
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Recursive use of Makefile.inc
\end_layout

\begin_layout Standard
The makefile glue described here is used when building Samba4 vfs modules.
\end_layout

\begin_layout Standard
If you are defining a module of an existing program you may find it easier
 to use a slight recursive make instead of including the makefile directly.
 This way there is less chance of definitions in Makefile.inc interfering
 with definitions in the main makefile, or with definitions in other Makefile.inc
 from other noweb modules.
\end_layout

\begin_layout Standard
The glue works by adding a .PHONY target to call the recursive make, and
 adding this target as an additional pre-requisite to the existing targets.
\end_layout

\begin_layout Standard
In this example, the existing build system already has a build target for
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
example.o
\end_layout

\end_inset

, so we just add another pre-requisite to that.
 In this case we use 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
example.tex.stamp
\end_layout

\end_inset

 as a pre-requisite, the stamp file's modified time indicating when all
 sources were extracted.
\end_layout

\begin_layout Chunk
makefile-glue
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(example_srcdir)/example.o: $(example_srcdir)/example.tex.stamp
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The target for this new pre-requisite is generated by a recursive make using
 Makefile.inc which will make sure that the source is up to date, before
 it is built by the main projects makefile.
\end_layout

\begin_layout Chunk
makefile-glue
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(example_srcdir)/example.tex.stamp: $(example_srcdir)/example.tex ; 
\backslash

\end_layout

\begin_layout Plain Layout

        cd $(example_srcdir) && 
\backslash

\end_layout

\begin_layout Plain Layout

        $(MAKE) -f Makefile.inc newfangle_sources
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can do similar glue for the docs, clean and distclean targets.
 In this example our build system is using a double colon for these targets,
 so we use the same in our glue.
\end_layout

\begin_layout Chunk
makefile-glue
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

docs:: docs_example
\end_layout

\begin_layout Plain Layout

.PHONY: docs_example
\end_layout

\begin_layout Plain Layout

docs_example:: ; cd $(example_srcdir) && 
\backslash

\end_layout

\begin_layout Plain Layout

        $(MAKE) -f Makefile.inc docs
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

clean:: clean_example
\end_layout

\begin_layout Plain Layout

.PHONEY: clean_example
\end_layout

\begin_layout Plain Layout

clean_example: ; cd $(example_srcdir) && 
\backslash

\end_layout

\begin_layout Plain Layout

        $(MAKE) -f Makefile.inc clean
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

distclean:: distclean_example
\end_layout

\begin_layout Plain Layout

.PHONY: distclean_example
\end_layout

\begin_layout Plain Layout

distclean_example: ; cd $(example_srcdir) && 
\backslash

\end_layout

\begin_layout Plain Layout

        $(MAKE) -f Makefile.inc distclean
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We could do similarly for install targets to install the generated docs.
\end_layout

\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sub:Converting-from-Lyx"

\end_inset

Converting from Lyx to LaTeX
\end_layout

\begin_layout Standard
The first stage will always be to convert the Lyx file to a LaTeX file;
 this must be so not only because newfangle needs to to run on a TeX file,
 but also because the Lyx command 
\emph on
server-goto-file-line
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
The Lyx command 
\emph on
server-goto-file-line
\emph default
 is used to position the Lyx cursor at the compiler errors.
\end_layout

\end_inset


\emph default
 insists that the line number provided is a line in the TeX file, and always
 reverse maps this to derive the line in the Lyx docment.
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
The tex file should probably be an automake extra dist sources or something,
 so that it gets produced and packaged by make dist
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The command [[lyx -e literate noweb-lyx.lyx]] will produce [[noweb-lyx.nw]]
 a tex file, so we define the noweb target to be the same as the Lyx file
 but with the .nw extension.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(TEX_SOURCE): $(LYX_SOURCE) ;
\backslash

\end_layout

\begin_layout Plain Layout

        lyx -e latex $<
\end_layout

\begin_layout Plain Layout

clean_tex: ; rm -f -- $(TEX_SOURCE)
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Extracting Program Source
\end_layout

\begin_layout Standard
The program source is extracted using newfangle, which is designed to operate
 on a LaTeX document.
 
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

NEWFANGLE_SOURCE=$(TEX_SOURCE)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The Lyx document can result in any number of source documents, but not all
 of these will be changed each time the Lyx document is updated.
 We certainly don't want to update the timestamps of these files and cause
 the whole source tree to be recompiled just because the Lyx document was
 edited.
 
\end_layout

\begin_layout Standard
To solve this problem we use a stamp file which is always updated each time
 the sources are extracted from the LaTeX document.
 If the stamp file is older than the LaTeX document, then we can make an
 attempt to re-extract the sources.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

NEWFANGLE_SOURCE_STAMP=$(NEWFANGLE_SOURCE).stamp
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(NEWFANGLE_SOURCE_STAMP): $(NEWFANGLE_SOURCE) 
\backslash

\end_layout

\begin_layout Plain Layout

                           $(NEWFANGLE_SOURCES) ; 
\backslash

\end_layout

\begin_layout Plain Layout

        echo > $(NEWFANGLE_SOURCE_STAMP)
\end_layout

\begin_layout Plain Layout

clean_stamp: ; rm -f $(NEWFANGLE_SOURCE_STAMP)
\end_layout

\begin_layout Plain Layout

clean: clean_stamp
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Extracting C sources
\end_layout

\begin_layout Standard
We compute 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
NEWFANGLE_SOURCES
\end_layout

\end_inset

 to hold the names of all the C source files defined in this document.
 We compute this only once, by means of := in assignent.
 The sed deletes the any <
\begin_inset space \hspace*{}
\length 0in
\end_inset

< and >
\begin_inset space \hspace*{}
\length 0in
\end_inset

> which may surround the roots names (for noroots compatibility).
 
\end_layout

\begin_layout Standard
As we use chunk names beginning with ./ to denote top level fragments that
 should be extracted, we filter out all fragments that do not begin with
 ./
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

NEWFANGLE_PREFIX:=
\backslash
.
\backslash
/
\end_layout

\begin_layout Plain Layout

NEWFANGLE_SOURCES:=$(shell 
\backslash

\end_layout

\begin_layout Plain Layout

  newfangle -r $(NEWFANGLE_SOURCE) |
\backslash

\end_layout

\begin_layout Plain Layout

  sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(NEWFANGLE_PREFIX)/!d' 
\backslash

\end_layout

\begin_layout Plain Layout

      -e 's/^$(NEWFANGLE_PREFIX)/
\backslash
.
\backslash
//' )
\end_layout

\begin_layout Plain Layout

@
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

.PHONY: echo_newfangle_sources
\end_layout

\begin_layout Plain Layout

echo_newfangle_sources: ; @echo $(NEWFANGLE_SOURCES)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We define a convenient target called 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle_sources
\end_layout

\end_inset

 to re-extract the source if the LaTeX file has been updated.
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

.PHONY: newfangle_sources
\end_layout

\begin_layout Plain Layout

newfangle_sources: $(NEWFANGLE_SOURCE_STAMP)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
And also a convenient target to remove extracted sources.
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

.PHONY: clean_newfangle_sources
\end_layout

\begin_layout Plain Layout

clean_newfangle_sources: ; 
\backslash

\end_layout

\begin_layout Plain Layout

        rm -f -- $(NEWFANGLE_SOURCE_STAMP) $(NEWFANGLE_SOURCES)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
if_extension
\end_layout

\end_inset

 macro takes 4 arguments: the filename (1), some extensions to match (2)
 and a some shell command to return if the filename matches the exentions
 (3), or not (4).
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
\end_layout

\end_inset


\end_layout

\begin_layout Standard
For some source files like C files, we want to output the line number and
 filename of the original LaTeX document from which the source came.
\end_layout

\begin_layout Standard
To make this easier we define the file extensions for which we want to do
 this.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

C_EXTENSIONS=.c .h
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can then use the if_extensions macro to define a macro which expands
 out to the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-L
\end_layout

\end_inset

 option if newfangle is being invoked in a C source file, so that C compile
 errors will refer to the line number in the Lyx document.
 
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

TABS=8
\end_layout

\begin_layout Plain Layout

nf_line=-L -T$(TABS)
\end_layout

\begin_layout Plain Layout

newfangle=newfangle 
\backslash

\end_layout

\begin_layout Plain Layout

  $(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line)) 
\backslash

\end_layout

\begin_layout Plain Layout

    -R"$(2)" $(1)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can use a similar trick to define an 
\emph on
indent
\emph default
 macro which takes just the filename as an argument and can return a pipeline
 stage calling the indent command.
 Indent can be turned off with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
make newfangle_sources indent=
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
\end_layout

\begin_layout Plain Layout

indent=$(call if_extension,$(1),$(C_EXTENSIONS),
\backslash

\end_layout

\begin_layout Plain Layout

              | indent $(indent_options))
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We now define the pattern for extracting a file.
 The files are written using noweb's 
\emph on
cpif
\begin_inset Foot
status collapsed

\begin_layout Plain Layout

\emph on
So you still need noweb installed in order to use cpif
\end_layout

\end_inset


\begin_inset Note Note
status collapsed

\begin_layout Plain Layout

\emph on
Write an awk version
\end_layout

\end_inset


\emph default
 so that the file timestamp will not be touched if the contents haven't
 changed.
 This avoids the need to rebuild the entire project because of a typographical
 change in the documentation, or if only a few C source files have changed.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

newfangle_extract=@mkdir -p $(dir $(1)) && 
\backslash

\end_layout

\begin_layout Plain Layout

  $(call newfangle,$(2),$(1)) > "$(1).tmp" && 
\backslash

\end_layout

\begin_layout Plain Layout

  cat "$(1).tmp" $(indent) | cpif "$(1)" 
\backslash

\end_layout

\begin_layout Plain Layout

  && rm -- "$(1).tmp" || 
\backslash

\end_layout

\begin_layout Plain Layout

  (echo error newfangling $(1) from $(2) ; exit 1)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We define a target which will extract or update all sources.
 To do this we first defined a makefile template that can do this for any
 source file in the LaTeX document.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

define NEWFANGLE_template
\end_layout

\begin_layout Plain Layout

  $(1): $(2); 
\backslash

\end_layout

\begin_layout Plain Layout

    $$(call newfangle_extract,$(1),$(2))
\end_layout

\begin_layout Plain Layout

  NEWFANGLE_TARGETS+=$(1)
\end_layout

\begin_layout Plain Layout

endef
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then enumerate the discovered 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
NEWTANGLE_SOURCES
\end_layout

\end_inset

 to generate a makefile rule for each one using the makefile template we
 defined above.
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(foreach source,$(NEWFANGLE_SOURCES),
\backslash

\end_layout

\begin_layout Plain Layout

  $(eval $(call NEWFANGLE_template,$(source),$(NEWFANGLE_SOURCE))) 
\backslash

\end_layout

\begin_layout Plain Layout

)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
These will all be built with NEWFANGLE_SOURCE_STAMP.
\end_layout

\begin_layout Standard
We also remove the generated sources on a 
\emph on
make distclean
\emph default
.
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

_distclean: clean_newfangle_sources
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Extracting Documentation
\end_layout

\begin_layout Standard
We then identify the intermediate stages of the documentation and their
 build and clean targets.
\end_layout

\begin_layout Subsubsection
Running pdflatex
\end_layout

\begin_layout Standard
We produce a pdf file from the tex file.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

NEWFANGLE_PDF=$(TEX_SOURCE:.tex=.pdf)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We run pdflatex twice to be sure that the contents and aux files are up
 to date.
 We certainly are required to run pdflatex twice if these files do not exist!
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

$(NEWFANGLE_PDF): $(TEX_SOURCE); pdflatex $< && pdflatex $<
\end_layout

\begin_layout Plain Layout

clean_pdf: ; rm -f -- $(NEWFANGLE_PDF) 
\backslash

\end_layout

\begin_layout Plain Layout

                      $(TEX_SOURCE:.tex=.toc) 
\backslash

\end_layout

\begin_layout Plain Layout

                      $(TEX_SOURCE:.tex=.log) 
\backslash

\end_layout

\begin_layout Plain Layout

                      $(TEX_SOURCE:.tex=.aux)
\end_layout

\end_inset


\end_layout

\begin_layout Subsubsection
The docs as a whole
\end_layout

\begin_layout Standard
Currently we only build pdf as a final format, but NEWFANGLE_DOCS may later
 hold other output formats.
\end_layout

\begin_layout Chunk
Makefile.inc-vars
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

NEWFANGLE_DOCS=$(NEWFANGLE_PDF)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also define newfangle_docs as a convenient phony target<
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

.PHONY: newfangle_docs
\end_layout

\begin_layout Plain Layout

newfangle_docs: $(NEWFANGLE_DOCS)
\end_layout

\begin_layout Plain Layout

docs: newfangle_docs
\end_layout

\end_inset


\end_layout

\begin_layout Standard
And define a convenient clean_noweb_docs which we add to the regular clean
 target
\end_layout

\begin_layout Chunk
Makefile.inc-targets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

.PHONEY: clean_newfangle_docs
\end_layout

\begin_layout Plain Layout

clean_newfangle_docs: clean_tex clean_pdf
\end_layout

\begin_layout Plain Layout

clean: clean_newfangle_docs
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

distclean_newfangle_docs: clean_tex clean_newfangle_docs
\end_layout

\begin_layout Plain Layout

distclean: clean distclean_newfangle_docs
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Other helpers
\end_layout

\begin_layout Standard
If Makefile.inc is included into Makefile, then extracted files can be updated
 with this command:
\end_layout

\begin_layout LyX-Code
make newfangle_sources
\end_layout

\begin_layout Standard
otherwise, with:
\end_layout

\begin_layout LyX-Code
make -f Makefile.inc newfangle_sources
\end_layout

\begin_layout Part
Source Code
\end_layout

\begin_layout Chapter
Newfangle awk source code
\end_layout

\begin_layout Standard
We use the copyright notice from chapter 
\begin_inset CommandInset ref
LatexCommand vref
reference "cha:License"

\end_inset

.
\end_layout

\begin_layout Chunk
./newfangle,language=awk,morestring=[b]{/},morekeywords=else
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

#! /usr/bin/awk -f
\end_layout

\begin_layout Plain Layout

# =<
\backslash
chunkref{gpl3-copyright}>
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also use code from Arnold Robbins public domain getopt (1993 revision)
 defined in chapter 
\begin_inset CommandInset ref
LatexCommand ref
reference "cha:getopt"

\end_inset

, and naturally want to attribute this appropriately.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

# NOTE: Arnold Robbins public domain getopt for awk is also used:
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-header}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-getopt()}>
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
And include the following chunks
\end_layout

\begin_layout Chunk
./newfangle
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{helper-functions}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode-tracker}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{parse_chunk_args}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{chunk-storage-functions}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{output_chunk_names()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{output_chunks()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{write_chunk()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{expand_chunk_args()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{begin}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{recognize-chunk}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{end}>
\end_layout

\end_inset


\end_layout

\begin_layout Section
AWK tricks
\end_layout

\begin_layout Standard
The portable way to erase an array in awk is to split the empty string,
 like this:
\end_layout

\begin_layout Chunk
awk-delete-array,params=ARRAY
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

split("", ${ARRAY});
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
dump-array,params=ARRAY
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

print "
\backslash
nDump: ${ARRAY}
\backslash
n--------
\backslash
n" > "/dev/stderr";
\end_layout

\begin_layout Plain Layout

for (_x in ${ARRAY}) {
\end_layout

\begin_layout Plain Layout

  print _x "=" ${ARRAY}[_x] "
\backslash
n" > "/dev/stderr";
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

print "========
\backslash
n" > "/dev/stderr";
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
,params=
\end_layout

\begin_layout Section
Catching errors
\end_layout

\begin_layout Standard
Fatal errors are issued with the error function:
\end_layout

\begin_layout Chunk
error(),append=helper-functions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function error(message)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
\end_layout

\begin_layout Plain Layout

  exit 1;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function warning(message)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
\end_layout

\begin_layout Plain Layout

  warnings++;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
lstlistings
\end_layout

\begin_layout Standard
LaTeX arguments to lstlistings macros are a comma seperated list of key-value
 pairs.
 Values containing commas are enclosed in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

 braces 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}
\end_layout

\end_inset

, which is to be expected for LaTeX.
\end_layout

\begin_layout Standard
A sample expressions is:
\end_layout

\begin_layout LyX-Code
name=thomas, params={a, b}, something, something-else
\end_layout

\begin_layout Standard
but we see that this is just a simpler form of this expression:
\end_layout

\begin_layout LyX-Code
name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
\end_layout

\begin_layout Standard
We may consider that we need a function that can parse such LaTeX expressions
 and assign the values to an 
\noun on
AWK
\noun default
 associated array, perhaps using a recursive parser into a multi-dimensional
 hash
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
as AWK doesn't have nested-hash support
\end_layout

\end_inset

, resulting in:
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features>
<column alignment="left" valignment="top" width="0">
<column alignment="left" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
key
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
value
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[name]
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
freddie
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[foo, bar]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
baz
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[foo, quux, quirk]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[foo, quux, a]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fleeg
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[etc]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Oon reflection it seems that sometimes such nesting is not desirable, as
 the braces are also used to delimit values that contain commas --- we may
 consider that
\end_layout

\begin_layout LyX-Code
name={williamson, freddie}
\end_layout

\begin_layout Standard
should assign 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
williamson, freddie
\end_layout

\end_inset

 to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
name
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
In fact we are not so interested in the detail so as to be bothered by this,
 which turns out to be a good thing for two reasons.
 Firstly LaTeX has a malleable parser with no strict syntax, and secondly
 whether or not 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
williamson
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
freddie
\end_layout

\end_inset

 should count as two items will be context dependant anyway.
\end_layout

\begin_layout Standard
We need to parse this latex for only one reason; which is that we are extending
 lstlistings to add some additional arguments which will be used to express
 chunk parameters and other chunk options.
\end_layout

\begin_layout Section
Additional lstlstings parameters
\end_layout

\begin_layout Standard
Further on we define a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 LaTeX macro whose arguments will consist of a the chunk name, optionally
 followed by a comma and then a comma separated list of arguments.
 In fact we will just need to prefix 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
name=
\end_layout

\end_inset

 to the arguments to in order to create valid lstlistings arguments.
 
\end_layout

\begin_layout Standard
There will be other arguments supported too; 
\end_layout

\begin_layout Description
params As an extension to many literate-programming styles, newfangle permits
 code chunks to take parameters and thus operate somewhat like C pre-processor
 macros, or like C++ templates.
 Chunk parameters are declared with a chunk argument called 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
params
\end_layout

\end_inset

, which holds a semi-colon separated list of parameters, like this:
\end_layout

\begin_layout LyX-Code
achunk,language=C,params=name;address
\end_layout

\begin_layout Description
addto a named chunk that this chunk is to be included into.
 This saves the effort of having to declare another listing of the named
 chunk merely to include this one.
\end_layout

\begin_layout Standard
Function 
\begin_inset Flex Chunkref
status collapsed

\begin_layout Plain Layout
get_chunk_args()
\end_layout

\end_inset

 will accept two paramters, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
text
\end_layout

\end_inset

 being the text to parse, and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
values
\end_layout

\end_inset

 being an array to receive the parsed values as described above.
 The optional parameter 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
path
\end_layout

\end_inset

 is used during recursion to build up the multi-dimensional array path.
\end_layout

\begin_layout Chunk
./newfangle
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{get_chunk_args()}>
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
get_chunk_args()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function get_chunk_args(text, values,
\end_layout

\begin_layout Plain Layout

  # optional parameters
\end_layout

\begin_layout Plain Layout

  path, # hierarchical precursors
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  a, name)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The strategy is to parse the name, and then look for a value.
 If the value begins with a brace 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

, then we recurse and consume as much of the text as necessary, returning
 the remaining text when we encounter a leading close-brace 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}
\end_layout

\end_inset

.
 This being the strategy --- and executed in a loop --- we realise that
 we must first look for the closing brace (perhaps preceded by white space)
 in order to terminate the recursion, and returning remaining text.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  split("", next_chunk_args);
\end_layout

\begin_layout Plain Layout

  while(length(text)) {
\end_layout

\begin_layout Plain Layout

    if (match(text, "^ *}(.*)", a)) {
\end_layout

\begin_layout Plain Layout

      return a[1];
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

    =<
\backslash
chunkref{parse-chunk-args}>
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  return text;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
Use BNF package here
\end_layout

\end_inset

We can see that the text could be inspected with this regex:
\end_layout

\begin_layout Chunk
parse-chunk-args
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a))
 {
\end_layout

\begin_layout Plain Layout

  return text;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
and that 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
a
\end_layout

\end_inset

 will have the following values:
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="2">
<features>
<column alignment="center" valignment="top" width="0">
<column alignment="left" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a[n]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
assigned text
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
freddie
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
=
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
freddie
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
, foo={bar=baz, quux={quirk, a=fleeg}}, etc
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
a[3] will be either 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=
\end_layout

\end_inset

 or 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
,
\end_layout

\end_inset

 and signify whether the option named in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
a[1]
\end_layout

\end_inset

 has a value or not (respectively).
\end_layout

\begin_layout Standard
If the option does have a value, then if the expression 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
substr(a[4],1,1)
\end_layout

\end_inset

 returns a brace 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

 it will signify that we need to recurse:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

name=a[1];
\end_layout

\begin_layout Plain Layout

if (a[3] == "=") {
\end_layout

\begin_layout Plain Layout

  if (substr(a[4],1,1) == "{") {
\end_layout

\begin_layout Plain Layout

    text = get_chunk_args(substr(a[4],2), values, path name SUBSEP);
\end_layout

\begin_layout Plain Layout

  } else {
\end_layout

\begin_layout Plain Layout

    values[path name]=a[5];
\end_layout

\begin_layout Plain Layout

    text = a[6];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  values[path name]="";
\end_layout

\begin_layout Plain Layout

  text = a[2];
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can test this function like this:
\end_layout

\begin_layout Chunk
gca-test.awk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{get_chunk_args()}>
\end_layout

\begin_layout Plain Layout

BEGIN {
\end_layout

\begin_layout Plain Layout

  SUBSEP=".";
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  print get_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}},
 etc", a);
\end_layout

\begin_layout Plain Layout

  for (b in a) {
\end_layout

\begin_layout Plain Layout

    print "a[" b "] => " a[b];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
which should give this output:
\end_layout

\begin_layout Chunk
gca-test.awk-results
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

a[foo.quux.quirk] => 
\end_layout

\begin_layout Plain Layout

a[foo.quux.a] => fleeg
\end_layout

\begin_layout Plain Layout

a[foo.bar] => baz
\end_layout

\begin_layout Plain Layout

a[etc] => 
\end_layout

\begin_layout Plain Layout

a[name] => freddie
\end_layout

\end_inset


\end_layout

\begin_layout Section
Parsing chunk arguments
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "cha:Chunk Arguments"

\end_inset


\end_layout

\begin_layout Standard
Arguments to paramterized chunks are expressed in round brackets as a comma
 separated list of optional arguments.
 For example, a chunk that is defined with:
\end_layout

\begin_layout LyX-Code

\backslash
Chunk{achunk, params=name ; address}
\end_layout

\begin_layout Standard
could be invoked as:
\end_layout

\begin_layout LyX-Code

\backslash
chunkref{achunk}(John Jones, jones@example.com)
\end_layout

\begin_layout Standard
An argument list may be as simple as in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref{pull}(thing, otherthing)
\end_layout

\end_inset

 or as complex as:
\end_layout

\begin_layout LyX-Code

\backslash
chunkref{pull}(things[x, y], get_other_things(a, "(all)")) 
\end_layout

\begin_layout Standard
--- which for all it's commas and quotes and parenthesis represents only
 two parameters: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
things[x, y]
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
get_other_things(a, "(all)")
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
If we simply split parameter list on commas, then the comma in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
things[x,y]
\end_layout

\end_inset

 would split into two seperate arguments: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
things[x
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
y]
\end_layout

\end_inset

--- neither of which make sense on their own.
\end_layout

\begin_layout Standard
One way to prevent this would be by refusing to split text between matching
 delimiters, such as 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
[
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
]
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
(
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
)
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}
\end_layout

\end_inset

 and most likely also 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
"
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
"
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
'
\end_layout

\end_inset

, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
'
\end_layout

\end_inset

.
 Of course this also makes it impossible to pass such mis-matched code fragments
 as parameters, but I think that it would be hard for readers to cope with
 authors who would pass such code unbalanced fragments as chunk parameters
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
I know that I couldn't cope with users doing such things, and although the
 GPL3 license prevents me from actually forbidding anyone from trying, if
 they want it to work they'll have to write the code themselves and not
 expect any support from me.
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Unfortunately, the full set of matching delimiters may vary from language
 to language.
 In certain C++ template contexts, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
>
\end_layout

\end_inset

 would count as delimiters, and yet in other contexts they would not.
\end_layout

\begin_layout Standard
This puts me in the unfortunate position of having to parse-somewhat all
 programming languages without knowing what they are!
\end_layout

\begin_layout Standard
However, if this universal mode-tracking is possible, then parsing the arguments
 would be trivial.
 Such a mode tracker is described in chapter 
\begin_inset CommandInset ref
LatexCommand ref
reference "cha:modes"

\end_inset

 and used here with simplicity.
\end_layout

\begin_layout Chunk
parse_chunk_args
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function parse_chunk_args(language, text, values, mode,
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  c, context, rest)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{new-mode-tracker}(context, language, mode)>
\end_layout

\begin_layout Plain Layout

  rest = mode_tracker(context, text, values);
\end_layout

\begin_layout Plain Layout

  # extract values
\end_layout

\begin_layout Plain Layout

  for(c=1; c <= context[0, "values"]; c++) {
\end_layout

\begin_layout Plain Layout

    values[c] = context[0, "values", c];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  return rest;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Section
Expanding parameters in the text
\end_layout

\begin_layout Standard
Within the body of the chunk, the parameters are referred to with: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${name}
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${address}
\end_layout

\end_inset

.
 There is a strong case that a LaTeX style notation should be used, like
 
\backslash
param{name} which would be expressed in the listing as =<
\backslash
param{name}> and be rendered as 
\begin_inset listings
inline true
status open

\begin_layout Plain Layout

=<
\backslash
param{name}>
\end_layout

\end_inset

.
 Such notation would make me go blind, but I do intend to adopt it
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Probably when LyX's listings inset can recognize and render it, so that
 I no longer have to type =<\SpecialChar \ldots{}
>
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
We therefore need a function 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
expand_chunk_args
\end_layout

\end_inset

 which will take a block of text, a list of permitted parameters, and the
 arguments which must substitute for the parameters.
 
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "Here-we-split"

\end_inset

Here we split the text on 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${
\end_layout

\end_inset

 which means that all parts except the first will begin with a parameter
 name which will be terminated by 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}
\end_layout

\end_inset

.
 The split function will consume the literal 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${
\end_layout

\end_inset

 in each case.
\end_layout

\begin_layout Chunk
expand_chunk_args()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function expand_chunk_args(text, params, args,  
\end_layout

\begin_layout Plain Layout

  p, text_array, next_text, v, t, l)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  if (split(text, text_array, "
\backslash

\backslash
${")) {
\end_layout

\begin_layout Plain Layout

    =<
\backslash
chunkref{substitute-chunk-args}>
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  return text;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
First, we produce an associative array of substitution values indexed by
 parameter names.
 This will serve as a cache, allowing us to look up the replacement values
 as we extract each name.
\end_layout

\begin_layout Chunk
substitute-chunk-args
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

for(p in params) {
\end_layout

\begin_layout Plain Layout

  v[params[p]]=args[p];
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We accumulate substituted text in the variable 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
text
\end_layout

\end_inset

.
 As the first part of the split function is the part before the delimiter
 --- which is 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${
\end_layout

\end_inset

 in our case --- this part will never contain a parameter reference, so
 we assign this directly to the result kept in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
$text
\end_layout

\end_inset

.
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

text=text_array[1];
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then iterate over the remaining values in the array
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
I don't know why I think that it will enumerate the array in order, but
 it seems to work
\end_layout

\end_inset


\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
So fix it or porve it
\end_layout

\end_inset

, and substitute each reference for it's argument.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

for(t=2; t in text_array; t++) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{substitute-chunk-arg}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
After the split on 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
${
\end_layout

\end_inset

 a valid parameter reference will consist of valid parameter name terminated
 by a close-brace 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}
\end_layout

\end_inset

.
 A valid character name begins with the underscore or a letter, and may
 contain letters, digits or underscores.
\end_layout

\begin_layout Standard
A valid looking reference that is not actually the name of a parameter will
 be and not substituted.
 This is good because there is nothing to substitute anyway, and it avoids
 clashes when writing code for languages where ${\SpecialChar \ldots{}
} is a valid construct
 --- such constructs will not be interfered with unless the parameter name
 also matches.
\end_layout

\begin_layout Chunk
substitute-chunk-arg
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
\end_layout

\begin_layout Plain Layout

    l[1] in v) 
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  text = text v[l[1]] substr(text_array[t], length(l[1])+2);
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  text = text "${" text_array[t];
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Language Modes
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "cha:modes"

\end_inset


\end_layout

\begin_layout Section
Modes
\end_layout

\begin_layout Standard
lstlistings and newfangle both recognize source languages, and perform some
 basic parsing.
 lstlistings can detect strings and comments within a language definition
 and perform suitable rendering, such as italics for comments, and visible-space
s within strings.
\end_layout

\begin_layout Standard
Newfangle similarly can recognize strings, and comments, etc, within a language,
 so that any chunks included with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

 can be suitably escape or quoted.
\end_layout

\begin_layout Subsection
Modes to keep code together
\end_layout

\begin_layout Standard
As an example, in the C language there are a few parse modes, affecting
 the interpretation of characters.
\end_layout

\begin_layout Standard
One parse mode is the strings mode.
 The string mode is commenced by an un-escaped quotation mark 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
"
\end_layout

\end_inset

 and terminated by the same.
 Within the string mode, only one additional mode can be commenced, it is
 the backslash mode 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash

\end_layout

\end_inset

, which is always terminated by the folloing character.
\end_layout

\begin_layout Standard
Other modes are 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
[
\end_layout

\end_inset

 which is terminated by a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
]
\end_layout

\end_inset

 (unless it occurs in a string).
\end_layout

\begin_layout Standard
Consider this line of C code:
\end_layout

\begin_layout Standard
\begin_inset Formula $\mathtt{things\underbrace{[\mathtt{x,\ y}]}_{1.\ [\ mode},\ get\_other\_things\underbrace{(\mathtt{a,\overbrace{"\mathtt{(all)}"}})}_{2.\ (\ mode}^{3.\ "\ mode}}$
\end_inset


\end_layout

\begin_layout Standard
Mode nesting prevents the close parenthesis in quote mode (part 3) from
 terminating the parenthesis mode (part 2).
\end_layout

\begin_layout Standard
Each language has a set of modes, the default mode being the null mode.
 Each mode can lead to other modes.
\end_layout

\begin_layout Subsection
Modes to included chunks
\end_layout

\begin_layout Standard
For instance, consider this chunk with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
language=perl
\end_layout

\end_inset

:
\end_layout

\begin_layout Chunk
example-perl,language=perl
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

s/"$/'/;
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If it were included in a chunk with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
language=sh
\end_layout

\end_inset

, like this:
\end_layout

\begin_layout Chunk
example-sh,language=sh
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

perl -pe "=<
\backslash
chunkref{example-perl}>"
\end_layout

\end_inset


\end_layout

\begin_layout Standard
newfangle would need to generate output like this if it were to work: 
\end_layout

\begin_layout LyX-Code
perl -pe "s/
\backslash
"
\backslash
$/'/;"
\end_layout

\begin_layout Standard
See that the double quote 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
"
\end_layout

\end_inset

, and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
$
\end_layout

\end_inset

 in the regex have been quoted with a back-slash to protect them from shell
 interpretation.
\end_layout

\begin_layout Standard
If that were then included in a chunk with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
language=make
\end_layout

\end_inset

, like this:
\end_layout

\begin_layout Chunk
example-makefile,language=make
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

target: pre-req
\end_layout

\begin_layout Plain Layout

                =<
\backslash
chunkref{example-sh}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We would need the output to look like this --- note the $$:
\end_layout

\begin_layout LyX-Code
target: pre-req
\end_layout

\begin_layout LyX-Code
        perl -pe "s/
\backslash
"
\backslash
$$/'/;"
\end_layout

\begin_layout Standard
In order to make this work, we need to define a mode-tracker supporting
 each language, that can detect the various quoting modes, and provide a
 transformation that must be applied to any included text so that included
 text will be interpreted correctly after any interpolation that it may
 be subject to at run-time.
\end_layout

\begin_layout Standard
For example, the sed transformation for text to be inserted into sh double-quote
d strings would be something like:
\end_layout

\begin_layout LyX-Code
s/
\backslash

\backslash
/
\backslash

\backslash

\backslash

\backslash
/g;s/$/
\backslash

\backslash
$/g;s/"/
\backslash

\backslash
"/g;
\end_layout

\begin_layout Standard
which protects 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
 $ "
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
I don't think this example is true
\end_layout

\end_inset

The mode tracker must also track nested mode-changes, as in this 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
sh
\end_layout

\end_inset

 example.
\end_layout

\begin_layout LyX-Code
echo "hello `id \SpecialChar \ldots{}
`"
\end_layout

\begin_layout LyX-Code
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
noindent
\backslash
hphantom{echo "hello `id}
\end_layout

\end_inset

↑
\end_layout

\begin_layout Standard
Any characters inserted at the point marked ↑ would need to be escaped,
 including 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
` | *
\end_layout

\end_inset

 among others.
 First it would need escaping for the back-ticks `, and then for the double-quot
es ".
\end_layout

\begin_layout Standard
Escaping need not occur if the format and mode of the included chunk matches
 that of the including chunk.
\end_layout

\begin_layout Standard
As each chunk is output a new mode tracker for that language is initialized
 in it's normal state.
 As text is output for that chunk the output mode is tracked.
 When a new chunk is included, a transformation appropriate to that mode
 is selected and pushed onto a stack of transformations.
 Any text to be output is first passed through this stack of transformations.
\end_layout

\begin_layout Standard
It remains to consider if the chunk-include function should return it's
 generated text so that the caller can apply any transformations (and formatting
), or if it should apply the stack of transformations itself.
\end_layout

\begin_layout Standard
Note that the transformed text should have the property of not being able
 to change the mode in the current chunk.
\end_layout

\begin_layout Standard
\begin_inset Note Note
status open

\begin_layout Plain Layout
Note chunk parameters should probably also be transformed
\end_layout

\end_inset


\end_layout

\begin_layout Section
Language Mode Definitions
\end_layout

\begin_layout Standard
All modes are stored in a single multi-dimensional hash.
 The first index is the language, and the second index is the mode-identifier.
 The third indexes are terminators, and optionally, submodes, and delimiters.
\end_layout

\begin_layout Standard
A useful set of mode definitions for a nameless general C-type language
 is shown here.
 (Don't be confused by the double backslash escaping needed in awk.
 One set of escaping is for the string, and the second set of escaping is
 for the regex).
 
\begin_inset Note Note
status open

\begin_layout Plain Layout
TODO: Add =<
\backslash
mode{}> command which will allow us to signify that a string is regex and
 thus newfangle will quote it for us.
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Submodes are entered by the characters 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash

\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
"
\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
'
\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
(
\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
[
\end_layout

\end_inset

 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
/*
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
common-mode-definitions,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, "",  "submodes" ]="
\backslash

\backslash

\backslash

\backslash
|
\backslash
"|'|{|
\backslash

\backslash
(|
\backslash

\backslash
[";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
In the default mode, a comma surrounded by un-important white space is a
 delimiter of language items.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, "",  "delimiters"]=" *, *";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
and should pass this test:
\end_layout

\begin_layout Standard
\begin_inset Note Note
status open

\begin_layout Plain Layout
Why do the tests run in 
\begin_inset Quotes eld
\end_inset

(
\begin_inset Quotes erd
\end_inset

 mode and not 
\begin_inset Quotes eld
\end_inset


\begin_inset Quotes erd
\end_inset

 mode
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
test:mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

parse_chunk_args("", "1,2,3", a, "");
\end_layout

\begin_layout Plain Layout

if (a[1] != "1") e++;
\end_layout

\begin_layout Plain Layout

if (a[2] != "2") e++;
\end_layout

\begin_layout Plain Layout

if (a[3] != "3") e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 3) e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

parse_chunk_args("", "joe, red", a, "");
\end_layout

\begin_layout Plain Layout

if (a[1] != "joe") e++;
\end_layout

\begin_layout Plain Layout

if (a[2] != "red") e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 2) e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

parse_chunk_args("", "${colour}", a, "");
\end_layout

\begin_layout Plain Layout

if (a[1] != "${colour}") e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 1) e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Nested modes are identified by a backslash, a double or single quote, various
 bracket styles or a /* comment.
\end_layout

\begin_layout Standard
For each of these sub-modes modes we must also identify at a mode terminator,
 and any sub-modes or delimiters that may be entered
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Because we are using the sub-mode characters as the mode identifier it means
 we can't currently have a mode character dependant on it's context; i.e.
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
{
\end_layout

\end_inset

 can't behave differently when it is inside 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
[
\end_layout

\end_inset

.
\end_layout

\end_inset

.
\end_layout

\begin_layout Subsection
Backslash
\end_layout

\begin_layout Standard
The backslash mode has no submodes or delimiters, and is terminated by any
 character.
 Note that we are not so much interested in evaluating or interpolating
 content as we are in delineating content.
 It is no matter that a double backslash (
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash

\backslash

\end_layout

\end_inset

) may represent a single backslash while a backslash-newline may represent
 white space, but it does matter that the newline in a backslash newline
 should not be able to terminate a C pre-processor statement; and so the
 newline will be consumed by the backslash however it is to be interpreted.
\end_layout

\begin_layout Chunk
common-mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, "
\backslash

\backslash
", "terminators"]=".";
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Strings
\end_layout

\begin_layout Standard
Common languages support two kinds of strings quoting, double quotes and
 single quotes.
\end_layout

\begin_layout Chunk
mode:common-string,params=language;quote
\end_layout

\begin_layout Standard
In a string we have one special mode, which is the backslash.
 This may escape an embedded quote and prevent us thinking that it should
 terminate the string.
 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, ${quote}, "submodes" ]="
\backslash

\backslash

\backslash

\backslash
";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Otherwise, the string will be terminated by the same character that commenced
 it.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, ${quote}, "terminators"]=${quote};
\end_layout

\end_inset


\end_layout

\begin_layout Standard
In C type languages, certain escape sequences exist in strings.
 We need to define mechanism to enclode any chunks included in this mode
 using those escape sequences.
 These are expressed in two parts, s meaning search, and r meaning replace.
\end_layout

\begin_layout Standard
The first substitution is to replace a backslash with a double backslash.
 We do this first as other substitutions may introduce a backslash which
 we would not then want to escape again here.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="
\backslash

\backslash
";
\end_layout

\begin_layout Plain Layout

escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="
\backslash

\backslash

\backslash

\backslash
";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If the quote character occurs in the text, it should be preceded by a backslash,
 otherwise it would terminate the string unexpectedly.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]=${quote};
\end_layout

\begin_layout Plain Layout

escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="
\backslash

\backslash
" ${quote};
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Any newlines in the string, must be replaced by 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
n
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="
\backslash
n";
\end_layout

\begin_layout Plain Layout

escapes[${language}, ${quote},   escapes[${language}, ${quote}], "r"]="
\backslash

\backslash
n";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
For the common modes, we define this string handling for double and single
 quotes.
\end_layout

\begin_layout Chunk
common-mode-definitions,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:common-string}(${language}, "
\backslash
textbackslash{}"")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:common-string}(${language}, "'")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Working strings should pass this test:
\end_layout

\begin_layout Chunk
test:mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

parse_chunk_args("", "say 
\backslash
"I said, 
\backslash

\backslash

\backslash
"Hello, how are you
\backslash

\backslash

\backslash
".
\backslash
", for me", a, "");
\end_layout

\begin_layout Plain Layout

if (a[1] != "say 
\backslash
"I said, 
\backslash

\backslash

\backslash
"Hello, how are you
\backslash

\backslash

\backslash
".
\backslash
"") e++;
\end_layout

\begin_layout Plain Layout

if (a[2] != "for me") e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 2) e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Parentheses, Braces and Brackets
\end_layout

\begin_layout Standard
Where quotes are closed by the same character, parentheses, brackets and
 braces are closed by an alternate character.
\end_layout

\begin_layout Chunk
mode:common-brackets,params=language;open;close
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, ${open},  "submodes" ]="
\backslash

\backslash

\backslash

\backslash
|
\backslash
"|{|
\backslash

\backslash
(|
\backslash

\backslash
[|'|/
\backslash

\backslash
*";
\end_layout

\begin_layout Plain Layout

modes[${language}, ${open},  "delimiters"]=" *, *";
\end_layout

\begin_layout Plain Layout

modes[${language}, ${open},  "terminators"]=${close};
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Note that the open is NOT a regex but the close token IS.
 
\begin_inset Note Note
status open

\begin_layout Plain Layout
WHen we can quote regex we won't have to put the slashes in here
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
common-mode-definitions,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:common-brackets}(${language}, "{", "}")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:common-brackets}(${language}, "[", "
\backslash
textbackslash{}
\backslash
textbackslash{}]")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:common-brackets}(${language}, "(", "
\backslash
textbackslash{}
\backslash
textbackslash{})")>
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Customizing Standard Modes
\end_layout

\begin_layout Chunk
mode:add-submode,params=language;mode;submode
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

modes[${language}, ${mode}, "submodes"] = modes[${language}, ${mode}, "submodes"
] "|" ${submode};
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
mode:add-escapes,params=language;mode;search;replace
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "s"]=${search};
\end_layout

\begin_layout Plain Layout

escapes[${language}, ${mode},   escapes[${language}, ${mode}], "r"]=${replace};
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Comments
\end_layout

\begin_layout Standard
We can define 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
/* comment */
\end_layout

\end_inset

 style comments and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
//comment
\end_layout

\end_inset

 style comments to be added to any language:
\end_layout

\begin_layout Chunk
mode:multi-line-comments,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-submode}(${language}, "", "/
\backslash
textbackslash{}
\backslash
textbackslash{}*")>
\end_layout

\begin_layout Plain Layout

modes[${language}, "/*", "terminators"]="
\backslash

\backslash
*/";
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
mode:single-line-slash-comments,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-submode}(${language}, "", "//")>
\end_layout

\begin_layout Plain Layout

modes[${language}, "//", "terminators"]="
\backslash
n";
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-escapes}(${language}, "//", "
\backslash
textbackslash{}n", "
\backslash
textbackslash{}n//")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can also define 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
# comment
\end_layout

\end_inset

 style comments (as used in awk and shell scripts) in a similar manner.
\end_layout

\begin_layout Chunk
mode:add-hash-comments,params=language
\begin_inset Note Note
status open

\begin_layout Plain Layout
I'm having to use 
\backslash
# for hash and 
\backslash
textbackslash{} for 
\backslash
 and have hacky work-arounds in the parser for now
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-submode}(${language}, "", "
\backslash
#")>
\end_layout

\begin_layout Plain Layout

modes[${language}, "#", "terminators"]="
\backslash
n";
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-escapes}(${language}, "
\backslash
#", "
\backslash
textbackslash{}n", "
\backslash
textbackslash{}n
\backslash
#")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
In C, the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
#
\end_layout

\end_inset

 denotes pre-processor directives which can be multi-line 
\end_layout

\begin_layout Chunk
mode:add-hash-defines,params=language
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-submode}(${language}, "", "
\backslash
#")>
\end_layout

\begin_layout Plain Layout

modes[${language}, "#", "submodes" ]="
\backslash

\backslash

\backslash

\backslash
";
\end_layout

\begin_layout Plain Layout

modes[${language}, "#", "terminators"]="
\backslash
n";
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-escapes}(${language}, "
\backslash
#", "
\backslash
textbackslash{}n", "
\backslash
textbackslash{}
\backslash
textbackslash{}
\backslash
textbackslash{}
\backslash
textbackslash{}
\backslash
textbackslash{}n")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We can add these definitions to various languages
\end_layout

\begin_layout Chunk
mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{common-mode-definitions}("")>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{common-mode-definitions}("c")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:multi-line-comments}("c")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:single-line-slash-comments}("c")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-hash-defines}("c")>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{common-mode-definitions}("awk")>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode:add-hash-comments}("awk")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The awk definitions should allow a comment block like this:
\end_layout

\begin_layout Chunk
test:comment-quote,language=awk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# Comment: =<
\backslash
chunkref{test:comment-text}>
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
test:comment-text,language=
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

Now is the time for
\end_layout

\begin_layout Plain Layout

the quick brown fox to bring lemonade
\end_layout

\begin_layout Plain Layout

to the party
\end_layout

\end_inset


\end_layout

\begin_layout Standard
to come out like this:
\end_layout

\begin_layout Chunk
test:comment-quote:result
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# Comment: Now is the time for
\end_layout

\begin_layout Plain Layout

#the quick brown fox to bring lemonade
\end_layout

\begin_layout Plain Layout

#to the party
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The C definition for such a block should have it come out like this:
\end_layout

\begin_layout Chunk
test:comment-quote:C-result
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# Comment: Now is the time for
\backslash

\end_layout

\begin_layout Plain Layout

the quick brown fox to bring lemonade
\backslash

\end_layout

\begin_layout Plain Layout

to the party
\end_layout

\end_inset


\end_layout

\begin_layout Section
Some tests
\end_layout

\begin_layout Standard
Also, the parser must return any spare text at the end that has not been
 processed due to a mode terminator being found.
\end_layout

\begin_layout Chunk
test:mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

rest = parse_chunk_args("", "1, 2, 3) spare", a, "(");
\end_layout

\begin_layout Plain Layout

if (a[1] != 1) e++;
\end_layout

\begin_layout Plain Layout

if (a[2] != 2) e++;
\end_layout

\begin_layout Plain Layout

if (a[3] != 3) e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 3) e++;
\end_layout

\begin_layout Plain Layout

if (rest != " spare") e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We must also be able to parse the example given earlier.
\end_layout

\begin_layout Chunk
test:mode-definitions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

parse_chunk_args("", "things[x, y], get_other_things(a, 
\backslash
"(all)
\backslash
"), 99", a, "(");
\end_layout

\begin_layout Plain Layout

if (a[1] != "things[x, y]") e++;
\end_layout

\begin_layout Plain Layout

if (a[2] != "get_other_things(a, 
\backslash
"(all)
\backslash
")") e++;
\end_layout

\begin_layout Plain Layout

if (a[3] != "99") e++;
\end_layout

\begin_layout Plain Layout

if (length(a) != 3) e++;
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{pca-test.awk:summary}>
\end_layout

\end_inset


\end_layout

\begin_layout Section
A non-recursive mode tracker
\end_layout

\begin_layout Subsection
Constructor
\end_layout

\begin_layout Standard
The mode tracker holds its state in a stack based on a hash.
 This function, when passed an empty hash will intialize it.
\end_layout

\begin_layout Chunk
new_mode_tracker()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function new_mode_tracker(context, language, mode) {
\end_layout

\begin_layout Plain Layout

  context[""] = 0;
\end_layout

\begin_layout Plain Layout

  context[0, "language"] = language;
\end_layout

\begin_layout Plain Layout

  context[0, "mode"] = mode;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Because awk functions cannot return an array, we must create the array first
 and pass it in, so we have a newfangle macro to do this:
\end_layout

\begin_layout Chunk
new-mode-tracker,params=context;language;mode
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{awk-delete-array}(${context})>
\end_layout

\begin_layout Plain Layout

new_mode_tracker(${context}, ${language}, ${mode});
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Management
\end_layout

\begin_layout Standard
And for tracking modes, we dispatch to a mode-tracker action based on the
 current language
\end_layout

\begin_layout Chunk
mode_tracker
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function push_mode_tracker(context, language, mode,
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  top)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  if (! ("" in context)) {
\end_layout

\begin_layout Plain Layout

    =<
\backslash
chunkref{new-mode-tracker}(context, language, mode)>
\end_layout

\begin_layout Plain Layout

  } else {
\end_layout

\begin_layout Plain Layout

    top = context[""];
\end_layout

\begin_layout Plain Layout

    if (context[top, "language"] == language && mode=="") mode = context[top,
 "mode"];
\end_layout

\begin_layout Plain Layout

    top++;
\end_layout

\begin_layout Plain Layout

    context[top, "language"] = language;
\end_layout

\begin_layout Plain Layout

    context[top, "mode"] = mode;
\end_layout

\begin_layout Plain Layout

    context[""] = top;
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function finalize_mode_tracker(context,
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  top)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  if ( ("" in context) && context[""] != 0) return 0;
\end_layout

\begin_layout Plain Layout

  return 1;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This implies that any chunk must be syntactically whole; for instance, this
 is fine:
\end_layout

\begin_layout Chunk
test:whole-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (1) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{test:say-hello}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
test:say-hello
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

print "hello";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
But this is not fine; the chunk 
\begin_inset Flex Chunkref
status collapsed

\begin_layout Plain Layout
test:hidden-else
\end_layout

\end_inset

 is not properly cromulent.
\end_layout

\begin_layout Chunk
test:partial-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (1) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{test:hidden-else}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
test:hidden-else
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  print "I'm fine";
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  print "I'm not";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
These tests will check for correct behaviour:
\end_layout

\begin_layout Chunk
test:cromulence
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

echo Cromulence test
\end_layout

\begin_layout Plain Layout

passtest ./newfangle -Rtest:whole-chunk $TEX_SRC &>/dev/null || ( echo "Whole
 chunk failed" && exit 1 )
\end_layout

\begin_layout Plain Layout

failtest ./newfangle -Rtest:partial-chunk $TEX_SRC &>/dev/null || ( echo
 "Partial chunk failed" && exit 1 )
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Tracker
\end_layout

\begin_layout Standard
We must avoid recursion as a language construct because we intend to employ
 mode-tracking to track language mode of emitted code, and the code is emitted
 from a function which is itself recursive, so instead we implement psuedo-recur
sion using our own stack based on a hash.
\end_layout

\begin_layout Chunk
mode_tracker()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function mode_tracker(context, text, values, 
\end_layout

\begin_layout Plain Layout

  # optional parameters
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  mode, submodes, language,
\end_layout

\begin_layout Plain Layout

  cindex, c, a, part, item, name, result, new_values, new_mode, 
\end_layout

\begin_layout Plain Layout

  delimiters, terminators)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We could be re-commencing with a valid context, so we need to setup the
 state according to the last context.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  cindex = context[""] + 0;
\end_layout

\begin_layout Plain Layout

  mode = context[cindex, "mode"];
\end_layout

\begin_layout Plain Layout

  language = context[cindex, "language" ];
\end_layout

\end_inset


\end_layout

\begin_layout Standard
First we construct a single large regex combining the possible sub-modes
 for the current mode along with the terminators for the current mode.
\end_layout

\begin_layout Chunk
parse_chunk_args-reset-modes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  submodes=modes[language, mode, "submodes"];
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  if ((language, mode, "delimiters") in modes) {
\end_layout

\begin_layout Plain Layout

    delimiters = modes[language, mode, "delimiters"];
\end_layout

\begin_layout Plain Layout

    if (length(submodes)>0) submodes = submodes "|";
\end_layout

\begin_layout Plain Layout

    submodes=submodes delimiters;
\end_layout

\begin_layout Plain Layout

  } else delimiters="";
\end_layout

\begin_layout Plain Layout

  if ((language, mode, "terminators") in modes) {
\end_layout

\begin_layout Plain Layout

    terminators = modes[language, mode, "terminators"];
\end_layout

\begin_layout Plain Layout

    if (length(submodes)>0) submodes = submodes "|";
\end_layout

\begin_layout Plain Layout

    submodes=submodes terminators;
\end_layout

\begin_layout Plain Layout

  } else terminators="";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If we don't find anything to match on --- probably because the language
 is not supported --- then we return the entire text without matching anything.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  if (! length(submodes)) return text; 
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
mode_tracker()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{parse_chunk_args-reset-modes}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then iterate the text (until there is none left) looking for sub-modes
 or terminators in the regex.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  while((cindex >= 0) && length(text)) {
\end_layout

\begin_layout Plain Layout

    if (match(text, "(" submodes ")", a)) {
\end_layout

\end_inset


\end_layout

\begin_layout Standard
A bug that creeps in regularly during development is bad regexes of zero
 length which result in an infinite loop (as no text is consumed), so I
 catch that right away with this test.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

      if (RLENGTH<1) {
\end_layout

\begin_layout Plain Layout

        error(sprintf("Internal error, matched zero length submode, should
 be impossible - likely regex computation error
\backslash
n" 
\backslash

\end_layout

\begin_layout Plain Layout

                "Language=%s
\backslash
nmode=%s
\backslash
nmatch=%s
\backslash
n", language, mode, submodes));
\end_layout

\begin_layout Plain Layout

      }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
part
\end_layout

\end_inset

 is defined as the text up to the sub-mode or terminator, and this is appended
 to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
item
\end_layout

\end_inset

 --- which is the current text being gathered.
 If a mode has a delimiter, then item is reset each time a delimiter is
 found.
\end_layout

\begin_layout Standard
\begin_inset Formula $\mathtt{\overbrace{"\overbrace{hello}^{item},\ \overbrace{there}^{item}"}^{item},\ \overbrace{he\ said.}^{item}}$
\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

      part = substr(text, 1, RSTART -1);
\end_layout

\begin_layout Plain Layout

      item = item part;
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We must now determine what was matched.
 If it was a terminator, then we must restore the previous mode.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

      if (match(a[1], "^" terminators "$")) {
\end_layout

\begin_layout Plain Layout

#printf("%2d EXIT  MODE [%s] by [%s] [%s]
\backslash
n", cindex, mode, a[1], text) > "/dev/stderr"
\end_layout

\begin_layout Plain Layout

        context[cindex, "values", ++context[cindex, "values"]] = item;
\end_layout

\begin_layout Plain Layout

        delete context[cindex];
\end_layout

\begin_layout Plain Layout

        context[""] = --cindex;
\end_layout

\begin_layout Plain Layout

        if (cindex>=0) {
\end_layout

\begin_layout Plain Layout

          mode = context[cindex, "mode"];
\end_layout

\begin_layout Plain Layout

          language = context[cindex, "language"];
\end_layout

\begin_layout Plain Layout

          =<
\backslash
chunkref{parse_chunk_args-reset-modes}>
\end_layout

\begin_layout Plain Layout

        }
\end_layout

\begin_layout Plain Layout

        item = item a[1];
\end_layout

\begin_layout Plain Layout

        text = substr(text, 1 + length(part) + length(a[1]));
\end_layout

\begin_layout Plain Layout

      }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If a delimiter was matched, then we must store the current item in the parsed
 values array, and reset the item.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

      else if (match(a[1], "^" delimiters "$")) {
\end_layout

\begin_layout Plain Layout

        if (cindex==0) {
\end_layout

\begin_layout Plain Layout

          context[cindex, "values", ++context[cindex, "values"]] = item;
\end_layout

\begin_layout Plain Layout

          item = "";
\end_layout

\begin_layout Plain Layout

        } else {
\end_layout

\begin_layout Plain Layout

          item = item a[1];
\end_layout

\begin_layout Plain Layout

        }
\end_layout

\begin_layout Plain Layout

        text = substr(text, 1 + length(part) + length(a[1]));
\end_layout

\begin_layout Plain Layout

      }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
otherwise, if a new submode is detected (all submodes have terminators),
 we must create a nested parse context until we find the terminator for
 this mode.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

 else if ((language, a[1], "terminators") in modes) {
\end_layout

\begin_layout Plain Layout

        #check if new_mode is defined
\end_layout

\begin_layout Plain Layout

        item = item a[1];
\end_layout

\begin_layout Plain Layout

#printf("%2d ENTER MODE [%s] in [%s]
\backslash
n", cindex, a[1], text) > "/dev/stderr"
\end_layout

\begin_layout Plain Layout

        text = substr(text, 1 + length(part) + length(a[1]));
\end_layout

\begin_layout Plain Layout

        context[""] = ++cindex;
\end_layout

\begin_layout Plain Layout

        context[cindex, "mode"] = a[1];
\end_layout

\begin_layout Plain Layout

        context[cindex, "language"] = language;
\end_layout

\begin_layout Plain Layout

        mode = a[1];
\end_layout

\begin_layout Plain Layout

        =<
\backslash
chunkref{parse_chunk_args-reset-modes}>
\end_layout

\begin_layout Plain Layout

      } else {
\end_layout

\begin_layout Plain Layout

        error(sprintf("Submode '%s' set unknown mode in text: %s
\backslash
nLanguage %s Mode %s
\backslash
n", a[1], text, language, mode));
\end_layout

\begin_layout Plain Layout

        text = substr(text, 1 + length(part) + length(a[1]));
\end_layout

\begin_layout Plain Layout

      }
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
In the final case, we parsed to the end of the string.
 If the string was entire, then we should have no nested mode context, but
 if the string was just a fragment we may have a mode context which must
 be preserved for the next fragment.
 Todo: Consideration ought to be given if sub-mode strings are split over
 two fragments.
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
Consideration ought to be given if sub-mode strings are split over two fragments.
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

else {
\end_layout

\begin_layout Plain Layout

      context[cindex, "values", ++context[cindex, "values"]] = item text;
\end_layout

\begin_layout Plain Layout

      text = "";
\end_layout

\begin_layout Plain Layout

      item = "";
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  context["item"] = item;
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  if (length(item)) context[cindex, "values", ++context[cindex, "values"]]
 = item;
\end_layout

\begin_layout Plain Layout

  return text;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Subsubsection
One happy chunk
\end_layout

\begin_layout Standard
All the mode tracker chunks are referred to here:
\end_layout

\begin_layout Chunk
mode-tracker
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{new_mode_tracker()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode_tracker()}>
\end_layout

\end_inset


\end_layout

\begin_layout Subsubsection
Tests
\end_layout

\begin_layout Standard
We can test this function like this:
\end_layout

\begin_layout Chunk
pca-test.awk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{error()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{mode-tracker}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{parse_chunk_args()}>
\end_layout

\begin_layout Plain Layout

BEGIN {
\end_layout

\begin_layout Plain Layout

  SUBSEP=".";
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{mode-definitions}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{test:mode-definitions}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
pca-test.awk:summary
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (e) {
\end_layout

\begin_layout Plain Layout

  printf "Failed " e
\end_layout

\begin_layout Plain Layout

  for (b in a) {
\end_layout

\begin_layout Plain Layout

    print "a[" b "] => " a[b];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  print "Passed"
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

split("", a);
\end_layout

\begin_layout Plain Layout

e=0;
\end_layout

\end_inset


\end_layout

\begin_layout Standard
which should give this output:
\end_layout

\begin_layout Chunk
pca-test.awk-results
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

a[foo.quux.quirk] => 
\end_layout

\begin_layout Plain Layout

a[foo.quux.a] => fleeg
\end_layout

\begin_layout Plain Layout

a[foo.bar] => baz
\end_layout

\begin_layout Plain Layout

a[etc] => 
\end_layout

\begin_layout Plain Layout

a[name] => freddie
\end_layout

\end_inset


\end_layout

\begin_layout Section
Escaping and Quoting
\end_layout

\begin_layout Standard
Each nested mode can optionally define a set of transforms to be applied
 to any text that is included from another language.
\end_layout

\begin_layout Standard
This code can perform transforms
\end_layout

\begin_layout Chunk
mode_tracker
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function transform_escape(s, r, text,
\end_layout

\begin_layout Plain Layout

    # optional
\end_layout

\begin_layout Plain Layout

    max, 
\end_layout

\begin_layout Plain Layout

        # local vars
\end_layout

\begin_layout Plain Layout

        c)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  for(c=1; c <= max && (c in s); c++) {
\end_layout

\begin_layout Plain Layout

    gsub(s[c], r[c], text);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  return text;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This function must append from index 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
c
\end_layout

\end_inset

 onwards, and escape transforms from the supplied context, and return c
 + number of new transforms.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function mode_escaper(context, s, r, src,
\end_layout

\begin_layout Plain Layout

  c, cp, cpl)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

        for(c = context[""]; c >= 0; c--) {
\end_layout

\begin_layout Plain Layout

                if ( (context[c, "language"], context[c, "mode"]) in escapes) {
\end_layout

\begin_layout Plain Layout

                        cpl = escapes[context[c, "language"], context[c, "mode"]];
\end_layout

\begin_layout Plain Layout

                        for (cp = 1; cp <= cpl; cp ++) {
\end_layout

\begin_layout Plain Layout

                                ++src;
\end_layout

\begin_layout Plain Layout

                                s[src] = escapes[context[c, "language"], context[c, "mode"], c, "s"];
\end_layout

\begin_layout Plain Layout

                                r[src] = escapes[context[c, "language"], context[c, "mode"], c, "r"];
\end_layout

\begin_layout Plain Layout

                        }
\end_layout

\begin_layout Plain Layout

                }
\end_layout

\begin_layout Plain Layout

        }
\end_layout

\begin_layout Plain Layout

        return src;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
test:escapes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

echo escapes test
\end_layout

\begin_layout Plain Layout

passtest ./newfangle -Rtest:comment-quote $TEX_SRC &>/dev/null || ( echo
 "Comment-quote failed" && exit 1 )
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Recognizing Chunks
\end_layout

\begin_layout Standard
Newfangle recognizes noweb chunks, but as we also want better LaTeX integration
 we will recognize any of these:
\end_layout

\begin_layout Itemize
notangle chunks matching the pattern 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
^<
\begin_inset space \hspace*{}
\length 0in
\end_inset

<.*?>
\begin_inset space \hspace*{}
\length 0in
\end_inset

>=
\end_layout

\end_inset


\end_layout

\begin_layout Itemize
a chunks beginning with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{lstlistings}
\end_layout

\end_inset

, possibly with 
\backslash
Chunk{\SpecialChar \ldots{}
} on the previous line
\end_layout

\begin_layout Itemize
an older form I have used, beginning with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{Chunk}[options]
\end_layout

\end_inset

 --- also more suitable for plain LaTeX users
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Is there such a thing as plain LaTeX?
\end_layout

\end_inset

.
\end_layout

\begin_layout Section
Chunk start
\end_layout

\begin_layout Standard
The variable 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunking
\end_layout

\end_inset

 is used to signify that we are processing a code chunk and not document.
 In such a state, input lines will be assigned to the current chunk; otherwise
 they are ignored.
\end_layout

\begin_layout Subsection
lstlistings
\end_layout

\begin_layout Standard
Our current scheme is to recognize the new lstlisting chunks, but these
 may be preceded by a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 command which in LyX is a more convenient way to pass the chunk name to
 the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{lstlistings}
\end_layout

\end_inset

 command, and a more visible way to specify other 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
lstset
\end_layout

\end_inset

 settings.
\end_layout

\begin_layout Standard
The arguments to the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 command are a name, and then a comma-seperated list of key-value pairs
 after the manner of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

.
 (In fact within the LaTeX 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 macro (section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sub:The-chunk-command"

\end_inset

) the text 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
name=
\end_layout

\end_inset

 is prefixed to the argument which is then literally passed to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

).
\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

/^
\backslash

\backslash
Chunk{/ {
\end_layout

\begin_layout Plain Layout

  if (match($0, "^
\backslash

\backslash

\backslash

\backslash
Chunk{ *([^ ,}]*),?(.*)}", line)) {
\end_layout

\begin_layout Plain Layout

    next_chunk_name = line[1];
\end_layout

\begin_layout Plain Layout

    get_chunk_args(line[2], next_chunk_args);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  next;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also make a basic attempt to parse the name out of the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstlistings[name=
\begin_inset space \hspace{}
\length 0in
\end_inset

chunk-name]
\end_layout

\end_inset

 text, otherwise we fall back to the name found in the previous chunk command.
 This attempt is very basic and doesn't support commas or spaces or square
 brackets as part of the chunkname.
 We also recognize 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{Chunk}
\end_layout

\end_inset

 which is convenient for some users
\begin_inset Foot
status open

\begin_layout Plain Layout
but not yet supported in the LaTeX macros
\end_layout

\end_inset


\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
Add noweave support
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

/^
\backslash

\backslash
begin{lstlisting}|^
\backslash

\backslash
begin{Chunk}/ {
\end_layout

\begin_layout Plain Layout

  if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
\end_layout

\begin_layout Plain Layout

    new_chunk(line[1]);
\end_layout

\begin_layout Plain Layout

  } else {
\end_layout

\begin_layout Plain Layout

    new_chunk(next_chunk_name, next_chunk_args);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  chunking=1;
\end_layout

\begin_layout Plain Layout

  next;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Noweb
\end_layout

\begin_layout Standard
We recognize notangle style chunks too:
\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

/^[<]<.*[>]>=/ {
\end_layout

\begin_layout Plain Layout

  if (match($0, "^[<]<(.*)[>]>= *$", line)) {
\end_layout

\begin_layout Plain Layout

    chunking=1;
\end_layout

\begin_layout Plain Layout

    notangle_mode=1;
\end_layout

\begin_layout Plain Layout

    new_chunk(line[1]);
\end_layout

\begin_layout Plain Layout

    next;
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Section
Chunk end
\end_layout

\begin_layout Standard
Likewise, we need to recognize when a chunk ends.
\end_layout

\begin_layout Subsection
lstlistings
\end_layout

\begin_layout Standard
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
e
\end_layout

\end_inset

 in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
[e]end{lislisting}
\end_layout

\end_inset

 is surrounded by square brackets so that when this document is processed,
 this chunk doesn't terminate early when the lstlistings package recognizes
 it's own end-string! 
\begin_inset Note Greyedout
status collapsed

\begin_layout Plain Layout
This doesn't make sense as the regex is anchored with ^, which this line
 does not begin with!
\end_layout

\end_inset


\begin_inset Note Note
status open

\begin_layout Plain Layout
No, it doesn't.
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

/^
\backslash

\backslash
[e]nd{lstlisting}|^
\backslash

\backslash
[e]nd{Chunk}/ {
\end_layout

\begin_layout Plain Layout

  chunking=0;
\end_layout

\begin_layout Plain Layout

  active_chunk="";
\end_layout

\begin_layout Plain Layout

  next;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
noweb
\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

/^@ *$/ {
\end_layout

\begin_layout Plain Layout

  chunking=0;
\end_layout

\begin_layout Plain Layout

  active_chunk="";
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
All other recognizers are only of effect if we are chunking; there's no
 point in looking at lines if they aren't part of a chunk, so we just ignore
 them as efficiently as we can.
\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

! chunking { next; }
\end_layout

\end_inset


\end_layout

\begin_layout Section
Chunk contents
\end_layout

\begin_layout Standard
Chunk contents are any lines read while 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunking
\end_layout

\end_inset

 is true.
 Some chunk contents are special in that they refer to other chunks, and
 will be replaced by the contents of these chunks when the file is generated.
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "sub:ORS-chunk-text"

\end_inset

We add the output record separator 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
ORS
\end_layout

\end_inset

 to the line now, because we will set 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
ORS
\end_layout

\end_inset

 to the empty string when we generate the output
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
So that we can print partial lines using 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
print
\end_layout

\end_inset

 instead of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
printf
\end_layout

\end_inset


\end_layout

\end_inset

.
\end_layout

\begin_layout Chunk
recognize-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

length(active_chunk) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{process-chunk-tabs}>
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{process-chunk}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If a chunk just consisted of plain text, we could handle the chunk like
 this:
\end_layout

\begin_layout Chunk
process-chunk-simple
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

chunk_line(active_chunk, $0 ORS);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
but in fact a chunk can include references to other chunks.
 Chunk includes are traditionally written as 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<chunk-name>>
\end_layout

\end_inset

, but we support other variations.
\end_layout

\begin_layout Standard
However, we also process tabs at this point, a tab at input can be replaced
 by a number of spaces defined by the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
tabs
\end_layout

\end_inset

 variable, set by the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
-T
\end_layout

\end_inset

 option.
 Of course this is poor tab behaviour, we should probably have the option
 to use proper counted tab-stops and process this on output.
\end_layout

\begin_layout Chunk
process-chunk-tabs
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (length(tabs)) {
\end_layout

\begin_layout Plain Layout

  gsub("
\backslash
t", tabs);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sub:lstlistings-includes"

\end_inset

lstlistings
\end_layout

\begin_layout Standard
If 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset{escapeinside={=<}{>}}
\end_layout

\end_inset

 is set, then we can use 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\backslash
chunkref{
\begin_inset space \hspace{}
\length 0in
\end_inset

chunk-name}>
\end_layout

\end_inset

 in listings.
 The sequence 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\end_layout

\end_inset

 was chosen because:
\end_layout

\begin_layout Enumerate
it is a better mnemonic than 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<chunk-name>>
\end_layout

\end_inset

 in that the = sign signifies equivalent or substitutability, 
\end_layout

\begin_layout Enumerate
and because =< is not valid in C or in any language I can think of 
\end_layout

\begin_layout Enumerate
and also because lstlistings doesn't like 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
>>
\end_layout

\end_inset

 as an end delimiter for the 
\emph on
texcl
\emph default
 escape, so we must make do with a single 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
>
\end_layout

\end_inset

, which is better matched by 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\end_layout

\end_inset

 than 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Unfortunately 
\begin_inset Note Note
status open

\begin_layout Plain Layout
fix this then
\end_layout

\end_inset

 the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<\SpecialChar \ldots{}
>
\end_layout

\end_inset

 that we use re-enters a LaTeX parsing mode in which some characters are
 special, e.g.
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
# 
\backslash

\end_layout

\end_inset

, and so these cause trouble if used in arguments to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

.
 At some point I must fix the LaTeX command 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

 so that it can accept these literally, but until then, when writing chunkref
 argumemts that need these characters, I must use the forms 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
textbackslash{}
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
#
\end_layout

\end_inset

; so I also define a hacky chunk 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
delatex
\end_layout

\end_inset

 whose purpose it is to remove these from any arguments parsed by newfangle,
 and used further on.
\end_layout

\begin_layout Chunk
delatex,params=text
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# FILTHY HACK
\end_layout

\begin_layout Plain Layout

gsub("
\backslash

\backslash

\backslash

\backslash
#", "#", ${text});
\end_layout

\begin_layout Plain Layout

gsub("
\backslash

\backslash

\backslash

\backslash
textbackslash{}", "
\backslash

\backslash
", ${text});
\end_layout

\end_inset


\end_layout

\begin_layout Standard
As each chunk line may contain more than one chunk include, we will split
 out chunk includes in an iterative fashion
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Contrary to our use of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
split
\end_layout

\end_inset

 when substituting parameters in chapter 
\begin_inset CommandInset ref
LatexCommand ref
reference "Here-we-split"

\end_inset


\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
First, as long as the chunk contains a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

 command we take as much as we can up to the first 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

 command.
\end_layout

\begin_layout Chunk
process-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

chunk = $0;
\end_layout

\begin_layout Plain Layout

indent = 0;
\end_layout

\begin_layout Plain Layout

while(match(chunk, 
\end_layout

\begin_layout Plain Layout

            "([=]<
\backslash

\backslash

\backslash

\backslash
chunkref{([^}>]*)}(
\backslash

\backslash
(.*
\backslash

\backslash
)|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)", 
\end_layout

\begin_layout Plain Layout

            line)
\backslash

\end_layout

\begin_layout Plain Layout

) {
\end_layout

\begin_layout Plain Layout

  chunklet = substr(chunk, 1, RSTART - 1);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We keep track of the indent count, by counting the number of literal characters
 found.
 We can then preserve this indent on each output line when multi-line chunks
 are expanded.
\end_layout

\begin_layout Standard
We then process this first part literal text, and set the chunk which is
 still to be processed to be the text after the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref
\end_layout

\end_inset

 command, which we will process next as we continue around the loop.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  indent += length(chunklet);
\end_layout

\begin_layout Plain Layout

  chunk_line(active_chunk, chunklet);
\end_layout

\begin_layout Plain Layout

  chunk = substr(chunk, RSTART + RLENGTH);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then consider the type of chunk command we have found, whether it is
 the newfangle style command beginning with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\end_layout

\end_inset

 or the older notangle style beginning with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<
\end_layout

\end_inset

.
 
\end_layout

\begin_layout Standard
Newfangle chunks may have parameters contained within square brackets.
 These will be matched in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
line[3]
\end_layout

\end_inset

 and are considered at this stage of processing to be part of the name of
 the chunk to be included.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  if (substr(line[1], 1, 1) == "=") {
\end_layout

\begin_layout Plain Layout

    # chunk name up to }
\end_layout

\begin_layout Plain Layout

        =<
\backslash
chunkref{delatex}(line[3])>
\end_layout

\begin_layout Plain Layout

    chunk_include(active_chunk, line[2] line[3], indent);
\end_layout

\begin_layout Plain Layout

  } else if (substr(line[1], 1, 1) == "<") {
\end_layout

\begin_layout Plain Layout

    chunk_include(active_chunk, line[4], indent);
\end_layout

\begin_layout Plain Layout

  } else {
\end_layout

\begin_layout Plain Layout

    error("Unknown chunk fragment: " line[1]);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The loop will continue until there are no more chunkref statements in the
 text, at which point we process the final part of the chunk.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

chunk_line(active_chunk, chunk);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "lone-newline"

\end_inset

We add the newline character as a chunklet on it's own, to make it easier
 to detect new lines and thus manage indentation when processing the output.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

chunk_line(active_chunk, "
\backslash
n");
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We will also permit a chunk-part number to follow in square brackets, so
 that 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\backslash
chunkref{chunk-name[1]}>
\end_layout

\end_inset

 will refer to the first part only.
 This can make it easy to include a C function prototype in a header file,
 if the first part of the chunk is just the function prototype without the
 trailing semi-colon.
 The header file would include the prototype with the trailing semi-colon,
 like this:
\end_layout

\begin_layout LyX-Code
=<
\backslash
chunkref{chunk-name[1]}>;
\end_layout

\begin_layout Standard
This is handled in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sub:Chunk-parts"

\end_inset


\end_layout

\begin_layout Standard
We should perhaps introduce a notion of language specific chunk options;
 so that perhaps we could specify:
\end_layout

\begin_layout LyX-Code
=<
\backslash
chunkref{chunk-name[function-declaration]}>;
\end_layout

\begin_layout Standard
which applies a transform 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
function-declaration
\end_layout

\end_inset

 to the chunk --- which in this case would extract a function prototype
 from a function.
\begin_inset Note Note
status open

\begin_layout Plain Layout
So do it
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Processing Options
\end_layout

\begin_layout Standard
At the start, first we set the default options.
\end_layout

\begin_layout Chunk
default-options
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

debug=0;
\end_layout

\begin_layout Plain Layout

linenos=0;
\end_layout

\begin_layout Plain Layout

notangle_mode=0;
\end_layout

\begin_layout Plain Layout

root="*";
\end_layout

\begin_layout Plain Layout

tabs = "";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Then we use getopt the standard way, and null out ARGV afterwards in the
 normal AWK fashion.
\end_layout

\begin_layout Chunk
read-options
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

Optind = 1    # skip ARGV[0]
\end_layout

\begin_layout Plain Layout

while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{handle-options}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

for (i=1; i<Optind; i++) { ARGV[i]=""; }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This is how we handle our options:
\end_layout

\begin_layout Chunk
handle-options
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (Optopt == "R") root = Optarg;
\end_layout

\begin_layout Plain Layout

else if (Optopt == "r") root="";
\end_layout

\begin_layout Plain Layout

else if (Optopt == "L") linenos = 1;
\end_layout

\begin_layout Plain Layout

else if (Optopt == "d") debug = 1;
\end_layout

\begin_layout Plain Layout

else if (Optopt == "T") tabs = indent_string(Optarg+0);
\end_layout

\begin_layout Plain Layout

else if (Optopt == "h") help();
\end_layout

\begin_layout Plain Layout

else if (Optopt == "?") help();
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We do all of this at the beginning of the program
\end_layout

\begin_layout Chunk
begin
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

BEGIN {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{constants}>
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{mode-definitions}>
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{default-options}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{read-options}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
And have a simple help function
\end_layout

\begin_layout Chunk
help()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function help() {
\end_layout

\begin_layout Plain Layout

  print "Usage:"
\end_layout

\begin_layout Plain Layout

  print "  newfangle [-L] -R<rootname> [source.tex ...]"
\end_layout

\begin_layout Plain Layout

  print "  newfangle -r [source.tex ...]"
\end_layout

\begin_layout Plain Layout

  print "  If the filename, source.tex is not specified then stdin is used"
\end_layout

\begin_layout Plain Layout

  print
\end_layout

\begin_layout Plain Layout

  print "-L causes the C statement: #line <lineno> 
\backslash
"filename
\backslash
"" to be issued"
\end_layout

\begin_layout Plain Layout

  print "-R causes the named root to be written to stdout"
\end_layout

\begin_layout Plain Layout

  print "-r lists all roots in the file (even those used elsewhere)"
\end_layout

\begin_layout Plain Layout

  exit 1;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Generating the output
\end_layout

\begin_layout Standard
We generate output by calling output_chunk, or listing the chunk names.
\end_layout

\begin_layout Chunk
generate-output
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (length(root)) output_chunk(root);
\end_layout

\begin_layout Plain Layout

else output_chunk_names();
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also have some other output debugging:
\end_layout

\begin_layout Chunk
debug-output
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (debug) {
\end_layout

\begin_layout Plain Layout

  print "------ chunk names "
\end_layout

\begin_layout Plain Layout

  output_chunk_names();
\end_layout

\begin_layout Plain Layout

  print "====== chunks"
\end_layout

\begin_layout Plain Layout

  output_chunks();
\end_layout

\begin_layout Plain Layout

  print "++++++ debug"
\end_layout

\begin_layout Plain Layout

  for (a in chunks) {
\end_layout

\begin_layout Plain Layout

    print a "=" chunks[a];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We do both of these at the end.
 We also set 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
ORS=""
\end_layout

\end_inset

 because each chunklet is not necessarily a complete line, and we already
 added 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
ORS
\end_layout

\end_inset

 to each input line in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sub:ORS-chunk-text"

\end_inset

.
\end_layout

\begin_layout Chunk
end
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

END {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{debug-output}>
\end_layout

\begin_layout Plain Layout

  ORS="";
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{generate-output}>
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We write chunk names like this.
 If we seem to be running in notangle compatibility mode, then we enclose
 the name like this 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
<<name>>
\end_layout

\end_inset

 the same way notangle does:
\end_layout

\begin_layout Chunk
output_chunk_names()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function output_chunk_names(   c, prefix, suffix) 
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  if (notangle_mode) {
\end_layout

\begin_layout Plain Layout

    prefix="<<";
\end_layout

\begin_layout Plain Layout

    suffix=">>";
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  for (c in chunk_names) {
\end_layout

\begin_layout Plain Layout

    print prefix c suffix "
\backslash
n";
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This function would write out all chunks
\end_layout

\begin_layout Chunk
output_chunks()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function output_chunks(  a) 
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  for (a in chunk_names) {
\end_layout

\begin_layout Plain Layout

    output_chunk(chunk_names[a]);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

function output_chunk(chunk) {
\end_layout

\begin_layout Plain Layout

  newline = 1;
\end_layout

\begin_layout Plain Layout

  lineno_needed = linenos;
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  write_chunk(chunk);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Section
Assembling the chunks
\end_layout

\begin_layout Standard
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunk_path
\end_layout

\end_inset

 holds a string consisting of the names of all the chunks that resulted
 in this chunk being output.
 
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
Make sure it includes the line numbers too...
 
\end_layout

\end_inset

It should probably also contain the source line numbers at which each inclusion
 also occured.
\end_layout

\begin_layout Chunk
write_chunk()
\end_layout

\begin_layout Standard
We first initialize the mode tracker for this chunk.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function write_chunk(chunk_name) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{awk-delete-array}(context)>
\end_layout

\begin_layout Plain Layout

  return write_chunk_r(chunk_name, context);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
write_chunk(),emph={chunk_path}
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function write_chunk_r(chunk_name, context, indent, tail,
\end_layout

\begin_layout Plain Layout

  # optional vars
\end_layout

\begin_layout Plain Layout

  chunk_path, chunk_args, 
\end_layout

\begin_layout Plain Layout

  s, r, src, new_src, 
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  chunk_params, part, max_part, part_line, frag, max_frag, text, 
\end_layout

\begin_layout Plain Layout

  chunklet, only_part, call_chunk_args)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sub:Chunk-parts"

\end_inset

Chunk parts
\end_layout

\begin_layout Standard
As mentioned in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sub:lstlistings-includes"

\end_inset

, a chunk name may contain a part specifier in square brackets, limiting
 the parts that should be emitted.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  if (match(chunk_name, "^(.*)
\backslash

\backslash
[([0-9]*)
\backslash

\backslash
]$", chunk_name_parts)) {
\end_layout

\begin_layout Plain Layout

    chunk_name = chunk_name_parts[1];
\end_layout

\begin_layout Plain Layout

    only_part = chunk_name_parts[2];
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then create a mode tracker 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  =<
\backslash
chunkref{new-mode-tracker}(context, chunks[chunk_name, "language"], "")>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We extract into 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunk_params
\end_layout

\end_inset

 the names of the parameters that this chunk accepts, whose values were
 (optionally) passed in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunk_args
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  split(chunks[chunk_name, "params"], chunk_params, " *; *");
\end_layout

\end_inset


\end_layout

\begin_layout Standard
To assemble a chunk, we write out each part.
\end_layout

\begin_layout Chunk
write_chunk()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  if (! (chunk_name in chunk_names)) {
\end_layout

\begin_layout Plain Layout

    error(sprintf(_"The root module <<%s>> was not defined.
\backslash
nUsed by: %s",
\backslash

\end_layout

\begin_layout Plain Layout

                  chunk_name, chunk_path));
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  max_part = chunks[chunk_name, "part"];
\end_layout

\begin_layout Plain Layout

  for(part = 1; part <= max_part; part++) {
\end_layout

\begin_layout Plain Layout

    if (! only_part || part == only_part) {
\end_layout

\begin_layout Plain Layout

      =<
\backslash
chunkref{write-part}>
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  if (! finalize_mode_tracker(context)) {
\end_layout

\begin_layout Plain Layout

    error(sprintf(_"Module %s did not close context properly.
\backslash
nUsed by: %s
\backslash
n", chunk_name, chunk_path));
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
A part can either be a chunklet of lines, or an include of another chunk.
\end_layout

\begin_layout Standard
Chunks may also have parameters, specified in LaTeX style with braces after
 the chunk name --- looking like this in the document: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunkname{param1, param2}
\end_layout

\end_inset

.
 Arguments are passed in square brackets: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkref{chunkname}[arg1, arg2]
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Before we process each part, we check that the source position hasn't changed
 unexpectedly, so that we can know if we need to output a new file-line
 directive.
\end_layout

\begin_layout Chunk
write-part
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{check-source-jump}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

chunklet = chunks[chunk_name, "part", part];
\end_layout

\begin_layout Plain Layout

if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{write-included-chunk}>
\end_layout

\begin_layout Plain Layout

} else if (chunklet SUBSEP "line" in chunks) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{write-chunklets}>
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  # empty last chunklet
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
To write an included chunk, we must detect any optional chunk arguments
 in parenthesis.
 Then we recurse calling 
\begin_inset Flex Chunkref
status collapsed

\begin_layout Plain Layout
write_chunk()
\end_layout

\end_inset

.
\end_layout

\begin_layout Chunk
write-included-chunk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (match(chunklet, "^([^
\backslash

\backslash
[
\backslash

\backslash
(]*)
\backslash

\backslash
((.*)
\backslash

\backslash
)$", chunklet_parts)) {
\end_layout

\begin_layout Plain Layout

  chunklet = chunklet_parts[1];
\end_layout

\begin_layout Plain Layout

  parse_chunk_args("", chunklet_parts[2], call_chunk_args, "(");
\end_layout

\begin_layout Plain Layout

  for (c in call_chunk_args) {
\end_layout

\begin_layout Plain Layout

    call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params,
 chunk_args);
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

} else {
\end_layout

\begin_layout Plain Layout

  split("", call_chunk_args);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

# update the transforms arrays
\end_layout

\begin_layout Plain Layout

new_src = mode_escaper(context, s, r, src);
\end_layout

\begin_layout Plain Layout

write_chunk_r(chunklet, context,
\end_layout

\begin_layout Plain Layout

            chunks[chunk_name, "part", part, "indent"] indent,
\end_layout

\begin_layout Plain Layout

            chunks[chunk_name, "part", part, "tail"],
\end_layout

\begin_layout Plain Layout

            chunk_path "
\backslash
n         " chunk_name,
\end_layout

\begin_layout Plain Layout

            call_chunk_args,
\end_layout

\begin_layout Plain Layout

                        s, r, new_src);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Before we output a chunklet of lines, we first emit the file and line number
 if we have one, and if it is safe to do so.
 
\end_layout

\begin_layout Standard
Chunklets are generally broken up by includes, so the start of a chunklet
 is a good place to do this.
 Then we output each line of the chunklet.
\end_layout

\begin_layout Standard
When it is not safe, such as in the middle of a multi-line macro definition,
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
lineno_suppressed
\end_layout

\end_inset

 is set to true, and in such a case we note that we want to emit the line
 statement when it is next safe.
\end_layout

\begin_layout Chunk
write-chunklets
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

max_frag = chunks[chunklet, "line"];
\end_layout

\begin_layout Plain Layout

for(frag = 1; frag <= max_frag; frag++) {
\end_layout

\begin_layout Plain Layout

  =<
\backslash
chunkref{write-file-line}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then extract the chunklet text and expand any arguments.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

  text = chunks[chunklet, frag];
\end_layout

\begin_layout Plain Layout

 
\end_layout

\begin_layout Plain Layout

  /* check params */
\end_layout

\begin_layout Plain Layout

  text = expand_chunk_args(text, chunk_params, chunk_args);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If the text is a single newline (which we keep separate - see 
\begin_inset CommandInset ref
LatexCommand ref
reference "lone-newline"

\end_inset

) then we increment the line number.
 In the case where this is the last line of a chunk and it is not a top-level
 chunk we replace the newline with an empty string --- because the chunk
 that included this chunk will have the newline at the end of the line that
 included this chunk.
\end_layout

\begin_layout Standard
We also note by 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newline = 1
\end_layout

\end_inset

 that we have started a new line, so that indentation can be managed with
 the following piece of text.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

 if (text == "
\backslash
n") {
\end_layout

\begin_layout Plain Layout

    lineno++;
\end_layout

\begin_layout Plain Layout

    if (part == max_part && frag == max_frag && length(chunk_path)) {
\end_layout

\begin_layout Plain Layout

      text = "";
\end_layout

\begin_layout Plain Layout

      break;
\end_layout

\begin_layout Plain Layout

    } else {
\end_layout

\begin_layout Plain Layout

      newline = 1;
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If this text does not represent a newline, but we see that we are the first
 piece of text on a newline, then we prefix our text with the current indent.
 NOTE: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newline
\end_layout

\end_inset

 is a global output-state variable, but the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
indent
\end_layout

\end_inset

 is not.
 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  } else if (length(text) || length(tail)) {
\end_layout

\begin_layout Plain Layout

    if (newline) text = indent text;
\end_layout

\begin_layout Plain Layout

    newline = 0;
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
Tail will soon no longer be relevant once mode-detection is in place.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  text = text tail;
\end_layout

\begin_layout Plain Layout

  mode_tracker(context, text);
\end_layout

\begin_layout Plain Layout

  print transform_escape(s, r, text, src);
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If a line ends in a backslash --- suggesting continuation --- then we supress
 outputting file-line as it would probably break the continued lines.
 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  if (linenos) {
\end_layout

\begin_layout Plain Layout

    lineno_suppressed = substr(lastline, length(lastline)) == "
\backslash

\backslash
";
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Of course there is no point in actually outputting the source filename and
 line number (file-line) if they don't say anything new! We only need to
 emit them if they aren't what is expected, or if we we not able to emit
 one when they had changed.
\end_layout

\begin_layout Chunk
write-file-line
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (newline && lineno_needed && ! lineno_suppressed) {
\end_layout

\begin_layout Plain Layout

  filename = a_filename;
\end_layout

\begin_layout Plain Layout

  lineno = a_lineno;
\end_layout

\begin_layout Plain Layout

  print "#line " lineno " 
\backslash
"" filename "
\backslash
"
\backslash
n"
\end_layout

\begin_layout Plain Layout

  lineno_needed = 0;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We check if a new file-line is needed by checking if the source line matches
 what we (or a compiler) would expect.
 
\end_layout

\begin_layout Chunk
check-source-jump
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in
 chunks)) {
\end_layout

\begin_layout Plain Layout

  a_filename = chunks[chunk_name, "part", part, "FILENAME"];
\end_layout

\begin_layout Plain Layout

  a_lineno = chunks[chunk_name, "part", part, "LINENO"];
\end_layout

\begin_layout Plain Layout

  if (a_filename != filename || a_lineno != lineno) {
\end_layout

\begin_layout Plain Layout

    lineno_needed++;
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Storing chunks
\end_layout

\begin_layout Standard
Awk has pretty limited data structures, so we will use two main hashes.
 Uninterrupted sequences of a chunk will be stored in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunklets
\end_layout

\end_inset

 and the chunklets used in a chunk will be stored in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
chunks
\end_layout

\end_inset

.
\end_layout

\begin_layout Chunk
constants
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

part_type_chunk=1;
\end_layout

\begin_layout Plain Layout

SUBSEP=",";
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
params
\end_layout

\end_inset

 mentioned are not chunk parameters for parameterized chunks, as mentioned
 in 
\begin_inset CommandInset ref
LatexCommand ref
reference "cha:Chunk Arguments"

\end_inset

, but the lstlistings style parameters used in the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 command
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
The 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
params
\end_layout

\end_inset

 parameter is used to hold the parameters for parameterized chunks
\end_layout

\end_inset

.
\end_layout

\begin_layout Chunk
chunk-storage-functions
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function new_chunk(chunk_name, params,
\end_layout

\begin_layout Plain Layout

  # local vars
\end_layout

\begin_layout Plain Layout

  p, append )
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  # HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
\end_layout

\begin_layout Plain Layout

  gsub("
\backslash

\backslash
(
\backslash

\backslash
)$", "", chunk_name);
\end_layout

\begin_layout Plain Layout

  if (! (chunk_name in chunk_names)) {
\end_layout

\begin_layout Plain Layout

    if (debug) print "New chunk " chunk_name;
\end_layout

\begin_layout Plain Layout

    chunk_names[chunk_name];
\end_layout

\begin_layout Plain Layout

    for (p in params) {
\end_layout

\begin_layout Plain Layout

      chunks[chunk_name, p] = params[p];
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

    if ("append" in params) {
\end_layout

\begin_layout Plain Layout

      append=params["append"];
\end_layout

\begin_layout Plain Layout

      if (! (append in chunk_names)) {
\end_layout

\begin_layout Plain Layout

        warning("Chunk " chunk_name " is appended to chunk " append " which
 is not defined yet");
\end_layout

\begin_layout Plain Layout

        new_chunk(append);
\end_layout

\begin_layout Plain Layout

      }
\end_layout

\begin_layout Plain Layout

      chunk_include(append, chunk_name);
\end_layout

\begin_layout Plain Layout

      chunk_line(append, ORS);
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

  }
\end_layout

\begin_layout Plain Layout

  active_chunk = chunk_name;
\end_layout

\begin_layout Plain Layout

  prime_chunk(chunk_name);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

function prime_chunk(chunk_name)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = 
\backslash

\end_layout

\begin_layout Plain Layout

         chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"]
;
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR
 + 1;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

function chunk_line(chunk_name, line){
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
\end_layout

\begin_layout Plain Layout

         ++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
 "line"]  ] = line;
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
Chunk include represents a 
\emph on
chunkref
\emph default
 statement, and stores the requirement to include another chunk.
 The parameter indent represents the quanity of literal text characters
 that preceded this 
\emph on
chunkref
\emph default
 statement and therefore by how much additional lines of the included chunk
 should be indented.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function chunk_include(chunk_name, chunk_ref, indent, tail)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_ch
unk;
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_str
ing(indent);
\end_layout

\begin_layout Plain Layout

  chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
\end_layout

\begin_layout Plain Layout

  prime_chunk(chunk_name);
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
The indent is calculated by indent_string, which may in future convert some
 spaces into tab characters.
 This function works by generating a printf padded format string, like 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
%22s
\end_layout

\end_inset

 for an indent of 22, and then printing an empty string using that format.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function indent_string(indent) {
\end_layout

\begin_layout Plain Layout

  return sprintf("%" indent "s", "");
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
\begin_inset CommandInset label
LatexCommand label
name "cha:getopt"

\end_inset

getopt
\end_layout

\begin_layout Standard
I use Arnold Robbins public domain getopt (1993 revision).
 This is probably the same one that is covered in chapter 12 of 
\begin_inset Quotes eld
\end_inset

Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk
\begin_inset Quotes erd
\end_inset

 but as that is licensed under the GNU Free Documentation License, Version
 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompany
ing explanations), so I do my best to explain how it works here.
\end_layout

\begin_layout Standard
The getopt.awk header is:
\end_layout

\begin_layout Chunk
getopt.awk-header,language=awk,morestring=[b]{/},morekeywords=else
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# getopt.awk --- do C library getopt(3) function in awk
\end_layout

\begin_layout Plain Layout

#
\end_layout

\begin_layout Plain Layout

# Arnold Robbins, arnold@skeeve.com, Public Domain
\end_layout

\begin_layout Plain Layout

#
\end_layout

\begin_layout Plain Layout

# Initial version: March, 1991
\end_layout

\begin_layout Plain Layout

# Revised: May, 1993
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
The provided explanation is:
\end_layout

\begin_layout Chunk
getopt.awk-notes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# External variables:
\end_layout

\begin_layout Plain Layout

#    Optind -- index in ARGV of first nonoption argument
\end_layout

\begin_layout Plain Layout

#    Optarg -- string value of argument to current option
\end_layout

\begin_layout Plain Layout

#    Opterr -- if nonzero, print our own diagnostic
\end_layout

\begin_layout Plain Layout

#    Optopt -- current option letter
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

# Returns:
\end_layout

\begin_layout Plain Layout

#    -1     at end of options
\end_layout

\begin_layout Plain Layout

#    ?      for unrecognized option
\end_layout

\begin_layout Plain Layout

#    <c>    a character representing the current option
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

# Private Data:
\end_layout

\begin_layout Plain Layout

#    _opti  -- index in multi-flag option, e.g., -abc
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
The function follows.
 The final two parameters, 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
thisopt
\end_layout

\end_inset

 and 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
i
\end_layout

\end_inset

 are local variables and not parameters --- as indicated by the multiple
 spaces preceding them.
 Awk doesn't care, the multiple spaces are a convention to help us humans.
\end_layout

\begin_layout Chunk
getopt.awk-getopt()
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

function getopt(argc, argv, options,    thisopt, i)
\end_layout

\begin_layout Plain Layout

{
\end_layout

\begin_layout Plain Layout

    if (length(options) == 0)    # no options given
\end_layout

\begin_layout Plain Layout

        return -1
\end_layout

\begin_layout Plain Layout

    if (argv[Optind] == "--") {  # all done
\end_layout

\begin_layout Plain Layout

        Optind++
\end_layout

\begin_layout Plain Layout

        _opti = 0
\end_layout

\begin_layout Plain Layout

        return -1
\end_layout

\begin_layout Plain Layout

    } else if (argv[Optind] !~ /^-[^: 
\backslash
t
\backslash
n
\backslash
f
\backslash
r
\backslash
v
\backslash
b]/) {
\end_layout

\begin_layout Plain Layout

        _opti = 0
\end_layout

\begin_layout Plain Layout

        return -1
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

    if (_opti == 0)
\end_layout

\begin_layout Plain Layout

        _opti = 2
\end_layout

\begin_layout Plain Layout

    thisopt = substr(argv[Optind], _opti, 1)
\end_layout

\begin_layout Plain Layout

    Optopt = thisopt
\end_layout

\begin_layout Plain Layout

    i = index(options, thisopt)
\end_layout

\begin_layout Plain Layout

    if (i == 0) {
\end_layout

\begin_layout Plain Layout

        if (Opterr)
\end_layout

\begin_layout Plain Layout

            printf("%c -- invalid option
\backslash
n",
\end_layout

\begin_layout Plain Layout

                                  thisopt) > "/dev/stderr"
\end_layout

\begin_layout Plain Layout

        if (_opti >= length(argv[Optind])) {
\end_layout

\begin_layout Plain Layout

            Optind++
\end_layout

\begin_layout Plain Layout

            _opti = 0
\end_layout

\begin_layout Plain Layout

        } else
\end_layout

\begin_layout Plain Layout

            _opti++
\end_layout

\begin_layout Plain Layout

        return "?"
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\end_inset


\end_layout

\begin_layout Standard
At this point, the option has been found and we need to know if it takes
 any arguments.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

    if (substr(options, i + 1, 1) == ":") {
\end_layout

\begin_layout Plain Layout

        # get option argument
\end_layout

\begin_layout Plain Layout

        if (length(substr(argv[Optind], _opti + 1)) > 0)
\end_layout

\begin_layout Plain Layout

            Optarg = substr(argv[Optind], _opti + 1)
\end_layout

\begin_layout Plain Layout

        else
\end_layout

\begin_layout Plain Layout

            Optarg = argv[++Optind]
\end_layout

\begin_layout Plain Layout

        _opti = 0
\end_layout

\begin_layout Plain Layout

    } else
\end_layout

\begin_layout Plain Layout

        Optarg = ""
\end_layout

\begin_layout Plain Layout

    if (_opti == 0 || _opti >= length(argv[Optind])) {
\end_layout

\begin_layout Plain Layout

        Optind++
\end_layout

\begin_layout Plain Layout

        _opti = 0
\end_layout

\begin_layout Plain Layout

    } else
\end_layout

\begin_layout Plain Layout

        _opti++
\end_layout

\begin_layout Plain Layout

    return thisopt
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset

A test program is built in, too
\end_layout

\begin_layout Chunk
getopt.awk-begin
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

BEGIN {
\end_layout

\begin_layout Plain Layout

    Opterr = 1    # default is to diagnose
\end_layout

\begin_layout Plain Layout

    Optind = 1    # skip ARGV[0]
\end_layout

\begin_layout Plain Layout

    # test program
\end_layout

\begin_layout Plain Layout

    if (_getopt_test) {
\end_layout

\begin_layout Plain Layout

        while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
\end_layout

\begin_layout Plain Layout

            printf("c = <%c>, optarg = <%s>
\backslash
n",
\end_layout

\begin_layout Plain Layout

                                       _go_c, Optarg)
\end_layout

\begin_layout Plain Layout

        printf("non-option arguments:
\backslash
n")
\end_layout

\begin_layout Plain Layout

        for (; Optind < ARGC; Optind++)
\end_layout

\begin_layout Plain Layout

            printf("
\backslash
tARGV[%d] = <%s>
\backslash
n",
\end_layout

\begin_layout Plain Layout

                                    Optind, ARGV[Optind])
\end_layout

\begin_layout Plain Layout

    }
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The entire getopt.awk is made out of these chunks in order
\end_layout

\begin_layout Chunk
getopt.awk
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-header}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-notes}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-getopt()}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-begin}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Although we only want the header and function:
\end_layout

\begin_layout Chunk
getopt
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

# try: locate getopt.awk for the full original file
\end_layout

\begin_layout Plain Layout

# as part of your standard awk installation
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-header}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{getopt.awk-getopt()}>
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Newfangle LaTeX source code
\end_layout

\begin_layout Section
newfangle module
\end_layout

\begin_layout Standard
Here we define a Lyx .module file that makes it convenient to use LyX for
 writing such literate programs.
\end_layout

\begin_layout Standard
This file 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
./newfangle.module
\end_layout

\end_inset

 can be installed in your personal 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
.lyx/layouts folder
\end_layout

\end_inset

.
 You will need to Tools Reconfigure so that LyX notices it.
 It adds a new format Chunk, which should precede every listing and contain
 the chunk name.
 
\end_layout

\begin_layout Chunk
./newfangle.module,language=
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

#
\backslash
DeclareLyXModule{Newfangle Literate Listings}
\end_layout

\begin_layout Plain Layout

#DescriptionBegin
\end_layout

\begin_layout Plain Layout

#  Newfangle literate listings allow one to write
\end_layout

\begin_layout Plain Layout

#   literate programs after the fashion of noweb, but without having
\end_layout

\begin_layout Plain Layout

#   to use noweave to generate the documentation.
 Instead the listings
\end_layout

\begin_layout Plain Layout

#   package is extended in conjunction with the noweb package to implement
\end_layout

\begin_layout Plain Layout

#   to code formating directly as latex.
\end_layout

\begin_layout Plain Layout

#  The newfangle awk script
\end_layout

\begin_layout Plain Layout

#DescriptionEnd
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

Format 11
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

AddToPreamble
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{./newfangle.sty}>
\end_layout

\begin_layout Plain Layout

EndPreamble
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{chunkstyle}>
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{chunkref}>
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
The Chunk style
\end_layout

\begin_layout Standard
The purpose of the 
\noun on
chunk
\noun default
 style is to make it easier for LyX users to provide the name to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstlistings
\end_layout

\end_inset

.
 Normally this requires right-clicking on the listing, choosing settings,
 advanced, and then typing 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
name=chunk-name
\end_layout

\end_inset

.
 This has the further disadvantage that the name (and other options) are
 not generally visible during document editing.
\end_layout

\begin_layout Standard
The chunk style is defined as a LaTeX command, so that all text on the same
 line is passed to the LaTeX command 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
Chunk
\end_layout

\end_inset

.
 This makes it easy to parse using 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
newfangle
\end_layout

\end_inset

, and easy to pass these options on to the listings package.
 The first word in a chunk section should be the chunk name, and will have
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
name=
\end_layout

\end_inset

 prepended to it.
 Any other words are accepted arguments to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
We set PassThru to 1 because the user is actually entering raw latex.
\end_layout

\begin_layout Chunk
chunkstyle
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

Style Chunk
\end_layout

\begin_layout Plain Layout

  LatexType             Command
\end_layout

\begin_layout Plain Layout

  LatexName             Chunk
\end_layout

\begin_layout Plain Layout

  Margin                First_Dynamic
\end_layout

\begin_layout Plain Layout

  LeftMargin            Chunk:xxx
\end_layout

\begin_layout Plain Layout

  LabelSep              xx
\end_layout

\begin_layout Plain Layout

  LabelType             Static
\end_layout

\begin_layout Plain Layout

  LabelString           "Chunk:"
\end_layout

\begin_layout Plain Layout

  Align                 Left
\end_layout

\begin_layout Plain Layout

  PassThru              1
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
To make the label very visible we choose a larger font coloured red.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  LabelFont
\end_layout

\begin_layout Plain Layout

    Family              Sans
\end_layout

\begin_layout Plain Layout

    Size                Large
\end_layout

\begin_layout Plain Layout

    Series              Bold
\end_layout

\begin_layout Plain Layout

    Shape               Italic
\end_layout

\begin_layout Plain Layout

    Color               red
\end_layout

\begin_layout Plain Layout

  EndFont
\end_layout

\begin_layout Plain Layout

End
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
The chunkref style
\end_layout

\begin_layout Standard
We also define the Chunkref style which can be used to express cross references
 to chunks.
\end_layout

\begin_layout Chunk
chunkref
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

InsetLayout Chunkref
\end_layout

\begin_layout Plain Layout

  LyxType               charstyle
\end_layout

\begin_layout Plain Layout

  LatexType             Command
\end_layout

\begin_layout Plain Layout

  LatexName             chunkref
\end_layout

\begin_layout Plain Layout

  PassThru              1
\end_layout

\begin_layout Plain Layout

  LabelFont             
\end_layout

\begin_layout Plain Layout

    Shape               Italic
\end_layout

\begin_layout Plain Layout

    Color               red
\end_layout

\begin_layout Plain Layout

  EndFont
\end_layout

\begin_layout Plain Layout

End
\end_layout

\end_inset


\end_layout

\begin_layout Section
\begin_inset CommandInset label
LatexCommand label
name "sec:Latex-Macros"

\end_inset

Latex Macros
\end_layout

\begin_layout Standard
We require the 
\noun on
listings
\noun default
, 
\noun on
noweb
\noun default
 and 
\noun on
xargs
\noun default
 packages.
 As noweb defines it's own 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
code
\end_layout

\end_inset

 environment, we re-define the one that LyX logical markup module expects
 here.
\end_layout

\begin_layout Chunk
./newfangle.sty,language=tex,basicstyle=
\backslash
ttfamily
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
usepackage{listings}%
\end_layout

\begin_layout Plain Layout


\backslash
usepackage{noweb}%
\end_layout

\begin_layout Plain Layout


\backslash
usepackage{xargs}%
\end_layout

\begin_layout Plain Layout


\backslash
renewcommand{
\backslash
code}[1]{
\backslash
texttt{#1}}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also define a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
CChunk
\end_layout

\end_inset

 macro, for use as: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{CChunk}
\end_layout

\end_inset

 which will need renaming to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
begin{Chunk}
\end_layout

\end_inset

 when I can do this without clashing with 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
lstnewenvironment{Chunk}{
\backslash
relax}{
\backslash
relax}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also define a suitable 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstset
\end_layout

\end_inset

 of parameters that suit the literate programming style after the fashion
 of 
\noun on
noweave
\noun default
.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
lstset{numbers=left, stepnumber=5, numbersep=5pt,
\end_layout

\begin_layout Plain Layout

        breaklines=false,basicstyle=
\backslash
ttfamily,
\end_layout

\begin_layout Plain Layout

        numberstyle=
\backslash
tiny, language=C}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also define a notangle-like mechanism for 
\emph on
escaping
\emph default
 to LaTeX from the listing, and by which we can refer to other listings.
 We declare the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<\SpecialChar \ldots{}
>
\end_layout

\end_inset

 sequence to contain LaTeX code, and include another like this chunk: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<
\backslash
chunkref{chunkname}>
\end_layout

\end_inset

.
 However, because 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
=<\SpecialChar \ldots{}
>
\end_layout

\end_inset

 is already defined to contain LaTeX code for this document --- this is
 a 
\noun on
newfangle
\noun default
 document after all --- the code fragment below effectively contains the
 LaTeX code: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
}{
\end_layout

\end_inset

.
 To avoid problems with document generation, I had to declare an lstlistings
 property: 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
escapeinside={}
\end_layout

\end_inset

 for this listing only; which in LyX was done by right-clicking the listings
 inset, choosing 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
settings
\end_layout

\end_inset

\SpecialChar \menuseparator

\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
advanced
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
=< isn't enjoyed literally here, in a listing when the escape sequence is
 already defined as shown...
 we need to somehow escape this representation...
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
lstparams "escapeinside={}"
inline false
status open

\begin_layout Plain Layout


\backslash
lstset{escapeinside={=<}{>}}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Although our macros will contain the @ symbol, they will be included in
 a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
makeatletter
\end_layout

\end_inset

 section by LyX; however we keep the commented out 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
makeatletter
\end_layout

\end_inset

 as a reminder.
 The listings package likes to centre the titles, but noweb titles are specially
 formatted and must be left aligned.
 The simplest way to do this turned out to be by removing the definition
 of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lst@maketitle
\end_layout

\end_inset

.
 This may interact badly if other listings want a regular title or caption.
 We remember the old maketitle in case we need it.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

%
\backslash
makeatletter
\end_layout

\begin_layout Plain Layout

%somehow re-defining maketitle gives us a left-aligned title
\end_layout

\begin_layout Plain Layout

%which is extactly what our specially formatted title needs!
\end_layout

\begin_layout Plain Layout


\backslash
global
\backslash
let
\backslash
newfangle@lst@maketitle
\backslash
lst@maketitle%
\end_layout

\begin_layout Plain Layout


\backslash
global
\backslash
def
\backslash
lst@maketitle{}%
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sub:The-chunk-command"

\end_inset

The chunk command
\end_layout

\begin_layout Standard
Our chunk command accepts one argument, and calls 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
ltset
\end_layout

\end_inset

.
 Although 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
ltset
\end_layout

\end_inset

 will note the name, this is erased when the next 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lstlisting
\end_layout

\end_inset

 starts, so we make a note of this in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lst@chunkname
\end_layout

\end_inset

 and restore in in lstlistings Init hook.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
def
\backslash
Chunk#1{%
\end_layout

\begin_layout Plain Layout

  
\backslash
lstset{title={
\backslash
newfanglecaption},name=#1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
global
\backslash
edef
\backslash
lst@chunkname{
\backslash
lst@intname}%
\end_layout

\begin_layout Plain Layout

}%
\end_layout

\begin_layout Plain Layout


\backslash
def
\backslash
lst@chunkname{
\backslash
empty}%
\end_layout

\end_inset


\end_layout

\begin_layout Subsubsection
Chunk parameters
\end_layout

\begin_layout Standard
Newfangle permits parameterized chunks, and requires the paramters to be
 specified as listings options.
 The newfangle script uses this, and although we don't do anything with
 these in the LaTeX code right now, we need to stop the listings package
 complaining.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
lst@Key{params}
\backslash
relax{
\backslash
def
\backslash
newfangle@chunk@params{#1}}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
As it is common to define a chunk which then needs appending to another
 chunk, and annoying to have to declare a single line chunk to manage the
 include, we support an 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
append=
\end_layout

\end_inset

 option.
 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
lst@Key{append}
\backslash
relax{
\backslash
def
\backslash
newfangle@chunk@append{#1}}%
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
The noweb styled caption
\end_layout

\begin_layout Standard
We define a public macro 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
newfanglecaption
\end_layout

\end_inset

 which can be set as a regular title.
 By means of 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
protect
\end_layout

\end_inset

, It expands to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
newfangle@caption
\end_layout

\end_inset

 at the appriate time when the caption is emitted.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
def
\backslash
newfanglecaption{
\backslash
protect
\backslash
newfangle@caption}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status collapsed

\begin_layout Plain Layout
\begin_inset Box Boxed
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
status open

\begin_layout Plain Layout
22c
\begin_inset space \qquad{}
\end_inset

⟨
\shape italic
some-chunk
\shape default
 19b⟩
\begin_inset Formula $\equiv+$
\end_inset


\begin_inset space \qquad{}
\end_inset


\begin_inset space \qquad{}
\end_inset


\begin_inset space \qquad{}
\end_inset


\begin_inset Formula $\triangleleft$
\end_inset

22b
\begin_inset space \quad{}
\end_inset

24d
\begin_inset Formula $\triangleright$
\end_inset


\end_layout

\begin_layout Plain Layout

\size footnotesize
In this example, the current chunk is 22c, and therefore the third chunk
 on page 22.
\end_layout

\begin_layout Plain Layout

\size footnotesize
It's name is 
\emph on
some-chunk
\emph default
.
 
\end_layout

\begin_layout Plain Layout

\size footnotesize
The first chunk with this name (19b) occurs as the second chunk on page
 19.
\end_layout

\begin_layout Plain Layout

\size footnotesize
The previous chunk (22d) with the same name is the second chunk on page
 22.
\end_layout

\begin_layout Plain Layout

\size footnotesize
The next chunk (24d) is the fourth chunk on page 24.
\end_layout

\begin_layout Plain Layout
\begin_inset Caption

\begin_layout Plain Layout
noweb heading
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\end_inset

The general noweb output format compactly identifies the current chunk,
 and references to the first chunk, and the previous and next chunks that
 have the same name.
 
\end_layout

\begin_layout Standard
This means that we need to keep a counter for each chunk-name, that we use
 to count chunks of the same name.
 
\end_layout

\begin_layout Subsection
The chunk counter
\end_layout

\begin_layout Standard
It would be natural to have a counter for each chunk name, but TeX would
 soon run out of counters
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
\SpecialChar \ldots{}
soon 
\emph on
did
\emph default
 run out of counters and so I had to re-write the LaTeX macros to share
 a counter as described here
\end_layout

\end_inset

, so we have one counter which we save at the end of a chunk and restore
 at the beginning of a chunk.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
newcounter{newfangle@chunkcounter}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We construct the name of this variable to store the counter to be the text
 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
lst-chunk-
\end_layout

\end_inset

 prefixed onto the chunks own name, and store it in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkcount
\end_layout

\end_inset

.
 
\end_layout

\begin_layout Standard
We save the counter like this:
\end_layout

\begin_layout Chunk
save-counter
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
global
\backslash
expandafter
\backslash
edef
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname{
\backslash
arabic{newfangle@chunkcounter}}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
and restore the counter like this:
\end_layout

\begin_layout Chunk
restore-counter
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
setcounter{newfangle@chunkcounter}{
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname}%
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
./newfangle.sty
\end_layout

\begin_layout Standard
If there does not already exist a variable whose name is stored in 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
chunkcount
\end_layout

\end_inset

, then we know we are the first chunk with this name, and then define a
 counter.
 
\end_layout

\begin_layout Standard
Although chunks of the same name share a common counter, they must still
 be distinguished.
 We use is the internal name of the listing, suffixed by the counter value.
 So the first chunk might be 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
something-1
\end_layout

\end_inset

 and the second chunk be 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
something-2
\end_layout

\end_inset

, etc.
\end_layout

\begin_layout Standard
We also calculate the name of the previous chunk if we can (before we increment
 the chunk counter).
 If this is the first chunk of that name, then 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
prevchunkname
\end_layout

\end_inset

 is set to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
relax
\end_layout

\end_inset

 which the noweb package will interpret as not existing.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
def
\backslash
newfangle@caption{%
\end_layout

\begin_layout Plain Layout

  
\backslash
edef
\backslash
chunkcount{lst-chunk-
\backslash
lst@intname}%
\end_layout

\begin_layout Plain Layout

  
\backslash
@ifundefined{
\backslash
chunkcount}{%
\end_layout

\begin_layout Plain Layout

    
\backslash
expandafter
\backslash
gdef
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname{0}%
\end_layout

\begin_layout Plain Layout

    
\backslash
setcounter{newfangle@chunkcounter}{
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname}%
\end_layout

\begin_layout Plain Layout

    
\backslash
let
\backslash
prevchunkname
\backslash
relax%
\end_layout

\begin_layout Plain Layout

  }{%
\end_layout

\begin_layout Plain Layout

    
\backslash
setcounter{newfangle@chunkcounter}{
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname}%
\end_layout

\begin_layout Plain Layout

    
\backslash
edef
\backslash
prevchunkname{
\backslash
lst@intname-
\backslash
arabic{newfangle@chunkcounter}}%
\end_layout

\begin_layout Plain Layout

  }%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
After incrementing the chunk counter, we then define the name of this chunk,
 as well as the name of the first chunk.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  
\backslash
addtocounter{newfangle@chunkcounter}{1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
global
\backslash
expandafter
\backslash
edef
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname{
\backslash
arabic{newfangle@chunkcounter}}%
\end_layout

\begin_layout Plain Layout

  
\backslash
edef
\backslash
chunkname{
\backslash
lst@intname-
\backslash
arabic{newfangle@chunkcounter}}%
\end_layout

\begin_layout Plain Layout

  
\backslash
edef
\backslash
firstchunkname{
\backslash
lst@intname-1}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We now need to calculate the name of the next chunk.
 We do this by temporarily skipping the counter on by one; however there
 may not actually be another chunk with this name! We detect this by also
 defining a label for each chunk based on the chunkname.
 If there is a next chunkname then it will define a label with that name.
 As labels are persistent, we can at least tell the second time LaTeX is
 run.
 If we don't find such a defined label then we define 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
nextchunkname
\end_layout

\end_inset

 to 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
relax
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  
\backslash
addtocounter{newfangle@chunkcounter}{1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
edef
\backslash
nextchunkname{
\backslash
lst@intname-
\backslash
arabic{newfangle@chunkcounter}}%
\end_layout

\begin_layout Plain Layout

  
\backslash
@ifundefined{r@label-
\backslash
nextchunkname}{
\backslash
let
\backslash
nextchunkname
\backslash
relax}{}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The noweb package requires that we define a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
sublabel
\end_layout

\end_inset

 for every chunk, with a unique name, which is then used to print out it's
 navigation hints.
\end_layout

\begin_layout Standard
We also define a regular label for this chunk, as was mentioned above when
 we calculated 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
nextchunkname
\end_layout

\end_inset

.
 This requires LaTeX to be run at least twice after new chunk sections are
 added --- but noweb requried that anyway.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  
\backslash
sublabel{
\backslash
chunkname}%
\end_layout

\begin_layout Plain Layout

% define this label for every chunk instance, so we
\end_layout

\begin_layout Plain Layout

% can tell when we are the last chunk of this name
\end_layout

\begin_layout Plain Layout

  
\backslash
label{label-
\backslash
chunkname}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We also try and add the chunk to the list of listings, but I'm afraid we
 don't do very well.
 We want each chunk name listing once, with all of it's references.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  
\backslash
addcontentsline{lol}{lstlisting}{
\backslash
lst@name~[
\backslash
protect
\backslash
subpageref{
\backslash
chunkname}]}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
We then call the noweb output macros in the same way that noweave generates
 them, except that we don't need to call 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
nwstartdeflinemarkup
\end_layout

\end_inset

 or 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
nwenddeflinemarkup
\end_layout

\end_inset

 -- and if we do it messes up the output somewhat.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  
\backslash
nwmargintag{%
\end_layout

\begin_layout Plain Layout

    {%
\end_layout

\begin_layout Plain Layout

      
\backslash
nwtagstyle{}%
\end_layout

\begin_layout Plain Layout

      
\backslash
subpageref{
\backslash
chunkname}%
\end_layout

\begin_layout Plain Layout

    }%
\end_layout

\begin_layout Plain Layout

  }%
\end_layout

\begin_layout Plain Layout

%
\end_layout

\begin_layout Plain Layout

  
\backslash
moddef{%
\end_layout

\begin_layout Plain Layout

    {
\backslash
lst@name}%
\end_layout

\begin_layout Plain Layout

    {%
\end_layout

\begin_layout Plain Layout

      
\backslash
nwtagstyle{}
\backslash
/%
\end_layout

\begin_layout Plain Layout

      
\backslash
@ifundefined{newfangle@chunk@params}{}{%
\end_layout

\begin_layout Plain Layout

        (
\backslash
newfangle@chunk@params)%
\end_layout

\begin_layout Plain Layout

      }%
\end_layout

\begin_layout Plain Layout

      [
\backslash
csname 
\backslash
chunkcount
\backslash
endcsname]~%
\end_layout

\begin_layout Plain Layout

      
\backslash
subpageref{
\backslash
firstchunkname}%
\end_layout

\begin_layout Plain Layout

    }%
\end_layout

\begin_layout Plain Layout

    
\backslash
@ifundefined{newfangle@chunk@append}{}{%
\end_layout

\begin_layout Plain Layout

    
\backslash
ifx{}
\backslash
newfangle@chunk@append{x}
\backslash
else%
\end_layout

\begin_layout Plain Layout

        ,~add~to~
\backslash
newfangle@chunk@append%
\end_layout

\begin_layout Plain Layout

    
\backslash
fi%
\end_layout

\begin_layout Plain Layout

    }%
\end_layout

\begin_layout Plain Layout


\backslash
global
\backslash
def
\backslash
newfangle@chunk@append{}%
\end_layout

\begin_layout Plain Layout


\backslash
lstset{append=x}%
\end_layout

\begin_layout Plain Layout

  }%
\end_layout

\begin_layout Plain Layout

%
\end_layout

\begin_layout Plain Layout

  
\backslash
ifx
\backslash
relax
\backslash
prevchunkname
\backslash
endmoddef
\backslash
else
\backslash
plusendmoddef
\backslash
fi%
\end_layout

\begin_layout Plain Layout

%  
\backslash
nwstartdeflinemarkup%
\end_layout

\begin_layout Plain Layout

  
\backslash
nwprevnextdefs{
\backslash
prevchunkname}{
\backslash
nextchunkname}%
\end_layout

\begin_layout Plain Layout

%  
\backslash
nwenddeflinemarkup%
\end_layout

\begin_layout Plain Layout

}%
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Originally this was developed as a 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout
listings
\end_layout

\end_inset

 aspect, in the Init hook, but it was found easier to affect the title without
 using a hook --- 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
lst@AddToHookExe{PreSet}
\end_layout

\end_inset

 is still required to set the listings name to the name passed to the 
\begin_inset Flex CharStyle:Code
status collapsed

\begin_layout Plain Layout

\backslash
Chunk
\end_layout

\end_inset

 command, though.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

%
\backslash
lst@BeginAspect{newfangle}
\end_layout

\begin_layout Plain Layout

%
\backslash
lst@Key{newfangle}{true}[t]{
\backslash
lstKV@SetIf{#1}{true}}
\end_layout

\begin_layout Plain Layout


\backslash
lst@AddToHookExe{PreSet}{
\backslash
global
\backslash
let
\backslash
lst@intname
\backslash
lst@chunkname}
\end_layout

\begin_layout Plain Layout


\backslash
lst@AddToHook{Init}{}%
\backslash
newfangle@caption}
\end_layout

\begin_layout Plain Layout

%
\backslash
lst@EndAspect
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
Cross references
\end_layout

\begin_layout Standard
We define the 
\backslash
chunkref command which makes it easy to generate visual references to different
 code chunks, e.g.
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features>
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Macro
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Appearance
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\backslash
chunkref{preamble}
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
chunkref{preamble}
\end_layout

\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\backslash
chunkref[3]{preamble}
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
chunkref[3]{preamble}
\end_layout

\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\backslash
chunkref{preamble}[arg1, arg2]
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
chunkref{preamble}[arg1, arg2]
\end_layout

\end_inset


\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Chunkref can also be used within a code chunk to include another code chunk.
 The third optional parameter to chunkref is a comma sepatarated list of
 arguments, which will replace defined parameters in the chunkref.
\begin_inset Note Note
status open

\begin_layout Plain Layout
Darn it, if I have: =<
\backslash
chunkref{new-mode-tracker}[{chunks[chunk_name, "language"]},{mode}]> the
 inner braces (inside [ ]) cause _ to signify subscript even though we have
 lst@ReplaceIn
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout


\backslash
def
\backslash
chunkref@args#1,{%
\end_layout

\begin_layout Plain Layout

  
\backslash
def
\backslash
arg{#1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
lst@ReplaceIn
\backslash
arg
\backslash
lst@filenamerpl%
\end_layout

\begin_layout Plain Layout

  
\backslash
arg%
\end_layout

\begin_layout Plain Layout

  
\backslash
@ifnextchar){
\backslash
relax}{, 
\backslash
chunkref@args}%
\end_layout

\begin_layout Plain Layout

}%
\end_layout

\begin_layout Plain Layout


\backslash
newcommand
\backslash
chunkref[2][0]{%
\end_layout

\begin_layout Plain Layout

  
\backslash
@ifnextchar({
\backslash
chunkref@i{#1}{#2}}{
\backslash
chunkref@i{#1}{#2}()}%
\end_layout

\begin_layout Plain Layout

}%
\end_layout

\begin_layout Plain Layout


\backslash
def
\backslash
chunkref@i#1#2(#3){%
\end_layout

\begin_layout Plain Layout

  
\backslash
def
\backslash
zero{0}%
\end_layout

\begin_layout Plain Layout

  
\backslash
def
\backslash
chunk{#2}%
\end_layout

\begin_layout Plain Layout

  
\backslash
def
\backslash
chunkno{#1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
def
\backslash
chunkargs{#3}%
\end_layout

\begin_layout Plain Layout

  
\backslash
ifx
\backslash
chunkno
\backslash
zero%
\end_layout

\begin_layout Plain Layout

    
\backslash
def
\backslash
chunkname{#2-1}%
\end_layout

\begin_layout Plain Layout

  
\backslash
else%
\end_layout

\begin_layout Plain Layout

    
\backslash
def
\backslash
chunkname{#2-
\backslash
chunkno}%
\end_layout

\begin_layout Plain Layout

  
\backslash
fi%
\end_layout

\begin_layout Plain Layout

  
\backslash
let
\backslash
lst@arg
\backslash
chunk%
\end_layout

\begin_layout Plain Layout

  
\backslash
lst@ReplaceIn
\backslash
chunk
\backslash
lst@filenamerpl%
\end_layout

\begin_layout Plain Layout

  
\backslash
LA{%
\backslash
moddef{%
\end_layout

\begin_layout Plain Layout

    {
\backslash
chunk}%
\end_layout

\begin_layout Plain Layout

    {%
\end_layout

\begin_layout Plain Layout

      
\backslash
nwtagstyle{}
\backslash
/%
\end_layout

\begin_layout Plain Layout

      
\backslash
ifx
\backslash
chunkno
\backslash
zero%
\end_layout

\begin_layout Plain Layout

      
\backslash
else%
\end_layout

\begin_layout Plain Layout

      [
\backslash
chunkno]%
\end_layout

\begin_layout Plain Layout

      
\backslash
fi%
\end_layout

\begin_layout Plain Layout

      
\backslash
ifx
\backslash
chunkargs
\backslash
empty%
\end_layout

\begin_layout Plain Layout

      
\backslash
else%
\end_layout

\begin_layout Plain Layout

        (
\backslash
chunkref@args #3,)%
\end_layout

\begin_layout Plain Layout

      
\backslash
fi%
\end_layout

\begin_layout Plain Layout

      ~
\backslash
subpageref{
\backslash
chunkname}%
\end_layout

\begin_layout Plain Layout

    }%
\end_layout

\begin_layout Plain Layout

  }%
\end_layout

\begin_layout Plain Layout

  
\backslash
RA%
\backslash
endmoddef%
\end_layout

\begin_layout Plain Layout

}%
\end_layout

\end_inset


\end_layout

\begin_layout Subsection
The end
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

%
\end_layout

\begin_layout Plain Layout

%
\backslash
makeatother
\end_layout

\end_inset


\end_layout

\begin_layout Chapter
Extracting newfangle
\end_layout

\begin_layout Section
Extracting from Lyx
\end_layout

\begin_layout Standard
To extract from LyX, you will need to configure LyX as explained in section
 
\begin_inset CommandInset ref
LatexCommand ref
reference "sub:Configuring-the-build"

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "lyx-build-script"

\end_inset

And this lyx-build scrap will extract newfangle for me.
\end_layout

\begin_layout Chunk
lyx-build,language=sh
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

#! /bin/sh
\end_layout

\begin_layout Plain Layout

set -x
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{lyx-build-helper}>
\end_layout

\begin_layout Plain Layout

cd $PROJECT_DIR || exit 1
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

/usr/local/bin/newfangle -R./newfangle $TEX_SRC > ./newfangle
\end_layout

\begin_layout Plain Layout

/usr/local/bin/newfangle -R./newfangle.module $TEX_SRC > ./newfangle.module
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{test:helpers}>
\end_layout

\begin_layout Plain Layout

# run tests
\end_layout

\begin_layout Plain Layout

./newfangle -Rpca-test.awk $TEX_SRC | awk -f - || exit 1
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{test:cromulence}>
\end_layout

\begin_layout Plain Layout

=<
\backslash
chunkref{test:escapes}>
\end_layout

\end_inset


\end_layout

\begin_layout Standard
With a lyx-build-helper
\end_layout

\begin_layout Chunk
lyx-build-helper,language=sh
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

PROJECT_DIR="$LYX_r"
\end_layout

\begin_layout Plain Layout

LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
\end_layout

\begin_layout Plain Layout

TEX_DIR="$LYX_p"
\end_layout

\begin_layout Plain Layout

TEX_SRC="$TEX_DIR/$LYX_i"
\end_layout

\end_inset


\end_layout

\begin_layout Section
Extracting from the command line
\end_layout

\begin_layout Standard
First you will need the tex output, then you can extract:
\end_layout

\begin_layout Chunk
lyx-build-manual,language=sh
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

lyx -e latex newfangle.lyx
\end_layout

\begin_layout Plain Layout

newfangle -R./newfangle newfangle.tex > ./newfangle
\end_layout

\begin_layout Plain Layout

newfangle -R./newfangle.module newfangle.tex > ./newfangle.module
\end_layout

\end_inset


\end_layout

\begin_layout Section
Testing
\end_layout

\begin_layout Chunk
test:helpers
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

passtest() {
\end_layout

\begin_layout Plain Layout

  if "$@"
\end_layout

\begin_layout Plain Layout

  then echo "Passed"
\end_layout

\begin_layout Plain Layout

  else echo "Failed"
\end_layout

\begin_layout Plain Layout

       return 1
\end_layout

\begin_layout Plain Layout

  fi
\end_layout

\begin_layout Plain Layout

}
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

failtest() {
\end_layout

\begin_layout Plain Layout

  if ! "$@"
\end_layout

\begin_layout Plain Layout

  then echo "Passed"
\end_layout

\begin_layout Plain Layout

  else echo "Failed"
\end_layout

\begin_layout Plain Layout

       return 1
\end_layout

\begin_layout Plain Layout

  fi
\end_layout

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Part
Tests
\end_layout

\begin_layout Chapter
Chunk Parameters
\end_layout

\begin_layout Chunk
tests-sub,language=,params=THING;colour
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

I see a ${THING} of 
\end_layout

\begin_layout Plain Layout

colour ${colour}, 
\end_layout

\begin_layout Plain Layout

looking closer =<
\backslash
chunkref{tests-sub-sub}(${colour})>
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
tests-sub-sub,params=colour
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

a funny shade of ${colour}
\end_layout

\end_inset


\end_layout

\begin_layout Chunk
tests
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

What do you see? "=<
\backslash
chunkref{tests-sub}(joe, red)>"
\end_layout

\begin_layout Plain Layout

Well, fancy!
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Should generate output:
\end_layout

\begin_layout Chunk
tests-result
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

What do you see? "I see a joe of 
\end_layout

\begin_layout Plain Layout

                  colour red, 
\end_layout

\begin_layout Plain Layout

                  looking closer a funny shade of red"
\end_layout

\begin_layout Plain Layout

Well, fancy!
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\end_body
\end_document