18 Fangle is a tool for fangled literate programming. Newfangled is defined as New and often needlessly novel by TheFreeDictionary.com.
19 In this case, fangled means yet another not-so-new1. but improved. ^1 method for literate programming.
20 Literate Programming has a long history starting with the great Donald Knuth himself, whose literate programming tools seem to make use of as many escape sequences for semantic markup as TeX (also by Donald Knuth).
21 Norman Ramsey wrote the Noweb set of tools (notangle, noweave and noroots) and helpfully reduced the amount of magic character sequences to pretty much just <<, >> and @, and in doing so brought the wonders of literate programming within my reach.
22 While using the L Y X editor for LaTeX editing I had various troubles with the noweb tools, some of which were my fault, some of which were noweb's fault and some of which were L Y X's fault.
23 Noweb generally brought literate programming to the masses through removing some of the complexity of the original literate programming, but this would be of no advantage to me if the L Y X / LaTeX combination brought more complications in their place.
24 Fangle was thus born (originally called Newfangle) as an awk replacement for notangle, adding some important features, like better integration with L Y X and LaTeX (and later TeXmacs), multiple output format conversions, and fixing notangle bugs like indentation when using -L for line numbers.
25 Significantly, fangle is just one program which replaces various programs in Noweb. Noweave is done away with and implemented directly as LaTeX macros, and noroots is implemented as a function of the untangler fangle.
26 Fangle is written in awk for portability reasons, awk being available for most platforms. A Python version2. hasn't anyone implemented awk in python yet? ^2 was considered for the benefit of L Y X but a scheme version for TeXmacs will probably materialise first; as TeXmacs macro capabilities help make edit-time and format-time rendering of fangle chunks simple enough for my weak brain.
27 As an extension to many literate-programming styles, Fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Name parameters (or even local variables in the callers scope) are anticipated, as parameterized chunks — useful though they are — are hard to comprehend in the literate document.
29 Fangle is licensed under the GPL 3 (or later).
30 This doesn't mean that sources generated by fangle must be licensed under the GPL 3.
31 This doesn't mean that you can't use or distribute fangle with sources of an incompatible license, but it means you must make the source of fangle available too.
32 As fangle is currently written in awk, an interpreted language, this should not be too hard.
34 4a <gpl3-copyright[1](
\v), lang=text> ≡
35 ________________________________________________________________________
36 1 | fangle - fully featured notangle replacement in awk
38 3 | Copyright (C) 2009-2010 Sam Liddicott <sam@liddicott.com>
40 5 | This program is free software: you can redistribute it and/or modify
41 6 | it under the terms of the GNU General Public License as published by
42 7 | the Free Software Foundation, either version 3 of the License, or
43 8 | (at your option) any later version.
45 10 | This program is distributed in the hope that it will be useful,
46 11 | but WITHOUT ANY WARRANTY; without even the implied warranty of
47 12 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
48 13 | GNU General Public License for more details.
50 15 | You should have received a copy of the GNU General Public License
51 16 | along with this program. If not, see <http://www.gnu.org/licenses/>.
52 |________________________________________________________________________
59 1 Introduction to Literate Programming 11
62 2.2 Extracting roots 13
63 2.3 Formatting the document 13
64 3 Using Fangle with L^ A T_ E X 15
65 4 Using Fangle with L Y X 17
66 4.1 Installing the L Y X module 17
67 4.2 Obtaining a decent mono font 17
71 4.3 Formatting your Lyx document 18
72 4.3.1 Customising the listing appearance 18
73 4.3.2 Global customisations 18
74 4.4 Configuring the build script 19
76 5 Using Fangle with T_ E X_( M A CS) 21
77 6 Fangle with Makefiles 23
78 6.1 A word about makefiles formats 23
79 6.2 Extracting Sources 23
80 6.2.1 Converting from L Y X to L^ A T_ E X 24
81 6.2.2 Converting from T_ E X_( M A CS) 24
82 6.3 Extracting Program Source 25
83 6.4 Extracting Source Files 25
84 6.5 Extracting Documentation 27
85 6.5.1 Formatting T_ E X 27
86 6.5.1.1 Running pdflatex 27
87 6.5.2 Formatting T_ E X_( M A CS) 28
88 6.5.3 Building the Documentation as a Whole 28
90 6.7 Boot-strapping the extraction 29
91 6.8 Incorporating Makefile.inc into existing projects 30
94 7 Fangle awk source code 33
96 7.2 Catching errors 34
97 8 T_ E X_( M A CS) args 35
98 9 L^ A T_ E X and lstlistings 37
99 9.1 Additional lstlstings parameters 37
100 9.2 Parsing chunk arguments 39
101 9.3 Expanding parameters in the text 40
102 10 Language Modes & Quoting 43
104 10.1.1 Modes to keep code together 43
105 10.1.2 Modes affect included chunks 43
106 10.2 Language Mode Definitions 44
109 10.2.3 Parentheses, Braces and Brackets 47
110 10.2.4 Customizing Standard Modes 47
116 10.4 A non-recursive mode tracker 50
117 10.4.1 Constructor 50
120 10.4.3.1 One happy chunk 54
122 10.5 Escaping and Quoting 54
123 11 Recognizing Chunks 57
125 11.1.1 T_ E X_( M A CS) 57
126 11.1.2 lstlistings 58
128 11.2.1 T_ E X_( M A CS) 59
131 11.3.1 lstlistings 60
133 11.4 Chunk contents 61
134 11.4.1 lstlistings 62
135 12 Processing Options 65
136 13 Generating the Output 67
137 13.1 Assembling the Chunks 68
138 13.1.1 Chunk Parts 68
141 16 Fangle LaTeX source code 79
142 16.1 fangle module 79
143 16.1.1 The Chunk style 79
144 16.1.2 The chunkref style 80
146 16.2.1 The chunk command 81
147 16.2.1.1 Chunk parameters 82
148 16.2.2 The noweb styled caption 82
149 16.2.3 The chunk counter 82
150 16.2.4 Cross references 85
152 17 Extracting fangle 87
153 17.1 Extracting from Lyx 87
154 17.2 Extracting documentation 87
155 17.3 Extracting from the command line 88
158 18 Chunk Parameters 91
159 19 Compile-log-lyx 91
161 Chapter 1Introduction to Literate Programming
162 Todo: Should really follow on from a part-0 explanation of what literate programming is.
163 Chapter 2Running Fangle
164 Fangle is a replacement for noweb, which consists of notangle, noroots and noweave.
165 Like notangle and noroots, fangle can read multiple named files, or from stdin.
167 The -r option causes fangle to behave like noroots.
168 fangle -r filename.tex
169 will print out the fangle roots of a tex file.
170 Unlike the noroots command, the printed roots are not enclosed in angle brackets e.g. <<name>>, unless at least one of the roots is defined using the notangle notation <<name>>=.
171 Also, unlike noroots, it prints out all roots --- not just those that are not used elsewhere. I find that a root not being used doesn't make it particularly top level — and so-called top level roots could also be included in another root as well.
172 My convention is that top level roots to be extracted begin with ./ and have the form of a filename.
173 Makefile.inc, discussed in 6, can automatically extract all such sources prefixed with ./
175 notangle's -R and -L options are supported.
176 If you are using L Y X or LaTeX, the standard way to extract a file would be:
177 fangle -R./Makefile.inc fangle.tex > ./Makefile.inc
178 If you are using TeXmacs, the standard way to extract a file would similarly be:
179 fangle -R./Makefile.inc fangle.txt > ./Makefile.inc
180 TeXmacs users would obtain the text file with a verbatim export from TeXmacs which can be done on the command line with texmacs -s -c fangle.tm fangle.txt -q
181 Unlike the noroots command, the -L option to generate C pre-preocessor #file style line-number directives,does not break indenting of the generated file..
182 Also, thanks to mode tracking (described in 10) the -L option does not interrupt (and break) multi-line C macros either.
183 This does mean that sometimes the compiler might calculate the source line wrongly when generating error messages in such cases, but there isn't any other way around if multi-line macros include other chunks.
184 Future releases will include a mapping file so that line/character references from the C compiler can be converted to the correct part of the source document.
185 2.3 Formatting the document
186 The noweave replacement built into the editing and formatting environment for TeXmacs, L Y X (which uses LaTeX), and even for raw LaTeX.
187 Use of fangle with TeXmacs, L Y X and LaTeX are explained the the next few chapters.
188 Chapter 3Using Fangle with LaTeX
189 Because the noweave replacement is impemented in LaTeX, there is no processing stage required before running the LaTeX command. Of course, LaTeX may need running two or more times, so that the code chunk references can be fully calculated.
190 The formatting is managed by a set of macros shown in 16, and can be included with:
191 \usepackage{fangle.sty}
192 Norman Ramsay's origial noweb.sty package is currently required as it is used for formatting the code chunk captions.
193 The listings.sty package is required, and is used for formatting the code chunks and syntax highlighting.
194 The xargs.sty package is also required, and makes writing LaTeX macro so much more pleasant.
195 To do: Add examples of use of Macros
197 Chapter 4Using Fangle with L Y X
198 L Y X uses the same LaTeX macros shown in 16 as part of a L Y X module file fangle.module, which automatically includes the macros in the document pre-amble provided that the fangle L Y X module is used in the document.
199 4.1 Installing the L Y X module
200 Copy fangle.module to your L Y X layouts directory, which for unix users will be ~/.lyx/layouts
201 In order to make the new literate styles availalble, you will need to reconfigure L Y X by clicking Tools->Reconfigure, and then re-start L Y X.
202 4.2 Obtaining a decent mono font
203 The syntax high-lighting features of lstlistings makes use of bold; however a mono-space tt font is used to typeset the listings. Obtaining a bold tt font can be impossibly difficult and amazingly easy. I spent many hours at it, following complicated instructions from those who had spend many hours over it, and was finally delivered the simple solution on the lyx mailing list.
205 The simple way was to add this to my preamble:
207 \renewcommand{\ttdefault}{txtt}
210 The next simplest way was to use ams poor-mans-bold, by adding this to the pre-amble:
212 %\renewcommand{\ttdefault}{txtt}
213 %somehow make \pmb be the command for bold, forgot how, sorry, above line not work
214 It works, but looks wretched on the dvi viewer.
216 The lstlistings documention suggests using Luximono.
217 Luximono was installed according to the instructions in Ubuntu Forums thread 11591811. http://ubuntuforums.org/showthread.php?t=1159181 ^1 with tips from miknight2. http://miknight.blogspot.com/2005/11/how-to-install-luxi-mono-font-in.html ^2 stating that sudo updmap --enable MixedMap ul9.map is required. It looks fine in PDF and PS view but still looks rotten in dvi view.
218 4.3 Formatting your Lyx document
219 It is not necessary to base your literate document on any of the original L Y X literate classes; so select a regular class for your document type.
220 Add the new module Fangle Literate Listings and also Logical Markup which is very useful.
221 In the drop-down style listbox you should notice a new style defined, called Chunk.
222 When you wish to insert a literate chunk, you enter it's plain name in the Chunk style, instead of the old noweb method that uses <<name>>= type tags. In the line (or paragraph) following the chunk name, you insert a listing with: Insert->Program Listing.
223 Inside the white listing box you can type (or paste using shift+ctrl+V) your listing. There is no need to use ctrl+enter at the end of lines as with some older L Y X literate techniques --- just press enter as normal.
224 4.3.1 Customising the listing appearance
225 The code is formatted using the lstlistings package. The chunk style doesn't just define the chunk name, but can also define any other chunk options supported by the lstlistings package \lstset command. In fact, what you type in the chunk style is raw latex. If you want to set the chunk language without having to right-click the listing, just add ,lanuage=C after the chunk name. (Currently the language will affect all subsequent listings, so you may need to specify ,language= quite a lot).
226 To do: so fix the bug
228 Of course you can do this by editing the listings box advanced properties by right-clicking on the listings box, but that takes longer, and you can't see at-a-glance what the advanced settings are while editing the document; also advanced settings apply only to that box --- the chunk settings apply through the rest of the document3. It ought to apply only to subsequent chunks of the same name. I'll fix that later ^3.
229 To do: So make sure they only apply to chunks of that name
231 4.3.2 Global customisations
232 As lstlistings is used to set the code chunks, it's \lstset command can be used in the pre-amble to set some document wide settings.
233 If your source has many words with long sequences of capital letters, then columns=fullflexible may be a good idea, or the capital letters will get crowded. (I think lstlistings ought to use a slightly smaller font for captial letters so that they still fit).
234 The font family \ttfamily looks more normal for code, but has no bold (an alternate typewriter font is used).
235 With \ttfamily, I must also specify columns=fullflexible or the wrong letter spacing is used.
236 In my LaTeX pre-amble I usually specialise my code format with:
238 19a <document-preamble[1](
\v), lang=tex> ≡
239 ________________________________________________________________________
241 2 | numbers=left, stepnumber=1, numbersep=5pt,
242 3 | breaklines=false,
243 4 | basicstyle=\footnotesize\ttfamily,
244 5 | numberstyle=\tiny,
246 7 | columns=fullflexible,
247 8 | numberfirstline=true
249 |________________________________________________________________________
253 4.4 Configuring the build script
254 You can invoke code extraction and building from the L Y X menu option Document->Build Program.
255 First, make sure you don't have a conversion defined for Lyx->Program
256 From the menu Tools->Preferences, add a conversion from Latex(Plain)->Program as:
257 set -x ; fangle -Rlyx-build $$i |
258 env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
259 (But don't cut-n-paste it from this document or you may be be pasting a multi-line string which will break your lyx preferences file).
260 I hope that one day, L Y X will set these into the environment when calling the build script.
261 You may also want to consider adding options to this conversion...
262 parselog=/usr/share/lyx/scripts/listerrors
263 ...but if you do you will lose your stderr4. There is some bash plumbing to get a copy of stderr but this footnote is too small ^4.
264 Now, a shell script chunk called lyx-build will be extracted and run whenever you choose the Document->Build Program menu item.
265 This document was originally managed using L Y X and lyx-build script for this document is shown here for historical reference.
266 lyx -e latex fangle.lyx && \
267 fangle fangle.lyx > ./autoboot
268 This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
270 When the lyx-build chunk is executed, the current directory will be a temporary directory, and LYX_SOURCE will refer to the tex file in this temporary directory. This is unfortunate as our makefile wants to run from the project directory where the Lyx file is kept.
271 We can extract the project directory from $$r, and derive the probable Lyx filename from the noweb file that Lyx generated.
273 19b <lyx-build-helper[1](
\v), lang=sh> ≡ 87c⊳
274 ________________________________________________________________________
275 1 | PROJECT_DIR="$LYX_r"
276 2 | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
278 4 | TEX_SRC="$TEX_DIR/$LYX_i"
279 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
280 And then we can define a lyx-build fragment similar to the autoboot fragment
282 20a <lyx-build[1](
\v), lang=sh> ≡ 87a⊳
283 ________________________________________________________________________
285 2 | «lyx-build-helper 19b»
286 3 | cd $PROJECT_DIR || exit 1
288 5 | #/usr/bin/fangle -filter ./notanglefix-filter \
289 6 | # -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx" \
290 7 | # | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/' \
291 8 | # > ./Makefile.inc
293 10 | #make -f ./Makefile.inc fangle_sources
294 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
296 Chapter 5Using Fangle with TeXmacs
297 To do: Write this chapter
299 Chapter 6Fangle with Makefiles
300 Here we describe a Makefile.inc that you can include in your own Makefiles, or glue as a recursive make to other projects.
301 Makefile.inc will cope with extracting all the other source files from this or any specified literate document and keeping them up to date.
302 It may also be included by a Makefile or Makefile.am defined in a literate document to automatically deal with the extraction of source files and documents during normal builds.
303 Thus, if Makefile.inc is included into a main project makefile it add rules for the source files, capable of extracting the source files from the literate document.
304 6.1 A word about makefiles formats
305 Whitespace formatting is very important in a Makefile. The first character of each action line must be a TAB.
306 target: pre-requisite
309 This requires that the literate programming environment have the ability to represent a TAB character in a way that fangle will generate an actual TAB character.
310 We also adopt a convention that code chunks whose names beginning with ./ should always be automatically extracted from the document. Code chunks whose names do not begin with ./ are for internal reference. Such chunks may be extracted directly, but will not be automatically extracted by this Makefile.
311 6.2 Extracting Sources
312 Our makefile has two parts; variables must be defined before the targets that use them.
313 As we progress through this chapter, explaining concepts, we will be adding lines to <Makefile.inc-vars 23b> and <Makefile.inc-targets 24c> which are included in <./Makefile.inc 23a> below.
315 23a <./Makefile.inc[1](
\v), lang=make> ≡
316 ________________________________________________________________________
317 1 | «Makefile.inc-vars 23b»
318 2 | «Makefile.inc-default-targets ?a»
319 3 | «Makefile.inc-targets 24c»
320 |________________________________________________________________________
323 We first define a placeholder for the tool fangle in case it cannot be found in the path.
325 23b <Makefile.inc-vars[1](
\v), lang=> ≡ 24a⊳
326 ________________________________________________________________________
328 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
329 We also define a placeholder for LITERATE_SOURCE to hold the name of this document. This will normally be passed on the command line.
331 24a <Makefile.inc-vars[2](
\v) ⇑23b, lang=> +≡ ⊲23b 24b▿
332 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
334 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
335 Fangle cannot process L Y X or TeXmacs documents directly, so the first stage is to convert these to more suitable text based formats1. L Y X and TeXmacs formats are text-based, but not suitable for fangle ^1.
336 6.2.1 Converting from L Y X to LaTeX
337 The first stage will always be to convert the L Y X file to a LaTeX file. Fangle must run on a TeX file because the L Y X command server-goto-file-line2. The Lyx command server-goto-file-line is used to position the Lyx cursor at the compiler errors. ^2 requries that the line number provided be a line of the TeX file and always maps this the line in the L Y X docment. We use server-goto-file-line when moving the cursor to error lines during compile failures.
338 The command lyx -e literate fangle.lyx will produce fangle.tex, a TeX file; so we define a make target to be the same as the L Y X file but with the .tex extension.
339 The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.
341 24b <Makefile.inc-vars[3](
\v) ⇑23b, lang=> +≡ ▵24a 25a⊳
342 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
343 3 | LYX_SOURCE=$(LITERATE_SOURCE) # but only the .lyx files
344 4 | TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
345 5 | EXTRA_DIST+=$(TEX_SOURCE)
346 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
347 We then specify that the TeX source is to be generated from the L Y X source.
349 24c <Makefile.inc-targets[1](
\v), lang=> ≡ 24a▿
350 ________________________________________________________________________
351 1 | .SUFFIXES: .tex .lyx
355 5 | ↦rm -f -- $(TEX_SOURCE)
357 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
358 6.2.2 Converting from TeXmacs
359 Fangle cannot process TeXmacs files directly3. but this is planned when TeXmacs uses xml as it's native format ^3, but must first convert them to text files.
360 The command texmacs -c fangle.tm fangle.txt -q will produce fangle.txt, a text file; so we define a make target to be the same as the TeXmacs file but with the .txt extension.
361 The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.
363 25a <Makefile.inc-vars[4](
\v) ⇑23b, lang=> +≡ ⊲24b 25a▿
364 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
365 6 | TEXMACS_SOURCE=$(LITERATE_SOURCE) # but only the .tm files
366 7 | TXT_SOURCE=$(LITERATE_SOURCE:.tm=.txt)
367 8 | EXTRA_DIST+=$(TXT_SOURCE)
368 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
369 To do: Add loop around each $< so multiple targets can be specified
372 24a <Makefile.inc-targets[2](
\v) ⇑24c, lang=> +≡ ▵24c 25a⊳
373 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
374 7 | .SUFFIXES: .txt .tm
376 9 | ↦texmacs -s -c $< $@ -q
377 10 | .PHONEY: clean_txt
379 12 | ↦rm -f -- $(TXT_SOURCE)
380 13 | clean: clean_txt
381 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
382 6.3 Extracting Program Source
383 The program source is extracted using fangle, which is designed to operate on text or a LaTeX documents4. LaTeX documents are just slightly special text documents ^4.
385 25a <Makefile.inc-vars[5](
\v) ⇑23b, lang=> +≡ ▵25a 26a⊳
386 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
387 9 | FANGLE_SOURCE=$(TXT_SOURCE)
388 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
389 The literate document can result in any number of source files, but not all of these will be changed each time the document is updated. We certainly don't want to update the timestamps of these files and cause the whole source tree to be recompiled just because the literate explanation was revised. We use CPIF from the Noweb tools to avoid updating the file if the content has not changed, but should probably write our own.
390 However, if a source file is not updated, then the fangle file will always have a newer time-stamp and the makefile would always re-attempt to extact a newer source file which would be a waste of time.
391 Because of this, we use a stamp file which is always updated each time the sources are fully extracted from the LaTeX document. If the stamp file is newer than the document, then we can avoid an attempt to re-extract any of the sources. Because this stamp file is only updated when extraction is complete, it is safe for the user to interrupt the build-process mid-extraction.
392 We use echo rather than touch to update the stamp file beause the touch command does not work very well over an sshfs mount that I was using.
394 26a <Makefile.inc-vars[6](
\v) ⇑23b, lang=> +≡ ⊲25a 26a▿
395 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
396 10 | FANGLE_SOURCE_STAMP=$(FANGLE_SOURCE).stamp
397 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
399 25a <Makefile.inc-targets[3](
\v) ⇑24c, lang=> +≡ ⊲24a 26b⊳
400 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
401 14 | $(FANGLE_SOURCE_STAMP): $(FANGLE_SOURCE) \
402 15 | ↦ $(FANGLE_SOURCES) ; \
403 16 | ↦echo -n > $(FANGLE_SOURCE_STAMP)
405 18 | ↦rm -f $(FANGLE_SOURCE_STAMP)
406 19 | clean: clean_stamp
407 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
408 6.4 Extracting Source Files
409 We compute FANGLE_SOURCES to hold the names of all the source files defined in the document. We compute this only once, by means of := in assignent. The sed deletes the any << and >> which may surround the roots names (for compatibility with Noweb's noroots command).
410 As we use chunk names beginning with ./ to denote top level fragments that should be extracted, we filter out all fragments that do not begin with ./
411 Note 1. FANGLE_PREFIX is set to ./ by default, but whatever it may be overridden to, the prefix is replaced by a literal ./ before extraction so that files will be extracted in the current directory whatever the prefix. This helps namespace or sub-project prefixes like documents: for chunks like documents:docbook/intro.xml
412 To do: This doesn't work though, because it loses the full name and doesn't know what to extact!
415 26a <Makefile.inc-vars[7](
\v) ⇑23b, lang=> +≡ ▵26a 26e▿
416 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
417 11 | FANGLE_PREFIX:=\.\/
418 12 | FANGLE_SOURCES:=$(shell \
419 13 | $(FANGLE) -r $(FANGLE_SOURCE) |\
420 14 | sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(FANGLE_PREFIX)/!d' \
421 15 | -e 's/^$(FANGLE_PREFIX)/\.\//' )
422 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
423 The target below, echo_fangle_sources is a helpful debugging target and shows the names of the files that would be extracted.
425 26b <Makefile.inc-targets[4](
\v) ⇑24c, lang=> +≡ ⊲25a 26c▿
426 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
427 20 | .PHONY: echo_fangle_sources
428 21 | echo_fangle_sources: ; @echo $(FANGLE_SOURCES)
429 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
430 We define a convenient target called fangle_sources so that make -f fangle_sources will re-extract the source if the literate document has been updated.
432 26c <Makefile.inc-targets[5](
\v) ⇑24c, lang=> +≡ ▵26b 26d▿
433 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
434 22 | .PHONY: fangle_sources
435 23 | fangle_sources: $(FANGLE_SOURCE_STAMP)
436 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
437 And also a convenient target to remove extracted sources.
439 26d <Makefile.inc-targets[6](
\v) ⇑24c, lang=> +≡ ▵26c 27a⊳
440 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
441 24 | .PHONY: clean_fangle_sources
442 25 | clean_fangle_sources: ; \
443 26 | rm -f -- $(FANGLE_SOURCE_STAMP) $(FANGLE_SOURCES)
444 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
445 We now look at the extraction of the source files.
446 This makefile macro if_extension takes 4 arguments: the filename $(1), some extensions to match $(2) and a shell command to return if the filename does match the exensions $(3), and a shell command to return if it does not match the extensions $(4).
448 26e <Makefile.inc-vars[8](
\v) ⇑23b, lang=> +≡ ▵26a 26f▿
449 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
450 16 | if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
451 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
452 For some source files like C files, we want to output the line number and filename of the original LaTeX document from which the source came5. I plan to replace this option with a separate mapping file so as not to pollute the generated source, and also to allow a code pretty-printing reformatter like indent be able to re-format the file and adjust for changes through comparing the character streams. ^5.
453 To make this easier we define the file extensions for which we want to do this.
455 26f <Makefile.inc-vars[9](
\v) ⇑23b, lang=> +≡ ▵26e 27a⊳
456 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
457 17 | C_EXTENSIONS=.c .h
458 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
459 We can then use the if_extensions macro to define a macro which expands out to the -L option if fangle is being invoked in a C source file, so that C compile errors will refer to the line number in the TeX document.
461 27a <Makefile.inc-vars[10](
\v) ⇑23b, lang=> +≡ ⊲26f 27b▿
462 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
464 19 | nf_line=-L -T$(TABS)
465 20 | fangle=$(FANGLE) $(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line)) -R"$(2)" $(1)
466 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
467 We can use a similar trick to define an indent macro which takes just the filename as an argument and can return a pipeline stage calling the indent command. Indent can be turned off with make fangle_sources indent=
469 27b <Makefile.inc-vars[11](
\v) ⇑23b, lang=> +≡ ▵27a 27c▿
470 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
471 21 | indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
472 22 | indent=$(call if_extension,$(1),$(C_EXTENSIONS), | indent $(indent_options))
473 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
474 We now define the pattern for extracting a file. The files are written using noweb's cpif so that the file timestamp will not be touched if the contents haven't changed. This avoids the need to rebuild the entire project because of a typographical change in the documentation, or if none or a few C source files have changed.
476 27c <Makefile.inc-vars[12](
\v) ⇑23b, lang=> +≡ ▵27b 28a⊳
477 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
478 23 | fangle_extract=@mkdir -p $(dir $(1)) && \
479 24 | $(call fangle,$(2),$(1)) > "$(1).tmp" && \
480 25 | cat "$(1).tmp" $(indent) | cpif "$(1)" \
481 26 | && rm -f -- "$(1).tmp" || \
482 27 | (echo error fangling $(1) from $(2) ; exit 1)
483 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
484 We define a target which will extract or update all sources. To do this we first defined a makefile template that can do this for any source file in the LaTeX document.
486 28a <Makefile.inc-vars[13](
\v) ⇑23b, lang=> +≡ ⊲27c 28a▿
487 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
488 28 | define FANGLE_template
490 30 | ↦$$(call fangle_extract,$(1),$(2))
491 31 | FANGLE_TARGETS+=$(1)
493 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
494 We then enumerate the discovered FANGLE_SOURCES to generate a makefile rule for each one using the makefile template we defined above.
496 27a <Makefile.inc-targets[7](
\v) ⇑24c, lang=> +≡ ⊲26d 27b▿
497 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
498 27 | $(foreach source,$(FANGLE_SOURCES),\
499 28 | $(eval $(call FANGLE_template,$(source),$(FANGLE_SOURCE))) \
501 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
502 These will all be built with FANGLE_SOURCE_STAMP.
503 We also remove the generated sources on a make distclean.
505 27b <Makefile.inc-targets[8](
\v) ⇑24c, lang=> +≡ ▵27a 28b⊳
506 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
507 30 | _distclean: clean_fangle_sources
508 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
509 6.5 Extracting Documentation
510 We then identify the intermediate stages of the documentation and their build and clean targets.
512 ?a <Makefile.inc-default-targets[1](
\v), lang=> ≡
513 ________________________________________________________________________
514 1 | .PHONEY : clean_pdf
515 |________________________________________________________________________
519 6.5.1.1 Running pdflatex
520 We produce a pdf file from the tex file.
522 28a <Makefile.inc-vars[14](
\v) ⇑23b, lang=> +≡ ▵28a 28c▿
523 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
524 33 | FANGLE_PDF+=$(TEX_SOURCE:.tex=.pdf)
525 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
526 We run pdflatex twice to be sure that the contents and aux files are up to date. We certainly are required to run pdflatex at least twice if these files do not exist.
528 28b <Makefile.inc-targets[9](
\v) ⇑24c, lang=> +≡ ⊲27b 28d▿
529 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
530 31 | .SUFFIXES: .tex .pdf
532 33 | ↦pdflatex $< && pdflatex $<
535 36 | ↦rm -f -- $(FANGLE_PDF) $(TEX_SOURCE:.tex=.toc) \
536 37 | ↦ $(TEX_SOURCE:.tex=.log) $(TEX_SOURCE:.tex=.aux)
537 38 | clean_pdf: clean_pdf_tex
538 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
539 6.5.2 Formatting TeXmacs
540 TeXmacs can produce a PDF file directly.
542 28c <Makefile.inc-vars[15](
\v) ⇑23b, lang=> +≡ ▵28a
543 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
544 34 | FANGLE_PDF+=$(LITERATE_SOURCE:.tm=.pdf)
545 |________________________________________________________________________
548 To do: Outputting the PDF may not be enough to update the links and page references. I think
549 we need to update twice, generate a pdf, update twice mode and generate a new PDF.
550 Basically the PDF export of TeXmacs is pretty rotten and doesn't work properly from the CLI
553 28d <Makefile.inc-targets[10](
\v) ⇑24c, lang=> +≡ ▵28b 28a▿
554 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
555 39 | .SUFFIXES: .tm .pdf
557 41 | ↦texmacs -s -c $< $@ -q
559 43 | clean_pdf_texmacs:
560 44 | ↦rm -f -- $(FANGLE_PDF)
561 45 | clean_pdf: clean_pdf_texmacs
562 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
563 6.5.3 Building the Documentation as a Whole
564 Currently we only build pdf as a final format, but FANGLE_DOCS may later hold other output formats.
566 ?a <Makefile.inc-vars[16](
\v) ⇑23b, lang=> +≡ ⊲28c
567 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
568 1 | FANGLE_DOCS=$(FANGLE_PDF)
569 |________________________________________________________________________
572 We also define fangle_docs as a convenient phony target.
574 28a <Makefile.inc-targets[11](
\v) ⇑24c, lang=> +≡ ▵28d 28b▿
575 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
576 46 | .PHONY: fangle_docs
577 47 | fangle_docs: $(FANGLE_DOCS)
578 48 | docs: fangle_docs
579 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
580 And define a convenient clean_fangle_docs which we add to the regular clean target
582 28b <Makefile.inc-targets[12](
\v) ⇑24c, lang=> +≡ ▵28a
583 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
584 49 | .PHONEY: clean_fangle_docs
585 50 | clean_fangle_docs: clean_tex clean_pdf
586 51 | clean: clean_fangle_docs
588 53 | distclean_fangle_docs: clean_tex clean_fangle_docs
589 54 | distclean: clean distclean_fangle_docs
590 |________________________________________________________________________
594 If Makefile.inc is included into Makefile, then extracted files can be updated with this command:
597 make -f Makefile.inc fangle_sources
598 6.7 Boot-strapping the extraction
599 As well as having the makefile extract or update the source files as part of it's operation, it also seems convenient to have the makefile re-extracted itself from this document.
600 It would also be convenient to have the code that extracts the makefile from this document to also be part of this document, however we have to start somewhere and this unfortunately requires us to type at least a few words by hand to start things off.
601 Therefore we will have a minimal root fragment, which, when extracted, can cope with extracting the rest of the source. This shell script fragment can do that. It's name is * — out of regard for Noweb, but when extracted might better be called autoupdate.
605 29a <*[1](
\v), lang=sh> ≡
606 ________________________________________________________________________
609 3 | MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
610 4 | MAKE_SRC=‘dirname "$MAKE_SRC"‘/‘basename "$MAKE_SRC" .lyx‘
611 5 | NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
612 6 | lyx -e latex $MAKE_SRC
614 8 | fangle -R./Makefile.inc ${MAKE_SRC}.tex \
615 9 | | sed "/FANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$FANGLE_SRC" \
616 10 | | cpif ./Makefile.inc
618 12 | make -f ./Makefile.inc fangle_sources
619 |________________________________________________________________________
622 The general Makefile can be invoked with ./autoboot and can also be included into any automake file to automatically re-generate the source files.
623 The autoboot can be extracted with this command:
624 lyx -e latex fangle.lyx && \
625 fangle fangle.lyx > ./autoboot
626 This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
627 On a unix system this will extract fangle.module and the fangle awk script, and run some basic tests.
628 To do: cross-ref to test chapter when it is a chapter all on its own
630 6.8 Incorporating Makefile.inc into existing projects
631 If you are writing a literate module of an existing non-literate program you may find it easier to use a slight recursive make instead of directly including Makefile.inc in the projects makefile.
632 This way there is less chance of definitions in Makefile.inc interfering with definitions in the main makefile, or with definitions in other Makefile.inc from other literate modules of the same project.
633 To do this we add some glue to the project makefile that invokes Makefile.inc in the right way. The glue works by adding a .PHONY target to call the recursive make, and adding this target as an additional pre-requisite to the existing targets.
634 Example Sub-module of existing system
635 In this example, we are building module.so as a literate module of a larger project.
636 We will show the sort glue that can be inserted into the projects Makefile — or more likely — a regular Makefile included in or invoked by the projects Makefile.
638 30a <makefile-glue[1](
\v), lang=> ≡ 30b▿
639 ________________________________________________________________________
640 1 | module_srcdir=modules/module
641 2 | MODULE_SOURCE=module.tm
642 3 | MODULE_STAMP=$(MODULE_SOURCE).stamp
643 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
644 The existing build system may already have a build target for module.o, but we just add another pre-requisite to that. In this case we use module.tm.stamp as a pre-requisite, the stamp file's modified time indicating when all sources were extracted6. If the projects build system does not know how to build the module from the extracted sources, then just add build actions here as normal. ^6.
646 30b <makefile-glue[2](
\v) ⇑30a, lang=make> +≡ ▵30a 30c▿
647 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
648 4 | $(module_srcdir)/module.o: $(module_srcdir)/$(MODULE_STAMP)
649 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
650 The target for this new pre-requisite will be generated by a recursive make using Makefile.inc which will make sure that the source is up to date, before it is built by the main projects makefile.
652 30c <makefile-glue[3](
\v) ⇑30a, lang=> +≡ ▵30b 30d▿
653 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
654 5 | $(module_srcdir)/$(MODULE_STAMP): $(module_srcdir)/$(MODULE_SOURCE)
655 6 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc fangle_sources LITERATE_SOURCE=$(MODULE_SOURCE)
656 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
657 We can do similar glue for the docs, clean and distclean targets. In this example the main prject was using a double colon for these targets, so we must use the same in our glue.
659 30d <makefile-glue[4](
\v) ⇑30a, lang=> +≡ ▵30c
660 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
661 7 | docs:: docs_module
662 8 | .PHONY: docs_module
664 10 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc docs LITERATE_SOURCE=$(MODULE_SOURCE)
666 12 | clean:: clean_module
667 13 | .PHONEY: clean_module
669 15 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc clean LITERATE_SOURCE=$(MODULE_SOURCE)
671 17 | distclean:: distclean_module
672 18 | .PHONY: distclean_module
673 19 | distclean_module:
674 20 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc distclean LITERATE_SOURCE=$(MODULE_SOURCE)
675 |________________________________________________________________________
678 We could do similarly for install targets to install the generated docs.
680 Chapter 7Fangle awk source code
681 We use the copyright notice from chapter 2.
683 33a <./fangle[1](
\v), lang=awk> ≡ 33b▿
684 ________________________________________________________________________
685 1 | #! /usr/bin/awk -f
686 2 | # «gpl3-copyright 4a»
687 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
688 We also use code from Arnold Robbins public domain getopt (1993 revision) defined in 77a, and naturally want to attribute this appropriately.
690 33b <./fangle[2](
\v) ⇑33a, lang=> +≡ ▵33a 33c▿
691 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
692 3 | # NOTE: Arnold Robbins public domain getopt for awk is also used:
693 4 | «getopt.awk-header 75a»
694 5 | «getopt.awk-getopt() 75c»
696 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
697 And include the following chunks (which are explained further on) to make up the program:
699 33c <./fangle[3](
\v) ⇑33a, lang=> +≡ ▵33b 38a⊳
700 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
701 7 | «helper-functions 34d»
702 8 | «mode-tracker 54a»
703 9 | «parse_chunk_args 40a»
704 10 | «chunk-storage-functions 73b»
705 11 | «output_chunk_names() 67d»
706 12 | «output_chunks() 67e»
707 13 | «write_chunk() 68a»
708 14 | «expand_chunk_args() 40b»
711 17 | «recognize-chunk 57a»
713 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
715 The portable way to erase an array in awk is to split the empty string, so we define a fangle macro that can split an array, like this:
717 33d <awk-delete-array[1](ARRAY
\v\v), lang=awk> ≡
718 ________________________________________________________________________
719 1 | split("", ${ARRAY});
720 |________________________________________________________________________
723 For debugging it is sometimes convenient to be able to dump the contents of an array to stderr, and so this macro is also useful.
725 33e <dump-array[1](ARRAY
\v\v), lang=awk> ≡
726 ________________________________________________________________________
727 1 | print "\nDump: ${ARRAY}\n--------\n" > "/dev/stderr";
728 2 | for (_x in ${ARRAY}) {
729 3 | print _x "=" ${ARRAY}[_x] "\n" > "/dev/stderr";
731 5 | print "========\n" > "/dev/stderr";
732 |________________________________________________________________________
736 Fatal errors are issued with the error function:
738 34a <error()[1](
\v), lang=awk> ≡ 34b▿
739 ________________________________________________________________________
740 1 | function error(message)
742 3 | print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
745 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
746 and likewise for non-fatal warnings:
748 34b <error()[2](
\v) ⇑34a, lang=awk> +≡ ▵34a 34c▿
749 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
750 6 | function warning(message)
752 8 | print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
755 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
756 and debug output too:
758 34c <error()[3](
\v) ⇑34a, lang=awk> +≡ ▵34b
759 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
760 11 | function debug_log(message)
762 13 | print "DEBUG: " FILENAME ":" FNR " " message > "/dev/stderr";
764 |________________________________________________________________________
767 To do: append=helper-functions
770 34d <helper-functions[1](
\v), lang=> ≡
771 ________________________________________________________________________
773 |________________________________________________________________________
776 Chapter 8TeXmacs args
777 TeXmacs functions with arguments1. or function declarations with parameters ^1 appear like this:
778 blah((I came, I saw, I conquered)<wide-overbrace>^(argument 1)(^K, )<wide-overbrace>^(sep.)(and then went home asd)<wide-overbrace>^(argument 3)(^K))<wide-overbrace>^(term.)_arguments
779 Arguments commence after the opening parenthesis. The first argument runs up till the next ^K.
780 If the following character is a , then another argument follows. If the next character after the , is a space character, then it is also eaten. The fangle stylesheet emits ^K,space as separators, but the fangle untangler will forgive a missing space.
781 If the following character is ) then this is a terminator and there are no more arguments.
783 35a <constants[1](
\v), lang=> ≡ 73a⊳
784 ________________________________________________________________________
785 1 | ARG_SEPARATOR=sprintf("%c", 11);
786 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
787 To process the text in this fashion, we split the string on ^K
790 35b <get_chunk_args[1](
\v), lang=> ≡
791 ________________________________________________________________________
792 1 | function get_texmacs_chunk_args(text, args, a, done) {
793 2 | split(text, args, ARG_SEPARATOR);
796 5 | for (a=1; (a in args); a++) if (a>1) {
797 6 | if (args[a] == "" || substr(args[a], 1, 1) == ")") done=1;
803 12 | if (substr(args[a], 1, 2) == ", ") args[a]=substr(args[a], 3);
804 13 | else if (substr(args[a], 1, 1) == ",") args[a]=substr(args[a], 2);
807 |________________________________________________________________________
810 Chapter 9LaTeX and lstlistings
811 To do: Split LyX and TeXmacs parts
813 For L Y X and LaTeX, the lstlistings package is used to format the lines of code chunks. You may recal from chapter XXX that arguments to a chunk definition are pure LaTeX code. This means that fangle needs to be able to parse LaTeX a little.
814 LaTeX arguments to lstlistings macros are a comma seperated list of key-value pairs, and values containing commas are enclosed in { braces } (which is to be expected for LaTeX).
815 A sample expressions is:
816 name=thomas, params={a, b}, something, something-else
817 but we see that this is just a simpler form of this expression:
818 name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
819 We may consider that we need a function that can parse such LaTeX expressions and assign the values to an AWK associated array, perhaps using a recursive parser into a multi-dimensional hash1. as AWK doesn't have nested-hash support ^1, resulting in:
824 a[foo, quux, a] fleeg
827 Yet, also, on reflection it seems that sometimes such nesting is not desirable, as the braces are also used to delimit values that contain commas --- we may consider that
828 name={williamson, freddie}
829 should assign williamson, freddie to name.
830 In fact we are not so interested in the detail so as to be bothered by this, which turns out to be a good thing for two reasons. Firstly TeX has a malleable parser with no strict syntax, and secondly whether or not williamson and freddie should count as two items will be context dependant anyway.
831 We need to parse this latex for only one reason; which is that we are extending lstlistings to add some additional arguments which will be used to express chunk parameters and other chunk options.
832 9.1 Additional lstlstings parameters
833 Further on we define a \Chunk LaTeX macro whose arguments will consist of a the chunk name, optionally followed by a comma and then a comma separated list of arguments. In fact we will just need to prefix name= to the arguments to in order to create valid lstlistings arguments.
834 There will be other arguments supported too;
835 params.As an extension to many literate-programming styles, fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Chunk parameters are declared with a chunk argument called params, which holds a semi-colon separated list of parameters, like this:
836 achunk,language=C,params=name;address
837 addto.a named chunk that this chunk is to be included into. This saves the effort of having to declare another listing of the named chunk merely to include this one.
838 Function get_chunk_args() will accept two paramters, text being the text to parse, and values being an array to receive the parsed values as described above. The optional parameter path is used during recursion to build up the multi-dimensional array path.
840 38a <./fangle[4](
\v) ⇑33a, lang=> +≡ ⊲33c
841 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
842 19 | «get_chunk_args() 38b»
843 |________________________________________________________________________
847 38b <get_chunk_args()[1](
\v), lang=> ≡ 38c▿
848 ________________________________________________________________________
849 1 | function get_tex_chunk_args(text, values,
850 2 | # optional parameters
851 3 | path, # hierarchical precursors
854 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
855 The strategy is to parse the name, and then look for a value. If the value begins with a brace {, then we recurse and consume as much of the text as necessary, returning the remaining text when we encounter a leading close-brace }. This being the strategy --- and executed in a loop --- we realise that we must first look for the closing brace (perhaps preceded by white space) in order to terminate the recursion, and returning remaining text.
857 38c <get_chunk_args()[2](
\v) ⇑38b, lang=> +≡ ▵38b
858 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
860 7 | split("", values);
861 8 | while(length(text)) {
862 9 | if (match(text, "^ *}(.*)", a)) {
865 12 | «parse-chunk-args 38d»
869 |________________________________________________________________________
872 We can see that the text could be inspected with this regex:
874 38d <parse-chunk-args[1](
\v), lang=> ≡ 39a⊳
875 ________________________________________________________________________
876 1 | if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a)) {
879 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
880 and that a will have the following values:
883 2 =freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
885 4 freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
887 6 , foo={bar=baz, quux={quirk, a=fleeg}}, etc
889 a[3] will be either = or , and signify whether the option named in a[1] has a value or not (respectively).
890 If the option does have a value, then if the expression substr(a[4],1,1) returns a brace { it will signify that we need to recurse:
892 39a <parse-chunk-args[2](
\v) ⇑38d, lang=> +≡ ⊲38d
893 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
895 5 | if (a[3] == "=") {
896 6 | if (substr(a[4],1,1) == "{") {
897 7 | text = get_tex_chunk_args(substr(a[4],2), values, path name SUBSEP);
899 9 | values[path name]=a[5];
903 13 | values[path name]="";
906 |________________________________________________________________________
909 We can test this function like this:
911 39b <gca-test.awk[1](
\v), lang=> ≡
912 ________________________________________________________________________
913 1 | «get_chunk_args() 38b»
917 5 | print get_tex_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc", a);
919 7 | print "a[" b "] => " a[b];
922 |________________________________________________________________________
925 which should give this output:
927 39c <gca-test.awk-results[1](
\v), lang=> ≡
928 ________________________________________________________________________
929 1 | a[foo.quux.quirk] =>
930 2 | a[foo.quux.a] => fleeg
931 3 | a[foo.bar] => baz
933 5 | a[name] => freddie
934 |________________________________________________________________________
937 9.2 Parsing chunk arguments
938 Arguments to paramterized chunks are expressed in round brackets as a comma separated list of optional arguments. For example, a chunk that is defined with:
939 \Chunk{achunk, params=name ; address}
941 \chunkref{achunk}(John Jones, jones@example.com)
942 An argument list may be as simple as in \chunkref{pull}(thing, otherthing) or as complex as:
943 \chunkref{pull}(things[x, y], get_other_things(a, "(all)"))
944 --- which for all it's commas and quotes and parenthesis represents only two parameters: things[x, y] and get_other_things(a, "(all)").
945 If we simply split parameter list on commas, then the comma in things[x,y] would split into two seperate arguments: things[x and y]--- neither of which make sense on their own.
946 One way to prevent this would be by refusing to split text between matching delimiters, such as [, ], (, ), {, } and most likely also ", " and ', '. Of course this also makes it impossible to pass such mis-matched code fragments as parameters, but I think that it would be hard for readers to cope with authors who would pass such code unbalanced fragments as chunk parameters2. I know that I couldn't cope with users doing such things, and although the GPL3 license prevents me from actually forbidding anyone from trying, if they want it to work they'll have to write the code themselves and not expect any support from me. ^2.
947 Unfortunately, the full set of matching delimiters may vary from language to language. In certain C++ template contexts, < and > would count as delimiters, and yet in other contexts they would not.
948 This puts me in the unfortunate position of having to parse-somewhat all programming languages without knowing what they are!
949 However, if this universal mode-tracking is possible, then parsing the arguments would be trivial. Such a mode tracker is described in chapter 10 and used here with simplicity.
951 40a <parse_chunk_args[1](
\v), lang=> ≡
952 ________________________________________________________________________
953 1 | function parse_chunk_args(language, text, values, mode,
955 3 | c, context, rest)
957 5 | «new-mode-tracker
\v(context
\v, language
\v, mode
\v) 50b»
958 6 | rest = mode_tracker(context, text, values);
960 8 | for(c=1; c <= context[0, "values"]; c++) {
961 9 | values[c] = context[0, "values", c];
965 |________________________________________________________________________
968 9.3 Expanding parameters in the text
969 Within the body of the chunk, the parameters are referred to with: ${name} and ${address}. There is a strong case that a LaTeX style notation should be used, like \param{name} which would be expressed in the listing as =<\param{name}> and be rendered as ${name}. Such notation would make me go blind, but I do intend to adopt it.
970 We therefore need a function expand_chunk_args which will take a block of text, a list of permitted parameters, and the arguments which must substitute for the parameters.
971 Here we split the text on ${ which means that all parts except the first will begin with a parameter name which will be terminated by }. The split function will consume the literal ${ in each case.
973 40b <expand_chunk_args()[1](
\v), lang=> ≡
974 ________________________________________________________________________
975 1 | function expand_chunk_args(text, params, args,
976 2 | p, text_array, next_text, v, t, l)
978 4 | if (split(text, text_array, "\\${")) {
979 5 | «substitute-chunk-args 41a»
984 |________________________________________________________________________
987 First, we produce an associative array of substitution values indexed by parameter names. This will serve as a cache, allowing us to look up the replacement values as we extract each name.
989 41a <substitute-chunk-args[1](
\v), lang=> ≡ 41b▿
990 ________________________________________________________________________
991 1 | for(p in params) {
992 2 | v[params[p]]=args[p];
994 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
995 We accumulate substituted text in the variable text. As the first part of the split function is the part before the delimiter --- which is ${ in our case --- this part will never contain a parameter reference, so we assign this directly to the result kept in $text.
997 41b <substitute-chunk-args[2](
\v) ⇑41a, lang=> +≡ ▵41a 41c▿
998 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
999 4 | text=text_array[1];
1000 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1001 We then iterate over the remaining values in the array3. I don't know why I think that it will enumerate the array in order, but it seems to work ^3
1002 To do: fix or prove it
1003 , and substitute each reference for it's argument.
1005 41c <substitute-chunk-args[3](
\v) ⇑41a, lang=> +≡ ▵41b
1006 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1007 5 | for(t=2; t in text_array; t++) {
1008 6 | «substitute-chunk-arg 41d»
1010 |________________________________________________________________________
1013 After the split on ${ a valid parameter reference will consist of valid parameter name terminated by a close-brace }. A valid character name begins with the underscore or a letter, and may contain letters, digits or underscores.
1014 A valid looking reference that is not actually the name of a parameter will be and not substituted. This is good because there is nothing to substitute anyway, and it avoids clashes when writing code for languages where ${...} is a valid construct --- such constructs will not be interfered with unless the parameter name also matches.
1016 41d <substitute-chunk-arg[1](
\v), lang=> ≡
1017 ________________________________________________________________________
1018 1 | if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
1021 4 | text = text v[l[1]] substr(text_array[t], length(l[1])+2);
1023 6 | text = text "${" text_array[t];
1025 |________________________________________________________________________
1028 Chapter 10Language Modes & Quoting
1030 lstlistings and fangle both recognize source languages, and perform some basic parsing. lstlistings can detect strings and comments within a language definition and perform suitable rendering, such as italics for comments, and visible-spaces within strings.
1031 Fangle similarly can recognize strings, and comments, etc, within a language, so that any chunks included with \chunkref can be suitably escape or quoted.
1032 10.1.1 Modes to keep code together
1033 As an example, in the C language there are a few parse modes, affecting the interpretation of characters.
1034 One parse mode is the strings mode. The string mode is commenced by an un-escaped quotation mark " and terminated by the same. Within the string mode, only one additional mode can be commenced, it is the backslash mode \, which is always terminated after the folloing character.
1035 Another mode is [ which is terminated by a ] (unless it occurs in a string).
1036 Consider this fragment of C code:
1038 things([x, y])<wide-overbrace>^(1. [ mode), get_other_things((a, "(all)"_(3. " mode)))<wide-overbrace>^(2. ( mode)
1040 Mode nesting prevents the close parenthesis in the quoted string (part 3) from terminating the parenthesis mode (part 2).
1041 Each language has a set of modes, the default mode being the null mode. Each mode can lead to other modes.
1042 10.1.2 Modes affect included chunks
1043 For instance, consider this chunk with language=perl:
1045 43a <example-perl[1](
\v), lang=perl> ≡
1046 ________________________________________________________________________
1047 print "hello world $0\n";
1048 |________________________________________________________________________
1051 If it were included in a chunk with language=sh, like this:
1053 43b <example-sh[1](
\v), lang=sh> ≡
1054 ________________________________________________________________________
1055 perl -e "«example-perl 43a»"
1056 |________________________________________________________________________
1059 fangle would want to generate output like this:
1060 perl -e "print \"hello world \$0\\n\";"
1061 See that the double quote ", back-slash \ and $ have been quoted with a back-slash to protect them from shell interpretation.
1062 If that were then included in a chunk with language=make, like this:
1064 44a <example-makefile[1](
\v), lang=make> ≡
1065 ________________________________________________________________________
1067 2 | «example-sh 43b»
1068 |________________________________________________________________________
1071 We would need the output to look like this --- note the $$:
1073 perl -e "print \"hello world \$$0\\n\";"
1074 In order to make this work, we need to define a mode-tracker supporting each language, that can detect the various quoting modes, and provide a transformation that must be applied to any included text so that included text will be interpreted correctly after any interpolation that it may be subject to at run-time.
1075 For example, the sed transformation for text to be inserted into shell double-quoted strings would be something like:
1076 s/\\/\\\\/g;s/$/\\$/g;s/"/\\"/g;
1077 which protects \ $ ".
1078 To do: I don't think this example is true
1079 The mode tracker must also track nested mode-changes, as in this sh example.
1080 echo "hello ‘id ...‘"
1082 Any characters inserted at the point marked ↑ would need to be escaped, including ‘ | * among others. First it would need escaping for the back-ticks ‘, and then for the double-quotes ".
1084 Escaping need not occur if the format and mode of the included chunk matches that of the including chunk.
1085 As each chunk is output a new mode tracker for that language is initialized in it's normal state. As text is output for that chunk the output mode is tracked. When a new chunk is included, a transformation appropriate to that mode is selected and pushed onto a stack of transformations. Any text to be output is first passed through this stack of transformations.
1086 It remains to consider if the chunk-include function should return it's generated text so that the caller can apply any transformations (and formatting), or if it should apply the stack of transformations itself.
1087 Note that the transformed text should have the property of not being able to change the mode in the current chunk.
1088 To do: Note chunk parameters should probably also be transformed
1090 10.2 Language Mode Definitions
1091 All modes are stored in a single multi-dimensional hash. The first index is the language, and the second index is the mode-identifier. The third indexes are terminators, and optionally, submodes, and delimiters.
1092 A useful set of mode definitions for a nameless general C-type language is shown here. (Don't be confused by the double backslash escaping needed in awk. One set of escaping is for the string, and the second set of escaping is for the regex).
1093 To do: TODO: Add =<\mode{}> command which will allow us to signify that a string is
1094 regex and thus fangle will quote it for us.
1096 Submodes are entered by the characters " ' { ( [ /*
1098 45a <common-mode-definitions[1](language
\v\v), lang=> ≡ 45b▿
1099 ________________________________________________________________________
1100 1 | modes[${language}, "", "submodes"]="\\\\|\"|'|{|\\(|\\[";
1101 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1102 In the default mode, a comma surrounded by un-important white space is a delimiter of language items1. whatever a language item might be ^1.
1104 45b <common-mode-definitions[2](language
\v\v) ⇑45a, lang=> +≡ ▵45a 45d▿
1105 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1106 2 | modes[${language}, "", "delimiters"]=" *, *";
1107 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1108 and should pass this test:
1109 To do: Why do the tests run in ?(? mode and not ?? mode
1112 45c <test:mode-definitions[1](
\v), lang=> ≡ 46g⊳
1113 ________________________________________________________________________
1114 1 | parse_chunk_args("c-like", "1,2,3", a, "");
1115 2 | if (a[1] != "1") e++;
1116 3 | if (a[2] != "2") e++;
1117 4 | if (a[3] != "3") e++;
1118 5 | if (length(a) != 3) e++;
1119 6 | «pca-test.awk:summary 54c»
1121 8 | parse_chunk_args("c-like", "joe, red", a, "");
1122 9 | if (a[1] != "joe") e++;
1123 10 | if (a[2] != "red") e++;
1124 11 | if (length(a) != 2) e++;
1125 12 | «pca-test.awk:summary 54c»
1127 14 | parse_chunk_args("c-like", "${colour}", a, "");
1128 15 | if (a[1] != "${colour}") e++;
1129 16 | if (length(a) != 1) e++;
1130 17 | «pca-test.awk:summary 54c»
1131 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1132 Nested modes are identified by a backslash, a double or single quote, various bracket styles or a /* comment.
1133 For each of these sub-modes modes we must also identify at a mode terminator, and any sub-modes or delimiters that may be entered2. Because we are using the sub-mode characters as the mode identifier it means we can't currently have a mode character dependant on it's context; i.e. { can't behave differently when it is inside [. ^2.
1135 The backslash mode has no submodes or delimiters, and is terminated by any character. Note that we are not so much interested in evaluating or interpolating content as we are in delineating content. It is no matter that a double backslash (\\) may represent a single backslash while a backslash-newline may represent white space, but it does matter that the newline in a backslash newline should not be able to terminate a C pre-processor statement; and so the newline will be consumed by the backslash however it is to be interpreted.
1137 45d <common-mode-definitions[3](language
\v\v) ⇑45a, lang=> +≡ ▵45b 46f⊳
1138 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1139 3 | modes[${language}, "\\", "terminators"]=".";
1140 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1142 Common languages support two kinds of strings quoting, double quotes and single quotes.
1143 In a string we have one special mode, which is the backslash. This may escape an embedded quote and prevent us thinking that it should terminate the string.
1145 46a <mode:common-string[1](language
\v, quote
\v\v), lang=> ≡ 46b▿
1146 ________________________________________________________________________
1147 1 | modes[${language}, ${quote}, "submodes"]="\\\\";
1148 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1149 Otherwise, the string will be terminated by the same character that commenced it.
1151 46b <mode:common-string[2](language
\v, quote
\v\v) ⇑46a, lang=> +≡ ▵46a 46c▿
1152 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1153 2 | modes[${language}, ${quote}, "terminators"]=${quote};
1154 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1155 In C type languages, certain escape sequences exist in strings. We need to define mechanism to enclode any chunks included in this mode using those escape sequences. These are expressed in two parts, s meaning search, and r meaning replace.
1156 The first substitution is to replace a backslash with a double backslash. We do this first as other substitutions may introduce a backslash which we would not then want to escape again here.
1157 Note: Backslashes need double-escaping in the search pattern but not in the replacement string, hence we are replacing a literal \ with a literal \\.
1159 46c <mode:common-string[3](language
\v, quote
\v\v) ⇑46a, lang=> +≡ ▵46b 46d▿
1160 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1161 3 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\\\";
1162 4 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\\\";
1163 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1164 If the quote character occurs in the text, it should be preceded by a backslash, otherwise it would terminate the string unexpectedly.
1166 46d <mode:common-string[4](language
\v, quote
\v\v) ⇑46a, lang=> +≡ ▵46c 46e▿
1167 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1168 5 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]=${quote};
1169 6 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\" ${quote};
1170 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1171 Any newlines in the string, must be replaced by \n.
1173 46e <mode:common-string[5](language
\v, quote
\v\v) ⇑46a, lang=> +≡ ▵46d
1174 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1175 7 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\n";
1176 8 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\n";
1177 |________________________________________________________________________
1180 For the common modes, we define this string handling for double and single quotes.
1182 46f <common-mode-definitions[4](language
\v\v) ⇑45a, lang=> +≡ ⊲45d 47b⊳
1183 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1184 4 | «mode:common-string
\v(${language}
\v, "\""
\v) 46a»
1185 5 | «mode:common-string
\v(${language}
\v, "'"
\v) 46a»
1186 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1187 Working strings should pass this test:
1189 46g <test:mode-definitions[2](
\v) ⇑45c, lang=> +≡ ⊲45c 49d⊳
1190 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1191 18 | parse_chunk_args("c-like", "say \"I said, \\\"Hello, how are you\\\".\", for me", a, "");
1192 19 | if (a[1] != "say \"I said, \\\"Hello, how are you\\\".\"") e++;
1193 20 | if (a[2] != "for me") e++;
1194 21 | if (length(a) != 2) e++;
1195 22 | «pca-test.awk:summary 54c»
1196 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1197 10.2.3 Parentheses, Braces and Brackets
1198 Where quotes are closed by the same character, parentheses, brackets and braces are closed by an alternate character.
1200 47a <mode:common-brackets[1](language
\v, open
\v, close
\v\v), lang=> ≡
1201 ________________________________________________________________________
1202 1 | modes[${language}, ${open}, "submodes" ]="\\\\|\"|{|\\(|\\[|'|/\\*";
1203 2 | modes[${language}, ${open}, "delimiters"]=" *, *";
1204 3 | modes[${language}, ${open}, "terminators"]=${close};
1205 |________________________________________________________________________
1208 Note that the open is NOT a regex but the close token IS.
1209 To do: When we can quote regex we won't have to put the slashes in here
1212 47b <common-mode-definitions[5](language
\v\v) ⇑45a, lang=> +≡ ⊲46f
1213 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1214 6 | «mode:common-brackets
\v(${language}
\v, "{"
\v, "}"
\v) 47a»
1215 7 | «mode:common-brackets
\v(${language}
\v, "["
\v, "\\]"
\v) 47a»
1216 8 | «mode:common-brackets
\v(${language}
\v, "("
\v, "\\)"
\v) 47a»
1217 |________________________________________________________________________
1220 10.2.4 Customizing Standard Modes
1222 47c <mode:add-submode[1](language
\v, mode
\v, submode
\v\v), lang=> ≡
1223 ________________________________________________________________________
1224 1 | modes[${language}, ${mode}, "submodes"] = modes[${language}, ${mode}, "submodes"] "|" ${submode};
1225 |________________________________________________________________________
1229 47d <mode:add-escapes[1](language
\v, mode
\v, search
\v, replace
\v\v), lang=> ≡
1230 ________________________________________________________________________
1231 1 | escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "s"]=${search};
1232 2 | escapes[${language}, ${mode}, escapes[${language}, ${mode}], "r"]=${replace};
1233 |________________________________________________________________________
1238 We can define /* comment */ style comments and //comment style comments to be added to any language:
1240 47e <mode:multi-line-comments[1](language
\v\v), lang=> ≡
1241 ________________________________________________________________________
1242 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "/\\*"
\v) 47c»
1243 2 | modes[${language}, "/*", "terminators"]="\\*/";
1244 |________________________________________________________________________
1248 47f <mode:single-line-slash-comments[1](language
\v\v), lang=> ≡
1249 ________________________________________________________________________
1250 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "//"
\v) 47c»
1251 2 | modes[${language}, "//", "terminators"]="\n";
1252 3 | «mode:add-escapes
\v(${language}
\v, "//"
\v, "\n"
\v, "\n//"
\v) 47d»
1253 |________________________________________________________________________
1256 We can also define # comment style comments (as used in awk and shell scripts) in a similar manner.
1257 To do: I'm having to use # for hash and ¯extbackslash{} for and have hacky work-arounds in the parser for now
1260 47g <mode:add-hash-comments[1](language
\v\v), lang=> ≡
1261 ________________________________________________________________________
1262 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "#"
\v) 47c»
1263 2 | modes[${language}, "#", "terminators"]="\n";
1264 3 | «mode:add-escapes
\v(${language}
\v, "#"
\v, "\n"
\v, "\n#"
\v) 47d»
1265 |________________________________________________________________________
1268 In C, the # denotes pre-processor directives which can be multi-line
1270 48a <mode:add-hash-defines[1](language
\v\v), lang=> ≡
1271 ________________________________________________________________________
1272 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "#"
\v) 47c»
1273 2 | modes[${language}, "#", "submodes" ]="\\\\";
1274 3 | modes[${language}, "#", "terminators"]="\n";
1275 4 | «mode:add-escapes
\v(${language}
\v, "#"
\v, "\n"
\v, "\\\\\n"
\v) 47d»
1276 |________________________________________________________________________
1280 48b <mode:quote-dollar-escape[1](language
\v, quote
\v\v), lang=> ≡
1281 ________________________________________________________________________
1282 1 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\$";
1283 2 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\$";
1284 |________________________________________________________________________
1287 We can add these definitions to various languages
1289 48c <mode-definitions[1](
\v), lang=> ≡ 49b⊳
1290 ________________________________________________________________________
1291 1 | «common-mode-definitions
\v("c-like"
\v) 45a»
1293 3 | «common-mode-definitions
\v("c"
\v) 45a»
1294 4 | «mode:multi-line-comments
\v("c"
\v) 47e»
1295 5 | «mode:single-line-slash-comments
\v("c"
\v) 47f»
1296 6 | «mode:add-hash-defines
\v("c"
\v) 48a»
1298 8 | «common-mode-definitions
\v("awk"
\v) 45a»
1299 9 | «mode:add-hash-comments
\v("awk"
\v) 47g»
1300 10 | «mode:add-naked-regex
\v("awk"
\v) 49a»
1301 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1302 The awk definitions should allow a comment block like this:
1304 48d <test:comment-quote[1](
\v), lang=awk> ≡
1305 ________________________________________________________________________
1306 1 | # Comment: «test:comment-text 48e»
1307 |________________________________________________________________________
1311 48e <test:comment-text[1](
\v), lang=> ≡
1312 ________________________________________________________________________
1313 1 | Now is the time for
1314 2 | the quick brown fox to bring lemonade
1316 |________________________________________________________________________
1319 to come out like this:
1321 48f <test:comment-quote:result[1](
\v), lang=> ≡
1322 ________________________________________________________________________
1323 1 | # Comment: Now is the time for
1324 2 | #the quick brown fox to bring lemonade
1326 |________________________________________________________________________
1329 The C definition for such a block should have it come out like this:
1331 48g <test:comment-quote:C-result[1](
\v), lang=> ≡
1332 ________________________________________________________________________
1333 1 | # Comment: Now is the time for\
1334 2 | the quick brown fox to bring lemonade\
1336 |________________________________________________________________________
1340 This pattern is incomplete, but meant to detect naked regular expressions in awk and perl; e.g. /.*$/, however required capabilities are not present.
1341 Current it only detects regexes anchored with ^ as used in fangle.
1342 For full regex support, modes need to be named not after their starting character, but some other more fully qualified name.
1344 49a <mode:add-naked-regex[1](language
\v\v), lang=> ≡
1345 ________________________________________________________________________
1346 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "/\\^"
\v) 47c»
1347 2 | modes[${language}, "/^", "terminators"]="/";
1348 |________________________________________________________________________
1353 49b <mode-definitions[2](
\v) ⇑48c, lang=> +≡ ⊲48c 49c▿
1354 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1355 11 | «common-mode-definitions
\v("perl"
\v) 45a»
1356 12 | «mode:multi-line-comments
\v("perl"
\v) 47e»
1357 13 | «mode:add-hash-comments
\v("perl"
\v) 47g»
1358 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1359 Still need to add add s/, submode /, terminate both with //. This is likely to be impossible as perl regexes can contain perl.
1362 49c <mode-definitions[3](
\v) ⇑48c, lang=awk> +≡ ▵49b
1363 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1364 14 | «common-mode-definitions
\v("sh"
\v) 45a»
1365 15 | «mode:add-hash-comments
\v("sh"
\v) 47g»
1366 16 | «mode:quote-dollar-escape
\v("sh"
\v, "\""
\v) 48b»
1367 |________________________________________________________________________
1371 Also, the parser must return any spare text at the end that has not been processed due to a mode terminator being found.
1373 49d <test:mode-definitions[3](
\v) ⇑45c, lang=> +≡ ⊲46g 49e▿
1374 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1375 23 | rest = parse_chunk_args("c-like", "1, 2, 3) spare", a, "(");
1376 24 | if (a[1] != 1) e++;
1377 25 | if (a[2] != 2) e++;
1378 26 | if (a[3] != 3) e++;
1379 27 | if (length(a) != 3) e++;
1380 28 | if (rest != " spare") e++;
1381 29 | «pca-test.awk:summary 54c»
1382 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1383 We must also be able to parse the example given earlier.
1385 49e <test:mode-definitions[4](
\v) ⇑45c, lang=> +≡ ▵49d
1386 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1387 30 | parse_chunk_args("c-like", "things[x, y], get_other_things(a, \"(all)\"), 99", a, "(");
1388 31 | if (a[1] != "things[x, y]") e++;
1389 32 | if (a[2] != "get_other_things(a, \"(all)\")") e++;
1390 33 | if (a[3] != "99") e++;
1391 34 | if (length(a) != 3) e++;
1392 35 | «pca-test.awk:summary 54c»
1393 |________________________________________________________________________
1396 10.4 A non-recursive mode tracker
1398 The mode tracker holds its state in a stack based on a numerically indexed hash. This function, when passed an empty hash, will intialize it.
1400 50a <new_mode_tracker()[1](
\v), lang=> ≡
1401 ________________________________________________________________________
1402 1 | function new_mode_tracker(context, language, mode) {
1403 2 | context[""] = 0;
1404 3 | context[0, "language"] = language;
1405 4 | context[0, "mode"] = mode;
1407 |________________________________________________________________________
1410 Because awk functions cannot return an array, we must create the array first and pass it in, so we have a fangle macro to do this:
1412 50b <new-mode-tracker[1](context
\v, language
\v, mode
\v\v), lang=awk> ≡
1413 ________________________________________________________________________
1414 1 | «awk-delete-array
\v(context
\v) 33d»
1415 2 | new_mode_tracker(${context}, ${language}, ${mode});
1416 |________________________________________________________________________
1420 And for tracking modes, we dispatch to a mode-tracker action based on the current language
1422 50c <mode_tracker[1](
\v), lang=awk> ≡ 50d▿
1423 ________________________________________________________________________
1424 1 | function push_mode_tracker(context, language, mode,
1428 5 | if (! ("" in context)) {
1429 6 | «new-mode-tracker
\v(context
\v, language
\v, mode
\v) 50b»
1431 8 | top = context[""];
1432 9 | if (context[top, "language"] == language && mode=="") mode = context[top, "mode"];
1434 11 | context[top, "language"] = language;
1435 12 | context[top, "mode"] = mode;
1436 13 | context[""] = top;
1439 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1441 50d <mode_tracker[2](
\v) ⇑50c, lang=> +≡ ▵50c 50e▿
1442 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1443 16 | function dump_mode_tracker(context,
1446 19 | for(c=0; c <= context[""]; c++) {
1447 20 | printf(" %2d %s:%s\n", c, context[c, "language"], context[c, "mode"]) > "/dev/stderr";
1448 21 | for(d=1; ( (c, "values", d) in context); d++) {
1449 22 | printf(" %2d %s\n", d, context[c, "values", d]) > "/dev/stderr";
1453 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1455 50e <mode_tracker[3](
\v) ⇑50c, lang=> +≡ ▵50d 55a⊳
1456 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1457 26 | function finalize_mode_tracker(context)
1459 28 | if ( ("" in context) && context[""] != 0) return 0;
1462 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1463 This implies that any chunk must be syntactically whole; for instance, this is fine:
1465 51a <test:whole-chunk[1](
\v), lang=> ≡
1466 ________________________________________________________________________
1468 2 | «test:say-hello 51b»
1470 |________________________________________________________________________
1474 51b <test:say-hello[1](
\v), lang=> ≡
1475 ________________________________________________________________________
1477 |________________________________________________________________________
1480 But this is not fine; the chunk <test:hidden-else 51d> is not properly cromulent.
1482 51c <test:partial-chunk[1](
\v), lang=> ≡
1483 ________________________________________________________________________
1485 2 | «test:hidden-else 51d»
1487 |________________________________________________________________________
1491 51d <test:hidden-else[1](
\v), lang=> ≡
1492 ________________________________________________________________________
1493 1 | print "I'm fine";
1495 3 | print "I'm not";
1496 |________________________________________________________________________
1499 These tests will check for correct behaviour:
1501 51e <test:cromulence[1](
\v), lang=> ≡
1502 ________________________________________________________________________
1503 1 | echo Cromulence test
1504 2 | passtest $FANGLE -Rtest:whole-chunk $TEX_SRC &>/dev/null || ( echo "Whole chunk failed" && exit 1 )
1505 3 | failtest $FANGLE -Rtest:partial-chunk $TEX_SRC &>/dev/null || ( echo "Partial chunk failed" && exit 1 )
1506 |________________________________________________________________________
1510 We must avoid recursion as a language construct because we intend to employ mode-tracking to track language mode of emitted code, and the code is emitted from a function which is itself recursive, so instead we implement psuedo-recursion using our own stack based on a hash.
1512 51f <mode_tracker()[1](
\v), lang=awk> ≡ 51g▿
1513 ________________________________________________________________________
1514 1 | function mode_tracker(context, text, values,
1515 2 | # optional parameters
1517 4 | mode, submodes, language,
1518 5 | cindex, c, a, part, item, name, result, new_values, new_mode,
1519 6 | delimiters, terminators)
1521 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1522 We could be re-commencing with a valid context, so we need to setup the state according to the last context.
1524 51g <mode_tracker()[2](
\v) ⇑51f, lang=> +≡ ▵51f 52c⊳
1525 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1526 8 | cindex = context[""] + 0;
1527 9 | mode = context[cindex, "mode"];
1528 10 | language = context[cindex, "language" ];
1529 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1530 First we construct a single large regex combining the possible sub-modes for the current mode along with the terminators for the current mode.
1532 52a <parse_chunk_args-reset-modes[1](
\v), lang=> ≡ 52b▿
1533 ________________________________________________________________________
1534 1 | submodes=modes[language, mode, "submodes"];
1536 3 | if ((language, mode, "delimiters") in modes) {
1537 4 | delimiters = modes[language, mode, "delimiters"];
1538 5 | if (length(submodes)>0) submodes = submodes "|";
1539 6 | submodes=submodes delimiters;
1540 7 | } else delimiters="";
1541 8 | if ((language, mode, "terminators") in modes) {
1542 9 | terminators = modes[language, mode, "terminators"];
1543 10 | if (length(submodes)>0) submodes = submodes "|";
1544 11 | submodes=submodes terminators;
1545 12 | } else terminators="";
1546 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1547 If we don't find anything to match on --- probably because the language is not supported --- then we return the entire text without matching anything.
1549 52b <parse_chunk_args-reset-modes[2](
\v) ⇑52a, lang=> +≡ ▵52a
1550 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1551 13 | if (! length(submodes)) return text;
1552 |________________________________________________________________________
1556 52c <mode_tracker()[3](
\v) ⇑51f, lang=> +≡ ⊲51g 52d▿
1557 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1558 11 | «parse_chunk_args-reset-modes 52a»
1559 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1560 We then iterate the text (until there is none left) looking for sub-modes or terminators in the regex.
1562 52d <mode_tracker()[4](
\v) ⇑51f, lang=> +≡ ▵52c 52e▿
1563 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1564 12 | while((cindex >= 0) && length(text)) {
1565 13 | if (match(text, "(" submodes ")", a)) {
1566 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1567 A bug that creeps in regularly during development is bad regexes of zero length which result in an infinite loop (as no text is consumed), so I catch that right away with this test.
1569 52e <mode_tracker()[5](
\v) ⇑51f, lang=> +≡ ▵52d 52f▿
1570 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1571 14 | if (RLENGTH<1) {
1572 15 | error(sprintf("Internal error, matched zero length submode, should be impossible - likely regex computation error\n" \
1573 16 | "Language=%s\nmode=%s\nmatch=%s\n", language, mode, submodes));
1575 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1576 part is defined as the text up to the sub-mode or terminator, and this is appended to item --- which is the current text being gathered. If a mode has a delimiter, then item is reset each time a delimiter is found.
1577 ("hello_item, there_item")<wide-overbrace>^item, (he said.)<wide-overbrace>^item
1579 52f <mode_tracker()[6](
\v) ⇑51f, lang=> +≡ ▵52e 52g▿
1580 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1581 18 | part = substr(text, 1, RSTART -1);
1582 19 | item = item part;
1583 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1584 We must now determine what was matched. If it was a terminator, then we must restore the previous mode.
1586 52g <mode_tracker()[7](
\v) ⇑51f, lang=> +≡ ▵52f 53a⊳
1587 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1588 20 | if (match(a[1], "^" terminators "$")) {
1589 21 | #printf("%2d EXIT MODE [%s] by [%s] [%s]\n", cindex, mode, a[1], text) > "/dev/stderr"
1590 22 | context[cindex, "values", ++context[cindex, "values"]] = item;
1591 23 | delete context[cindex];
1592 24 | context[""] = --cindex;
1593 25 | if (cindex>=0) {
1594 26 | mode = context[cindex, "mode"];
1595 27 | language = context[cindex, "language"];
1596 28 | «parse_chunk_args-reset-modes 52a»
1598 30 | item = item a[1];
1599 31 | text = substr(text, 1 + length(part) + length(a[1]));
1601 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1602 If a delimiter was matched, then we must store the current item in the parsed values array, and reset the item.
1604 53a <mode_tracker()[8](
\v) ⇑51f, lang=> +≡ ⊲52g 53b▿
1605 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1606 33 | else if (match(a[1], "^" delimiters "$")) {
1607 34 | if (cindex==0) {
1608 35 | context[cindex, "values", ++context[cindex, "values"]] = item;
1611 38 | item = item a[1];
1613 40 | text = substr(text, 1 + length(part) + length(a[1]));
1615 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1616 otherwise, if a new submode is detected (all submodes have terminators), we must create a nested parse context until we find the terminator for this mode.
1618 53b <mode_tracker()[9](
\v) ⇑51f, lang=> +≡ ▵53a 53c▿
1619 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1620 42 | else if ((language, a[1], "terminators") in modes) {
1621 43 | #check if new_mode is defined
1622 44 | item = item a[1];
1623 45 | #printf("%2d ENTER MODE [%s] in [%s]\n", cindex, a[1], text) > "/dev/stderr"
1624 46 | text = substr(text, 1 + length(part) + length(a[1]));
1625 47 | context[""] = ++cindex;
1626 48 | context[cindex, "mode"] = a[1];
1627 49 | context[cindex, "language"] = language;
1629 51 | «parse_chunk_args-reset-modes 52a»
1631 53 | error(sprintf("Submode '%s' set unknown mode in text: %s\nLanguage %s Mode %s\n", a[1], text, language, mode));
1632 54 | text = substr(text, 1 + length(part) + length(a[1]));
1635 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1636 In the final case, we parsed to the end of the string. If the string was entire, then we should have no nested mode context, but if the string was just a fragment we may have a mode context which must be preserved for the next fragment. Todo: Consideration ought to be given if sub-mode strings are split over two fragments.
1638 53c <mode_tracker()[10](
\v) ⇑51f, lang=> +≡ ▵53b
1639 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1641 58 | context[cindex, "values", ++context[cindex, "values"]] = item text;
1647 64 | context["item"] = item;
1649 66 | if (length(item)) context[cindex, "values", ++context[cindex, "values"]] = item;
1652 |________________________________________________________________________
1655 10.4.3.1 One happy chunk
1656 All the mode tracker chunks are referred to here:
1658 54a <mode-tracker[1](
\v), lang=> ≡
1659 ________________________________________________________________________
1660 1 | «new_mode_tracker() 50a»
1661 2 | «mode_tracker() 51f»
1662 |________________________________________________________________________
1666 We can test this function like this:
1668 54b <pca-test.awk[1](
\v), lang=awk> ≡
1669 ________________________________________________________________________
1671 2 | «mode-tracker 54a»
1672 3 | «parse_chunk_args() ?»
1675 6 | «mode-definitions 48c»
1677 8 | «test:mode-definitions 45c»
1679 |________________________________________________________________________
1683 54c <pca-test.awk:summary[1](
\v), lang=awk> ≡
1684 ________________________________________________________________________
1686 2 | printf "Failed " e
1688 4 | print "a[" b "] => " a[b];
1695 |________________________________________________________________________
1698 which should give this output:
1700 54d <pca-test.awk-results[1](
\v), lang=> ≡
1701 ________________________________________________________________________
1702 1 | a[foo.quux.quirk] =>
1703 2 | a[foo.quux.a] => fleeg
1704 3 | a[foo.bar] => baz
1706 5 | a[name] => freddie
1707 |________________________________________________________________________
1710 10.5 Escaping and Quoting
1711 For the time being and to get around TeXmacs inability to export a TAB character, the right arrow ↦ whose UTF-8 sequence is ...
1714 Another special character is used, the left-arrow ↤ with UTF-8 sequence 0xE2 0x86 0xA4 is used to strip any preceding white space as a way of un-tabbing and removing indent that has been applied — this is important for bash here documents, and the like. It's a filthy hack.
1715 To do: remove the hack
1718 55a <mode_tracker[4](
\v) ⇑50c, lang=> +≡ ⊲50e 55b▿
1719 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1721 31 | function untab(text) {
1722 32 | gsub("[[:space:]]*\xE2\x86\xA4","", text);
1725 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1726 Each nested mode can optionally define a set of transforms to be applied to any text that is included from another language.
1727 This code can perform transforms
1729 55b <mode_tracker[5](
\v) ⇑50c, lang=awk> +≡ ▵55a 55c▿
1730 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1731 35 | function transform_escape(s, r, text,
1737 41 | for(c=1; c <= max && (c in s); c++) {
1738 42 | gsub(s[c], r[c], text);
1742 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1743 This function must append from index c onwards, and escape transforms from the supplied context, and return c + number of new transforms.
1745 55c <mode_tracker[6](
\v) ⇑50c, lang=awk> +≡ ▵55b
1746 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1747 46 | function mode_escaper(context, s, r, src,
1750 49 | for(c = context[""]; c >= 0; c--) {
1751 50 | if ( (context[c, "language"], context[c, "mode"]) in escapes) {
1752 51 | cpl = escapes[context[c, "language"], context[c, "mode"]];
1753 52 | for (cp = 1; cp <= cpl; cp ++) {
1755 54 | s[src] = escapes[context[c, "language"], context[c, "mode"], cp, "s"];
1756 55 | r[src] = escapes[context[c, "language"], context[c, "mode"], cp, "r"];
1762 61 | function dump_escaper(c, s, r, cc) {
1763 62 | for(cc=1; cc<=c; cc++) {
1764 63 | printf("%2d s[%s] r[%s]\n", cc, s[cc], r[cc]) > "/dev/stderr"
1767 |________________________________________________________________________
1771 55d <test:escapes[1](
\v), lang=sh> ≡
1772 ________________________________________________________________________
1773 1 | echo escapes test
1774 2 | passtest $FANGLE -Rtest:comment-quote $TEX_SRC &>/dev/null || ( echo "Comment-quote failed" && exit 1 )
1775 |________________________________________________________________________
1778 Chapter 11Recognizing Chunks
1779 Fangle recognizes noweb chunks, but as we also want better LaTeX integration we will recognize any of these:
1780 • notangle chunks matching the pattern ^<<.*?>>=
1781 • chunks beginning with \begin{lstlistings}, possibly with \Chunk{...} on the previous line
1782 • an older form I have used, beginning with \begin{Chunk}[options] --- also more suitable for plain LaTeX users1. Is there such a thing as plain LaTeX? ^1.
1784 The variable chunking is used to signify that we are processing a code chunk and not document. In such a state, input lines will be assigned to the current chunk; otherwise they are ignored.
1786 We don't handle TeXmacs files natively yet, but rather instead emit unicode character sequences to mark up the text-export file which we do process.
1787 These hacks detect the unicode character sequences and retro-fit in the old TeX parsing.
1788 We convert ↦ into a tab character.
1790 57a <recognize-chunk[1](
\v), lang=> ≡ 57b▿
1791 ________________________________________________________________________
1794 2 | # gsub("\n*$","");
1795 3 | # gsub("\n", " ");
1798 6 | /\xE2\x86\xA6/ {
1799 7 | gsub("\\xE2\\x86\\xA6", "\x09");
1801 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1802 TeXmacs back-tick handling is obscure, and a cut-n-paste back-tick from a shell window comes out as a unicode sequence2. that won't export to html, except as a NULL character (literal 0x00) ^2 that is fixed-up here.
1804 57b <recognize-chunk[2](
\v) ⇑57a, lang=> +≡ ▵57a 58a⊳
1805 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1807 10 | /\xE2\x80\x98/ {
1808 11 | gsub("\\xE2\\x80\\x98", "‘");
1810 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1811 In the TeXmacs output, the start of a chunk will appear like this:
1812 5b<example-chunk^K[1](arg1,^K arg2^K^K), lang=C> ≡
1813 We detect the the start of a TeXmacs chunk by detecting the ≡ symbol which occurs near the end of the line. We obtain the chunk name, the chunk parameters, and the chunk language.
1815 58a <recognize-chunk[3](
\v) ⇑57a, lang=> +≡ ⊲57b 58b▿
1816 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1818 14 | /\xE2\x89\xA1/ {
1819 15 | if (match($0, "^ *([^[ ]* |)<([^[ ]*)\\[[0-9]*\\][(](.*)[)].*, lang=([^ ]*)>", line)) {
1820 16 | next_chunk_name=line[2];
1821 17 | get_texmacs_chunk_args(line[3], next_chunk_params);
1822 18 | gsub(ARG_SEPARATOR ",? ?", ";", line[3]);
1823 19 | params = "params=" line[3];
1824 20 | if ((line[4])) {
1825 21 | params = params ",language=" line[4]
1827 23 | get_tex_chunk_args(params, next_chunk_opts);
1828 24 | new_chunk(next_chunk_name, next_chunk_opts, next_chunk_params);
1829 25 | texmacs_chunking = 1;
1831 27 | warning(sprintf("Unexpected chunk match: %s\n", $_))
1835 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1837 Our current scheme is to recognize the new lstlisting chunks, but these may be preceded by a \Chunk command which in L Y X is a more convenient way to pass the chunk name to the \begin{lstlistings} command, and a more visible way to specify other lstset settings.
1838 The arguments to the \Chunk command are a name, and then a comma-seperated list of key-value pairs after the manner of \lstset. (In fact within the LaTeX \Chunk macro (section 16.2.1) the text name= is prefixed to the argument which is then literally passed to \lstset).
1840 58b <recognize-chunk[4](
\v) ⇑57a, lang=awk> +≡ ▵58a 58c▿
1841 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1843 32 | if (match($0, "^\\\\Chunk{ *([^ ,}]*),?(.*)}", line)) {
1844 33 | next_chunk_name = line[1];
1845 34 | get_tex_chunk_args(line[2], next_chunk_opts);
1849 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1850 We also make a basic attempt to parse the name out of the \lstlistings[name=chunk-name] text, otherwise we fall back to the name found in the previous chunk command. This attempt is very basic and doesn't support commas or spaces or square brackets as part of the chunkname. We also recognize \begin{Chunk} which is convenient for some users3. but not yet supported in the LaTeX macros ^3.
1852 58c <recognize-chunk[5](
\v) ⇑57a, lang=> +≡ ▵58b 59a⊳
1853 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1854 38 | /^\\begin{lstlisting}|^\\begin{Chunk}/ {
1855 39 | if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
1856 40 | new_chunk(line[1]);
1858 42 | new_chunk(next_chunk_name, next_chunk_opts);
1863 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1866 A chunk body in TeXmacs ends with |________... if it is the final chunklet of a chunk, or if there are further chunklets it ends with |\/\/\/... which is a depiction of a jagged line of torn paper.
1868 59a <recognize-chunk[6](
\v) ⇑57a, lang=> +≡ ⊲58c 59b▿
1869 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1870 47 | /^ *\|____________*/ && texmacs_chunking {
1871 48 | active_chunk="";
1872 49 | texmacs_chunking=0;
1875 52 | /^ *\|\/\\/ && texmacs_chunking {
1876 53 | texmacs_chunking=0;
1878 55 | active_chunk="";
1880 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1881 It has been observed that not every line of output when a TeXmacs chunk is active is a line of chunk. This may no longer be true, but we set a variable texmacs_chunk if the current line is a chunk line.
1882 Initially we set this to zero...
1884 59b <recognize-chunk[7](
\v) ⇑57a, lang=> +≡ ▵59a 59c▿
1885 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1886 57 | texmacs_chunk=0;
1887 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1888 ...and then we look to see if the current line is a chunk line.
1889 TeXmacs lines look like this: 3 | main() { so we detect the lines by leading white space, digits, more whiter space and a vertical bar followed by at least once space.
1890 If we find such a line, we remove this line-header and set texmacs_chunk=1 as well as chunking=1
1892 59c <recognize-chunk[8](
\v) ⇑57a, lang=> +≡ ▵59b 59d▿
1893 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1894 58 | /^ *[1-9][0-9]* *\| / {
1895 59 | if (texmacs_chunking) {
1897 61 | texmacs_chunk=1;
1898 62 | gsub("^ *[1-9][0-9]* *\\| ", "")
1901 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1902 When TeXmacs chunking, lines that commence with \/ or __ are not chunk content but visual framing, and are skipped.
1904 59d <recognize-chunk[9](
\v) ⇑57a, lang=> +≡ ▵59c 60a⊳
1905 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1906 65 | /^ *\.\/\\/ && texmacs_chunking {
1909 68 | /^ *__*$/ && texmacs_chunking {
1912 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1913 Any other line when TeXmacs chunking is considered to be a line-wrapped line.
1915 60a <recognize-chunk[10](
\v) ⇑57a, lang=> +≡ ⊲59d 60b▿
1916 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1917 71 | texmacs_chunking {
1918 72 | if (! texmacs_chunk) {
1919 73 | # must be a texmacs continued line
1921 75 | texmacs_chunk=1;
1924 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1925 This final chunklet seems bogus and probably stops L Y X working.
1927 60b <recognize-chunk[11](
\v) ⇑57a, lang=> +≡ ▵60a 60c▿
1928 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1929 78 | ! texmacs_chunk {
1930 79 | # texmacs_chunking=0;
1933 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1935 We recognize notangle style chunks too:
1937 60c <recognize-chunk[12](
\v) ⇑57a, lang=awk> +≡ ▵60b 60d▿
1938 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1939 82 | /^[<]<.*[>]>=/ {
1940 83 | if (match($0, "^[<]<(.*)[>]>= *$", line)) {
1942 85 | notangle_mode=1;
1943 86 | new_chunk(line[1]);
1947 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1949 Likewise, we need to recognize when a chunk ends.
1951 The e in [e]nd{lislisting} is surrounded by square brackets so that when this document is processed, this chunk doesn't terminate early when the lstlistings package recognizes it's own end-string!4. This doesn't make sense as the regex is anchored with ^, which this line does not begin with! ^4
1953 60d <recognize-chunk[13](
\v) ⇑57a, lang=> +≡ ▵60c 61a⊳
1954 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1955 90 | /^\\[e]nd{lstlisting}|^\\[e]nd{Chunk}/ {
1957 92 | active_chunk="";
1960 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1963 61a <recognize-chunk[14](
\v) ⇑57a, lang=> +≡ ⊲60d 61b▿
1964 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1967 97 | active_chunk="";
1969 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1970 All other recognizers are only of effect if we are chunking; there's no point in looking at lines if they aren't part of a chunk, so we just ignore them as efficiently as we can.
1972 61b <recognize-chunk[15](
\v) ⇑57a, lang=> +≡ ▵61a 61c▿
1973 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1974 99 | ! chunking { next; }
1975 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1977 Chunk contents are any lines read while chunking is true. Some chunk contents are special in that they refer to other chunks, and will be replaced by the contents of these chunks when the file is generated.
1978 We add the output record separator ORS to the line now, because we will set ORS to the empty string when we generate the output5. So that we can partial print lines using print instead of printf.
1979 To do: This does't make sense
1982 61c <recognize-chunk[16](
\v) ⇑57a, lang=> +≡ ▵61b
1983 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1984 100 | length(active_chunk) {
1985 101 | «process-chunk-tabs 61e»
1986 102 | «process-chunk 62b»
1988 |________________________________________________________________________
1991 If a chunk just consisted of plain text, we could handle the chunk like this:
1993 61d <process-chunk-simple[1](
\v), lang=> ≡
1994 ________________________________________________________________________
1995 1 | chunk_line(active_chunk, $0 ORS);
1996 |________________________________________________________________________
1999 but in fact a chunk can include references to other chunks. Chunk includes are traditionally written as <<chunk-name>> but we support other variations, some of which are more suitable for particular editing systems.
2000 However, we also process tabs at this point. A tab at input can be replaced by a number of spaces defined by the tabs variable, set by the -T option. Of course this is poor tab behaviour, we should probably have the option to use proper counted tab-stops and process this on output.
2002 61e <process-chunk-tabs[1](
\v), lang=> ≡
2003 ________________________________________________________________________
2004 1 | if (length(tabs)) {
2005 2 | gsub("\t", tabs);
2007 |________________________________________________________________________
2011 If \lstset{escapeinside={=<}{>}} is set, then we can use <chunk-name ?> in listings. The sequence =< was chosen because:
2012 1.it is a better mnemonic than <<chunk-name>> in that the = sign signifies equivalence or substitutability.
2013 2.and because =< is not valid in C or any language I can think of.
2014 3.and also because lstlistings doesn't like >> as an end delimiter for the texcl escape, so we must make do with a single > which is better complemented by =< than by <<.
2015 Unfortunately the =<...> that we use re-enters a LaTeX parsing mode in which some characters are special, e.g. # \ and so these cause trouble if used in arguments to \chunkref. At some point I must fix the LaTeX command \chunkref so that it can accept these literally, but until then, when writing chunkref argumemts that need these characters, I must use the forms \textbackslash{} and \#; so I also define a hacky chunk delatex to be used further on whose purpose it is to remove these from any arguments parsed by fangle.
2017 62a <delatex[1](text
\v\v), lang=> ≡
2018 ________________________________________________________________________
2020 2 | gsub("\\\\#", "#", ${text});
2021 3 | gsub("\\\\textbackslash{}", "\\", ${text});
2022 4 | gsub("\\\\\\^", "^", ${text});
2023 |________________________________________________________________________
2026 As each chunk line may contain more than one chunk include, we will split out chunk includes in an iterative fashion6. Contrary to our use of split when substituting parameters in chapter ? ^6.
2027 First, as long as the chunk contains a \chunkref command we take as much as we can up to the first \chunkref command.
2028 TeXmacs text output uses ⟨...⟩ which comes out as unicode sequences 0xC2 0xAB ... 0xC2 0xBB. Modern awk will interpret [^\xC2\xBB] as a single unicode character if LANG is set correctly to the sub-type UTF-8, e.g. LANG=en_GB.UTF-8, otherwise [^\xC2\xBB] will be treated as a two character negated match — but this should not interfere with the function.
2030 62b <process-chunk[1](
\v), lang=> ≡ 62c▿
2031 ________________________________________________________________________
2034 3 | while(match(chunk,"(\xC2\xAB)([^\xC2\xBB]*) [^\xC2\xBB]*\xC2\xBB", line) ||
2036 5 | "([=]<\\\\chunkref{([^}>]*)}(\\(.*\\)|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)",
2039 8 | chunklet = substr(chunk, 1, RSTART - 1);
2040 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2041 We keep track of the indent count, by counting the number of literal characters found. We can then preserve this indent on each output line when multi-line chunks are expanded.
2042 We then process this first part literal text, and set the chunk which is still to be processed to be the text after the \chunkref command, which we will process next as we continue around the loop.
2044 62c <process-chunk[2](
\v) ⇑62b, lang=> +≡ ▵62b 63a⊳
2045 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2046 9 | indent += length(chunklet);
2047 10 | chunk_line(active_chunk, chunklet);
2048 11 | chunk = substr(chunk, RSTART + RLENGTH);
2049 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2050 We then consider the type of chunk command we have found, whether it is the fangle style command beginning with =< the older notangle style beginning with <<.
2051 Fangle chunks may have parameters contained within square brackets. These will be matched in line[3] and are considered at this stage of processing to be part of the name of the chunk to be included.
2053 63a <process-chunk[3](
\v) ⇑62b, lang=> +≡ ⊲62c 63b▿
2054 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2055 12 | if (substr(line[1], 1, 1) == "=") {
2056 13 | # chunk name up to }
2057 14 | «delatex
\v(line[3]
\v) 62a»
2058 15 | chunk_include(active_chunk, line[2] line[3], indent);
2059 16 | } else if (substr(line[1], 1, 1) == "<") {
2060 17 | chunk_include(active_chunk, line[4], indent);
2061 18 | } else if (line[1] == "\xC2\xAB") {
2062 19 | chunk_include(active_chunk, line[2], indent);
2064 21 | error("Unknown chunk fragment: " line[1]);
2066 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2067 The loop will continue until there are no more chunkref statements in the text, at which point we process the final part of the chunk.
2069 63b <process-chunk[4](
\v) ⇑62b, lang=> +≡ ▵63a 63c▿
2070 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2072 24 | chunk_line(active_chunk, chunk);
2073 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2074 We add the newline character as a chunklet on it's own, to make it easier to detect new lines and thus manage indentation when processing the output.
2076 63c <process-chunk[5](
\v) ⇑62b, lang=> +≡ ▵63b
2077 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2078 25 | chunk_line(active_chunk, "\n");
2079 |________________________________________________________________________
2082 We will also permit a chunk-part number to follow in square brackets, so that <chunk-name[1] ?> will refer to the first part only. This can make it easy to include a C function prototype in a header file, if the first part of the chunk is just the function prototype without the trailing semi-colon. The header file would include the prototype with the trailing semi-colon, like this:
2084 This is handled in section 13.1.1
2085 We should perhaps introduce a notion of language specific chunk options; so that perhaps we could specify:
2086 =<\chunkref{chunk-name[function-declaration]}
2087 which applies a transform function-declaration to the chunk --- which in this case would extract a function prototype from a function.
2090 Chapter 12Processing Options
2091 At the start, first we set the default options.
2093 65a <default-options[1](
\v), lang=> ≡
2094 ________________________________________________________________________
2097 3 | notangle_mode=0;
2100 |________________________________________________________________________
2103 Then we use getopt the standard way, and null out ARGV afterwards in the normal AWK fashion.
2105 65b <read-options[1](
\v), lang=> ≡
2106 ________________________________________________________________________
2107 1 | Optind = 1 # skip ARGV[0]
2108 2 | while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
2109 3 | «handle-options 65c»
2111 5 | for (i=1; i<Optind; i++) { ARGV[i]=""; }
2112 |________________________________________________________________________
2115 This is how we handle our options:
2117 65c <handle-options[1](
\v), lang=> ≡
2118 ________________________________________________________________________
2119 1 | if (Optopt == "R") root = Optarg;
2120 2 | else if (Optopt == "r") root="";
2121 3 | else if (Optopt == "L") linenos = 1;
2122 4 | else if (Optopt == "d") debug = 1;
2123 5 | else if (Optopt == "T") tabs = indent_string(Optarg+0);
2124 6 | else if (Optopt == "h") help();
2125 7 | else if (Optopt == "?") help();
2126 |________________________________________________________________________
2129 We do all of this at the beginning of the program
2131 65d <begin[1](
\v), lang=> ≡
2132 ________________________________________________________________________
2135 3 | «mode-definitions 48c»
2136 4 | «default-options 65a»
2138 6 | «read-options 65b»
2140 |________________________________________________________________________
2143 And have a simple help function
2145 65e <help()[1](
\v), lang=> ≡
2146 ________________________________________________________________________
2147 1 | function help() {
2149 3 | print " fangle [-L] -R<rootname> [source.tex ...]"
2150 4 | print " fangle -r [source.tex ...]"
2151 5 | print " If the filename, source.tex is not specified then stdin is used"
2153 7 | print "-L causes the C statement: #line <lineno> \"filename\"" to be issued"
2154 8 | print "-R causes the named root to be written to stdout"
2155 9 | print "-r lists all roots in the file (even those used elsewhere)"
2158 |________________________________________________________________________
2161 Chapter 13Generating the Output
2162 We generate output by calling output_chunk, or listing the chunk names.
2164 67a <generate-output[1](
\v), lang=> ≡
2165 ________________________________________________________________________
2166 1 | if (length(root)) output_chunk(root);
2167 2 | else output_chunk_names();
2168 |________________________________________________________________________
2171 We also have some other output debugging:
2173 67b <debug-output[1](
\v), lang=> ≡
2174 ________________________________________________________________________
2176 2 | print "------ chunk names "
2177 3 | output_chunk_names();
2178 4 | print "====== chunks"
2179 5 | output_chunks();
2180 6 | print "++++++ debug"
2181 7 | for (a in chunks) {
2182 8 | print a "=" chunks[a];
2185 |________________________________________________________________________
2188 We do both of these at the end. We also set ORS="" because each chunklet is not necessarily a complete line, and we already added ORS to each input line in section 11.4.
2190 67c <end[1](
\v), lang=> ≡
2191 ________________________________________________________________________
2193 2 | «debug-output 67b»
2195 4 | «generate-output 67a»
2197 |________________________________________________________________________
2200 We write chunk names like this. If we seem to be running in notangle compatibility mode, then we enclose the name like this <<name>> the same way notangle does:
2202 67d <output_chunk_names()[1](
\v), lang=> ≡
2203 ________________________________________________________________________
2204 1 | function output_chunk_names( c, prefix, suffix)
2206 3 | if (notangle_mode) {
2210 7 | for (c in chunk_names) {
2211 8 | print prefix c suffix "\n";
2214 |________________________________________________________________________
2217 This function would write out all chunks
2219 67e <output_chunks()[1](
\v), lang=> ≡
2220 ________________________________________________________________________
2221 1 | function output_chunks( a)
2223 3 | for (a in chunk_names) {
2224 4 | output_chunk(a);
2228 8 | function output_chunk(chunk) {
2230 10 | lineno_needed = linenos;
2232 12 | write_chunk(chunk);
2235 |________________________________________________________________________
2238 13.1 Assembling the Chunks
2239 chunk_path holds a string consisting of the names of all the chunks that resulted in this chunk being output. It should probably also contain the source line numbers at which each inclusion also occured.
2240 We first initialize the mode tracker for this chunk.
2242 68a <write_chunk()[1](
\v), lang=> ≡ 68b▿
2243 ________________________________________________________________________
2244 1 | function write_chunk(chunk_name) {
2245 2 | «awk-delete-array
\v(context
\v) 33d»
2246 3 | return write_chunk_r(chunk_name, context);
2249 6 | function write_chunk_r(chunk_name, context, indent, tail,
2251 8 | chunk_path, chunk_args,
2252 9 | s, r, src, new_src,
2254 11 | chunk_params, part, max_part, part_line, frag, max_frag, text,
2255 12 | chunklet, only_part, call_chunk_args, new_context)
2257 14 | if (debug) debug_log("write_chunk_r(" chunk_name ")");
2258 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2260 As mentioned in section ?, a chunk name may contain a part specifier in square brackets, limiting the parts that should be emitted.
2262 68b <write_chunk()[2](
\v) ⇑68a, lang=> +≡ ▵68a 68c▿
2263 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2264 15 | if (match(chunk_name, "^(.*)\\[([0-9]*)\\]$", chunk_name_parts)) {
2265 16 | chunk_name = chunk_name_parts[1];
2266 17 | only_part = chunk_name_parts[2];
2268 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2269 We then create a mode tracker
2271 68c <write_chunk()[3](
\v) ⇑68a, lang=> +≡ ▵68b 69a⊳
2272 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2273 19 | «new-mode-tracker
\v(context
\v, chunks[chunk_name, "language"]
\v, ""
\v) 50b»
2274 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2275 We extract into chunk_params the names of the parameters that this chunk accepts, whose values were (optionally) passed in chunk_args.
2277 69a <write_chunk()[4](
\v) ⇑68a, lang=> +≡ ⊲68c 69b▿
2278 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2279 20 | split(chunks[chunk_name, "params"], chunk_params, " *; *");
2280 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2281 To assemble a chunk, we write out each part.
2283 69b <write_chunk()[5](
\v) ⇑68a, lang=> +≡ ▵69a
2284 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2285 21 | if (! (chunk_name in chunk_names)) {
2286 22 | error(sprintf(_"The root module <<%s>> was not defined.\nUsed by: %s",\
2287 23 | chunk_name, chunk_path));
2290 26 | max_part = chunks[chunk_name, "part"];
2291 27 | for(part = 1; part <= max_part; part++) {
2292 28 | if (! only_part || part == only_part) {
2293 29 | «write-part 69c»
2296 32 | if (! finalize_mode_tracker(context)) {
2297 33 | dump_mode_tracker(context);
2298 34 | error(sprintf(_"Module %s did not close context properly.\nUsed by: %s\n", chunk_name, chunk_path));
2301 |________________________________________________________________________
2304 A part can either be a chunklet of lines, or an include of another chunk.
2305 Chunks may also have parameters, specified in LaTeX style with braces after the chunk name --- looking like this in the document: chunkname{param1, param2}. Arguments are passed in square brackets: \chunkref{chunkname}[arg1, arg2].
2306 Before we process each part, we check that the source position hasn't changed unexpectedly, so that we can know if we need to output a new file-line directive.
2308 69c <write-part[1](
\v), lang=> ≡
2309 ________________________________________________________________________
2310 1 | «check-source-jump 71d»
2312 3 | chunklet = chunks[chunk_name, "part", part];
2313 4 | if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
2314 5 | «write-included-chunk 69d»
2315 6 | } else if (chunklet SUBSEP "line" in chunks) {
2316 7 | «write-chunklets 70a»
2318 9 | # empty last chunklet
2320 |________________________________________________________________________
2323 To write an included chunk, we must detect any optional chunk arguments in parenthesis. Then we recurse calling write_chunk().
2325 69d <write-included-chunk[1](
\v), lang=> ≡
2326 ________________________________________________________________________
2327 1 | if (match(chunklet, "^([^\\[\\(]*)\\((.*)\\)$", chunklet_parts)) {
2328 2 | chunklet = chunklet_parts[1];
2330 4 | gsub(sprintf("%c",11), "", chunklet);
2331 5 | gsub(sprintf("%c",11), "", chunklet_parts[2]);
2332 6 | parse_chunk_args("c-like", chunklet_parts[2], call_chunk_args, "(");
2333 7 | for (c in call_chunk_args) {
2334 8 | call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params, chunk_args);
2337 11 | split("", call_chunk_args);
2339 13 | # update the transforms arrays
2340 14 | new_src = mode_escaper(context, s, r, src);
2341 15 | «awk-delete-array
\v(new_context
\v) 33d»
2342 16 | write_chunk_r(chunklet, new_context,
2343 17 | chunks[chunk_name, "part", part, "indent"] indent,
2344 18 | chunks[chunk_name, "part", part, "tail"],
2345 19 | chunk_path "\n " chunk_name,
2346 20 | call_chunk_args,
2347 21 | s, r, new_src);
2348 |________________________________________________________________________
2351 Before we output a chunklet of lines, we first emit the file and line number if we have one, and if it is safe to do so.
2352 Chunklets are generally broken up by includes, so the start of a chunklet is a good place to do this. Then we output each line of the chunklet.
2353 When it is not safe, such as in the middle of a multi-line macro definition, lineno_suppressed is set to true, and in such a case we note that we want to emit the line statement when it is next safe.
2355 70a <write-chunklets[1](
\v), lang=> ≡ 70b▿
2356 ________________________________________________________________________
2357 1 | max_frag = chunks[chunklet, "line"];
2358 2 | for(frag = 1; frag <= max_frag; frag++) {
2359 3 | «write-file-line 71c»
2360 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2361 We then extract the chunklet text and expand any arguments.
2363 70b <write-chunklets[2](
\v) ⇑70a, lang=> +≡ ▵70a 70c▿
2364 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2366 5 | text = chunks[chunklet, frag];
2368 7 | /* check params */
2369 8 | text = expand_chunk_args(text, chunk_params, chunk_args);
2370 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2371 If the text is a single newline (which we keep separate - see 6) then we increment the line number. In the case where this is the last line of a chunk and it is not a top-level chunk we replace the newline with an empty string --- because the chunk that included this chunk will have the newline at the end of the line that included this chunk.
2372 We also note by newline = 1 that we have started a new line, so that indentation can be managed with the following piece of text.
2374 70c <write-chunklets[3](
\v) ⇑70a, lang=> +≡ ▵70b 70d▿
2375 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2377 10 | if (text == "\n") {
2379 12 | if (part == max_part && frag == max_frag && length(chunk_path)) {
2385 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2386 If this text does not represent a newline, but we see that we are the first piece of text on a newline, then we prefix our text with the current indent.
2387 Note 1. newline is a global output-state variable, but the indent is not.
2389 70d <write-chunklets[4](
\v) ⇑70a, lang=> +≡ ▵70c 71a⊳
2390 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2391 18 | } else if (length(text) || length(tail)) {
2392 19 | if (newline) text = indent text;
2396 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2397 Tail will soon no longer be relevant once mode-detection is in place.
2399 71a <write-chunklets[5](
\v) ⇑70a, lang=> +≡ ⊲70d 71b▿
2400 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2401 23 | text = text tail;
2402 24 | mode_tracker(context, text);
2403 25 | print untab(transform_escape(s, r, text, src));
2404 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2405 If a line ends in a backslash --- suggesting continuation --- then we supress outputting file-line as it would probably break the continued lines.
2407 71b <write-chunklets[6](
\v) ⇑70a, lang=> +≡ ▵71a
2408 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2410 27 | lineno_suppressed = substr(lastline, length(lastline)) == "\\";
2413 |________________________________________________________________________
2416 Of course there is no point in actually outputting the source filename and line number (file-line) if they don't say anything new! We only need to emit them if they aren't what is expected, or if we we not able to emit one when they had changed.
2418 71c <write-file-line[1](
\v), lang=> ≡
2419 ________________________________________________________________________
2420 1 | if (newline && lineno_needed && ! lineno_suppressed) {
2421 2 | filename = a_filename;
2422 3 | lineno = a_lineno;
2423 4 | print "#line " lineno " \"" filename "\"\n"
2424 5 | lineno_needed = 0;
2426 |________________________________________________________________________
2429 We check if a new file-line is needed by checking if the source line matches what we (or a compiler) would expect.
2431 71d <check-source-jump[1](
\v), lang=> ≡
2432 ________________________________________________________________________
2433 1 | if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in chunks)) {
2434 2 | a_filename = chunks[chunk_name, "part", part, "FILENAME"];
2435 3 | a_lineno = chunks[chunk_name, "part", part, "LINENO"];
2436 4 | if (a_filename != filename || a_lineno != lineno) {
2437 5 | lineno_needed++;
2440 |________________________________________________________________________
2443 Chapter 14Storing Chunks
2444 Awk has pretty limited data structures, so we will use two main hashes. Uninterrupted sequences of a chunk will be stored in chunklets and the chunklets used in a chunk will be stored in chunks.
2446 73a <constants[2](
\v) ⇑35a, lang=> +≡ ⊲35a
2447 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2448 2 | part_type_chunk=1;
2450 |________________________________________________________________________
2453 The params mentioned are not chunk parameters for parameterized chunks, as mentioned in 9.2, but the lstlistings style parameters used in the \Chunk command1. The params parameter is used to hold the parameters for parameterized chunks ^1.
2455 73b <chunk-storage-functions[1](
\v), lang=> ≡ 73c▿
2456 ________________________________________________________________________
2457 1 | function new_chunk(chunk_name, opts, args,
2461 5 | # HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
2462 6 | gsub("\\(\\)$", "", chunk_name);
2463 7 | if (! (chunk_name in chunk_names)) {
2464 8 | if (debug) print "New chunk " chunk_name;
2465 9 | chunk_names[chunk_name];
2466 10 | for (p in opts) {
2467 11 | chunks[chunk_name, p] = opts[p];
2468 12 | if (debug) print "chunks[" chunk_name "," p "] = " opts[p];
2470 14 | for (p in args) {
2471 15 | chunks[chunk_name, "params", p] = args[p];
2473 17 | if ("append" in opts) {
2474 18 | append=opts["append"];
2475 19 | if (! (append in chunk_names)) {
2476 20 | warning("Chunk " chunk_name " is appended to chunk " append " which is not defined yet");
2477 21 | new_chunk(append);
2479 23 | chunk_include(append, chunk_name);
2480 24 | chunk_line(append, ORS);
2483 27 | active_chunk = chunk_name;
2484 28 | prime_chunk(chunk_name);
2486 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2488 73c <chunk-storage-functions[2](
\v) ⇑73b, lang=> +≡ ▵73b 74a⊳
2489 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2491 31 | function prime_chunk(chunk_name)
2493 33 | chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = \
2494 34 | chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"];
2495 35 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
2496 36 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR + 1;
2499 39 | function chunk_line(chunk_name, line){
2500 40 | chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
2501 41 | ++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"], "line"] ] = line;
2504 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2505 Chunk include represents a chunkref statement, and stores the requirement to include another chunk. The parameter indent represents the quanity of literal text characters that preceded this chunkref statement and therefore by how much additional lines of the included chunk should be indented.
2507 74a <chunk-storage-functions[3](
\v) ⇑73b, lang=> +≡ ⊲73c 74b▿
2508 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2509 44 | function chunk_include(chunk_name, chunk_ref, indent, tail)
2511 46 | chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
2512 47 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_chunk;
2513 48 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_string(indent);
2514 49 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
2515 50 | prime_chunk(chunk_name);
2518 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2519 The indent is calculated by indent_string, which may in future convert some spaces into tab characters. This function works by generating a printf padded format string, like %22s for an indent of 22, and then printing an empty string using that format.
2521 74b <chunk-storage-functions[4](
\v) ⇑73b, lang=> +≡ ▵74a
2522 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2523 53 | function indent_string(indent) {
2524 54 | return sprintf("%" indent "s", "");
2526 |________________________________________________________________________
2530 I use Arnold Robbins public domain getopt (1993 revision). This is probably the same one that is covered in chapter 12 of âĂIJEdition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU AwkâĂİ but as that is licensed under the GNU Free Documentation License, Version 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompanying explanations), so I do my best to explain how it works here.
2531 The getopt.awk header is:
2533 75a <getopt.awk-header[1](
\v), lang=> ≡
2534 ________________________________________________________________________
2535 1 | # getopt.awk --- do C library getopt(3) function in awk
2537 3 | # Arnold Robbins, arnold@skeeve.com, Public Domain
2539 5 | # Initial version: March, 1991
2540 6 | # Revised: May, 1993
2542 |________________________________________________________________________
2545 The provided explanation is:
2547 75b <getopt.awk-notes[1](
\v), lang=> ≡
2548 ________________________________________________________________________
2549 1 | # External variables:
2550 2 | # Optind -- index in ARGV of first nonoption argument
2551 3 | # Optarg -- string value of argument to current option
2552 4 | # Opterr -- if nonzero, print our own diagnostic
2553 5 | # Optopt -- current option letter
2556 8 | # -1 at end of options
2557 9 | # ? for unrecognized option
2558 10 | # <c> a character representing the current option
2560 12 | # Private Data:
2561 13 | # _opti -- index in multi-flag option, e.g., -abc
2563 |________________________________________________________________________
2566 The function follows. The final two parameters, thisopt and i are local variables and not parameters --- as indicated by the multiple spaces preceding them. Awk doesn't care, the multiple spaces are a convention to help us humans.
2568 75c <getopt.awk-getopt()[1](
\v), lang=> ≡ 76a⊳
2569 ________________________________________________________________________
2570 1 | function getopt(argc, argv, options, thisopt, i)
2572 3 | if (length(options) == 0) # no options given
2574 5 | if (argv[Optind] == "--") { # all done
2578 9 | } else if (argv[Optind] !~ /^-[^: \t\n\f\r\v\b]/) {
2582 13 | if (_opti == 0)
2584 15 | thisopt = substr(argv[Optind], _opti, 1)
2585 16 | Optopt = thisopt
2586 17 | i = index(options, thisopt)
2589 20 | printf("%c -- invalid option\n",
2590 21 | thisopt) > "/dev/stderr"
2591 22 | if (_opti >= length(argv[Optind])) {
2598 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2599 At this point, the option has been found and we need to know if it takes any arguments.
2601 76a <getopt.awk-getopt()[2](
\v) ⇑75c, lang=> +≡ ⊲75c
2602 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2603 29 | if (substr(options, i + 1, 1) == ":") {
2604 30 | # get option argument
2605 31 | if (length(substr(argv[Optind], _opti + 1)) > 0)
2606 32 | Optarg = substr(argv[Optind], _opti + 1)
2608 34 | Optarg = argv[++Optind]
2612 38 | if (_opti == 0 || _opti >= length(argv[Optind])) {
2619 |________________________________________________________________________
2622 A test program is built in, too
2624 76b <getopt.awk-begin[1](
\v), lang=> ≡
2625 ________________________________________________________________________
2627 2 | Opterr = 1 # default is to diagnose
2628 3 | Optind = 1 # skip ARGV[0]
2630 5 | if (_getopt_test) {
2631 6 | while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
2632 7 | printf("c = <%c>, optarg = <%s>\n",
2634 9 | printf("non-option arguments:\n")
2635 10 | for (; Optind < ARGC; Optind++)
2636 11 | printf("\tARGV[%d] = <%s>\n",
2637 12 | Optind, ARGV[Optind])
2640 |________________________________________________________________________
2643 The entire getopt.awk is made out of these chunks in order
2645 76c <getopt.awk[1](
\v), lang=> ≡
2646 ________________________________________________________________________
2647 1 | «getopt.awk-header 75a»
2649 3 | «getopt.awk-notes 75b»
2650 4 | «getopt.awk-getopt() 75c»
2651 5 | «getopt.awk-begin 76b»
2652 |________________________________________________________________________
2655 Although we only want the header and function:
2657 77a <getopt[1](
\v), lang=> ≡
2658 ________________________________________________________________________
2659 1 | # try: locate getopt.awk for the full original file
2660 2 | # as part of your standard awk installation
2661 3 | «getopt.awk-header 75a»
2663 5 | «getopt.awk-getopt() 75c»
2664 |________________________________________________________________________
2667 Chapter 16Fangle LaTeX source code
2669 Here we define a L Y X .module file that makes it convenient to use L Y X for writing such literate programs.
2670 This file ./fangle.module can be installed in your personal .lyx/layouts folder. You will need to Tools Reconfigure so that L Y X notices it. It adds a new format Chunk, which should precede every listing and contain the chunk name.
2672 79a <./fangle.module[1](
\v), lang=lyx-module> ≡
2673 ________________________________________________________________________
2674 1 | #\DeclareLyXModule{Fangle Literate Listings}
2675 2 | #DescriptionBegin
2676 3 | # Fangle literate listings allow one to write
2677 4 | # literate programs after the fashion of noweb, but without having
2678 5 | # to use noweave to generate the documentation. Instead the listings
2679 6 | # package is extended in conjunction with the noweb package to implement
2680 7 | # to code formating directly as latex.
2681 8 | # The fangle awk script
2684 11 | «gpl3-copyright.hashed 79b»
2689 16 | «./fangle.sty 80d»
2692 19 | «chunkstyle 80a»
2695 |________________________________________________________________________
2698 Because L Y X modules are not yet a language supported by fangle or lstlistings, we resort to this fake awk chunk below in order to have each line of the GPL3 license commence with a #
2700 79b <gpl3-copyright.hashed[1](
\v), lang=awk> ≡
2701 ________________________________________________________________________
2702 1 | #«gpl3-copyright 4a»
2704 |________________________________________________________________________
2707 16.1.1 The Chunk style
2708 The purpose of the chunk style is to make it easier for L Y X users to provide the name to lstlistings. Normally this requires right-clicking on the listing, choosing settings, advanced, and then typing name=chunk-name. This has the further disadvantage that the name (and other options) are not generally visible during document editing.
2709 The chunk style is defined as a LaTeX command, so that all text on the same line is passed to the LaTeX command Chunk. This makes it easy to parse using fangle, and easy to pass these options on to the listings package. The first word in a chunk section should be the chunk name, and will have name= prepended to it. Any other words are accepted arguments to lstset.
2710 We set PassThru to 1 because the user is actually entering raw latex.
2712 80a <chunkstyle[1](
\v), lang=> ≡ 80b▿
2713 ________________________________________________________________________
2715 2 | LatexType Command
2717 4 | Margin First_Dynamic
2718 5 | LeftMargin Chunk:xxx
2720 7 | LabelType Static
2721 8 | LabelString "Chunk:"
2725 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2726 To make the label very visible we choose a larger font coloured red.
2728 80b <chunkstyle[2](
\v) ⇑80a, lang=> +≡ ▵80a
2729 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2738 |________________________________________________________________________
2741 16.1.2 The chunkref style
2742 We also define the Chunkref style which can be used to express cross references to chunks.
2744 80c <chunkref[1](
\v), lang=> ≡
2745 ________________________________________________________________________
2746 1 | InsetLayout Chunkref
2747 2 | LyxType charstyle
2748 3 | LatexType Command
2749 4 | LatexName chunkref
2756 |________________________________________________________________________
2760 We require the listings, noweb and xargs packages. As noweb defines it's own \code environment, we re-define the one that L Y X logical markup module expects here.
2762 80d <./fangle.sty[1](
\v), lang=tex> ≡ 81a⊳
2763 ________________________________________________________________________
2764 1 | \usepackage{listings}%
2765 2 | \usepackage{noweb}%
2766 3 | \usepackage{xargs}%
2767 4 | \renewcommand{\code}[1]{\texttt{#1}}%
2768 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2769 We also define a CChunk macro, for use as: \begin{CChunk} which will need renaming to \begin{Chunk} when I can do this without clashing with \Chunk.
2771 81a <./fangle.sty[2](
\v) ⇑80d, lang=> +≡ ⊲80d 81b▿
2772 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2773 5 | \lstnewenvironment{Chunk}{\relax}{\relax}%
2774 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2775 We also define a suitable \lstset of parameters that suit the literate programming style after the fashion of noweave.
2777 81b <./fangle.sty[3](
\v) ⇑80d, lang=> +≡ ▵81a 81c▿
2778 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2779 6 | \lstset{numbers=left, stepnumber=5, numbersep=5pt,
2780 7 | breaklines=false,basicstyle=\ttfamily,
2781 8 | numberstyle=\tiny, language=C}%
2782 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2783 We also define a notangle-like mechanism for escaping to LaTeX from the listing, and by which we can refer to other listings. We declare the =<...> sequence to contain LaTeX code, and include another like this chunk: <chunkname ?>. However, because =<...> is already defined to contain LaTeX code for this document --- this is a fangle document after all --- the code fragment below effectively contains the LaTeX code: }{. To avoid problems with document generation, I had to declare an lstlistings property: escapeinside={} for this listing only; which in L Y X was done by right-clicking the listings inset, choosing settings->advanced. Therefore =< isn't interpreted literally here, in a listing when the escape sequence is already defined as shown... we need to somehow escape this representation...
2785 81c <./fangle.sty[4](
\v) ⇑80d, lang=> +≡ ▵81b 81d▿
2786 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2787 9 | \lstset{escapeinside={=<}{>}}%
2788 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2789 Although our macros will contain the @ symbol, they will be included in a \makeatletter section by L Y X; however we keep the commented out \makeatletter as a reminder. The listings package likes to centre the titles, but noweb titles are specially formatted and must be left aligned. The simplest way to do this turned out to be by removing the definition of \lst@maketitle. This may interact badly if other listings want a regular title or caption. We remember the old maketitle in case we need it.
2791 81d <./fangle.sty[5](
\v) ⇑80d, lang=> +≡ ▵81c 81e▿
2792 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2794 11 | %somehow re-defining maketitle gives us a left-aligned title
2795 12 | %which is extactly what our specially formatted title needs!
2796 13 | \global\let\fangle@lst@maketitle\lst@maketitle%
2797 14 | \global\def\lst@maketitle{}%
2798 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2799 16.2.1 The chunk command
2800 Our chunk command accepts one argument, and calls \ltset. Although \ltset will note the name, this is erased when the next \lstlisting starts, so we make a note of this in \lst@chunkname and restore in in lstlistings Init hook.
2802 81e <./fangle.sty[6](
\v) ⇑80d, lang=> +≡ ▵81d 82a⊳
2803 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2805 16 | \lstset{title={\fanglecaption},name=#1}%
2806 17 | \global\edef\lst@chunkname{\lst@intname}%
2808 19 | \def\lst@chunkname{\empty}%
2809 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2810 16.2.1.1 Chunk parameters
2811 Fangle permits parameterized chunks, and requires the paramters to be specified as listings options. The fangle script uses this, and although we don't do anything with these in the LaTeX code right now, we need to stop the listings package complaining.
2813 82a <./fangle.sty[7](
\v) ⇑80d, lang=> +≡ ⊲81e 82b▿
2814 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2815 20 | \lst@Key{params}\relax{\def\fangle@chunk@params{#1}}%
2816 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2817 As it is common to define a chunk which then needs appending to another chunk, and annoying to have to declare a single line chunk to manage the include, we support an append= option.
2819 82b <./fangle.sty[8](
\v) ⇑80d, lang=> +≡ ▵82a 82c▿
2820 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2821 21 | \lst@Key{append}\relax{\def\fangle@chunk@append{#1}}%
2822 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2823 16.2.2 The noweb styled caption
2824 We define a public macro \fanglecaption which can be set as a regular title. By means of \protect, It expands to \fangle@caption at the appopriate time when the caption is emitted.
2826 82c <./fangle.sty[9](
\v) ⇑80d, lang=> +≡ ▵82b 82d▿
2827 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2828 \def\fanglecaption{\protect\fangle@caption}%
2829 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2830 22c ⟨some-chunk 19b⟩≡+ ⊲22b 24d⊳
2832 In this example, the current chunk is 22c, and therefore the third chunk on page 22.
2833 It's name is some-chunk.
2834 The first chunk with this name (19b) occurs as the second chunk on page 19.
2835 The previous chunk (22d) with the same name is the second chunk on page 22.
2836 The next chunk (24d) is the fourth chunk on page 24.
2838 Figure 1. Noweb Heading
2840 The general noweb output format compactly identifies the current chunk, and references to the first chunk, and the previous and next chunks that have the same name.
2841 This means that we need to keep a counter for each chunk-name, that we use to count chunks of the same name.
2842 16.2.3 The chunk counter
2843 It would be natural to have a counter for each chunk name, but TeX would soon run out of counters1. ...soon did run out of counters and so I had to re-write the LaTeX macros to share a counter as described here. ^1, so we have one counter which we save at the end of a chunk and restore at the beginning of a chunk.
2845 82d <./fangle.sty[10](
\v) ⇑80d, lang=> +≡ ▵82c 83c⊳
2846 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2847 22 | \newcounter{fangle@chunkcounter}%
2848 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2849 We construct the name of this variable to store the counter to be the text lst-chunk- prefixed onto the chunks own name, and store it in \chunkcount.
2850 We save the counter like this:
2852 83a <save-counter[1](
\v), lang=> ≡
2853 ________________________________________________________________________
2854 \global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
2855 |________________________________________________________________________
2858 and restore the counter like this:
2860 83b <restore-counter[1](
\v), lang=> ≡
2861 ________________________________________________________________________
2862 \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
2863 |________________________________________________________________________
2866 If there does not already exist a variable whose name is stored in \chunkcount, then we know we are the first chunk with this name, and then define a counter.
2867 Although chunks of the same name share a common counter, they must still be distinguished. We use is the internal name of the listing, suffixed by the counter value. So the first chunk might be something-1 and the second chunk be something-2, etc.
2868 We also calculate the name of the previous chunk if we can (before we increment the chunk counter). If this is the first chunk of that name, then \prevchunkname is set to \relax which the noweb package will interpret as not existing.
2870 83c <./fangle.sty[11](
\v) ⇑80d, lang=> +≡ ⊲82d 83d▿
2871 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2872 23 | \def\fangle@caption{%
2873 24 | \edef\chunkcount{lst-chunk-\lst@intname}%
2874 25 | \@ifundefined{\chunkcount}{%
2875 26 | \expandafter\gdef\csname \chunkcount\endcsname{0}%
2876 27 | \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
2877 28 | \let\prevchunkname\relax%
2879 30 | \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
2880 31 | \edef\prevchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
2882 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2883 After incrementing the chunk counter, we then define the name of this chunk, as well as the name of the first chunk.
2885 83d <./fangle.sty[12](
\v) ⇑80d, lang=> +≡ ▵83c 83e▿
2886 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2887 33 | \addtocounter{fangle@chunkcounter}{1}%
2888 34 | \global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
2889 35 | \edef\chunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
2890 36 | \edef\firstchunkname{\lst@intname-1}%
2891 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2892 We now need to calculate the name of the next chunk. We do this by temporarily skipping the counter on by one; however there may not actually be another chunk with this name! We detect this by also defining a label for each chunk based on the chunkname. If there is a next chunkname then it will define a label with that name. As labels are persistent, we can at least tell the second time LaTeX is run. If we don't find such a defined label then we define \nextchunkname to \relax.
2894 83e <./fangle.sty[13](
\v) ⇑80d, lang=> +≡ ▵83d 84a⊳
2895 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2896 37 | \addtocounter{fangle@chunkcounter}{1}%
2897 38 | \edef\nextchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
2898 39 | \@ifundefined{r@label-\nextchunkname}{\let\nextchunkname\relax}{}%
2899 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2900 The noweb package requires that we define a \sublabel for every chunk, with a unique name, which is then used to print out it's navigation hints.
2901 We also define a regular label for this chunk, as was mentioned above when we calculated \nextchunkname. This requires LaTeX to be run at least twice after new chunk sections are added --- but noweb requried that anyway.
2903 84a <./fangle.sty[14](
\v) ⇑80d, lang=> +≡ ⊲83e 84b▿
2904 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2905 40 | \sublabel{\chunkname}%
2906 41 | % define this label for every chunk instance, so we
2907 42 | % can tell when we are the last chunk of this name
2908 43 | \label{label-\chunkname}%
2909 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2910 We also try and add the chunk to the list of listings, but I'm afraid we don't do very well. We want each chunk name listing once, with all of it's references.
2912 84b <./fangle.sty[15](
\v) ⇑80d, lang=> +≡ ▵84a 84c▿
2913 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2914 44 | \addcontentsline{lol}{lstlisting}{\lst@name~[\protect\subpageref{\chunkname}]}%
2915 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2916 We then call the noweb output macros in the same way that noweave generates them, except that we don't need to call \nwstartdeflinemarkup or \nwenddeflinemarkup — and if we do, it messes up the output somewhat.
2918 84c <./fangle.sty[16](
\v) ⇑80d, lang=> +≡ ▵84b 84d▿
2919 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2923 48 | \subpageref{\chunkname}%
2930 55 | \nwtagstyle{}\/%
2931 56 | \@ifundefined{fangle@chunk@params}{}{%
2932 57 | (\fangle@chunk@params)%
2934 59 | [\csname \chunkcount\endcsname]~%
2935 60 | \subpageref{\firstchunkname}%
2937 62 | \@ifundefined{fangle@chunk@append}{}{%
2938 63 | \ifx{}\fangle@chunk@append{x}\else%
2939 64 | ,~add~to~\fangle@chunk@append%
2942 67 | \global\def\fangle@chunk@append{}%
2943 68 | \lstset{append=x}%
2946 71 | \ifx\relax\prevchunkname\endmoddef\else\plusendmoddef\fi%
2947 72 | % \nwstartdeflinemarkup%
2948 73 | \nwprevnextdefs{\prevchunkname}{\nextchunkname}%
2949 74 | % \nwenddeflinemarkup%
2951 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2952 Originally this was developed as a listings aspect, in the Init hook, but it was found easier to affect the title without using a hook — \lst@AddToHookExe{PreSet} is still required to set the listings name to the name passed to the \Chunk command, though.
2954 84d <./fangle.sty[17](
\v) ⇑80d, lang=> +≡ ▵84c 85a⊳
2955 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2956 76 | %\lst@BeginAspect{fangle}
2957 77 | %\lst@Key{fangle}{true}[t]{\lstKV@SetIf{#1}{true}}
2958 78 | \lst@AddToHookExe{PreSet}{\global\let\lst@intname\lst@chunkname}
2959 79 | \lst@AddToHook{Init}{}%\fangle@caption}
2960 80 | %\lst@EndAspect
2961 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2962 16.2.4 Cross references
2963 We define the \chunkref command which makes it easy to generate visual references to different code chunks, e.g.
2966 \chunkref[3]{preamble}
2967 \chunkref{preamble}[arg1, arg2]
2969 Chunkref can also be used within a code chunk to include another code chunk. The third optional parameter to chunkref is a comma sepatarated list of arguments, which will replace defined parameters in the chunkref.
2970 Note 1. Darn it, if I have: =<\chunkref{new-mode-tracker}[{chunks[chunk_name, "language"]},{mode}]> the inner braces (inside [ ]) cause _ to signify subscript even though we have lst@ReplaceIn
2972 85a <./fangle.sty[18](
\v) ⇑80d, lang=> +≡ ⊲84d 86a⊳
2973 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2974 81 | \def\chunkref@args#1,{%
2976 83 | \lst@ReplaceIn\arg\lst@filenamerpl%
2978 85 | \@ifnextchar){\relax}{, \chunkref@args}%
2980 87 | \newcommand\chunkref[2][0]{%
2981 88 | \@ifnextchar({\chunkref@i{#1}{#2}}{\chunkref@i{#1}{#2}()}%
2983 90 | \def\chunkref@i#1#2(#3){%
2985 92 | \def\chunk{#2}%
2986 93 | \def\chunkno{#1}%
2987 94 | \def\chunkargs{#3}%
2988 95 | \ifx\chunkno\zero%
2989 96 | \def\chunkname{#2-1}%
2991 98 | \def\chunkname{#2-\chunkno}%
2993 100 | \let\lst@arg\chunk%
2994 101 | \lst@ReplaceIn\chunk\lst@filenamerpl%
2995 102 | \LA{%\moddef{%
2998 105 | \nwtagstyle{}\/%
2999 106 | \ifx\chunkno\zero%
3003 110 | \ifx\chunkargs\empty%
3005 112 | (\chunkref@args #3,)%
3007 114 | ~\subpageref{\chunkname}%
3010 117 | \RA%\endmoddef%
3012 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3015 86a <./fangle.sty[19](
\v) ⇑80d, lang=> +≡ ⊲85a
3016 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3019 |________________________________________________________________________
3022 Chapter 17Extracting fangle
3023 17.1 Extracting from Lyx
3024 To extract from L Y X, you will need to configure L Y X as explained in section ?.
3025 And this lyx-build scrap will extract fangle for me.
3027 87a <lyx-build[2](
\v) ⇑20a, lang=sh> +≡ ⊲20a
3028 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3032 14 | «lyx-build-helper 19b»
3033 15 | cd $PROJECT_DIR || exit 1
3035 17 | /usr/local/bin/fangle -R./fangle $TEX_SRC > ./fangle
3036 18 | /usr/local/bin/fangle -R./fangle.module $TEX_SRC > ./fangle.module
3038 20 | «test:helpers 88b»
3039 21 | export FANGLE=./fangle
3040 22 | export TMP=${TMP:-/tmp}
3041 23 | «test:run-tests 87b»
3042 24 | # Now check that we can extract a fangle that also passes the tests!
3043 25 | $FANGLE -R./fangle $TEX_SRC > ./new-fangle
3044 26 | export FANGLE=./new-fangle
3045 27 | «test:run-tests 87b»
3046 |________________________________________________________________________
3050 87b <test:run-tests[1](
\v), lang=sh> ≡
3051 ________________________________________________________________________
3053 2 | $FANGLE -Rpca-test.awk $TEX_SRC | awk -f - || exit 1
3054 3 | «test:cromulence 51e»
3055 4 | «test:escapes 55d»
3056 5 | «test:chunk-params 91e»
3057 |________________________________________________________________________
3060 With a lyx-build-helper
3062 87c <lyx-build-helper[2](
\v) ⇑19b, lang=sh> +≡ ⊲19b
3063 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3064 5 | PROJECT_DIR="$LYX_r"
3065 6 | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
3066 7 | TEX_DIR="$LYX_p"
3067 8 | TEX_SRC="$TEX_DIR/$LYX_i"
3068 |________________________________________________________________________
3071 17.2 Extracting documentation
3073 87d <./gen-www[1](
\v), lang=> ≡
3074 ________________________________________________________________________
3075 1 | #python -m elyxer --css lyx.css $LYX_SRC | \
3076 2 | # iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT | \
3077 3 | # sed 's/UTF-8"\(.\)>/ISO-8859-1"\1>/' > www/docs/fangle.html
3079 5 | python -m elyxer --css lyx.css --iso885915 --html --destdirectory www/docs/fangle.e \
3080 6 | fangle.lyx > www/docs/fangle.e/fangle.html
3082 8 | ( mkdir -p www/docs/fangle && cd www/docs/fangle && \
3083 9 | lyx -e latex ../../../fangle.lyx && \
3084 10 | htlatex ../../../fangle.tex "xhtml,fn-in" && \
3085 11 | sed -i -e 's/<!--l\. [0-9][0-9]* *-->//g' fangle.html
3088 14 | ( mkdir -p www/docs/literate && cd www/docs/literate && \
3089 15 | lyx -e latex ../../../literate.lyx && \
3090 16 | htlatex ../../../literate.tex "xhtml,fn-in" && \
3091 17 | sed -i -e 's/<!--l\. [0-9][0-9]* *-->$//g' literate.html
3093 |________________________________________________________________________
3096 17.3 Extracting from the command line
3097 First you will need the tex output, then you can extract:
3099 88a <lyx-build-manual[1](
\v), lang=sh> ≡
3100 ________________________________________________________________________
3101 1 | lyx -e latex fangle.lyx
3102 2 | fangle -R./fangle fangle.tex > ./fangle
3103 3 | fangle -R./fangle.module fangle.tex > ./fangle.module
3104 |________________________________________________________________________
3109 88b <test:helpers[1](
\v), lang=> ≡
3110 ________________________________________________________________________
3113 3 | then echo "Passed"
3114 4 | else echo "Failed"
3121 11 | then echo "Passed"
3122 12 | else echo "Failed"
3126 |________________________________________________________________________
3130 Chapter 18Chunk Parameters
3133 91a <test:lyx:chunk-params:sub[1](THING
\v, colour
\v\v), lang=> ≡
3134 ________________________________________________________________________
3135 1 | I see a ${THING},
3136 2 | a ${THING} of colour ${colour},
3137 3 | and looking closer =<\chunkref{test:lyx:chunk-params:sub:sub}(${colour)>
3138 |________________________________________________________________________
3142 91b <test:lyx:chunk-params:sub:sub[1](colour
\v\v), lang=> ≡
3143 ________________________________________________________________________
3144 1 | a funny shade of ${colour}
3145 |________________________________________________________________________
3149 91c <test:chunk-params:text[1](
\v), lang=> ≡ 92a⊳
3150 ________________________________________________________________________
3151 1 | What do you see? "=<\chunkref{test:lyx:chunk-params:sub}(joe, red)>"
3153 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3154 Should generate output:
3156 91d <test:lyx:chunk-params:result[1](
\v), lang=> ≡
3157 ________________________________________________________________________
3158 1 | What do you see? "I see a joe,
3159 2 | a joe of colour red,
3160 3 | and looking closer a funny shade of red"
3162 |________________________________________________________________________
3165 And this chunk will perform the test:
3167 91e <test:chunk-params[1](
\v), lang=> ≡ 92c⊳
3168 ________________________________________________________________________
3169 1 | $FANGLE -Rtest:lyx:chunk-params:result $TEX_SRC > $TMP/answer || exit 1
3170 2 | $FANGLE -Rtest:lyx:chunk-params:text $TEX_SRC > $TMP/result || exit 1
3171 3 | passtest diff $TMP/answer $TMP/result || (echo test:lyx:chunk-params:text failed ; exit 1)
3172 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3175 91f <test:chunk-params:sub[1](THING
\v, colour
\v\v), lang=> ≡
3176 ________________________________________________________________________
3177 1 | I see a ${THING}
3178 2 | a ${THING} of colour ${colour},
3179 3 | and looking closer «test:chunk-params:sub:sub
\v(${colour}
\v) 91g»
3180 |________________________________________________________________________
3184 91g <test:chunk-params:sub:sub[1](colour
\v\v), lang=> ≡
3185 ________________________________________________________________________
3186 1 | a funny shade of ${colour}
3187 |________________________________________________________________________
3191 92a <test:chunk-params:text[2](
\v) ⇑91c, lang=> +≡ ⊲91c
3192 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3193 3 | What do you see? "«test:chunk-params:sub
\v(joe
\v, red
\v) 91f»
3195 |________________________________________________________________________
3198 Should generate output:
3200 92b <test:chunk-params:result[1](
\v), lang=> ≡
3201 ________________________________________________________________________
3202 1 | What do you see? "I see a joe,
3203 2 | a joe of colour red,
3204 3 | and looking closer a funny shade of red"
3206 |________________________________________________________________________
3209 And this chunk will perform the test:
3211 92c <test:chunk-params[2](
\v) ⇑91e, lang=> +≡ ⊲91e
3212 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3213 4 | $FANGLE -Rtest:chunk-params:result $TEX_SRC > $TMP/answer || exit 1
3214 5 | $FANGLE -Rtest:chunk-params:text $TEX_SRC > $TMP/result || exit 1
3215 6 | passtest diff $TMP/answer $TMP/result || (echo test:chunk-params:text failed ; exit 1)
3216 |________________________________________________________________________
3219 Chapter 19Compile-log-lyx
3221 93a <Chunk:./compile-log-lyx[1](
\v), lang=sh> ≡
3222 ________________________________________________________________________
3224 2 | # can't use gtkdialog -i, cos it uses the "source" command which ubuntu sh doesn't have
3227 5 | errors="/tmp/compile.log.$$"
3228 6 | # if grep '^[^ ]*:\( In \|[0-9][0-9]*: [^ ]*:\)' > $errors
3229 7 | if grep '^[^ ]*(\([0-9][0-9]*\)) *: *\(error\|warning\)' > $errors
3231 9 | sed -i -e 's/^[^ ]*[/\\]\([^/\\]*\)(\([ 0-9][ 0-9]*\)) *: */\1:\2|\2|/' $errors
3232 10 | COMPILE_DIALOG='
3235 13 | <label>Compiler errors:</label>
3237 15 | <tree exported_column="0">
3238 16 | <variable>LINE</variable>
3239 17 | <height>400</height><width>800</width>
3240 18 | <label>File | Line | Message</label>
3241 19 | <action>'". $SELF ; "'lyxgoto $LINE</action>
3242 20 | <input>'"cat $errors"'</input>
3245 23 | <button><label>Build</label>
3246 24 | <action>lyxclient -c "LYXCMD:build-program" &</action>
3248 26 | <button ok></button>
3252 30 | export COMPILE_DIALOG
3253 31 | ( gtkdialog --program=COMPILE_DIALOG ; rm $errors ) &
3260 38 | file="${LINE%:*}"
3261 39 | line="${LINE##*:}"
3262 40 | extraline=‘cat $file | head -n $line | tac | sed '/^\\\\begin{lstlisting}/q' | wc -l‘
3263 41 | extraline=‘expr $extraline - 1‘
3264 42 | lyxclient -c "LYXCMD:command-sequence server-goto-file-row $file $line ; char-forward ; repeat $extraline paragraph-down ; paragraph-up-select"
3268 46 | if test -z "$COMPILE_DIALOG"
3271 |________________________________________________________________________