18 Fangle is a tool for fangled literate programming. Newfangled is defined as New and often needlessly novel by TheFreeDictionary.com.
19 In this case, fangled means yet another not-so-new1. but improved. ^1 method for literate programming.
20 Literate Programming has a long history starting with the great Donald Knuth himself, whose literate programming tools seem to make use of as many escape sequences for semantic markup as TeX (also by Donald Knuth).
21 Norman Ramsey wrote the Noweb set of tools (notangle, noweave and noroots) and helpfully reduced the amount of magic character sequences to pretty much just <<, >> and @, and in doing so brought the wonders of literate programming within my reach.
22 While using the L Y X editor for LaTeX editing I had various troubles with the noweb tools, some of which were my fault, some of which were noweb's fault and some of which were L Y X's fault.
23 Noweb generally brought literate programming to the masses through removing some of the complexity of the original literate programming, but this would be of no advantage to me if the L Y X / LaTeX combination brought more complications in their place.
24 Fangle was thus born (originally called Newfangle) as an awk replacement for notangle, adding some important features, like better integration with L Y X and LaTeX (and later TeXmacs), multiple output format conversions, and fixing notangle bugs like indentation when using -L for line numbers.
25 Significantly, fangle is just one program which replaces various programs in Noweb. Noweave is done away with and implemented directly as LaTeX macros, and noroots is implemented as a function of the untangler fangle.
26 Fangle is written in awk for portability reasons, awk being available for most platforms. A Python version2. hasn't anyone implemented awk in python yet? ^2 was considered for the benefit of L Y X but a scheme version for TeXmacs will probably materialise first; as TeXmacs macro capabilities help make edit-time and format-time rendering of fangle chunks simple enough for my weak brain.
27 As an extension to many literate-programming styles, Fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Name parameters (or even local variables in the callers scope) are anticipated, as parameterized chunks — useful though they are — are hard to comprehend in the literate document.
29 Fangle is licensed under the GPL 3 (or later).
30 This doesn't mean that sources generated by fangle must be licensed under the GPL 3.
31 This doesn't mean that you can't use or distribute fangle with sources of an incompatible license, but it means you must make the source of fangle available too.
32 As fangle is currently written in awk, an interpreted language, this should not be too hard.
34 4a <gpl3-copyright[1](
\v), lang=text> ≡
35 ________________________________________________________________________
36 1 | fangle - fully featured notangle replacement in awk
38 3 | Copyright (C) 2009-2010 Sam Liddicott <sam@liddicott.com>
40 5 | This program is free software: you can redistribute it and/or modify
41 6 | it under the terms of the GNU General Public License as published by
42 7 | the Free Software Foundation, either version 3 of the License, or
43 8 | (at your option) any later version.
45 10 | This program is distributed in the hope that it will be useful,
46 11 | but WITHOUT ANY WARRANTY; without even the implied warranty of
47 12 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
48 13 | GNU General Public License for more details.
50 15 | You should have received a copy of the GNU General Public License
51 16 | along with this program. If not, see <http://www.gnu.org/licenses/>.
52 |________________________________________________________________________
59 1 Introduction to Literate Programming 11
62 2.2 Extracting roots 13
63 2.3 Formatting the document 13
64 3 Using Fangle with L^ A T_ E X 15
65 4 Using Fangle with L Y X 17
66 4.1 Installing the L Y X module 17
67 4.2 Obtaining a decent mono font 17
71 4.3 Formatting your Lyx document 18
72 4.3.1 Customising the listing appearance 18
73 4.3.2 Global customisations 18
74 4.4 Configuring the build script 19
76 5 Using Fangle with T_ E X_( M A CS) 21
77 6 Fangle with Makefiles 23
78 6.1 A word about makefiles formats 23
79 6.2 Extracting Sources 23
80 6.2.1 Converting from L Y X to L^ A T_ E X 24
81 6.2.2 Converting from T_ E X_( M A CS) 24
82 6.3 Extracting Program Source 25
83 6.4 Extracting Source Files 25
84 6.5 Extracting Documentation 28
85 6.5.1 Formatting T_ E X 28
86 6.5.1.1 Running pdflatex 28
87 6.5.2 Formatting T_ E X_( M A CS) 28
88 6.5.3 Building the Documentation as a Whole 28
90 6.7 Boot-strapping the extraction 29
91 6.8 Incorporating Makefile.inc into existing projects 30
94 7 Fangle awk source code 35
96 7.2 Catching errors 37
97 8 T_ E X_( M A CS) args 38
98 9 L^ A T_ E X and lstlistings 39
99 9.1 Additional lstlstings parameters 41
100 9.2 Parsing chunk arguments 41
101 9.3 Expanding parameters in the text 43
102 10 Language Modes & Quoting 44
103 10.1 Modes to keep code together 47
104 10.2 Modes affect included chunks 47
105 10.3 Modes operation 47
106 10.4 Quoting scenarios 48
107 10.4.1 Direct quoting 49
108 10.5 Language Mode Definitions 49
111 10.5.3 Parentheses, Braces and Brackets 51
112 10.5.4 Customizing Standard Modes 52
119 10.7 A non-recursive mode tracker 57
120 10.7.1 Constructor 58
123 10.7.3.1 One happy chunk 62
125 10.8 Escaping and Quoting 63
126 11 Recognizing Chunks 65
128 11.1.1 T_ E X_( M A CS) 65
129 11.1.2 lstlistings 66
131 11.2.1 T_ E X_( M A CS) 67
134 11.3.1 lstlistings 68
136 11.4 Chunk contents 69
137 11.4.1 lstlistings 70
138 12 Processing Options 73
139 13 Generating the Output 75
140 13.1 Assembling the Chunks 76
141 13.1.1 Chunk Parts 76
144 16 Fangle LaTeX source code 87
145 16.1 fangle module 87
146 16.1.1 The Chunk style 87
147 16.1.2 The chunkref style 88
149 16.2.1 The chunk command 89
150 16.2.1.1 Chunk parameters 90
151 16.2.2 The noweb styled caption 90
152 16.2.3 The chunk counter 90
153 16.2.4 Cross references 93
155 17 Extracting fangle 95
156 17.1 Extracting from Lyx 95
157 17.2 Extracting documentation 95
158 17.3 Extracting from the command line 96
161 19 Chunk Parameters 101
163 19.2 T_ E X_( M A CS) 101
164 20 Compile-log-lyx 103
166 Chapter 1Introduction to Literate Programming
167 Todo: Should really follow on from a part-0 explanation of what literate programming is.
168 Chapter 2Running Fangle
169 Fangle is a replacement for noweb, which consists of notangle, noroots and noweave.
170 Like notangle and noroots, fangle can read multiple named files, or from stdin.
172 The -r option causes fangle to behave like noroots.
173 fangle -r filename.tex
174 will print out the fangle roots of a tex file.
175 Unlike the noroots command, the printed roots are not enclosed in angle brackets e.g. <<name>>, unless at least one of the roots is defined using the notangle notation <<name>>=.
176 Also, unlike noroots, it prints out all roots --- not just those that are not used elsewhere. I find that a root not being used doesn't make it particularly top level — and so-called top level roots could also be included in another root as well.
177 My convention is that top level roots to be extracted begin with ./ and have the form of a filename.
178 Makefile.inc, discussed in 6, can automatically extract all such sources prefixed with ./
180 notangle's -R and -L options are supported.
181 If you are using L Y X or LaTeX, the standard way to extract a file would be:
182 fangle -R./Makefile.inc fangle.tex > ./Makefile.inc
183 If you are using TeXmacs, the standard way to extract a file would similarly be:
184 fangle -R./Makefile.inc fangle.txt > ./Makefile.inc
185 TeXmacs users would obtain the text file with a verbatim export from TeXmacs which can be done on the command line with texmacs -s -c fangle.tm fangle.txt -q
186 Unlike the noroots command, the -L option to generate C pre-preocessor #file style line-number directives,does not break indenting of the generated file..
187 Also, thanks to mode tracking (described in 11) the -L option does not interrupt (and break) multi-line C macros either.
188 This does mean that sometimes the compiler might calculate the source line wrongly when generating error messages in such cases, but there isn't any other way around if multi-line macros include other chunks.
189 Future releases will include a mapping file so that line/character references from the C compiler can be converted to the correct part of the source document.
190 2.3 Formatting the document
191 The noweave replacement built into the editing and formatting environment for TeXmacs, L Y X (which uses LaTeX), and even for raw LaTeX.
192 Use of fangle with TeXmacs, L Y X and LaTeX are explained the the next few chapters.
193 Chapter 3Using Fangle with LaTeX
194 Because the noweave replacement is impemented in LaTeX, there is no processing stage required before running the LaTeX command. Of course, LaTeX may need running two or more times, so that the code chunk references can be fully calculated.
195 The formatting is managed by a set of macros shown in 17, and can be included with:
196 \usepackage{fangle.sty}
197 Norman Ramsay's origial noweb.sty package is currently required as it is used for formatting the code chunk captions.
198 The listings.sty package is required, and is used for formatting the code chunks and syntax highlighting.
199 The xargs.sty package is also required, and makes writing LaTeX macro so much more pleasant.
200 To do: Add examples of use of Macros
202 Chapter 4Using Fangle with L Y X
203 L Y X uses the same LaTeX macros shown in 17 as part of a L Y X module file fangle.module, which automatically includes the macros in the document pre-amble provided that the fangle L Y X module is used in the document.
204 4.1 Installing the L Y X module
205 Copy fangle.module to your L Y X layouts directory, which for unix users will be ~/.lyx/layouts
206 In order to make the new literate styles availalble, you will need to reconfigure L Y X by clicking Tools->Reconfigure, and then re-start L Y X.
207 4.2 Obtaining a decent mono font
208 The syntax high-lighting features of lstlistings makes use of bold; however a mono-space tt font is used to typeset the listings. Obtaining a bold tt font can be impossibly difficult and amazingly easy. I spent many hours at it, following complicated instructions from those who had spend many hours over it, and was finally delivered the simple solution on the lyx mailing list.
210 The simple way was to add this to my preamble:
212 \renewcommand{\ttdefault}{txtt}
215 The next simplest way was to use ams poor-mans-bold, by adding this to the pre-amble:
217 %\renewcommand{\ttdefault}{txtt}
218 %somehow make \pmb be the command for bold, forgot how, sorry, above line not work
219 It works, but looks wretched on the dvi viewer.
221 The lstlistings documention suggests using Luximono.
222 Luximono was installed according to the instructions in Ubuntu Forums thread 11591811. http://ubuntuforums.org/showthread.php?t=1159181 ^1 with tips from miknight2. http://miknight.blogspot.com/2005/11/how-to-install-luxi-mono-font-in.html ^2 stating that sudo updmap --enable MixedMap ul9.map is required. It looks fine in PDF and PS view but still looks rotten in dvi view.
223 4.3 Formatting your Lyx document
224 It is not necessary to base your literate document on any of the original L Y X literate classes; so select a regular class for your document type.
225 Add the new module Fangle Literate Listings and also Logical Markup which is very useful.
226 In the drop-down style listbox you should notice a new style defined, called Chunk.
227 When you wish to insert a literate chunk, you enter it's plain name in the Chunk style, instead of the old noweb method that uses <<name>>= type tags. In the line (or paragraph) following the chunk name, you insert a listing with: Insert->Program Listing.
228 Inside the white listing box you can type (or paste using shift+ctrl+V) your listing. There is no need to use ctrl+enter at the end of lines as with some older L Y X literate techniques --- just press enter as normal.
229 4.3.1 Customising the listing appearance
230 The code is formatted using the lstlistings package. The chunk style doesn't just define the chunk name, but can also define any other chunk options supported by the lstlistings package \lstset command. In fact, what you type in the chunk style is raw latex. If you want to set the chunk language without having to right-click the listing, just add ,lanuage=C after the chunk name. (Currently the language will affect all subsequent listings, so you may need to specify ,language= quite a lot).
231 To do: so fix the bug
233 Of course you can do this by editing the listings box advanced properties by right-clicking on the listings box, but that takes longer, and you can't see at-a-glance what the advanced settings are while editing the document; also advanced settings apply only to that box --- the chunk settings apply through the rest of the document3. It ought to apply only to subsequent chunks of the same name. I'll fix that later ^3.
234 To do: So make sure they only apply to chunks of that name
236 4.3.2 Global customisations
237 As lstlistings is used to set the code chunks, it's \lstset command can be used in the pre-amble to set some document wide settings.
238 If your source has many words with long sequences of capital letters, then columns=fullflexible may be a good idea, or the capital letters will get crowded. (I think lstlistings ought to use a slightly smaller font for captial letters so that they still fit).
239 The font family \ttfamily looks more normal for code, but has no bold (an alternate typewriter font is used).
240 With \ttfamily, I must also specify columns=fullflexible or the wrong letter spacing is used.
241 In my LaTeX pre-amble I usually specialise my code format with:
243 19a <document-preamble[1](
\v), lang=tex> ≡
244 ________________________________________________________________________
246 2 | numbers=left, stepnumber=1, numbersep=5pt,
247 3 | breaklines=false,
248 4 | basicstyle=\footnotesize\ttfamily,
249 5 | numberstyle=\tiny,
251 7 | columns=fullflexible,
252 8 | numberfirstline=true
254 |________________________________________________________________________
258 4.4 Configuring the build script
259 You can invoke code extraction and building from the L Y X menu option Document->Build Program.
260 First, make sure you don't have a conversion defined for Lyx->Program
261 From the menu Tools->Preferences, add a conversion from Latex(Plain)->Program as:
262 set -x ; fangle -Rlyx-build $$i |
263 env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
264 (But don't cut-n-paste it from this document or you may be be pasting a multi-line string which will break your lyx preferences file).
265 I hope that one day, L Y X will set these into the environment when calling the build script.
266 You may also want to consider adding options to this conversion...
267 parselog=/usr/share/lyx/scripts/listerrors
268 ...but if you do you will lose your stderr4. There is some bash plumbing to get a copy of stderr but this footnote is too small ^4.
269 Now, a shell script chunk called lyx-build will be extracted and run whenever you choose the Document->Build Program menu item.
270 This document was originally managed using L Y X and lyx-build script for this document is shown here for historical reference.
271 lyx -e latex fangle.lyx && \
272 fangle fangle.lyx > ./autoboot
273 This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
275 When the lyx-build chunk is executed, the current directory will be a temporary directory, and LYX_SOURCE will refer to the tex file in this temporary directory. This is unfortunate as our makefile wants to run from the project directory where the Lyx file is kept.
276 We can extract the project directory from $$r, and derive the probable Lyx filename from the noweb file that Lyx generated.
278 19b <lyx-build-helper[1](
\v), lang=sh> ≡ 95b⊳
279 ________________________________________________________________________
280 1 | PROJECT_DIR="$LYX_r"
281 2 | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
283 4 | TEX_SRC="$TEX_DIR/$LYX_i"
284 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
285 And then we can define a lyx-build fragment similar to the autoboot fragment
287 20a <lyx-build[1](
\v), lang=sh> ≡ 95a⊳
288 ________________________________________________________________________
290 2 | «lyx-build-helper 19b»
291 3 | cd $PROJECT_DIR || exit 1
293 5 | #/usr/bin/fangle -filter ./notanglefix-filter \
294 6 | # -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx" \
295 7 | # | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/' \
296 8 | # > ./Makefile.inc
298 10 | #make -f ./Makefile.inc fangle_sources
299 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
301 Chapter 5Using Fangle with TeXmacs
302 To do: Write this chapter
304 Chapter 6Fangle with Makefiles
305 Here we describe a Makefile.inc that you can include in your own Makefiles, or glue as a recursive make to other projects.
306 Makefile.inc will cope with extracting all the other source files from this or any specified literate document and keeping them up to date.
307 It may also be included by a Makefile or Makefile.am defined in a literate document to automatically deal with the extraction of source files and documents during normal builds.
308 Thus, if Makefile.inc is included into a main project makefile it add rules for the source files, capable of extracting the source files from the literate document.
309 6.1 A word about makefiles formats
310 Whitespace formatting is very important in a Makefile. The first character of each action line must be a TAB.
311 target: pre-requisite
314 This requires that the literate programming environment have the ability to represent a TAB character in a way that fangle will generate an actual TAB character.
315 We also adopt a convention that code chunks whose names beginning with ./ should always be automatically extracted from the document. Code chunks whose names do not begin with ./ are for internal reference. Such chunks may be extracted directly, but will not be automatically extracted by this Makefile.
316 6.2 Extracting Sources
317 Our makefile has two parts; variables must be defined before the targets that use them.
318 As we progress through this chapter, explaining concepts, we will be adding lines to <Makefile.inc-vars 23b> and <Makefile.inc-targets 24c> which are included in <./Makefile.inc 23a> below.
320 23a <./Makefile.inc[1](
\v), lang=make> ≡
321 ________________________________________________________________________
322 1 | «Makefile.inc-vars 23b»
323 2 | «Makefile.inc-default-targets 28a»
324 3 | «Makefile.inc-targets 24c»
325 |________________________________________________________________________
328 We first define a placeholder for the tool fangle in case it cannot be found in the path.
330 23b <Makefile.inc-vars[1](
\v), lang=make> ≡ 24a⊳
331 ________________________________________________________________________
334 3 | RUN_FANGLE=$(AWK) -f $(FANGLE)
335 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
336 We also define a placeholder for LITERATE_SOURCE to hold the name of this document. This will normally be passed on the command line or set by the including makefile.
338 24a <Makefile.inc-vars[2](
\v) ⇑23b, lang=> +≡ ⊲23b 24b▿
339 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
340 4 | #LITERATE_SOURCE=
341 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
342 Fangle cannot process L Y X or TeXmacs documents directly, so the first stage is to convert these to more suitable text based formats1. L Y X and TeXmacs formats are text-based, but not suitable for fangle ^1.
343 6.2.1 Converting from L Y X to LaTeX
344 The first stage will always be to convert the L Y X file to a LaTeX file. Fangle must run on a TeX file because the L Y X command server-goto-file-line2. The Lyx command server-goto-file-line is used to position the Lyx cursor at the compiler errors. ^2 requries that the line number provided be a line of the TeX file and always maps this the line in the L Y X docment. We use server-goto-file-line when moving the cursor to error lines during compile failures.
345 The command lyx -e literate fangle.lyx will produce fangle.tex, a TeX file; so we define a make target to be the same as the L Y X file but with the .tex extension.
346 The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.
348 24b <Makefile.inc-vars[3](
\v) ⇑23b, lang=> +≡ ▵24a 24d▿
349 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
350 5 | LYX_SOURCE=$(LITERATE_SOURCE) # but only the .lyx files
351 6 | TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
352 7 | EXTRA_DIST+=$(TEX_SOURCE)
353 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
354 We then specify that the TeX source is to be generated from the L Y X source.
356 24c <Makefile.inc-targets[1](
\v), lang=make> ≡ 25a⊳
357 ________________________________________________________________________
358 1 | .SUFFIXES: .tex .lyx
362 5 | ↦rm -f -- $(TEX_SOURCE)
364 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
365 6.2.2 Converting from TeXmacs
366 Fangle cannot process TeXmacs files directly3. but this is planned when TeXmacs uses xml as it's native format ^3, but must first convert them to text files.
367 The command texmacs -c fangle.tm fangle.txt -q will produce fangle.txt, a text file; so we define a make target to be the same as the TeXmacs file but with the .txt extension.
368 The EXTRA_DIST is for automake support so that the TeX files will automaticaly be distributed with the source, to help those who don't have L Y X installed.
370 24d <Makefile.inc-vars[4](
\v) ⇑23b, lang=> +≡ ▵24b 25b⊳
371 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
372 8 | TEXMACS_SOURCE=$(LITERATE_SOURCE) # but only the .tm files
373 9 | TXT_SOURCE=$(LITERATE_SOURCE:.tm=.txt)
374 10 | EXTRA_DIST+=$(TXT_SOURCE)
375 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
376 To do: Add loop around each $< so multiple targets can be specified
379 25a <Makefile.inc-targets[2](
\v) ⇑24c, lang=> +≡ ⊲24c 25d▿
380 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
381 7 | .SUFFIXES: .txt .tm
383 9 | ↦texmacs -s -c $< $@ -q
384 10 | .PHONEY: clean_txt
386 12 | ↦rm -f -- $(TXT_SOURCE)
387 13 | clean: clean_txt
388 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
389 6.3 Extracting Program Source
390 The program source is extracted using fangle, which is designed to operate on text or a LaTeX documents4. LaTeX documents are just slightly special text documents ^4.
392 25b <Makefile.inc-vars[5](
\v) ⇑23b, lang=> +≡ ⊲24d 25c▿
393 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
394 11 | FANGLE_SOURCE=$(TXT_SOURCE)
395 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
396 The literate document can result in any number of source files, but not all of these will be changed each time the document is updated. We certainly don't want to update the timestamps of these files and cause the whole source tree to be recompiled just because the literate explanation was revised. We use CPIF from the Noweb tools to avoid updating the file if the content has not changed, but should probably write our own.
397 However, if a source file is not updated, then the fangle file will always have a newer time-stamp and the makefile would always re-attempt to extact a newer source file which would be a waste of time.
398 Because of this, we use a stamp file which is always updated each time the sources are fully extracted from the LaTeX document. If the stamp file is newer than the document, then we can avoid an attempt to re-extract any of the sources. Because this stamp file is only updated when extraction is complete, it is safe for the user to interrupt the build-process mid-extraction.
399 We use echo rather than touch to update the stamp file beause the touch command does not work very well over an sshfs mount that I was using.
401 25c <Makefile.inc-vars[6](
\v) ⇑23b, lang=> +≡ ▵25b 26a⊳
402 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
403 12 | FANGLE_SOURCE_STAMP=$(FANGLE_SOURCE).stamp
404 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
406 25d <Makefile.inc-targets[3](
\v) ⇑24c, lang=> +≡ ▵25a 26b⊳
407 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
408 14 | $(FANGLE_SOURCE_STAMP): $(FANGLE_SOURCE) \
409 15 | ↦ $(FANGLE_SOURCES) ; \
410 16 | ↦echo -n > $(FANGLE_SOURCE_STAMP)
412 18 | ↦rm -f $(FANGLE_SOURCE_STAMP)
413 19 | clean: clean_stamp
414 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
415 6.4 Extracting Source Files
416 We compute FANGLE_SOURCES to hold the names of all the source files defined in the document. We compute this only once, by means of := in assignent. The sed deletes the any << and >> which may surround the roots names (for compatibility with Noweb's noroots command).
417 As we use chunk names beginning with ./ to denote top level fragments that should be extracted, we filter out all fragments that do not begin with ./
418 Note 1. FANGLE_PREFIX is set to ./ by default, but whatever it may be overridden to, the prefix is replaced by a literal ./ before extraction so that files will be extracted in the current directory whatever the prefix. This helps namespace or sub-project prefixes like documents: for chunks like documents:docbook/intro.xml
419 To do: This doesn't work though, because it loses the full name and doesn't know what to extact!
422 26a <Makefile.inc-vars[7](
\v) ⇑23b, lang=> +≡ ⊲25c 26e▿
423 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
424 13 | FANGLE_PREFIX:=\.\/
425 14 | FANGLE_SOURCES:=$(shell \
426 15 | $(RUN_FANGLE) -r $(FANGLE_SOURCE) |\
427 16 | sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(FANGLE_PREFIX)/!d' \
428 17 | -e 's/^$(FANGLE_PREFIX)/\.\//' )
429 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
430 The target below, echo_fangle_sources is a helpful debugging target and shows the names of the files that would be extracted.
432 26b <Makefile.inc-targets[4](
\v) ⇑24c, lang=> +≡ ⊲25d 26c▿
433 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
434 20 | .PHONY: echo_fangle_sources
435 21 | echo_fangle_sources: ; @echo $(FANGLE_SOURCES)
436 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
437 We define a convenient target called fangle_sources so that make -f fangle_sources will re-extract the source if the literate document has been updated.
439 26c <Makefile.inc-targets[5](
\v) ⇑24c, lang=> +≡ ▵26b 26d▿
440 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
441 22 | .PHONY: fangle_sources
442 23 | fangle_sources: $(FANGLE_SOURCE_STAMP)
443 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
444 And also a convenient target to remove extracted sources.
446 26d <Makefile.inc-targets[6](
\v) ⇑24c, lang=> +≡ ▵26c 27f⊳
447 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
448 24 | .PHONY: clean_fangle_sources
449 25 | clean_fangle_sources: ; \
450 26 | rm -f -- $(FANGLE_SOURCE_STAMP) $(FANGLE_SOURCES)
451 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
452 We now look at the extraction of the source files.
453 This makefile macro if_extension takes 4 arguments: the filename $(1), some extensions to match $(2) and a shell command to return if the filename does match the exensions $(3), and a shell command to return if it does not match the extensions $(4).
455 26e <Makefile.inc-vars[8](
\v) ⇑23b, lang=> +≡ ▵26a 27a⊳
456 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
457 18 | if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
458 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
459 For some source files like C files, we want to output the line number and filename of the original LaTeX document from which the source came5. I plan to replace this option with a separate mapping file so as not to pollute the generated source, and also to allow a code pretty-printing reformatter like indent be able to re-format the file and adjust for changes through comparing the character streams. ^5.
460 To make this easier we define the file extensions for which we want to do this.
462 27a <Makefile.inc-vars[9](
\v) ⇑23b, lang=> +≡ ⊲26e 27b▿
463 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
464 19 | C_EXTENSIONS=.c .h
465 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
466 We can then use the if_extensions macro to define a macro which expands out to the -L option if fangle is being invoked in a C source file, so that C compile errors will refer to the line number in the TeX document.
468 27b <Makefile.inc-vars[10](
\v) ⇑23b, lang=> +≡ ▵27a 27c▿
469 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
471 21 | nf_line=-L -T$(TABS)
472 22 | fangle=$(RUN_FANGLE) $(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line)) -R"$(2)" $(1)
473 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
474 We can use a similar trick to define an indent macro which takes just the filename as an argument and can return a pipeline stage calling the indent command. Indent can be turned off with make fangle_sources indent=
476 27c <Makefile.inc-vars[11](
\v) ⇑23b, lang=> +≡ ▵27b 27d▿
477 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
478 23 | indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
479 24 | indent=$(call if_extension,$(1),$(C_EXTENSIONS), | indent $(indent_options))
480 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
481 We now define the pattern for extracting a file. The files are written using noweb's cpif so that the file timestamp will not be touched if the contents haven't changed. This avoids the need to rebuild the entire project because of a typographical change in the documentation, or if none or a few C source files have changed.
483 27d <Makefile.inc-vars[12](
\v) ⇑23b, lang=> +≡ ▵27c 27e▿
484 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
485 25 | fangle_extract=@mkdir -p $(dir $(1)) && \
486 26 | $(call fangle,$(2),$(1)) > "$(1).tmp" && \
487 27 | cat "$(1).tmp" $(indent) | cpif "$(1)" \
488 28 | && rm -f -- "$(1).tmp" || \
489 29 | (echo error fangling $(1) from $(2) ; exit 1)
490 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
491 We define a target which will extract or update all sources. To do this we first defined a makefile template that can do this for any source file in the LaTeX document.
493 27e <Makefile.inc-vars[13](
\v) ⇑23b, lang=> +≡ ▵27d 28b⊳
494 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
495 30 | define FANGLE_template
497 32 | ↦$$(call fangle_extract,$(1),$(2))
498 33 | FANGLE_TARGETS+=$(1)
500 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
501 We then enumerate the discovered FANGLE_SOURCES to generate a makefile rule for each one using the makefile template we defined above.
503 27f <Makefile.inc-targets[7](
\v) ⇑24c, lang=> +≡ ⊲26d 27g▿
504 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
505 27 | $(foreach source,$(FANGLE_SOURCES),\
506 28 | $(eval $(call FANGLE_template,$(source),$(FANGLE_SOURCE))) \
508 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
509 These will all be built with FANGLE_SOURCE_STAMP.
510 We also remove the generated sources on a make distclean.
512 27g <Makefile.inc-targets[8](
\v) ⇑24c, lang=> +≡ ▵27f 28c⊳
513 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
514 30 | _distclean: clean_fangle_sources
515 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
516 6.5 Extracting Documentation
517 We then identify the intermediate stages of the documentation and their build and clean targets.
519 28a <Makefile.inc-default-targets[1](
\v), lang=> ≡
520 ________________________________________________________________________
521 1 | .PHONEY : clean_pdf
522 |________________________________________________________________________
526 6.5.1.1 Running pdflatex
527 We produce a pdf file from the tex file.
529 28b <Makefile.inc-vars[14](
\v) ⇑23b, lang=> +≡ ⊲27e 28d▿
530 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
531 35 | FANGLE_PDF+=$(TEX_SOURCE:.tex=.pdf)
532 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
533 We run pdflatex twice to be sure that the contents and aux files are up to date. We certainly are required to run pdflatex at least twice if these files do not exist.
535 28c <Makefile.inc-targets[9](
\v) ⇑24c, lang=> +≡ ⊲27g 28e▿
536 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
537 31 | .SUFFIXES: .tex .pdf
539 33 | ↦pdflatex $< && pdflatex $<
542 36 | ↦rm -f -- $(FANGLE_PDF) $(TEX_SOURCE:.tex=.toc) \
543 37 | ↦ $(TEX_SOURCE:.tex=.log) $(TEX_SOURCE:.tex=.aux)
544 38 | clean_pdf: clean_pdf_tex
545 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
546 6.5.2 Formatting TeXmacs
547 TeXmacs can produce a PDF file directly.
549 28d <Makefile.inc-vars[15](
\v) ⇑23b, lang=> +≡ ▵28b 29a⊳
550 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
551 36 | FANGLE_PDF+=$(LITERATE_SOURCE:.tm=.pdf)
552 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
553 To do: Outputting the PDF may not be enough to update the links and page references. I think
554 we need to update twice, generate a pdf, update twice mode and generate a new PDF.
555 Basically the PDF export of TeXmacs is pretty rotten and doesn't work properly from the CLI
558 28e <Makefile.inc-targets[10](
\v) ⇑24c, lang=> +≡ ▵28c 29b⊳
559 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
560 39 | .SUFFIXES: .tm .pdf
562 41 | ↦texmacs -s -c $< $@ -q
564 43 | clean_pdf_texmacs:
565 44 | ↦rm -f -- $(FANGLE_PDF)
566 45 | clean_pdf: clean_pdf_texmacs
567 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
568 6.5.3 Building the Documentation as a Whole
569 Currently we only build pdf as a final format, but FANGLE_DOCS may later hold other output formats.
571 29a <Makefile.inc-vars[16](
\v) ⇑23b, lang=> +≡ ⊲28d
572 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
573 37 | FANGLE_DOCS=$(FANGLE_PDF)
574 |________________________________________________________________________
577 We also define fangle_docs as a convenient phony target.
579 29b <Makefile.inc-targets[11](
\v) ⇑24c, lang=> +≡ ⊲28e 29c▿
580 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
581 46 | .PHONY: fangle_docs
582 47 | fangle_docs: $(FANGLE_DOCS)
583 48 | docs: fangle_docs
584 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
585 And define a convenient clean_fangle_docs which we add to the regular clean target
587 29c <Makefile.inc-targets[12](
\v) ⇑24c, lang=> +≡ ▵29b
588 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
589 49 | .PHONEY: clean_fangle_docs
590 50 | clean_fangle_docs: clean_tex clean_pdf
591 51 | clean: clean_fangle_docs
593 53 | distclean_fangle_docs: clean_tex clean_fangle_docs
594 54 | distclean: clean distclean_fangle_docs
595 |________________________________________________________________________
599 If Makefile.inc is included into Makefile, then extracted files can be updated with this command:
602 make -f Makefile.inc fangle_sources
603 6.7 Boot-strapping the extraction
604 As well as having the makefile extract or update the source files as part of it's operation, it also seems convenient to have the makefile re-extracted itself from this document.
605 It would also be convenient to have the code that extracts the makefile from this document to also be part of this document, however we have to start somewhere and this unfortunately requires us to type at least a few words by hand to start things off.
606 Therefore we will have a minimal root fragment, which, when extracted, can cope with extracting the rest of the source. This shell script fragment can do that. It's name is * — out of regard for Noweb, but when extracted might better be called autoupdate.
610 29d <*[1](
\v), lang=sh> ≡
611 ________________________________________________________________________
614 3 | MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
615 4 | MAKE_SRC=‘dirname "$MAKE_SRC"‘/‘basename "$MAKE_SRC" .lyx‘
616 5 | NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
617 6 | lyx -e latex $MAKE_SRC
619 8 | fangle -R./Makefile.inc ${MAKE_SRC}.tex \
620 9 | | sed "/FANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$FANGLE_SRC" \
621 10 | | cpif ./Makefile.inc
623 12 | make -f ./Makefile.inc fangle_sources
624 |________________________________________________________________________
627 The general Makefile can be invoked with ./autoboot and can also be included into any automake file to automatically re-generate the source files.
628 The autoboot can be extracted with this command:
629 lyx -e latex fangle.lyx && \
630 fangle fangle.lyx > ./autoboot
631 This looks simple enough, but as mentioned, fangle has to be had from somewhere before it can be extracted.
632 On a unix system this will extract fangle.module and the fangle awk script, and run some basic tests.
633 To do: cross-ref to test chapter when it is a chapter all on its own
635 6.8 Incorporating Makefile.inc into existing projects
636 If you are writing a literate module of an existing non-literate program you may find it easier to use a slight recursive make instead of directly including Makefile.inc in the projects makefile.
637 This way there is less chance of definitions in Makefile.inc interfering with definitions in the main makefile, or with definitions in other Makefile.inc from other literate modules of the same project.
638 To do this we add some glue to the project makefile that invokes Makefile.inc in the right way. The glue works by adding a .PHONY target to call the recursive make, and adding this target as an additional pre-requisite to the existing targets.
639 Example Sub-module of existing system
640 In this example, we are building module.so as a literate module of a larger project.
641 We will show the sort glue that can be inserted into the projects Makefile — or more likely — a regular Makefile included in or invoked by the projects Makefile.
643 30a <makefile-glue[1](
\v), lang=> ≡ 30b▿
644 ________________________________________________________________________
645 1 | module_srcdir=modules/module
646 2 | MODULE_SOURCE=module.tm
647 3 | MODULE_STAMP=$(MODULE_SOURCE).stamp
648 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
649 The existing build system may already have a build target for module.o, but we just add another pre-requisite to that. In this case we use module.tm.stamp as a pre-requisite, the stamp file's modified time indicating when all sources were extracted6. If the projects build system does not know how to build the module from the extracted sources, then just add build actions here as normal. ^6.
651 30b <makefile-glue[2](
\v) ⇑30a, lang=make> +≡ ▵30a 30c▿
652 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
653 4 | $(module_srcdir)/module.o: $(module_srcdir)/$(MODULE_STAMP)
654 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
655 The target for this new pre-requisite will be generated by a recursive make using Makefile.inc which will make sure that the source is up to date, before it is built by the main projects makefile.
657 30c <makefile-glue[3](
\v) ⇑30a, lang=> +≡ ▵30b 31a⊳
658 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
659 5 | $(module_srcdir)/$(MODULE_STAMP): $(module_srcdir)/$(MODULE_SOURCE)
660 6 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc fangle_sources LITERATE_SOURCE=$(MODULE_SOURCE)
661 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
662 We can do similar glue for the docs, clean and distclean targets. In this example the main prject was using a double colon for these targets, so we must use the same in our glue.
664 31a <makefile-glue[4](
\v) ⇑30a, lang=> +≡ ⊲30c
665 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
666 7 | docs:: docs_module
667 8 | .PHONY: docs_module
669 10 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc docs LITERATE_SOURCE=$(MODULE_SOURCE)
671 12 | clean:: clean_module
672 13 | .PHONEY: clean_module
674 15 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc clean LITERATE_SOURCE=$(MODULE_SOURCE)
676 17 | distclean:: distclean_module
677 18 | .PHONY: distclean_module
678 19 | distclean_module:
679 20 | ↦$(MAKE) -C $(module_srcdir) -f Makefile.inc distclean LITERATE_SOURCE=$(MODULE_SOURCE)
680 |________________________________________________________________________
683 We could do similarly for install targets to install the generated docs.
685 Chapter 7Fangle Makefile
686 We use the copyright notice from chapter 2, and the Makefile.inc from chapter 6
688 35a <./Makefile[1](
\v), lang=make> ≡
689 ________________________________________________________________________
690 1 | # «gpl3-copyright 4a»
692 3 | «make-fix-make-shell 55c»
694 5 | LITERATE_SOURCE=fangle.tm
696 7 | all: fangle_sources
697 8 | include Makefile.inc
702 13 | test: fangle.txt
703 14 | ↦$(RUN_FANGLE) -R"test:*" fangle.txt > test.sh
704 15 | ↦bash test.sh ; echo pass $$?
705 |________________________________________________________________________
708 Chapter 8Fangle awk source code
709 We use the copyright notice from chapter 2.
711 37a <./fangle[1](
\v), lang=awk> ≡ 37b▿
712 ________________________________________________________________________
713 1 | #! /usr/bin/awk -f
714 2 | # «gpl3-copyright 4a»
715 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
716 We also use code from Arnold Robbins public domain getopt (1993 revision) defined in 85a, and naturally want to attribute this appropriately.
718 37b <./fangle[2](
\v) ⇑37a, lang=> +≡ ▵37a 37c▿
719 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
720 3 | # NOTE: Arnold Robbins public domain getopt for awk is also used:
721 4 | «getopt.awk-header 83a»
722 5 | «getopt.awk-getopt() 83c»
724 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
725 And include the following chunks (which are explained further on) to make up the program:
727 37c <./fangle[3](
\v) ⇑37a, lang=> +≡ ▵37b 42a⊳
728 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
729 7 | «helper-functions 38d»
730 8 | «mode-tracker 62b»
731 9 | «parse_chunk_args 44a»
732 10 | «chunk-storage-functions 81b»
733 11 | «output_chunk_names() 75d»
734 12 | «output_chunks() 75e»
735 13 | «write_chunk() 76a»
736 14 | «expand_chunk_args() 44b»
739 17 | «recognize-chunk 65a»
741 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
743 The portable way to erase an array in awk is to split the empty string, so we define a fangle macro that can split an array, like this:
745 37d <awk-delete-array[1](ARRAY
\v\v), lang=awk> ≡
746 ________________________________________________________________________
747 1 | split("", ${ARRAY});
748 |________________________________________________________________________
751 For debugging it is sometimes convenient to be able to dump the contents of an array to stderr, and so this macro is also useful.
753 37e <dump-array[1](ARRAY
\v\v), lang=awk> ≡
754 ________________________________________________________________________
755 1 | print "\nDump: ${ARRAY}\n--------\n" > "/dev/stderr";
756 2 | for (_x in ${ARRAY}) {
757 3 | print _x "=" ${ARRAY}[_x] "\n" > "/dev/stderr";
759 5 | print "========\n" > "/dev/stderr";
760 |________________________________________________________________________
764 Fatal errors are issued with the error function:
766 38a <error()[1](
\v), lang=awk> ≡ 38b▿
767 ________________________________________________________________________
768 1 | function error(message)
770 3 | print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
773 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
774 and likewise for non-fatal warnings:
776 38b <error()[2](
\v) ⇑38a, lang=awk> +≡ ▵38a 38c▿
777 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
778 6 | function warning(message)
780 8 | print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
783 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
784 and debug output too:
786 38c <error()[3](
\v) ⇑38a, lang=awk> +≡ ▵38b
787 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
788 11 | function debug_log(message)
790 13 | print "DEBUG: " FILENAME ":" FNR " " message > "/dev/stderr";
792 |________________________________________________________________________
795 To do: append=helper-functions
798 38d <helper-functions[1](
\v), lang=> ≡
799 ________________________________________________________________________
801 |________________________________________________________________________
804 Chapter 9TeXmacs args
805 TeXmacs functions with arguments1. or function declarations with parameters ^1 appear like this:
806 blah((I came, I saw, I conquered)<wide-overbrace>^(argument 1)(^K, )<wide-overbrace>^(sep.)(and then went home asd)<wide-overbrace>^(argument 3)(^K))<wide-overbrace>^(term.)_arguments
807 Arguments commence after the opening parenthesis. The first argument runs up till the next ^K.
808 If the following character is a , then another argument follows. If the next character after the , is a space character, then it is also eaten. The fangle stylesheet emits ^K,space as separators, but the fangle untangler will forgive a missing space.
809 If the following character is ) then this is a terminator and there are no more arguments.
811 39a <constants[1](
\v), lang=> ≡ 81a⊳
812 ________________________________________________________________________
813 1 | ARG_SEPARATOR=sprintf("%c", 11);
814 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
815 To process the text in this fashion, we split the string on ^K
818 39b <get_chunk_args[1](
\v), lang=> ≡
819 ________________________________________________________________________
820 1 | function get_texmacs_chunk_args(text, args, a, done) {
821 2 | split(text, args, ARG_SEPARATOR);
824 5 | for (a=1; (a in args); a++) if (a>1) {
825 6 | if (args[a] == "" || substr(args[a], 1, 1) == ")") done=1;
831 12 | if (substr(args[a], 1, 2) == ", ") args[a]=substr(args[a], 3);
832 13 | else if (substr(args[a], 1, 1) == ",") args[a]=substr(args[a], 2);
835 |________________________________________________________________________
838 Chapter 10LaTeX and lstlistings
839 To do: Split LyX and TeXmacs parts
841 For L Y X and LaTeX, the lstlistings package is used to format the lines of code chunks. You may recal from chapter XXX that arguments to a chunk definition are pure LaTeX code. This means that fangle needs to be able to parse LaTeX a little.
842 LaTeX arguments to lstlistings macros are a comma seperated list of key-value pairs, and values containing commas are enclosed in { braces } (which is to be expected for LaTeX).
843 A sample expressions is:
844 name=thomas, params={a, b}, something, something-else
845 but we see that this is just a simpler form of this expression:
846 name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
847 We may consider that we need a function that can parse such LaTeX expressions and assign the values to an AWK associated array, perhaps using a recursive parser into a multi-dimensional hash1. as AWK doesn't have nested-hash support ^1, resulting in:
852 a[foo, quux, a] fleeg
855 Yet, also, on reflection it seems that sometimes such nesting is not desirable, as the braces are also used to delimit values that contain commas --- we may consider that
856 name={williamson, freddie}
857 should assign williamson, freddie to name.
858 In fact we are not so interested in the detail so as to be bothered by this, which turns out to be a good thing for two reasons. Firstly TeX has a malleable parser with no strict syntax, and secondly whether or not williamson and freddie should count as two items will be context dependant anyway.
859 We need to parse this latex for only one reason; which is that we are extending lstlistings to add some additional arguments which will be used to express chunk parameters and other chunk options.
860 10.1 Additional lstlstings parameters
861 Further on we define a \Chunk LaTeX macro whose arguments will consist of a the chunk name, optionally followed by a comma and then a comma separated list of arguments. In fact we will just need to prefix name= to the arguments to in order to create valid lstlistings arguments.
862 There will be other arguments supported too;
863 params.As an extension to many literate-programming styles, fangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Chunk parameters are declared with a chunk argument called params, which holds a semi-colon separated list of parameters, like this:
864 achunk,language=C,params=name;address
865 addto.a named chunk that this chunk is to be included into. This saves the effort of having to declare another listing of the named chunk merely to include this one.
866 Function get_chunk_args() will accept two paramters, text being the text to parse, and values being an array to receive the parsed values as described above. The optional parameter path is used during recursion to build up the multi-dimensional array path.
868 42a <./fangle[4](
\v) ⇑37a, lang=> +≡ ⊲37c
869 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
870 19 | «get_chunk_args() 42b»
871 |________________________________________________________________________
875 42b <get_chunk_args()[1](
\v), lang=> ≡ 42c▿
876 ________________________________________________________________________
877 1 | function get_tex_chunk_args(text, values,
878 2 | # optional parameters
879 3 | path, # hierarchical precursors
882 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
883 The strategy is to parse the name, and then look for a value. If the value begins with a brace {, then we recurse and consume as much of the text as necessary, returning the remaining text when we encounter a leading close-brace }. This being the strategy --- and executed in a loop --- we realise that we must first look for the closing brace (perhaps preceded by white space) in order to terminate the recursion, and returning remaining text.
885 42c <get_chunk_args()[2](
\v) ⇑42b, lang=> +≡ ▵42b
886 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
888 7 | split("", values);
889 8 | while(length(text)) {
890 9 | if (match(text, "^ *}(.*)", a)) {
893 12 | «parse-chunk-args 42d»
897 |________________________________________________________________________
900 We can see that the text could be inspected with this regex:
902 42d <parse-chunk-args[1](
\v), lang=> ≡ 43a⊳
903 ________________________________________________________________________
904 1 | if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a)) {
907 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
908 and that a will have the following values:
911 2 =freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
913 4 freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
915 6 , foo={bar=baz, quux={quirk, a=fleeg}}, etc
917 a[3] will be either = or , and signify whether the option named in a[1] has a value or not (respectively).
918 If the option does have a value, then if the expression substr(a[4],1,1) returns a brace { it will signify that we need to recurse:
920 43a <parse-chunk-args[2](
\v) ⇑42d, lang=> +≡ ⊲42d
921 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
923 5 | if (a[3] == "=") {
924 6 | if (substr(a[4],1,1) == "{") {
925 7 | text = get_tex_chunk_args(substr(a[4],2), values, path name SUBSEP);
927 9 | values[path name]=a[5];
931 13 | values[path name]="";
934 |________________________________________________________________________
937 We can test this function like this:
939 43b <gca-test.awk[1](
\v), lang=> ≡
940 ________________________________________________________________________
941 1 | «get_chunk_args() 42b»
945 5 | print get_tex_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc", a);
947 7 | print "a[" b "] => " a[b];
950 |________________________________________________________________________
953 which should give this output:
955 43c <gca-test.awk-results[1](
\v), lang=> ≡
956 ________________________________________________________________________
957 1 | a[foo.quux.quirk] =>
958 2 | a[foo.quux.a] => fleeg
959 3 | a[foo.bar] => baz
961 5 | a[name] => freddie
962 |________________________________________________________________________
965 10.2 Parsing chunk arguments
966 Arguments to paramterized chunks are expressed in round brackets as a comma separated list of optional arguments. For example, a chunk that is defined with:
967 \Chunk{achunk, params=name ; address}
969 \chunkref{achunk}(John Jones, jones@example.com)
970 An argument list may be as simple as in \chunkref{pull}(thing, otherthing) or as complex as:
971 \chunkref{pull}(things[x, y], get_other_things(a, "(all)"))
972 --- which for all it's commas and quotes and parenthesis represents only two parameters: things[x, y] and get_other_things(a, "(all)").
973 If we simply split parameter list on commas, then the comma in things[x,y] would split into two seperate arguments: things[x and y]--- neither of which make sense on their own.
974 One way to prevent this would be by refusing to split text between matching delimiters, such as [, ], (, ), {, } and most likely also ", " and ', '. Of course this also makes it impossible to pass such mis-matched code fragments as parameters, but I think that it would be hard for readers to cope with authors who would pass such code unbalanced fragments as chunk parameters2. I know that I couldn't cope with users doing such things, and although the GPL3 license prevents me from actually forbidding anyone from trying, if they want it to work they'll have to write the code themselves and not expect any support from me. ^2.
975 Unfortunately, the full set of matching delimiters may vary from language to language. In certain C++ template contexts, < and > would count as delimiters, and yet in other contexts they would not.
976 This puts me in the unfortunate position of having to parse-somewhat all programming languages without knowing what they are!
977 However, if this universal mode-tracking is possible, then parsing the arguments would be trivial. Such a mode tracker is described in chapter 11 and used here with simplicity.
979 44a <parse_chunk_args[1](
\v), lang=> ≡
980 ________________________________________________________________________
981 1 | function parse_chunk_args(language, text, values, mode,
983 3 | c, context, rest)
985 5 | «new-mode-tracker
\v(context
\v, language
\v, mode
\v) 58b»
986 6 | rest = mode_tracker(context, text, values);
988 8 | for(c=1; c <= context[0, "values"]; c++) {
989 9 | values[c] = context[0, "values", c];
993 |________________________________________________________________________
996 10.3 Expanding parameters in the text
997 Within the body of the chunk, the parameters are referred to with: ${name} and ${address}. There is a strong case that a LaTeX style notation should be used, like \param{name} which would be expressed in the listing as =<\param{name}> and be rendered as ${name}. Such notation would make me go blind, but I do intend to adopt it.
998 We therefore need a function expand_chunk_args which will take a block of text, a list of permitted parameters, and the arguments which must substitute for the parameters.
999 Here we split the text on ${ which means that all parts except the first will begin with a parameter name which will be terminated by }. The split function will consume the literal ${ in each case.
1001 44b <expand_chunk_args()[1](
\v), lang=> ≡
1002 ________________________________________________________________________
1003 1 | function expand_chunk_args(text, params, args,
1004 2 | p, text_array, next_text, v, t, l)
1006 4 | if (split(text, text_array, "\\${")) {
1007 5 | «substitute-chunk-args 45a»
1012 |________________________________________________________________________
1015 First, we produce an associative array of substitution values indexed by parameter names. This will serve as a cache, allowing us to look up the replacement values as we extract each name.
1017 45a <substitute-chunk-args[1](
\v), lang=> ≡ 45b▿
1018 ________________________________________________________________________
1019 1 | for(p in params) {
1020 2 | v[params[p]]=args[p];
1022 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1023 We accumulate substituted text in the variable text. As the first part of the split function is the part before the delimiter --- which is ${ in our case --- this part will never contain a parameter reference, so we assign this directly to the result kept in $text.
1025 45b <substitute-chunk-args[2](
\v) ⇑45a, lang=> +≡ ▵45a 45c▿
1026 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1027 4 | text=text_array[1];
1028 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1029 We then iterate over the remaining values in the array, and substitute each reference for it's argument.
1031 45c <substitute-chunk-args[3](
\v) ⇑45a, lang=> +≡ ▵45b
1032 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1033 5 | for(t=2; t in text_array; t++) {
1034 6 | «substitute-chunk-arg 45d»
1036 |________________________________________________________________________
1039 After the split on ${ a valid parameter reference will consist of valid parameter name terminated by a close-brace }. A valid character name begins with the underscore or a letter, and may contain letters, digits or underscores.
1040 A valid looking reference that is not actually the name of a parameter will be and not substituted. This is good because there is nothing to substitute anyway, and it avoids clashes when writing code for languages where ${...} is a valid construct --- such constructs will not be interfered with unless the parameter name also matches.
1042 45d <substitute-chunk-arg[1](
\v), lang=> ≡
1043 ________________________________________________________________________
1044 1 | if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
1047 4 | text = text v[l[1]] substr(text_array[t], length(l[1])+2);
1049 6 | text = text "${" text_array[t];
1051 |________________________________________________________________________
1054 Chapter 11Language Modes & Quoting
1055 lstlistings and fangle both recognize source languages, and perform some basic parsing and syntax highlighting in the rendered document1. although lstlisting supports many more languages ^1. lstlistings can detect strings and comments within a language definition and perform suitable rendering, such as italics for comments, and visible-spaces within strings.
1056 Fangle similarly can recognize strings, and comments, etc, within a language, so that any chunks included with \chunkref{a-chunk} or <a-chunk ?> can be suitably escape or quoted.
1057 11.1 Modes explanation
1058 As an example, the C language has a few parse modes, which affect the interpretation of characters.
1059 One parse mode is the string mode. The string mode is commenced by an un-escaped quotation mark " and terminated by the same. Within the string mode, only one additional mode can be commenced, it is the backslash mode \, which is always terminated by the following character.
1060 Another mode is [ which is terminated by a ] (unless it occurs in a string).
1061 Consider this fragment of C code:
1062 do_something((things([x, y])<wide-overbrace>^(2. [ mode), get_other_things((a, "(all)"_(4. " mode)))<wide-overbrace>^(3. ( mode)))<wide-overbrace>^(1. ( mode)
1064 Mode nesting prevents the close parenthesis in the quoted string (part 4) from terminating the parenthesis mode (part 3).
1065 Each language has a set of modes, the default mode being the null mode. Each mode can lead to other modes.
1066 11.2 Modes affect included chunks
1067 For instance, consider this chunk with language=perl:
1069 47a <test:example-perl[1](
\v), lang=perl> ≡
1070 ________________________________________________________________________
1071 1 | print "hello world $0\n";
1072 |________________________________________________________________________
1075 If it were included in a chunk with language=sh, like this:
1077 47b <test:example-sh[1](
\v), lang=sh> ≡
1078 ________________________________________________________________________
1079 1 | perl -e "«test:example-perl 47a»"
1080 |________________________________________________________________________
1083 we might want fangle would to generate output like this:
1085 48a <test:example-sh.result[1](
\v), lang=sh> ≡
1086 ________________________________________________________________________
1087 1 | perl -e "print \"hello world \$0\\n\";"
1088 |________________________________________________________________________
1091 See that the double quote ", back-slash \ and $ have been quoted with a back-slash to protect them from shell interpretation.
1092 If that were then included in a chunk with language=make, like this:
1094 48b <test:example-makefile[1](
\v), lang=make> ≡
1095 ________________________________________________________________________
1097 2 | ↦«test:example-sh 47b»
1098 |________________________________________________________________________
1101 We would need the output to look like this --- note the $$ as the single $ has been makefile-quoted with another $.
1103 48c <test:example-makefile.result[1](
\v), lang=make> ≡
1104 ________________________________________________________________________
1106 2 | ↦perl -e "print \"hello world \$$0\\n\";"
1107 |________________________________________________________________________
1110 11.3 Language Mode Definitions
1111 In order to make this work, we must define a mode-tracker supporting each language, that can detect the various quoting modes, and provide a transformation that may be applied to any included text so that included text will be interpreted correctly after any interpolation that it may be subject to at run-time.
1112 For example, the sed transformation for text to be inserted into shell double-quoted strings would be something like:
1113 s/\\/\\\\/g;s/$/\\$/g;s/"/\\"/g;
1114 which would protect \ $ "
1115 All modes definitions are stored in a single multi-dimensional hash called modes:
1116 modes[language, mode, properties]
1117 The first index is the language, and the second index is the mode. The third indexes hold properties such as terminators, possible submodes, transformations, and so forth.
1119 48d <xmode:set-terminators[1](language
\v, mode
\v, terminators
\v\v), lang=> ≡
1120 ________________________________________________________________________
1121 1 | modes["${language}", "${mode}", "terminators"]="${terminators}";
1122 |________________________________________________________________________
1126 48e <xmode:set-submodes[1](language
\v, mode
\v, submodes
\v\v), lang=> ≡
1127 ________________________________________________________________________
1128 1 | modes["${language}", "${mode}", "submodes"]="${submodes}";
1129 |________________________________________________________________________
1132 A useful set of mode definitions for a nameless general C-type language is shown here.
1133 Don't be confused by the double backslash escaping needed in awk. One set of escaping is for the string, and the second set of escaping is for the regex.
1134 To do: TODO: Add =<\mode{}> command which will allow us to signify that a string is
1135 regex and thus fangle will quote it for us.
1137 Sub-modes are identified by a backslash, a double or single quote, various bracket styles or a /* comment; specifically: \ " ' { ( [ /*
1138 For each of these sub-modes modes we must also identify at a mode terminator, and any sub-modes or delimiters that may be entered2. Because we are using the sub-mode characters as the mode identifier it means we can't currently have a mode character dependant on it's context; i.e. { can't behave differently when it is inside [. ^2.
1140 49a <common-mode-definitions[1](language
\v\v), lang=> ≡ 49b▿
1141 ________________________________________________________________________
1142 1 | modes[${language}, "", "submodes"]="\\\\|\"|'|{|\\(|\\[";
1143 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1144 In the default mode, a comma surrounded by un-important white space is a delimiter of language items3. whatever a language item might be ^3. Delimiters are used so that fangle can parse and recognise arguments individually.
1146 49b <common-mode-definitions[2](language
\v\v) ⇑49a, lang=> +≡ ▵49a 49d▿
1147 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1148 2 | modes[${language}, "", "delimiters"]=" *, *";
1149 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1150 and should pass this test:
1151 To do: Why do the tests run in ?(? mode and not ?? mode
1154 49c <test:mode-definitions[1](
\v), lang=> ≡ 50g⊳
1155 ________________________________________________________________________
1156 1 | parse_chunk_args("c-like", "1,2,3", a, "");
1157 2 | if (a[1] != "1") e++;
1158 3 | if (a[2] != "2") e++;
1159 4 | if (a[3] != "3") e++;
1160 5 | if (length(a) != 3) e++;
1161 6 | «pca-test.awk:summary 62d»
1163 8 | parse_chunk_args("c-like", "joe, red", a, "");
1164 9 | if (a[1] != "joe") e++;
1165 10 | if (a[2] != "red") e++;
1166 11 | if (length(a) != 2) e++;
1167 12 | «pca-test.awk:summary 62d»
1169 14 | parse_chunk_args("c-like", "${colour}", a, "");
1170 15 | if (a[1] != "${colour}") e++;
1171 16 | if (length(a) != 1) e++;
1172 17 | «pca-test.awk:summary 62d»
1173 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1175 The backslash mode has no submodes or delimiters, and is terminated by any character. Note that we are not so much interested in evaluating or interpolating content as we are in delineating content. It is no matter that a double backslash (\\) may represent a single backslash while a backslash-newline may represent white space, but it does matter that the newline in a backslash newline should not be able to terminate a C pre-processor statement; and so the newline will be consumed by the backslash terminator however it may uultimately be interpreted.
1177 49d <common-mode-definitions[3](language
\v\v) ⇑49a, lang=> +≡ ▵49b 50f⊳
1178 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1179 3 | modes[${language}, "\\", "terminators"]=".";
1180 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1182 Common languages support two kinds of strings quoting, double quotes and single quotes.
1183 In a string we have one special mode, which is the backslash. This may escape an embedded quote and prevent us thinking that it should terminate the string.
1185 50a <mode:common-string[1](language
\v, quote
\v\v), lang=> ≡ 50b▿
1186 ________________________________________________________________________
1187 1 | modes[${language}, ${quote}, "submodes"]="\\\\";
1188 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1189 Otherwise, the string will be terminated by the same character that commenced it.
1191 50b <mode:common-string[2](language
\v, quote
\v\v) ⇑50a, lang=> +≡ ▵50a 50c▿
1192 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1193 2 | modes[${language}, ${quote}, "terminators"]=${quote};
1194 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1195 In C type languages, certain escape sequences exist in strings. We need to define mechanism to enclode any chunks included in this mode using those escape sequences. These are expressed in two parts, s meaning search, and r meaning replace.
1196 The first substitution is to replace a backslash with a double backslash. We do this first as other substitutions may introduce a backslash which we would not then want to escape again here.
1197 Note: Backslashes need double-escaping in the search pattern but not in the replacement string, hence we are replacing a literal \ with a literal \\.
1199 50c <mode:common-string[3](language
\v, quote
\v\v) ⇑50a, lang=> +≡ ▵50b 50d▿
1200 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1201 3 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\\\";
1202 4 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\\\";
1203 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1204 If the quote character occurs in the text, it should be preceded by a backslash, otherwise it would terminate the string unexpectedly.
1206 50d <mode:common-string[4](language
\v, quote
\v\v) ⇑50a, lang=> +≡ ▵50c 50e▿
1207 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1208 5 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]=${quote};
1209 6 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\" ${quote};
1210 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1211 Any newlines in the string, must be replaced by \n.
1213 50e <mode:common-string[5](language
\v, quote
\v\v) ⇑50a, lang=> +≡ ▵50d
1214 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1215 7 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\n";
1216 8 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\n";
1217 |________________________________________________________________________
1220 For the common modes, we define this string handling for double and single quotes.
1222 50f <common-mode-definitions[4](language
\v\v) ⇑49a, lang=> +≡ ⊲49d 51b⊳
1223 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1224 4 | «mode:common-string
\v(${language}
\v, "\""
\v) 50a»
1225 5 | «mode:common-string
\v(${language}
\v, "'"
\v) 50a»
1226 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1227 Working strings should pass this test:
1229 50g <test:mode-definitions[2](
\v) ⇑49c, lang=> +≡ ⊲49c 57b⊳
1230 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1231 18 | parse_chunk_args("c-like", "say \"I said, \\\"Hello, how are you\\\".\", for me", a, "");
1232 19 | if (a[1] != "say \"I said, \\\"Hello, how are you\\\".\"") e++;
1233 20 | if (a[2] != "for me") e++;
1234 21 | if (length(a) != 2) e++;
1235 22 | «pca-test.awk:summary 62d»
1236 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1237 11.3.3 Parentheses, Braces and Brackets
1238 Where quotes are closed by the same character, parentheses, brackets and braces are closed by an alternate character.
1240 51a <mode:common-brackets[1](language
\v, open
\v, close
\v\v), lang=> ≡
1241 ________________________________________________________________________
1242 1 | modes[${language}, ${open}, "submodes" ]="\\\\|\"|{|\\(|\\[|'|/\\*";
1243 2 | modes[${language}, ${open}, "delimiters"]=" *, *";
1244 3 | modes[${language}, ${open}, "terminators"]=${close};
1245 |________________________________________________________________________
1248 Note that the open is NOT a regex but the close token IS.
1249 To do: When we can quote regex we won't have to put the slashes in here
1252 51b <common-mode-definitions[5](language
\v\v) ⇑49a, lang=> +≡ ⊲50f
1253 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1254 6 | «mode:common-brackets
\v(${language}
\v, "{"
\v, "}"
\v) 51a»
1255 7 | «mode:common-brackets
\v(${language}
\v, "["
\v, "\\]"
\v) 51a»
1256 8 | «mode:common-brackets
\v(${language}
\v, "("
\v, "\\)"
\v) 51a»
1257 |________________________________________________________________________
1260 11.3.4 Customizing Standard Modes
1262 51c <mode:add-submode[1](language
\v, mode
\v, submode
\v\v), lang=> ≡
1263 ________________________________________________________________________
1264 1 | modes[${language}, ${mode}, "submodes"] = modes[${language}, ${mode}, "submodes"] "|" ${submode};
1265 |________________________________________________________________________
1269 51d <mode:add-escapes[1](language
\v, mode
\v, search
\v, replace
\v\v), lang=> ≡
1270 ________________________________________________________________________
1271 1 | escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "s"]=${search};
1272 2 | escapes[${language}, ${mode}, escapes[${language}, ${mode}], "r"]=${replace};
1273 |________________________________________________________________________
1278 We can define /* comment */ style comments and //comment style comments to be added to any language:
1280 51e <mode:multi-line-comments[1](language
\v\v), lang=> ≡
1281 ________________________________________________________________________
1282 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "/\\*"
\v) 51c»
1283 2 | modes[${language}, "/*", "terminators"]="\\*/";
1284 |________________________________________________________________________
1288 51f <mode:single-line-slash-comments[1](language
\v\v), lang=> ≡
1289 ________________________________________________________________________
1290 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "//"
\v) 51c»
1291 2 | modes[${language}, "//", "terminators"]="\n";
1292 3 | «mode:add-escapes
\v(${language}
\v, "//"
\v, "\n"
\v, "\n//"
\v) 51d»
1293 |________________________________________________________________________
1296 We can also define # comment style comments (as used in awk and shell scripts) in a similar manner.
1297 To do: I'm having to use # for hash and ¯extbackslash{} for and have hacky work-arounds in the parser for now
1300 51g <mode:add-hash-comments[1](language
\v\v), lang=> ≡
1301 ________________________________________________________________________
1302 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "#"
\v) 51c»
1303 2 | modes[${language}, "#", "terminators"]="\n";
1304 3 | «mode:add-escapes
\v(${language}
\v, "#"
\v, "\n"
\v, "\n#"
\v) 51d»
1305 |________________________________________________________________________
1308 In C, the # denotes pre-processor directives which can be multi-line
1310 51h <mode:add-hash-defines[1](language
\v\v), lang=> ≡
1311 ________________________________________________________________________
1312 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "#"
\v) 51c»
1313 2 | modes[${language}, "#", "submodes" ]="\\\\";
1314 3 | modes[${language}, "#", "terminators"]="\n";
1315 4 | «mode:add-escapes
\v(${language}
\v, "#"
\v, "\n"
\v, "\\\\\n"
\v) 51d»
1316 |________________________________________________________________________
1320 52a <mode:quote-dollar-escape[1](language
\v, quote
\v\v), lang=> ≡
1321 ________________________________________________________________________
1322 1 | escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\$";
1323 2 | escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\$";
1324 |________________________________________________________________________
1327 We can add these definitions to various languages
1329 52b <mode-definitions[1](
\v), lang=> ≡ 53a⊳
1330 ________________________________________________________________________
1331 1 | «common-mode-definitions
\v("c-like"
\v) 49a»
1333 3 | «common-mode-definitions
\v("c"
\v) 49a»
1334 4 | «mode:multi-line-comments
\v("c"
\v) 51e»
1335 5 | «mode:single-line-slash-comments
\v("c"
\v) 51f»
1336 6 | «mode:add-hash-defines
\v("c"
\v) 51h»
1338 8 | «common-mode-definitions
\v("awk"
\v) 49a»
1339 9 | «mode:add-hash-comments
\v("awk"
\v) 51g»
1340 10 | «mode:add-naked-regex
\v("awk"
\v) 52g»
1341 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1342 The awk definitions should allow a comment block like this:
1344 52c <test:comment-quote[1](
\v), lang=awk> ≡
1345 ________________________________________________________________________
1346 1 | # Comment: «test:comment-text 52d»
1347 |________________________________________________________________________
1351 52d <test:comment-text[1](
\v), lang=> ≡
1352 ________________________________________________________________________
1353 1 | Now is the time for
1354 2 | the quick brown fox to bring lemonade
1356 |________________________________________________________________________
1359 to come out like this:
1361 52e <test:comment-quote:result[1](
\v), lang=> ≡
1362 ________________________________________________________________________
1363 1 | # Comment: Now is the time for
1364 2 | #the quick brown fox to bring lemonade
1366 |________________________________________________________________________
1369 The C definition for such a block should have it come out like this:
1371 52f <test:comment-quote:C-result[1](
\v), lang=> ≡
1372 ________________________________________________________________________
1373 1 | # Comment: Now is the time for\
1374 2 | the quick brown fox to bring lemonade\
1376 |________________________________________________________________________
1380 This pattern is incomplete, but meant to detect naked regular expressions in awk and perl; e.g. /.*$/, however required capabilities are not present.
1381 Current it only detects regexes anchored with ^ as used in fangle.
1382 For full regex support, modes need to be named not after their starting character, but some other more fully qualified name.
1384 52g <mode:add-naked-regex[1](language
\v\v), lang=> ≡
1385 ________________________________________________________________________
1386 1 | «mode:add-submode
\v(${language}
\v, ""
\v, "/\\^"
\v) 51c»
1387 2 | modes[${language}, "/^", "terminators"]="/";
1388 |________________________________________________________________________
1393 53a <mode-definitions[2](
\v) ⇑52b, lang=> +≡ ⊲52b 53b▿
1394 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1395 11 | «common-mode-definitions
\v("perl"
\v) 49a»
1396 12 | «mode:multi-line-comments
\v("perl"
\v) 51e»
1397 13 | «mode:add-hash-comments
\v("perl"
\v) 51g»
1398 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1399 Still need to add add s/, submode /, terminate both with //. This is likely to be impossible as perl regexes can contain perl.
1401 Shell single-quote strings are different to other strings and have no escape characters. The only special character is the single quote ' which always closes the string. Therefore we cannot use <common-mode-definitions
\v("sh"
\v) 49a> but we will invoke most of it's definition apart from single-quote strings.
1403 53b <mode-definitions[3](
\v) ⇑52b, lang=awk> +≡ ▵53a 54a⊳
1404 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1405 14 | modes["sh", "", "submodes"]="\\\\|\"|'|{|\\(|\\[|\\$\\(";
1406 15 | modes["sh", "\\", "terminators"]=".";
1408 17 | modes["sh", "\"", "submodes"]="\\\\|\\$\\(";
1409 18 | modes["sh", "\"", "terminators"]="\"";
1410 19 | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\\\\";
1411 20 | escapes["sh", "\"", escapes["sh", "\""], "r"]="\\\\";
1412 21 | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\"";
1413 22 | escapes["sh", "\"", escapes["sh", "\""], "r"]="\\" "\"";
1414 23 | escapes["sh", "\"", ++escapes["sh", "\""], "s"]="\n";
1415 24 | escapes["sh", "\"", escapes["sh", "\""], "r"]="\\n";
1417 26 | modes["sh", "'", "terminators"]="'";
1418 27 | escapes["sh", "'", ++escapes["sh", "'"], "s"]="'";
1419 28 | escapes["sh", "'", escapes["sh", "'"], "r"]="'\\'" "'";
1420 29 | «mode:common-brackets
\v("sh"
\v, "$("
\v, "\\)"
\v) 51a»
1421 30 | «mode:add-tunnel
\v("sh"
\v, "$("
\v, ""
\v) 53c»
1422 31 | «mode:common-brackets
\v("sh"
\v, "{"
\v, "}"
\v) 51a»
1423 32 | «mode:common-brackets
\v("sh"
\v, "["
\v, "\\]"
\v) 51a»
1424 33 | «mode:common-brackets
\v("sh"
\v, "("
\v, "\\)"
\v) 51a»
1425 34 | «mode:add-hash-comments
\v("sh"
\v) 51g»
1426 35 | «mode:quote-dollar-escape
\v("sh"
\v, "\""
\v) 52a»
1427 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1428 The definition of add-tunnel is:
1430 53c <mode:add-tunnel[1](language
\v, mode
\v, tunnel
\v\v), lang=> ≡
1431 ________________________________________________________________________
1432 1 | escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "tunnel"]=${tunnel};
1433 |________________________________________________________________________
1437 BUGS: makefile tab mode is terminated by newline, but chunks never end in a newline! So tab mode is never closed unless there is a trailing blank line!
1438 For makefiles, we currently recognize 2 modes: the null mode and ↦ mode, which is tabbed mode and contains the makefile recipie.
1441 54a <mode-definitions[4](
\v) ⇑52b, lang=awk> +≡ ⊲53b 54b▿
1442 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1443 36 | modes["make", "", "submodes"]="↦";
1444 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1445 In the null mode the only escape is $ which must be converted to $$, and hash-style comments. POSIX requires that line-continuations extend hash-style comments and so fangle-style transformations to replicate the hash at the start of each line is not strictly required, however it is harmless, easier to read, and required by some implementations of make which do not implement POSIX requirements correctly.
1447 54b <mode-definitions[5](
\v) ⇑52b, lang=awk> +≡ ▵54a 55d⊳
1448 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1449 37 | escapes["make", "", ++escapes["make", ""], "s"]="\\$";
1450 38 | escapes["make", "", escapes["make", ""], "r"]="$$";
1451 39 | «mode:add-hash-comments
\v("make"
\v) 51g»
1452 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1453 Tabbed mode is harder to manage, as the GNU Make Manual says in the section on splitting lines4. http://www.gnu.org/s/hello/manual/make/Splitting-Lines.html ^4. There is no obvious way to escape a multi-line text that occurs as part of a makefile recipe.
1454 Traditionally, if the newline's in the shell script all occur at points of top-level shell syntax, then we could replace them with ;\n↦and largely get the right effect.
1456 54c <test:make:1[1](
\v), lang=make> ≡
1457 ________________________________________________________________________
1460 3 | ↦«test:make:1-inc
\v($@) 54d»
1461 |________________________________________________________________________
1467 54d <test:make:1-inc[1](target
\v\v), lang=sh> ≡
1468 ________________________________________________________________________
1469 1 | if test "${target}" = "all"
1470 2 | then echo yes, all
1471 3 | else echo "${target}" | sed -e '/^\//{
1475 |________________________________________________________________________
1478 The two chunks above could reasonably produce something like this:
1480 ?a <test:make:1.result.bad[1](
\v), lang=make> ≡
1481 ________________________________________________________________________
1484 3 | ↦if test "$@" = "all" ;\
1485 4 | ↦then echo yes, all ;\
1486 5 | ↦else echo "$@" | sed -e '/^\//{ ;\
1490 |________________________________________________________________________
1493 However ;\ is not a proper continuation inside a multi-line sed script. There is no simple continuation that fangle could use — and in any case it would depend on what type of quote marks were used in the bash that contained the sed.
1494 We would prefer to use a more intuitive single backslash at the end of the line, giving these results.
1496 54a <test:make:1.result[1](
\v), lang=make> ≡
1497 ________________________________________________________________________
1500 3 | ↦if test "$$@" = "all"\
1501 4 | ↦ then echo yes, all\
1502 5 | ↦ else echo "$$@" | sed -e '/^\//{\
1506 |________________________________________________________________________
1509 The difficulty lies in the way that make handles the recipe. Each line of the recipe is invoked as a separate shell command (using $(SHELL) -c) unless the last character of the line was a backslash. In such a case, the backslash and the newline and the nextline are handed to the shell (although the tab character that prefixes the next line is stripped).
1510 This behaviour makes it impossible to hand a newline character to the shell unless it is prefixed by a backslash. If an included shell fragment contained strings with literal newline characters then there would be no easy way to escape these and preserve the value of the string.
1511 A different style of makefile construction might be used — the recipe could be stored in a target specific variable5. http://www.gnu.org/s/hello/manual/make/Target_002dspecific.html ^5 which contains the recipe with a more normal escape mechanism.
1512 A better solution is to use a shell helper that strips the back-slash which precedes the newline character and then passes the arguments to the normal shell.
1513 Because this is a simple operation and because bash is so flexible, this can be managed in a single line within the makefile itself.
1514 As a newline will only exist when preceded by the backslash, and as the purpose of the backash is to protect th newline, that is needed is to remove any backslash that is followed by a newline.
1515 Bash is capable of doing this with its pattern substitution. If A=123:=456:=789 then ${A//:=/=} will be 123=456=789. We don't want to just perform the substitution in a single variable but in fact in all of $@'', however bash will repeat substitution over all members of an array, so this is done automatically.
1516 In bash, $'\012' represents the newline character (expressed as an octal escape sequence), so this expression will replace backslash-newline with a single newline.
1518 55a <fix-requote-newline[1](
\v), lang=sh> ≡
1519 ________________________________________________________________________
1520 1 | "${@//\\$'\012'/$'\012'}"
1521 |________________________________________________________________________
1524 We use this as part of a larger statement which will invoke such a transformed command ine using any particular shell. The trailing -- prevents any options in the command line from being interpreted as options to our bash command — instead they will be transformed and passed to the inner shell which is invoked with exec so that our fixup-shell does not hang around longer than is needed.
1526 55b <fix-make-shell[1](shell
\v\v), lang=sh> ≡
1527 ________________________________________________________________________
1528 1 | bash -c 'exec ${shell} «fix-requote-newline 55a»' --
1529 |________________________________________________________________________
1532 We can then cinlude a line like this in our makefiles. We should rather pass $(SHELL) as the chunk argument than bash, but currently fangle will not track which nested-inclusion level the argument comes from and will quote the $ in $(SHELL) in the same way it quotes a $ that may occur in the bash script, so this would come out as $$(SHELL) and have the wrong effect.
1534 55c <make-fix-make-shell[1](
\v), lang=> ≡
1535 ________________________________________________________________________
1536 1 | SHELL:=«fix-make-shell
\v(bash
\v) 55b»
1537 |________________________________________________________________________
1540 The full escaped and quoted text with $(SHELL) and suitale for inclusion in a Makefile is:
1541 SHELL:=bash -c 'exec $(SHELL) "$${@//\\$$'\''\012'\''/$$'\''\012'\''}"' --
1542 Based on this, we just need to escape newlines (in tabbed mode) with a regular backslash:
1543 Note that terminators applies to literal, not included text, escapes apply to included, not literal text; also that the tab character is hard-wired into the pattern, and that the make variable .RECIPEPREFIX might change this to something else.
1545 55d <mode-definitions[6](
\v) ⇑52b, lang=awk> +≡ ⊲54b
1546 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1547 40 | modes["make", "↦", "terminators"]="\\n";
1548 41 | escapes["make", "↦", ++escapes["make", "↦"], "s"]="\\n";
1549 42 | escapes["make", "↦", escapes["make", "↦"], "r"]="\\\n↦";
1550 |________________________________________________________________________
1553 With this improved quoting, the test on 54c will actually produce this:
1555 55e <test:make:1.result-actual[1](
\v), lang=make> ≡
1556 ________________________________________________________________________
1559 3 | ↦if test "$$@" = "all"\
1560 4 | ↦ then echo yes, all\
1561 5 | ↦ else echo not all\
1563 |________________________________________________________________________
1566 The chunk argument $@ has been quoted (which would have been fine if we were passing the name of a shell variable), and the other shell lines are (harmlessly) indented by 1 space as part of fangle indent-matching which should have taken into account the expanded tab size, and should generally take into account the expanded prefix of the line whose indent it is trying to match, but which in this case we want to have no effect at all!
1567 To do: The $@ was passed from a make fragment. In what cases should it be converted to $$@?
1568 Do we need to track the language of sources of arguments?
1570 A more ugly work-around until this problem can be solved would be to use this notation:
1572 56a <test:make:2[1](
\v), lang=make> ≡
1573 ________________________________________________________________________
1576 3 | ↦ARG="$@"; «test:make:1-inc
\v($ARG) 54d»
1577 |________________________________________________________________________
1580 which produces this output which is more useful (because it works):
1582 56b <test:make:2.result[1](
\v), lang=make> ≡
1583 ________________________________________________________________________
1586 3 | ↦ARG="$@"; if test "$$ARG" = "all"\
1587 4 | ↦ then echo yes, all\
1588 5 | ↦ else echo "$$ARG" | sed -e '/^\//{\
1592 |________________________________________________________________________
1595 11.4 Quoting scenarios
1596 11.4.1 Direct quoting
1597 He we give examples of various quoting scenarios and discuss what the expected outcome might be and how this could be obtained.
1599 56c <test:q:1[1](
\v), lang=sh> ≡
1600 ________________________________________________________________________
1601 1 | echo "$(«test:q:1-inc 56d»)"
1602 |________________________________________________________________________
1606 56d <test:q:1-inc[1](
\v), lang=sh> ≡
1607 ________________________________________________________________________
1609 |________________________________________________________________________
1612 Should this examples produce echo "$(echo "hello")" or echo "$(echo \"hello\")" ?
1613 This depends on what the author intended, but we must provde a way to express that intent.
1614 We might argue that as both chunks have lang=sh the intent must have been to quote the included chunk — but consider that this might be shell script that writes shell script.
1615 If <test:q:1-inc 56d> had lang=text then it certainly would have been right to quote it, which leads us to ask: in what ways can we reduce quoting if lang of the included chunk is compatible with the lang of the including chunk?
1616 If we take a completely nested approach then even though $( mode might do no quoting of it's own, " mode will still do it's own quoting. We need a model where the nested $( mode will prevent " from quoting.
1617 This leads rise to the tunneling feature. In bash, the $( gives rise to a new top-level parsing scenario, so we need to enter the null mode, and also ignore any quoting and then undo-this when the $( mode is terminated by the corresponding close ).
1618 We shall say that tunneling is when a mode in a language ignores other modes in the same language and arrives back at an earlier null mode of the same language.
1619 In example <test:q:1 56c> above, the nesting of modes is: null, ", $(
1620 When mode $( is commenced, the stack of nest modes will be traversed. If the null mode can be found in the same language, without the language varying, then a tunnel will be established so that the intervening modes, " in this case, can be skipped when the modes are enumerated to quote the texted being emitted.
1621 In such a case, the correct result would be:
1623 57a <test:q:1.result[1](
\v), lang=sh> ≡
1624 ________________________________________________________________________
1625 1 | echo "$(echo "hello")"
1626 |________________________________________________________________________
1630 Also, the parser must return any spare text at the end that has not been processed due to a mode terminator being found.
1632 57b <test:mode-definitions[3](
\v) ⇑49c, lang=> +≡ ⊲50g 57c▿
1633 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1634 23 | rest = parse_chunk_args("c-like", "1, 2, 3) spare", a, "(");
1635 24 | if (a[1] != 1) e++;
1636 25 | if (a[2] != 2) e++;
1637 26 | if (a[3] != 3) e++;
1638 27 | if (length(a) != 3) e++;
1639 28 | if (rest != " spare") e++;
1640 29 | «pca-test.awk:summary 62d»
1641 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1642 We must also be able to parse the example given earlier.
1644 57c <test:mode-definitions[4](
\v) ⇑49c, lang=> +≡ ▵57b
1645 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1646 30 | parse_chunk_args("c-like", "things[x, y], get_other_things(a, \"(all)\"), 99", a, "(");
1647 31 | if (a[1] != "things[x, y]") e++;
1648 32 | if (a[2] != "get_other_things(a, \"(all)\")") e++;
1649 33 | if (a[3] != "99") e++;
1650 34 | if (length(a) != 3) e++;
1651 35 | «pca-test.awk:summary 62d»
1652 |________________________________________________________________________
1655 11.6 A non-recursive mode tracker
1656 As each chunk is output a new mode tracker for that language is initialized in it's normal state. As text is output for that chunk the output mode is tracked. When a new chunk is included, a transformation appropriate to that mode is selected and pushed onto a stack of transformations. Any text to be output is passed through this stack of transformations.
1657 It remains to consider if the chunk-include function should return it's generated text so that the caller can apply any transformations (and formatting), or if it should apply the stack of transformations itself.
1658 Note that the transformed included text should have the property of not being able to change the mode in the current chunk.
1659 To do: Note chunk parameters should probably also be transformed
1662 The mode tracker holds its state in a stack based on a numerically indexed hash. This function, when passed an empty hash, will intialize it.
1664 58a <new_mode_tracker()[1](
\v), lang=> ≡
1665 ________________________________________________________________________
1666 1 | function new_mode_tracker(context, language, mode) {
1667 2 | context[""] = 0;
1668 3 | context[0, "language"] = language;
1669 4 | context[0, "mode"] = mode;
1671 |________________________________________________________________________
1674 Awk functions cannot return an array, but arrays are passed by reference. Because of this we must create the array first and pass it in, so we have a fangle macro to do this:
1676 58b <new-mode-tracker[1](context
\v, language
\v, mode
\v\v), lang=awk> ≡
1677 ________________________________________________________________________
1678 1 | «awk-delete-array
\v(${context}
\v) 37d»
1679 2 | new_mode_tracker(${context}, ${language}, ${mode});
1680 |________________________________________________________________________
1684 And for tracking modes, we dispatch to a mode-tracker action based on the current language
1686 58c <mode_tracker[1](
\v), lang=awk> ≡ 58d▿
1687 ________________________________________________________________________
1688 1 | function push_mode_tracker(context, language, mode,
1692 5 | if (! ("" in context)) {
1693 6 | «new-mode-tracker
\v(context
\v, language
\v, mode
\v) 58b»
1696 9 | top = context[""];
1697 10 | # if (context[top, "language"] == language && mode=="") mode = context[top, "mode"];
1698 11 | if (context[top, "language"] == language && context[top, "mode"] == mode) return top - 1;
1701 14 | context[top, "language"] = language;
1702 15 | context[top, "mode"] = mode;
1703 16 | context[""] = top;
1705 18 | return old_top;
1707 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1709 58d <mode_tracker[2](
\v) ⇑58c, lang=> +≡ ▵58c 58e▿
1710 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1711 20 | function dump_mode_tracker(context,
1714 23 | for(c=0; c <= context[""]; c++) {
1715 24 | printf(" %2d %s:%s\n", c, context[c, "language"], context[c, "mode"]) > "/dev/stderr";
1716 25 | # for(d=1; ( (c, "values", d) in context); d++) {
1717 26 | # printf(" %2d %s\n", d, context[c, "values", d]) > "/dev/stderr";
1721 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1723 58e <mode_tracker[3](
\v) ⇑58c, lang=> +≡ ▵58d 63a⊳
1724 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1725 30 | function pop_mode_tracker(context, context_origin)
1727 32 | if ( (context_origin) && ("" in context) && context[""] != (1+context_origin) && context[""] != context_origin) {
1728 33 | print "Context level: " context[""] ", origin: " context_origin "\n" > "/dev/stderr"
1731 36 | context[""] = context_origin;
1734 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1735 This implies that any chunk must be syntactically whole; for instance, this is fine:
1737 59a <test:whole-chunk[1](
\v), lang=> ≡
1738 ________________________________________________________________________
1740 2 | «test:say-hello 59b»
1742 |________________________________________________________________________
1746 59b <test:say-hello[1](
\v), lang=> ≡
1747 ________________________________________________________________________
1749 |________________________________________________________________________
1752 But this is not fine; the chunk <test:hidden-else 59d> is not properly cromulent.
1754 59c <test:partial-chunk[1](
\v), lang=> ≡
1755 ________________________________________________________________________
1757 2 | «test:hidden-else 59d»
1759 |________________________________________________________________________
1763 59d <test:hidden-else[1](
\v), lang=> ≡
1764 ________________________________________________________________________
1765 1 | print "I'm fine";
1767 3 | print "I'm not";
1768 |________________________________________________________________________
1771 These tests will check for correct behaviour:
1773 59e <test:cromulence[1](
\v), lang=> ≡
1774 ________________________________________________________________________
1775 1 | echo Cromulence test
1776 2 | passtest $FANGLE -Rtest:whole-chunk $TXT_SRC &>/dev/null || ( echo "Whole chunk failed" && exit 1 )
1777 3 | failtest $FANGLE -Rtest:partial-chunk $TXT_SRC &>/dev/null || ( echo "Partial chunk failed" && exit 1 )
1778 |________________________________________________________________________
1782 We must avoid recursion as a language construct because we intend to employ mode-tracking to track language mode of emitted code, and the code is emitted from a function which is itself recursive, so instead we implement psuedo-recursion using our own stack based on a hash.
1784 59f <mode_tracker()[1](
\v), lang=awk> ≡ 60a⊳
1785 ________________________________________________________________________
1786 1 | function mode_tracker(context, text, values,
1787 2 | # optional parameters
1789 4 | mode, submodes, language,
1790 5 | cindex, c, a, part, item, name, result, new_values, new_mode,
1791 6 | delimiters, terminators)
1793 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1794 We could be re-commencing with a valid context, so we need to setup the state according to the last context.
1796 60a <mode_tracker()[2](
\v) ⇑59f, lang=> +≡ ⊲59f 60d▿
1797 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1798 8 | cindex = context[""] + 0;
1799 9 | mode = context[cindex, "mode"];
1800 10 | language = context[cindex, "language" ];
1801 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1802 First we construct a single large regex combining the possible sub-modes for the current mode along with the terminators for the current mode.
1804 60b <parse_chunk_args-reset-modes[1](
\v), lang=> ≡ 60c▿
1805 ________________________________________________________________________
1806 1 | submodes=modes[language, mode, "submodes"];
1808 3 | if ((language, mode, "delimiters") in modes) {
1809 4 | delimiters = modes[language, mode, "delimiters"];
1810 5 | if (length(submodes)>0) submodes = submodes "|";
1811 6 | submodes=submodes delimiters;
1812 7 | } else delimiters="";
1813 8 | if ((language, mode, "terminators") in modes) {
1814 9 | terminators = modes[language, mode, "terminators"];
1815 10 | if (length(submodes)>0) submodes = submodes "|";
1816 11 | submodes=submodes terminators;
1817 12 | } else terminators="";
1818 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1819 If we don't find anything to match on --- probably because the language is not supported --- then we return the entire text without matching anything.
1821 60c <parse_chunk_args-reset-modes[2](
\v) ⇑60b, lang=> +≡ ▵60b
1822 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1823 13 | if (! length(submodes)) return text;
1824 |________________________________________________________________________
1828 60d <mode_tracker()[3](
\v) ⇑59f, lang=> +≡ ▵60a 60e▿
1829 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1830 11 | «parse_chunk_args-reset-modes 60b»
1831 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1832 We then iterate the text (until there is none left) looking for sub-modes or terminators in the regex.
1834 60e <mode_tracker()[4](
\v) ⇑59f, lang=> +≡ ▵60d 60f▿
1835 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1836 12 | while((cindex >= 0) && length(text)) {
1837 13 | if (match(text, "(" submodes ")", a)) {
1838 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1839 A bug that creeps in regularly during development is bad regexes of zero length which result in an infinite loop (as no text is consumed), so I catch that right away with this test.
1841 60f <mode_tracker()[5](
\v) ⇑59f, lang=> +≡ ▵60e 61a⊳
1842 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1843 14 | if (RLENGTH<1) {
1844 15 | error(sprintf("Internal error, matched zero length submode, should be impossible - likely regex computation error\n" \
1845 16 | "Language=%s\nmode=%s\nmatch=%s\n", language, mode, submodes));
1847 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1848 part is defined as the text up to the sub-mode or terminator, and this is appended to item --- which is the current text being gathered. If a mode has a delimiter, then item is reset each time a delimiter is found.
1849 ("hello_item, there_item")<wide-overbrace>^item, (he said.)<wide-overbrace>^item
1851 61a <mode_tracker()[6](
\v) ⇑59f, lang=> +≡ ⊲60f 61b▿
1852 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1853 18 | part = substr(text, 1, RSTART -1);
1854 19 | item = item part;
1855 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1856 We must now determine what was matched. If it was a terminator, then we must restore the previous mode.
1858 61b <mode_tracker()[7](
\v) ⇑59f, lang=> +≡ ▵61a 61c▿
1859 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1860 20 | if (match(a[1], "^" terminators "$")) {
1861 21 | #printf("%2d EXIT MODE [%s] by [%s] [%s]\n", cindex, mode, a[1], text) > "/dev/stderr"
1862 22 | context[cindex, "values", ++context[cindex, "values"]] = item;
1863 23 | delete context[cindex];
1864 24 | context[""] = --cindex;
1865 25 | if (cindex>=0) {
1866 26 | mode = context[cindex, "mode"];
1867 27 | language = context[cindex, "language"];
1868 28 | «parse_chunk_args-reset-modes 60b»
1870 30 | item = item a[1];
1871 31 | text = substr(text, 1 + length(part) + length(a[1]));
1873 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1874 If a delimiter was matched, then we must store the current item in the parsed values array, and reset the item.
1876 61c <mode_tracker()[8](
\v) ⇑59f, lang=> +≡ ▵61b 61d▿
1877 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1878 33 | else if (match(a[1], "^" delimiters "$")) {
1879 34 | if (cindex==0) {
1880 35 | context[cindex, "values", ++context[cindex, "values"]] = item;
1883 38 | item = item a[1];
1885 40 | text = substr(text, 1 + length(part) + length(a[1]));
1887 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1888 otherwise, if a new submode is detected (all submodes have terminators), we must create a nested parse context until we find the terminator for this mode.
1890 61d <mode_tracker()[9](
\v) ⇑59f, lang=> +≡ ▵61c 62a⊳
1891 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1892 42 | else if ((language, a[1], "terminators") in modes) {
1893 43 | #check if new_mode is defined
1894 44 | item = item a[1];
1895 45 | #printf("%2d ENTER MODE [%s] in [%s]\n", cindex, a[1], text) > "/dev/stderr"
1896 46 | text = substr(text, 1 + length(part) + length(a[1]));
1897 47 | context[""] = ++cindex;
1898 48 | context[cindex, "mode"] = a[1];
1899 49 | context[cindex, "language"] = language;
1901 51 | «parse_chunk_args-reset-modes 60b»
1903 53 | error(sprintf("Submode '%s' set unknown mode in text: %s\nLanguage %s Mode %s\n", a[1], text, language, mode));
1904 54 | text = substr(text, 1 + length(part) + length(a[1]));
1907 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1908 In the final case, we parsed to the end of the string. If the string was entire, then we should have no nested mode context, but if the string was just a fragment we may have a mode context which must be preserved for the next fragment. Todo: Consideration ought to be given if sub-mode strings are split over two fragments.
1910 62a <mode_tracker()[10](
\v) ⇑59f, lang=> +≡ ⊲61d
1911 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1913 58 | context[cindex, "values", ++context[cindex, "values"]] = item text;
1919 64 | context["item"] = item;
1921 66 | if (length(item)) context[cindex, "values", ++context[cindex, "values"]] = item;
1924 |________________________________________________________________________
1927 11.6.3.1 One happy chunk
1928 All the mode tracker chunks are referred to here:
1930 62b <mode-tracker[1](
\v), lang=> ≡
1931 ________________________________________________________________________
1932 1 | «new_mode_tracker() 58a»
1933 2 | «mode_tracker() 59f»
1934 |________________________________________________________________________
1938 We can test this function like this:
1940 62c <pca-test.awk[1](
\v), lang=awk> ≡
1941 ________________________________________________________________________
1943 2 | «mode-tracker 62b»
1944 3 | «parse_chunk_args() ?»
1947 6 | «mode-definitions 52b»
1949 8 | «test:mode-definitions 49c»
1951 |________________________________________________________________________
1955 62d <pca-test.awk:summary[1](
\v), lang=awk> ≡
1956 ________________________________________________________________________
1958 2 | printf "Failed " e
1960 4 | print "a[" b "] => " a[b];
1967 |________________________________________________________________________
1970 which should give this output:
1972 62e <pca-test.awk-results[1](
\v), lang=> ≡
1973 ________________________________________________________________________
1974 1 | a[foo.quux.quirk] =>
1975 2 | a[foo.quux.a] => fleeg
1976 3 | a[foo.bar] => baz
1978 5 | a[name] => freddie
1979 |________________________________________________________________________
1982 11.7 Escaping and Quoting
1983 For the time being and to get around TeXmacs inability to export a TAB character, the right arrow ↦ whose UTF-8 sequence is ...
1986 Another special character is used, the left-arrow ↤ with UTF-8 sequence 0xE2 0x86 0xA4 is used to strip any preceding white space as a way of un-tabbing and removing indent that has been applied — this is important for bash here documents, and the like. It's a filthy hack.
1987 To do: remove the hack
1990 63a <mode_tracker[4](
\v) ⇑58c, lang=> +≡ ⊲58e 63b▿
1991 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1993 39 | function untab(text) {
1994 40 | gsub("[[:space:]]*\xE2\x86\xA4","", text);
1997 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1998 Each nested mode can optionally define a set of transforms to be applied to any text that is included from another language.
1999 This code can perform transforms from index c downwards.
2001 63b <mode_tracker[5](
\v) ⇑58c, lang=awk> +≡ ▵63a 58c⊳
2002 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2003 43 | function transform_escape(context, text, top,
2004 44 | c, cp, cpl, s, r)
2006 46 | for(c = top; c >= 0; c--) {
2007 47 | if ( (context[c, "language"], context[c, "mode"]) in escapes) {
2008 48 | cpl = escapes[context[c, "language"], context[c, "mode"]];
2009 49 | for (cp = 1; cp <= cpl; cp ++) {
2010 50 | s = escapes[context[c, "language"], context[c, "mode"], cp, "s"];
2011 51 | r = escapes[context[c, "language"], context[c, "mode"], cp, "r"];
2012 52 | if (length(s)) {
2013 53 | gsub(s, r, text);
2015 55 | if ( (context[c, "language"], context[c, "mode"], cp, "t") in escapes ) {
2016 56 | quotes[src, "t"] = escapes[context[c, "language"], context[c, "mode"], cp, "t"];
2023 63 | function dump_escaper(quotes, r, cc) {
2024 64 | for(cc=1; cc<=c; cc++) {
2025 65 | printf("%2d s[%s] r[%s]\n", cc, quotes[cc, "s"], quotes[cc, "r"]) > "/dev/stderr"
2028 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2030 63c <test:escapes[1](
\v), lang=sh> ≡
2031 ________________________________________________________________________
2032 1 | echo escapes test
2033 2 | passtest $FANGLE -Rtest:comment-quote $TXT_SRC &>/dev/null || ( echo "Comment-quote failed" && exit 1 )
2034 |________________________________________________________________________
2037 Chapter 12Recognizing Chunks
2038 Fangle recognizes noweb chunks, but as we also want better LaTeX integration we will recognize any of these:
2039 • notangle chunks matching the pattern ^<<.*?>>=
2040 • chunks beginning with \begin{lstlistings}, possibly with \Chunk{...} on the previous line
2041 • an older form I have used, beginning with \begin{Chunk}[options] --- also more suitable for plain LaTeX users1. Is there such a thing as plain LaTeX? ^1.
2043 The variable chunking is used to signify that we are processing a code chunk and not document. In such a state, input lines will be assigned to the current chunk; otherwise they are ignored.
2045 We don't handle TeXmacs files natively yet, but rather instead emit unicode character sequences to mark up the text-export file which we do process.
2046 These hacks detect the unicode character sequences and retro-fit in the old TeX parsing.
2047 We convert ↦ into a tab character.
2049 65a <recognize-chunk[1](
\v), lang=> ≡ 65b▿
2050 ________________________________________________________________________
2053 2 | # gsub("\n*$","");
2054 3 | # gsub("\n", " ");
2057 6 | /\xE2\x86\xA6/ {
2058 7 | gsub("\\xE2\\x86\\xA6", "\x09");
2060 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2061 TeXmacs back-tick handling is obscure, and a cut-n-paste back-tick from a shell window comes out as a unicode sequence2. that won't export to html, except as a NULL character (literal 0x00) ^2 that is fixed-up here.
2063 65b <recognize-chunk[2](
\v) ⇑65a, lang=> +≡ ▵65a 66a⊳
2064 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2066 10 | /\xE2\x80\x98/ {
2067 11 | gsub("\\xE2\\x80\\x98", "‘");
2069 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2070 In the TeXmacs output, the start of a chunk will appear like this:
2071 5b<example-chunk^K[1](arg1,^K arg2^K^K), lang=C> ≡
2072 We detect the the start of a TeXmacs chunk by detecting the ≡ symbol which occurs near the end of the line. We obtain the chunk name, the chunk parameters, and the chunk language.
2074 66a <recognize-chunk[3](
\v) ⇑65a, lang=> +≡ ⊲65b 66b▿
2075 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2077 14 | /\xE2\x89\xA1/ {
2078 15 | if (match($0, "^ *([^[ ]* |)<([^[ ]*)\\[[0-9]*\\][(](.*)[)].*, lang=([^ ]*)>", line)) {
2079 16 | next_chunk_name=line[2];
2080 17 | get_texmacs_chunk_args(line[3], next_chunk_params);
2081 18 | gsub(ARG_SEPARATOR ",? ?", ";", line[3]);
2082 19 | params = "params=" line[3];
2083 20 | if ((line[4])) {
2084 21 | params = params ",language=" line[4]
2086 23 | get_tex_chunk_args(params, next_chunk_opts);
2087 24 | new_chunk(next_chunk_name, next_chunk_opts, next_chunk_params);
2088 25 | texmacs_chunking = 1;
2090 27 | # warning(sprintf("Unexpected chunk match: %s\n", $_))
2094 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2096 Our current scheme is to recognize the new lstlisting chunks, but these may be preceded by a \Chunk command which in L Y X is a more convenient way to pass the chunk name to the \begin{lstlistings} command, and a more visible way to specify other lstset settings.
2097 The arguments to the \Chunk command are a name, and then a comma-seperated list of key-value pairs after the manner of \lstset. (In fact within the LaTeX \Chunk macro (section 17.2.1) the text name= is prefixed to the argument which is then literally passed to \lstset).
2099 66b <recognize-chunk[4](
\v) ⇑65a, lang=awk> +≡ ▵66a 66c▿
2100 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2102 32 | if (match($0, "^\\\\Chunk{ *([^ ,}]*),?(.*)}", line)) {
2103 33 | next_chunk_name = line[1];
2104 34 | get_tex_chunk_args(line[2], next_chunk_opts);
2108 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2109 We also make a basic attempt to parse the name out of the \lstlistings[name=chunk-name] text, otherwise we fall back to the name found in the previous chunk command. This attempt is very basic and doesn't support commas or spaces or square brackets as part of the chunkname. We also recognize \begin{Chunk} which is convenient for some users3. but not yet supported in the LaTeX macros ^3.
2111 66c <recognize-chunk[5](
\v) ⇑65a, lang=> +≡ ▵66b 67a⊳
2112 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2113 38 | /^\\begin{lstlisting}|^\\begin{Chunk}/ {
2114 39 | if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
2115 40 | new_chunk(line[1]);
2117 42 | new_chunk(next_chunk_name, next_chunk_opts);
2122 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2125 A chunk body in TeXmacs ends with |________... if it is the final chunklet of a chunk, or if there are further chunklets it ends with |\/\/\/... which is a depiction of a jagged line of torn paper.
2127 67a <recognize-chunk[6](
\v) ⇑65a, lang=> +≡ ⊲66c 67b▿
2128 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2129 47 | /^ *\|____________*/ && texmacs_chunking {
2130 48 | active_chunk="";
2131 49 | texmacs_chunking=0;
2134 52 | /^ *\|\/\\/ && texmacs_chunking {
2135 53 | texmacs_chunking=0;
2137 55 | active_chunk="";
2139 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2140 It has been observed that not every line of output when a TeXmacs chunk is active is a line of chunk. This may no longer be true, but we set a variable texmacs_chunk if the current line is a chunk line.
2141 Initially we set this to zero...
2143 67b <recognize-chunk[7](
\v) ⇑65a, lang=> +≡ ▵67a 67c▿
2144 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2145 57 | texmacs_chunk=0;
2146 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2147 ...and then we look to see if the current line is a chunk line.
2148 TeXmacs lines look like this: 3 | main() { so we detect the lines by leading white space, digits, more whiter space and a vertical bar followed by at least once space.
2149 If we find such a line, we remove this line-header and set texmacs_chunk=1 as well as chunking=1
2151 67c <recognize-chunk[8](
\v) ⇑65a, lang=> +≡ ▵67b 67d▿
2152 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2153 58 | /^ *[1-9][0-9]* *\| / {
2154 59 | if (texmacs_chunking) {
2156 61 | texmacs_chunk=1;
2157 62 | gsub("^ *[1-9][0-9]* *\\| ", "")
2160 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2161 When TeXmacs chunking, lines that commence with \/ or __ are not chunk content but visual framing, and are skipped.
2163 67d <recognize-chunk[9](
\v) ⇑65a, lang=> +≡ ▵67c 68a⊳
2164 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2165 65 | /^ *\.\/\\/ && texmacs_chunking {
2168 68 | /^ *__*$/ && texmacs_chunking {
2171 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2172 Any other line when TeXmacs chunking is considered to be a line-wrapped line.
2174 68a <recognize-chunk[10](
\v) ⇑65a, lang=> +≡ ⊲67d 68b▿
2175 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2176 71 | texmacs_chunking {
2177 72 | if (! texmacs_chunk) {
2178 73 | # must be a texmacs continued line
2180 75 | texmacs_chunk=1;
2183 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2184 This final chunklet seems bogus and probably stops L Y X working.
2186 68b <recognize-chunk[11](
\v) ⇑65a, lang=> +≡ ▵68a 68c▿
2187 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2188 78 | ! texmacs_chunk {
2189 79 | # texmacs_chunking=0;
2192 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2194 We recognize notangle style chunks too:
2196 68c <recognize-chunk[12](
\v) ⇑65a, lang=awk> +≡ ▵68b 68d▿
2197 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2198 82 | /^[<]<.*[>]>=/ {
2199 83 | if (match($0, "^[<]<(.*)[>]>= *$", line)) {
2201 85 | notangle_mode=1;
2202 86 | new_chunk(line[1]);
2206 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2208 Likewise, we need to recognize when a chunk ends.
2210 The e in [e]nd{lislisting} is surrounded by square brackets so that when this document is processed, this chunk doesn't terminate early when the lstlistings package recognizes it's own end-string!4. This doesn't make sense as the regex is anchored with ^, which this line does not begin with! ^4
2212 68d <recognize-chunk[13](
\v) ⇑65a, lang=> +≡ ▵68c 69a⊳
2213 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2214 90 | /^\\[e]nd{lstlisting}|^\\[e]nd{Chunk}/ {
2216 92 | active_chunk="";
2219 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2222 69a <recognize-chunk[14](
\v) ⇑65a, lang=> +≡ ⊲68d 69b▿
2223 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2226 97 | active_chunk="";
2228 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2229 All other recognizers are only of effect if we are chunking; there's no point in looking at lines if they aren't part of a chunk, so we just ignore them as efficiently as we can.
2231 69b <recognize-chunk[15](
\v) ⇑65a, lang=> +≡ ▵69a 69c▿
2232 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2233 99 | ! chunking { next; }
2234 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2236 Chunk contents are any lines read while chunking is true. Some chunk contents are special in that they refer to other chunks, and will be replaced by the contents of these chunks when the file is generated.
2237 We add the output record separator ORS to the line now, because we will set ORS to the empty string when we generate the output5. So that we can partial print lines using print instead of printf.
2238 To do: This does't make sense
2241 69c <recognize-chunk[16](
\v) ⇑65a, lang=> +≡ ▵69b
2242 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2243 100 | length(active_chunk) {
2244 101 | «process-chunk-tabs 69e»
2245 102 | «process-chunk 70b»
2247 |________________________________________________________________________
2250 If a chunk just consisted of plain text, we could handle the chunk like this:
2252 69d <process-chunk-simple[1](
\v), lang=> ≡
2253 ________________________________________________________________________
2254 1 | chunk_line(active_chunk, $0 ORS);
2255 |________________________________________________________________________
2258 but in fact a chunk can include references to other chunks. Chunk includes are traditionally written as <<chunk-name>> but we support other variations, some of which are more suitable for particular editing systems.
2259 However, we also process tabs at this point. A tab at input can be replaced by a number of spaces defined by the tabs variable, set by the -T option. Of course this is poor tab behaviour, we should probably have the option to use proper counted tab-stops and process this on output.
2261 69e <process-chunk-tabs[1](
\v), lang=> ≡
2262 ________________________________________________________________________
2263 1 | if (length(tabs)) {
2264 2 | gsub("\t", tabs);
2266 |________________________________________________________________________
2270 If \lstset{escapeinside={=<}{>}} is set, then we can use <chunk-name ?> in listings. The sequence =< was chosen because:
2271 1.it is a better mnemonic than <<chunk-name>> in that the = sign signifies equivalence or substitutability.
2272 2.and because =< is not valid in C or any language I can think of.
2273 3.and also because lstlistings doesn't like >> as an end delimiter for the texcl escape, so we must make do with a single > which is better complemented by =< than by <<.
2274 Unfortunately the =<...> that we use re-enters a LaTeX parsing mode in which some characters are special, e.g. # \ and so these cause trouble if used in arguments to \chunkref. At some point I must fix the LaTeX command \chunkref so that it can accept these literally, but until then, when writing chunkref argumemts that need these characters, I must use the forms \textbackslash{} and \#; so I also define a hacky chunk delatex to be used further on whose purpose it is to remove these from any arguments parsed by fangle.
2276 70a <delatex[1](text
\v\v), lang=> ≡
2277 ________________________________________________________________________
2279 2 | gsub("\\\\#", "#", ${text});
2280 3 | gsub("\\\\textbackslash{}", "\\", ${text});
2281 4 | gsub("\\\\\\^", "^", ${text});
2282 |________________________________________________________________________
2285 As each chunk line may contain more than one chunk include, we will split out chunk includes in an iterative fashion6. Contrary to our use of split when substituting parameters in chapter ? ^6.
2286 First, as long as the chunk contains a \chunkref command we take as much as we can up to the first \chunkref command.
2287 TeXmacs text output uses ⟨...⟩ which comes out as unicode sequences 0xC2 0xAB ... 0xC2 0xBB. Modern awk will interpret [^\xC2\xBB] as a single unicode character if LANG is set correctly to the sub-type UTF-8, e.g. LANG=en_GB.UTF-8, otherwise [^\xC2\xBB] will be treated as a two character negated match — but this should not interfere with the function.
2289 70b <process-chunk[1](
\v), lang=> ≡ 70c▿
2290 ________________________________________________________________________
2293 3 | while(match(chunk,"(\xC2\xAB)([^\xC2\xBB]*) [^\xC2\xBB]*\xC2\xBB", line) ||
2295 5 | "([=]<\\\\chunkref{([^}>]*)}(\\(.*\\)|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)",
2298 8 | chunklet = substr(chunk, 1, RSTART - 1);
2299 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2300 We keep track of the indent count, by counting the number of literal characters found. We can then preserve this indent on each output line when multi-line chunks are expanded.
2301 We then process this first part literal text, and set the chunk which is still to be processed to be the text after the \chunkref command, which we will process next as we continue around the loop.
2303 70c <process-chunk[2](
\v) ⇑70b, lang=> +≡ ▵70b 71a⊳
2304 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2305 9 | indent += length(chunklet);
2306 10 | chunk_line(active_chunk, chunklet);
2307 11 | chunk = substr(chunk, RSTART + RLENGTH);
2308 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2309 We then consider the type of chunk command we have found, whether it is the fangle style command beginning with =< the older notangle style beginning with <<.
2310 Fangle chunks may have parameters contained within square brackets. These will be matched in line[3] and are considered at this stage of processing to be part of the name of the chunk to be included.
2312 71a <process-chunk[3](
\v) ⇑70b, lang=> +≡ ⊲70c 71b▿
2313 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2314 12 | if (substr(line[1], 1, 1) == "=") {
2315 13 | # chunk name up to }
2316 14 | «delatex
\v(line[3]
\v) 70a»
2317 15 | chunk_include(active_chunk, line[2] line[3], indent);
2318 16 | } else if (substr(line[1], 1, 1) == "<") {
2319 17 | chunk_include(active_chunk, line[4], indent);
2320 18 | } else if (line[1] == "\xC2\xAB") {
2321 19 | chunk_include(active_chunk, line[2], indent);
2323 21 | error("Unknown chunk fragment: " line[1]);
2325 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2326 The loop will continue until there are no more chunkref statements in the text, at which point we process the final part of the chunk.
2328 71b <process-chunk[4](
\v) ⇑70b, lang=> +≡ ▵71a 71c▿
2329 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2331 24 | chunk_line(active_chunk, chunk);
2332 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2333 We add the newline character as a chunklet on it's own, to make it easier to detect new lines and thus manage indentation when processing the output.
2335 71c <process-chunk[5](
\v) ⇑70b, lang=> +≡ ▵71b
2336 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2337 25 | chunk_line(active_chunk, "\n");
2338 |________________________________________________________________________
2341 We will also permit a chunk-part number to follow in square brackets, so that <chunk-name[1] ?> will refer to the first part only. This can make it easy to include a C function prototype in a header file, if the first part of the chunk is just the function prototype without the trailing semi-colon. The header file would include the prototype with the trailing semi-colon, like this:
2343 This is handled in section 14.1.1
2344 We should perhaps introduce a notion of language specific chunk options; so that perhaps we could specify:
2345 =<\chunkref{chunk-name[function-declaration]}
2346 which applies a transform function-declaration to the chunk --- which in this case would extract a function prototype from a function.
2349 Chapter 13Processing Options
2350 At the start, first we set the default options.
2352 73a <default-options[1](
\v), lang=> ≡
2353 ________________________________________________________________________
2356 3 | notangle_mode=0;
2359 |________________________________________________________________________
2362 Then we use getopt the standard way, and null out ARGV afterwards in the normal AWK fashion.
2364 73b <read-options[1](
\v), lang=> ≡
2365 ________________________________________________________________________
2366 1 | Optind = 1 # skip ARGV[0]
2367 2 | while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
2368 3 | «handle-options 73c»
2370 5 | for (i=1; i<Optind; i++) { ARGV[i]=""; }
2371 |________________________________________________________________________
2374 This is how we handle our options:
2376 73c <handle-options[1](
\v), lang=> ≡
2377 ________________________________________________________________________
2378 1 | if (Optopt == "R") root = Optarg;
2379 2 | else if (Optopt == "r") root="";
2380 3 | else if (Optopt == "L") linenos = 1;
2381 4 | else if (Optopt == "d") debug = 1;
2382 5 | else if (Optopt == "T") tabs = indent_string(Optarg+0);
2383 6 | else if (Optopt == "h") help();
2384 7 | else if (Optopt == "?") help();
2385 |________________________________________________________________________
2388 We do all of this at the beginning of the program
2390 73d <begin[1](
\v), lang=> ≡
2391 ________________________________________________________________________
2394 3 | «mode-definitions 52b»
2395 4 | «default-options 73a»
2397 6 | «read-options 73b»
2399 |________________________________________________________________________
2402 And have a simple help function
2404 73e <help()[1](
\v), lang=> ≡
2405 ________________________________________________________________________
2406 1 | function help() {
2408 3 | print " fangle [-L] -R<rootname> [source.tex ...]"
2409 4 | print " fangle -r [source.tex ...]"
2410 5 | print " If the filename, source.tex is not specified then stdin is used"
2412 7 | print "-L causes the C statement: #line <lineno> \"filename\"" to be issued"
2413 8 | print "-R causes the named root to be written to stdout"
2414 9 | print "-r lists all roots in the file (even those used elsewhere)"
2417 |________________________________________________________________________
2420 Chapter 14Generating the Output
2421 We generate output by calling output_chunk, or listing the chunk names.
2423 75a <generate-output[1](
\v), lang=> ≡
2424 ________________________________________________________________________
2425 1 | if (length(root)) output_chunk(root);
2426 2 | else output_chunk_names();
2427 |________________________________________________________________________
2430 We also have some other output debugging:
2432 75b <debug-output[1](
\v), lang=> ≡
2433 ________________________________________________________________________
2435 2 | print "------ chunk names "
2436 3 | output_chunk_names();
2437 4 | print "====== chunks"
2438 5 | output_chunks();
2439 6 | print "++++++ debug"
2440 7 | for (a in chunks) {
2441 8 | print a "=" chunks[a];
2444 |________________________________________________________________________
2447 We do both of these at the end. We also set ORS="" because each chunklet is not necessarily a complete line, and we already added ORS to each input line in section 12.4.
2449 75c <end[1](
\v), lang=> ≡
2450 ________________________________________________________________________
2452 2 | «debug-output 75b»
2454 4 | «generate-output 75a»
2456 |________________________________________________________________________
2459 We write chunk names like this. If we seem to be running in notangle compatibility mode, then we enclose the name like this <<name>> the same way notangle does:
2461 75d <output_chunk_names()[1](
\v), lang=> ≡
2462 ________________________________________________________________________
2463 1 | function output_chunk_names( c, prefix, suffix)
2465 3 | if (notangle_mode) {
2469 7 | for (c in chunk_names) {
2470 8 | print prefix c suffix "\n";
2473 |________________________________________________________________________
2476 This function would write out all chunks
2478 75e <output_chunks()[1](
\v), lang=> ≡
2479 ________________________________________________________________________
2480 1 | function output_chunks( a)
2482 3 | for (a in chunk_names) {
2483 4 | output_chunk(a);
2487 8 | function output_chunk(chunk) {
2489 10 | lineno_needed = linenos;
2491 12 | write_chunk(chunk);
2494 |________________________________________________________________________
2497 14.1 Assembling the Chunks
2498 chunk_path holds a string consisting of the names of all the chunks that resulted in this chunk being output. It should probably also contain the source line numbers at which each inclusion also occured.
2499 We first initialize the mode tracker for this chunk.
2501 76a <write_chunk()[1](
\v), lang=awk> ≡ 76b▿
2502 ________________________________________________________________________
2503 1 | function write_chunk(chunk_name) {
2504 2 | «awk-delete-array
\v(context
\v) 37d»
2505 3 | return write_chunk_r(chunk_name, context);
2508 6 | function write_chunk_r(chunk_name, context, indent, tail,
2510 8 | chunk_path, chunk_args,
2512 10 | context_origin,
2513 11 | chunk_params, part, max_part, part_line, frag, max_frag, text,
2514 12 | chunklet, only_part, call_chunk_args, new_context)
2516 14 | if (debug) debug_log("write_chunk_r(" chunk_name ")");
2517 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2519 As mentioned in section ?, a chunk name may contain a part specifier in square brackets, limiting the parts that should be emitted.
2521 76b <write_chunk()[2](
\v) ⇑76a, lang=> +≡ ▵76a 76c▿
2522 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2523 15 | if (match(chunk_name, "^(.*)\\[([0-9]*)\\]$", chunk_name_parts)) {
2524 16 | chunk_name = chunk_name_parts[1];
2525 17 | only_part = chunk_name_parts[2];
2527 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2528 We then create a mode tracker
2530 76c <write_chunk()[3](
\v) ⇑76a, lang=> +≡ ▵76b 77a⊳
2531 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2532 19 | context_origin = context[""];
2533 20 | new_context = push_mode_tracker(context, chunks[chunk_name, "language"], "");
2534 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2535 We extract into chunk_params the names of the parameters that this chunk accepts, whose values were (optionally) passed in chunk_args.
2537 77a <write_chunk()[4](
\v) ⇑76a, lang=> +≡ ⊲76c 77b▿
2538 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2539 21 | split(chunks[chunk_name, "params"], chunk_params, " *; *");
2540 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2541 To assemble a chunk, we write out each part.
2543 77b <write_chunk()[5](
\v) ⇑76a, lang=> +≡ ▵77a
2544 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2545 22 | if (! (chunk_name in chunk_names)) {
2546 23 | error(sprintf(_"The root module <<%s>> was not defined.\nUsed by: %s",\
2547 24 | chunk_name, chunk_path));
2550 27 | max_part = chunks[chunk_name, "part"];
2551 28 | for(part = 1; part <= max_part; part++) {
2552 29 | if (! only_part || part == only_part) {
2553 30 | «write-part 77c»
2556 33 | if (! pop_mode_tracker(context, context_origin)) {
2557 34 | dump_mode_tracker(context);
2558 35 | error(sprintf(_"Module %s did not close context properly.\nUsed by: %s\n", chunk_name, chunk_path));
2561 |________________________________________________________________________
2564 A part can either be a chunklet of lines, or an include of another chunk.
2565 Chunks may also have parameters, specified in LaTeX style with braces after the chunk name --- looking like this in the document: chunkname{param1, param2}. Arguments are passed in square brackets: \chunkref{chunkname}[arg1, arg2].
2566 Before we process each part, we check that the source position hasn't changed unexpectedly, so that we can know if we need to output a new file-line directive.
2568 77c <write-part[1](
\v), lang=> ≡
2569 ________________________________________________________________________
2570 1 | «check-source-jump 79d»
2572 3 | chunklet = chunks[chunk_name, "part", part];
2573 4 | if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
2574 5 | «write-included-chunk 77d»
2575 6 | } else if (chunklet SUBSEP "line" in chunks) {
2576 7 | «write-chunklets 78a»
2578 9 | # empty last chunklet
2580 |________________________________________________________________________
2583 To write an included chunk, we must detect any optional chunk arguments in parenthesis. Then we recurse calling write_chunk().
2585 77d <write-included-chunk[1](
\v), lang=> ≡
2586 ________________________________________________________________________
2587 1 | if (match(chunklet, "^([^\\[\\(]*)\\((.*)\\)$", chunklet_parts)) {
2588 2 | chunklet = chunklet_parts[1];
2590 4 | gsub(sprintf("%c",11), "", chunklet);
2591 5 | gsub(sprintf("%c",11), "", chunklet_parts[2]);
2592 6 | parse_chunk_args("c-like", chunklet_parts[2], call_chunk_args, "(");
2593 7 | for (c in call_chunk_args) {
2594 8 | call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params, chunk_args);
2597 11 | split("", call_chunk_args);
2600 14 | write_chunk_r(chunklet, context,
2601 15 | chunks[chunk_name, "part", part, "indent"] indent,
2602 16 | chunks[chunk_name, "part", part, "tail"],
2603 17 | chunk_path "\n " chunk_name,
2604 18 | call_chunk_args);
2605 |________________________________________________________________________
2608 Before we output a chunklet of lines, we first emit the file and line number if we have one, and if it is safe to do so.
2609 Chunklets are generally broken up by includes, so the start of a chunklet is a good place to do this. Then we output each line of the chunklet.
2610 When it is not safe, such as in the middle of a multi-line macro definition, lineno_suppressed is set to true, and in such a case we note that we want to emit the line statement when it is next safe.
2612 78a <write-chunklets[1](
\v), lang=> ≡ 78b▿
2613 ________________________________________________________________________
2614 1 | max_frag = chunks[chunklet, "line"];
2615 2 | for(frag = 1; frag <= max_frag; frag++) {
2616 3 | «write-file-line 79c»
2617 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2618 We then extract the chunklet text and expand any arguments.
2620 78b <write-chunklets[2](
\v) ⇑78a, lang=> +≡ ▵78a 78c▿
2621 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2623 5 | text = chunks[chunklet, frag];
2625 7 | /* check params */
2626 8 | text = expand_chunk_args(text, chunk_params, chunk_args);
2627 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2628 If the text is a single newline (which we keep separate - see 6) then we increment the line number. In the case where this is the last line of a chunk and it is not a top-level chunk we replace the newline with an empty string --- because the chunk that included this chunk will have the newline at the end of the line that included this chunk.
2629 We also note by newline = 1 that we have started a new line, so that indentation can be managed with the following piece of text.
2631 78c <write-chunklets[3](
\v) ⇑78a, lang=> +≡ ▵78b 78d▿
2632 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2634 10 | if (text == "\n") {
2636 12 | if (part == max_part && frag == max_frag && length(chunk_path)) {
2642 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2643 If this text does not represent a newline, but we see that we are the first piece of text on a newline, then we prefix our text with the current indent.
2644 Note 1. newline is a global output-state variable, but the indent is not.
2646 78d <write-chunklets[4](
\v) ⇑78a, lang=> +≡ ▵78c 79a⊳
2647 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2648 18 | } else if (length(text) || length(tail)) {
2649 19 | if (newline) text = indent text;
2653 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2654 Tail will soon no longer be relevant once mode-detection is in place.
2656 79a <write-chunklets[5](
\v) ⇑78a, lang=> +≡ ⊲78d 79b▿
2657 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2658 23 | text = text tail;
2659 24 | mode_tracker(context, text);
2660 25 | print untab(transform_escape(context, text, new_context));
2661 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2662 If a line ends in a backslash --- suggesting continuation --- then we supress outputting file-line as it would probably break the continued lines.
2664 79b <write-chunklets[6](
\v) ⇑78a, lang=> +≡ ▵79a
2665 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2667 27 | lineno_suppressed = substr(lastline, length(lastline)) == "\\";
2670 |________________________________________________________________________
2673 Of course there is no point in actually outputting the source filename and line number (file-line) if they don't say anything new! We only need to emit them if they aren't what is expected, or if we we not able to emit one when they had changed.
2675 79c <write-file-line[1](
\v), lang=> ≡
2676 ________________________________________________________________________
2677 1 | if (newline && lineno_needed && ! lineno_suppressed) {
2678 2 | filename = a_filename;
2679 3 | lineno = a_lineno;
2680 4 | print "#line " lineno " \"" filename "\"\n"
2681 5 | lineno_needed = 0;
2683 |________________________________________________________________________
2686 We check if a new file-line is needed by checking if the source line matches what we (or a compiler) would expect.
2688 79d <check-source-jump[1](
\v), lang=> ≡
2689 ________________________________________________________________________
2690 1 | if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in chunks)) {
2691 2 | a_filename = chunks[chunk_name, "part", part, "FILENAME"];
2692 3 | a_lineno = chunks[chunk_name, "part", part, "LINENO"];
2693 4 | if (a_filename != filename || a_lineno != lineno) {
2694 5 | lineno_needed++;
2697 |________________________________________________________________________
2700 Chapter 15Storing Chunks
2701 Awk has pretty limited data structures, so we will use two main hashes. Uninterrupted sequences of a chunk will be stored in chunklets and the chunklets used in a chunk will be stored in chunks.
2703 81a <constants[2](
\v) ⇑39a, lang=> +≡ ⊲39a
2704 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2705 2 | part_type_chunk=1;
2707 |________________________________________________________________________
2710 The params mentioned are not chunk parameters for parameterized chunks, as mentioned in 10.2, but the lstlistings style parameters used in the \Chunk command1. The params parameter is used to hold the parameters for parameterized chunks ^1.
2712 81b <chunk-storage-functions[1](
\v), lang=> ≡ 81c▿
2713 ________________________________________________________________________
2714 1 | function new_chunk(chunk_name, opts, args,
2718 5 | # HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
2719 6 | gsub("\\(\\)$", "", chunk_name);
2720 7 | if (! (chunk_name in chunk_names)) {
2721 8 | if (debug) print "New chunk " chunk_name;
2722 9 | chunk_names[chunk_name];
2723 10 | for (p in opts) {
2724 11 | chunks[chunk_name, p] = opts[p];
2725 12 | if (debug) print "chunks[" chunk_name "," p "] = " opts[p];
2727 14 | for (p in args) {
2728 15 | chunks[chunk_name, "params", p] = args[p];
2730 17 | if ("append" in opts) {
2731 18 | append=opts["append"];
2732 19 | if (! (append in chunk_names)) {
2733 20 | warning("Chunk " chunk_name " is appended to chunk " append " which is not defined yet");
2734 21 | new_chunk(append);
2736 23 | chunk_include(append, chunk_name);
2737 24 | chunk_line(append, ORS);
2740 27 | active_chunk = chunk_name;
2741 28 | prime_chunk(chunk_name);
2743 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2745 81c <chunk-storage-functions[2](
\v) ⇑81b, lang=> +≡ ▵81b 82a⊳
2746 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2748 31 | function prime_chunk(chunk_name)
2750 33 | chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = \
2751 34 | chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"];
2752 35 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
2753 36 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR + 1;
2756 39 | function chunk_line(chunk_name, line){
2757 40 | chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
2758 41 | ++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"], "line"] ] = line;
2761 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2762 Chunk include represents a chunkref statement, and stores the requirement to include another chunk. The parameter indent represents the quanity of literal text characters that preceded this chunkref statement and therefore by how much additional lines of the included chunk should be indented.
2764 82a <chunk-storage-functions[3](
\v) ⇑81b, lang=> +≡ ⊲81c 82b▿
2765 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2766 44 | function chunk_include(chunk_name, chunk_ref, indent, tail)
2768 46 | chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
2769 47 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_chunk;
2770 48 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_string(indent);
2771 49 | chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
2772 50 | prime_chunk(chunk_name);
2775 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2776 The indent is calculated by indent_string, which may in future convert some spaces into tab characters. This function works by generating a printf padded format string, like %22s for an indent of 22, and then printing an empty string using that format.
2778 82b <chunk-storage-functions[4](
\v) ⇑81b, lang=> +≡ ▵82a
2779 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2780 53 | function indent_string(indent) {
2781 54 | return sprintf("%" indent "s", "");
2783 |________________________________________________________________________
2787 I use Arnold Robbins public domain getopt (1993 revision). This is probably the same one that is covered in chapter 12 of âĂIJEdition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU AwkâĂİ but as that is licensed under the GNU Free Documentation License, Version 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompanying explanations), so I do my best to explain how it works here.
2788 The getopt.awk header is:
2790 83a <getopt.awk-header[1](
\v), lang=> ≡
2791 ________________________________________________________________________
2792 1 | # getopt.awk --- do C library getopt(3) function in awk
2794 3 | # Arnold Robbins, arnold@skeeve.com, Public Domain
2796 5 | # Initial version: March, 1991
2797 6 | # Revised: May, 1993
2799 |________________________________________________________________________
2802 The provided explanation is:
2804 83b <getopt.awk-notes[1](
\v), lang=> ≡
2805 ________________________________________________________________________
2806 1 | # External variables:
2807 2 | # Optind -- index in ARGV of first nonoption argument
2808 3 | # Optarg -- string value of argument to current option
2809 4 | # Opterr -- if nonzero, print our own diagnostic
2810 5 | # Optopt -- current option letter
2813 8 | # -1 at end of options
2814 9 | # ? for unrecognized option
2815 10 | # <c> a character representing the current option
2817 12 | # Private Data:
2818 13 | # _opti -- index in multi-flag option, e.g., -abc
2820 |________________________________________________________________________
2823 The function follows. The final two parameters, thisopt and i are local variables and not parameters --- as indicated by the multiple spaces preceding them. Awk doesn't care, the multiple spaces are a convention to help us humans.
2825 83c <getopt.awk-getopt()[1](
\v), lang=> ≡ 84a⊳
2826 ________________________________________________________________________
2827 1 | function getopt(argc, argv, options, thisopt, i)
2829 3 | if (length(options) == 0) # no options given
2831 5 | if (argv[Optind] == "--") { # all done
2835 9 | } else if (argv[Optind] !~ /^-[^: \t\n\f\r\v\b]/) {
2839 13 | if (_opti == 0)
2841 15 | thisopt = substr(argv[Optind], _opti, 1)
2842 16 | Optopt = thisopt
2843 17 | i = index(options, thisopt)
2846 20 | printf("%c -- invalid option\n",
2847 21 | thisopt) > "/dev/stderr"
2848 22 | if (_opti >= length(argv[Optind])) {
2855 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2856 At this point, the option has been found and we need to know if it takes any arguments.
2858 84a <getopt.awk-getopt()[2](
\v) ⇑83c, lang=> +≡ ⊲83c
2859 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2860 29 | if (substr(options, i + 1, 1) == ":") {
2861 30 | # get option argument
2862 31 | if (length(substr(argv[Optind], _opti + 1)) > 0)
2863 32 | Optarg = substr(argv[Optind], _opti + 1)
2865 34 | Optarg = argv[++Optind]
2869 38 | if (_opti == 0 || _opti >= length(argv[Optind])) {
2876 |________________________________________________________________________
2879 A test program is built in, too
2881 84b <getopt.awk-begin[1](
\v), lang=> ≡
2882 ________________________________________________________________________
2884 2 | Opterr = 1 # default is to diagnose
2885 3 | Optind = 1 # skip ARGV[0]
2887 5 | if (_getopt_test) {
2888 6 | while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
2889 7 | printf("c = <%c>, optarg = <%s>\n",
2891 9 | printf("non-option arguments:\n")
2892 10 | for (; Optind < ARGC; Optind++)
2893 11 | printf("\tARGV[%d] = <%s>\n",
2894 12 | Optind, ARGV[Optind])
2897 |________________________________________________________________________
2900 The entire getopt.awk is made out of these chunks in order
2902 84c <getopt.awk[1](
\v), lang=> ≡
2903 ________________________________________________________________________
2904 1 | «getopt.awk-header 83a»
2906 3 | «getopt.awk-notes 83b»
2907 4 | «getopt.awk-getopt() 83c»
2908 5 | «getopt.awk-begin 84b»
2909 |________________________________________________________________________
2912 Although we only want the header and function:
2914 85a <getopt[1](
\v), lang=> ≡
2915 ________________________________________________________________________
2916 1 | # try: locate getopt.awk for the full original file
2917 2 | # as part of your standard awk installation
2918 3 | «getopt.awk-header 83a»
2920 5 | «getopt.awk-getopt() 83c»
2921 |________________________________________________________________________
2924 Chapter 17Fangle LaTeX source code
2926 Here we define a L Y X .module file that makes it convenient to use L Y X for writing such literate programs.
2927 This file ./fangle.module can be installed in your personal .lyx/layouts folder. You will need to Tools Reconfigure so that L Y X notices it. It adds a new format Chunk, which should precede every listing and contain the chunk name.
2929 87a <./fangle.module[1](
\v), lang=lyx-module> ≡
2930 ________________________________________________________________________
2931 1 | #\DeclareLyXModule{Fangle Literate Listings}
2932 2 | #DescriptionBegin
2933 3 | # Fangle literate listings allow one to write
2934 4 | # literate programs after the fashion of noweb, but without having
2935 5 | # to use noweave to generate the documentation. Instead the listings
2936 6 | # package is extended in conjunction with the noweb package to implement
2937 7 | # to code formating directly as latex.
2938 8 | # The fangle awk script
2941 11 | «gpl3-copyright.hashed 87b»
2946 16 | «./fangle.sty 88d»
2949 19 | «chunkstyle 88a»
2952 |________________________________________________________________________
2955 Because L Y X modules are not yet a language supported by fangle or lstlistings, we resort to this fake awk chunk below in order to have each line of the GPL3 license commence with a #
2957 87b <gpl3-copyright.hashed[1](
\v), lang=awk> ≡
2958 ________________________________________________________________________
2959 1 | #«gpl3-copyright 4a»
2961 |________________________________________________________________________
2964 17.1.1 The Chunk style
2965 The purpose of the chunk style is to make it easier for L Y X users to provide the name to lstlistings. Normally this requires right-clicking on the listing, choosing settings, advanced, and then typing name=chunk-name. This has the further disadvantage that the name (and other options) are not generally visible during document editing.
2966 The chunk style is defined as a LaTeX command, so that all text on the same line is passed to the LaTeX command Chunk. This makes it easy to parse using fangle, and easy to pass these options on to the listings package. The first word in a chunk section should be the chunk name, and will have name= prepended to it. Any other words are accepted arguments to lstset.
2967 We set PassThru to 1 because the user is actually entering raw latex.
2969 88a <chunkstyle[1](
\v), lang=> ≡ 88b▿
2970 ________________________________________________________________________
2972 2 | LatexType Command
2974 4 | Margin First_Dynamic
2975 5 | LeftMargin Chunk:xxx
2977 7 | LabelType Static
2978 8 | LabelString "Chunk:"
2982 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2983 To make the label very visible we choose a larger font coloured red.
2985 88b <chunkstyle[2](
\v) ⇑88a, lang=> +≡ ▵88a
2986 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
2995 |________________________________________________________________________
2998 17.1.2 The chunkref style
2999 We also define the Chunkref style which can be used to express cross references to chunks.
3001 88c <chunkref[1](
\v), lang=> ≡
3002 ________________________________________________________________________
3003 1 | InsetLayout Chunkref
3004 2 | LyxType charstyle
3005 3 | LatexType Command
3006 4 | LatexName chunkref
3013 |________________________________________________________________________
3017 We require the listings, noweb and xargs packages. As noweb defines it's own \code environment, we re-define the one that L Y X logical markup module expects here.
3019 88d <./fangle.sty[1](
\v), lang=tex> ≡ 89a⊳
3020 ________________________________________________________________________
3021 1 | \usepackage{listings}%
3022 2 | \usepackage{noweb}%
3023 3 | \usepackage{xargs}%
3024 4 | \renewcommand{\code}[1]{\texttt{#1}}%
3025 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3026 We also define a CChunk macro, for use as: \begin{CChunk} which will need renaming to \begin{Chunk} when I can do this without clashing with \Chunk.
3028 89a <./fangle.sty[2](
\v) ⇑88d, lang=> +≡ ⊲88d 89b▿
3029 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3030 5 | \lstnewenvironment{Chunk}{\relax}{\relax}%
3031 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3032 We also define a suitable \lstset of parameters that suit the literate programming style after the fashion of noweave.
3034 89b <./fangle.sty[3](
\v) ⇑88d, lang=> +≡ ▵89a 89c▿
3035 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3036 6 | \lstset{numbers=left, stepnumber=5, numbersep=5pt,
3037 7 | breaklines=false,basicstyle=\ttfamily,
3038 8 | numberstyle=\tiny, language=C}%
3039 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3040 We also define a notangle-like mechanism for escaping to LaTeX from the listing, and by which we can refer to other listings. We declare the =<...> sequence to contain LaTeX code, and include another like this chunk: <chunkname ?>. However, because =<...> is already defined to contain LaTeX code for this document --- this is a fangle document after all --- the code fragment below effectively contains the LaTeX code: }{. To avoid problems with document generation, I had to declare an lstlistings property: escapeinside={} for this listing only; which in L Y X was done by right-clicking the listings inset, choosing settings->advanced. Therefore =< isn't interpreted literally here, in a listing when the escape sequence is already defined as shown... we need to somehow escape this representation...
3042 89c <./fangle.sty[4](
\v) ⇑88d, lang=> +≡ ▵89b 89d▿
3043 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3044 9 | \lstset{escapeinside={=<}{>}}%
3045 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3046 Although our macros will contain the @ symbol, they will be included in a \makeatletter section by L Y X; however we keep the commented out \makeatletter as a reminder. The listings package likes to centre the titles, but noweb titles are specially formatted and must be left aligned. The simplest way to do this turned out to be by removing the definition of \lst@maketitle. This may interact badly if other listings want a regular title or caption. We remember the old maketitle in case we need it.
3048 89d <./fangle.sty[5](
\v) ⇑88d, lang=> +≡ ▵89c 89e▿
3049 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3051 11 | %somehow re-defining maketitle gives us a left-aligned title
3052 12 | %which is extactly what our specially formatted title needs!
3053 13 | \global\let\fangle@lst@maketitle\lst@maketitle%
3054 14 | \global\def\lst@maketitle{}%
3055 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3056 17.2.1 The chunk command
3057 Our chunk command accepts one argument, and calls \ltset. Although \ltset will note the name, this is erased when the next \lstlisting starts, so we make a note of this in \lst@chunkname and restore in in lstlistings Init hook.
3059 89e <./fangle.sty[6](
\v) ⇑88d, lang=> +≡ ▵89d 90a⊳
3060 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3062 16 | \lstset{title={\fanglecaption},name=#1}%
3063 17 | \global\edef\lst@chunkname{\lst@intname}%
3065 19 | \def\lst@chunkname{\empty}%
3066 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3067 17.2.1.1 Chunk parameters
3068 Fangle permits parameterized chunks, and requires the paramters to be specified as listings options. The fangle script uses this, and although we don't do anything with these in the LaTeX code right now, we need to stop the listings package complaining.
3070 90a <./fangle.sty[7](
\v) ⇑88d, lang=> +≡ ⊲89e 90b▿
3071 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3072 20 | \lst@Key{params}\relax{\def\fangle@chunk@params{#1}}%
3073 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3074 As it is common to define a chunk which then needs appending to another chunk, and annoying to have to declare a single line chunk to manage the include, we support an append= option.
3076 90b <./fangle.sty[8](
\v) ⇑88d, lang=> +≡ ▵90a 90c▿
3077 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3078 21 | \lst@Key{append}\relax{\def\fangle@chunk@append{#1}}%
3079 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3080 17.2.2 The noweb styled caption
3081 We define a public macro \fanglecaption which can be set as a regular title. By means of \protect, It expands to \fangle@caption at the appopriate time when the caption is emitted.
3083 90c <./fangle.sty[9](
\v) ⇑88d, lang=> +≡ ▵90b 90d▿
3084 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3085 \def\fanglecaption{\protect\fangle@caption}%
3086 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3087 22c ⟨some-chunk 19b⟩≡+ ⊲22b 24d⊳
3089 In this example, the current chunk is 22c, and therefore the third chunk on page 22.
3090 It's name is some-chunk.
3091 The first chunk with this name (19b) occurs as the second chunk on page 19.
3092 The previous chunk (22d) with the same name is the second chunk on page 22.
3093 The next chunk (24d) is the fourth chunk on page 24.
3095 Figure 1. Noweb Heading
3097 The general noweb output format compactly identifies the current chunk, and references to the first chunk, and the previous and next chunks that have the same name.
3098 This means that we need to keep a counter for each chunk-name, that we use to count chunks of the same name.
3099 17.2.3 The chunk counter
3100 It would be natural to have a counter for each chunk name, but TeX would soon run out of counters1. ...soon did run out of counters and so I had to re-write the LaTeX macros to share a counter as described here. ^1, so we have one counter which we save at the end of a chunk and restore at the beginning of a chunk.
3102 90d <./fangle.sty[10](
\v) ⇑88d, lang=> +≡ ▵90c 91c⊳
3103 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3104 22 | \newcounter{fangle@chunkcounter}%
3105 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3106 We construct the name of this variable to store the counter to be the text lst-chunk- prefixed onto the chunks own name, and store it in \chunkcount.
3107 We save the counter like this:
3109 91a <save-counter[1](
\v), lang=> ≡
3110 ________________________________________________________________________
3111 \global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
3112 |________________________________________________________________________
3115 and restore the counter like this:
3117 91b <restore-counter[1](
\v), lang=> ≡
3118 ________________________________________________________________________
3119 \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
3120 |________________________________________________________________________
3123 If there does not already exist a variable whose name is stored in \chunkcount, then we know we are the first chunk with this name, and then define a counter.
3124 Although chunks of the same name share a common counter, they must still be distinguished. We use is the internal name of the listing, suffixed by the counter value. So the first chunk might be something-1 and the second chunk be something-2, etc.
3125 We also calculate the name of the previous chunk if we can (before we increment the chunk counter). If this is the first chunk of that name, then \prevchunkname is set to \relax which the noweb package will interpret as not existing.
3127 91c <./fangle.sty[11](
\v) ⇑88d, lang=> +≡ ⊲90d 91d▿
3128 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3129 23 | \def\fangle@caption{%
3130 24 | \edef\chunkcount{lst-chunk-\lst@intname}%
3131 25 | \@ifundefined{\chunkcount}{%
3132 26 | \expandafter\gdef\csname \chunkcount\endcsname{0}%
3133 27 | \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
3134 28 | \let\prevchunkname\relax%
3136 30 | \setcounter{fangle@chunkcounter}{\csname \chunkcount\endcsname}%
3137 31 | \edef\prevchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
3139 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3140 After incrementing the chunk counter, we then define the name of this chunk, as well as the name of the first chunk.
3142 91d <./fangle.sty[12](
\v) ⇑88d, lang=> +≡ ▵91c 91e▿
3143 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3144 33 | \addtocounter{fangle@chunkcounter}{1}%
3145 34 | \global\expandafter\edef\csname \chunkcount\endcsname{\arabic{fangle@chunkcounter}}%
3146 35 | \edef\chunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
3147 36 | \edef\firstchunkname{\lst@intname-1}%
3148 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3149 We now need to calculate the name of the next chunk. We do this by temporarily skipping the counter on by one; however there may not actually be another chunk with this name! We detect this by also defining a label for each chunk based on the chunkname. If there is a next chunkname then it will define a label with that name. As labels are persistent, we can at least tell the second time LaTeX is run. If we don't find such a defined label then we define \nextchunkname to \relax.
3151 91e <./fangle.sty[13](
\v) ⇑88d, lang=> +≡ ▵91d 92a⊳
3152 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3153 37 | \addtocounter{fangle@chunkcounter}{1}%
3154 38 | \edef\nextchunkname{\lst@intname-\arabic{fangle@chunkcounter}}%
3155 39 | \@ifundefined{r@label-\nextchunkname}{\let\nextchunkname\relax}{}%
3156 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3157 The noweb package requires that we define a \sublabel for every chunk, with a unique name, which is then used to print out it's navigation hints.
3158 We also define a regular label for this chunk, as was mentioned above when we calculated \nextchunkname. This requires LaTeX to be run at least twice after new chunk sections are added --- but noweb requried that anyway.
3160 92a <./fangle.sty[14](
\v) ⇑88d, lang=> +≡ ⊲91e 92b▿
3161 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3162 40 | \sublabel{\chunkname}%
3163 41 | % define this label for every chunk instance, so we
3164 42 | % can tell when we are the last chunk of this name
3165 43 | \label{label-\chunkname}%
3166 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3167 We also try and add the chunk to the list of listings, but I'm afraid we don't do very well. We want each chunk name listing once, with all of it's references.
3169 92b <./fangle.sty[15](
\v) ⇑88d, lang=> +≡ ▵92a 92c▿
3170 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3171 44 | \addcontentsline{lol}{lstlisting}{\lst@name~[\protect\subpageref{\chunkname}]}%
3172 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3173 We then call the noweb output macros in the same way that noweave generates them, except that we don't need to call \nwstartdeflinemarkup or \nwenddeflinemarkup — and if we do, it messes up the output somewhat.
3175 92c <./fangle.sty[16](
\v) ⇑88d, lang=> +≡ ▵92b 92d▿
3176 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3180 48 | \subpageref{\chunkname}%
3187 55 | \nwtagstyle{}\/%
3188 56 | \@ifundefined{fangle@chunk@params}{}{%
3189 57 | (\fangle@chunk@params)%
3191 59 | [\csname \chunkcount\endcsname]~%
3192 60 | \subpageref{\firstchunkname}%
3194 62 | \@ifundefined{fangle@chunk@append}{}{%
3195 63 | \ifx{}\fangle@chunk@append{x}\else%
3196 64 | ,~add~to~\fangle@chunk@append%
3199 67 | \global\def\fangle@chunk@append{}%
3200 68 | \lstset{append=x}%
3203 71 | \ifx\relax\prevchunkname\endmoddef\else\plusendmoddef\fi%
3204 72 | % \nwstartdeflinemarkup%
3205 73 | \nwprevnextdefs{\prevchunkname}{\nextchunkname}%
3206 74 | % \nwenddeflinemarkup%
3208 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3209 Originally this was developed as a listings aspect, in the Init hook, but it was found easier to affect the title without using a hook — \lst@AddToHookExe{PreSet} is still required to set the listings name to the name passed to the \Chunk command, though.
3211 92d <./fangle.sty[17](
\v) ⇑88d, lang=> +≡ ▵92c 93a⊳
3212 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3213 76 | %\lst@BeginAspect{fangle}
3214 77 | %\lst@Key{fangle}{true}[t]{\lstKV@SetIf{#1}{true}}
3215 78 | \lst@AddToHookExe{PreSet}{\global\let\lst@intname\lst@chunkname}
3216 79 | \lst@AddToHook{Init}{}%\fangle@caption}
3217 80 | %\lst@EndAspect
3218 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3219 17.2.4 Cross references
3220 We define the \chunkref command which makes it easy to generate visual references to different code chunks, e.g.
3223 \chunkref[3]{preamble}
3224 \chunkref{preamble}[arg1, arg2]
3226 Chunkref can also be used within a code chunk to include another code chunk. The third optional parameter to chunkref is a comma sepatarated list of arguments, which will replace defined parameters in the chunkref.
3227 Note 1. Darn it, if I have: =<\chunkref{new-mode-tracker}[{chunks[chunk_name, "language"]},{mode}]> the inner braces (inside [ ]) cause _ to signify subscript even though we have lst@ReplaceIn
3229 93a <./fangle.sty[18](
\v) ⇑88d, lang=> +≡ ⊲92d 94a⊳
3230 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3231 81 | \def\chunkref@args#1,{%
3233 83 | \lst@ReplaceIn\arg\lst@filenamerpl%
3235 85 | \@ifnextchar){\relax}{, \chunkref@args}%
3237 87 | \newcommand\chunkref[2][0]{%
3238 88 | \@ifnextchar({\chunkref@i{#1}{#2}}{\chunkref@i{#1}{#2}()}%
3240 90 | \def\chunkref@i#1#2(#3){%
3242 92 | \def\chunk{#2}%
3243 93 | \def\chunkno{#1}%
3244 94 | \def\chunkargs{#3}%
3245 95 | \ifx\chunkno\zero%
3246 96 | \def\chunkname{#2-1}%
3248 98 | \def\chunkname{#2-\chunkno}%
3250 100 | \let\lst@arg\chunk%
3251 101 | \lst@ReplaceIn\chunk\lst@filenamerpl%
3252 102 | \LA{%\moddef{%
3255 105 | \nwtagstyle{}\/%
3256 106 | \ifx\chunkno\zero%
3260 110 | \ifx\chunkargs\empty%
3262 112 | (\chunkref@args #3,)%
3264 114 | ~\subpageref{\chunkname}%
3267 117 | \RA%\endmoddef%
3269 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3272 94a <./fangle.sty[19](
\v) ⇑88d, lang=> +≡ ⊲93a
3273 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3276 |________________________________________________________________________
3279 Chapter 18Extracting fangle
3280 18.1 Extracting from Lyx
3281 To extract from L Y X, you will need to configure L Y X as explained in section ?.
3282 And this lyx-build scrap will extract fangle for me.
3284 95a <lyx-build[2](
\v) ⇑20a, lang=sh> +≡ ⊲20a
3285 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3289 14 | «lyx-build-helper 19b»
3290 15 | cd $PROJECT_DIR || exit 1
3292 17 | /usr/local/bin/fangle -R./fangle $TEX_SRC > ./fangle
3293 18 | /usr/local/bin/fangle -R./fangle.module $TEX_SRC > ./fangle.module
3295 20 | export FANGLE=./fangle
3296 21 | export TMP=${TMP:-/tmp}
3298 |________________________________________________________________________
3301 With a lyx-build-helper
3303 95b <lyx-build-helper[2](
\v) ⇑19b, lang=sh> +≡ ⊲19b
3304 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3305 5 | PROJECT_DIR="$LYX_r"
3306 6 | LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
3307 7 | TEX_DIR="$LYX_p"
3308 8 | TEX_SRC="$TEX_DIR/$LYX_i"
3309 9 | TXT_SRC="$TEX_SRC"
3310 |________________________________________________________________________
3313 18.2 Extracting documentation
3315 95c <./gen-www[1](
\v), lang=> ≡
3316 ________________________________________________________________________
3317 1 | #python -m elyxer --css lyx.css $LYX_SRC | \
3318 2 | # iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT | \
3319 3 | # sed 's/UTF-8"\(.\)>/ISO-8859-1"\1>/' > www/docs/fangle.html
3321 5 | python -m elyxer --css lyx.css --iso885915 --html --destdirectory www/docs/fangle.e \
3322 6 | fangle.lyx > www/docs/fangle.e/fangle.html
3324 8 | ( mkdir -p www/docs/fangle && cd www/docs/fangle && \
3325 9 | lyx -e latex ../../../fangle.lyx && \
3326 10 | htlatex ../../../fangle.tex "xhtml,fn-in" && \
3327 11 | sed -i -e 's/<!--l\. [0-9][0-9]* *-->//g' fangle.html
3330 14 | ( mkdir -p www/docs/literate && cd www/docs/literate && \
3331 15 | lyx -e latex ../../../literate.lyx && \
3332 16 | htlatex ../../../literate.tex "xhtml,fn-in" && \
3333 17 | sed -i -e 's/<!--l\. [0-9][0-9]* *-->$//g' literate.html
3335 |________________________________________________________________________
3338 18.3 Extracting from the command line
3339 First you will need the tex output, then you can extract:
3341 96a <lyx-build-manual[1](
\v), lang=sh> ≡
3342 ________________________________________________________________________
3343 1 | lyx -e latex fangle.lyx
3344 2 | fangle -R./fangle fangle.tex > ./fangle
3345 3 | fangle -R./fangle.module fangle.tex > ./fangle.module
3346 |________________________________________________________________________
3353 99a <test:*[1](
\v), lang=> ≡
3354 ________________________________________________________________________
3357 3 | export SRC="${SRC:-./fangle.tm}"
3358 4 | export FANGLE="${FANGLE:-./fangle}"
3359 5 | export TMP="${TMP:-/tmp}"
3360 6 | export TESTDIR="$TMP/$USER/fangle.tests"
3361 7 | export TXT_SRC="${TXT_SRC:-$TESTDIR/fangle.txt}"
3362 8 | export AWK="${AWK:-awk}"
3363 9 | export RUN_FANGLE="${RUN_FANGLE:-$AWK -f}"
3366 12 | ${AWK} -f ${FANGLE} "$@"
3369 15 | mkdir -p "$TESTDIR"
3371 17 | tm -s -c "$SRC" "$TXT_SRC" -q
3373 19 | «test:helpers 99c»
3375 21 | «test:run-tests 99b»
3378 24 | # test current fangle
3379 25 | echo Testing current fangle
3382 28 | # extract new fangle
3383 29 | echo testing new fangle
3384 30 | fangle -R./fangle "$TXT_SRC" > "$TESTDIR/fangle"
3385 31 | export FANGLE="$TESTDIR/fangle"
3388 34 | # Now check that it can extract a fangle that also passes the tests!
3389 35 | echo testing if new fangle can generate itself
3390 36 | fangle -R./fangle "$TXT_SRC" > "$TESTDIR/fangle.new"
3391 37 | passtest diff -bwu "$FANGLE" "$TESTDIR/fangle.new"
3392 38 | export FANGLE="$TESTDIR/fangle.new"
3394 |________________________________________________________________________
3398 99b <test:run-tests[1](
\v), lang=sh> ≡
3399 ________________________________________________________________________
3401 2 | fangle -Rpca-test.awk $TXT_SRC | awk -f - || exit 1
3402 3 | «test:cromulence 59e»
3403 4 | «test:escapes 63c»
3404 5 | «test:test-chunk
\v(test:example-sh
\v) 100a»
3405 6 | «test:test-chunk
\v(test:example-makefile
\v) 100a»
3406 7 | «test:test-chunk
\v(test:q:1
\v) 100a»
3407 8 | «test:test-chunk
\v(test:make:1
\v) 100a»
3408 9 | «test:test-chunk
\v(test:make:2
\v) 100a»
3409 10 | «test:chunk-params 101e»
3410 |________________________________________________________________________
3414 99c <test:helpers[1](
\v), lang=> ≡
3415 ________________________________________________________________________
3418 3 | then echo "Passed $TEST"
3419 4 | else echo "Failed $TEST"
3426 11 | then echo "Passed $TEST"
3427 12 | else echo "Failed $TEST"
3431 |________________________________________________________________________
3434 This chunk will render a named chunk and compare it to another rendered nameed chunk
3436 100a <test:test-chunk[1](chunk
\v\v), lang=sh> ≡
3437 ________________________________________________________________________
3438 1 | «test:test-chunk-result
\v(${chunk}
\v, ${chunk}.result
\v) 100b»
3439 |________________________________________________________________________
3443 100b <test:test-chunk-result[1](chunk
\v, result
\v\v), lang=sh> ≡
3444 ________________________________________________________________________
3445 1 | TEST="${result}" passtest diff -u --label "EXPECTED: ${result}" <( fangle -R${result} $TXT_SRC ) \
3446 2 | --label "ACTUAL: ${chunk}" <( fangle -R${chunk} $TXT_SRC )
3447 |________________________________________________________________________
3450 Chapter 20Chunk Parameters
3453 101a <test:lyx:chunk-params:sub[1](THING
\v, colour
\v\v), lang=> ≡
3454 ________________________________________________________________________
3455 1 | I see a ${THING},
3456 2 | a ${THING} of colour ${colour},
3457 3 | and looking closer =<\chunkref{test:lyx:chunk-params:sub:sub}(${colour})>
3458 |________________________________________________________________________
3462 101b <test:lyx:chunk-params:sub:sub[1](colour
\v\v), lang=> ≡
3463 ________________________________________________________________________
3464 1 | a funny shade of ${colour}
3465 |________________________________________________________________________
3469 101c <test:lyx:chunk-params:text[1](
\v), lang=> ≡
3470 ________________________________________________________________________
3471 1 | What do you see? "=<\chunkref{test:lyx:chunk-params:sub}(joe, red)>"
3473 |________________________________________________________________________
3476 Should generate output:
3478 101d <test:lyx:chunk-params:result[1](
\v), lang=> ≡
3479 ________________________________________________________________________
3480 1 | What do you see? "I see a joe,
3481 2 | a joe of colour red,
3482 3 | and looking closer a funny shade of red"
3484 |________________________________________________________________________
3487 And this chunk will perform the test:
3489 101e <test:chunk-params[1](
\v), lang=> ≡ 102b⊳
3490 ________________________________________________________________________
3491 1 | «test:test-chunk-result
\v(test:lyx:chunk-params:text
\v, test:lyx:chunk-params:result
\v) 100b» || exit 1
3492 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3495 101f <test:chunk-params:sub[1](THING
\v, colour
\v\v), lang=> ≡
3496 ________________________________________________________________________
3497 1 | I see a ${THING},
3498 2 | a ${THING} of colour ${colour},
3499 3 | and looking closer «test:chunk-params:sub:sub
\v(${colour}
\v) 101g»
3500 |________________________________________________________________________
3504 101g <test:chunk-params:sub:sub[1](colour
\v\v), lang=> ≡
3505 ________________________________________________________________________
3506 1 | a funny shade of ${colour}
3507 |________________________________________________________________________
3511 101h <test:chunk-params:text[1](
\v), lang=> ≡ 96a⊳
3512 ________________________________________________________________________
3513 1 | What do you see? "«test:chunk-params:sub
\v(joe
\v, red
\v) 101f»"
3515 |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3516 Should generate output:
3518 102a <test:chunk-params:result[1](
\v), lang=> ≡
3519 ________________________________________________________________________
3520 1 | What do you see? "I see a joe,
3521 2 | a joe of colour red,
3522 3 | and looking closer a funny shade of red"
3524 |________________________________________________________________________
3527 And this chunk will perform the test:
3529 102b <test:chunk-params[2](
\v) ⇑101e, lang=> +≡ ⊲101e
3530 ./\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
3531 2 | «test:test-chunk-result
\v(test:chunk-params:text
\v, test:chunk-params:result
\v) 100b» || exit 1
3532 |________________________________________________________________________
3535 Chapter 21Compile-log-lyx
3537 103a <Chunk:./compile-log-lyx[1](
\v), lang=sh> ≡
3538 ________________________________________________________________________
3540 2 | # can't use gtkdialog -i, cos it uses the "source" command which ubuntu sh doesn't have
3543 5 | errors="/tmp/compile.log.$$"
3544 6 | # if grep '^[^ ]*:\( In \|[0-9][0-9]*: [^ ]*:\)' > $errors
3545 7 | if grep '^[^ ]*(\([0-9][0-9]*\)) *: *\(error\|warning\)' > $errors
3547 9 | sed -i -e 's/^[^ ]*[/\\]\([^/\\]*\)(\([ 0-9][ 0-9]*\)) *: */\1:\2|\2|/' $errors
3548 10 | COMPILE_DIALOG='
3551 13 | <label>Compiler errors:</label>
3553 15 | <tree exported_column="0">
3554 16 | <variable>LINE</variable>
3555 17 | <height>400</height><width>800</width>
3556 18 | <label>File | Line | Message</label>
3557 19 | <action>'". $SELF ; "'lyxgoto $LINE</action>
3558 20 | <input>'"cat $errors"'</input>
3561 23 | <button><label>Build</label>
3562 24 | <action>lyxclient -c "LYXCMD:build-program" &</action>
3564 26 | <button ok></button>
3568 30 | export COMPILE_DIALOG
3569 31 | ( gtkdialog --program=COMPILE_DIALOG ; rm $errors ) &
3576 38 | file="${LINE%:*}"
3577 39 | line="${LINE##*:}"
3578 40 | extraline=‘cat $file | head -n $line | tac | sed '/^\\\\begin{lstlisting}/q' | wc -l‘
3579 41 | extraline=‘expr $extraline - 1‘
3580 42 | lyxclient -c "LYXCMD:command-sequence server-goto-file-row $file $line ; char-forward ; repeat $extraline paragraph-down ; paragraph-up-select"
3584 46 | if test -z "$COMPILE_DIALOG"
3587 |________________________________________________________________________