1 #LyX 1.6.4 created this file. For more info see http://www.lyx.org/
8 %\definecolor{darkgreen}{rgb}{0,0.5,0}
9 \lstset{numbers=left, stepnumber=1, numbersep=5pt, breaklines=false,
10 basicstyle=\footnotesize\ttfamily,
11 %keywordstyle=\color{darkgreen},
12 numberstyle=\tiny,language=C,columns=fullflexible,
15 \use_default_options true
24 \font_typewriter default
25 \font_default_family default
32 \paperfontsize default
35 \pdf_title "Newfangle"
36 \pdf_author "Sam Liddicott"
37 \pdf_subject "Literate Programing"
38 \pdf_keywords "notangle noweb noweave literate programming cweb"
40 \pdf_bookmarksnumbered false
41 \pdf_bookmarksopen false
42 \pdf_bookmarksopenlevel 1
54 \paperorientation portrait
57 \paragraph_separation skip
59 \quotes_language english
62 \paperpagestyle default
63 \tracking_changes false
83 \begin_layout Chapter*
87 \begin_layout Standard
92 is a tool for newfangled literate programming.
93 Newfangled is defined as
95 New and often needlessly novel
104 \begin_layout Standard
105 In this case, newfangled means yet another new and improved method for literate
109 \begin_layout Standard
114 has a long history starting with the great
118 himself, whose literate programming tools seem to make use of as many escaped
119 abbreviations for semantic markup as TeX itself.
122 \begin_layout Standard
131 set of tools (notangle, noweave and noroots) and helpfully reduced the
132 amount of magic character sequences to just
133 \begin_inset Flex CharStyle:Code
136 \begin_layout Plain Layout
143 \begin_inset Flex CharStyle:Code
146 \begin_layout Plain Layout
152 , and in doing so brought the wonders of literate programming within my
156 \begin_layout Standard
157 While using the LyX editor for LaTeX editing I had various troubles with
158 the noweb tools, some of which were my fault, some of which were noweb's
159 fault and some of which were LyX's fault.
162 \begin_layout Standard
167 generally brought literate programming to the masses through removing some
168 of the complexity of the original literate programming, but this would
169 be of no advantage to me if the LyX / LaTeX combination brought more complicati
173 \begin_layout Standard
178 was thus born --- as an awk replacement for notangle, adding some important
179 features, like better integration with LyX and LaTeX, multiple output format
180 conversions, and fixing notangle bugs like indentation when using -L for
184 \begin_layout Standard
185 Significantly, newfangle is just one program which replaces various programs
187 Specifically noweave is done away with and implemented directly as LaTeX
188 macros, and noroots is implemented as a function of the untangler
195 \begin_layout Standard
196 Newfangle is written in awk for portability reasons, awk being available
198 A python conversion will probably be attempted for the benefit of LyX.
199 (Hasn't anyone implemented awk in python yet?)
202 \begin_layout Section*
206 \begin_layout Enumerate
207 ^^ is always going to be a problem, see texbytopic 1.2.2 (Work out what I
211 \begin_layout Enumerate
212 copy over up to date Makefile guide from noweb-lyx document
215 \begin_layout Enumerate
216 Make chunk-name settings only apply to chunks with that name
219 \begin_layout Enumerate
220 indent of multi-line chunks may be mode dependant (i.e.
221 not in string literals)
224 \begin_layout Enumerate
225 support chunk-param usage =<
230 \begin_layout Enumerate
231 trim spaces from param
234 \begin_layout Enumerate
235 add support for other commands in =<...>, starting with
237 label which takes the line-number within the chunk, and maybe should also
238 take the chunk name/page
241 \begin_layout Enumerate
242 cant have listing inside a ruled box
245 \begin_layout Enumerate
246 when a parameterized chunk is included as well as the #line emission, say
247 what the paremeters were for that invocation.
250 \begin_layout Enumerate
251 with 2 macro expansions on one line ${} ${} the first is too greedy and
255 \begin_layout Enumerate
258 chunkref[3]{preamble} to include a certain chunk needs to work in newfangle.awk
259 instead of failing to be recognized at all
262 \begin_layout Enumerate
263 make in-listins labels track the chunk ref too, and make
265 chunref{[2],thing}> resolve to 41c (or d, or whatever chunk the 2nd chunk
269 \begin_layout Enumerate
272 chunkref in text needs a trailing space maybe, it keeps butting up to the
276 \begin_layout Enumerate
277 because the white-space indent is output by the parent chunk, the #line
278 is that of the parent chunk.
279 White space indents must be passed to the child chunk
282 \begin_layout Chapter*
286 \begin_layout Standard
287 \begin_inset CommandInset label
293 Newfangle is licensed under the GPL 3
294 \begin_inset CommandInset citation
301 This doesn't mean that you can't use or distribute newfangle with sources
302 of an incompatible license, but it means you must make the source of newfangle
307 gpl3-copyright,language=
310 \begin_layout Standard
311 \begin_inset listings
315 \begin_layout Plain Layout
317 #newfangle - fully featured notangle replacement in awk
320 \begin_layout Plain Layout
325 \begin_layout Plain Layout
327 #Copyright (C) Sam Liddicott 2009
330 \begin_layout Plain Layout
335 \begin_layout Plain Layout
337 #This program is free software: you can redistribute it and/or modify
340 \begin_layout Plain Layout
342 #it under the terms of the GNU General Public License as published by
345 \begin_layout Plain Layout
347 #the Free Software Foundation, either version 3 of the License, or
350 \begin_layout Plain Layout
352 #(at your option) any later version.
355 \begin_layout Plain Layout
360 \begin_layout Plain Layout
362 #This program is distributed in the hope that it will be useful,
365 \begin_layout Plain Layout
367 #but WITHOUT ANY WARRANTY; without even the implied warranty of
370 \begin_layout Plain Layout
372 #MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
376 \begin_layout Plain Layout
378 #GNU General Public License for more details.
381 \begin_layout Plain Layout
386 \begin_layout Plain Layout
388 #You should have received a copy of the GNU General Public License
391 \begin_layout Plain Layout
393 #along with this program.
394 If not, see <http://www.gnu.org/licenses/>.
402 \begin_layout Standard
403 \begin_inset CommandInset toc
404 LatexCommand tableofcontents
415 \begin_layout Chapter
419 \begin_layout Standard
420 Newfangle is a replacement for noweb, which consists of
421 \begin_inset Flex CharStyle:Code
424 \begin_layout Plain Layout
431 \begin_inset Flex CharStyle:Code
434 \begin_layout Plain Layout
441 \begin_inset Flex CharStyle:Code
444 \begin_layout Plain Layout
453 \begin_layout Standard
455 \begin_inset Flex CharStyle:Code
458 \begin_layout Plain Layout
465 \begin_inset Flex CharStyle:Code
468 \begin_layout Plain Layout
474 it can read multiple named files, or from stdin.
477 \begin_layout Section
481 \begin_layout Standard
482 The -r option causes newfangle to behave like noroots.
485 \begin_layout LyX-Code
486 newfangle -r filename.tex
489 \begin_layout Standard
490 will print out the newfangle roots of a tex file.
494 \begin_layout Standard
496 \begin_inset Flex CharStyle:Code
499 \begin_layout Plain Layout
505 command, the roots are not enclosed in
506 \begin_inset Flex CharStyle:Code
509 \begin_layout Plain Layout
515 , unless at least one of the roots is defined using the
516 \begin_inset Flex CharStyle:Code
519 \begin_layout Plain Layout
526 \begin_inset Flex CharStyle:Code
529 \begin_layout Plain Layout
538 \begin_layout Standard
539 Also, unlike noroots, it prints out all roots --- not just those that are
541 I find that a root not being used, doesn't make it particularly top level.
542 My convention is that top level roots to be extracted begin with
543 \begin_inset Flex CharStyle:Code
546 \begin_layout Plain Layout
552 and have the form of a filename.
555 \begin_layout Section
559 \begin_layout Standard
561 \begin_inset Flex CharStyle:Code
564 \begin_layout Plain Layout
571 \begin_inset Flex CharStyle:Code
574 \begin_layout Plain Layout
580 options are supported.
583 \begin_layout Standard
584 The standard way to extract a file would be:
587 \begin_layout LyX-Code
588 newfangle -R./Makefile.inc newfangle.tex > ./Makefile.inc
591 \begin_layout Standard
593 \begin_inset Flex CharStyle:Code
596 \begin_layout Plain Layout
603 \begin_inset Flex CharStyle:Code
606 \begin_layout Plain Layout
612 option does not break indenting; also the
613 \begin_inset Flex CharStyle:Code
616 \begin_layout Plain Layout
622 option does not interrupt (and break) multi-line C macros --- or indeed
623 any line ending with a backslash.
624 This does mean that sometimes the compiler might calculate the source line
625 wrongly when generating error messages in such cases, but there isn't any
626 other way around if multi-line macros include other chunks.
629 \begin_layout Section
630 Formatting source in LaTeX
633 \begin_layout Standard
634 The noweave replacement is a set of LaTeX macros dependant upon
638 , and which can be included with:
641 \begin_layout LyX-Code
644 usepackage{newfangle.sty}
647 \begin_layout Standard
648 The LaTeX macros are shown in section
649 \begin_inset CommandInset ref
651 reference "sec:Latex-Macros"
655 , and are part of a LyX module file
656 \begin_inset Flex CharStyle:Code
659 \begin_layout Plain Layout
665 , which automatically includes the macros in the document pre-amble when
666 the newfangle LyX module is used.
669 \begin_layout Standard
670 Because the noweave replacement is impemented in LaTeX, there is no processing
671 stage required before running the
672 \begin_inset Flex CharStyle:Code
675 \begin_layout Plain Layout
682 LaTeX may need running two or more times, so that the code chunk references
683 can be fully calculated.
686 \begin_layout Standard
688 \begin_inset Flex CharStyle:Code
691 \begin_layout Plain Layout
697 package is required as it is used for formatting the code chunk captions
700 \begin_layout Standard
702 \begin_inset Flex CharStyle:Code
705 \begin_layout Plain Layout
711 package is also required, as it is used for formatting the code chunks
715 \begin_layout Standard
717 \begin_inset Flex CharStyle:Code
720 \begin_layout Plain Layout
726 package is also required.
729 \begin_layout Chapter
730 Literate Programming with Newfangle
733 \begin_layout Standard
735 Should really follow on from a part-0 explanation of what literate programming
739 \begin_layout Chapter
740 Using Newfangle with LyX
743 \begin_layout Section
747 \begin_layout Subsection
748 Installing the LyX module
751 \begin_layout Standard
753 \begin_inset Flex CharStyle:Code
756 \begin_layout Plain Layout
762 to your LyX layouts directory, which for unix users will be
763 \begin_inset Flex CharStyle:Code
766 \begin_layout Plain Layout
775 \begin_layout Standard
776 You will need to reconfigure LyX by clicking Tools\SpecialChar \menuseparator
777 Reconfigure, and then
781 \begin_layout Subsection
782 \begin_inset CommandInset label
784 name "sub:Configuring-the-build"
788 Configuring the build script
791 \begin_layout Standard
792 Make sure you don't have a conversion defined for Lyx → Program
795 \begin_layout Standard
796 From the menu Tools\SpecialChar \menuseparator
797 Preferences, add a conversion from Latex(Plain) → Program
801 \begin_layout LyX-Code
802 set -x ; newfangle -Rlyx-build $$i |
805 \begin_layout LyX-Code
806 env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
809 \begin_layout Standard
810 (But don't cut-n-paste it from this document or you'll be pasting a multi-line
811 string which will break your lyx preferences file).
815 \begin_layout Standard
816 I hope that one day, LyX will set these into the environment when calling
820 \begin_layout Standard
821 You may also want to consider adding options to this conversion\SpecialChar \ldots{}
825 \begin_layout LyX-Code
826 parselog=/usr/share/lyx/scripts/listerrors
829 \begin_layout Standard
830 \SpecialChar \ldots{}
831 but if you do you will lose your stderr
835 \begin_layout Plain Layout
836 There is some bash plumbing to get a copy of stderr but this footnote is
845 \begin_layout Standard
846 Now, a shell script chunk called
847 \begin_inset Flex CharStyle:Code
850 \begin_layout Plain Layout
856 will be extracted and run whenever you choose the Document\SpecialChar \menuseparator
861 \begin_layout Standard
862 The lyx-build script for this document is in section
863 \begin_inset CommandInset ref
865 reference "lyx-build-script"
869 and on a unix system will extract
870 \begin_inset Flex CharStyle:Code
873 \begin_layout Plain Layout
880 \begin_inset Flex CharStyle:Code
883 \begin_layout Plain Layout
892 \begin_layout Subsection
893 Preparing your Lyx document
896 \begin_layout Standard
897 It is not necessary to base your literate document on any of the original
898 LyX literate classes; so select a regular class for your document type.
901 \begin_layout Standard
917 \begin_layout Standard
918 In the drop-down style listbox you should notice a new style defined, called
926 \begin_layout Standard
927 When you wish to insert a literate chunk, you enter it's plain name in the
928 Chunk style, instead of the older method that used
929 \begin_inset Flex CharStyle:Code
932 \begin_layout Plain Layout
939 Following the chunk name, you insert a listing with: Insert\SpecialChar \menuseparator
943 \begin_layout Standard
944 Inside the white listing box you can type (or paste using shift+ctrl+V)
946 There is not need to use ctrl+enter at the end of lines as with some older
947 LyX literate techniques --- just press enter as normal.
950 \begin_layout Subsubsection
951 Customising the listing appearance
954 \begin_layout Standard
955 In the final document, the code is formatted using the
960 The chunk style doesn't just define the chunk name, but can also define
961 any other chunk options supported by the lstlistings package
962 \begin_inset Flex CharStyle:Code
965 \begin_layout Plain Layout
974 In fact, what you type in the chunk style is raw latex.
975 If you want to set the chunk language without having to right-click the
977 \begin_inset Flex CharStyle:Code
980 \begin_layout Plain Layout
986 after the chunk name.
989 \begin_layout Standard
990 Of course you can do this by editing the listings box advanced properties
991 by right-clicking on the listings box, but that takes longer, and you can't
992 see at-a-glance what the advanced settings are while editing the document;
993 also advanced settings apply only to that box --- the chunk settings apply
994 through the rest of the document
998 \begin_layout Plain Layout
999 It ought to apply only to subsequent chunks of the same name.
1006 \begin_inset Note Note
1009 \begin_layout Plain Layout
1010 So make sure they only apply to chunks of that name
1018 \begin_layout Subsubsection
1019 Global customisations
1022 \begin_layout Standard
1027 is used to set the code chunks, it's
1028 \begin_inset Flex CharStyle:Code
1031 \begin_layout Plain Layout
1039 command can be used in the pre-amble to set some document wide settings.
1042 \begin_layout Standard
1043 If your source has many words with long sequences of capital letters, then
1045 \begin_inset Flex CharStyle:Code
1048 \begin_layout Plain Layout
1049 columns=fullflexible
1054 may be a good idea, or the capital letters will get crowded.
1055 (I think lstlistings ought to use a slightly smaller font for captial letters
1056 so that they still fit).
1059 \begin_layout Standard
1061 \begin_inset Flex CharStyle:Code
1064 \begin_layout Plain Layout
1072 looks more normal for code, but has no bold (unless luximono is used, but
1073 it doesn't work for me); so I use
1074 \begin_inset Flex CharStyle:Code
1077 \begin_layout Plain Layout
1087 \begin_inset Flex CharStyle:Code
1090 \begin_layout Plain Layout
1099 \begin_inset Flex CharStyle:Code
1102 \begin_layout Plain Layout
1103 columns=fullflexible
1108 is used or the wrong letter spacing is used.
1111 \begin_layout Standard
1112 In my LeTeX pre-amble I usually specialise my code format with:
1116 document-preamble,language=tex
1119 \begin_layout Standard
1120 \begin_inset listings
1124 \begin_layout Plain Layout
1131 \begin_layout Plain Layout
1135 definecolor{darkgreen}{rgb}{0,0.5,0}
1138 \begin_layout Plain Layout
1142 lstset{numbers=left, stepnumber=5, numbersep=5pt, breaklines=false,
1145 \begin_layout Plain Layout
1154 \begin_layout Plain Layout
1161 \begin_layout Plain Layout
1165 tiny,language=C,columns=fullflexible,
1168 \begin_layout Plain Layout
1170 numberfirstline=true
1173 \begin_layout Plain Layout
1183 \begin_layout Chapter
1184 Newfangle with Makefiles
1187 \begin_layout Standard
1188 \begin_inset Note Note
1191 \begin_layout Plain Layout
1192 This chapter needs revising
1198 \begin_inset Note Greyedout
1201 \begin_layout Plain Layout
1202 This chapter needs revising
1207 Here we describe a Makefile.inc that you can include in your own Makefiles,
1208 or glue as a recursive make to other projects.
1211 \begin_layout Standard
1212 The Makefile.inc described here was put together for a Samba4 vfs module,
1213 but can be used in any Make project, including automake projects.
1216 \begin_layout Section
1217 A word about makefiles formats
1220 \begin_layout Standard
1221 Whitespace formatting is very important in a Makefile.
1222 The first character of each command line must be a TAB.
1225 \begin_layout LyX-Code
1226 target: pre-requisite
1227 \begin_inset Newline newline
1231 \begin_inset Newline newline
1237 \begin_layout Standard
1238 But a TAB is pretty hard to enter into most of the Lyx formats and insets
1240 An alternative is to use a semi-colon after the pre-requisite, and a backslash
1241 at the end of each line (except the last).
1242 Then any whitespace (or none) can prefix each action.
1245 \begin_layout LyX-Code
1246 target: pre-requisite ;
1249 \begin_inset Newline newline
1255 \begin_inset Newline newline
1261 \begin_layout Standard
1262 This is the style that we use and it works pretty well for GNU make at least.
1265 \begin_layout Standard
1266 We also adopt a convention that code chunks whose names beginning with ./
1267 should always be automatically extracted from the document.
1268 Code chunks whose names do not begin with ./ are for internal reference.
1269 (This doesn't prevent such chunks from being extracted directly).
1272 \begin_layout Section
1273 Boot-strapping the extraction
1276 \begin_layout Subsection
1280 \begin_layout Standard
1281 \begin_inset CommandInset label
1283 name "sub:Bootstrap-Using-a-Makefile"
1287 It seems convenient to have the makefile extract or update the C source
1288 files as part of it's operation.
1289 It also seems convenient to have the makefile itself extracted from this
1293 \begin_layout Standard
1294 It would also be convenient to have the code to extract the makefile from
1295 this document to also be part of this document, however we have to start
1296 somewhere and this unfortunately requires us to type at least a few words
1297 by hand to start things off.
1300 \begin_layout Standard
1301 Therefore we will have a minimal root fragment, which, when extracted, can
1302 cope with extracting the rest of the source.
1303 perhaps with this shell script, which could be called
1308 \begin_inset Note Note
1311 \begin_layout Plain Layout
1312 FIX THIS CHUNK AND TEST IT
1324 \begin_layout Standard
1325 \begin_inset listings
1329 \begin_layout Plain Layout
1334 \begin_layout Plain Layout
1338 \begin_layout Plain Layout
1340 MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
1343 \begin_layout Plain Layout
1345 MAKE_SRC=`dirname "$MAKE_SRC"`/`basename "$MAKE_SRC" .lyx`
1348 \begin_layout Plain Layout
1350 NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
1353 \begin_layout Plain Layout
1355 lyx -e latex $MAKE_SRC
1358 \begin_layout Plain Layout
1362 \begin_layout Plain Layout
1364 newfangle -R./Makefile.inc ${MAKE_SRC}.tex
1369 \begin_layout Plain Layout
1371 | sed "/NEWFANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$NEWFANGLE_SRC"
1376 \begin_layout Plain Layout
1378 | cpif ./Makefile.inc
1381 \begin_layout Plain Layout
1385 \begin_layout Plain Layout
1387 make -f ./Makefile.inc newfangle_sources
1395 \begin_layout Standard
1396 The general Makefile can be invoked with
1400 and can also be included into any automake file to automatically re-generate
1404 \begin_layout Standard
1409 can be extracted with this command:
1412 \begin_layout LyX-Code
1413 lyx -e latex newfangle.lyx &&
1418 \begin_layout LyX-Code
1419 newfangle newfangle.lyx > ./autoboot
1422 \begin_layout Standard
1423 This looks simple enough, but as mentioned, newfangle has to be had from
1424 somewhere before it can be extracted.
1427 \begin_layout Subsection
1428 \begin_inset Note Note
1431 \begin_layout Plain Layout
1432 MERGE THIS WITH THE SECTIONS OF THIS DOCUMENT
1437 \SpecialChar \ldots{}
1441 \begin_layout Standard
1442 When the lyx-build chunk is executed, the current directory will be a temporary
1444 \begin_inset Flex CharStyle:Code
1447 \begin_layout Plain Layout
1453 will refer to the tex file in this temporary directory.
1454 This is unfortunate as our makefile wants to run from the project directory
1455 where the Lyx file is kept.
1458 \begin_layout Standard
1459 We can extract the project directory from $$r, and derive the probable Lyx
1460 filename from the noweb file that Lyx generated.
1467 \begin_layout Standard
1468 \begin_inset listings
1472 \begin_layout Plain Layout
1474 PROJECT_DIR="$LYX_r"
1477 \begin_layout Plain Layout
1479 LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
1482 \begin_layout Plain Layout
1487 \begin_layout Plain Layout
1489 TEX_SRC="$TEX_DIR/$LYX_i"
1497 \begin_layout Standard
1498 And then we can define a lyx-build fragment similar to the autoboot fragment
1505 \begin_layout Standard
1506 \begin_inset listings
1510 \begin_layout Plain Layout
1515 \begin_layout Plain Layout
1519 chunkref{lyx-build-helper}>
1522 \begin_layout Plain Layout
1524 cd $PROJECT_DIR || exit 1
1527 \begin_layout Plain Layout
1531 \begin_layout Plain Layout
1533 #/usr/bin/newfangle -filter ./notanglefix-filter
1538 \begin_layout Plain Layout
1540 # -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx"
1545 \begin_layout Plain Layout
1547 # | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/'
1552 \begin_layout Plain Layout
1557 \begin_layout Plain Layout
1562 \begin_layout Plain Layout
1564 #make -f ./Makefile.inc newfangle_sources
1572 \begin_layout Section
1576 \begin_layout Subsection
1577 Including Makefile.inc
1580 \begin_layout Standard
1581 \begin_inset CommandInset label
1583 name "sub:Keeping-extracted-files"
1587 Makefile.inc will cope with extracting all the other source files from this
1588 document and keeping them up to date.
1592 \begin_layout Standard
1593 It may also be included by a Makefile or Makefile.am defined in a Lyx document
1594 to automatically deal with the extraction of source files and documents.
1597 \begin_layout Standard
1598 A makefile has two parts; variables must be defined before the targets that
1606 \begin_layout Standard
1607 \begin_inset listings
1611 \begin_layout Plain Layout
1615 chunkref{Makefile.inc-vars}>
1618 \begin_layout Plain Layout
1622 chunkref{Makefile.inc-targets}>
1630 \begin_layout Standard
1632 \begin_inset Flex CharStyle:Code
1635 \begin_layout Plain Layout
1641 to hold the name of this Lyx file.
1648 \begin_layout Standard
1649 \begin_inset listings
1653 \begin_layout Plain Layout
1658 \begin_layout Plain Layout
1660 LITERATE_SOURCE=$(LYX_SOURCE)
1668 \begin_layout Subsection
1669 Recursive use of Makefile.inc
1672 \begin_layout Standard
1673 The makefile glue described here is used when building Samba4 vfs modules.
1676 \begin_layout Standard
1677 If you are defining a module of an existing program you may find it easier
1678 to use a slight recursive make instead of including the makefile directly.
1679 This way there is less chance of definitions in Makefile.inc interfering
1680 with definitions in the main makefile, or with definitions in other Makefile.inc
1681 from other noweb modules.
1684 \begin_layout Standard
1685 The glue works by adding a .PHONY target to call the recursive make, and
1686 adding this target as an additional pre-requisite to the existing targets.
1689 \begin_layout Standard
1690 In this example, the existing build system already has a build target for
1692 \begin_inset Flex CharStyle:Code
1695 \begin_layout Plain Layout
1701 , so we just add another pre-requisite to that.
1703 \begin_inset Flex CharStyle:Code
1706 \begin_layout Plain Layout
1712 as a pre-requisite, the stamp file's modified time indicating when all
1713 sources were extracted.
1720 \begin_layout Standard
1721 \begin_inset listings
1725 \begin_layout Plain Layout
1727 $(example_srcdir)/example.o: $(example_srcdir)/example.tex.stamp
1735 \begin_layout Standard
1736 The target for this new pre-requisite is generated by a recursive make using
1737 Makefile.inc which will make sure that the source is up to date, before
1738 it is built by the main projects makefile.
1745 \begin_layout Standard
1746 \begin_inset listings
1750 \begin_layout Plain Layout
1752 $(example_srcdir)/example.tex.stamp: $(example_srcdir)/example.tex ;
1757 \begin_layout Plain Layout
1759 cd $(example_srcdir) &&
1764 \begin_layout Plain Layout
1766 $(MAKE) -f Makefile.inc newfangle_sources
1774 \begin_layout Standard
1775 We can do similar glue for the docs, clean and distclean targets.
1776 In this example our build system is using a double colon for these targets,
1777 so we use the same in our glue.
1784 \begin_layout Standard
1785 \begin_inset listings
1789 \begin_layout Plain Layout
1794 \begin_layout Plain Layout
1796 .PHONY: docs_example
1799 \begin_layout Plain Layout
1801 docs_example:: ; cd $(example_srcdir) &&
1806 \begin_layout Plain Layout
1808 $(MAKE) -f Makefile.inc docs
1811 \begin_layout Plain Layout
1815 \begin_layout Plain Layout
1817 clean:: clean_example
1820 \begin_layout Plain Layout
1822 .PHONEY: clean_example
1825 \begin_layout Plain Layout
1827 clean_example: ; cd $(example_srcdir) &&
1832 \begin_layout Plain Layout
1834 $(MAKE) -f Makefile.inc clean
1837 \begin_layout Plain Layout
1841 \begin_layout Plain Layout
1843 distclean:: distclean_example
1846 \begin_layout Plain Layout
1848 .PHONY: distclean_example
1851 \begin_layout Plain Layout
1853 distclean_example: ; cd $(example_srcdir) &&
1858 \begin_layout Plain Layout
1860 $(MAKE) -f Makefile.inc distclean
1868 \begin_layout Standard
1869 We could do similarly for install targets to install the generated docs.
1872 \begin_layout Subsection
1873 \begin_inset CommandInset label
1875 name "sub:Converting-from-Lyx"
1879 Converting from Lyx to LaTeX
1882 \begin_layout Standard
1883 The first stage will always be to convert the Lyx file to a LaTeX file;
1884 this must be so not only because newfangle needs to to run on a TeX file,
1885 but also because the Lyx command
1887 server-goto-file-line
1891 \begin_layout Plain Layout
1894 server-goto-file-line
1896 is used to position the Lyx cursor at the compiler errors.
1903 insists that the line number provided is a line in the TeX file, and always
1904 reverse maps this to derive the line in the Lyx docment.
1905 \begin_inset Note Note
1908 \begin_layout Plain Layout
1909 The tex file should probably be an automake extra dist sources or something,
1910 so that it gets produced and packaged by make dist
1918 \begin_layout Standard
1919 The command [[lyx -e literate noweb-lyx.lyx]] will produce [[noweb-lyx.nw]]
1920 a tex file, so we define the noweb target to be the same as the Lyx file
1921 but with the .nw extension.
1928 \begin_layout Standard
1929 \begin_inset listings
1933 \begin_layout Plain Layout
1935 TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
1944 Makefile.inc-targets
1947 \begin_layout Standard
1948 \begin_inset listings
1952 \begin_layout Plain Layout
1954 $(TEX_SOURCE): $(LYX_SOURCE) ;
1959 \begin_layout Plain Layout
1964 \begin_layout Plain Layout
1966 clean_tex: ; rm -f -- $(TEX_SOURCE)
1974 \begin_layout Subsection
1975 Extracting Program Source
1978 \begin_layout Standard
1979 The program source is extracted using newfangle, which is designed to operate
1980 on a LaTeX document.
1988 \begin_layout Standard
1989 \begin_inset listings
1993 \begin_layout Plain Layout
1995 NEWFANGLE_SOURCE=$(TEX_SOURCE)
2003 \begin_layout Standard
2004 The Lyx document can result in any number of source documents, but not all
2005 of these will be changed each time the Lyx document is updated.
2006 We certainly don't want to update the timestamps of these files and cause
2007 the whole source tree to be recompiled just because the Lyx document was
2012 \begin_layout Standard
2013 To solve this problem we use a stamp file which is always updated each time
2014 the sources are extracted from the LaTeX document.
2015 If the stamp file is older than the LaTeX document, then we can make an
2016 attempt to re-extract the sources.
2023 \begin_layout Standard
2024 \begin_inset listings
2028 \begin_layout Plain Layout
2030 NEWFANGLE_SOURCE_STAMP=$(NEWFANGLE_SOURCE).stamp
2039 Makefile.inc-targets
2042 \begin_layout Standard
2043 \begin_inset listings
2047 \begin_layout Plain Layout
2049 $(NEWFANGLE_SOURCE_STAMP): $(NEWFANGLE_SOURCE)
2054 \begin_layout Plain Layout
2056 $(NEWFANGLE_SOURCES) ;
2061 \begin_layout Plain Layout
2063 echo > $(NEWFANGLE_SOURCE_STAMP)
2066 \begin_layout Plain Layout
2068 clean_stamp: ; rm -f $(NEWFANGLE_SOURCE_STAMP)
2071 \begin_layout Plain Layout
2081 \begin_layout Subsection
2082 Extracting C sources
2085 \begin_layout Standard
2087 \begin_inset Flex CharStyle:Code
2090 \begin_layout Plain Layout
2096 to hold the names of all the C source files defined in this document.
2097 We compute this only once, by means of := in assignent.
2098 The sed deletes the any <
2099 \begin_inset space \hspace*{}
2104 \begin_inset space \hspace*{}
2108 > which may surround the roots names (for noroots compatibility).
2112 \begin_layout Standard
2113 As we use chunk names beginning with ./ to denote top level fragments that
2114 should be extracted, we filter out all fragments that do not begin with
2122 \begin_layout Standard
2123 \begin_inset listings
2127 \begin_layout Plain Layout
2136 \begin_layout Plain Layout
2138 NEWFANGLE_SOURCES:=$(shell
2143 \begin_layout Plain Layout
2145 newfangle -r $(NEWFANGLE_SOURCE) |
2150 \begin_layout Plain Layout
2152 sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(NEWFANGLE_PREFIX)/!d'
2157 \begin_layout Plain Layout
2159 -e 's/^$(NEWFANGLE_PREFIX)/
2166 \begin_layout Plain Layout
2177 Makefile.inc-targets
2180 \begin_layout Standard
2181 \begin_inset listings
2185 \begin_layout Plain Layout
2187 .PHONY: echo_newfangle_sources
2190 \begin_layout Plain Layout
2192 echo_newfangle_sources: ; @echo $(NEWFANGLE_SOURCES)
2200 \begin_layout Standard
2201 We define a convenient target called
2202 \begin_inset Flex CharStyle:Code
2205 \begin_layout Plain Layout
2211 to re-extract the source if the LaTeX file has been updated.
2215 Makefile.inc-targets
2218 \begin_layout Standard
2219 \begin_inset listings
2223 \begin_layout Plain Layout
2225 .PHONY: newfangle_sources
2228 \begin_layout Plain Layout
2230 newfangle_sources: $(NEWFANGLE_SOURCE_STAMP)
2238 \begin_layout Standard
2239 And also a convenient target to remove extracted sources.
2243 Makefile.inc-targets
2246 \begin_layout Standard
2247 \begin_inset listings
2251 \begin_layout Plain Layout
2253 .PHONY: clean_newfangle_sources
2256 \begin_layout Plain Layout
2258 clean_newfangle_sources: ;
2263 \begin_layout Plain Layout
2265 rm -f -- $(NEWFANGLE_SOURCE_STAMP) $(NEWFANGLE_SOURCES)
2273 \begin_layout Standard
2275 \begin_inset Flex CharStyle:Code
2278 \begin_layout Plain Layout
2284 macro takes 4 arguments: the filename (1), some extensions to match (2)
2285 and a some shell command to return if the filename matches the exentions
2293 \begin_layout Standard
2294 \begin_inset listings
2298 \begin_layout Plain Layout
2300 if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
2308 \begin_layout Standard
2309 For some source files like C files, we want to output the line number and
2310 filename of the original LaTeX document from which the source came.
2313 \begin_layout Standard
2314 To make this easier we define the file extensions for which we want to do
2322 \begin_layout Standard
2323 \begin_inset listings
2327 \begin_layout Plain Layout
2337 \begin_layout Standard
2338 We can then use the if_extensions macro to define a macro which expands
2340 \begin_inset Flex CharStyle:Code
2343 \begin_layout Plain Layout
2349 option if newfangle is being invoked in a C source file, so that C compile
2350 errors will refer to the line number in the Lyx document.
2358 \begin_layout Standard
2359 \begin_inset listings
2363 \begin_layout Plain Layout
2368 \begin_layout Plain Layout
2370 nf_line=-L -T$(TABS)
2373 \begin_layout Plain Layout
2380 \begin_layout Plain Layout
2382 $(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line))
2387 \begin_layout Plain Layout
2397 \begin_layout Standard
2398 We can use a similar trick to define an
2402 macro which takes just the filename as an argument and can return a pipeline
2403 stage calling the indent command.
2404 Indent can be turned off with
2405 \begin_inset Flex CharStyle:Code
2408 \begin_layout Plain Layout
2409 make newfangle_sources indent=
2421 \begin_layout Standard
2422 \begin_inset listings
2426 \begin_layout Plain Layout
2428 indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
2431 \begin_layout Plain Layout
2433 indent=$(call if_extension,$(1),$(C_EXTENSIONS),
2438 \begin_layout Plain Layout
2440 | indent $(indent_options))
2448 \begin_layout Standard
2449 We now define the pattern for extracting a file.
2450 The files are written using noweb's
2456 \begin_layout Plain Layout
2459 So you still need noweb installed in order to use cpif
2465 \begin_inset Note Note
2468 \begin_layout Plain Layout
2471 Write an awk version
2478 so that the file timestamp will not be touched if the contents haven't
2480 This avoids the need to rebuild the entire project because of a typographical
2481 change in the documentation, or if only a few C source files have changed.
2488 \begin_layout Standard
2489 \begin_inset listings
2493 \begin_layout Plain Layout
2495 newfangle_extract=@mkdir -p $(dir $(1)) &&
2500 \begin_layout Plain Layout
2502 $(call newfangle,$(2),$(1)) > "$(1).tmp" &&
2507 \begin_layout Plain Layout
2509 cat "$(1).tmp" $(indent) | cpif "$(1)"
2514 \begin_layout Plain Layout
2516 && rm -- "$(1).tmp" ||
2521 \begin_layout Plain Layout
2523 (echo error newfangling $(1) from $(2) ; exit 1)
2531 \begin_layout Standard
2532 We define a target which will extract or update all sources.
2533 To do this we first defined a makefile template that can do this for any
2534 source file in the LaTeX document.
2541 \begin_layout Standard
2542 \begin_inset listings
2546 \begin_layout Plain Layout
2548 define NEWFANGLE_template
2551 \begin_layout Plain Layout
2558 \begin_layout Plain Layout
2560 $$(call newfangle_extract,$(1),$(2))
2563 \begin_layout Plain Layout
2565 NEWFANGLE_TARGETS+=$(1)
2568 \begin_layout Plain Layout
2578 \begin_layout Standard
2579 We then enumerate the discovered
2580 \begin_inset Flex CharStyle:Code
2583 \begin_layout Plain Layout
2589 to generate a makefile rule for each one using the makefile template we
2594 Makefile.inc-targets
2597 \begin_layout Standard
2598 \begin_inset listings
2602 \begin_layout Plain Layout
2604 $(foreach source,$(NEWFANGLE_SOURCES),
2609 \begin_layout Plain Layout
2611 $(eval $(call NEWFANGLE_template,$(source),$(NEWFANGLE_SOURCE)))
2616 \begin_layout Plain Layout
2626 \begin_layout Standard
2627 These will all be built with NEWFANGLE_SOURCE_STAMP.
2630 \begin_layout Standard
2631 We also remove the generated sources on a
2639 Makefile.inc-targets
2642 \begin_layout Standard
2643 \begin_inset listings
2647 \begin_layout Plain Layout
2649 _distclean: clean_newfangle_sources
2657 \begin_layout Subsection
2658 Extracting Documentation
2661 \begin_layout Standard
2662 We then identify the intermediate stages of the documentation and their
2663 build and clean targets.
2666 \begin_layout Subsubsection
2670 \begin_layout Standard
2671 We produce a pdf file from the tex file.
2678 \begin_layout Standard
2679 \begin_inset listings
2683 \begin_layout Plain Layout
2685 NEWFANGLE_PDF=$(TEX_SOURCE:.tex=.pdf)
2693 \begin_layout Standard
2694 We run pdflatex twice to be sure that the contents and aux files are up
2696 We certainly are required to run pdflatex twice if these files do not exist!
2700 Makefile.inc-targets
2703 \begin_layout Standard
2704 \begin_inset listings
2708 \begin_layout Plain Layout
2710 $(NEWFANGLE_PDF): $(TEX_SOURCE); pdflatex $< && pdflatex $<
2713 \begin_layout Plain Layout
2715 clean_pdf: ; rm -f -- $(NEWFANGLE_PDF)
2720 \begin_layout Plain Layout
2722 $(TEX_SOURCE:.tex=.toc)
2727 \begin_layout Plain Layout
2729 $(TEX_SOURCE:.tex=.log)
2734 \begin_layout Plain Layout
2736 $(TEX_SOURCE:.tex=.aux)
2744 \begin_layout Subsubsection
2748 \begin_layout Standard
2749 Currently we only build pdf as a final format, but NEWFANGLE_DOCS may later
2750 hold other output formats.
2757 \begin_layout Standard
2758 \begin_inset listings
2762 \begin_layout Plain Layout
2764 NEWFANGLE_DOCS=$(NEWFANGLE_PDF)
2772 \begin_layout Standard
2773 We also define newfangle_docs as a convenient phony target<
2777 Makefile.inc-targets
2780 \begin_layout Standard
2781 \begin_inset listings
2785 \begin_layout Plain Layout
2787 .PHONY: newfangle_docs
2790 \begin_layout Plain Layout
2792 newfangle_docs: $(NEWFANGLE_DOCS)
2795 \begin_layout Plain Layout
2797 docs: newfangle_docs
2805 \begin_layout Standard
2806 And define a convenient clean_noweb_docs which we add to the regular clean
2811 Makefile.inc-targets
2814 \begin_layout Standard
2815 \begin_inset listings
2819 \begin_layout Plain Layout
2821 .PHONEY: clean_newfangle_docs
2824 \begin_layout Plain Layout
2826 clean_newfangle_docs: clean_tex clean_pdf
2829 \begin_layout Plain Layout
2831 clean: clean_newfangle_docs
2834 \begin_layout Plain Layout
2838 \begin_layout Plain Layout
2840 distclean_newfangle_docs: clean_tex clean_newfangle_docs
2843 \begin_layout Plain Layout
2845 distclean: clean distclean_newfangle_docs
2853 \begin_layout Subsection
2857 \begin_layout Standard
2858 If Makefile.inc is included into Makefile, then extracted files can be updated
2862 \begin_layout LyX-Code
2863 make newfangle_sources
2866 \begin_layout Standard
2870 \begin_layout LyX-Code
2871 make -f Makefile.inc newfangle_sources
2878 \begin_layout Chapter
2879 Newfangle awk source code
2882 \begin_layout Standard
2883 We use the copyright notice from chapter
2884 \begin_inset CommandInset ref
2886 reference "cha:License"
2894 ./newfangle,language=awk,morestring=[b]{/},morekeywords=else
2897 \begin_layout Standard
2898 \begin_inset listings
2902 \begin_layout Plain Layout
2907 \begin_layout Plain Layout
2911 chunkref{gpl3-copyright}>
2919 \begin_layout Standard
2920 We also use code from Arnold Robbins public domain getopt (1993 revision)
2922 \begin_inset CommandInset ref
2924 reference "cha:getopt"
2928 , and naturally want to attribute this appropriately.
2931 \begin_layout Standard
2932 \begin_inset listings
2936 \begin_layout Plain Layout
2940 \begin_layout Plain Layout
2942 # NOTE: Arnold Robbins public domain getopt for awk is also used:
2945 \begin_layout Plain Layout
2949 chunkref{getopt.awk-header}>
2952 \begin_layout Plain Layout
2956 \begin_layout Plain Layout
2960 chunkref{getopt.awk-getopt()}>
2963 \begin_layout Plain Layout
2972 \begin_layout Standard
2973 And include the following chunks
2980 \begin_layout Standard
2981 \begin_inset listings
2985 \begin_layout Plain Layout
2989 chunkref{helper-functions}>
2992 \begin_layout Plain Layout
2996 chunkref{mode-tracker}>
2999 \begin_layout Plain Layout
3003 chunkref{parse_chunk_args}>
3006 \begin_layout Plain Layout
3010 chunkref{chunk-storage-functions}>
3013 \begin_layout Plain Layout
3017 chunkref{output_chunk_names()}>
3020 \begin_layout Plain Layout
3024 chunkref{output_chunks()}>
3027 \begin_layout Plain Layout
3031 chunkref{write_chunk()}>
3034 \begin_layout Plain Layout
3038 chunkref{expand_chunk_args()}>
3041 \begin_layout Plain Layout
3048 \begin_layout Plain Layout
3052 chunkref{recognize-chunk}>
3055 \begin_layout Plain Layout
3067 \begin_layout Section
3071 \begin_layout Standard
3072 The portable way to erase an array in awk is to split the empty string,
3077 awk-delete-array,params=ARRAY
3080 \begin_layout Standard
3081 \begin_inset listings
3085 \begin_layout Plain Layout
3087 split("", ${ARRAY});
3096 dump-array,params=ARRAY
3099 \begin_layout Standard
3100 \begin_inset listings
3104 \begin_layout Plain Layout
3115 \begin_layout Plain Layout
3117 for (_x in ${ARRAY}) {
3120 \begin_layout Plain Layout
3122 print _x "=" ${ARRAY}[_x] "
3127 \begin_layout Plain Layout
3132 \begin_layout Plain Layout
3148 \begin_layout Section
3152 \begin_layout Standard
3153 Fatal errors are issued with the error function:
3157 error(),append=helper-functions
3160 \begin_layout Standard
3161 \begin_inset listings
3165 \begin_layout Plain Layout
3167 function error(message)
3170 \begin_layout Plain Layout
3175 \begin_layout Plain Layout
3177 print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
3180 \begin_layout Plain Layout
3185 \begin_layout Plain Layout
3195 \begin_layout Standard
3196 \begin_inset listings
3200 \begin_layout Plain Layout
3202 function warning(message)
3205 \begin_layout Plain Layout
3210 \begin_layout Plain Layout
3212 print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
3215 \begin_layout Plain Layout
3220 \begin_layout Plain Layout
3230 \begin_layout Chapter
3234 \begin_layout Standard
3235 LaTeX arguments to lstlistings macros are a comma seperated list of key-value
3237 Values containing commas are enclosed in { braces }.
3240 \begin_layout Standard
3241 We need a function that can parse such an expression and assign the values
3249 \begin_layout Standard
3250 A sample expressions is:
3253 \begin_layout LyX-Code
3254 name=thomas, params={a, b}, something, something-else
3257 \begin_layout Standard
3258 but we see that this is just a simpler form of this expression:
3261 \begin_layout LyX-Code
3262 name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
3265 \begin_layout Standard
3266 And that it would be a good idea to use a recursive parser into a multi-dimensio
3271 \begin_layout Plain Layout
3272 as AWK doesn't have nested-hash support
3280 \begin_layout Standard
3281 \begin_inset Tabular
3282 <lyxtabular version="3" rows="6" columns="2">
3284 <column alignment="left" valignment="top" width="0">
3285 <column alignment="left" valignment="top" width="0">
3287 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
3290 \begin_layout Plain Layout
3296 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
3299 \begin_layout Plain Layout
3307 <cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
3310 \begin_layout Plain Layout
3316 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3319 \begin_layout Plain Layout
3327 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3330 \begin_layout Plain Layout
3336 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3339 \begin_layout Plain Layout
3347 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3350 \begin_layout Plain Layout
3356 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3359 \begin_layout Plain Layout
3367 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3370 \begin_layout Plain Layout
3376 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3379 \begin_layout Plain Layout
3387 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
3390 \begin_layout Plain Layout
3396 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
3399 \begin_layout Plain Layout
3413 \begin_layout Standard
3414 On reflection it seems that sometimes such nesting is not desirable, as
3415 the braces are also used to delimit values that contain commas --- we may
3419 \begin_layout LyX-Code
3420 name={williamson, freddie}
3423 \begin_layout Standard
3425 \begin_inset Flex CharStyle:Code
3428 \begin_layout Plain Layout
3435 \begin_inset Flex CharStyle:Code
3438 \begin_layout Plain Layout
3444 --- so I may change this behaviour.
3445 \begin_inset Note Note
3448 \begin_layout Plain Layout
3457 \begin_layout Standard
3459 \begin_inset Flex Chunkref
3462 \begin_layout Plain Layout
3468 will accept two paramters,
3469 \begin_inset Flex CharStyle:Code
3472 \begin_layout Plain Layout
3478 being the text to parse, and
3479 \begin_inset Flex CharStyle:Code
3482 \begin_layout Plain Layout
3488 being an array to receive the parsed values as described above.
3489 The optional parameter
3490 \begin_inset Flex CharStyle:Code
3493 \begin_layout Plain Layout
3499 is used during recursion to build up the multi-dimensional array path.
3506 \begin_layout Standard
3507 \begin_inset listings
3511 \begin_layout Plain Layout
3515 chunkref{get_chunk_args()}>
3527 \begin_layout Standard
3528 \begin_inset listings
3532 \begin_layout Plain Layout
3534 function get_chunk_args(text, values,
3537 \begin_layout Plain Layout
3539 # optional parameters
3542 \begin_layout Plain Layout
3544 path, # hierarchical precursors
3547 \begin_layout Plain Layout
3552 \begin_layout Plain Layout
3562 \begin_layout Standard
3563 The strategy is to parse the name, and then look for a value.
3564 If the value begins with a brace
3565 \begin_inset Flex CharStyle:Code
3568 \begin_layout Plain Layout
3574 , then we recurse and consume as much of the text as necessary, returning
3575 the remaining text when we encounter a leading close-brace
3576 \begin_inset Flex CharStyle:Code
3579 \begin_layout Plain Layout
3586 This being the strategy --- and executed in a loop --- we realise that
3587 we must first look for the closing brace (perhaps preceded by white space)
3588 in order to terminate the recursion, and returning remaining text.
3591 \begin_layout Standard
3592 \begin_inset listings
3596 \begin_layout Plain Layout
3601 \begin_layout Plain Layout
3603 split("", next_chunk_args);
3606 \begin_layout Plain Layout
3608 while(length(text)) {
3611 \begin_layout Plain Layout
3613 if (match(text, "^ *}(.*)", a)) {
3616 \begin_layout Plain Layout
3621 \begin_layout Plain Layout
3626 \begin_layout Plain Layout
3630 chunkref{parse-chunk-args}>
3633 \begin_layout Plain Layout
3638 \begin_layout Plain Layout
3643 \begin_layout Plain Layout
3653 \begin_layout Standard
3654 \begin_inset Note Note
3657 \begin_layout Plain Layout
3658 Use BNF package here
3663 We can see that the text could be inspected with this regex:
3670 \begin_layout Standard
3671 \begin_inset listings
3675 \begin_layout Plain Layout
3677 if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a))
3681 \begin_layout Plain Layout
3686 \begin_layout Plain Layout
3696 \begin_layout Standard
3698 \begin_inset Flex CharStyle:Code
3701 \begin_layout Plain Layout
3707 will have the following values:
3710 \begin_layout Standard
3711 \begin_inset Tabular
3712 <lyxtabular version="3" rows="7" columns="2">
3714 <column alignment="center" valignment="top" width="0">
3715 <column alignment="left" valignment="top" width="0">
3717 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
3720 \begin_layout Plain Layout
3726 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
3729 \begin_layout Plain Layout
3737 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3740 \begin_layout Plain Layout
3746 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3749 \begin_layout Plain Layout
3757 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3760 \begin_layout Plain Layout
3766 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3769 \begin_layout Plain Layout
3770 =freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
3777 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3780 \begin_layout Plain Layout
3786 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3789 \begin_layout Plain Layout
3797 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3800 \begin_layout Plain Layout
3806 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3809 \begin_layout Plain Layout
3810 freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
3817 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3820 \begin_layout Plain Layout
3826 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3829 \begin_layout Plain Layout
3837 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
3840 \begin_layout Plain Layout
3846 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
3849 \begin_layout Plain Layout
3850 , foo={bar=baz, quux={quirk, a=fleeg}}, etc
3863 \begin_layout Standard
3865 \begin_inset Flex CharStyle:Code
3868 \begin_layout Plain Layout
3875 \begin_inset Flex CharStyle:Code
3878 \begin_layout Plain Layout
3884 and signify whether the option named in
3885 \begin_inset Flex CharStyle:Code
3888 \begin_layout Plain Layout
3894 has a value or not (respectively).
3897 \begin_layout Standard
3898 If the option does have a value, then if the expression
3899 \begin_inset Flex CharStyle:Code
3902 \begin_layout Plain Layout
3909 \begin_inset Flex CharStyle:Code
3912 \begin_layout Plain Layout
3918 it will signify that we need to recurse:
3921 \begin_layout Standard
3922 \begin_inset listings
3926 \begin_layout Plain Layout
3931 \begin_layout Plain Layout
3936 \begin_layout Plain Layout
3938 if (substr(a[4],1,1) == "{") {
3941 \begin_layout Plain Layout
3943 text = get_chunk_args(substr(a[4],2), values, path name SUBSEP);
3946 \begin_layout Plain Layout
3951 \begin_layout Plain Layout
3953 values[path name]=a[5];
3956 \begin_layout Plain Layout
3961 \begin_layout Plain Layout
3966 \begin_layout Plain Layout
3971 \begin_layout Plain Layout
3973 values[path name]="";
3976 \begin_layout Plain Layout
3981 \begin_layout Plain Layout
3991 \begin_layout Standard
3992 We can test this function like this:
3999 \begin_layout Standard
4000 \begin_inset listings
4004 \begin_layout Plain Layout
4008 chunkref{get_chunk_args()}>
4011 \begin_layout Plain Layout
4016 \begin_layout Plain Layout
4021 \begin_layout Plain Layout
4025 \begin_layout Plain Layout
4027 print get_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}},
4031 \begin_layout Plain Layout
4036 \begin_layout Plain Layout
4038 print "a[" b "] => " a[b];
4041 \begin_layout Plain Layout
4046 \begin_layout Plain Layout
4056 \begin_layout Standard
4057 which should give this output:
4061 gca-test.awk-results
4064 \begin_layout Standard
4065 \begin_inset listings
4069 \begin_layout Plain Layout
4071 a[foo.quux.quirk] =>
4074 \begin_layout Plain Layout
4076 a[foo.quux.a] => fleeg
4079 \begin_layout Plain Layout
4084 \begin_layout Plain Layout
4089 \begin_layout Plain Layout
4099 \begin_layout Chapter
4103 \begin_layout Standard
4104 \begin_inset CommandInset label
4111 \begin_inset Note Greyedout
4114 \begin_layout Plain Layout
4115 This feature is in-development and does not work yet
4121 \begin_inset Note Note
4124 \begin_layout Plain Layout
4133 \begin_layout Standard
4134 lstlistings and newfangle both recognize source languages, and perform some
4136 lstlistings can detect strings and comments within a language definition
4137 and perform suitable rendering, such as italics for comments, and visible-space
4141 \begin_layout Standard
4142 Newfangle similarly can recognize strings, and comments, etc, within a language,
4143 so that any chunks included with
4144 \begin_inset Flex CharStyle:Code
4147 \begin_layout Plain Layout
4155 can be suitably escape or quoted.
4158 \begin_layout Standard
4159 For instance, consider this chunk with
4160 \begin_inset Flex CharStyle:Code
4163 \begin_layout Plain Layout
4173 example-perl,language=perl
4176 \begin_layout Standard
4177 \begin_inset listings
4181 \begin_layout Plain Layout
4191 \begin_layout Standard
4192 If it were included in a chunk with
4193 \begin_inset Flex CharStyle:Code
4196 \begin_layout Plain Layout
4206 example-sh,language=sh
4209 \begin_layout Standard
4210 \begin_inset listings
4214 \begin_layout Plain Layout
4218 chunkref{example-perl}>"
4226 \begin_layout Standard
4227 newfangle would need to generate output like this if it were to work:
4230 \begin_layout LyX-Code
4238 \begin_layout Standard
4239 See that the double quote
4240 \begin_inset Flex CharStyle:Code
4243 \begin_layout Plain Layout
4250 \begin_inset Flex CharStyle:Code
4253 \begin_layout Plain Layout
4259 in the regex have been quoted with a back-slash to protect them from shell
4263 \begin_layout Standard
4264 If that were then included in a chunk with
4265 \begin_inset Flex CharStyle:Code
4268 \begin_layout Plain Layout
4278 example-makefile,language=make
4281 \begin_layout Standard
4282 \begin_inset listings
4286 \begin_layout Plain Layout
4291 \begin_layout Plain Layout
4295 chunkref{example-sh}>
4303 \begin_layout Standard
4304 We would need the output to look like this --- note the $$:
4307 \begin_layout LyX-Code
4311 \begin_layout LyX-Code
4319 \begin_layout Standard
4320 In order to make this work, we need to define a mode-tracker supporting
4321 each language, that can detect the various quoting modes, and provide a
4322 transformation that must be applied to any included text so that included
4323 text will be interpreted correctly after any interpolation that it may
4324 be subject to at run-time.
4327 \begin_layout Standard
4328 For example, the sed transformation for text to be inserted into sh double-quote
4329 d strings would be something like:
4332 \begin_layout LyX-Code
4356 \begin_layout Standard
4358 \begin_inset Flex CharStyle:Code
4361 \begin_layout Plain Layout
4372 \begin_layout Standard
4373 \begin_inset Note Note
4376 \begin_layout Plain Layout
4377 I don't think this example is true
4382 The mode tracker must also track nested mode-changes, as in this
4383 \begin_inset Flex CharStyle:Code
4386 \begin_layout Plain Layout
4395 \begin_layout LyX-Code
4396 echo "hello `id \SpecialChar \ldots{}
4400 \begin_layout LyX-Code
4404 \begin_layout Plain Layout
4410 hphantom{echo "hello `id}
4418 \begin_layout Standard
4419 Any characters inserted at the point marked ↑ would need to be escaped,
4421 \begin_inset Flex CharStyle:Code
4424 \begin_layout Plain Layout
4431 First it would need escaping for the back-ticks `, and then for the double-quot
4435 \begin_layout Standard
4436 Escaping need not occur if the format and mode of the included chunk matches
4437 that of the including chunk.
4440 \begin_layout Standard
4441 As each chunk is output a new mode tracker for that language is initialized
4442 in it's normal state.
4443 As text is output for that chunk the output mode is tracked.
4444 When a new chunk is included, a transformation appropriate to that mode
4445 is selected and pushed onto a stack of transformations.
4446 Any text to be output is first passed through this stack of transformations.
4449 \begin_layout Standard
4450 It remains to consider if the chunk-include function should return it's
4451 generated text so that the caller can apply any transformations (and formatting
4452 ), or if it should apply the stack of transformations itself.
4455 \begin_layout Standard
4456 Note that the transformed text should have the property of not being able
4457 to change the mode in the current chunk.
4460 \begin_layout Standard
4461 \begin_inset Note Note
4464 \begin_layout Plain Layout
4465 Note chunk parameters should probably also be transformed
4473 \begin_layout Section
4477 \begin_layout Standard
4478 The mode tracker holds its state in a stack based on a hash.
4479 This function, when passed an empty hash will intialize it.
4486 \begin_layout Standard
4487 \begin_inset listings
4491 \begin_layout Plain Layout
4493 function new_mode_tracker(context, language, mode) {
4496 \begin_layout Plain Layout
4501 \begin_layout Plain Layout
4503 context[0, "language"] = language;
4506 \begin_layout Plain Layout
4508 context[0, "mode"] = mode;
4511 \begin_layout Plain Layout
4521 \begin_layout Standard
4522 Because awk functions cannot return an array, we must create the array first
4523 and pass it in, so we have a newfangle macro to do this:
4527 new-mode-tracker,params=context;language;mode
4530 \begin_layout Standard
4531 \begin_inset listings
4535 \begin_layout Plain Layout
4539 chunkref{awk-delete-array}(${context})>
4542 \begin_layout Plain Layout
4544 new_mode_tracker(${context}, ${language}, ${mode});
4552 \begin_layout Standard
4553 And for tracking modes, we dispatch to a mode-tracker action based on the
4561 \begin_layout Standard
4562 \begin_inset listings
4566 \begin_layout Plain Layout
4568 function push_mode_tracker(context, language, mode,
4571 \begin_layout Plain Layout
4576 \begin_layout Plain Layout
4581 \begin_layout Plain Layout
4586 \begin_layout Plain Layout
4588 if (! ("" in context)) {
4591 \begin_layout Plain Layout
4595 chunkref{new-mode-tracker}(context, language, mode)>
4598 \begin_layout Plain Layout
4603 \begin_layout Plain Layout
4608 \begin_layout Plain Layout
4610 if (context[top, "language"] == language && mode=="") mode = context[top,
4614 \begin_layout Plain Layout
4619 \begin_layout Plain Layout
4621 context[top, "language"] = language;
4624 \begin_layout Plain Layout
4626 context[top, "mode"] = mode;
4629 \begin_layout Plain Layout
4634 \begin_layout Plain Layout
4639 \begin_layout Plain Layout
4649 \begin_layout Standard
4650 \begin_inset listings
4654 \begin_layout Plain Layout
4656 function finalize_mode_tracker(context,
4659 \begin_layout Plain Layout
4664 \begin_layout Plain Layout
4669 \begin_layout Plain Layout
4674 \begin_layout Plain Layout
4676 if ( ("" in context) && context[""] != 0) return 0;
4679 \begin_layout Plain Layout
4684 \begin_layout Plain Layout
4694 \begin_layout Standard
4695 This implies that any chunk must be syntactically whole; for instance, this
4703 \begin_layout Standard
4704 \begin_inset listings
4708 \begin_layout Plain Layout
4713 \begin_layout Plain Layout
4717 chunkref{test:say-hello}>
4720 \begin_layout Plain Layout
4734 \begin_layout Standard
4735 \begin_inset listings
4739 \begin_layout Plain Layout
4749 \begin_layout Standard
4750 But this is not fine; the chunk
4751 \begin_inset Flex Chunkref
4754 \begin_layout Plain Layout
4760 is not properly cromulent.
4767 \begin_layout Standard
4768 \begin_inset listings
4772 \begin_layout Plain Layout
4777 \begin_layout Plain Layout
4781 chunkref{test:hidden-else}>
4784 \begin_layout Plain Layout
4798 \begin_layout Standard
4799 \begin_inset listings
4803 \begin_layout Plain Layout
4808 \begin_layout Plain Layout
4813 \begin_layout Plain Layout
4823 \begin_layout Standard
4824 These tests will check for correct behaviour:
4831 \begin_layout Standard
4832 \begin_inset listings
4836 \begin_layout Plain Layout
4838 echo Cromulence test
4841 \begin_layout Plain Layout
4843 passtest ./newfangle -Rtest:whole-chunk $TEX_SRC &>/dev/null || ( echo "Whole
4844 chunk failed" && exit 1 )
4847 \begin_layout Plain Layout
4849 failtest ./newfangle -Rtest:partial-chunk $TEX_SRC &>/dev/null || ( echo
4850 "Partial chunk failed" && exit 1 )
4858 \begin_layout Section
4862 \begin_layout Standard
4863 In the C language there are a few parse modes, affecting the interpretation
4867 \begin_layout Standard
4868 One parse mode is the strings mode.
4869 The string mode is commenced by an un-escaped quotation mark
4870 \begin_inset Flex CharStyle:Code
4873 \begin_layout Plain Layout
4879 and terminated by the same.
4880 Within the string mode, only one additional mode can be commenced, it is
4882 \begin_inset Flex CharStyle:Code
4885 \begin_layout Plain Layout
4893 , which is always terminated by the folloing character.
4896 \begin_layout Standard
4898 \begin_inset Flex CharStyle:Code
4901 \begin_layout Plain Layout
4907 which is terminated by a
4908 \begin_inset Flex CharStyle:Code
4911 \begin_layout Plain Layout
4917 (unless it occurs in a string).
4920 \begin_layout Standard
4921 Consider this line of C code:
4924 \begin_layout Standard
4925 \begin_inset Formula $\mathtt{things\underbrace{[\mathtt{x,\ y}]}_{1.\ [\ mode},\ get\_other\_things\underbrace{(\mathtt{a,\overbrace{"\mathtt{(all)}"}})}_{2.\ (\ mode}^{3.\ "\ mode}}$
4931 \begin_layout Standard
4932 Mode nesting prevents the close parenthesis in quote mode (part 3) from
4933 terminating the parenthesis mode (part 2).
4936 \begin_layout Standard
4937 Each language has a set of modes, the default mode being the null mode.
4938 Each mode can lead to other modes.
4941 \begin_layout Subsection
4945 \begin_layout Standard
4946 All modes are stored in a single multi-dimensional hash.
4947 The first index is the language, and the second index is the mode-identifier.
4948 The third indexes are terminators, and optionally, submodes, and delimiters.
4951 \begin_layout Standard
4952 A useful set of mode definitions for a nameless general C-type language
4954 (Don't be confused by the double backslash escaping needed in awk.
4955 One set of escaping is for the string, and the second set of escaping is
4959 mode{}> command which will allow us to signify that a string is regex and
4960 thus newfangle will quote it for us.
4963 \begin_layout Standard
4964 Submodes are entered by the characters
4965 \begin_inset Flex CharStyle:Code
4968 \begin_layout Plain Layout
4977 \begin_inset Flex CharStyle:Code
4980 \begin_layout Plain Layout
4981 \begin_inset Quotes eld
4990 \begin_inset Flex CharStyle:Code
4993 \begin_layout Plain Layout
5000 \begin_inset Flex CharStyle:Code
5003 \begin_layout Plain Layout
5010 \begin_inset Flex CharStyle:Code
5013 \begin_layout Plain Layout
5020 \begin_inset Flex CharStyle:Code
5023 \begin_layout Plain Layout
5036 \begin_layout Standard
5037 \begin_inset listings
5041 \begin_layout Plain Layout
5043 modes["", "", "submodes" ]="
5073 \begin_layout Standard
5074 In the default mode, a comma surrounded by un-important white space is a
5075 delimiter of language items.
5078 \begin_layout Standard
5079 \begin_inset listings
5083 \begin_layout Plain Layout
5085 modes["", "", "delimiters"]=" *, *";
5093 \begin_layout Standard
5094 and should pass this test:
5097 \begin_layout Standard
5098 \begin_inset Note Note
5101 \begin_layout Plain Layout
5102 Why do the tests run in
5103 \begin_inset Quotes eld
5107 \begin_inset Quotes erd
5111 \begin_inset Quotes eld
5115 \begin_inset Quotes erd
5127 test:mode-definitions
5130 \begin_layout Standard
5131 \begin_inset listings
5135 \begin_layout Plain Layout
5137 parse_chunk_args("", "1,2,3", a, "");
5140 \begin_layout Plain Layout
5142 if (a[1] != "1") e++;
5145 \begin_layout Plain Layout
5147 if (a[2] != "2") e++;
5150 \begin_layout Plain Layout
5152 if (a[3] != "3") e++;
5155 \begin_layout Plain Layout
5157 if (length(a) != 3) e++;
5160 \begin_layout Plain Layout
5164 chunkref{pca-test.awk:summary}>
5167 \begin_layout Plain Layout
5171 \begin_layout Plain Layout
5173 parse_chunk_args("", "joe, red", a, "");
5176 \begin_layout Plain Layout
5178 if (a[1] != "joe") e++;
5181 \begin_layout Plain Layout
5183 if (a[2] != "red") e++;
5186 \begin_layout Plain Layout
5188 if (length(a) != 2) e++;
5191 \begin_layout Plain Layout
5195 chunkref{pca-test.awk:summary}>
5198 \begin_layout Plain Layout
5202 \begin_layout Plain Layout
5204 parse_chunk_args("", "${colour}", a, "");
5207 \begin_layout Plain Layout
5209 if (a[1] != "${colour}") e++;
5212 \begin_layout Plain Layout
5214 if (length(a) != 1) e++;
5217 \begin_layout Plain Layout
5221 chunkref{pca-test.awk:summary}>
5229 \begin_layout Standard
5230 Nested modes are identified by a backslash, a double or single quote, various
5231 bracket styles or a /* comment.
5234 \begin_layout Standard
5235 For each of these sub-modes modes we must also identify at a mode terminator,
5236 and any sub-modes or delimiters that may be entered
5240 \begin_layout Plain Layout
5241 Because we are using the sub-mode characters as the mode identifier it means
5242 we can't currently have a mode character dependant on it's context; i.e.
5244 \begin_inset Flex CharStyle:Code
5247 \begin_layout Plain Layout
5253 can't behave differently when it is inside
5254 \begin_inset Flex CharStyle:Code
5257 \begin_layout Plain Layout
5271 \begin_layout Subsection
5275 \begin_layout Standard
5276 The backslash mode has no submodes or delimiters, and is terminated by any
5278 Note that we are not so much interested in evaluating or interpolating
5279 content as we are in delineating content.
5280 It is no matter that a double backslash (
5281 \begin_inset Flex CharStyle:Code
5284 \begin_layout Plain Layout
5294 ) may represent a single backslash while a backslash-newline may represent
5295 white space, but it does matter that the newline in a backslash newline
5296 should not be able to terminate a C pre-processor statement; and so the
5297 newline will be consumed by the backslash however it is to be interpreted.
5304 \begin_layout Standard
5305 \begin_inset listings
5309 \begin_layout Plain Layout
5315 ", "terminators"]=".";
5323 \begin_layout Subsection
5327 \begin_layout Standard
5328 In a string we have one special mode, which is the backslash.
5329 This may escape an embedded quote and prevent us thinking that it should
5330 terminate the string.
5331 Otherwise, the string will be terminated by a double-quote.
5334 \begin_layout Standard
5335 \begin_inset listings
5339 \begin_layout Plain Layout
5354 \begin_layout Plain Layout
5358 "", "terminators"]="
5368 \begin_layout Standard
5369 Working strings should pass this test:
5373 test:mode-definitions
5376 \begin_layout Standard
5377 \begin_inset listings
5381 \begin_layout Plain Layout
5383 parse_chunk_args("", "say
5402 \begin_layout Plain Layout
5423 \begin_layout Plain Layout
5425 if (a[2] != "for me") e++;
5428 \begin_layout Plain Layout
5430 if (length(a) != 2) e++;
5433 \begin_layout Plain Layout
5437 chunkref{pca-test.awk:summary}>
5445 \begin_layout Standard
5446 Also, the parser must return any spare text at the end that has not been
5447 processed due to a mode terminator being found.
5450 \begin_layout Standard
5451 \begin_inset listings
5455 \begin_layout Plain Layout
5457 rest = parse_chunk_args("", "1, 2, 3) spare", a, "(");
5460 \begin_layout Plain Layout
5465 \begin_layout Plain Layout
5470 \begin_layout Plain Layout
5475 \begin_layout Plain Layout
5477 if (length(a) != 3) e++;
5480 \begin_layout Plain Layout
5482 if (rest != " spare") e++;
5485 \begin_layout Plain Layout
5489 chunkref{pca-test.awk:summary}>
5501 \begin_layout Standard
5502 \begin_inset listings
5506 \begin_layout Plain Layout
5508 modes["", "{", "submodes" ]="
5533 \begin_layout Plain Layout
5535 modes["", "{", "delimeters"]=" *, *";
5538 \begin_layout Plain Layout
5540 modes["", "{", "terminators"]="}";
5543 \begin_layout Plain Layout
5545 modes["", "[", "submodes" ]="
5570 \begin_layout Plain Layout
5572 modes["", "[", "delimiters"]=" *, *";
5575 \begin_layout Plain Layout
5577 modes["", "[", "terminators"]="
5584 \begin_layout Plain Layout
5586 modes["", "(", "submodes" ]="
5611 \begin_layout Plain Layout
5613 modes["", "(", "delimiters"]=" *, *";
5616 \begin_layout Plain Layout
5618 modes["", "(", "terminators"]="
5625 \begin_layout Plain Layout
5627 modes["", "'", "submodes" ]="
5638 \begin_layout Plain Layout
5640 modes["", "'", "terminators"]="'";
5643 \begin_layout Plain Layout
5645 modes["", "/*", "submodes"]="
5652 \begin_layout Plain Layout
5654 modes["", "/*", "terminators"]="*/";
5657 \begin_layout Plain Layout
5659 modes["", "//", "submodes"]="
5664 \begin_layout Plain Layout
5666 modes["", "//", "terminators"]="
5671 \begin_layout Plain Layout
5673 modes["", "", "submodes" ]="
5703 \begin_layout Standard
5704 This test should also pass.
5708 test:mode-definitions
5711 \begin_layout Standard
5712 \begin_inset listings
5716 \begin_layout Plain Layout
5718 parse_chunk_args("", "things[x, y], get_other_things(a,
5725 \begin_layout Plain Layout
5727 if (a[1] != "things[x, y]") e++;
5730 \begin_layout Plain Layout
5732 if (a[2] != "get_other_things(a,
5739 \begin_layout Plain Layout
5741 if (a[3] != "99") e++;
5744 \begin_layout Plain Layout
5746 if (length(a) != 3) e++;
5749 \begin_layout Plain Layout
5753 chunkref{pca-test.awk:summary}>
5761 \begin_layout Subsection
5762 A non-recursive mode tracker
5765 \begin_layout Standard
5766 We must avoid recursion as a language construct because we intend to employ
5767 mode-tracking to track language mode of emitted code, and the code is emitted
5768 from a function which is itself recursive, so instead we implement psuedo-recur
5769 sion using our own stack based on a hash.
5776 \begin_layout Standard
5777 \begin_inset listings
5781 \begin_layout Plain Layout
5783 function mode_tracker(context, text, values,
5786 \begin_layout Plain Layout
5788 # optional parameters
5791 \begin_layout Plain Layout
5796 \begin_layout Plain Layout
5798 mode, submodes, language,
5801 \begin_layout Plain Layout
5803 cindex, c, a, part, item, name, result, new_values, new_mode,
5806 \begin_layout Plain Layout
5808 delimiters, terminators)
5811 \begin_layout Plain Layout
5821 \begin_layout Standard
5822 We could be re-commencing with a valid context, so we need to setup the
5823 state according to the last context.
5826 \begin_layout Standard
5827 \begin_inset listings
5831 \begin_layout Plain Layout
5833 cindex = context[""] + 0;
5836 \begin_layout Plain Layout
5838 mode = context[cindex, "mode"];
5841 \begin_layout Plain Layout
5843 language = context[cindex, "language" ];
5851 \begin_layout Standard
5852 First we construct a single large regex combining the possile sub-modes
5853 for the current mode along with the terminators for the current mode.
5857 parse_chunk_args-reset-modes
5860 \begin_layout Standard
5861 \begin_inset listings
5865 \begin_layout Plain Layout
5867 submodes=modes[language, mode, "submodes"];
5870 \begin_layout Plain Layout
5872 if ((language, mode, "delimiters") in modes) {
5875 \begin_layout Plain Layout
5877 delimiters = modes[language, mode, "delimiters"];
5880 \begin_layout Plain Layout
5882 submodes=submodes "|" delimiters;
5885 \begin_layout Plain Layout
5890 \begin_layout Plain Layout
5892 if ((language, mode, "terminators") in modes) {
5895 \begin_layout Plain Layout
5897 terminators = modes[language, mode, "terminators"];
5900 \begin_layout Plain Layout
5902 submodes=submodes "|" terminators;
5905 \begin_layout Plain Layout
5919 \begin_layout Standard
5920 \begin_inset listings
5924 \begin_layout Plain Layout
5928 chunkref{parse_chunk_args-reset-modes}>
5936 \begin_layout Standard
5937 We then iterate the text (until there is none left) looking for sub-modes
5938 or terminators in the regex.
5941 \begin_layout Standard
5942 \begin_inset listings
5946 \begin_layout Plain Layout
5948 while((cindex >= 0) && length(text)) {
5951 \begin_layout Plain Layout
5953 if (match(text, "(" submodes ")", a)) {
5961 \begin_layout Standard
5962 A bug that creeps in regularly during development is bad regexes of zero
5963 length which result in an infinite loop (as no text is consumed), so I
5964 catch that right away with this test.
5967 \begin_layout Standard
5968 \begin_inset listings
5972 \begin_layout Plain Layout
5977 \begin_layout Plain Layout
5979 error("Internal error, matched zero length submode, should be impossible
5980 - likely regex computation error");
5983 \begin_layout Plain Layout
5993 \begin_layout Standard
5994 \begin_inset Flex CharStyle:Code
5997 \begin_layout Plain Layout
6003 is defined as the text up to the sub-mode or terminator, and this is appended
6005 \begin_inset Flex CharStyle:Code
6008 \begin_layout Plain Layout
6014 --- which is the current text being gathered.
6015 If a mode has a delimiter, then item is reset each time a delimiter is
6019 \begin_layout Standard
6020 \begin_inset Formula $\mathtt{\overbrace{"\overbrace{hello}^{item},\ \overbrace{there}^{item}"}^{item},\ \overbrace{he\ said.}^{item}}$
6026 \begin_layout Standard
6027 \begin_inset listings
6031 \begin_layout Plain Layout
6033 part = substr(text, 1, RSTART -1);
6036 \begin_layout Plain Layout
6046 \begin_layout Standard
6047 We must now determine what was matched.
6048 If it was a terminator, then we must restore the previous mode.
6051 \begin_layout Standard
6052 \begin_inset listings
6056 \begin_layout Plain Layout
6058 if (match(a[1], "^" terminators "$")) {
6061 \begin_layout Plain Layout
6063 context[cindex, "values", ++context[cindex, "values"]] = item;
6066 \begin_layout Plain Layout
6068 delete context[cindex];
6071 \begin_layout Plain Layout
6073 context[""] = --cindex;
6076 \begin_layout Plain Layout
6081 \begin_layout Plain Layout
6083 mode = context[cindex, "mode"];
6086 \begin_layout Plain Layout
6088 language = context[cindex, "language"];
6091 \begin_layout Plain Layout
6095 chunkref{parse_chunk_args-reset-modes}>
6098 \begin_layout Plain Layout
6103 \begin_layout Plain Layout
6108 \begin_layout Plain Layout
6110 text = substr(text, 1 + length(part) + length(a[1]));
6113 \begin_layout Plain Layout
6123 \begin_layout Standard
6124 If a delimiter was matched, then we must store the current item in the parsed
6125 values array, and reset the item.
6128 \begin_layout Standard
6129 \begin_inset listings
6133 \begin_layout Plain Layout
6135 else if (match(a[1], "^" delimiters "$")) {
6138 \begin_layout Plain Layout
6143 \begin_layout Plain Layout
6145 context[cindex, "values", ++context[cindex, "values"]] = item;
6148 \begin_layout Plain Layout
6153 \begin_layout Plain Layout
6158 \begin_layout Plain Layout
6163 \begin_layout Plain Layout
6168 \begin_layout Plain Layout
6170 text = substr(text, 1 + length(part) + length(a[1]));
6173 \begin_layout Plain Layout
6183 \begin_layout Standard
6184 otherwise, if a new submode is detected (all submodes have terminators),
6185 we must create a nested parse context until we find the terminator for
6189 \begin_layout Standard
6190 \begin_inset listings
6194 \begin_layout Plain Layout
6196 else if ((language, a[1], "terminators") in modes) {
6199 \begin_layout Plain Layout
6201 #check if new_mode is defined
6204 \begin_layout Plain Layout
6209 \begin_layout Plain Layout
6211 text = substr(text, 1 + length(part) + length(a[1]));
6214 \begin_layout Plain Layout
6218 \begin_layout Plain Layout
6220 context[""] = ++cindex;
6223 \begin_layout Plain Layout
6225 context[cindex, "mode"] = a[1];
6228 \begin_layout Plain Layout
6230 context[cindex, "language"] = language;
6233 \begin_layout Plain Layout
6238 \begin_layout Plain Layout
6242 chunkref{parse_chunk_args-reset-modes}>
6245 \begin_layout Plain Layout
6250 \begin_layout Plain Layout
6252 error(sprintf("Submode '%s' set unknown mode in text: %s", a[1],
6256 \begin_layout Plain Layout
6258 text = substr(text, 1 + length(part) + length(a[1]));
6261 \begin_layout Plain Layout
6266 \begin_layout Plain Layout
6276 \begin_layout Standard
6277 In the final case, we parsed to the end of the string.
6278 If the string was entire, then we should have no nested mode context, but
6279 if the string was just a fragment we may have a mode context which must
6280 be preserved for the next fragment.
6281 Todo: Consideration ought to be given if sub-mode strings are split over
6283 \begin_inset Note Note
6286 \begin_layout Plain Layout
6287 Consideration ought to be given if sub-mode strings are split over two fragments.
6295 \begin_layout Standard
6296 \begin_inset listings
6300 \begin_layout Plain Layout
6305 \begin_layout Plain Layout
6307 context[cindex, "values", ++context[cindex, "values"]] = item text;
6310 \begin_layout Plain Layout
6315 \begin_layout Plain Layout
6320 \begin_layout Plain Layout
6325 \begin_layout Plain Layout
6330 \begin_layout Plain Layout
6334 \begin_layout Plain Layout
6336 context["item"] = item;
6339 \begin_layout Plain Layout
6343 \begin_layout Plain Layout
6345 if (length(item)) context[cindex, "values", ++context[cindex, "values"]]
6349 \begin_layout Plain Layout
6354 \begin_layout Plain Layout
6364 \begin_layout Subsection
6368 \begin_layout Standard
6369 All the mode tracker chunks are referred to here:
6376 \begin_layout Standard
6377 \begin_inset listings
6381 \begin_layout Plain Layout
6385 chunkref{new_mode_tracker()}>
6388 \begin_layout Plain Layout
6392 chunkref{mode_tracker()}>
6400 \begin_layout Subsection
6404 \begin_layout Standard
6405 We can test this function like this:
6412 \begin_layout Standard
6413 \begin_inset listings
6417 \begin_layout Plain Layout
6424 \begin_layout Plain Layout
6428 chunkref{mode-tracker}>
6431 \begin_layout Plain Layout
6435 chunkref{parse_chunk_args()}>
6438 \begin_layout Plain Layout
6443 \begin_layout Plain Layout
6448 \begin_layout Plain Layout
6452 chunkref{mode-definitions}>
6455 \begin_layout Plain Layout
6459 \begin_layout Plain Layout
6463 chunkref{test:mode-definitions}>
6466 \begin_layout Plain Layout
6477 pca-test.awk:summary
6480 \begin_layout Standard
6481 \begin_inset listings
6485 \begin_layout Plain Layout
6490 \begin_layout Plain Layout
6495 \begin_layout Plain Layout
6500 \begin_layout Plain Layout
6502 print "a[" b "] => " a[b];
6505 \begin_layout Plain Layout
6510 \begin_layout Plain Layout
6515 \begin_layout Plain Layout
6520 \begin_layout Plain Layout
6525 \begin_layout Plain Layout
6530 \begin_layout Plain Layout
6540 \begin_layout Standard
6541 which should give this output:
6545 pca-test.awk-results
6548 \begin_layout Standard
6549 \begin_inset listings
6553 \begin_layout Plain Layout
6555 a[foo.quux.quirk] =>
6558 \begin_layout Plain Layout
6560 a[foo.quux.a] => fleeg
6563 \begin_layout Plain Layout
6568 \begin_layout Plain Layout
6573 \begin_layout Plain Layout
6583 \begin_layout Chapter
6584 Expanding chunk arguments
6587 \begin_layout Standard
6588 \begin_inset CommandInset label
6590 name "cha:Chunk Arguments"
6595 \begin_inset Note Note
6598 \begin_layout Plain Layout
6599 Explain this in the documentation section too
6604 As an extension to many literate-programming styles, newfangle permits code
6605 chunks to take parameters and thus operate somewhat like C pre-processor
6606 macros, or like C++ templates.
6609 \begin_layout Standard
6610 Chunk parameters are declared with a chunk argument called
6611 \begin_inset Flex CharStyle:Code
6614 \begin_layout Plain Layout
6620 , which holds a semi-colon separated list of parameters, like this:
6623 \begin_layout LyX-Code
6624 achunk,language=C,params=name;address
6627 \begin_layout Standard
6628 When such a chunk is included, the arguments are expressed in round brackets
6629 as a comma separated list of optional arguments:
6630 \begin_inset Note Note
6633 \begin_layout Plain Layout
6634 We ought to support qouting in {} like ({Jones, John}, Jones@example.com)
6642 \begin_layout LyX-Code
6645 chunkref{achunk}(John Jones, jones@example.com)
6648 \begin_layout Standard
6649 Within the body of a chunk, the parameters are referred to with:
6650 \begin_inset Flex CharStyle:Code
6653 \begin_layout Plain Layout
6660 \begin_inset Flex CharStyle:Code
6663 \begin_layout Plain Layout
6670 There is a strong case that a LaTeX style notation should be used, like
6673 param{name} which would be expressed in the listing as =<
6675 param{name}> and be rendered as
6676 \begin_inset listings
6680 \begin_layout Plain Layout
6690 Such notation would make me go blind, but I do intend to adopt it.
6693 \begin_layout Standard
6694 We therefore need a function
6695 \begin_inset Flex CharStyle:Code
6698 \begin_layout Plain Layout
6704 which will take a block of text, a list of permitted parameters and the
6705 arguments which must substitute for the parameters.
6709 \begin_layout Standard
6710 We also need to be able to parse a parameter list into an array of parameters.
6713 \begin_layout Section
6714 Parsing argument lists
6717 \begin_layout Standard
6718 An argument list may be as simple as in
6719 \begin_inset Flex CharStyle:Code
6722 \begin_layout Plain Layout
6725 chunkref{pull}(thing, otherthing)
6733 \begin_layout LyX-Code
6736 chunkref{pull}(things[x, y], get_other_things(a, "(all)"))
6739 \begin_layout Standard
6740 --- which for all it's commas and quotes and parenthesis represents only
6742 \begin_inset Flex CharStyle:Code
6745 \begin_layout Plain Layout
6752 \begin_inset Flex CharStyle:Code
6755 \begin_layout Plain Layout
6756 get_other_things(a, "(all)")
6764 \begin_layout Standard
6765 If we simply split parameter list on commas, then the comma in
6766 \begin_inset Flex CharStyle:Code
6769 \begin_layout Plain Layout
6775 would split into two seperate arguments:
6776 \begin_inset Flex CharStyle:Code
6779 \begin_layout Plain Layout
6786 \begin_inset Flex CharStyle:Code
6789 \begin_layout Plain Layout
6795 --- neither of which make sense on their own.
6798 \begin_layout Standard
6799 One way to prevent this would be by refusing to split text between maching
6801 \begin_inset Flex CharStyle:Code
6804 \begin_layout Plain Layout
6811 \begin_inset Flex CharStyle:Code
6814 \begin_layout Plain Layout
6821 \begin_inset Flex CharStyle:Code
6824 \begin_layout Plain Layout
6831 \begin_inset Flex CharStyle:Code
6834 \begin_layout Plain Layout
6841 \begin_inset Flex CharStyle:Code
6844 \begin_layout Plain Layout
6851 \begin_inset Flex CharStyle:Code
6854 \begin_layout Plain Layout
6860 and most likely also
6861 \begin_inset Flex CharStyle:Code
6864 \begin_layout Plain Layout
6871 \begin_inset Flex CharStyle:Code
6874 \begin_layout Plain Layout
6881 \begin_inset Flex CharStyle:Code
6884 \begin_layout Plain Layout
6891 \begin_inset Flex CharStyle:Code
6894 \begin_layout Plain Layout
6901 Of course this also makes it impossible to pass such mis-matched code fragments
6902 as parameters, but I think that it would be hard for readers to cope with
6903 authors who would pass such code unbalanced fragments as chunk parameters
6907 \begin_layout Plain Layout
6908 I know that I couldn't cope with users doing such things, and although the
6909 GPL3 license prevents me from actually forbidding anyone from trying, if
6910 they want it to work they'll have to write the code themselves and not
6911 expect any support from me.
6919 \begin_layout Standard
6920 Unfortunately, the full set of matching delimiters may vary from language
6922 In certain C++ template contexts,
6923 \begin_inset Flex CharStyle:Code
6926 \begin_layout Plain Layout
6933 \begin_inset Flex CharStyle:Code
6936 \begin_layout Plain Layout
6942 would count as delimiters, and yet in other contexts they would not.
6945 \begin_layout Standard
6946 This puts me in the unfortunate position of having to parse-somewhat all
6947 programming languages without knowing what they are!
6950 \begin_layout Standard
6951 This is the basis of mode-tracking, by tracking parse-modes for different
6959 \begin_layout Standard
6960 \begin_inset listings
6964 \begin_layout Plain Layout
6966 function parse_chunk_args(language, text, values, mode,
6969 \begin_layout Plain Layout
6974 \begin_layout Plain Layout
6979 \begin_layout Plain Layout
6984 \begin_layout Plain Layout
6988 chunkref{new-mode-tracker}(context, language, mode)>
6991 \begin_layout Plain Layout
6993 rest = mode_tracker(context, text, values);
6996 \begin_layout Plain Layout
7001 \begin_layout Plain Layout
7003 for(c=1; c <= context[0, "values"]; c++) {
7006 \begin_layout Plain Layout
7008 values[c] = context[0, "values", c];
7011 \begin_layout Plain Layout
7016 \begin_layout Plain Layout
7021 \begin_layout Plain Layout
7031 \begin_layout Section
7032 Expanding parameters
7035 \begin_layout Standard
7036 \begin_inset CommandInset label
7038 name "Here-we-split"
7042 Here we split the text on
7043 \begin_inset Flex CharStyle:Code
7046 \begin_layout Plain Layout
7052 which means that all parts except the first will begin with a parameter
7054 The split function will consume the literal
7055 \begin_inset Flex CharStyle:Code
7058 \begin_layout Plain Layout
7071 \begin_layout Standard
7072 \begin_inset listings
7076 \begin_layout Plain Layout
7078 function expand_chunk_args(text, params, args,
7081 \begin_layout Plain Layout
7083 p, text_array, next_text, v, t, l)
7086 \begin_layout Plain Layout
7091 \begin_layout Plain Layout
7093 if (split(text, text_array, "
7100 \begin_layout Plain Layout
7104 chunkref{substitute-chunk-args}>
7107 \begin_layout Plain Layout
7112 \begin_layout Plain Layout
7116 \begin_layout Plain Layout
7121 \begin_layout Plain Layout
7131 \begin_layout Standard
7132 First, we produce an associative array of substitution values indexed by
7137 substitute-chunk-args
7140 \begin_layout Standard
7141 \begin_inset listings
7145 \begin_layout Plain Layout
7150 \begin_layout Plain Layout
7152 v[params[p]]=args[p];
7155 \begin_layout Plain Layout
7165 \begin_layout Standard
7166 We accumulate substituted text in the variable
7167 \begin_inset Flex CharStyle:Code
7170 \begin_layout Plain Layout
7177 As the first part of the split function is the part before the delimiter
7179 \begin_inset Flex CharStyle:Code
7182 \begin_layout Plain Layout
7188 in our case --- this part will never contain a parameter reference, so
7189 we assign this directly to the result kept in
7190 \begin_inset Flex CharStyle:Code
7193 \begin_layout Plain Layout
7200 \begin_inset listings
7204 \begin_layout Plain Layout
7214 \begin_layout Standard
7215 We then iterate over the remaining values in the array
7219 \begin_layout Plain Layout
7220 I don't know why I think that it will enumerate the array in order, but
7227 \begin_inset Note Note
7230 \begin_layout Plain Layout
7231 So fix it or porve it
7236 , and substitute each reference for it's argument.
7239 \begin_layout Standard
7240 \begin_inset listings
7244 \begin_layout Plain Layout
7246 for(t=2; t in text_array; t++) {
7249 \begin_layout Plain Layout
7253 chunkref{substitute-chunk-arg}>
7256 \begin_layout Plain Layout
7266 \begin_layout Standard
7268 \begin_inset Flex CharStyle:Code
7271 \begin_layout Plain Layout
7277 a valid parameter reference will consist of valid parameter name terminated
7279 \begin_inset Flex CharStyle:Code
7282 \begin_layout Plain Layout
7289 A valid character name begins with the underscore or a letter, and may
7290 contain letters, digits or underscores.
7293 \begin_layout Standard
7294 A valid looking reference that is not actually the name of a parameter will
7295 be and not substituted.
7296 This is good because there is nothing to substitute anyway, and it avoids
7297 clashes when writing code for languages where ${\SpecialChar \ldots{}
7298 } is a valid construct
7299 --- such constructs will not be interfered with unless the parameter name
7304 substitute-chunk-arg
7307 \begin_layout Standard
7308 \begin_inset listings
7312 \begin_layout Plain Layout
7314 if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
7317 \begin_layout Plain Layout
7322 \begin_layout Plain Layout
7327 \begin_layout Plain Layout
7329 text = text v[l[1]] substr(text_array[t], length(l[1])+2);
7332 \begin_layout Plain Layout
7337 \begin_layout Plain Layout
7339 text = text "${" text_array[t];
7342 \begin_layout Plain Layout
7352 \begin_layout Chapter
7356 \begin_layout Standard
7357 Newfangle recognizes noweb chunks, but as we also want better LaTeX integration
7358 we will recognize any of these:
7361 \begin_layout Itemize
7362 notangle chunks matching the pattern
7363 \begin_inset Flex CharStyle:Code
7366 \begin_layout Plain Layout
7368 \begin_inset space \hspace*{}
7373 \begin_inset space \hspace*{}
7385 \begin_layout Itemize
7386 a chunks beginning with
7387 \begin_inset Flex CharStyle:Code
7390 \begin_layout Plain Layout
7400 Chunk{\SpecialChar \ldots{}
7401 } on the previous line
7404 \begin_layout Itemize
7405 an older form I have used, beginning with
7406 \begin_inset Flex CharStyle:Code
7409 \begin_layout Plain Layout
7412 begin{Chunk}[options]
7417 --- also more suitable for plain LaTeX users
7421 \begin_layout Plain Layout
7422 Is there such a thing as plain LaTeX?
7430 \begin_layout Section
7434 \begin_layout Standard
7436 \begin_inset Flex CharStyle:Code
7439 \begin_layout Plain Layout
7445 is used to signify that we are processing a code chunk and not document.
7446 In such a state, input lines will be assigned to the current chunk; otherwise
7450 \begin_layout Subsection
7454 \begin_layout Standard
7455 Our current scheme is to recognize the new lstlisting chunks, but these
7456 may be preceded by a
7457 \begin_inset Flex CharStyle:Code
7460 \begin_layout Plain Layout
7468 command which in LyX is a more convenient way to pass the chunk name to
7470 \begin_inset Flex CharStyle:Code
7473 \begin_layout Plain Layout
7481 command, and a more visible way to specify other
7482 \begin_inset Flex CharStyle:Code
7485 \begin_layout Plain Layout
7494 \begin_layout Standard
7495 The arguments to the
7496 \begin_inset Flex CharStyle:Code
7499 \begin_layout Plain Layout
7507 command are a name, and then a comma-seperated list of key-value pairs
7509 \begin_inset Flex CharStyle:Code
7512 \begin_layout Plain Layout
7521 (In fact within the LaTeX
7522 \begin_inset Flex CharStyle:Code
7525 \begin_layout Plain Layout
7534 \begin_inset CommandInset ref
7536 reference "sub:The-chunk-command"
7541 \begin_inset Flex CharStyle:Code
7544 \begin_layout Plain Layout
7550 is prefixed to the argument which is then literally passed to
7551 \begin_inset Flex CharStyle:Code
7554 \begin_layout Plain Layout
7569 \begin_layout Standard
7570 \begin_inset listings
7574 \begin_layout Plain Layout
7583 \begin_layout Plain Layout
7593 Chunk{ *([^ ,}]*),?(.*)}", line)) {
7596 \begin_layout Plain Layout
7598 next_chunk_name = line[1];
7601 \begin_layout Plain Layout
7603 get_chunk_args(line[2], next_chunk_args);
7606 \begin_layout Plain Layout
7611 \begin_layout Plain Layout
7616 \begin_layout Plain Layout
7626 \begin_layout Standard
7627 We also make a basic attempt to parse the name out of the
7628 \begin_inset Flex CharStyle:Code
7631 \begin_layout Plain Layout
7635 \begin_inset space \hspace{}
7644 text, otherwise we fall back to the name found in the previous chunk command.
7645 This attempt is very basic and doesn't support commas or spaces or square
7646 brackets as part of the chunkname.
7648 \begin_inset Flex CharStyle:Code
7651 \begin_layout Plain Layout
7659 which is convenient for some users
7663 \begin_layout Plain Layout
7664 but not yet supported in the LaTeX macros
7670 \begin_inset Note Note
7673 \begin_layout Plain Layout
7682 \begin_layout Standard
7683 \begin_inset listings
7687 \begin_layout Plain Layout
7700 \begin_layout Plain Layout
7702 if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
7705 \begin_layout Plain Layout
7710 \begin_layout Plain Layout
7715 \begin_layout Plain Layout
7717 new_chunk(next_chunk_name, next_chunk_args);
7720 \begin_layout Plain Layout
7725 \begin_layout Plain Layout
7730 \begin_layout Plain Layout
7735 \begin_layout Plain Layout
7745 \begin_layout Subsection
7749 \begin_layout Standard
7750 We recognize notangle style chunks too:
7757 \begin_layout Standard
7758 \begin_inset listings
7762 \begin_layout Plain Layout
7767 \begin_layout Plain Layout
7769 if (match($0, "^[<]<(.*)[>]>= *$", line)) {
7772 \begin_layout Plain Layout
7777 \begin_layout Plain Layout
7782 \begin_layout Plain Layout
7787 \begin_layout Plain Layout
7792 \begin_layout Plain Layout
7797 \begin_layout Plain Layout
7807 \begin_layout Section
7811 \begin_layout Standard
7812 Likewise, we need to recognize when a chunk ends.
7815 \begin_layout Subsection
7819 \begin_layout Standard
7821 \begin_inset Flex CharStyle:Code
7824 \begin_layout Plain Layout
7831 \begin_inset Flex CharStyle:Code
7834 \begin_layout Plain Layout
7840 is surrounded by square brackets so that when this document is processed,
7841 this chunk doesn't terminate early when the lstlistings package recognizes
7842 it's own end-string!
7843 \begin_inset Note Greyedout
7846 \begin_layout Plain Layout
7847 This doesn't make sense as the regex is anchored with ^, which this line
7848 does not begin with!
7854 \begin_inset Note Note
7857 \begin_layout Plain Layout
7870 \begin_layout Standard
7871 \begin_inset listings
7875 \begin_layout Plain Layout
7888 \begin_layout Plain Layout
7893 \begin_layout Plain Layout
7898 \begin_layout Plain Layout
7903 \begin_layout Plain Layout
7913 \begin_layout Subsection
7921 \begin_layout Standard
7922 \begin_inset listings
7926 \begin_layout Plain Layout
7931 \begin_layout Plain Layout
7936 \begin_layout Plain Layout
7941 \begin_layout Plain Layout
7951 \begin_layout Standard
7952 All other recognizers are only of effect if we are chunking; there's no
7953 point in looking at lines if they aren't part of a chunk, so we just ignore
7954 them as efficiently as we can.
7961 \begin_layout Standard
7962 \begin_inset listings
7966 \begin_layout Plain Layout
7968 ! chunking { next; }
7976 \begin_layout Section
7980 \begin_layout Standard
7981 Chunk contents are any lines read while
7982 \begin_inset Flex CharStyle:Code
7985 \begin_layout Plain Layout
7992 Some chunk contents are special in that they refer to other chunks, and
7993 will be replaced by the contents of these chunks when the file is generated.
7996 \begin_layout Standard
7997 \begin_inset CommandInset label
7999 name "sub:ORS-chunk-text"
8003 We add the output record separator
8004 \begin_inset Flex CharStyle:Code
8007 \begin_layout Plain Layout
8013 to the line now, because we will set
8014 \begin_inset Flex CharStyle:Code
8017 \begin_layout Plain Layout
8023 to the empty string when we generate the output
8027 \begin_layout Plain Layout
8028 So that we can print partial lines using
8029 \begin_inset Flex CharStyle:Code
8032 \begin_layout Plain Layout
8039 \begin_inset Flex CharStyle:Code
8042 \begin_layout Plain Layout
8060 \begin_layout Standard
8061 \begin_inset listings
8065 \begin_layout Plain Layout
8067 length(active_chunk) {
8070 \begin_layout Plain Layout
8074 chunkref{process-chunk-tabs}>
8077 \begin_layout Plain Layout
8081 chunkref{process-chunk}>
8084 \begin_layout Plain Layout
8094 \begin_layout Standard
8095 If a chunk just consisted of plain text, we could handle the chunk like
8100 process-chunk-simple
8103 \begin_layout Standard
8104 \begin_inset listings
8108 \begin_layout Plain Layout
8110 chunk_line(active_chunk, $0 ORS);
8118 \begin_layout Standard
8119 but in fact a chunk can include references to other chunks.
8120 Chunk includes are traditionally written as
8121 \begin_inset Flex CharStyle:Code
8124 \begin_layout Plain Layout
8130 , but we support other variations.
8133 \begin_layout Standard
8134 However, we also process tabs at this point, a tab at input can be replaced
8135 by a number of spaces defined by the
8136 \begin_inset Flex CharStyle:Code
8139 \begin_layout Plain Layout
8145 variable, set by the
8146 \begin_inset Flex CharStyle:Code
8149 \begin_layout Plain Layout
8156 Of course this is poor tab behaviour, we should probably have the option
8157 to use proper counted tab-stops and process this on output.
8164 \begin_layout Standard
8165 \begin_inset listings
8169 \begin_layout Plain Layout
8174 \begin_layout Plain Layout
8181 \begin_layout Plain Layout
8191 \begin_layout Subsection
8192 \begin_inset CommandInset label
8194 name "sub:lstlistings-includes"
8201 \begin_layout Standard
8203 \begin_inset Flex CharStyle:Code
8206 \begin_layout Plain Layout
8209 lstset{escapeinside={=<}{>}}
8214 is set, then we can use
8215 \begin_inset Flex CharStyle:Code
8218 \begin_layout Plain Layout
8222 \begin_inset space \hspace{}
8233 \begin_inset Flex CharStyle:Code
8236 \begin_layout Plain Layout
8245 \begin_layout Enumerate
8246 it is a better mnemonic than
8247 \begin_inset Flex CharStyle:Code
8250 \begin_layout Plain Layout
8256 in that the = sign signifies equivalent or substitutability,
8259 \begin_layout Enumerate
8260 and because =< is not valid in C or in any language I can think of
8263 \begin_layout Enumerate
8264 and also because lstlistings doesn't like
8265 \begin_inset Flex CharStyle:Code
8268 \begin_layout Plain Layout
8274 as an end delimiter for the
8278 escape, so we must make do with a single
8279 \begin_inset Flex CharStyle:Code
8282 \begin_layout Plain Layout
8288 , which is better matched by
8289 \begin_inset Flex CharStyle:Code
8292 \begin_layout Plain Layout
8299 \begin_inset Flex CharStyle:Code
8302 \begin_layout Plain Layout
8311 \begin_layout Standard
8312 As each chunk line may contain more than one chunk include, we will split
8313 out chunk includes in an iterative fashion
8317 \begin_layout Plain Layout
8318 Contrary to our use of
8319 \begin_inset Flex CharStyle:Code
8322 \begin_layout Plain Layout
8328 when substituting parameters in chapter
8329 \begin_inset CommandInset ref
8331 reference "Here-we-split"
8343 \begin_layout Standard
8344 First, as long as the chunk contains a
8345 \begin_inset Flex CharStyle:Code
8348 \begin_layout Plain Layout
8356 command we take as much as we can up to the first
8357 \begin_inset Flex CharStyle:Code
8360 \begin_layout Plain Layout
8375 \begin_layout Standard
8376 \begin_inset listings
8380 \begin_layout Plain Layout
8385 \begin_layout Plain Layout
8390 \begin_layout Plain Layout
8395 \begin_layout Plain Layout
8413 )|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)",
8416 \begin_layout Plain Layout
8423 \begin_layout Plain Layout
8428 \begin_layout Plain Layout
8430 chunklet = substr(chunk, 1, RSTART - 1);
8438 \begin_layout Standard
8439 We keep track of the indent count, by counting the number of literal characters
8441 We can then preserve this indent on each output line when multi-line chunks
8445 \begin_layout Standard
8446 We then process this first part literal text, and set the chunk which is
8447 still to be processed to be the text after the
8448 \begin_inset Flex CharStyle:Code
8451 \begin_layout Plain Layout
8459 command, which we will process next as we continue around the loop.
8462 \begin_layout Standard
8463 \begin_inset listings
8467 \begin_layout Plain Layout
8469 indent += length(chunklet);
8472 \begin_layout Plain Layout
8474 chunk_line(active_chunk, chunklet);
8477 \begin_layout Plain Layout
8479 chunk = substr(chunk, RSTART + RLENGTH);
8487 \begin_layout Standard
8488 We then consider the type of chunk command we have found, whether it is
8489 the newfangle style command beginning with
8490 \begin_inset Flex CharStyle:Code
8493 \begin_layout Plain Layout
8499 or the older notangle style beginning with
8500 \begin_inset Flex CharStyle:Code
8503 \begin_layout Plain Layout
8513 \begin_layout Standard
8514 Newfangle chunks may have parameters contained within square brackets.
8515 These will be matched in
8516 \begin_inset Flex CharStyle:Code
8519 \begin_layout Plain Layout
8525 and are considered at this stage of processing to be part of the name of
8526 the chunk to be included.
8529 \begin_layout Standard
8530 \begin_inset listings
8534 \begin_layout Plain Layout
8536 if (substr(line[1], 1, 1) == "=") {
8539 \begin_layout Plain Layout
8541 # chunk name up to }
8544 \begin_layout Plain Layout
8546 chunk_include(active_chunk, line[2] line[3], indent);
8549 \begin_layout Plain Layout
8551 } else if (substr(line[1], 1, 1) == "<") {
8554 \begin_layout Plain Layout
8556 chunk_include(active_chunk, line[4], indent);
8559 \begin_layout Plain Layout
8564 \begin_layout Plain Layout
8566 error("Unknown chunk fragment: " line[1]);
8569 \begin_layout Plain Layout
8579 \begin_layout Standard
8580 The loop will continue until there are no more chunkref statements in the
8581 text, at which point we process the final part of the chunk.
8584 \begin_layout Standard
8585 \begin_inset listings
8589 \begin_layout Plain Layout
8594 \begin_layout Plain Layout
8596 chunk_line(active_chunk, chunk);
8604 \begin_layout Standard
8605 \begin_inset CommandInset label
8611 We add the newline character as a chunklet on it's own, to make it easier
8612 to detect new lines and thus manage indentation when processing the output.
8615 \begin_layout Standard
8616 \begin_inset listings
8620 \begin_layout Plain Layout
8622 chunk_line(active_chunk, "
8632 \begin_layout Standard
8633 We will also permit a chunk-part number to follow in square brackets, so
8635 \begin_inset Flex CharStyle:Code
8638 \begin_layout Plain Layout
8641 chunkref{chunk-name[1]}>
8646 will refer to the first part only.
8647 This can make it easy to include a C function prototype in a header file,
8648 if the first part of the chunk is just the function prototype without the
8649 trailing semi-colon.
8650 The header file would include the prototype with the trailing semi-colon,
8654 \begin_layout LyX-Code
8657 chunkref{chunk-name[1]}>;
8660 \begin_layout Standard
8661 This is handled in section
8662 \begin_inset CommandInset ref
8664 reference "sub:Chunk-parts"
8671 \begin_layout Standard
8672 We should perhaps introduce a notion of language specific chunk options;
8673 so that perhaps we could specify:
8676 \begin_layout LyX-Code
8679 chunkref{chunk-name[function-declaration]}>;
8682 \begin_layout Standard
8683 which applies a transform
8684 \begin_inset Flex CharStyle:Code
8687 \begin_layout Plain Layout
8688 function-declaration
8693 to the chunk --- which in this case would extract a function prototype
8695 \begin_inset Note Note
8698 \begin_layout Plain Layout
8707 \begin_layout Chapter
8711 \begin_layout Standard
8712 At the start, first we set the default options.
8719 \begin_layout Standard
8720 \begin_inset listings
8724 \begin_layout Plain Layout
8729 \begin_layout Plain Layout
8734 \begin_layout Plain Layout
8739 \begin_layout Plain Layout
8744 \begin_layout Plain Layout
8754 \begin_layout Standard
8755 Then we use getopt the standard way, and null out ARGV afterwards in the
8763 \begin_layout Standard
8764 \begin_inset listings
8768 \begin_layout Plain Layout
8770 Optind = 1 # skip ARGV[0]
8773 \begin_layout Plain Layout
8775 while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
8778 \begin_layout Plain Layout
8782 chunkref{handle-options}>
8785 \begin_layout Plain Layout
8790 \begin_layout Plain Layout
8792 for (i=1; i<Optind; i++) { ARGV[i]=""; }
8800 \begin_layout Standard
8801 This is how we handle our options:
8808 \begin_layout Standard
8809 \begin_inset listings
8813 \begin_layout Plain Layout
8815 if (Optopt == "R") root = Optarg;
8818 \begin_layout Plain Layout
8820 else if (Optopt == "r") root="";
8823 \begin_layout Plain Layout
8825 else if (Optopt == "L") linenos = 1;
8828 \begin_layout Plain Layout
8830 else if (Optopt == "d") debug = 1;
8833 \begin_layout Plain Layout
8835 else if (Optopt == "T") tabs = indent_string(Optarg+0);
8838 \begin_layout Plain Layout
8840 else if (Optopt == "h") help();
8843 \begin_layout Plain Layout
8845 else if (Optopt == "?") help();
8853 \begin_layout Standard
8854 We do all of this at the beginning of the program
8861 \begin_layout Standard
8862 \begin_inset listings
8866 \begin_layout Plain Layout
8871 \begin_layout Plain Layout
8875 chunkref{constants}>
8878 \begin_layout Plain Layout
8882 chunkref{mode-definitions}>
8885 \begin_layout Plain Layout
8889 chunkref{default-options}>
8892 \begin_layout Plain Layout
8896 \begin_layout Plain Layout
8900 chunkref{read-options}>
8903 \begin_layout Plain Layout
8913 \begin_layout Standard
8914 And have a simple help function
8921 \begin_layout Standard
8922 \begin_inset listings
8926 \begin_layout Plain Layout
8931 \begin_layout Plain Layout
8936 \begin_layout Plain Layout
8938 print " newfangle [-L] -R<rootname> [source.tex ...]"
8941 \begin_layout Plain Layout
8943 print " newfangle -r [source.tex ...]"
8946 \begin_layout Plain Layout
8948 print " If the filename, source.tex is not specified then stdin is used"
8951 \begin_layout Plain Layout
8956 \begin_layout Plain Layout
8958 print "-L causes the C statement: #line <lineno>
8965 \begin_layout Plain Layout
8967 print "-R causes the named root to be written to stdout"
8970 \begin_layout Plain Layout
8972 print "-r lists all roots in the file (even those used elsewhere)"
8975 \begin_layout Plain Layout
8980 \begin_layout Plain Layout
8990 \begin_layout Chapter
8991 Generating the output
8994 \begin_layout Standard
8995 We generate output by calling output_chunk, or listing the chunk names.
9002 \begin_layout Standard
9003 \begin_inset listings
9007 \begin_layout Plain Layout
9009 if (length(root)) output_chunk(root);
9012 \begin_layout Plain Layout
9014 else output_chunk_names();
9022 \begin_layout Standard
9023 We also have some other output debugging:
9030 \begin_layout Standard
9031 \begin_inset listings
9035 \begin_layout Plain Layout
9040 \begin_layout Plain Layout
9042 print "------ chunk names "
9045 \begin_layout Plain Layout
9047 output_chunk_names();
9050 \begin_layout Plain Layout
9052 print "====== chunks"
9055 \begin_layout Plain Layout
9060 \begin_layout Plain Layout
9062 print "++++++ debug"
9065 \begin_layout Plain Layout
9070 \begin_layout Plain Layout
9072 print a "=" chunks[a];
9075 \begin_layout Plain Layout
9080 \begin_layout Plain Layout
9090 \begin_layout Standard
9091 We do both of these at the end.
9093 \begin_inset Flex CharStyle:Code
9096 \begin_layout Plain Layout
9102 because each chunklet is not necessarily a complete line, and we already
9104 \begin_inset Flex CharStyle:Code
9107 \begin_layout Plain Layout
9113 to each input line in section
9114 \begin_inset CommandInset ref
9116 reference "sub:ORS-chunk-text"
9127 \begin_layout Standard
9128 \begin_inset listings
9132 \begin_layout Plain Layout
9137 \begin_layout Plain Layout
9141 chunkref{debug-output}>
9144 \begin_layout Plain Layout
9149 \begin_layout Plain Layout
9153 chunkref{generate-output}>
9156 \begin_layout Plain Layout
9166 \begin_layout Standard
9167 We write chunk names like this.
9168 If we seem to be running in notangle compatibility mode, then we enclose
9170 \begin_inset Flex CharStyle:Code
9173 \begin_layout Plain Layout
9179 the same way notangle does:
9183 output_chunk_names()
9186 \begin_layout Standard
9187 \begin_inset listings
9191 \begin_layout Plain Layout
9193 function output_chunk_names( c, prefix, suffix)
9196 \begin_layout Plain Layout
9201 \begin_layout Plain Layout
9203 if (notangle_mode) {
9206 \begin_layout Plain Layout
9211 \begin_layout Plain Layout
9216 \begin_layout Plain Layout
9221 \begin_layout Plain Layout
9223 for (c in chunk_names) {
9226 \begin_layout Plain Layout
9228 print prefix c suffix "
9233 \begin_layout Plain Layout
9238 \begin_layout Plain Layout
9248 \begin_layout Standard
9249 This function would write out all chunks
9256 \begin_layout Standard
9257 \begin_inset listings
9261 \begin_layout Plain Layout
9263 function output_chunks( a)
9266 \begin_layout Plain Layout
9271 \begin_layout Plain Layout
9273 for (a in chunk_names) {
9276 \begin_layout Plain Layout
9278 output_chunk(chunk_names[a]);
9281 \begin_layout Plain Layout
9286 \begin_layout Plain Layout
9291 \begin_layout Plain Layout
9295 \begin_layout Plain Layout
9297 function output_chunk(chunk) {
9300 \begin_layout Plain Layout
9305 \begin_layout Plain Layout
9307 lineno_needed = linenos;
9310 \begin_layout Plain Layout
9314 \begin_layout Plain Layout
9319 \begin_layout Plain Layout
9324 \begin_layout Plain Layout
9333 \begin_layout Section
9334 Assembling the chunks
9337 \begin_layout Standard
9338 \begin_inset Flex CharStyle:Code
9341 \begin_layout Plain Layout
9347 holds a string consisting of the names of all the chunks that resulted
9348 in this chunk being output.
9350 \begin_inset Note Note
9353 \begin_layout Plain Layout
9354 Make sure it includes the line numbers too...
9360 It should probably also contain the source line numbers at which each inclusion
9368 \begin_layout Standard
9369 We first initialize the mode tracker for this chunk.
9372 \begin_layout Standard
9373 \begin_inset listings
9377 \begin_layout Plain Layout
9379 function write_chunk(chunk_name) {
9382 \begin_layout Plain Layout
9386 chunkref{awk-delete-array}(context)>
9389 \begin_layout Plain Layout
9391 return write_chunk_r(chunk_name, context);
9394 \begin_layout Plain Layout
9405 write_chunk(),emph={chunk_path}
9408 \begin_layout Standard
9409 \begin_inset listings
9413 \begin_layout Plain Layout
9415 function write_chunk_r(chunk_name, context, indent, tail,
9418 \begin_layout Plain Layout
9423 \begin_layout Plain Layout
9425 chunk_path, chunk_args,
9428 \begin_layout Plain Layout
9433 \begin_layout Plain Layout
9435 chunk_params, part, max_part, part_line, frag, max_frag, text,
9438 \begin_layout Plain Layout
9440 chunklet, only_part, call_chunk_args)
9443 \begin_layout Plain Layout
9453 \begin_layout Subsection
9454 \begin_inset CommandInset label
9456 name "sub:Chunk-parts"
9463 \begin_layout Standard
9464 As mentioned in section
9465 \begin_inset CommandInset ref
9467 reference "sub:lstlistings-includes"
9471 , a chunk name may contain a part specifier in square brackets, limiting
9472 the parts that should be emitted.
9475 \begin_layout Standard
9476 \begin_inset listings
9480 \begin_layout Plain Layout
9482 if (match(chunk_name, "^(.*)
9490 ]$", chunk_name_parts)) {
9493 \begin_layout Plain Layout
9495 chunk_name = chunk_name_parts[1];
9498 \begin_layout Plain Layout
9500 only_part = chunk_name_parts[2];
9503 \begin_layout Plain Layout
9513 \begin_layout Standard
9514 We then create a mode tracker
9517 \begin_layout Standard
9518 \begin_inset listings
9522 \begin_layout Plain Layout
9526 chunkref{new-mode-tracker}(context, chunks[chunk_name, "language"], "")>
9534 \begin_layout Standard
9536 \begin_inset Flex CharStyle:Code
9539 \begin_layout Plain Layout
9545 the names of the parameters that this chunk accepts, whose values were
9546 (optionally) passed in
9547 \begin_inset Flex CharStyle:Code
9550 \begin_layout Plain Layout
9559 \begin_layout Standard
9560 \begin_inset listings
9564 \begin_layout Plain Layout
9566 split(chunks[chunk_name, "params"], chunk_params, " *; *");
9574 \begin_layout Standard
9575 To assemble a chunk, we write out each part.
9582 \begin_layout Standard
9583 \begin_inset listings
9587 \begin_layout Plain Layout
9589 if (! (chunk_name in chunk_names)) {
9592 \begin_layout Plain Layout
9594 error(sprintf(_"The root module <<%s>> was not defined.
9601 \begin_layout Plain Layout
9603 chunk_name, chunk_path));
9606 \begin_layout Plain Layout
9611 \begin_layout Plain Layout
9615 \begin_layout Plain Layout
9617 max_part = chunks[chunk_name, "part"];
9620 \begin_layout Plain Layout
9622 for(part = 1; part <= max_part; part++) {
9625 \begin_layout Plain Layout
9627 if (! only_part || part == only_part) {
9630 \begin_layout Plain Layout
9634 chunkref{write-part}>
9637 \begin_layout Plain Layout
9642 \begin_layout Plain Layout
9647 \begin_layout Plain Layout
9649 if (! finalize_mode_tracker(context)) {
9652 \begin_layout Plain Layout
9654 error(sprintf(_"Module %s did not close context properly.
9658 n", chunk_name, chunk_path));
9661 \begin_layout Plain Layout
9666 \begin_layout Plain Layout
9676 \begin_layout Standard
9677 A part can either be a chunklet of lines, or an include of another chunk.
9680 \begin_layout Standard
9681 Chunks may also have parameters, specified in LaTeX style with braces after
9682 the chunk name --- looking like this in the document:
9683 \begin_inset Flex CharStyle:Code
9686 \begin_layout Plain Layout
9687 chunkname{param1, param2}
9693 Arguments are passed in square brackets:
9694 \begin_inset Flex CharStyle:Code
9697 \begin_layout Plain Layout
9700 chunkref{chunkname}[arg1, arg2]
9708 \begin_layout Standard
9709 Before we process each part, we check that the source position hasn't changed
9710 unexpectedly, so that we can know if we need to output a new file-line
9718 \begin_layout Standard
9719 \begin_inset listings
9723 \begin_layout Plain Layout
9727 chunkref{check-source-jump}>
9730 \begin_layout Plain Layout
9734 \begin_layout Plain Layout
9736 chunklet = chunks[chunk_name, "part", part];
9739 \begin_layout Plain Layout
9741 if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
9744 \begin_layout Plain Layout
9748 chunkref{write-included-chunk}>
9751 \begin_layout Plain Layout
9753 } else if (chunklet SUBSEP "line" in chunks) {
9756 \begin_layout Plain Layout
9760 chunkref{write-chunklets}>
9763 \begin_layout Plain Layout
9768 \begin_layout Plain Layout
9770 # empty last chunklet
9773 \begin_layout Plain Layout
9783 \begin_layout Standard
9784 To write an included chunk, we must detect any optional chunk arguments
9786 Then we recurse calling
9787 \begin_inset Flex Chunkref
9790 \begin_layout Plain Layout
9800 write-included-chunk
9803 \begin_layout Standard
9804 \begin_inset listings
9808 \begin_layout Plain Layout
9810 if (match(chunklet, "^([^
9822 )$", chunklet_parts)) {
9825 \begin_layout Plain Layout
9827 chunklet = chunklet_parts[1];
9830 \begin_layout Plain Layout
9832 parse_chunk_args("", chunklet_parts[2], call_chunk_args, "(");
9835 \begin_layout Plain Layout
9837 for (c in call_chunk_args) {
9840 \begin_layout Plain Layout
9842 call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params,
9846 \begin_layout Plain Layout
9851 \begin_layout Plain Layout
9856 \begin_layout Plain Layout
9858 split("", call_chunk_args);
9861 \begin_layout Plain Layout
9866 \begin_layout Plain Layout
9868 write_chunk_r(chunklet, context,
9871 \begin_layout Plain Layout
9873 chunks[chunk_name, "part", part, "indent"] indent,
9876 \begin_layout Plain Layout
9878 chunks[chunk_name, "part", part, "tail"],
9881 \begin_layout Plain Layout
9888 \begin_layout Plain Layout
9898 \begin_layout Standard
9899 Before we output a chunklet of lines, we first emit the file and line number
9900 if we have one, and if it is safe to do so.
9904 \begin_layout Standard
9905 Chunklets are generally broken up by includes, so the start of a chunklet
9906 is a good place to do this.
9907 Then we output each line of the chunklet.
9910 \begin_layout Standard
9911 When it is not safe, such as in the middle of a multi-line macro definition,
9913 \begin_inset Flex CharStyle:Code
9916 \begin_layout Plain Layout
9922 is set to true, and in such a case we note that we want to emit the line
9923 statement when it is next safe.
9930 \begin_layout Standard
9931 \begin_inset listings
9935 \begin_layout Plain Layout
9937 max_frag = chunks[chunklet, "line"];
9940 \begin_layout Plain Layout
9942 for(frag = 1; frag <= max_frag; frag++) {
9945 \begin_layout Plain Layout
9949 chunkref{write-file-line}>
9957 \begin_layout Standard
9958 We then extract the chunklet text and expand any arguments.
9961 \begin_layout Standard
9962 \begin_inset listings
9966 \begin_layout Plain Layout
9970 \begin_layout Plain Layout
9972 text = chunks[chunklet, frag];
9975 \begin_layout Plain Layout
9980 \begin_layout Plain Layout
9985 \begin_layout Plain Layout
9987 text = expand_chunk_args(text, chunk_params, chunk_args);
9995 \begin_layout Standard
9996 If the text is a single newline (which we keep separate - see
9997 \begin_inset CommandInset ref
9999 reference "lone-newline"
10003 ) then we increment the line number.
10004 In the case where this is the last line of a chunk and it is not a top-level
10005 chunk we replace the newline with an empty string --- because the chunk
10006 that included this chunk will have the newline at the end of the line that
10007 included this chunk.
10010 \begin_layout Standard
10012 \begin_inset Flex CharStyle:Code
10015 \begin_layout Plain Layout
10021 that we have started a new line, so that indentation can be managed with
10022 the following piece of text.
10025 \begin_layout Standard
10026 \begin_inset listings
10030 \begin_layout Plain Layout
10034 \begin_layout Plain Layout
10041 \begin_layout Plain Layout
10046 \begin_layout Plain Layout
10048 if (part == max_part && frag == max_frag && length(chunk_path)) {
10051 \begin_layout Plain Layout
10056 \begin_layout Plain Layout
10061 \begin_layout Plain Layout
10066 \begin_layout Plain Layout
10071 \begin_layout Plain Layout
10081 \begin_layout Standard
10082 If this text does not represent a newline, but we see that we are the first
10083 piece of text on a newline, then we prefix our text with the current indent.
10085 \begin_inset Flex CharStyle:Code
10088 \begin_layout Plain Layout
10094 is a global output-state variable, but the
10095 \begin_inset Flex CharStyle:Code
10098 \begin_layout Plain Layout
10108 \begin_layout Standard
10109 \begin_inset listings
10113 \begin_layout Plain Layout
10115 } else if (length(text) || length(tail)) {
10118 \begin_layout Plain Layout
10120 if (newline) text = indent text;
10123 \begin_layout Plain Layout
10128 \begin_layout Plain Layout
10133 \begin_layout Plain Layout
10142 \begin_layout Standard
10143 Tail will soon no longer be relevant once mode-detection is in place.
10146 \begin_layout Standard
10147 \begin_inset listings
10151 \begin_layout Plain Layout
10156 \begin_layout Plain Layout
10158 mode_tracker(context, text);
10161 \begin_layout Plain Layout
10171 \begin_layout Standard
10172 If a line ends in a backslash --- suggesting continuation --- then we supress
10173 outputting file-line as it would probably break the continued lines.
10177 \begin_layout Standard
10178 \begin_inset listings
10182 \begin_layout Plain Layout
10187 \begin_layout Plain Layout
10189 lineno_suppressed = substr(lastline, length(lastline)) == "
10196 \begin_layout Plain Layout
10201 \begin_layout Plain Layout
10211 \begin_layout Standard
10212 Of course there is no point in actually outputting the source filename and
10213 line number (file-line) if they don't say anything new! We only need to
10214 emit them if they aren't what is expected, or if we we not able to emit
10215 one when they had changed.
10218 \begin_layout Chunk
10222 \begin_layout Standard
10223 \begin_inset listings
10227 \begin_layout Plain Layout
10229 if (newline && lineno_needed && ! lineno_suppressed) {
10232 \begin_layout Plain Layout
10234 filename = a_filename;
10237 \begin_layout Plain Layout
10242 \begin_layout Plain Layout
10244 print "#line " lineno "
10253 \begin_layout Plain Layout
10258 \begin_layout Plain Layout
10268 \begin_layout Standard
10269 We check if a new file-line is needed by checking if the source line matches
10270 what we (or a compiler) would expect.
10274 \begin_layout Chunk
10278 \begin_layout Standard
10279 \begin_inset listings
10283 \begin_layout Plain Layout
10285 if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in
10289 \begin_layout Plain Layout
10291 a_filename = chunks[chunk_name, "part", part, "FILENAME"];
10294 \begin_layout Plain Layout
10296 a_lineno = chunks[chunk_name, "part", part, "LINENO"];
10299 \begin_layout Plain Layout
10301 if (a_filename != filename || a_lineno != lineno) {
10304 \begin_layout Plain Layout
10309 \begin_layout Plain Layout
10314 \begin_layout Plain Layout
10324 \begin_layout Chapter
10328 \begin_layout Standard
10329 Awk has pretty limited data structures, so we will use two main hashes.
10330 Uninterrupted sequences of a chunk will be stored in
10331 \begin_inset Flex CharStyle:Code
10334 \begin_layout Plain Layout
10340 and the chunklets used in a chunk will be stored in
10341 \begin_inset Flex CharStyle:Code
10344 \begin_layout Plain Layout
10353 \begin_layout Chunk
10357 \begin_layout Standard
10358 \begin_inset listings
10362 \begin_layout Plain Layout
10367 \begin_layout Plain Layout
10377 \begin_layout Standard
10379 \begin_inset Flex CharStyle:Code
10382 \begin_layout Plain Layout
10388 mentioned are not chunk parameters for parameterized chunks, as mentioned
10390 \begin_inset CommandInset ref
10392 reference "cha:Chunk Arguments"
10396 , but the lstlistings style parameters used in the
10397 \begin_inset Flex CharStyle:Code
10400 \begin_layout Plain Layout
10412 \begin_layout Plain Layout
10414 \begin_inset Flex CharStyle:Code
10417 \begin_layout Plain Layout
10423 parameter is used to hold the parameters for parameterized chunks
10431 \begin_layout Chunk
10432 chunk-storage-functions
10435 \begin_layout Standard
10436 \begin_inset listings
10440 \begin_layout Plain Layout
10442 function new_chunk(chunk_name, params,
10445 \begin_layout Plain Layout
10450 \begin_layout Plain Layout
10455 \begin_layout Plain Layout
10460 \begin_layout Plain Layout
10462 # HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
10465 \begin_layout Plain Layout
10475 )$", "", chunk_name);
10478 \begin_layout Plain Layout
10480 if (! (chunk_name in chunk_names)) {
10483 \begin_layout Plain Layout
10485 if (debug) print "New chunk " chunk_name;
10488 \begin_layout Plain Layout
10490 chunk_names[chunk_name];
10493 \begin_layout Plain Layout
10495 for (p in params) {
10498 \begin_layout Plain Layout
10500 chunks[chunk_name, p] = params[p];
10503 \begin_layout Plain Layout
10508 \begin_layout Plain Layout
10510 if ("append" in params) {
10513 \begin_layout Plain Layout
10515 append=params["append"];
10518 \begin_layout Plain Layout
10520 if (! (append in chunk_names)) {
10523 \begin_layout Plain Layout
10525 warning("Chunk " chunk_name " is appended to chunk " append " which
10526 is not defined yet");
10529 \begin_layout Plain Layout
10534 \begin_layout Plain Layout
10539 \begin_layout Plain Layout
10541 chunk_include(append, chunk_name);
10544 \begin_layout Plain Layout
10546 chunk_line(append, ORS);
10549 \begin_layout Plain Layout
10554 \begin_layout Plain Layout
10559 \begin_layout Plain Layout
10561 active_chunk = chunk_name;
10564 \begin_layout Plain Layout
10566 prime_chunk(chunk_name);
10569 \begin_layout Plain Layout
10579 \begin_layout Standard
10580 \begin_inset listings
10584 \begin_layout Plain Layout
10588 \begin_layout Plain Layout
10590 function prime_chunk(chunk_name)
10593 \begin_layout Plain Layout
10598 \begin_layout Plain Layout
10600 chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] =
10605 \begin_layout Plain Layout
10607 chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"]
10611 \begin_layout Plain Layout
10613 chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
10616 \begin_layout Plain Layout
10618 chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR
10622 \begin_layout Plain Layout
10627 \begin_layout Plain Layout
10631 \begin_layout Plain Layout
10633 function chunk_line(chunk_name, line){
10636 \begin_layout Plain Layout
10638 chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
10641 \begin_layout Plain Layout
10643 ++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
10647 \begin_layout Plain Layout
10652 \begin_layout Plain Layout
10661 \begin_layout Standard
10662 Chunk include represents a
10666 statement, and stores the requirement to include another chunk.
10667 The parameter indent represents the quanity of literal text characters
10672 statement and therefore by how much additional lines of the included chunk
10673 should be indented.
10676 \begin_layout Standard
10677 \begin_inset listings
10681 \begin_layout Plain Layout
10683 function chunk_include(chunk_name, chunk_ref, indent, tail)
10686 \begin_layout Plain Layout
10691 \begin_layout Plain Layout
10693 chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
10696 \begin_layout Plain Layout
10698 chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_ch
10702 \begin_layout Plain Layout
10704 chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_str
10708 \begin_layout Plain Layout
10710 chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
10713 \begin_layout Plain Layout
10715 prime_chunk(chunk_name);
10718 \begin_layout Plain Layout
10723 \begin_layout Plain Layout
10732 \begin_layout Standard
10733 The indent is calculated by indent_string, which may in future convert some
10734 spaces into tab characters.
10735 This function works by generating a printf padded format string, like
10736 \begin_inset Flex CharStyle:Code
10739 \begin_layout Plain Layout
10745 for an indent of 22, and then printing an empty string using that format.
10748 \begin_layout Standard
10749 \begin_inset listings
10753 \begin_layout Plain Layout
10755 function indent_string(indent) {
10758 \begin_layout Plain Layout
10760 return sprintf("%" indent "s", "");
10763 \begin_layout Plain Layout
10773 \begin_layout Chapter
10774 \begin_inset CommandInset label
10783 \begin_layout Standard
10784 I use Arnold Robbins public domain getopt (1993 revision).
10785 This is probably the same one that is covered in chapter 12 of
10786 \begin_inset Quotes eld
10789 Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk
10790 \begin_inset Quotes erd
10793 but as that is licensed under the GNU Free Documentation License, Version
10794 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompany
10795 ing explanations), so I do my best to explain how it works here.
10798 \begin_layout Standard
10799 The getopt.awk header is:
10802 \begin_layout Chunk
10803 getopt.awk-header,language=awk,morestring=[b]{/},morekeywords=else
10806 \begin_layout Standard
10807 \begin_inset listings
10811 \begin_layout Plain Layout
10813 # getopt.awk --- do C library getopt(3) function in awk
10816 \begin_layout Plain Layout
10821 \begin_layout Plain Layout
10823 # Arnold Robbins, arnold@skeeve.com, Public Domain
10826 \begin_layout Plain Layout
10831 \begin_layout Plain Layout
10833 # Initial version: March, 1991
10836 \begin_layout Plain Layout
10838 # Revised: May, 1993
10846 \begin_layout Standard
10847 The provided explanation is:
10850 \begin_layout Chunk
10854 \begin_layout Standard
10855 \begin_inset listings
10859 \begin_layout Plain Layout
10861 # External variables:
10864 \begin_layout Plain Layout
10866 # Optind -- index in ARGV of first nonoption argument
10869 \begin_layout Plain Layout
10871 # Optarg -- string value of argument to current option
10874 \begin_layout Plain Layout
10876 # Opterr -- if nonzero, print our own diagnostic
10879 \begin_layout Plain Layout
10881 # Optopt -- current option letter
10884 \begin_layout Plain Layout
10888 \begin_layout Plain Layout
10893 \begin_layout Plain Layout
10895 # -1 at end of options
10898 \begin_layout Plain Layout
10900 # ? for unrecognized option
10903 \begin_layout Plain Layout
10905 # <c> a character representing the current option
10908 \begin_layout Plain Layout
10912 \begin_layout Plain Layout
10917 \begin_layout Plain Layout
10919 # _opti -- index in multi-flag option, e.g., -abc
10927 \begin_layout Standard
10928 The function follows.
10929 The final two parameters,
10930 \begin_inset Flex CharStyle:Code
10933 \begin_layout Plain Layout
10940 \begin_inset Flex CharStyle:Code
10943 \begin_layout Plain Layout
10949 are local variables and not parameters --- as indicated by the multiple
10950 spaces preceding them.
10951 Awk doesn't care, the multiple spaces are a convention to help us humans.
10954 \begin_layout Chunk
10955 getopt.awk-getopt()
10958 \begin_layout Standard
10959 \begin_inset listings
10963 \begin_layout Plain Layout
10965 function getopt(argc, argv, options, thisopt, i)
10968 \begin_layout Plain Layout
10973 \begin_layout Plain Layout
10975 if (length(options) == 0) # no options given
10978 \begin_layout Plain Layout
10983 \begin_layout Plain Layout
10985 if (argv[Optind] == "--") { # all done
10988 \begin_layout Plain Layout
10993 \begin_layout Plain Layout
10998 \begin_layout Plain Layout
11003 \begin_layout Plain Layout
11005 } else if (argv[Optind] !~ /^-[^:
11020 \begin_layout Plain Layout
11025 \begin_layout Plain Layout
11030 \begin_layout Plain Layout
11035 \begin_layout Plain Layout
11040 \begin_layout Plain Layout
11045 \begin_layout Plain Layout
11047 thisopt = substr(argv[Optind], _opti, 1)
11050 \begin_layout Plain Layout
11055 \begin_layout Plain Layout
11057 i = index(options, thisopt)
11060 \begin_layout Plain Layout
11065 \begin_layout Plain Layout
11070 \begin_layout Plain Layout
11072 printf("%c -- invalid option
11077 \begin_layout Plain Layout
11079 thisopt) > "/dev/stderr"
11082 \begin_layout Plain Layout
11084 if (_opti >= length(argv[Optind])) {
11087 \begin_layout Plain Layout
11092 \begin_layout Plain Layout
11097 \begin_layout Plain Layout
11102 \begin_layout Plain Layout
11107 \begin_layout Plain Layout
11112 \begin_layout Plain Layout
11122 \begin_layout Standard
11123 At this point, the option has been found and we need to know if it takes
11127 \begin_layout Standard
11128 \begin_inset listings
11132 \begin_layout Plain Layout
11134 if (substr(options, i + 1, 1) == ":") {
11137 \begin_layout Plain Layout
11139 # get option argument
11142 \begin_layout Plain Layout
11144 if (length(substr(argv[Optind], _opti + 1)) > 0)
11147 \begin_layout Plain Layout
11149 Optarg = substr(argv[Optind], _opti + 1)
11152 \begin_layout Plain Layout
11157 \begin_layout Plain Layout
11159 Optarg = argv[++Optind]
11162 \begin_layout Plain Layout
11167 \begin_layout Plain Layout
11172 \begin_layout Plain Layout
11177 \begin_layout Plain Layout
11179 if (_opti == 0 || _opti >= length(argv[Optind])) {
11182 \begin_layout Plain Layout
11187 \begin_layout Plain Layout
11192 \begin_layout Plain Layout
11197 \begin_layout Plain Layout
11202 \begin_layout Plain Layout
11207 \begin_layout Plain Layout
11214 A test program is built in, too
11217 \begin_layout Chunk
11221 \begin_layout Standard
11222 \begin_inset listings
11226 \begin_layout Plain Layout
11231 \begin_layout Plain Layout
11233 Opterr = 1 # default is to diagnose
11236 \begin_layout Plain Layout
11238 Optind = 1 # skip ARGV[0]
11241 \begin_layout Plain Layout
11246 \begin_layout Plain Layout
11248 if (_getopt_test) {
11251 \begin_layout Plain Layout
11253 while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
11256 \begin_layout Plain Layout
11258 printf("c = <%c>, optarg = <%s>
11263 \begin_layout Plain Layout
11268 \begin_layout Plain Layout
11270 printf("non-option arguments:
11275 \begin_layout Plain Layout
11277 for (; Optind < ARGC; Optind++)
11280 \begin_layout Plain Layout
11289 \begin_layout Plain Layout
11291 Optind, ARGV[Optind])
11294 \begin_layout Plain Layout
11299 \begin_layout Plain Layout
11309 \begin_layout Standard
11310 The entire getopt.awk is made out of these chunks in order
11313 \begin_layout Chunk
11317 \begin_layout Standard
11318 \begin_inset listings
11322 \begin_layout Plain Layout
11326 chunkref{getopt.awk-header}>
11329 \begin_layout Plain Layout
11333 \begin_layout Plain Layout
11337 chunkref{getopt.awk-notes}>
11340 \begin_layout Plain Layout
11344 chunkref{getopt.awk-getopt()}>
11347 \begin_layout Plain Layout
11351 chunkref{getopt.awk-begin}>
11359 \begin_layout Standard
11360 Although we only want the header and function:
11363 \begin_layout Chunk
11367 \begin_layout Standard
11368 \begin_inset listings
11372 \begin_layout Plain Layout
11374 # try: locate getopt.awk for the full original file
11377 \begin_layout Plain Layout
11379 # as part of your standard awk installation
11382 \begin_layout Plain Layout
11386 chunkref{getopt.awk-header}>
11389 \begin_layout Plain Layout
11393 \begin_layout Plain Layout
11397 chunkref{getopt.awk-getopt()}>
11405 \begin_layout Chapter
11406 Newfangle LaTeX source code
11409 \begin_layout Section
11413 \begin_layout Standard
11414 Here we define a Lyx .module file that makes it convenient to use LyX for
11415 writing such literate programs.
11418 \begin_layout Standard
11420 \begin_inset Flex CharStyle:Code
11423 \begin_layout Plain Layout
11429 can be installed in your personal
11430 \begin_inset Flex CharStyle:Code
11433 \begin_layout Plain Layout
11434 .lyx/layouts folder
11440 You will need to Tools Reconfigure so that LyX notices it.
11441 It adds a new format Chunk, which should precede every listing and contain
11446 \begin_layout Chunk
11447 ./newfangle.module,language=
11450 \begin_layout Standard
11451 \begin_inset listings
11455 \begin_layout Plain Layout
11459 DeclareLyXModule{Newfangle Literate Listings}
11462 \begin_layout Plain Layout
11467 \begin_layout Plain Layout
11469 # Newfangle literate listings allow one to write
11472 \begin_layout Plain Layout
11474 # literate programs after the fashion of noweb, but without having
11477 \begin_layout Plain Layout
11479 # to use noweave to generate the documentation.
11480 Instead the listings
11483 \begin_layout Plain Layout
11485 # package is extended in conjunction with the noweb package to implement
11488 \begin_layout Plain Layout
11490 # to code formating directly as latex.
11493 \begin_layout Plain Layout
11495 # The newfangle awk script
11498 \begin_layout Plain Layout
11503 \begin_layout Plain Layout
11507 \begin_layout Plain Layout
11512 \begin_layout Plain Layout
11516 \begin_layout Plain Layout
11521 \begin_layout Plain Layout
11525 chunkref{./newfangle.sty}>
11528 \begin_layout Plain Layout
11533 \begin_layout Plain Layout
11537 \begin_layout Plain Layout
11541 chunkref{chunkstyle}>
11544 \begin_layout Plain Layout
11548 \begin_layout Plain Layout
11552 chunkref{chunkref}>
11560 \begin_layout Subsection
11564 \begin_layout Standard
11569 style is to make it easier for LyX users to provide the name to
11570 \begin_inset Flex CharStyle:Code
11573 \begin_layout Plain Layout
11582 Normally this requires right-clicking on the listing, choosing settings,
11583 advanced, and then typing
11584 \begin_inset Flex CharStyle:Code
11587 \begin_layout Plain Layout
11594 This has the further disadvantage that the name (and other options) are
11595 not generally visible during document editing.
11598 \begin_layout Standard
11599 The chunk style is defined as a LaTeX command, so that all text on the same
11600 line is passed to the LaTeX command
11601 \begin_inset Flex CharStyle:Code
11604 \begin_layout Plain Layout
11611 This makes it easy to parse using
11612 \begin_inset Flex CharStyle:Code
11615 \begin_layout Plain Layout
11621 , and easy to pass these options on to the listings package.
11622 The first word in a chunk section should be the chunk name, and will have
11624 \begin_inset Flex CharStyle:Code
11627 \begin_layout Plain Layout
11634 Any other words are accepted arguments to
11635 \begin_inset Flex CharStyle:Code
11638 \begin_layout Plain Layout
11649 \begin_layout Standard
11650 We set PassThru to 1 because the user is actually entering raw latex.
11653 \begin_layout Chunk
11657 \begin_layout Standard
11658 \begin_inset listings
11662 \begin_layout Plain Layout
11667 \begin_layout Plain Layout
11672 \begin_layout Plain Layout
11677 \begin_layout Plain Layout
11679 Margin First_Dynamic
11682 \begin_layout Plain Layout
11684 LeftMargin Chunk:xxx
11687 \begin_layout Plain Layout
11692 \begin_layout Plain Layout
11697 \begin_layout Plain Layout
11699 LabelString "Chunk:"
11702 \begin_layout Plain Layout
11707 \begin_layout Plain Layout
11712 \begin_layout Plain Layout
11721 \begin_layout Standard
11722 To make the label very visible we choose a larger font coloured red.
11725 \begin_layout Standard
11726 \begin_inset listings
11730 \begin_layout Plain Layout
11735 \begin_layout Plain Layout
11740 \begin_layout Plain Layout
11745 \begin_layout Plain Layout
11750 \begin_layout Plain Layout
11755 \begin_layout Plain Layout
11760 \begin_layout Plain Layout
11765 \begin_layout Plain Layout
11775 \begin_layout Subsection
11779 \begin_layout Standard
11780 We also define the Chunkref style which can be used to express cross references
11784 \begin_layout Chunk
11788 \begin_layout Standard
11789 \begin_inset listings
11793 \begin_layout Plain Layout
11795 InsetLayout Chunkref
11798 \begin_layout Plain Layout
11803 \begin_layout Plain Layout
11808 \begin_layout Plain Layout
11813 \begin_layout Plain Layout
11818 \begin_layout Plain Layout
11823 \begin_layout Plain Layout
11828 \begin_layout Plain Layout
11833 \begin_layout Plain Layout
11838 \begin_layout Plain Layout
11848 \begin_layout Section
11849 \begin_inset CommandInset label
11851 name "sec:Latex-Macros"
11858 \begin_layout Standard
11872 As noweb defines it's own
11873 \begin_inset Flex CharStyle:Code
11876 \begin_layout Plain Layout
11884 environment, we re-define the one that LyX logical markup module expects
11888 \begin_layout Chunk
11889 ./newfangle.sty,language=tex,basicstyle=
11894 \begin_layout Standard
11895 \begin_inset listings
11899 \begin_layout Plain Layout
11903 usepackage{listings}%
11906 \begin_layout Plain Layout
11913 \begin_layout Plain Layout
11920 \begin_layout Plain Layout
11936 \begin_layout Standard
11938 \begin_inset Flex CharStyle:Code
11941 \begin_layout Plain Layout
11948 \begin_inset Flex CharStyle:Code
11951 \begin_layout Plain Layout
11959 which will need renaming to
11960 \begin_inset Flex CharStyle:Code
11963 \begin_layout Plain Layout
11971 when I can do this without clashing with
11972 \begin_inset Flex CharStyle:Code
11975 \begin_layout Plain Layout
11986 \begin_layout Standard
11987 \begin_inset listings
11991 \begin_layout Plain Layout
11995 lstnewenvironment{Chunk}{
12007 \begin_layout Standard
12008 We also define a suitable
12009 \begin_inset Flex CharStyle:Code
12012 \begin_layout Plain Layout
12020 of parameters that suit the literate programming style after the fashion
12028 \begin_layout Standard
12029 \begin_inset listings
12033 \begin_layout Plain Layout
12037 lstset{numbers=left, stepnumber=5, numbersep=5pt,
12040 \begin_layout Plain Layout
12042 breaklines=false,basicstyle=
12047 \begin_layout Plain Layout
12059 \begin_layout Standard
12060 We also define a notangle-like mechanism for
12064 to LaTeX from the listing, and by which we can refer to other listings.
12066 \begin_inset Flex CharStyle:Code
12069 \begin_layout Plain Layout
12070 =<\SpecialChar \ldots{}
12076 sequence to contain LaTeX code, and include another like this chunk:
12077 \begin_inset Flex CharStyle:Code
12080 \begin_layout Plain Layout
12083 chunkref{chunkname}>
12090 \begin_inset Flex CharStyle:Code
12093 \begin_layout Plain Layout
12094 =<\SpecialChar \ldots{}
12100 is already defined to contain LaTeX code for this document --- this is
12105 document after all --- the code fragment below effectively contains the
12107 \begin_inset Flex CharStyle:Code
12110 \begin_layout Plain Layout
12117 To avoid problems with document generation, I had to declare an lstlistings
12119 \begin_inset Flex CharStyle:Code
12122 \begin_layout Plain Layout
12128 for this listing only; which in LyX was done by right-clicking the listings
12130 \begin_inset Flex CharStyle:Code
12133 \begin_layout Plain Layout
12139 \SpecialChar \menuseparator
12141 \begin_inset Flex CharStyle:Code
12144 \begin_layout Plain Layout
12153 \begin_layout Standard
12154 \begin_inset Note Note
12157 \begin_layout Plain Layout
12158 =< isn't enjoyed literally here, in a listing when the escape sequence is
12159 already defined as shown...
12160 we need to somehow escape this representation...
12168 \begin_layout Standard
12169 \begin_inset listings
12170 lstparams "escapeinside={}"
12174 \begin_layout Plain Layout
12178 lstset{escapeinside={=<}{>}}%
12186 \begin_layout Standard
12187 Although our macros will contain the @ symbol, they will be included in
12189 \begin_inset Flex CharStyle:Code
12192 \begin_layout Plain Layout
12200 section by LyX; however we keep the commented out
12201 \begin_inset Flex CharStyle:Code
12204 \begin_layout Plain Layout
12213 The listings package likes to centre the titles, but noweb titles are specially
12214 formatted and must be left aligned.
12215 The simplest way to do this turned out to be by removing the definition
12217 \begin_inset Flex CharStyle:Code
12220 \begin_layout Plain Layout
12229 This may interact badly if other listings want a regular title or caption.
12230 We remember the old maketitle in case we need it.
12233 \begin_layout Standard
12234 \begin_inset listings
12238 \begin_layout Plain Layout
12245 \begin_layout Plain Layout
12247 %somehow re-defining maketitle gives us a left-aligned title
12250 \begin_layout Plain Layout
12252 %which is extactly what our specially formatted title needs!
12255 \begin_layout Plain Layout
12263 newfangle@lst@maketitle
12268 \begin_layout Plain Layout
12284 \begin_layout Subsection
12285 \begin_inset CommandInset label
12287 name "sub:The-chunk-command"
12294 \begin_layout Standard
12295 Our chunk command accepts one argument, and calls
12296 \begin_inset Flex CharStyle:Code
12299 \begin_layout Plain Layout
12309 \begin_inset Flex CharStyle:Code
12312 \begin_layout Plain Layout
12320 will note the name, this is erased when the next
12321 \begin_inset Flex CharStyle:Code
12324 \begin_layout Plain Layout
12332 starts, so we make a note of this in
12333 \begin_inset Flex CharStyle:Code
12336 \begin_layout Plain Layout
12344 and restore in in lstlistings Init hook.
12347 \begin_layout Standard
12348 \begin_inset listings
12352 \begin_layout Plain Layout
12361 \begin_layout Plain Layout
12367 newfanglecaption},name=#1}%
12370 \begin_layout Plain Layout
12383 \begin_layout Plain Layout
12388 \begin_layout Plain Layout
12404 \begin_layout Subsubsection
12408 \begin_layout Standard
12409 Newfangle permits parameterized chunks, and requires the paramters to be
12410 specified as listings options.
12411 The newfangle script uses this, and although we don't do anything with
12412 these in the LaTeX code right now, we need to stop the listings package
12416 \begin_layout Standard
12417 \begin_inset listings
12421 \begin_layout Plain Layout
12431 newfangle@chunk@params{#1}}%
12439 \begin_layout Standard
12440 As it is common to define a chunk which then needs appending to another
12441 chunk, and annoying to have to declare a single line chunk to manage the
12442 include, we support an
12443 \begin_inset Flex CharStyle:Code
12446 \begin_layout Plain Layout
12456 \begin_layout Standard
12457 \begin_inset listings
12461 \begin_layout Plain Layout
12471 newfangle@chunk@append{#1}}%
12479 \begin_layout Subsection
12480 The noweb styled caption
12483 \begin_layout Standard
12484 We define a public macro
12485 \begin_inset Flex CharStyle:Code
12488 \begin_layout Plain Layout
12496 which can be set as a regular title.
12498 \begin_inset Flex CharStyle:Code
12501 \begin_layout Plain Layout
12510 \begin_inset Flex CharStyle:Code
12513 \begin_layout Plain Layout
12521 at the appriate time when the caption is emitted.
12524 \begin_layout Standard
12525 \begin_inset listings
12529 \begin_layout Plain Layout
12539 newfangle@caption}%
12547 \begin_layout Standard
12548 \begin_inset Float figure
12554 \begin_layout Plain Layout
12555 \begin_inset Box Boxed
12564 height_special "totalheight"
12567 \begin_layout Plain Layout
12569 \begin_inset space \qquad{}
12577 \begin_inset Formula $\equiv+$
12581 \begin_inset space \qquad{}
12585 \begin_inset space \qquad{}
12589 \begin_inset space \qquad{}
12593 \begin_inset Formula $\triangleleft$
12597 \begin_inset space \quad{}
12601 \begin_inset Formula $\triangleright$
12607 \begin_layout Plain Layout
12610 In this example, the current chunk is 22c, and therefore the third chunk
12614 \begin_layout Plain Layout
12625 \begin_layout Plain Layout
12628 The first chunk with this name (19b) occurs as the second chunk on page
12632 \begin_layout Plain Layout
12635 The previous chunk (22d) with the same name is the second chunk on page
12639 \begin_layout Plain Layout
12642 The next chunk (24d) is the fourth chunk on page 24.
12645 \begin_layout Plain Layout
12646 \begin_inset Caption
12648 \begin_layout Plain Layout
12664 The general noweb output format compactly identifies the current chunk,
12665 and references to the first chunk, and the previous and next chunks that
12666 have the same name.
12670 \begin_layout Standard
12671 This means that we need to keep a counter for each chunk-name, that we use
12672 to count chunks of the same name.
12676 \begin_layout Subsection
12680 \begin_layout Standard
12681 It would be natural to have a counter for each chunk name, but TeX would
12682 soon run out of counters
12686 \begin_layout Plain Layout
12687 \SpecialChar \ldots{}
12692 run out of counters and so I had to re-write the LaTeX macros to share
12693 a counter as described here
12698 , so we have one counter which we save at the end of a chunk and restore
12699 at the beginning of a chunk.
12702 \begin_layout Standard
12703 \begin_inset listings
12707 \begin_layout Plain Layout
12711 newcounter{newfangle@chunkcounter}%
12719 \begin_layout Standard
12720 We construct the name of this variable to store the counter to be the text
12722 \begin_inset Flex CharStyle:Code
12725 \begin_layout Plain Layout
12731 prefixed onto the chunks own name, and store it in
12732 \begin_inset Flex CharStyle:Code
12735 \begin_layout Plain Layout
12747 \begin_layout Standard
12748 We save the counter like this:
12751 \begin_layout Chunk
12755 \begin_layout Standard
12756 \begin_inset listings
12760 \begin_layout Plain Layout
12776 arabic{newfangle@chunkcounter}}%
12784 \begin_layout Standard
12785 and restore the counter like this:
12788 \begin_layout Chunk
12792 \begin_layout Standard
12793 \begin_inset listings
12797 \begin_layout Plain Layout
12801 setcounter{newfangle@chunkcounter}{
12815 \begin_layout Chunk
12819 \begin_layout Standard
12820 If there does not already exist a variable whose name is stored in
12821 \begin_inset Flex CharStyle:Code
12824 \begin_layout Plain Layout
12832 , then we know we are the first chunk with this name, and then define a
12837 \begin_layout Standard
12838 Although chunks of the same name share a common counter, they must still
12840 We use is the internal name of the listing, suffixed by the counter value.
12841 So the first chunk might be
12842 \begin_inset Flex CharStyle:Code
12845 \begin_layout Plain Layout
12851 and the second chunk be
12852 \begin_inset Flex CharStyle:Code
12855 \begin_layout Plain Layout
12864 \begin_layout Standard
12865 We also calculate the name of the previous chunk if we can (before we increment
12866 the chunk counter).
12867 If this is the first chunk of that name, then
12868 \begin_inset Flex CharStyle:Code
12871 \begin_layout Plain Layout
12880 \begin_inset Flex CharStyle:Code
12883 \begin_layout Plain Layout
12891 which the noweb package will interpret as not existing.
12894 \begin_layout Standard
12895 \begin_inset listings
12899 \begin_layout Plain Layout
12905 newfangle@caption{%
12908 \begin_layout Plain Layout
12914 chunkcount{lst-chunk-
12919 \begin_layout Plain Layout
12928 \begin_layout Plain Layout
12943 \begin_layout Plain Layout
12947 setcounter{newfangle@chunkcounter}{
12956 \begin_layout Plain Layout
12967 \begin_layout Plain Layout
12972 \begin_layout Plain Layout
12976 setcounter{newfangle@chunkcounter}{
12985 \begin_layout Plain Layout
12995 arabic{newfangle@chunkcounter}}%
12998 \begin_layout Plain Layout
13008 \begin_layout Standard
13009 After incrementing the chunk counter, we then define the name of this chunk,
13010 as well as the name of the first chunk.
13013 \begin_layout Standard
13014 \begin_inset listings
13018 \begin_layout Plain Layout
13022 addtocounter{newfangle@chunkcounter}{1}%
13025 \begin_layout Plain Layout
13041 arabic{newfangle@chunkcounter}}%
13044 \begin_layout Plain Layout
13054 arabic{newfangle@chunkcounter}}%
13057 \begin_layout Plain Layout
13073 \begin_layout Standard
13074 We now need to calculate the name of the next chunk.
13075 We do this by temporarily skipping the counter on by one; however there
13076 may not actually be another chunk with this name! We detect this by also
13077 defining a label for each chunk based on the chunkname.
13078 If there is a next chunkname then it will define a label with that name.
13079 As labels are persistent, we can at least tell the second time LaTeX is
13081 If we don't find such a defined label then we define
13082 \begin_inset Flex CharStyle:Code
13085 \begin_layout Plain Layout
13094 \begin_inset Flex CharStyle:Code
13097 \begin_layout Plain Layout
13108 \begin_layout Standard
13109 \begin_inset listings
13113 \begin_layout Plain Layout
13117 addtocounter{newfangle@chunkcounter}{1}%
13120 \begin_layout Plain Layout
13130 arabic{newfangle@chunkcounter}}%
13133 \begin_layout Plain Layout
13137 @ifundefined{r@label-
13153 \begin_layout Standard
13154 The noweb package requires that we define a
13155 \begin_inset Flex CharStyle:Code
13158 \begin_layout Plain Layout
13166 for every chunk, with a unique name, which is then used to print out it's
13170 \begin_layout Standard
13171 We also define a regular label for this chunk, as was mentioned above when
13173 \begin_inset Flex CharStyle:Code
13176 \begin_layout Plain Layout
13185 This requires LaTeX to be run at least twice after new chunk sections are
13186 added --- but noweb requried that anyway.
13189 \begin_layout Standard
13190 \begin_inset listings
13194 \begin_layout Plain Layout
13203 \begin_layout Plain Layout
13205 % define this label for every chunk instance, so we
13208 \begin_layout Plain Layout
13210 % can tell when we are the last chunk of this name
13213 \begin_layout Plain Layout
13227 \begin_layout Standard
13228 We also try and add the chunk to the list of listings, but I'm afraid we
13229 don't do very well.
13230 We want each chunk name listing once, with all of it's references.
13233 \begin_layout Standard
13234 \begin_inset listings
13238 \begin_layout Plain Layout
13242 addcontentsline{lol}{lstlisting}{
13258 \begin_layout Standard
13259 We then call the noweb output macros in the same way that noweave generates
13260 them, except that we don't need to call
13261 \begin_inset Flex CharStyle:Code
13264 \begin_layout Plain Layout
13267 nwstartdeflinemarkup
13273 \begin_inset Flex CharStyle:Code
13276 \begin_layout Plain Layout
13284 -- and if we do it messes up the output somewhat.
13287 \begin_layout Standard
13288 \begin_inset listings
13292 \begin_layout Plain Layout
13299 \begin_layout Plain Layout
13304 \begin_layout Plain Layout
13311 \begin_layout Plain Layout
13320 \begin_layout Plain Layout
13325 \begin_layout Plain Layout
13330 \begin_layout Plain Layout
13335 \begin_layout Plain Layout
13342 \begin_layout Plain Layout
13349 \begin_layout Plain Layout
13354 \begin_layout Plain Layout
13363 \begin_layout Plain Layout
13367 @ifundefined{newfangle@chunk@params}{}{%
13370 \begin_layout Plain Layout
13374 newfangle@chunk@params)%
13377 \begin_layout Plain Layout
13382 \begin_layout Plain Layout
13393 \begin_layout Plain Layout
13402 \begin_layout Plain Layout
13407 \begin_layout Plain Layout
13411 @ifundefined{newfangle@chunk@append}{}{%
13414 \begin_layout Plain Layout
13420 newfangle@chunk@append{x}
13425 \begin_layout Plain Layout
13429 newfangle@chunk@append%
13432 \begin_layout Plain Layout
13439 \begin_layout Plain Layout
13444 \begin_layout Plain Layout
13452 newfangle@chunk@append{}%
13455 \begin_layout Plain Layout
13462 \begin_layout Plain Layout
13467 \begin_layout Plain Layout
13472 \begin_layout Plain Layout
13491 \begin_layout Plain Layout
13495 nwstartdeflinemarkup%
13498 \begin_layout Plain Layout
13509 \begin_layout Plain Layout
13513 nwenddeflinemarkup%
13516 \begin_layout Plain Layout
13526 \begin_layout Standard
13527 Originally this was developed as a
13528 \begin_inset Flex CharStyle:Code
13531 \begin_layout Plain Layout
13537 aspect, in the Init hook, but it was found easier to affect the title without
13539 \begin_inset Flex CharStyle:Code
13542 \begin_layout Plain Layout
13545 lst@AddToHookExe{PreSet}
13550 is still required to set the listings name to the name passed to the
13551 \begin_inset Flex CharStyle:Code
13554 \begin_layout Plain Layout
13565 \begin_layout Standard
13566 \begin_inset listings
13570 \begin_layout Plain Layout
13574 lst@BeginAspect{newfangle}
13577 \begin_layout Plain Layout
13581 lst@Key{newfangle}{true}[t]{
13583 lstKV@SetIf{#1}{true}}
13586 \begin_layout Plain Layout
13590 lst@AddToHookExe{PreSet}{
13601 \begin_layout Plain Layout
13605 lst@AddToHook{Init}{}%
13610 \begin_layout Plain Layout
13622 \begin_layout Subsection
13626 \begin_layout Standard
13629 chunkref command which makes it easy to generate visual references to different
13633 \begin_layout Standard
13634 \begin_inset Tabular
13635 <lyxtabular version="3" rows="4" columns="2">
13637 <column alignment="center" valignment="top" width="0">
13638 <column alignment="center" valignment="top" width="0">
13640 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
13643 \begin_layout Plain Layout
13649 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
13652 \begin_layout Plain Layout
13660 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
13663 \begin_layout Plain Layout
13671 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
13674 \begin_layout Plain Layout
13678 \begin_layout Plain Layout
13694 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
13697 \begin_layout Plain Layout
13700 chunkref[3]{preamble}
13705 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
13708 \begin_layout Plain Layout
13712 \begin_layout Plain Layout
13716 chunkref[3]{preamble}
13728 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
13731 \begin_layout Plain Layout
13734 chunkref{preamble}[arg1, arg2]
13739 <cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
13742 \begin_layout Plain Layout
13746 \begin_layout Plain Layout
13750 chunkref{preamble}[arg1, arg2]
13768 \begin_layout Standard
13769 Chunkref can also be used within a code chunk to include another code chunk.
13770 The third optional parameter to chunkref is a comma sepatarated list of
13771 arguments, which will replace defined parameters in the chunkref.
13772 \begin_inset Note Note
13775 \begin_layout Plain Layout
13776 Darn it, if I have: =<
13778 chunkref{new-mode-tracker}[{chunks[chunk_name, "language"]},{mode}]> the
13779 inner braces (inside [ ]) cause _ to signify subscript even though we have
13788 \begin_layout Standard
13789 \begin_inset listings
13793 \begin_layout Plain Layout
13802 \begin_layout Plain Layout
13811 \begin_layout Plain Layout
13822 \begin_layout Plain Layout
13829 \begin_layout Plain Layout
13840 \begin_layout Plain Layout
13845 \begin_layout Plain Layout
13854 \begin_layout Plain Layout
13860 chunkref@i{#1}{#2}}{
13862 chunkref@i{#1}{#2}()}%
13865 \begin_layout Plain Layout
13870 \begin_layout Plain Layout
13876 chunkref@i#1#2(#3){%
13879 \begin_layout Plain Layout
13888 \begin_layout Plain Layout
13897 \begin_layout Plain Layout
13906 \begin_layout Plain Layout
13915 \begin_layout Plain Layout
13926 \begin_layout Plain Layout
13935 \begin_layout Plain Layout
13942 \begin_layout Plain Layout
13953 \begin_layout Plain Layout
13960 \begin_layout Plain Layout
13971 \begin_layout Plain Layout
13982 \begin_layout Plain Layout
13991 \begin_layout Plain Layout
13998 \begin_layout Plain Layout
14003 \begin_layout Plain Layout
14012 \begin_layout Plain Layout
14023 \begin_layout Plain Layout
14030 \begin_layout Plain Layout
14037 \begin_layout Plain Layout
14044 \begin_layout Plain Layout
14055 \begin_layout Plain Layout
14062 \begin_layout Plain Layout
14066 chunkref@args #3,)%
14069 \begin_layout Plain Layout
14076 \begin_layout Plain Layout
14085 \begin_layout Plain Layout
14090 \begin_layout Plain Layout
14095 \begin_layout Plain Layout
14104 \begin_layout Plain Layout
14114 \begin_layout Subsection
14118 \begin_layout Standard
14119 \begin_inset listings
14123 \begin_layout Plain Layout
14128 \begin_layout Plain Layout
14140 \begin_layout Chapter
14141 Extracting newfangle
14144 \begin_layout Section
14145 Extracting from Lyx
14148 \begin_layout Standard
14149 To extract from LyX, you will need to configure LyX as explained in section
14151 \begin_inset CommandInset ref
14153 reference "sub:Configuring-the-build"
14160 \begin_layout Standard
14161 \begin_inset CommandInset label
14163 name "lyx-build-script"
14167 And this lyx-build scrap will extract newfangle for me.
14170 \begin_layout Chunk
14171 lyx-build,language=sh
14174 \begin_layout Standard
14175 \begin_inset listings
14179 \begin_layout Plain Layout
14184 \begin_layout Plain Layout
14189 \begin_layout Plain Layout
14193 \begin_layout Plain Layout
14197 chunkref{lyx-build-helper}>
14200 \begin_layout Plain Layout
14202 cd $PROJECT_DIR || exit 1
14205 \begin_layout Plain Layout
14209 \begin_layout Plain Layout
14211 /usr/local/bin/newfangle -R./newfangle $TEX_SRC > ./newfangle
14214 \begin_layout Plain Layout
14216 /usr/local/bin/newfangle -R./newfangle.module $TEX_SRC > ./newfangle.module
14219 \begin_layout Plain Layout
14223 \begin_layout Plain Layout
14227 chunkref{test:helpers}>
14230 \begin_layout Plain Layout
14235 \begin_layout Plain Layout
14237 ./newfangle -Rpca-test.awk $TEX_SRC | awk -f - || exit 1
14240 \begin_layout Plain Layout
14244 chunkref{test:cromulence}>
14252 \begin_layout Standard
14253 With a lyx-build-helper
14256 \begin_layout Chunk
14257 lyx-build-helper,language=sh
14260 \begin_layout Standard
14261 \begin_inset listings
14265 \begin_layout Plain Layout
14267 PROJECT_DIR="$LYX_r"
14270 \begin_layout Plain Layout
14272 LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
14275 \begin_layout Plain Layout
14280 \begin_layout Plain Layout
14282 TEX_SRC="$TEX_DIR/$LYX_i"
14290 \begin_layout Section
14291 Extracting from the command line
14294 \begin_layout Standard
14295 First you will need the tex output, then you can extract:
14298 \begin_layout Chunk
14299 lyx-build-manual,language=sh
14302 \begin_layout Standard
14303 \begin_inset listings
14307 \begin_layout Plain Layout
14309 lyx -e latex newfangle.lyx
14312 \begin_layout Plain Layout
14314 newfangle -R./newfangle newfangle.tex > ./newfangle
14317 \begin_layout Plain Layout
14319 newfangle -R./newfangle.module newfangle.tex > ./newfangle.module
14327 \begin_layout Section
14331 \begin_layout Chunk
14335 \begin_layout Standard
14336 \begin_inset listings
14340 \begin_layout Plain Layout
14345 \begin_layout Plain Layout
14350 \begin_layout Plain Layout
14355 \begin_layout Plain Layout
14360 \begin_layout Plain Layout
14365 \begin_layout Plain Layout
14370 \begin_layout Plain Layout
14375 \begin_layout Plain Layout
14379 \begin_layout Plain Layout
14384 \begin_layout Plain Layout
14389 \begin_layout Plain Layout
14394 \begin_layout Plain Layout
14399 \begin_layout Plain Layout
14404 \begin_layout Plain Layout
14409 \begin_layout Plain Layout
14423 \begin_layout Chapter
14427 \begin_layout Chunk
14428 tests-sub,params=THING;colour
14431 \begin_layout Standard
14432 \begin_inset listings
14436 \begin_layout Plain Layout
14438 I see a ${THING} of
14441 \begin_layout Plain Layout
14446 \begin_layout Plain Layout
14450 chunkref{tests-sub-sub}(${colour})>
14458 \begin_layout Chunk
14459 tests-sub-sub,params=colour
14462 \begin_layout Standard
14463 \begin_inset listings
14467 \begin_layout Plain Layout
14469 a funny shade of ${colour}
14477 \begin_layout Chunk
14481 \begin_layout Standard
14482 \begin_inset listings
14486 \begin_layout Plain Layout
14488 What do you see? "=<
14490 chunkref{tests-sub}(joe, red)>"
14493 \begin_layout Plain Layout
14503 \begin_layout Standard
14504 Should generate output:
14507 \begin_layout Chunk
14511 \begin_layout Standard
14512 \begin_inset listings
14516 \begin_layout Plain Layout
14518 What do you see? "I see a joe of
14521 \begin_layout Plain Layout
14526 \begin_layout Plain Layout
14528 looking closer a funny shade of red"
14531 \begin_layout Plain Layout
14536 \begin_layout Plain Layout