| blob | 03fe878ee136daaabcf51f34be1f3e0f82a775b1 |
1 From: Tony Balinski <ajbj@free.fr>
2 Subject: Unlimited macro string literal length and single-quoted strings
4 Available as a patch:
6 http://sourceforge.net/tracker/?func=detail&atid=311005&aid=1598271&group_id=11005
7 [ 1598271 ] Unlimited macro string literal length, single-quoted strings
8 macroStringLiterals.diff 2006-11-21
10 String literals are scanned twice, firstly to calculate their space
11 requirements, secondly to read their contents into allocated memory.
13 Separate string literals that follow one another are combined into one,
14 avoiding run-time concatenation of the pieces. Also single-quoted string
15 literals are allowed, within which backslash ('\') has no special meaning
16 (so you can't include a single-quote in a single-quoted string).
18 Note that a double-quoted string can be continued over multiple lines by
19 ending each line but the last with a backslash, like in C.
21 2006-11-21:
23 Fixed adjacent string literal merging to allow concatenation with "".
25 ---
27 doc/help.etx | 41 +++++---
28 source/parse.y | 291 ++++++++++++++++++++++++++++++++++-----------------------
29 2 files changed, 206 insertions(+), 126 deletions(-)
31 diff --quilt old/doc/help.etx new/doc/help.etx
32 --- old/doc/help.etx
33 +++ new/doc/help.etx
34 @@ -1945,15 +1945,16 @@ Macro Language
35 conditionally, such as the body of a loop, are surrounded by curly braces
36 "{}".
38 Blank lines and comments are also allowed. Comments begin with a "#" and end
39 with a newline, and can appear either on a line by themselves, or at the end
40 - of a statement.
41 + of a statement line.
43 Statements which are too long to fit on a single line may be split across
44 several lines, by placing a backslash "\" character at the end of each line
45 - to be continued.
46 + to be continued. Note that a comment with a backslash at the end is treated
47 + as a continuation in this way too.
50 3>Data Types
52 The NEdit macro language recognizes only three data types, dynamic character
53 @@ -1971,16 +1972,18 @@ Macro Language
54 a = -1
55 b = 1000
57 4>Character String Constants
59 - Character string constants are enclosed in double quotes. For example:
60 + Character string constants are enclosed in single or double quotes, but the
61 + start and end quotes must be the same character. For example:
63 a = "a string"
64 - dialog("Hi there!", "OK")
65 + dialog('Hi there!', "OK")
67 - Strings may also include C-language style escape sequences:
68 + A double-quoted string may also include C-language style escape
69 + sequences:
71 \\ Backslash \t Tab \f Form feed
72 \" Double quote \b Backspace \a Alert
73 \n Newline \r Carriage return \v Vertical tab
75 @@ -1994,32 +1997,48 @@ Macro Language
76 explicit newlines, and also buffers its output on a per-line basis:
78 t_print("a = " a "\n")
80 Other characters can be expressed as backslash-escape sequences in macro
81 - strings. The format is the same as for regular expressions, described in the
82 - paragraphs headed "Octal and Hex Escape Sequences" of the section
83 - "Metacharacters_", except that an octal escape sequence can start with any
84 - octal digit, not just 0, so the single character string "\0033" is the same
85 - as "\33", "\x1B" and "\e" (for an ASCII version of NEdit).
86 + double-quoted strings. The format is the same as for regular expressions,
87 + described in the paragraphs headed "Octal and Hex Escape Sequences" of the
88 + section "Metacharacters_", except that an octal escape sequence can start with
89 + any octal digit, not just 0, so the single character string "\0033" is the
90 + same as "\33", "\x1B" and "\e" (for an ASCII version of NEdit).
92 Note that if you want to define a regular expression in a macro string,
93 you need to "double-up" the backslashes for the metacharacters with
94 special meaning in regular expressions. For example, the expression
96 (?N(\s|/\*(?n(?:(?!\*/).)*)\*/|//.*\n|\n)+)
98 which matches whitespace or C/C++/Java-style comments, should be written as
99 - a macro string as
100 + a macro double-quoted string as
102 "(?N(\\s|/\\*(?n(?:(?!\\*/).)*)\\*/|//.*\n|\n)+)"
104 (The "\n"s towards the end add literal newline characters to the string. The
105 regular expression interpretation treats the newlines as themselves. It can
106 also interpret the sequence "\\n" as a newline, although the macro string here
107 would then contain a literal backslash followed by a lowercase `N'.)
109 + Alternatively, if you don't need special escapes or a single quote
110 + (apostrophe) in your string (true for this example), just turn the expression
111 + into a single-quoted string, as
112 +
113 + '(?N(\s|/\*(?n(?:(?!\*/).)*)\*/|//.*\n|\n)+)'
114 +
115 + Neighboring string literals (separated by whitespace or line continuations)
116 + are combined, as if by the concatenation operation before use. For example
117 +
118 + "The backslash '" '\' "' is an " \
119 + 'escape only in "double-quoted" strings' "\n"
120 +
121 + is treated as a single string ending with a newline character, looking like
122 +
123 + The backslash '\' is an escape only in "double-quoted" strings
124 +
126 3>Variables
128 Variable names must begin either with a letter (local variables), or a $
129 (global variables). Beyond the first character, variables may also contain
130 diff --quilt old/source/parse.y new/source/parse.y
131 --- old/source/parse.y
132 +++ new/source/parse.y
133 @@ -67,10 +67,11 @@ static int yylex(void);
134 int yyparse(void);
135 static int follow(char expect, int yes, int no);
136 static int follow2(char expect1, int yes1, char expect2, int yes2, int no);
137 static int follow_non_whitespace(char expect, int yes, int no);
138 static Symbol *matchesActionRoutine(char **inPtr);
139 +static int scanString(void);
141 static char *ErrMsg;
142 static char *InPtr;
143 extern Inst *LoopStack[]; /* addresses of break, cont stmts */
144 extern Inst **LoopStackPtr; /* to fill at the end of a loop */
145 @@ -673,16 +674,10 @@ static char skipWhitespace(void)
146 static int yylex(void)
147 {
148 int i, len;
149 Symbol *s;
150 static DataValue value = {NO_TAG, {0}};
151 - static char escape[] = "\\\"ntbrfave";
152 -#ifdef EBCDIC_CHARSET
153 - static char replace[] = "\\\"\n\t\b\r\f\a\v\x27"; /* EBCDIC escape */
154 -#else
155 - static char replace[] = "\\\"\n\t\b\r\f\a\v\x1B"; /* ASCII escape */
156 -#endif
157 int result;
159 skipWhitespace();
161 /* return end of input at the end of the string */
162 @@ -737,119 +732,14 @@ static int yylex(void)
163 }
164 yylval.sym = s;
165 return SYMBOL;
166 }
168 - /* Process quoted strings with embedded escape sequences:
169 - For backslashes we recognise hexadecimal values with initial 'x' such
170 - as "\x1B"; octal value (upto 3 oct digits with a possible leading zero)
171 - such as "\33", "\033" or "\0033", and the C escapes: \", \', \n, \t, \b,
172 - \r, \f, \a, \v, and the added \e for the escape character, as for REs.
173 - Disallow hex/octal zero values (NUL): instead ignore the introductory
174 - backslash, eg "\x0xyz" becomes "x0xyz" and "\0000hello" becomes
175 - "0000hello". */
176 -
177 - if (*InPtr == '\"') {
178 - char string[MAX_STRING_CONST_LEN], *p = string;
179 - char *backslash;
180 - InPtr++;
181 - while (*InPtr != '\0' && *InPtr != '\"' && *InPtr != '\n') {
182 - if (p >= string + MAX_STRING_CONST_LEN) {
183 - InPtr++;
184 - continue;
185 - }
186 - if (*InPtr == '\\') {
187 - backslash = InPtr;
188 - InPtr++;
189 - if (*InPtr == '\n') {
190 - InPtr++;
191 - continue;
192 - }
193 - if (*InPtr == 'x') {
194 - /* a hex introducer */
195 - int hexValue = 0;
196 - const char *hexDigits = "0123456789abcdef";
197 - const char *hexD;
198 - InPtr++;
199 - if (*InPtr == '\0' ||
200 - (hexD = strchr(hexDigits, tolower(*InPtr))) == NULL) {
201 - *p++ = 'x';
202 - }
203 - else {
204 - hexValue = hexD - hexDigits;
205 - InPtr++;
206 - /* now do we have another digit? only accept one more */
207 - if (*InPtr != '\0' &&
208 - (hexD = strchr(hexDigits,tolower(*InPtr))) != NULL){
209 - hexValue = hexD - hexDigits + (hexValue << 4);
210 - InPtr++;
211 - }
212 - if (hexValue != 0) {
213 - *p++ = (char)hexValue;
214 - }
215 - else {
216 - InPtr = backslash + 1; /* just skip the backslash */
217 - }
218 - }
219 - continue;
220 - }
221 - /* the RE documentation requires \0 as the octal introducer;
222 - here you can start with any octal digit, but you are only
223 - allowed up to three (or four if the first is '0'). */
224 - if ('0' <= *InPtr && *InPtr <= '7') {
225 - if (*InPtr == '0') {
226 - InPtr++; /* octal introducer: don't count this digit */
227 - }
228 - if ('0' <= *InPtr && *InPtr <= '7') {
229 - /* treat as octal - first digit */
230 - char octD = *InPtr++;
231 - int octValue = octD - '0';
232 - if ('0' <= *InPtr && *InPtr <= '7') {
233 - /* second digit */
234 - octD = *InPtr++;
235 - octValue = (octValue << 3) + octD - '0';
236 - /* now do we have another digit? can we add it?
237 - if value is going to be too big for char (greater
238 - than 0377), stop converting now before adding the
239 - third digit */
240 - if ('0' <= *InPtr && *InPtr <= '7' &&
241 - octValue <= 037) {
242 - /* third digit is acceptable */
243 - octD = *InPtr++;
244 - octValue = (octValue << 3) + octD - '0';
245 - }
246 - }
247 - if (octValue != 0) {
248 - *p++ = (char)octValue;
249 - }
250 - else {
251 - InPtr = backslash + 1; /* just skip the backslash */
252 - }
253 - }
254 - else { /* \0 followed by non-digits: go back to 0 */
255 - InPtr = backslash + 1; /* just skip the backslash */
256 - }
257 - continue;
258 - }
259 - for (i=0; escape[i]!='\0'; i++) {
260 - if (escape[i] == *InPtr) {
261 - *p++ = replace[i];
262 - InPtr++;
263 - break;
264 - }
265 - }
266 - /* if we get here, we didn't recognise the character after
267 - the backslash: just copy it next time round the loop */
268 - }
269 - else {
270 - *p++= *InPtr++;
271 - }
272 - }
273 - *p = '\0';
274 - InPtr++;
275 - yylval.sym = InstallStringConstSymbol(string);
276 - return STRING;
277 + /* Process quoted strings */
278 +
279 + if (*InPtr == '\"' || *InPtr == '\'') {
280 + return scanString();
281 }
283 /* process remaining two character tokens or return single char as token */
284 result = *InPtr++;
285 switch (result) {
286 @@ -949,10 +839,181 @@ static Symbol *matchesActionRoutine(char
287 *inPtr = c;
288 return s;
289 }
291 /*
292 +** Process quoted string literals. These can be in single or double quotes.
293 +** A sequence of string literals separated by whitespace (see skipWhitespace())
294 +** are read as a single string.
295 +**
296 +** Double-quoted string literals allow embedded escape sequences:
297 +** For backslashes we recognise hexadecimal values with initial 'x' such
298 +** as "\x1B"; octal value (upto 3 oct digits with a possible leading zero)
299 +** such as "\33", "\033" or "\0033", and the C escapes: \", \', \n, \t, \b,
300 +** \r, \f, \a, \v, and the added \e for the escape character, as for REs.
301 +** We disallow hex/octal zero values (NUL): instead ignore the introductory
302 +** backslash, eg "\x0xyz" becomes "x0xyz" and "\0000hello" becomes "0000hello".
303 +** An escaped newline is elided, and the string content continues on the next
304 +** source line.
305 +*/
306 +static int scanString(void)
307 +{
308 +# define SCANSTRING_WRITE_TO_STRING(p, len, val) \
309 + do { char mc = (val); if (p) { *p++ = mc; } else { ++len; } } while (0)
310 +
311 + /* scan the string twice: once to get its size, then again to build it */
312 + char *startPtr = InPtr;
313 + char *p = NULL, *string = NULL;
314 + int len, scan, i;
315 + char stopper, first_stopper = *startPtr;
316 + char *backslash;
317 + int handleBackslash;
318 +
319 + static char escape[] = "\\\"ntbrfave";
320 +#ifdef EBCDIC_CHARSET
321 + static char replace[] = "\\\"\n\t\b\r\f\a\v\x27"; /* EBCDIC escape */
322 +#else
323 + static char replace[] = "\\\"\n\t\b\r\f\a\v\x1B"; /* ASCII escape */
324 +#endif
325 +
326 + if (first_stopper != '\"' && first_stopper != '\'')
327 + return yyerror("expected a string");
328 +
329 + for (scan = 0; scan < 2; ++scan)
330 + {
331 + InPtr = startPtr;
332 + stopper = first_stopper;
333 + handleBackslash = (stopper == '\"');
334 + len = 0;
335 + InPtr++;
336 + while (*InPtr != '\0' && *InPtr != '\n') {
337 + if (*InPtr == stopper) {
338 + char *endPtr = InPtr++;
339 + skipWhitespace();
340 + /* is this followed by another string literal? */
341 + if (*InPtr == '\"' || *InPtr == '\'') {
342 + stopper = *InPtr++; /* add it to the end of the first */
343 + handleBackslash = (stopper == '\"');
344 + }
345 + else {
346 + InPtr = endPtr; /* no further string: restore position */
347 + break;
348 + }
349 + }
350 + else if (handleBackslash && *InPtr == '\\') {
351 + backslash = InPtr;
352 + InPtr++;
353 + if (*InPtr == '\n') { /* allows newline to be skipped */
354 + InPtr++;
355 + continue;
356 + }
357 + if (*InPtr == 'x') {
358 + /* a hex introducer */
359 + int hexValue = 0;
360 + const char *hexDigits = "0123456789abcdef";
361 + const char *hexD;
362 + InPtr++;
363 + if (*InPtr == '\0')
364 + break;
365 + if ((hexD = strchr(hexDigits, tolower(*InPtr))) == NULL) {
366 + SCANSTRING_WRITE_TO_STRING(p, len, 'x');
367 + }
368 + else {
369 + hexValue = hexD - hexDigits;
370 + InPtr++;
371 + if (*InPtr == '\0')
372 + break;
373 + /* now do we have another digit? only accept one more */
374 + if ((hexD = strchr(hexDigits,tolower(*InPtr))) != NULL){
375 + hexValue = hexD - hexDigits + (hexValue << 4);
376 + InPtr++;
377 + }
378 + if (hexValue != 0) {
379 + SCANSTRING_WRITE_TO_STRING(p, len, (char)hexValue);
380 + }
381 + else {
382 + InPtr = backslash + 1; /* just skip the backslash */
383 + }
384 + }
385 + continue;
386 + }
387 + /* the RE documentation requires \0 as the octal introducer;
388 + here you can start with any octal digit, but you are only
389 + allowed up to three (or four if the first is '0'). */
390 + if ('0' <= *InPtr && *InPtr <= '7') {
391 + if (*InPtr == '0') {
392 + InPtr++; /* octal introducer: don't count this digit */
393 + }
394 + if ('0' <= *InPtr && *InPtr <= '7') {
395 + /* treat as octal - first digit */
396 + char octD = *InPtr++;
397 + int octValue = octD - '0';
398 + if ('0' <= *InPtr && *InPtr <= '7') {
399 + /* second digit */
400 + octD = *InPtr++;
401 + octValue = (octValue << 3) + octD - '0';
402 + /* now do we have another digit? can we add it?
403 + if value is going to be too big for char (greater
404 + than 0377), stop converting now before adding the
405 + third digit */
406 + if ('0' <= *InPtr && *InPtr <= '7' &&
407 + octValue <= 037) {
408 + /* third digit is acceptable */
409 + octD = *InPtr++;
410 + octValue = (octValue << 3) + octD - '0';
411 + }
412 + }
413 + if (octValue != 0) {
414 + SCANSTRING_WRITE_TO_STRING(p, len, (char)octValue);
415 + }
416 + else {
417 + InPtr = backslash + 1; /* just skip the backslash */
418 + }
419 + }
420 + else { /* \0 followed by non-digits: go back to 0 */
421 + InPtr = backslash + 1; /* just skip the backslash */
422 + }
423 + continue;
424 + }
425 + /* check for a valid c-style escape character */
426 + for (i = 0; escape[i] != '\0'; i++) {
427 + if (escape[i] == *InPtr) {
428 + SCANSTRING_WRITE_TO_STRING(p, len, replace[i]);
429 + InPtr++;
430 + break;
431 + }
432 + }
433 + /* if we get here, we didn't recognise the character after
434 + the backslash: just copy it next time round the loop */
435 + }
436 + else {
437 + SCANSTRING_WRITE_TO_STRING(p, len, *InPtr++);
438 + }
439 + }
440 + /* terminate the string content */
441 + SCANSTRING_WRITE_TO_STRING(p, len, '\0');
442 + if (*InPtr == stopper) {
443 + if (!p) {
444 + /* this was the size measurement and validation */
445 + p = string = AllocString(len);
446 + }
447 + else {
448 + /* OK: string now contains our string text */
449 + InPtr++; /* skip past stopper */
450 + yylval.sym = InstallStringConstSymbol(string);
451 + return STRING;
452 + }
453 + }
454 + else {
455 + /* failure: end quote doesn't match start quote */
456 + break;
457 + }
458 + }
459 + return yyerror("unterminated string");
460 +}
461 +
462 +/*
463 ** Called by yacc to report errors (just stores for returning when
464 ** parsing is aborted. The error token action is to immediate abort
465 ** parsing, so this message is immediately reported to the caller
466 ** of ParseExpr)
467 */