1 This is flex.info, produced by makeinfo version 4.5 from flex.texi.
3 INFO-DIR-SECTION Programming
5 * flex: (flex). Fast lexical analyzer generator (lex replacement).
9 The flex manual is placed under the same licensing conditions as the
12 Copyright (C) 1990, 1997 The Regents of the University of California.
15 This code is derived from software contributed to Berkeley by Vern
18 The United States Government has rights in this work pursuant to
19 contract no. DE-AC03-76SF00098 between the United States Department of
20 Energy and the University of California.
22 Redistribution and use in source and binary forms, with or without
23 modification, are permitted provided that the following conditions are
26 1. Redistributions of source code must retain the above copyright
27 notice, this list of conditions and the following disclaimer.
29 2. Redistributions in binary form must reproduce the above copyright
30 notice, this list of conditions and the following disclaimer in the
31 documentation and/or other materials provided with the
33 Neither the name of the University nor the names of its contributors
34 may be used to endorse or promote products derived from this software
35 without specific prior written permission.
37 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
38 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
39 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
41 File: flex.info, Node: unnamed-faq-76, Next: unnamed-faq-77, Prev: unnamed-faq-75, Up: FAQ
47 To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com>
48 Subject: Re: FLEX 2.5 & THE YEAR 2000
49 In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT.
50 Date: Wed, 03 Jun 1998 10:22:26 PDT
51 From: Vern Paxson <vern>
53 > I am researching the Y2K problem with General Electric R&D
54 > and need to know if there are any known issues concerning
55 > the above mentioned software and Y2K regardless of version.
57 There shouldn't be, all it ever does with the date is ask the system
58 for it and then print it out.
63 File: flex.info, Node: unnamed-faq-77, Next: unnamed-faq-78, Prev: unnamed-faq-76, Up: FAQ
69 To: "Hans Dermot Doran" <htd@ibhdoran.com>
70 Subject: Re: flex problem
71 In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT.
72 Date: Tue, 21 Jul 1998 14:23:34 PDT
73 From: Vern Paxson <vern>
75 > To overcome this, I gets() the stdin into a string and lex the string. The
76 > string is lexed OK except that the end of string isn't lexed properly
77 > (yy_scan_string()), that is the lexer dosn't recognise the end of string.
79 Flex doesn't contain mechanisms for recognizing buffer endpoints. But if
80 you use fgets instead (which you should anyway, to protect against buffer
81 overflows), then the final \n will be preserved in the string, and you can
82 scan that in order to find the end of the string.
87 File: flex.info, Node: unnamed-faq-78, Next: unnamed-faq-79, Prev: unnamed-faq-77, Up: FAQ
93 To: soumen@almaden.ibm.com
94 Subject: Re: Flex++ 2.5.3 instance member vs. static member
95 In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT.
96 Date: Tue, 28 Jul 1998 01:10:34 PDT
97 From: Vern Paxson <vern>
107 > Now you'd expect mylineno to be a member of each instance of class
108 > yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to
109 > indicate otherwise; unless I am missing something the declaration of
110 > mylineno seems to be outside any class scope.
112 > How will this work if I want to run a multi-threaded application with each
113 > thread creating a FlexLexer instance?
115 Derive your own subclass and make mylineno a member variable of it.
120 File: flex.info, Node: unnamed-faq-79, Next: unnamed-faq-80, Prev: unnamed-faq-78, Up: FAQ
126 To: Adoram Rogel <adoram@hybridge.com>
127 Subject: Re: More than 32K states change hangs
128 In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT.
129 Date: Tue, 04 Aug 1998 22:28:45 PDT
130 From: Vern Paxson <vern>
134 > I followed your advice, posted on Usenet bu you, and emailed to me
135 > personally by you, on how to overcome the 32K states limit. I'm running
137 > I took the full source of version 2.5.4 and did the following changes in
139 > #define JAMSTATE -327660
140 > #define MAXIMUM_MNS 319990
141 > #define BAD_SUBSCRIPT -327670
142 > #define MAX_SHORT 327000
145 > All looked fine, including check and bigcheck, so I installed.
147 Hmmm, you shouldn't increase MAX_SHORT, though looking through my email
148 archives I see that I did indeed recommend doing so. Try setting it back
149 to 32700; that should suffice that you no longer need -Ca. If it still
150 hangs, then the interesting question is - where?
152 > Compiling the same hanged program with a out-of-the-box (RedHat 4.2
153 > distribution of Linux)
154 > flex 2.5.4 binary works.
156 Since Linux comes with source code, you should diff it against what
157 you have to see what problems they missed.
159 > Should I always compile with the -Ca option now ? even short and simple
162 No, definitely not. It's meant to be for those situations where you
163 absolutely must squeeze every last cycle out of your scanner.
168 File: flex.info, Node: unnamed-faq-80, Next: unnamed-faq-81, Prev: unnamed-faq-79, Up: FAQ
174 To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com>
175 Subject: Re: flex output for static code portion
176 In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT.
177 Date: Mon, 17 Aug 1998 23:57:42 PDT
178 From: Vern Paxson <vern>
180 > I would like to use flex under the hood to generate a binary file
181 > containing the data structures that control the parse.
183 This has been on the wish-list for a long time. In principle it's
184 straight-forward - you redirect mkdata() et al's I/O to another file,
185 and modify the skeleton to have a start-up function that slurps these
186 into dynamic arrays. The concerns are (1) the scanner generation code
187 is hairy and full of corner cases, so it's easy to get surprised when
188 going down this path :-( ; and (2) being careful about buffering so
189 that when the tables change you make sure the scanner starts in the
190 correct state and reading at the right point in the input file.
192 > I was wondering if you know of anyone who has used flex in this way.
194 I don't - but it seems like a reasonable project to undertake (unlike
195 numerous other flex tweaks :-).
200 File: flex.info, Node: unnamed-faq-81, Next: unnamed-faq-82, Prev: unnamed-faq-80, Up: FAQ
206 Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11])
207 by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838
208 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT)
209 Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2])
210 by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694
211 for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200
212 Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200
213 From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de>
214 Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de>
215 Subject: "flex scanner push-back overflow"
217 Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST)
218 Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
219 X-NoJunk: Do NOT send commercial mail, spam or ads to this address!
220 X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/
221 X-Mailer: ELM [version 2.4ME+ PL28 (25)]
223 Content-Type: text/plain; charset=US-ASCII
224 Content-Transfer-Encoding: 7bit
228 Yesterday, I encountered a strange problem: I use the macro processor m4
229 to include some lengthy lists into a .l file. Following is a flex macro
230 definition that causes some serious pain in my neck:
232 AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...])
234 The complete list contains about 10kB. When I try to "flex" this file
235 (on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased
236 some of the predefined values in flexdefs.h) I get the error:
238 myflex/flex -8 sentag.tmp.l
239 flex scanner push-back overflow
241 When I remove the slashes in the macro definition everything works fine.
242 As I understand it, the double quotes escape the slash-character so it
243 really means "/" and not "trailing context". Furthermore, I tried to
244 escape the slashes with backslashes, but with no use, the same error message
245 appeared when flexing the code.
247 Do you have an idea what's going on here?
249 Greetings from Germany,
252 Georg Rehm georg@cl-ki.uni-osnabrueck.de
253 Institute for Semantic Information Processing, University of Osnabrueck, FRG
256 File: flex.info, Node: unnamed-faq-82, Next: unnamed-faq-83, Prev: unnamed-faq-81, Up: FAQ
262 To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
263 Subject: Re: "flex scanner push-back overflow"
264 In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT.
265 Date: Thu, 20 Aug 1998 07:05:35 PDT
266 From: Vern Paxson <vern>
268 > myflex/flex -8 sentag.tmp.l
269 > flex scanner push-back overflow
271 Flex itself uses a flex scanner. That scanner is running out of buffer
272 space when it tries to unput() the humongous macro you've defined. When
273 you remove the '/'s, you make it small enough so that it fits in the buffer;
274 removing spaces would do the same thing.
276 The fix is to either rethink how come you're using such a big macro and
277 perhaps there's another/better way to do it; or to rebuild flex's own
278 scan.c with a larger value for
280 #define YY_BUF_SIZE 16384
285 File: flex.info, Node: unnamed-faq-83, Next: unnamed-faq-84, Prev: unnamed-faq-82, Up: FAQ
291 To: Jan Kort <jan@research.techforce.nl>
293 In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200.
294 Date: Sat, 05 Sep 1998 00:59:49 PDT
295 From: Vern Paxson <vern>
299 > "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); }
300 > ^\n { fprintf(stderr, "empty line\n"); }
302 > \n { fprintf(stderr, "new line\n"); }
305 > -- input ---------------------------------------
307 > -- output --------------------------------------
310 > ------------------------------------------------
312 IMHO, it's not clear whether or not this is in fact a bug. It depends
313 on whether you view yyless() as backing up in the input stream, or as
314 pushing new characters onto the beginning of the input stream. Flex
315 interprets it as the latter (for implementation convenience, I'll admit),
316 and so considers the newline as in fact matching at the beginning of a
317 line, as after all the last token scanned an entire line and so the
318 scanner is now at the beginning of a new line.
320 I agree that this is counter-intuitive for yyless(), given its
321 functional description (it's less so for unput(), depending on whether
322 you're unput()'ing new text or scanned text). But I don't plan to
323 change it any time soon, as it's a pain to do so. Consequently,
324 you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak
325 your scanner into the behavior you desire.
327 Sorry for the less-than-completely-satisfactory answer.
332 File: flex.info, Node: unnamed-faq-84, Next: unnamed-faq-85, Prev: unnamed-faq-83, Up: FAQ
338 To: Patrick Krusenotto <krusenot@mac-info-link.de>
339 Subject: Re: Problems with restarting flex-2.5.2-generated scanner
340 In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT.
341 Date: Thu, 24 Sep 1998 23:28:43 PDT
342 From: Vern Paxson <vern>
344 > I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately
345 > trying to make my scanner restart with a new file after my parser stops
346 > with a parse error. When my compiler restarts, the parser always
347 > receives the token after the token (in the old file!) that caused the
350 I suspect the problem is that your parser has read ahead in order
351 to attempt to resolve an ambiguity, and when it's restarted it picks
352 up with that token rather than reading a fresh one. If you're using
353 yacc, then the special "error" production can sometimes be used to
354 consume tokens in an attempt to get the parser into a consistent state.
359 File: flex.info, Node: unnamed-faq-85, Next: unnamed-faq-86, Prev: unnamed-faq-84, Up: FAQ
365 To: Henric Jungheim <junghelh@pe-nelson.com>
366 Subject: Re: flex 2.5.4a
367 In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST.
368 Date: Tue, 27 Oct 1998 16:50:14 PST
369 From: Vern Paxson <vern>
371 > This brings up a feature request: How about a command line
372 > option to specify the filename when reading from stdin? That way one
373 > doesn't need to create a temporary file in order to get the "#line"
374 > directives to make sense.
376 Use -o combined with -t (per the man page description of -o).
378 > P.S., Is there any simple way to use non-blocking IO to parse multiple
383 One approach might be to return a magic character on EWOULDBLOCK and
386 .*<magic-character> // put back .*, eat magic character
388 This is off the top of my head, not sure it'll work.
393 File: flex.info, Node: unnamed-faq-86, Next: unnamed-faq-87, Prev: unnamed-faq-85, Up: FAQ
399 To: "Repko, Billy D" <billy.d.repko@intel.com>
400 Subject: Re: Compiling scanners
401 In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST.
402 Date: Thu, 14 Jan 1999 00:25:30 PST
403 From: Vern Paxson <vern>
405 > It appears that maybe it cannot find the lfl library.
407 The Makefile in the distribution builds it, so you should have it.
408 It's exceedingly trivial, just a main() that calls yylex() and
409 a yyrap() that always returns 1.
412 > \n ++num_lines; ++num_chars;
415 You can't indent your rules like this - that's where the errors are coming
416 from. Flex copies indented text to the output file, it's how you do things
419 int num_lines_seen = 0;
421 to declare local variables.
426 File: flex.info, Node: unnamed-faq-87, Next: unnamed-faq-88, Prev: unnamed-faq-86, Up: FAQ
432 To: Erick Branderhorst <Erick.Branderhorst@asml.nl>
433 Subject: Re: flex input buffer
434 In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST.
435 Date: Tue, 09 Feb 1999 21:03:37 PST
436 From: Vern Paxson <vern>
438 > In the flex.skl file the size of the default input buffers is set. Can you
439 > explain why this size is set and why it is such a high number.
441 It's large to optimize performance when scanning large files. You can
442 safely make it a lot lower if needed.
447 File: flex.info, Node: unnamed-faq-88, Next: unnamed-faq-90, Prev: unnamed-faq-87, Up: FAQ
453 To: "Guido Minnen" <guidomi@cogs.susx.ac.uk>
454 Subject: Re: Flex error message
455 In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST.
456 Date: Thu, 25 Feb 1999 00:11:31 PST
457 From: Vern Paxson <vern>
459 > I'm extending a larger scanner written in Flex and I keep running into
460 > problems. More specifically, I get the error message:
461 > "flex: input rules are too complicated (>= 32000 NFA states)"
463 Increase the definitions in flexdef.h for:
465 #define JAMSTATE -32766 /* marks a reference to the state that always j
467 #define MAXIMUM_MNS 31999
468 #define BAD_SUBSCRIPT -32767
470 recompile everything, and it should all work.
475 File: flex.info, Node: unnamed-faq-90, Next: unnamed-faq-91, Prev: unnamed-faq-88, Up: FAQ
481 To: "Dmitriy Goldobin" <gold@ems.chel.su>
482 Subject: Re: FLEX trouble
483 In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT.
484 Date: Tue, 01 Jun 1999 00:15:07 PDT
485 From: Vern Paxson <vern>
487 > I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20
488 > but rule "/*"(.|\n)*"*/" don't work ?
490 The second of these will have to scan the entire input stream (because
491 "(.|\n)*" matches an arbitrary amount of any text) in order to see if
492 it ends with "*/", terminating the comment. That potentially will overflow
495 > More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error
496 > 'unrecognized rule'.
498 You can't use the '/' operator inside parentheses. It's not clear
499 what "(a/b)*" actually means.
501 > I now use workaround with state <comment>, but single-rule is
504 Single-rule is nice but will always have the problem of either setting
505 restrictions on comments (like not allowing multi-line comments) and/or
506 running the risk of consuming the entire input stream, as noted above.
511 File: flex.info, Node: unnamed-faq-91, Next: unnamed-faq-92, Prev: unnamed-faq-90, Up: FAQ
517 Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18])
518 by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100
519 for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT)
520 Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999
522 Date: Tue, 15 Jun 1999 08:55:43 -0700
523 From: "Aki Niimura" <neko@my-deja.com>
524 Message-ID: <KNONDOHDOBGAEAAA@my-deja.com>
529 X-Mailer: MailCity Service
530 Subject: A question on flex C++ scanner
531 X-Sender-Ip: 12.72.207.61
532 Organization: My Deja Email (http://www.my-deja.com:80)
533 Content-Type: text/plain; charset=us-ascii
534 Content-Transfer-Encoding: 7bit
538 I have been using flex for years.
539 It works very well on many projects.
540 Most case, I used it to generate a scanner on C language.
541 However, one project I needed to generate a scanner
542 on C++ lanuage. Thanks to your enhancement, flex did
545 Currently, I'm working on enhancing my previous project.
546 I need to deal with multiple input streams (recursive
547 inclusion) in this scanner (C++).
548 I did similar thing for another scanner (C) as you
549 explained in your documentation.
551 The generated scanner (C++) has necessary methods:
552 - switch_to_buffer(struct yy_buffer_state *b)
553 - yy_create_buffer(istream *is, int sz)
554 - yy_delete_buffer(struct yy_buffer_state *b)
556 However, I couldn't figure out how to access current
557 buffer (yy_current_buffer).
559 yy_current_buffer is a protected member of yyFlexLexer.
560 I can't access it directly.
561 Then, I thought yy_create_buffer() with is = 0 might
562 return current stream buffer. But it seems not as far
563 as I checked the source. (flex 2.5.4)
565 I went through the Web in addition to Flex documentation.
566 However, it hasn't been successful, so far.
568 It is not my intention to bother you, but, can you
569 comment about how to obtain the current stream buffer?
571 Your response would be highly appreciated.
576 --== Sent via Deja.com http://www.deja.com/ ==--
577 Share what you know. Learn what you don't.
580 File: flex.info, Node: unnamed-faq-92, Next: unnamed-faq-93, Prev: unnamed-faq-91, Up: FAQ
587 Subject: Re: A question on flex C++ scanner
588 In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT.
589 Date: Tue, 15 Jun 1999 09:04:24 PDT
590 From: Vern Paxson <vern>
592 > However, I couldn't figure out how to access current
593 > buffer (yy_current_buffer).
595 Derive your own subclass from yyFlexLexer.
600 File: flex.info, Node: unnamed-faq-93, Next: unnamed-faq-94, Prev: unnamed-faq-92, Up: FAQ
606 To: "Stones, Darren" <Darren.Stones@nectech.co.uk>
607 Subject: Re: You're the man to see?
608 In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT.
609 Date: Wed, 23 Jun 1999 09:01:40 PDT
610 From: Vern Paxson <vern>
612 > I hope you can help me. I am using Flex and Bison to produce an interpreted
613 > language. However all goes well until I try to implement an IF statement or
614 > a WHILE. I cannot get this to work as the parser parses all the conditions
615 > eg. the TRUE and FALSE conditons to check for a rule match. So I cannot
618 You need to use the parser to build a parse tree (= abstract syntax trwee),
619 and when that's all done you recursively evaluate the tree, binding variables
620 to values at that time.
625 File: flex.info, Node: unnamed-faq-94, Next: unnamed-faq-95, Prev: unnamed-faq-93, Up: FAQ
631 To: Petr Danecek <petr@ics.cas.cz>
632 Subject: Re: flex - question
633 In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT.
634 Date: Fri, 02 Jul 1999 16:52:13 PDT
635 From: Vern Paxson <vern>
637 > file, it takes an enormous amount of time. It is funny, because the
638 > source code has only 12 rules!!! I think it looks like an exponencial
641 Right, that's the problem - some patterns (those with a lot of
642 ambiguity, where yours has because at any given time the scanner can
643 be in the middle of all sorts of combinations of the different
644 rules) blow up exponentially.
646 For your rules, there is an easy fix. Change the ".*" that comes fater
647 the directory name to "[^ ]*". With that in place, the rules are no
648 longer nearly so ambiguous, because then once one of the directories
649 has been matched, no other can be matched (since they all require a
652 If that's not an acceptable solution, then you can enter a start state
653 to pick up the .*\n after each directory is matched.
655 Also note that for speed, you'll want to add a ".*" rule at the end,
656 otherwise rules that don't match any of the patterns will be matched
657 very slowly, a character at a time.
662 File: flex.info, Node: unnamed-faq-95, Next: unnamed-faq-96, Prev: unnamed-faq-94, Up: FAQ
668 To: Tielman Koekemoer <tielman@spi.co.za>
669 Subject: Re: Please help.
670 In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT.
671 Date: Thu, 08 Jul 1999 08:20:39 PDT
672 From: Vern Paxson <vern>
674 > I was hoping you could help me with my problem.
676 > I tried compiling (gnu)flex on a Solaris 2.4 machine
677 > but when I ran make (after configure) I got an error.
679 > --------------------------------------------------------------
680 > gcc -c -I. -I. -g -O parse.c
681 > ./flex -t -p ./scan.l >scan.c
682 > sh: ./flex: not found
684 > make: Fatal error: Command failed for target `scan.c'
685 > -------------------------------------------------------------
687 > What's strange to me is that I'm only
688 > trying to install flex now. I then edited the Makefile to
689 > and changed where it says "FLEX = flex" to "FLEX = lex"
690 > ( lex: the native Solaris one ) but then it complains about
691 > the "-p" option. Is there any way I can compile flex without
694 > Thanks so much for your time.
696 You managed to step on the bootstrap sequence, which first copies
697 initscan.c to scan.c in order to build flex. Try fetching a fresh
698 distribution from ftp.ee.lbl.gov. (Or you can first try removing
699 ".bootstrap" and doing a make again.)
704 File: flex.info, Node: unnamed-faq-96, Next: unnamed-faq-97, Prev: unnamed-faq-95, Up: FAQ
710 To: Tielman Koekemoer <tielman@spi.co.za>
711 Subject: Re: Please help.
712 In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT.
713 Date: Fri, 09 Jul 1999 00:27:20 PDT
714 From: Vern Paxson <vern>
716 > First I removed .bootstrap (and ran make) - no luck. I downloaded the
717 > software but I still have the same problem. Is there anything else I
726 If this last tries to first build scan.c from scan.l using ./flex, then
727 your "make" is broken, in which case compile scan.c to scan.o by hand.
732 File: flex.info, Node: unnamed-faq-97, Next: unnamed-faq-98, Prev: unnamed-faq-96, Up: FAQ
738 To: Sumanth Kamenani <skamenan@crl.nmsu.edu>
740 In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT.
741 Date: Tue, 20 Jul 1999 00:18:26 PDT
742 From: Vern Paxson <vern>
744 > I am getting a compilation error. The error is given as "unknown symbol- yylex".
746 The parser relies on calling yylex(), but you're instead using the C++ scanning
747 class, so you need to supply a yylex() "glue" function that calls an instance
748 scanner of the scanner (e.g., "scanner->yylex()").
753 File: flex.info, Node: unnamed-faq-98, Next: unnamed-faq-99, Prev: unnamed-faq-97, Up: FAQ
759 To: daniel@synchrods.synchrods.COM (Daniel Senderowicz)
761 In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST.
762 Date: Tue, 23 Nov 1999 15:54:30 PST
763 From: Vern Paxson <vern>
765 Well, your problem is the
767 switch (yybgin-yysvec-1) { /* witchcraft */
769 at the beginning of lex rules. "witchcraft" == "non-portable". It's
770 assuming knowledge of the AT&T lex's internal variables.
772 For flex, you can probably do the equivalent using a switch on YYSTATE.
777 File: flex.info, Node: unnamed-faq-99, Next: unnamed-faq-100, Prev: unnamed-faq-98, Up: FAQ
783 To: archow@hss.hns.com
784 Subject: Re: Regarding distribution of flex and yacc based grammars
785 In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530.
786 Date: Wed, 22 Dec 1999 01:56:24 PST
787 From: Vern Paxson <vern>
789 > When we provide the customer with an object code distribution, is it
790 > necessary for us to provide source
791 > for the generated C files from flex and bison since they are generated by
794 For flex, no. I don't know what the current state of this is for bison.
796 > Also, is there any requrirement for us to neccessarily provide source for
797 > the grammar files which are fed into flex and bison ?
801 See the file "COPYING" in the flex distribution for the legalese.
806 File: flex.info, Node: unnamed-faq-100, Next: unnamed-faq-101, Prev: unnamed-faq-99, Up: FAQ
812 To: Martin Gallwey <gallweym@hyperion.moe.ul.ie>
813 Subject: Re: Flex, and self referencing rules
814 In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST.
815 Date: Sat, 19 Feb 2000 18:33:16 PST
816 From: Vern Paxson <vern>
818 > However, I do not use unput anywhere. I do use self-referencing
821 > UnaryExpr ({UnionExpr})|("-"{UnaryExpr})
823 You can't do this - flex is *not* a parser like yacc (which does indeed
824 allow recursion), it is a scanner that's confined to regular expressions.
829 File: flex.info, Node: unnamed-faq-101, Next: What is the difference between YYLEX_PARAM and YY_DECL?, Prev: unnamed-faq-100, Up: FAQ
835 To: slg3@lehigh.edu (SAMUEL L. GULDEN)
836 Subject: Re: Flex problem
837 In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST.
838 Date: Thu, 02 Mar 2000 23:00:46 PST
839 From: Vern Paxson <vern>
841 If this is exactly your program:
845 > whitespace [ \t\n]+
848 > "[" { printf("open_brac\n");}
849 > "]" { printf("close_brac\n");}
850 > "+" { printf("addop\n");}
851 > "*" { printf("multop\n");}
852 > {digits} { printf("NUMBER = %s\n", yytext);}
855 then the problem is that the last rule needs to be "{whitespace}" !
860 File: flex.info, Node: What is the difference between YYLEX_PARAM and YY_DECL?, Next: Why do I get "conflicting types for yylex" error?, Prev: unnamed-faq-101, Up: FAQ
862 What is the difference between YYLEX_PARAM and YY_DECL?
863 =======================================================
865 YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to
866 pass extra params when it calls yylex() from the parser.
868 YY_DECL is the Flex declaration of yylex. The default is similar to
872 #define int yy_lex ()
875 File: flex.info, Node: Why do I get "conflicting types for yylex" error?, Next: How do I access the values set in a Flex action from within a Bison action?, Prev: What is the difference between YYLEX_PARAM and YY_DECL?, Up: FAQ
877 Why do I get "conflicting types for yylex" error?
878 =================================================
880 This is a compiler error regarding a generated Bison parser, not a
881 Flex scanner. It means you need a prototype of yylex() in the top of
882 the Bison file. Be sure the prototype matches YY_DECL.
885 File: flex.info, Node: How do I access the values set in a Flex action from within a Bison action?, Prev: Why do I get "conflicting types for yylex" error?, Up: FAQ
887 How do I access the values set in a Flex action from within a Bison action?
888 ===========================================================================
890 With $1, $2, $3, etc. These are called "Semantic Values" in the
891 Bison manual. See *Note Top: (bison)Top.
894 File: flex.info, Node: Appendices, Next: Indices, Prev: FAQ, Up: Top
901 * Makefiles and Flex::
906 File: flex.info, Node: Makefiles and Flex, Next: Bison Bridge, Prev: Appendices, Up: Appendices
911 In this appendix, we provide tips for writing Makefiles to build
914 In a traditional build environment, we say that the `.c' files are
915 the sources, and the `.o' files are the intermediate files. When using
916 `flex', however, the `.l' files are the sources, and the generated `.c'
917 files (along with the `.o' files) are the intermediate files. This
918 requires you to carefully plan your Makefile.
920 Modern `make' programs understand that `foo.l' is intended to
921 generate `lex.yy.c' or `foo.c', and will behave accordingly(1). The
922 following Makefile does not explicitly instruct `make' how to build
923 `foo.c' from `foo.l'. Instead, it relies on the implicit rules of the
924 `make' program to build the intermediate file, `scan.c':
927 # Basic Makefile -- relies on implicit rules
928 # Creates "myprogram" from "scan.l" and "myprogram.c"
931 myprogram: scan.o myprogram.o
934 For simple cases, the above may be sufficient. For other cases, you
935 may have to explicitly instruct `make' how to build your scanner. The
936 following is an example of a Makefile containing explicit rules:
939 # Basic Makefile -- provides explicit rules
940 # Creates "myprogram" from "scan.l" and "myprogram.c"
943 myprogram: scan.o myprogram.o
944 $(CC) -o $@ $(LDFLAGS) $^
946 myprogram.o: myprogram.c
947 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
950 $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
953 $(LEX) $(LFLAGS) -o $@ $^
958 Notice in the above example that `scan.c' is in the `clean' target.
959 This is because we consider the file `scan.c' to be an intermediate
962 Finally, we provide a realistic example of a `flex' scanner used
963 with a `bison' parser(2). There is a tricky problem we have to deal
964 with. Since a `flex' scanner will typically include a header file
965 (e.g., `y.tab.h') generated by the parser, we need to be sure that the
966 header file is generated BEFORE the scanner is compiled. We handle this
967 case in the following example:
970 # Makefile example -- scanner and parser.
971 # Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c"
976 objects = scan.o parse.o myprogram.o
978 myprogram: $(objects)
979 scan.o: scan.l parse.c
981 myprogram.o: myprogram.c
983 In the above example, notice the line,
986 scan.o: scan.l parse.c
988 , which lists the file `parse.c' (the generated parser) as a
989 dependency of `scan.o'. We want to ensure that the parser is created
990 before the scanner is compiled, and the above line seems to do the
991 trick. Feel free to experiment with your specific implementation of
994 For more details on writing Makefiles, see *Note Top: (make)Top.
996 ---------- Footnotes ----------
998 (1) GNU `make' and GNU `automake' are two such programs that provide
999 implicit rules for flex-generated scanners.
1001 (2) This example also applies to yacc parsers.
1004 File: flex.info, Node: Bison Bridge, Next: M4 Dependency, Prev: Makefiles and Flex, Up: Appendices
1006 C Scanners with Bison Parsers
1007 =============================
1009 This section describes the `flex' features useful when integrating
1010 `flex' with `GNU bison'(1). Skip this section if you are not using
1011 `bison' with your scanner. Here we discuss only the `flex' half of the
1012 `flex' and `bison' pair. We do not discuss `bison' in any detail. For
1013 more information about generating `bison' parsers, see *Note Top:
1016 A compatible `bison' scanner is generated by declaring `%option
1017 bison-bridge' or by supplying `--bison-bridge' when invoking `flex'
1018 from the command line. This instructs `flex' that the macro `yylval'
1019 may be used. The data type for `yylval', `YYSTYPE', is typically
1020 defined in a header file, included in section 1 of the `flex' input
1021 file. For a list of functions and macros available, *Note
1024 The declaration of yylex becomes,
1027 int yylex ( YYSTYPE * lvalp, yyscan_t scanner );
1029 If `%option bison-locations' is specified, then the declaration
1033 int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner );
1035 Note that the macros `yylval' and `yylloc' evaluate to pointers.
1036 Support for `yylloc' is optional in `bison', so it is optional in
1037 `flex' as well. The following is an example of a `flex' scanner that is
1038 compatible with `bison'.
1041 /* Scanner for "C" assignment statements... sort of. */
1043 #include "y.tab.h" /* Generated by bison. */
1046 %option bison-bridge bison-locations
1049 [[:digit:]]+ { yylval->num = atoi(yytext); return NUMBER;}
1050 [[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;}
1051 "="|";" { return yytext[0];}
1055 As you can see, there really is no magic here. We just use `yylval'
1056 as we would any other variable. The data type of `yylval' is generated
1057 by `bison', and included in the file `y.tab.h'. Here is the
1058 corresponding `bison' parser:
1061 /* Parser to convert "C" assignments to lisp. */
1063 /* Pass the argument to yyparse through to yylex. */
1064 #define YYPARSE_PARAM scanner
1065 #define YYLEX_PARAM scanner
1077 STRING '=' NUMBER ';' {
1078 printf( "(setf %s %d)", $1, $3 );
1082 ---------- Footnotes ----------
1084 (1) The features described here are purely optional, and are by no
1085 means the only way to use flex with bison. We merely provide some glue
1086 to ease development of your parser-scanner pair.
1089 File: flex.info, Node: M4 Dependency, Prev: Bison Bridge, Up: Appendices
1094 The macro processor `m4'(1) must be installed wherever flex is
1095 installed. `flex' invokes `m4', found by searching the directories in
1096 the `PATH' environment variable. Any code you place in section 1 or in
1097 the actions will be sent through m4. Please follow these rules to
1098 protect your code from unwanted `m4' processing.
1100 * Do not use symbols that begin with, `m4_', such as, `m4_define',
1101 or `m4_include', since those are reserved for `m4' macro names. If
1102 for some reason you need m4_ as a prefix, use a preprocessor
1103 #define to get your symbol past m4 unmangled.
1105 * Do not use the strings `[[' or `]]' anywhere in your code. The
1106 former is not valid in C, except within comments and strings, but
1107 the latter is valid in code such as `x[y[z]]'. The solution is
1108 simple. To get the literal string `"]]"', use `"]""]"'. To get the
1109 array notation `x[y[z]]', use `x[y[z] ]'.
1112 `m4' is only required at the time you run `flex'. The generated
1113 scanner is ordinary C or C++, and does _not_ require `m4'.
1115 ---------- Footnotes ----------
1117 (1) The use of m4 is subject to change in future revisions of flex.
1118 It is not part of the public API of flex. Do not depend on it.
1121 File: flex.info, Node: Indices, Prev: Appendices, Up: Top
1129 * Index of Functions and Macros::
1130 * Index of Variables::
1131 * Index of Data Types::
1133 * Index of Scanner Options::