2 @setfilename ../../info/semantic
3 @set TITLE Semantic Manual
4 @set AUTHOR Eric M. Ludlam, David Ponce, and Richard Y. Kim
5 @settitle @value{TITLE}
7 @c *************************************************************************
9 @c *************************************************************************
11 @c Merge all indexes into a single index for now.
12 @c We can always separate them later into two or more as needed.
19 @c @footnotestyle separate
25 This manual documents the Semantic library and utilities.
27 Copyright @copyright{} 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007,
28 2009, 2010 Free Software Foundation, Inc.
31 Permission is granted to copy, distribute and/or modify this document
32 under the terms of the GNU Free Documentation License, Version 1.3 or
33 any later version published by the Free Software Foundation; with no
34 Invariant Sections, with the Front-Cover texts being ``A GNU Manual,''
35 and with the Back-Cover Texts as in (a) below. A copy of the license
36 is included in the section entitled ``GNU Free Documentation License.''
38 (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and
39 modify this GNU manual. Buying copies from the FSF supports it in
40 developing GNU and promoting software freedom.''
46 * Semantic: (semantic). Source code parser library and utilities.
50 @center @titlefont{Semantic}
52 @center by @value{AUTHOR}
65 @macro obsolete{old,new}
67 @strong{Compatibility}:
68 @code{\new\} introduced in @semantic{} version 2.0 supercedes
69 @code{\old\} which is now obsolete.
72 @c *************************************************************************
74 @c *************************************************************************
80 @semantic{} is a suite of Emacs libraries and utilities for parsing
81 source code. At its core is a lexical analyzer and two parser
82 generators (@code{bovinator} and @code{wisent}) written in Emacs Lisp.
83 @semantic{} provides a variety of tools for making use of the parser
84 output, including user commands for code navigation and completion, as
85 well as enhancements for imenu, speedbar, whichfunc, eldoc,
86 hippie-expand, and several other parts of Emacs.
88 To send bug reports, or participate in discussions about semantic,
89 use the mailing list cedet-semantic@@sourceforge.net via the URL:
90 @url{http://lists.sourceforge.net/lists/listinfo/cedet-semantic}
99 * Semantic Internals::
101 * GNU Free Documentation License::
106 @chapter Introduction
108 This chapter gives an overview of @semantic{} and its goals.
110 Ordinarily, Emacs uses regular expressions (and syntax tables) to
111 analyze source code for purposes such as syntax highlighting. This
112 approach, though simple and efficient, has its limitations: roughly
113 speaking, it only ``guesses'' the meaning of each piece of source code
114 in the context of the programming language, instead of rigorously
115 ``understanding'' it.
117 @semantic{} provides a new infrastructure to analyze source code using
118 @dfn{parsers} instead of regular expressions. It contains two
119 built-in parser generators (an @acronym{LL} generator named
120 @code{Bovine} and an @acronym{LALR} generator named @code{Wisent},
121 both written in Emacs Lisp), and parsers for several common
122 programming languages. It can also make use of @dfn{external
123 parsers}---programs such as GNU Global and GNU IDUtils.
125 @semantic{} provides a uniform, language-independent @acronym{API} for
126 accessing the parser output. This output can be used by other Emacs
127 Lisp programs to implement ``syntax-aware'' behavior. @semantic{}
128 itself includes several such utilities, including user-level Emacs
129 commands for navigating, searching, and completing source code.
131 The following diagram illustrates the structure of the @semantic{}
136 The words in all-capital are those that @semantic{} itself provides.
137 Others are current or future languages or applications that are not
138 distributed along with @semantic{}.
147 +---------------+ +--------+ +--------+
148 C --->| C PARSER |--->| | | |
149 +---------------+ | | | |
150 +---------------+ | COMMON | | COMMON |<--- SPEEDBAR
151 Java --->| JAVA PARSER |--->| PARSE | | |
152 +---------------+ | TREE | | PARSE |<--- SEMANTICDB
153 +---------------+ | FORMAT | | API |
154 Scheme --->| SCHEME PARSER |--->| | | |<--- ecb
155 +---------------+ | | | |
156 +---------------+ | | | |
157 Texinfo --->| TEXI. PARSER |--->| | | |
158 +---------------+ | | | |
162 +---------------+ | | | |
163 Lang. Y --->| Y Parser |--->| | | |<--- app. ?
164 +---------------+ | | | |
165 +---------------+ | | | |<--- app. ?
166 Lang. Z --->| Z Parser |--->| | | |
167 +---------------+ +--------+ +--------+
171 * Semantic Components::
174 @node Semantic Components
175 @section Semantic Components
177 In this section, we provide a more detailed description of the major
178 components of @semantic{}, and how they interact with one another.
180 The first step in parsing a source code file is to break it up into
181 its fundamental components. This step is called lexical analysis:
184 syntax table, keywords list, and options
188 input file ----> Lexer ----> token stream
192 The output of the lexical analyzer is a list of tokens that make up
193 the file. The next step is the actual parsing, shown below:
199 token stream ---> Parser ----> parse tree
203 The end result, the parse tree, is @semantic{}'s internal
204 representation of the language grammar. @semantic{} provides an
205 @acronym{API} for Emacs Lisp programs to access the parse tree.
207 Parsing large files can take several seconds or more. By default,
208 @semantic{} automatically caches parse trees by saving them in your
209 @file{.emacs.d} directory. When you revisit a previously-parsed file,
210 the parse tree is automatically reloaded from this cache, to save
211 time. @xref{SemanticDB}.
214 @chapter Using Semantic
216 @include sem-user.texi
218 @node Semantic Internals
219 @chapter Semantic Internals
221 This chapter provides an overview of the internals of @semantic{}.
222 This information is usually not needed by application developers or
223 grammar developers; it is useful mostly for the hackers who would like
224 to learn more about how @semantic{} works.
227 * Parser code :: Code used for the parsers
228 * Tag handling :: Code used for manipulating tags
229 * Semanticdb Internals :: Code used in the semantic database
230 * Analyzer Internals :: Code used in the code analyzer
231 * Tools :: Code used in user tools
232 * Tests :: Code used for testing
238 @semantic{} parsing code is spread across a range of files.
242 The core infrastructure sets up buffers for parsing, and has all the
243 core parsing routines. Most parsing routines are overloadable, so the
244 actual implementation may be somewhere else.
246 @item semantic-edit.el
247 Incremental reparse based on user edits.
249 @item semantic-grammar.el
250 @itemx semantic-grammar.wy
251 Parser for the different grammar languages, and a major mode for
252 editing grammars in Emacs.
254 @item semantic-lex.el
255 Infrastructure for implementing lexical analyzers. Provides macros
256 for creating individual analyzers for specific features, and a way to
257 combine them together.
259 @item semantic-lex-spp.el
260 Infrastructure for a lexical symbolic preprocessor. This was written
261 to implement the C preprocessor, but could be used for other lexical
264 @item bovine/bovine-grammar.el
265 @itemx bovine/bovine-grammar-macros.el
266 @itemx bovine/semantic-bovine.el
267 The ``bovine'' grammar. This is the first grammar mode written for
268 @semantic{} and is useful for simple creating simple parsers.
270 @item wisent/wisent.el
271 @itemx wisent/bison-wisent.el
272 @itemx wisent/semantic-wisent.el
273 @itemx wisent/semantic-debug-grammar.el
274 A port of bison to Emacs. This infrastructure lets you create LALR
275 based parsers for @semantic{}.
277 @item semantic-ast.el
278 Manage Abstract Syntax Trees for parsers.
280 @item semantic-debug.el
281 Infrastructure for debugging grammars.
283 @item semantic-util.el
284 Various utilities for manipulating tags, such as describing the tag
285 under point, adding labels, and the all important
286 @code{semantic-something-to-tag-table}.
291 @section Tag handling
293 A tag represents an individual item found in a buffer, such as a
294 function or variable. Tag handling is handled in several source
298 @item semantic-tag.el
299 Basic tag creation, queries, cloning, binding, and unbinding.
301 @item semantic-tag-write.el
302 Write a tag or tag list to a stream. These routines are used by
303 @file{semanticdb-file.el} when saving a list of tags.
305 @item semantic-tag-file.el
306 Files associated with tags. Goto-tag, file for include, and file for
309 @item semantic-tag-ls.el
310 Language dependant features of a tag, such as parent calculation, slot
311 protection, and other states like abstract, virtual, static, and leaf.
313 @item semantic-dep.el
314 Include file handling. Contains the include path concepts, and
315 routines for looking up file names in the include path.
317 @item semantic-format.el
318 Convert a tag into a nicely formatted and colored string. Use
319 @code{semantic-test-all-format-tag-functions} to test different output
322 @item semantic-find.el
323 Find tags matching different conditions in a tag table.
324 These routines are used by @file{semanticdb-find.el} once the database
325 has been converted into a simpler tag table.
327 @item semantic-sort.el
328 Sorting lists of tags in different ways. Includes sorting a plain
329 list of tags forward or backward. Includes binning tags based on
330 attributes (bucketize), and tag adoption for multiple references to
333 @item semantic-doc.el
334 Capture documentation comments from near a tag.
338 @node Semanticdb Internals
339 @section Semanticdb Internals
341 @acronym{Semanticdb} complexity is certainly an issue. It is a rather
342 hairy problem to try and solve.
346 Defines a @dfn{database} and a @dfn{table} base class. You can
347 instantiate these classes, and use them, but they are not persistent.
349 This file also provides support for @code{semanticdb-minor-mode},
350 which automatically associates files with tables in databases so that
351 tags are @emph{saved} while a buffer is not in memory.
353 The database and tables both also provide applicate cache information,
354 and cache flushing system. The semanticdb search routines use caches
355 to save datastructures that are complex to calculate.
357 Lastly, it provides the concept of @dfn{project root}. It is a system
358 by which a file can be associated with the root of a project, so if
359 you have a tree of directories and source files, it can find the root,
360 and allow a tag-search to span all available databases in that
363 @item semanticdb-file.el
364 Provides a subclass of the basic table so that it can be saved to
365 disk. Implements all the code needed to unbind/rebind tags to a
366 buffer and writing them to a file.
368 @item semanticdb-el.el
369 Implements a special kind of @dfn{system} database that uses Emacs
370 internals to perform queries.
372 @item semanticdb-ebrowse.el
373 Implements a system database that uses Ebrowse to parse files into a
374 table that can be queried for tag names. Successful tag hits during a
375 find causes @semantic{} to pick up and parse the reference files to
376 get the full details.
378 @item semanticdb-find.el
379 Infrastructure for searching groups @semantic{} databases, and dealing
380 with the search results format.
382 @item semanticdb-ref.el
383 Tracks crossreferences. Cross references are needed when buffer is
384 reparsed, and must alert other tables that any dependant caches may
385 need to be flushed. References are in the form of include files.
389 @node Analyzer Internals
390 @section Analyzer Internals
392 The @semantic{} analyzer is a complex engine which has been broken
393 down across several modules. When the @semantic{} analyzer fails,
394 start with @code{semantic-analyze-debug-assist}, then dive into some
398 @item semantic-analyze.el
399 The core analyzer for defining the @dfn{current context}. The
400 current context is an object that contains references to aspects of
401 the local context including the current prefix, and a tag list
402 defining what the prefix means.
404 @item semantic-analyze-complete.el
405 Provides @code{semantic-analyze-possible-completions}.
407 @item semantic-analyze-debug.el
408 The analyzer debugger. Useful when attempting to get everything
411 @item semantic-analyze-fcn.el
412 Various support functions needed by the analyzer.
414 @item semantic-ctxt.el
415 Local context parser. Contains overloadable functions used to move
416 around through different scopes, get local variables, and collect the
417 current prefix used when doing completion.
419 @item semantic-scope.el
420 Calculate @dfn{scope} for a location in a buffer. The scope includes
421 local variables, and tag lists in scope for various reasons, such as
422 C++ using statements.
424 @item semanticdb-typecache.el
425 The typecache is part of @code{semanticdb}, but is used primarilly by
426 the analyzer to look up datatypes and complex names. The typecache is
427 bound across source files and builds a master lookup table for data
431 Interactive Analyzer functions. Simple routines that do completion or
432 lookups based on the results from the Analyzer. These routines are
433 meant as examples for application writers, but are quite useful as
436 @item semantic-ia-sb.el
437 Speedbar support for the analyzer, displaying context info, and
445 These files contain various tools a user can use.
448 @item semantic-idle.el
449 Idle scheduler for @semantic{}. Manages reparsing buffers after
450 edits, and large work tasks in idle time. Includes modes for showing
451 summary help and pop-up completion.
454 The @semantic{} navigator. Provides many ways to move through a
455 buffer based on the active tag table.
457 @item semantic-decorate.el
458 A minor mode for decorating tags based on details from the parser.
459 Includes overlines for functions, or coloring class fields based on
462 @item semantic-decorate-include.el
463 A decoration mode for include files, which assists users in setting up
464 parsing for their includes.
466 @item semantic-complete.el
467 Advanced completion prompts for reading tag names in the minibuffer, or
470 @item semantic-imenu.el
471 Imenu support for using @semantic{} tags in imenu.
473 @item semantic-mru-bookmark.el
474 Automatic bookmarking based on tags. Jump to locations you've been
475 before based on tag name.
478 Support for @semantic{} tag usage in Speedbar.
480 @item semantic-util-modes.el
481 A bunch of small minor-modes that exposes aspects of the semantic
482 parser state. Includes @code{semantic-stickyfunc-mode}.
485 @itemx document-vars.el
486 Create an update comments for tags.
488 @item semantic-adebug.el
489 Extensions of @file{data-debug.el} for @semantic{}.
491 @item semantic-chart.el
492 Draw some charts from stats generated from parsing.
495 @item semantic-elp.el
496 Profiler for helping to optimize the @semantic{} analyzer.
506 @item semantic-utest.el
507 Basic testing of parsing and incremental parsing for most supported
510 @item semantic-ia-utest.el
511 Test the semantic analyzer's ability to provide smart completions.
513 @item semantic-utest-c.el
514 Tests for the C parser's lexical pre-processor.
516 @item semantic-regtest.el
517 Regression tests from the older Semantic 1.x API.
526 In semantic 1.4, a BNF file represented ``Bovine Normal Form'', the
527 grammar file used for the 1.4 parser generator. This was a play on
528 Backus-Naur Form which proved too confusing.
531 A verb representing what happens when a bovine parser parses a file.
534 In a bovine, or LL parser, the bovine lambda is a function to execute
535 when a specific set of match rules has succeeded in matching text from
539 A parser using the bovine parser generator. It is an LL parser
540 suitible for small simple languages.
547 A program which converts text into a stream of tokens by analyzing
548 them lexically. Lexers will commonly create strings, symbols,
549 keywords and punctuation, and strip whitespaces and comments.
554 A nonterminal symbol or simply a nonterminal stands for a class of
555 syntactically equivalent groupings. A nonterminal symbol name is used
556 in writing grammar rules.
559 Some functions are defined via @code{define-overload}.
560 These can be overloaded via ....
563 A program that converts @b{tokens} to @b{tags}.
566 A tag is a representation of some entity in a language file, such as a
567 function, variable, or include statement. In semantic, the word tag is
568 used the same way it is used for the etags or ctags tools.
570 A tag is usually bound to a buffer region via overlay, or it just
571 specifies character locations in a file.
574 A single atomic item returned from a lexer. It represents some set
575 of characters found in a buffer.
578 The output of the lexer as well as the input to the parser.
581 A parser using the wisent parser generator. It is a port of bison to
582 Emacs Lisp. It is an LALR parser suitable for complex languages.
586 @node GNU Free Documentation License
587 @appendix GNU Free Documentation License
588 @include doclicense.texi
601 @c Following comments are for the benefit of ispell.
603 @c LocalWords: alist API APIs arg argc args argv asis assoc autoload Wisent
604 @c LocalWords: backquote bnf bovinate bovinates LALR
605 @c LocalWords: bovinating bovination bovinator bucketize
606 @c LocalWords: cb cdr charquote checkcache cindex CLOS
607 @c LocalWords: concat concocting const constantness ctxt Decl defcustom
608 @c LocalWords: deffn deffnx defun defvar destructor's dfn diff dir
609 @c LocalWords: doc docstring EDE EIEIO elisp emacsman emph enum
610 @c LocalWords: eq Exp EXPANDFULL expresssion fn foo func funcall
611 @c LocalWords: ia ids iff ifinfo imenu imenus init int isearch itemx java kbd
612 @c LocalWords: keymap keywordtable lang languagemode lexer lexing Ludlam
613 @c LocalWords: menubar metaparent metaparents min minibuffer Misc mode's
614 @c LocalWords: multitable NAvigaTOR noindent nomedian nonterm noselect
615 @c LocalWords: nosnarf obarray OLE OO outputfile paren parsetable POINT's
616 @c LocalWords: popup positionalonly positiononly positionormarker pre
617 @c LocalWords: printf printindex Programmatically pt punctuations quotemode
618 @c LocalWords: ref regex regexp Regexps reparse resetfile samp sb
619 @c LocalWords: scopestart SEmantic semanticdb setfilename setq
620 @c LocalWords: settitle setupfunction sexp sp SPC speedbar speedbar's
621 @c LocalWords: streamorbuffer struct subalist submenu submenus
622 @c LocalWords: subsubsection sw sym texi texinfo titlefont titlepage
623 @c LocalWords: tok TOKEN's toplevel typemodifiers uml unset untar
624 @c LocalWords: uref usedb var vskip xref yak
627 arch-tag: cbc6e78c-4ff1-410e-9fc7-936487e39bbf