7 :Contact: milde@users.berlios.de
9 :Copyright: © 2007, 2009 G. Milde,
10 Released without warranties or conditions of any kind
11 under the terms of the Apache License, Version 2.0
12 http://www.apache.org/licenses/LICENSE-2.0
13 :Abstract: Proposal to add syntax highlight of code blocks to the
14 capabilities of Docutils_.
19 Syntax highlighting significantly enhances the readability of code. However,
20 in the current version, docutils does not highlight literal blocks.
22 This sandbox project aims to add syntax highlight of code blocks to the
23 capabilities of docutils. To find its way into the docutils core, it should
24 meet the requirements laid out in a mail on `Questions about writing
25 programming manuals and scientific documents`__, by docutils main developer
28 I'd be happy to include Python source colouring support, and other
29 languages would be welcome too. A multi-language solution would be
30 useful, of course. My issue is providing support for all output formats
31 -- HTML and LaTeX and XML and anything in the future -- simultaneously.
32 Just HTML isn't good enough. Until there is a generic-output solution,
33 this will be something users will have to put together themselves.
35 __ http://sourceforge.net/mailarchive/message.php?msg_id=12921194
37 Some older ideas are gathered in Docutils TODO_ document.
39 .. _TODO: ../../../docutils/docs/dev/todo.html#colorize-python
44 There are already docutils extensions providing syntax colouring, e.g:
47 Since Docutils 0.5, the "latex2e" writer supports syntax highlight of
48 literal blocks via the `listings` package with the
49 ``--literal-block-env=lstlistings`` option. You need to provide a custom
50 style sheet. The stylesheets_ repository provides two LaTeX style sheets
51 for highlighting literal-blocks with "listings".
53 Odtwriter_, experimental writer for Docutils OpenOffice export supports syntax
54 colours using Pygments_. See also the (outdated) section `Odtwriter syntax`_.
57 is a generic syntax highlighter written completely in Python.
59 * Usable as a command-line tool and as a Python package.
60 * Supports about 200 `languages and markup formats`_ (version 1.4).
61 * Already used by the odtwriter_ and Sphinx.
62 * Support for new languages, formats, and styles is added easily (modular
63 structure, Python code, existing documentation).
64 * Well documented and actively maintained.
65 * The web site provides a recipe for `using Pygments in ReST documents`_
66 (used in the legacy `Pygments enhanced docutils front-ends`_).
69 the "site builder" provides the `colorize`__ macro (using the
70 `Moin-Moin Python colorizer`_)
72 __ http://www.voidspace.org.uk/python/rest2web/macros.html#colorize
75 a C++ library and Python extension that can provide lexical
76 analysis for over 20 different programming languages. A recipe__ for a
77 "code-block" directive provides syntax highlight by SilverCity.
79 __ http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252170
82 features automatic highlighting using the Pygments_ highlighter.
83 It introduces the custom directives
85 :code-block: similar to the proposal below,
86 :sourcecode: an alias to "code-block", and
87 :highlight: configre highlight of "literal blocks".
89 (see http://sphinx.pocoo.org/markup/code.html).
92 has `reStructuredText support`__ and offers syntax highlighting with
93 a "code-block" directive using GNU Enscript_, SilverCity_, or Pygments_.
95 __ http://trac.edgewall.org/wiki/WikiRestructuredText
101 On 2009-02-20, David Goodger wrote in docutils-devel
103 I'd like to see the extensions implemented in Bruce and Sphinx etc.
104 folded back into core Docutils eventually. Otherwise we'll end up with
105 incompatible systems.
107 Pygments_ seems to be the most promising Docutils highlighter.
109 For printed output and PDFs via LaTeX, the listings_ package is a viable
113 Pygments enhanced docutils front-ends
114 -------------------------------------
116 Syntax highlight can be achieved by `front-end scripts`_ combining docutils and
119 "something users [will have to] put together themselves"
122 + Easy implementation with no changes to the stock docutils_.
123 + Separation of code blocks and ordinary literal blocks.
126 1. "code-block" content is formatted by `pygments`_ and inserted in the
127 document tree as a "raw" node making the approach writer-dependant.
128 2. documents are incompatible with the standard docutils because of the
129 locally defined directive.
130 3. more "invasive" markup distracting from content
131 (no "minimal" code block marker -- three additional lines per code block)
134 Point 1 and 2 lead to the `code-block directive proposal`_.
136 Point 3 becomes an issue in software documentation and literate programming
137 where a code block is the most used block markup. It is addressed in the
138 proposal for a `configurable literal block directive`_).
141 `code-block` directive proposal
142 -------------------------------
147 .. note:: This is the first draft for a reStructuredText definition,
148 analogue to other directives in ``directives.txt``.
150 :Directive Type: "code"
151 :Doctree Element: literal_block
152 :Directive Arguments: One (`language`), optional.
153 :Directive Options: name, class, number-lines.
154 :Directive Content: Becomes the body of the literal block.
156 The "code-block" directive constructs a literal block where the content is
157 parsed as source code and syntax highlight rules for `language` are applied.
158 If syntax rules for `language` are not known to Docutils, a warning is
159 issued and the content is rendered as ordinary literal block with
160 additional class arguments: "code" and the value of `language`.
162 :number-lines: let pygments include line-numbers
165 The following options are recognized:
167 ``number-lines`` : [start line number]
168 Precede every code line with a line number.
169 The optional argument is the number of the first line (defaut 1).
171 and the common options `:class:`_ and `:name:`_.
174 The content of the following directive ::
182 is parsed and marked up as Python source code. The actual rendering
183 depends on the style-sheet.
189 * Without language argument, the parsing step is skipped. Use cases:
191 * Mark a literal block as pseudo-code.
193 * Suppress warnings about a missing Pygments_ module or unknown languages.
195 * Do the parsing in the writer or output processor (e.g. LaTeX with
196 the listings_ package).
198 The language's name can be given as `class` option.
201 make the `language` argument compulsory and add a "no-highlight" option.
203 * TODO: Pygments_ provides filters like VisibleWhitespaceFilter
204 add options to use them?
208 Include directive option
209 """"""""""""""""""""""""
211 The include directive should get a matching new option:
214 The entire included text is inserted into the document as if it were the
215 content of a code-block directive (useful for program listings).
220 For inline code snippets, a `code` role should be implemented. Roles for
221 specific languages might be defined via the `role` directive based on the
230 Felix Wiemann provided a `proof of concept`_ script that utilizes the
231 pygments_ parser to parse a source code string and store the result in
234 This concept is used in a `pygments_code_block_directive`_ (Source:
235 `pygments_code_block_directive.py`_), to define and register a "code-block"
238 * The ``DocutilsInterface`` class uses pygments to parse the content of the
239 directive and classify the tokens using short CSS class names identical to
240 pygments HTML output. If pygments is not available, the unparsed code is
241 returned. TODO: issue a warning.
243 * The ``code_block_directive`` function inserts the tokens in a "rich"
244 <literal_block> element with "classified" <inline> nodes.
249 The writers can use the class information in the <inline> elements to render
250 the tokens. They should ignore the class information if they are unable to
251 use it or to pass it on.
253 Running the test script `<../tools/test_pygments_code_block_directive.py>`_
254 produces example output for a set of writers.
257 The "html" writer works out of the box.
259 * The rst2html-highlight_ front end registers the "code-block" directive and
260 converts an input file to html.
262 * Styling is done with the adapted CSS style sheet `pygments-default.css`_
263 based on docutils' default stylesheet and the output of
264 ``pygmentize -S default -f html``.
266 The conversion of `<myfunction.py.txt>`_ looks like
267 `<myfunction.py.htm>`_.
269 The "s5" and "pep" writers are not tested yet.
272 "xml" and "pseudoxml" work out of the box.
274 The conversion of `myfunction.py.txt`_ looks like
275 `<myfunction.py.xml>`_ respective `<myfunction.py.pseudoxml>`_
278 "latex2e" (SVN version) works out of the box.
280 * A style file, e.g. `<pygments-docutilsroles.sty>`_, is required to actually
281 highlight the code in the output. (As with HTML, the pygments-produced
282 style file will not work with docutils' output.)
284 * Alternatively, the latex writer could reconstruct the original
285 content and pass it to a ``lstlistings`` environment.
287 TODO: This should be the default behaviour with
288 ``--literal-block-env=lstlistings``.
290 The LaTeX output of `myfunction.py.txt`_ looks like `<myfunction.py.tex>`_
291 and corresponding PDF like `<myfunction.py.pdf>`_.
294 The `odtwriter` provides syntax highlight with pygments but uses a
295 different syntax and implementation.
301 1. Minimal implementation:
303 * move the code from `pygments_code_block_directive.py`_ to "the right
306 * add the CSS rules to the default style-sheet (see pygments-default.css_)
308 * provide a LaTeX style.
310 2. Write functional test case and sample.
312 3. Think about an interface for pygments' options (like "encoding" or
316 Configurable literal block directive
317 ------------------------------------
322 A clean and simple syntax for highlighted code blocks -- preserving the
323 space saving feature of the "minimised" literal block marker (``::`` at the
324 end of a text paragraph). This is especially desirable in documents with
325 many code blocks like tutorials or literate programs.
330 The *role* of inline `interpreted text` can be customised with the
331 "default-role" directive. This allows the use of the concise "backtick"
332 syntax for the most often used role, e.g. in a chemical paper, one could
335 .. default-role:: subscript
337 The triple point of H\ `2`\O is at 0°C.
339 .. default-role:: subscript
343 The triple point of H\ `2`\O is at 0°C.
345 This customisation is currently not possible for block markup.
350 * Define a new "literal-block" directive syntax for an ordinary literal
351 block. This would simply insert the block content into the document
352 tree as "literal-block" element.
354 * Define a "default-literal-block" setting that controls which
355 directive is called on a block following ``::``. Default would be the
356 "literal-block" directive (backwards compatible).
361 Analogue to customising the default role of "interpreted text" with the
362 "default-role" directive, the concise ``::`` literal-block markup could be
365 * a "code-block" directive for syntax highight
367 * the "line-block" directive for poems or addresses
369 * the "parsed-literal" directive
373 ordinary literal block::
375 some text typeset in monospace
377 .. default-literal-block:: code-block python
379 this is colourful Python code::
385 In the same line, a "default-block-quote" setting or directive could be
386 considered to configure the role of a block quote.
392 The content of this section relates to an old version of the
393 `odtwriter`. Things changed with the inclusion of the `odtwriter` into
396 This is only kept for historical reasons.
398 Dave Kuhlman's odtwriter_ extension can add syntax highlighting
399 to ordinary literal blocks.
401 The ``--add-syntax-highlighting`` command line flag activates syntax
402 highlighting in literal blocks. By default, the "python" lexer is used.
404 You can change this within your reST document with the `sourcecode`
409 ordinary literal block::
411 content set in teletype
414 .. sourcecode:: python
416 colourful Python code::
422 The "sourcecode" directive defined by the odtwriter is principally
423 different from the "code-block" directive of ``rst2html-pygments``:
425 * The odtwriter directive does not have content. It is a switch.
427 * The syntax highlighting state and language/lexer set by this directive
428 remain in effect until the next sourcecode directive is encountered in the
431 ``.. sourcecode:: <newstate>``
432 make highlighting active or inactive.
433 <newstate> is either ``on`` or ``off``.
435 ``.. sourcecode:: <lexer>``
436 change the lexer parsing literal code blocks.
437 <lexer> should be one of aliases listed at pygment's `languages and
440 I.e. the odtwriter implements a `configurable literal block directive`_
441 (but with a slightly different syntax than the proposal above).
445 .. _rest2web: http://www.voidspace.org.uk/python/rest2web/
446 .. _Enscript: http://www.gnu.org/software/enscript/enscript.html
447 .. _SilverCity: http://silvercity.sourceforge.net/
448 .. _Trac: http://trac.edgewall.org/
449 .. _Moin-Moin Python colorizer:
450 http://www.standards-schmandards.com/2005/fangs-093/
451 .. _odtwriter: http://www.rexx.com/~dkuhlman/odtwriter.html
452 .. _Sphinx: http://sphinx.pocoo.org
454 http://www.ctan.org/tex-archive/help/Catalogue/entries/listings.html
455 .. _PyLit: http://pylit.berlios.de
456 .. _PyLit Examples: http://pylit.berlios.de/examples/index.html#latex-packages
458 .. _Pygments: http://pygments.org/
459 .. _languages and markup formats: http://pygments.org/languages
460 .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/
462 .. _Docutils: http://docutils.sourceforge.net/
463 .. _Docutils Document Tree:
464 http://docutils.sf.net/docs/ref/doctree.html#classes
465 .. _latex-variants: http://docutils.sourceforge.net/sandbox/latex-variants/
466 .. _proof of concept:
467 http://article.gmane.org/gmane.text.docutils.user/3689
470 .. _front-end scripts: ../tools/pygments-enhanced-front-ends
471 .. _pygments-default.css: ../data/pygments-default.css
472 .. _pygments_code_block_directive.py: ../pygments_code_block_directive.py
473 .. _pygments_code_block_directive: pygments_code_block_directive-bunt.py.htm
474 .. _rst2html-highlight: ../rst2html-highlight
475 .. _pygments-long.css: ../data/pygments-long.css
476 .. _stylesheets: ../../stylesheets/