7 :Contact: milde@users.berlios.de
9 :Copyright: © 2007, 2009 G. Milde,
10 Released without warranties or conditions of any kind
11 under the terms of the Apache License, Version 2.0
12 http://www.apache.org/licenses/LICENSE-2.0
13 :Abstract: Proposal to add syntax highlight of code blocks to the
14 capabilities of Docutils_.
19 Syntax highlighting significantly enhances the readability of code. However,
20 in the current version, docutils does not highlight literal blocks.
22 This sandbox project aims to add syntax highlight of code blocks to the
23 capabilities of docutils. To find its way into the docutils core, it should
24 meet the requirements laid out in a mail on `Questions about writing
25 programming manuals and scientific documents`__, by docutils main developer
28 I'd be happy to include Python source colouring support, and other
29 languages would be welcome too. A multi-language solution would be
30 useful, of course. My issue is providing support for all output formats
31 -- HTML and LaTeX and XML and anything in the future -- simultaneously.
32 Just HTML isn't good enough. Until there is a generic-output solution,
33 this will be something users will have to put together themselves.
35 __ http://sourceforge.net/mailarchive/message.php?msg_id=12921194
41 There are already docutils extensions providing syntax colouring, e.g:
44 a C++ library and Python extension that can provide lexical
45 analysis for over 20 different programming languages. A recipe__ for a
46 "code-block" directive provides syntax highlight by SilverCity.
48 __ http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252170
51 a LaTeX package providing highly customisable and advanced syntax
52 highlight, though only for LaTeX (and LaTeX derived PS|PDF).
54 Since Docutils 0.5, the "latex2e" writer supports syntax highlight of
55 literal blocks by `listings` with the ``--literal-block-env=lstlistings``
56 option. You need to provide a custom style sheet. The stylesheets_
57 repository provides two LaTeX style sheets for highlighting literal-blocks
61 has `reStructuredText support`__ and offers syntax highlighting with
62 a "code-block" directive using GNU Enscript_, SilverCity_, or Pygments_.
64 __ http://trac.edgewall.org/wiki/WikiRestructuredText
67 the "site builder" provides the `colorize`__ macro (using the
68 `Moin-Moin Python colorizer`_)
70 __ http://www.voidspace.org.uk/python/rest2web/macros.html#colorize
74 features automatic highlighting using the Pygments_ highlighter.
75 It introduces the custom directives
77 :code-block: similar to the proposal below,
78 :sourcecode: an alias to "code-block", and
79 :highlight: configre highlight of "literal blocks".
81 (see http://sphinx.pocoo.org/markup/code.html).
84 is a generic syntax highlighter written completely in Python.
86 * Usable as a command-line tool and as a Python package.
87 * A wide range of common `languages and markup formats`_ is supported.
88 * Additionally, OpenOffice's ``*.odt`` is supported by the odtwriter_.
89 * The layout is configurable by style sheets.
90 * Several built-in styles and an option for line-numbering.
91 * Built-in output formats include HTML, LaTeX, rtf
92 * Support for new languages, formats, and styles is added easily (modular
93 structure, Python code, existing documentation).
94 * Well documented and actively maintained.
95 * The web site provides a recipe for `using Pygments in ReST documents`_.
96 It is used in the `Pygments enhanced docutils front-ends`_ below.
98 Odtwriter_, experimental writer for Docutils OpenOffice export supports syntax
99 colours using Pygments_. (See the (outdated) section `Odtwriter syntax`_.)
104 Pygments_ seems to be the most promising Docutils highlighter.
106 For printed output and PDFs via LaTeX, the listings_ package is a viable
110 Pygments enhanced docutils front-ends
111 -------------------------------------
113 Syntax highlight can be achieved by `front-end scripts`_ combining docutils and
116 "something users [will have to] put together themselves"
119 + Easy implementation with no changes to the stock docutils_.
120 + Separation of code blocks and ordinary literal blocks.
123 1. "code-block" content is formatted by `pygments`_ and inserted in the
124 document tree as a "raw" node making the approach writer-dependant.
125 2. documents are incompatible with the standard docutils because of the
126 locally defined directive.
127 3. more "invasive" markup distracting from content
128 (no "minimal" code block marker -- three additional lines per code block)
131 Point 1 and 2 lead to the `code-block directive proposal`_.
133 Point 3 becomes an issue in literate programming where a code block is
134 the most used block markup. It is addressed in the proposal for a
135 `configurable literal block directive`_).
138 `code-block` directive proposal
139 -------------------------------
144 .. note:: This is the first draft for a reStructuredText definition,
145 analogue to other directives in ``directives.txt``.
147 :Directive Type: "code-block"
148 :Doctree Element: literal_block
149 :Directive Arguments: One (`language`) or more (class names), optional.
150 :Directive Options: None.
151 :Directive Content: Becomes the body of the literal block.
153 The "code-block" directive constructs a literal block where the content is
154 parsed as source code and syntax highlight rules for `language` are
155 applied. If syntax rules for `language` are not known to Docutils, it
156 is rendered like an ordinary literal block.
159 A bit of Python code ::
161 .. code-block:: python
167 The directive content will be parsed and marked up as Python source
168 code. The actual rendering depends on the style-sheet.
172 * If the language argument is missing, a (configurable) default language
175 * Additional arguments might be defined and passed to the pygments_
176 parser or the output document (as class arguments), e.g.
178 :number-lines: let pygments include line-numbers
181 Include directive option
182 """"""""""""""""""""""""
184 The include directive should get a matching new option:
187 The entire included text is inserted into the document as if it were the
188 content of a code-block directive (useful for program listings).
196 Felix Wiemann provided a `proof of concept`_ script that utilizes the
197 pygments_ parser to parse a source code string and store the result in
200 This concept is used in a `pygments_code_block_directive`_ (Source:
201 `pygments_code_block_directive.py`_), to define and register a "code-block"
204 * The ``DocutilsInterface`` class uses pygments to parse the content of the
205 directive and classify the tokens using short CSS class names identical to
206 pygments HTML output. If pygments is not available, the unparsed code is
209 * The ``code_block_directive`` function inserts the tokens in a "rich"
210 <literal_block> element with "classified" <inline> nodes.
215 The writers can use the class information in the <inline> elements to render
216 the tokens. They should ignore the class information if they are unable to
217 use it or to pass it on.
219 Running the test script `<../tools/test_pygments_code_block_directive.py>`_
220 produces example output for a set of writers.
223 The "html" writer works out of the box.
225 * The rst2html-highlight_ front end registers the "code-block" directive and
226 converts an input file to html.
228 * Styling is done with the adapted CSS style sheet `pygments-default.css`_
229 based on docutils' default stylesheet and the output of
230 ``pygmentize -S default -f html``.
232 The conversion of `<myfunction.py.txt>`_ looks like
233 `<myfunction.py.htm>`_.
235 The "s5" and "pep" writers are not tested yet.
238 "xml" and "pseudoxml" work out of the box.
240 The conversion of `myfunction.py.txt`_ looks like
241 `<myfunction.py.xml>`_ respective `<myfunction.py.pseudoxml>`_
244 "latex2e" (SVN version) works out of the box.
246 * A style file, e.g. `<pygments-docutilsroles.sty>`_, is required to actually
247 highlight the code in the output. (As with HTML, the pygments-produced
248 style file will not work with docutils' output.)
250 * Alternatively, the latex writer could reconstruct the original
251 content and pass it to a ``lstlistings`` environment.
253 TODO: This should be the default behaviour with
254 ``--literal-block-env=lstlistings``.
256 The LaTeX output of `myfunction.py.txt`_ looks like `<myfunction.py.tex>`_
257 and corresponding PDF like `<myfunction.py.pdf>`_.
260 The sandbox project `odtwriter` provided syntax highlight with
261 pygments but used a different syntax and implementation.
263 (What is the status of the `odtwriter` now included in the
264 standard distribution?)
270 1. Minimal implementation:
272 * move the code from `pygments_code_block_directive.py`_ to "the right
275 * add the CSS rules to the default style-sheet (see pygments-default.css_)
277 * provide a LaTeX style.
279 2. Write functional test case and sample.
282 3. Think about an interface for pygments' options (like "encoding" or
286 Configurable literal block directive
287 ------------------------------------
292 A clean and simple syntax for highlighted code blocks -- preserving the
293 space saving feature of the "minimised" literal block marker (``::`` at the
294 end of a text paragraph). This is especially desirable in documents with
295 many code blocks like tutorials or literate programs.
300 The *role* of inline `interpreted text` can be customised with the
301 "default-role" directive. This allows the use of the concise "backtick"
302 syntax for the most often used role, e.g. in a chemical paper, one could
305 .. default-role:: subscript
307 The triple point of H\ `2`\O is at 0°C.
309 .. default-role:: subscript
313 The triple point of H\ `2`\O is at 0°C.
315 This customisation is currently not possible for block markup.
320 * Define a new "literal-block" directive syntax for an ordinary literal
321 block. This would simply insert the block content into the document
322 tree as "literal-block" element.
324 * Define a "default-literal-block" setting that controls which
325 directive is called on a block following ``::``. Default would be the
326 "literal-block" directive (backwards compatible).
331 Analogue to customising the default role of "interpreted text" with the
332 "default-role" directive, the concise ``::`` literal-block markup could be
335 * a "code-block" directive for syntax highight
337 * the "line-block" directive for poems or addresses
339 * the "parsed-literal" directive
343 ordinary literal block::
345 some text typeset in monospace
347 .. default-literal-block:: code-block python
349 this is colourful Python code::
355 In the same line, a "default-block-quote" setting or directive could be
356 considered to configure the role of a block quote.
362 The content of this section relates to an old version of the
363 `odtwriter`. Things changed with the inclusion of the `odtwriter` into
366 This is only kept for historical reasons.
368 Dave Kuhlman's odtwriter_ extension can add syntax highlighting
369 to ordinary literal blocks.
371 The ``--add-syntax-highlighting`` command line flag activates syntax
372 highlighting in literal blocks. By default, the "python" lexer is used.
374 You can change this within your reST document with the `sourcecode`
379 ordinary literal block::
381 content set in teletype
384 .. sourcecode:: python
386 colourful Python code::
392 The "sourcecode" directive defined by the odtwriter is principally
393 different from the "code-block" directive of ``rst2html-pygments``:
395 * The odtwriter directive does not have content. It is a switch.
397 * The syntax highlighting state and language/lexer set by this directive
398 remain in effect until the next sourcecode directive is encountered in the
401 ``.. sourcecode:: <newstate>``
402 make highlighting active or inactive.
403 <newstate> is either ``on`` or ``off``.
405 ``.. sourcecode:: <lexer>``
406 change the lexer parsing literal code blocks.
407 <lexer> should be one of aliases listed at pygment's `languages and
410 I.e. the odtwriter implements a `configurable literal block directive`_
411 (but with a slightly different syntax than the proposal above).
415 .. _rest2web: http://www.voidspace.org.uk/python/rest2web/
416 .. _Enscript: http://www.gnu.org/software/enscript/enscript.html
417 .. _SilverCity: http://silvercity.sourceforge.net/
418 .. _Trac: http://trac.edgewall.org/
419 .. _Moin-Moin Python colorizer:
420 http://www.standards-schmandards.com/2005/fangs-093/
421 .. _odtwriter: http://www.rexx.com/~dkuhlman/odtwriter.html
422 .. _Sphinx: http://sphinx.pocoo.org
424 http://www.ctan.org/tex-archive/help/Catalogue/entries/listings.html
425 .. _PyLit: http://pylit.berlios.de
426 .. _PyLit Examples: http://pylit.berlios.de/examples/index.html#latex-packages
428 .. _Pygments: http://pygments.org/
429 .. _languages and markup formats: http://pygments.org/languages
430 .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/
432 .. _Docutils: http://docutils.sourceforge.net/
433 .. _Docutils Document Tree:
434 http://docutils.sf.net/docs/ref/doctree.html#classes
435 .. _latex-variants: http://docutils.sourceforge.net/sandbox/latex-variants/
436 .. _proof of concept:
437 http://article.gmane.org/gmane.text.docutils.user/3689
440 .. _front-end scripts: ../tools/pygments-enhanced-front-ends
441 .. _pygments-default.css: ../data/pygments-default.css
442 .. _pygments_code_block_directive.py: ../pygments_code_block_directive.py
443 .. _pygments_code_block_directive: pygments_code_block_directive-bunt.py.htm
444 .. _rst2html-highlight: ../rst2html-highlight
445 .. _pygments-long.css: ../data/pygments-long.css
446 .. _stylesheets: ../../stylesheets/