todo: notes about xslt compilers

[gtk-doc.git] / TODO

The TODO list for the gtk-doc project is at Bugzilla,
the bugtracking system of the GNOME project.

Visit
 http://bugzilla.gnome.org/buglist.cgi?product=gtk-doc&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED
to see what is allready requested, or where you can help. :-)

To put an other request on the TODO list, visit
 http://bugzilla.gnome.org/enter_bug.cgi?product=gtk-doc

Also have a look at
 http://live.gnome.org/DocumentationProject/GtkDocFuture
and join discussion about future features.


Developers can also add items here :)

= Cleanups =
== Tracing ==
* there is a bunch of #print statements for tracing
  => add a sub Trace() to gtkdoc-common.pl
  => use @TRACE@ "..." and depending on configure flag
     turn that into
       [print __FILE__ . ":" . __LINE__ . ":" . ] or [#]
     should be a function and the function should support loglevels and an
       envar to enable/disable conditionally;
       
= More abbreviations =
* expand urls (needds more work, see gtkdoc-mkdb : ExpandAbbreviations)

= Testing =
cd test/gobject
diff -u --exclude="Makefile*" docs docs-tmpl | diffstat

= Issues =
* gtkdoc-fixxref makefile targets use $HTML_DIR
  * HTML_DIR: The directory where gtk-doc generated documentation is installed
    it comes from gtk-doc.m4 (--with-html-dir) but has no default
  * automake exports $htmldir which is by default:
    ${prefix}/share/doc/${PACKAGE_TARNAME}
  * the Makefile uses $(DESTDIR)$(TARGET_DIR)
    where TARGET_DIR = $(HTML_DIR)/$(DOC_MODULE)
    http://www.gnu.org/software/libtool/manual/automake/DESTDIR.html
    
* $MODULE-unused.txt is produced in gtkdoc-mktmpl only
  * we won't find unused doc blobs in notmpl build
  * we should add mktmpl::CheckAllDeclarationsOutput() to mkdb (call it after
    OutputSGML), but only call it if there is no tmpl dir or
    remove writing the unused.txt in mktmpl.txt

= Output =
* http://sagehill.net/docbookxsl/index.html
* multipage-html
  * would be good to be able to have page titles as a concatenation of document
    name and page name (gtk+:GtkWIdget)
* formats
  http://bugzilla.gnome.org/show_bug.cgi?id=531572 : html-single
  http://bugzilla.gnome.org/show_bug.cgi?id=466535 : pdf
  http://bugzilla.gnome.org/show_bug.cgi?id=467488 : man
  we need more configure options in gtk-doc.m4:
  --(enable|disable)-gtk-doc-(html|pdf|man|html-single|rtf)
  - html : enabled by default
  - html-single : is single page html
* validation
    xmllint --noout --xinclude --postvalid tester-docs.xml
    xmllint --noout --postvalid tester-docs.fo --dtdvalid file://$HOME/download/fo.dtd
    - fo.dtd : http://www.renderx.com/Tests/validator/fo.zip
* single page
  xsltproc  --nonet --xinclude -o gtk-docs.html /home/ensonic/projects/gtk-doc/gtk-doc-single.xsl  gtk-docs.sgml
  * need to check if we can pass the style-sheet class as a parameter (--stringparam gtkdoc.stylesheet=(chunk|docbook))
  * we might also need to reflow some things, as gtk-doc.xsl also runs the devhelp/devhelp2 generation
    - but then the urls in the devhelp file, refer to the chunked html anyway
* pdf
  * xmlto via passivetex
    xmlto --skip-validation pdf tester-docs.xml
  * fop
    ~/download/fop-0.95beta/fop -xsl /usr/share/xml/docbook/stylesheet/nwalsh/fo/docbook.xsl -xml tester-docs.xml -pdf tester-docs.pdf 
    ~/download/fop-0.94/fop  -xsl /usr/share/xml/docbook/stylesheet/nwalsh/fo/docbook.xsl -xml tester-docs.xml -pdf tester-docs.pdf
  * xsltproc + fop
    xsltproc  --nonet --xinclude -o tester-docs.fo ../../../gtk-doc-fo.xsl tester-docs.xml
    ~/download/fop-0.94/fop -fo tester-docs.fo -pdf tester-docs.pdf
  * xsltproc + passivetex
    pdflatex "&pdfxmltex" tester-docs.fo
    xmltex tester-docs.fo
  * jade
    docbook2pdf -e no-valid tester-docs.sgml

  * bugs/problems/howto
    * xmlto via passivetex
      http://bugs.gentoo.org/show_bug.cgi?id=224937
      http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=310148
      - info
        http://www.tug.org/texlive/devsrc/Master/texmf-dist/doc/xmltex/passivetex/
    * fop
      https://issues.apache.org/bugzilla/show_bug.cgi?id=46386
      - download fop (don't use 0.95, 0.94 seems to work)
        http://mirror.eunet.fi/apache/xmlgraphics/fop/binaries/
        http://xmlgraphics.apache.org/fop/0.94/running.html
        export FOP_OPTS="-Dhttp.proxyHost=xxx -Dhttp.proxyPort=8080"
      - download offo
        http://sourceforge.net/project/showfiles.php?group_id=116740&package_id=129569&release_id=267101
        and copy fop-hyph.jar to fop-0.9*/lib/
* rtf
 ~/download/fop-0.94/fop -fo tester-docs.fo -rtf tester-docs.rtf
 * unrtf
   unrtf -t ps tester-docs.rtf >tester-docs.ps
   unrtf -t latex tester-docs.rtf >tester-docs.tex
   - bad output
* man
  we shouldn't convert the whole document to man. We should convert e.g. tool
  sections to man pages.

= Indexes =
* http://www.w3.org/TR/2003/WD-xinclude-20031110/#syntax
<xi:include href="index-symbols.html">
  <xi:fallback><index /></xi:fallback>
</xi:include>
* index terms
  http://sourceforge.net/tracker/index.php?func=detail&aid=1986587&group_id=21935&atid=373747

* we could add smart navigation for index/glossary pages
  (like the subsections on the doc-pages)
  
= Cleanup =
* can we deprecate title in the sectionfile and request people to have this in
  the SECTION comment?

= Warnings =
* add some -Wxxx parameters to help qa work
  - 'deprecated' deprecating 'features'
  - 'dummy-docs' check if symbol docs are very short and repeat mainly words
    from the symbol.

= Markup =
* protected scope
  * we can have /* < protected > */ in classes
  * we can have <SUBSECTION Protected> in -section.txt
  * ideally we have Scope: {Public, Protected, Private} supported in doc comments
  * there is a bg for gir, https://bugzilla.gnome.org/show_bug.cgi?id=594125

= GIR =
== scanning ==
* ideas
  * use gir files
    1) replace gtkdoc-scan/gtkdoc-scangobject by gtkdoc-gir and output the classical files or
       patch gtkdoc-scan/gtkdoc-scangobject to output gir files
    2) patch gtkdoc-mkdb to read stuff from gir instead of classical files
  * if gir-files would have the comments too (they are getting this now):
    * we could even drop scanning the sources
    * IDEs could use the gir-files to show doc-tooltips for symbols
* perl and xml
  * http://www.xml.com/pub/a/2001/04/18/perlxmlqstart1.html
== binding docs ==
* simmillar workflow to gettext
* add gtkdoc-mk??? to generate binding doc templates
  * have c-comments there as comments
  * when updating templates, mark sections that have changed as fuzzy
* add options to gtkdoc-mkdb to build docbook from those templates
* questions
  * could we use the tmpl file mechanism?
  * directory structure?
    * we need to list the languages like ALL_LINGUAS for translations
    * we need to create subdirs for binding docs, ideally we use one for 'C' as well
* devhelp
  * devhelp files need a language attribute in the book-tag
    language={C,C++,JavaScript,Perl,Python,...}
  * devhelp could show a selector for the language
  * need to get existing python/~mm docs to use it, gtk-doc could output
    language=C for own docs 
== installation ==
* need to install each book with a prefix
* would be good to have a language attribute in book tag to allow filter by language
* look at /usr/share/gtk-doc/html/pygobject/index.html

= linking to sources =
- what about a template URL containing a %s for the "path/file". 
  http://svn.gnome.org/viewvc/gtk-doc/trunk/tests/gobject/src/gobject.c?view=markup
  http://buzztard.svn.sourceforge.net/viewvc/buzztard/trunk/buzztard/src/lib/core/core.c?view=markup
  - unfortunately we can't link to symbols
  - linking to files is difficult as in gtkdoc we have modules


= docbook xml =
Its tedious to write large amounts of docbook.

== more macros ==
We need parametric, user definable macros.
|[ ... ]| - programlistings
|macro(arg1,arg2,...)[ ... ]| - call macro
  - pass args as parameters (on the commandline)
  - pass some gtk-doc vars in environment
    (gtk-doc version, module, srcdir, buildir)
  - content of [] on stdin or as a file
  - get output on stdout or file
  - and replace the macro with it
The changes could be made in gtkdoc-mkdb::ExpandAbbreviations()
=== example macros ===
|highlight(c)[...]| - highlight source code for a specific language
|dot(abc.svg)[...]| - format dot graph and include result as mediaobject
|ditta(abc.svg)[...]| - parse ascii art and include result as mediaobject

=== where to define macros ===
* system wide and with the package, <prefix>/share/gtk-doc/macros, $(srcdir)
* prefix for custom macros?
* we could require stdin for input and stdout for output, the wrapper for the
  actual tool can ensure the convention

== ascii doc as a frontend ==
Can we offer integration with asciidoc (http://www.methods.co.nz/asciidoc/)?
This way the master document could be written much easier. It would be cool if
we could use the asciidoc markup in source-comments also.

== extract other bits and pieces ==
=== library api ==
gtkdoc-scan could be obsoleted and gtkdoc-mkdb would build docbook fragemnts for
api docs and their indexes
=== DBUs Interfaces ===
http://hal.freedesktop.org/docs/DeviceKit/DeviceKit.html
http://cgit.freedesktop.org/DeviceKit/DeviceKit/tree/doc/dbus
=== GConf schemas ===

=== Wiki imports ===



= styling =
== source code examples==
http://bugzilla.gnome.org/show_bug.cgi?id=536928
We could also run a postprocessing script in gtkdoc-mkhtml/gtkdoc-fixxref

tools:
  source-highlight (/usr/bin/source-highlight)
  source-highlight -itests/gobject/examples/gobject.c -o$HOME/temp/gobject.html -n -t4 -sc
  source-highlight -itests/gobject/examples/gobject.c -o$HOME/temp/gobject.html -n -t4 -sc -cstyle.css --no-doc
  source-highlight -itests/gobject/examples/gobject.c -o$HOME/temp/gobject.xml -n -t4 -sc -f docbook

  highlight -itests/gobject/examples/gobject.c -o$HOME/temp/gobject.xml -l -X -f -j2

some tips about styling code listings in html
http://www.tjkdesign.com/articles/how_to_style_a_code_listing.asp

=== process docbook ===
if we highlight to docbook, we just get emphasis (bold)
=== process html ===
if we hightlight to html we get colors, we need to check what tags we should process though:
<pre class="programlisting"> is used for all code boxes.
<div class="informalexample"><pre class="programlisting"> is used for examples.
problems:
* in html we don't know the language anymore
  * add another div
* with source-highlight, constants and types are not markedup.
  for types we might need to build an own lang file dynamically and include
  /usr/share/source-highlight/c.lang
=== |[ ... ]| does not allow setting the language ===
* check for vi/emacs/jedit modelines
  jedit: http://www.jedit.org/users-guide/buffer-local.html
  vim: http://vim.wikia.com/wiki/Modeline_magic
  emacs: http://www.delorie.com/gnu/docs/emacs/emacs_486.html
* allow <!-- language="C" --> comments after |[
* we need to catch those when processing the docbook and expanding the |[



= syntax =
== wildcards in symbol names ==
Somtimes one defines a set of function and macros with very simillar purpose, e.g.
READ_INT8, READ_INT16, READ_INT32. It would be great to allow documenting a symbol
READ_INT* instead of 3 docs which are copy'n'pasted anyway. In the output we will have
all matching declarations in one source listing. Multiple wildcards are okay.


= documentation best practises #518427 =
* we'd like offer a more complete skelleton
  * structure
  * docbook markup (part/chapter structure)
* structure
  Sugested structure for api-docs.
  Idea is to have more content that api reference. It would be good to have a
  gnome-platform document in devhelp, so that we could xref that instead of
  explaining 100 times how to use pkg-config.

  * examples
    * gobject in devhelp
      * concepts / api / tools / tutorial / related tools
    * gtk in devhelp
      * overview / api / migation / tools
    * qt reference docs in qt assistant
      * classes / overview / tutorial&examples
  * recommendation
    * front-page
      * table with details (http://www.docbook.org/tdg/en/html/bookinfo.html)
        (problem: what enclosing tag)
        Logo, Module Version
        Copyright and Legalnotice
        Links
        * homepage, mailing lists, irc channel
        * repository, source releases, bugtracker
      * TOC
    * introduction - what is is about
    * concepts - explain basic ideas and terminology
    * development - how to build and run, env-vars, different platforms
    * api - classic api docs
    * tutorial & examples - integrated to keep it up-to-date and cross referenced
    * migration - how to for api changes, deprecations
    * (releated) tools - tools part of the package or recommended for development
    * indexes - api-index, depretations, new api since xxx

proposed structure in docbook xml:
<book>
  <bookinfo>
  </bookinfo>
  <preface><title>Introduction</title>
    ...
  </preface>
  <part label="I"><title>xxx Overview</title>
    <xi:include href="building.xml" />
    ...
  </part>
  <reference label="II"><title>xxx Core Reference</title>
    <xi:include href="xml/gtkmain.xml" />
    ...
  </part>
  <reference label="III"><title>xxx Object Reference</title>
    <chapter><title>Object Hierarchy</title>
      <xi:include href="xml/tree_index.sgml" />
    </chapter>
    <chapter>...
  </part>
  <index>...</index>
</book>

some things to check:
* gtk,glib: can we make a <part> for the glosary and index's (according to docbook, yes)
  should we use <appendix>? its like a chapter.
* gobject: uses a <preface> for introductions
* gobject: uses <reference> as a parent for the xi:includeed <refentry> docs

= extra link for symbols =
need options for configure:
--enable-gtk-doc-codesearch-links
--enable-gtk-doc-liveedit-links
== viewvc,cgit,... ==
- link to some online service for the code
- problem: most don't have local anchors for the symbols
- where to set the uri (in the document, like for online url)?
== codesearch ==
- google (code) link : http://www.google.com/codesearch?q=g_object_unref
== live editing ==
The idea is to have an 'edit' link in an online version of the docs (build from
head development version) per doc-entry (symbols and section).
The link goes to a cgi and that gets following parameters: docmodule,symbol.
E.g. http://library.gnome.org/devel/references/edit?docmodule=glib&symbol=g_new
The cgi would need a hashmap to get from docmodule to the way to check it out
(ideally it has a recent checkout and only updates it).
problems:
- signal that this has been edited already?
- support for xi:included examples
- updating the checkout could be slow

= fix missing since docs =
cd gstreamer/gstreamer/docs/gst
gtkdoc-mkdb --module=gstreamer --source-dir=../../gst --outputsymbolswithoutsince
cd gstreamer/gstreamer/src
git bisect start
git bisect good
git bisect bad RELEASE-0_10_0
git bisect run script.sh

script:
#!/bin/sh
make ctags
grep "gst_caps_is_always_compatible" tags


= performance =
- timestamp each step
  make check >make.log
- try CFLAGS=-O0 for compiling the scanner, no need to optimize it
  CFLAGS="-O0" make check >make.log
  safes max 0.5 sec.
- xslt
  http://docbook2x.sourceforge.net/latest/doc/performance.html
  - play with xsltproc --profile --verbose --timing
    cd tests/gobject/docs/html
    time /usr/bin/xsltproc 2>xslt.log --profile --verbose --timing --path /home/ensonic/projects/gnome/gtk-doc/gtk-doc/tests/gobject/docs --nonet --xinclude --stringparam gtkdoc.bookname tester --stringparam gtkdoc.version 1.14 /home/ensonic/projects/gnome/gtk-doc/gtk-doc/gtk-doc.xsl ../tester-docs.xml
    - l10n.language is slow
      bug: https://sourceforge.net/tracker/index.php?func=detail&aid=2918673&group_id=21935&atid=373750
      see: http://www.mail-archive.com/docbook-apps@lists.oasis-open.org/msg05412.html
      - overide l10n.language
        glib/gobject
        real        user        sys
        2m15.221s   1m58.740s   0m1.456s
        >
        1m55.480s   1m44.296s   0m2.125s
      - removing the gentext calls for nav-bar alt tags does not help

      
  - try plain docbook xslt to see if maybe we have bad xslt templates in the 
    customisation layer (gtk-doc.xsl)

  - we could do the xinlcude processing once and use it for both html and pdf
    time /usr/bin/xsltproc 2>../xslt4.log --path /home/ensonic/projects/gnome/gtk-doc/gtk-doc/tests/gobject/docs --nonet --xinclude --stringparam gtkdoc.bookname tester --stringparam gtkdoc.version 1.14 /home/ensonic/projects/gnome/gtk-doc/gtk-doc/gtk-doc.xsl ../tester-docs.xml
    real        user       sys
    0m4.846s    0m4.434s   0m0.147s
    0m4.842s    0m4.386s   0m0.145s
    
    
    time xmllint --nonet --xinclude ../tester-docs.xml >./tester-docs-all.xml
    real        user       sys
    0m0.596s    0m0.546s   0m0.023s
    
    time /usr/bin/xsltproc 2>../xslt5.log --path /home/ensonic/projects/gnome/gtk-doc/gtk-doc/tests/gobject/docs --nonet --stringparam gtkdoc.bookname tester --stringparam gtkdoc.version 1.14 /home/ensonic/projects/gnome/gtk-doc/gtk-doc/gtk-doc.xsl ./tester-docs-all.xml
    real        user       sys
    0m4.167s    0m3.834s   0m0.106s
    0m4.248s    0m3.851s   0m0.114s
    
    
    time xmllint --nonet --c14n --xinclude ../tester-docs.xml >./tester-docs-all2.xml
    
    real        user       sys
    0m0.700s    0m0.636s   0m0.034s
    
    time /usr/bin/xsltproc 2>../xslt6.log --path /home/ensonic/projects/gnome/gtk-doc/gtk-doc/tests/gobject/docs --nonet --stringparam gtkdoc.bookname tester --stringparam gtkdoc.version 1.14 /home/ensonic/projects/gnome/gtk-doc/gtk-doc/gtk-doc.xsl ./tester-docs-all2.xml
    
    real        user       sys
    0m3.344s    0m3.026s   0m0.109s
    0m3.372s    0m3.037s   0m0.115s
    
    
    l ../tester-docs.xml ./tester-docs-all*.xml
    
  - we could also try to compact the installed xslt
    xmllint --nonet --c14n --xinclude gtk-doc.xsl | sed -ne '/<!--/ { :c; /-->/! { N; b c; }; /-->/s/<!--.*-->//g }; /^  *$/!p;' | sed '/^$/d' >gtk-doc.pre.xsl
    - unfortunately there is no way to ask xsltproc to pre-transform an xslt, that could
      - strip comments
      - process xsl:import and xsl:include
  - compile xslt
    http://sourceforge.net/projects/xsltc/
    http://www.xmlhack.com/read.php?item=618

- perl regexps
  not really an issue, but we can improve by compiling the regexps
  http://stackoverflow.com/questions/550258/does-the-o-modifier-for-perl-regular-expressions-still-provide-any-benefit