a lot test fixes, whitespaces, new error messages

[nobug.git] / pipadoc

#!/bin/sh
#
#TIT pipadoc - Documentation extractor
#TIT =================================
#TIT Christian Thaeter <ct@pipapo.org>
#TIT
#LIC Copyright (C)              Pipapo Project
#LIC  2009,                     Christian Thaeter <ct@pipapo.org>
#LIC
#LIC This program is free software; you can redistribute it and/or modify
#LIC it under the terms of the GNU General Public License as published by
#LIC the Free Software Foundation; either version 2, or (at your option)
#LIC any later version.
#LIC
#LIC This program is distributed in the hope that it will be useful,
#LIC but WITHOUT ANY WARRANTY; without even the implied warranty of
#LIC MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#LIC GNU General Public License for more details.
#LIC
#LIC You should have received a copy of the GNU General Public License
#LIC along with this program; if not, contact Christian Thaeter <ct@pipapo.org>.
#LIC

#INT Embedding documentation in program source files often yields the problem that the
#INT structure of a program is most often not the optimal structure for the associated documentation.
#INT Still there are good reasons to maintain documentation together with the source right at the code
#INT which defines the documented functionality. Pipadoc addresses this problem by extracting
#INT special comments out of a source file and let one define rules how to bring the
#INT documentation into proper order.
#INT
#INT Pipadoc only extracts and reorders the text from it special comments, it never ever looks at the
#INT sourcecode or the text it extracts.
#INT
#INT This is somewhat similar to ``Literate Programming'' but it puts the emphasis back to the code.
#INT There is no need to extract the source from a literate source and in contrast to ``Literate Programming''
#INT the order of source and text is defined by the programmer and programming language constraints.
#INT
#INT Pipadoc is programming language and documentation system agnostic, all it requires is that
#INT the programming language has some line comments or block comments where one places doc statements
#INT on each block line (see xref:c-example[Example for C]).
#INT
#BAS HEAD- Basic concepts; BAS; concepts
#BAS
#BAS NOTE: The following description uses the xref:ENV[default] settings for all examples.
#BAS
#BAS Pipadoc is controlled by special line comments.
#BAS
#DIR PARA Direct comments; DIR; low level control
#DIR
#DIR Line comments immediately followed by a special documentation character (the underscore `_` by default)
#DIR are treated as direct comments. They will appear in order of appearance in the generated output.
#DIR These can be used to do some boilerplate stuff. Usually one wants to define a controlling document and
#DIR use this direct comments only there, since using them in different files might yield unexpected results
#DIR since the order then depends on the load order of the files.
#DIR ----
#DIR //_ This is a direct comment,
#DIR //_ it will appear just verbatim in the generated output
#DIR ----
#DIR
#KEY PARA Keys; KEY; sectioning content
#KEY
#KEY A line comment immediately followed by a alphanumeric keyword (including the `_` underscore character)
#KEY is treated as key, all such keyed comments can later be placed in intended order with
#KEY a xref:SUB[substitution] comment.
#KEY ----
#KEY //example This text will later be inserted where one uses the `//=example` substitution.
#KEY //example All example lines are appended there in order even if they are defined at different
#KEY //example places or in different files
#KEY ----
#KEY
#SOR PARA Sorted Keys; SOR; sorted sections
#SOR A key can be appended with a dot `.` and a non-space string. This string will then be used to sort
#SOR these lines alphabetically. This can be used to create sorted indices and glossars, as well as reordering
#SOR paragraphs in stored under one primary key.
#SOR ----
#SOR //example.omega This is sorted after the next line
#SOR //example.alpha comes before the omega line above
#SOR ----
#SOR
#VAR PARA Variable Substitution; VAR; variable substitution
#VAR The strings `%%FILE%%` and `%%LINE%%` are substituted with the filename and line currently proccessed.
#VAR They can be used in the secondary sort key, as well as in documentation text directly. When one wants
#VAR to use this words verbatim then then percent signs must be doubled as in `%%%FILE%%%` or `%%%LINE%%%`.
#VAR

#SUB PARA Substitutions; SUB; paste sections
#SUB A line comment immediately followed by a special `substitution` character (the equal `=` sign by default)
#SUB followed by a xref:KEY[key] will be replaced by the text defined under that key. The rest of the line
#SUB is ignored and can be used as comment.
#SUB ----
#SUB //=example this will insert anything defined under `//example` here
#SUB ----
#SUB

#USE HEAD- Documenting Files; USE; usage
#USE
#USE Usually one wants to write documentation in more or less smaller blocks which later shall be
#USE brought into proper order. The xref:SUB[substitutions] feature is the key for this. One writes
#USE his documentation blocks with comments which later get replaced by the right sorting key and
#USE finally brought into (alphabetical) stable-sort order. You might take a look at the pipadoc
#USE source itself to see it in action.
#USE

#PLU HEAD- Plugins; PLU; extending pipadoc
#PLU Pipadoc can be extended with awk. This plugins are `awk` snippets which are included into
#PLU the processing stream. Take a look at the supplied plugins to figure out how it works.
#PLU
#PLU Supplied Plugins
#PLU ~~~~~~~~~~~~~~~~
#PLU
#PLU .asciidoc
#PLU Defines HEAD and PARA macros for headers and paragaphs with automatic index generation
#PLU in asciidoc
#PLU
#PLU .verbatim
#PLU Lets one include the code _before_ an pipadoc comment with the magic word `VERBATIM`.
#PLU Used to include code in the documentation.
#PLU

#ENV HEAD~ Environment Variables; ENV; configuration
#ENV
#ENV `COM`::
#ENV   Defines the line-comment character used in the source file.
#ENV   Defaults to `//` (C++/C99) if not set. Set this to `#` for shell
#ENV   like languages.
test "$COM" || COM='//'
#ENV `DOC`::
#ENV   The Documentation character which must follow a line comment to be recognized
#ENV   by pipadoc as documentation. Either one for local definitions or two for global
#ENV   definitions are used. Defaults to `_` and needs rarely to be changed.
test "$DOC" || DOC='_'
#ENV `SUB`::
#ENV   Substitution character. Defaults to `=` and rarely needs to be changed.
#ENV   See xref:SUB[substitutions] for details.
#ENV
test "$SUB" || SUB='='
#ENV [[ENVSXT]]
#ENV `SXT`::
#ENV   Section eXTention. Defaults to `.txt`.
#ENV   See xref:SXT[plaintext files] for details.
#ENV
test "$SXT" || SXT='.txt'
#ENV `PLG`::
#ENV   Extension for xref:PLU[awk plugins] to implement generic text manipulations and macros.
#ENV   Defaults to '.pawk'
#ENV
test "$PLG" || PLG='.pawk'

#SXT HEAD- Documentation Only Files; SXT; plain documentation
#SXT
#SXT One can write documentation without the need of pipadoc special comments in files
#SXT which have a configured extension (see the xref:ENVSXT[SXT] environment variable).
#SXT Each line in such a file which does not have a pipadoc special comment is then implicitly
#SXT prepended with the line comment sequence and the basename of that file, for example
#SXT lines in a file 'foo.txt' will be treated as if they where written with `//foo ` in front of
#SXT them. Later xref:SUB[substitutions] can be used to organize the document.
#SXT When such a file has an ordinary pipadoc special comment line then this takes precedence over
#SXT the implicit commenting.
#SXT
#INV HEAD- Invocation; INV; running pipadoc
#INV
#INV Pipadoc is called with a list of files from which the documentation shall be extracted.
#INV The extracted documentation is piped to stdout. No other command line options are available.
#INV
#INV There are few xref:ENV[environment variables] to do basic configuration.
#INV


sources=($(for element in "$@"; do [[ $element != *$PLG ]] && printf "%s " $element; done))
plugins=($(for element in "$@"; do [[ $element = *$PLG ]] && printf "%s " $element; done))


# here we go
gawk -f /dev/fd/3 ${COM:+-v com="$COM"} ${DOC:+-v doc="$DOC"} ${SUB:+-v subs="$SUB"} ${SXT:+-v sxt="$SXT"} "${sources[@]}" 3<<EOF

function append(idx, sort, text)
{
  if (idx in maybe)
    maybe[idx] = maybe[idx] "\n."sort" " text
  else
    maybe[idx] = "."sort" "text
}

$([[ ${#plugins[*]} -gt 0 ]] && cat "${plugins[@]}")

# Plaintext file handling
FILENAME ~ sxt "\$" && \$0 !~ "^" com {
  match(FILENAME, "/?([^/]*)" sxt "\$", p)
  \$0 = com p[1] " " \$0
}

# Substitution
match(\$0, com subs "([[:alpha:]][[:alnum:]_]*)", s) {
  ++n
  subst[n] = s[1]
  next
}

# doc comment
match(\$0, com doc "([[:space:]](.*))?\$", s) {
  ++n
  output[n] = s[2]
  next
}

# record all other comments which may be candidate comments
match(\$0, "(.*)" com "([[:alpha:]][[:alnum:]_]*)(([.]([^[:space:]]*)))?([[:space:]](.*))?", m) && m[1] !~ com {
  gsub("%%FILE%%", " FILE%FILE ", m[5])
  gsub("%%LINE%%", " LINE%LINE ", m[5])
  gsub("%LINE%", FNR, m[5])
  gsub("%FILE%", FILENAME, m[5])
  gsub(" FILE%FILE ", "%FILE%", m[5])
  gsub(" LINE%LINE ", "%LINE%", m[5])

  gsub("%%FILE%%", " FILE%FILE ", m[7])
  gsub("%%LINE%%", " LINE%LINE ", m[7])
  gsub("%LINE%", FNR, m[7])
  gsub("%FILE%", FILENAME, m[7])
  gsub(" FILE%FILE ", "%FILE%", m[7])
  gsub(" LINE%LINE ", "%LINE%", m[7])

  append(m[2], m[5] , m[7])
  next
}

# finally output
END {
  for (i=1; i<=n; ++i)
    {
      if (i in output)
        print output[i]
      else
        {
          nelements = split(maybe[subst[i]], s, "\n")
          delete tosort
          #split("", tosort)

          for (j=1; j <= nelements; ++j)
            {
              match(s[j], "[.]([^[:space:]]*) (.*)", entries)

              if (entries[1] in tosort)
                tosort[entries[1]] = tosort[entries[1]] "\n" entries[2]
              else
                tosort[entries[1]] = entries[2]
            }

          elements = asorti(tosort, sorted)
          for (k = 1; k <= elements; ++k)
            {
              print tosort[sorted[k]]
            }
        }
    }
}
EOF

#Document structure:
#=TIT Titles and stuff
#=INT Introduction
#=BAS Basics
#=DIR Direct comments
#=KEY Key comments
#=SOR sorted keys
#=VAR Variable substitutions
#=SUB Substitutions
#=INV Invocation
#=USE Real usage
#=SXT Plain Documentation
#=PLU Plugins
#_ Appendix
#_ --------
#=ENV Environment variables
#=APP Appendix
#_ License
#_ ~~~~~~~
#_ [[LIC]]
#_ ....
#=LIC
#_ ....
#_ Index
#_ -----
#=index

# Examples and appendixes:
#USE .A small C99 Program
#USE [source,c]
#USE ----
#USE //intro This is the well known ``Hello World'' example
#USE //glos.helloworld A common program to show examples for programming language and tools
#USE
#USE //depends Only the Standard C library is needed
#USE #include <stdio.h>
#USE
#USE int main(int argc, char* argv[])
#USE {
#USE   //hello print the first commandline argument
#USE   //glos.argument the text you pass to the programm when calling it from the shell
#USE   //hello if no argument is given, then exit silently
#USE   if (argc > 1)
#USE     printf("Hello %s\n", argv[1]);
#USE   return 0;
#USE }
#USE
#USE // Now the document structure with substitutions:
#USE //_ Yet another 'Hello' program
#USE //_
#USE //=intro introduction first right after the title
#USE //intro
#USE //=hello The main documentation
#USE //_
#USE //_ Appendix
#USE //_ Dependencies:
#USE //=depends
#USE //_ Glossary
#USE //=glos glossary will be sorted
#USE ----
#USE
#USE Runnning this through pipadoc gives following output:
#USE ----
#USE Yet another 'Hello' program
#USE
#USE This is the well known ``Hello World'' example
#USE
#USE print the first commandline argument
#USE if no argument is given, then exit silently
#USE
#USE Appendix
#USE Dependencies:
#USE Only the Standard C library is needed
#USE Glossary
#USE the text you pass to the programm when calling it from the shell
#USE A common program to show examples for programming language and tools
#USE ----
#USE

#APP [[c-example]]
#APP
#APP .Using C block comments with pipadoc: +example.c+
#APP ----
#APP /*
#APP //_ this is a documentation line
#APP */
#APP ----
#APP use `pipadoc example.c` to process the documentation.
#APP
#APP Creating this File
#APP ~~~~~~~~~~~~~~~~~~
#APP Pipadoc itself is documented with asciidoc, one can extract the asciidoc source with
#APP by:
#APP ----
#APP COM='#' ./pipadoc pipadoc asciidoc.pawk >pipadoc.txt
#APP ----
#APP And then further process the generated asciidoc.txt with the asciidoc toolchain.
#APP