updated NEWS

[luatex.git] / manual / luatex-lua.tex

\environment luatex-style
\environment luatex-logos

\startcomponent luatex-lua

\startchapter[reference=lua,title={\LUA\ general}]

\section[init]{Initialization}

\subsection{\LUATEX\ as a \LUA\ interpreter}

There are some situations that make \LUATEX\ behave like a standalone \LUA\
interpreter:

\startitemize[packed]
\startitem
    if a \type {--luaonly} option is given on the commandline, or
\stopitem
\startitem
    if the executable is named \type {texlua} or \type {luatexlua}, or
\stopitem
\startitem
    if the only non|-|option argument (file) on the commandline has the extension
    \type {lua} or \type {luc}.
\stopitem
\stopitemize

In this mode, it will set \LUA's \type {arg[0]} to the found script name, pushing
preceding options in negative values and the rest of the commandline in the
positive values, just like the \LUA\ interpreter.

\LUATEX\ will exit immediately after executing the specified \LUA\ script and is,
in effect, a somewhat bulky standalone \LUA\ interpreter with a bunch of extra
preloaded libraries.

\subsection{\LUATEX\ as a \LUA\ byte compiler}

There are two situations that make \LUATEX\ behave like the \LUA\ byte compiler:

\startitemize[packed]
\startitem if a \type {--luaconly} option is given on the commandline, or \stopitem
\startitem if the executable is named \type {texluac} \stopitem
\stopitemize

In this mode, \LUATEX\ is exactly like \type {luac} from the standalone \LUA\
distribution, except that it does not have the \type {-l} switch, and that it
accepts (but ignores) the \type {--luaconly} switch.

\subsection{Other commandline processing}

When the \LUATEX\ executable starts, it looks for the \type {--lua} commandline
option. If there is no \type {--lua} option, the commandline is interpreted in a
similar fashion as in traditional \PDFTEX\ and \ALEPH. Some options are accepted
but have no consequence. The following command|-|line options are understood:

\starttabulate[|lT|p|]
\NC --fmt=FORMAT                 \NC load the format file \type {FORMAT} \NC\NR
\NC --lua=FILE                   \NC load and execute a \LUA\ initialization script\NC\NR
\NC --safer                      \NC disable easily exploitable \LUA\ commands \NC\NR
\NC --nosocket                   \NC disable the \LUA\ socket library \NC\NR
\NC --help                       \NC display help and exit \NC\NR
\NC --ini                        \NC be iniluatex, for dumping formats \NC\NR
\NC --interaction=STRING         \NC set interaction mode: \type {batchmode}, \type {nonstopmode}
                                     \type {scrollmode} or \type {errorstopmode} \NC \NR
\NC --halt-on-error              \NC stop processing at the first error\NC \NR
\NC --kpathsea-debug=NUMBER      \NC set path searching debugging flags according to
                                     the bits of \type {NUMBER} \NC \NR
\NC --progname=STRING            \NC set the program name to \type {STRING} \NC \NR
\NC --version                    \NC display version and exit \NC \NR
\NC --credits                    \NC display credits and exit \NC \NR
\NC --recorder                   \NC enable filename recorder \NC \NR
\NC --etex                       \NC ignored \NC \NR
\NC --output-comment=STRING      \NC use \type {STRING} for \DVI\ file comment instead of
                                     date (no effect for \PDF) \NC \NR
\NC --output-directory=DIR       \NC use \type {DIR} as the directory to write files to \NC \NR
\NC --draftmode                  \NC switch on draft mode i.e.\ generate no output in \PDF\ mode \NC \NR
\NC --output-format=FORMAT       \NC use \type {FORMAT} for job output; \type {FORMAT} is \type {dvi} or
                                     \type {pdf} \NC \NR
\NC --[no-]shell-escape          \NC disable/enable \type {\write18{SHELL COMMAND}} \NC \NR
\NC --enable-write18             \NC enable \type {\write18{SHELL COMMAND}} \NC \NR
\NC --disable-write18            \NC disable \type {\write18{SHELL COMMAND}} \NC \NR
\NC --shell-restricted           \NC restrict \type {\write18} to a list of commands
                                     given in \type {texmf.cnf} \NC \NR
\NC --debug-format               \NC enable format debugging \NC \NR
\NC --[no-]file-line-error       \NC disable/enable \type {file:line:error} style messages  \NC \NR
\NC --[no-]file-line-error-style \NC aliases of \type {--[no-]file-line-error} \NC \NR
\NC --jobname=STRING             \NC set the job name to \type {STRING} \NC \NR
\NC --[no-]parse-first-line      \NC ignored \NC \NR
\NC --translate-file=            \NC ignored \NC \NR
\NC --default-translate-file=    \NC ignored \NC \NR
\NC --8bit                       \NC ignored \NC \NR
\NC --[no-]mktex=FMT             \NC disable/enable \type {mktexFMT} generation with \type {FMT}
                                     is \type {tex} or \type {tfm} \NC \NR
\NC --synctex=NUMBER             \NC enable \type {synctex} \NC \NR
\stoptabulate

A note on the creation of the various temporary files and the \type {\jobname}.
The value to use for \type {\jobname} is decided as follows:

\startitemize
\startitem
  If \type {--jobname} is given on the command line, its argument will be the
  value for \type {\jobname}, without any changes. The argument will not be used
  for actual input so it need not exist. The \type {--jobname} switch only
  controls the \type {\jobname} setting.
\stopitem
\startitem
    Otherwise, \type {\jobname} will be the name of the first file that is read
    from the file system, with any path components and the last extension (the
    part following the last \type {.}) stripped off.
\stopitem
\startitem
    An exception to the previous point: if the command line goes into interactive
    mode (by starting with a command) and there are no files input via \type
    {\everyjob} either, then the \type {\jobname} is set to \type {texput} as a
    last resort.
\stopitem
\stopitemize

The file names for output files that are generated automatically are created by
attaching the proper extension (\type {.log}, \type {.pdf}, etc.) to the found
\type {\jobname}. These files are created in the directory pointed to by \type
{--output-directory}, or in the current directory, if that switch is not present.

\blank

Without the \type {--lua} option, command line processing works like it does in
any other web2c-based typesetting engine, except that \LUATEX\ has a few extra
switches.

If the \type {--lua} option is present, \LUATEX\ will enter an alternative mode
of commandline processing in comparison to the standard web2c programs.

In this mode, a small series of actions is taken in order. First, it will parse
the commandline as usual, but it will only interpret a small subset of the
options immediately: \type {--safer}, \type {--nosocket}, \type
{--[no-]shell-escape}, \type {--enable-write18}, \type {--disable-write18}, \type
{--shell-restricted}, \type {--help}, \type {--version}, and \type {--credits}.

Now it searches for the requested \LUA\ initialization script. If it cannot be
found using the actual name given on the commandline, a second attempt is made by
prepending the value of the environment variable \type {LUATEXDIR}, if that
variable is defined in the environment.

Then it checks the various safety switches. You can use those to disable some
\LUA\ commands that can easily be abused by a malicious document. At the moment,
\type {--safer} \type {nil}s the following functions:

\starttabulate[|l|l|]
\NC \bf library \NC \bf functions \NC \NR
\NC \type {os}   \NC \type {execute} \type {exec} \type {setenv} \type {rename} \type {remove} \type {tmpdir} \NC \NR
\NC \type {io}   \NC \type {popen} \type {output} \type {tmpfile} \NC \NR
\NC \type {lfs}  \NC \type {rmdir} \type {mkdir} \type {chdir} \type {lock} \type {touch} \NC \NR
\stoptabulate

Furthermore, it disables loading of compiled \LUA\ libraries and it makes \type
{io.open()} fail on files that are opened for anything besides reading.

\type {--nosocket} makes the socket library unavailable, so that \LUA\ cannot use
networking.

The switches \type {--[no-]shell-escape}, \type {--[enable|disable]-write18}, and
\type {--shell-restricted} have the same effects as in \PDFTEX, and additionally
make \type {io.popen()}, \type {os.execute}, \type {os.exec} and \type {os.spawn}
adhere to the requested option.

Next the initialization script is loaded and executed. From within the script,
the entire commandline is available in the \LUA\ table \type {arg}, beginning with
\type {arg[0]}, containing the name of the executable. As consequence, the warning
about unrecognized option is suppressed.

Commandline processing happens very early on. So early, in fact, that none of
\TEX's initializations have taken place yet. For that reason, the tables that
deal with typesetting, like \type {tex}, \type {token}, \type {node} and
\type {pdf}, are off|-|limits during the execution of the startup file (they
are nilled). Special care is taken that \type {texio.write} and \type
{texio.write_nl} function properly, so that you can at least report your actions
to the log file when (and if) it eventually becomes opened (note that \TEX\ does
not even know its \type {\jobname} yet at this point). See \in {chapter} [libraries]
for more information about the \LUATEX-specific \LUA\ extension tables.

Everything you do in the \LUA\ initialization script will remain visible during
the rest of the run, with the exception of the aforementioned \type {tex},
\type {token}, \type {node} and \type {pdf} tables: those will be
initialized to their documented state after the execution of the script. You
should not store anything in variables or within tables with these four global
names, as they will be overwritten completely.

We recommend you use the startup file only for your own \TEX|-|independent
initializations (if you need any), to parse the commandline, set values in the
\type {texconfig} table, and register the callbacks you need.

\LUATEX\ allows some of the commandline options to be overridden by reading
values from the \type {texconfig} table at the end of script execution (see the
description of the \type {texconfig} table later on in this document for more
details on which ones exactly).

Unless the \type {texconfig} table tells \LUATEX\ not to initialize \KPATHSEA\
at all (set \type {texconfig.kpse_init} to \type {false} for that), \LUATEX\
acts on some more commandline options after the initialization script is
finished: in order to initialize the built|-|in \KPATHSEA\ library properly,
\LUATEX\ needs to know the correct program name to use, and for that it needs to
check \type {--progname}, or \type {--ini} and \type {--fmt}, if \type
{--progname} is missing.

\section{\LUA\ behaviour}

\LUA s \type {tonumber} function may return values in scientific notation,
thereby confusing the \TEX\ end of things when it is used as the right|-|hand
side of an assignment to a \type {\dimen} or \type {\count}.

Loading dynamic \LUA\ libraries will fail if there are two \LUA\ libraries loaded
at the same time (which will typically happen on \type {win32}, because there is
one \LUA\ 5.2 inside \LUATEX, and another will likely be linked to the \DLL\ file
of the module itself). We plan to fix that later by switching \LUATEX\ itself to
using de \DLL\ version of \LUA\ 5.2 inside \LUATEX\ instead of including a static
version in the binary.

\LUATEX\ is able to use the kpathsea library to find \type {require()}d modules.
For this purpose, \type {package.searchers[2]} is replaced by a different loader
function, that decides at runtime whether to use kpathsea or the built|-|in core
\LUA\ function. It uses \KPATHSEA\ when that is already initialized at that point
in time, otherwise it reverts to using the normal \type {package.path} loader.

Initialization of \KPATHSEA\ can happen either implicitly (when \LUATEX\ starts
up and the startup script has not set \type {texconfig.kpse_init} to false), or
explicitly by calling the \LUA\ function \type {kpse.set_program_name()}.

\LUATEX\ is able to use dynamically loadable \LUA\ libraries, unless
\type {--safer} was given as an option on the command line. For this purpose,
\type {package.searchers[3]} is replaced by a different loader function, that
decides at runtime whether to use \KPATHSEA\ or the built|-|in core \LUA\
function. It uses \KPATHSEA\ when that is already initialized at that point in
time, otherwise it reverts to using the normal \type {package.cpath} loader.

This functionality required an extension to kpathsea:

\startnarrower
There is a new kpathsea file format: \type {kpse_clua_format} that searches for
files with extension \type {.dll} and \type {.so}. The \type {texmf.cnf} setting
for this variable is \type {CLUAINPUTS}, and by default it has this value:

\starttyping
CLUAINPUTS=.:$SELFAUTOLOC/lib/{$progname,$engine,}/lua//
\stoptyping % $

This path is imperfect (it requires a \TDS\ subtree below the binaries
directory), but the architecture has to be in the path somewhere, and the
currently simplest way to do that is to search below the binaries directory only.
Of course it no big deal to write an alternative loader and use that in a macro
package.

One level up (a \type {lib} directory parallel to \type {bin}) would have been
nicer, but that is not doable because \TEXLIVE\ uses a \type {bin/<arch>}
structure.
\stopnarrower

In keeping with the other \TEX|-|like programs in \TEXLIVE, the two \LUA\ functions
\type {os.execute} and \type {io.popen}, as well as the two new functions \type
{os.exec} and \type {os.spawn} that are explained below, take the value of \type
{shell_escape} and|/|or \type {shell_escape_commands} in account. Whenever
\LUATEX\ is run with the assumed intention to typeset a document (and by that we
mean that it is called as \type {luatex}, as opposed to \type {texlua}, and that
the commandline option \type {--luaonly} was not given), it will only run the
four functions above if the matching \type {texmf.cnf} variable(s) or their \type
{texconfig} (see \in {section} [texconfig]) counterparts allow execution of the
requested system command. In \quote {script interpreter} runs of \LUATEX, these
settings have no effect, and all four functions function as normal.

The \type {f:read("*line")} and \type {f:lines()} functions from the io library
have been adjusted so that they are line|-|ending neutral: any of \type {LF},
\type {CR} or \type {CR+LF} are acceptable line endings.

\type {luafilesystem} has been extended: there are two extra boolean functions
(\type {lfs.isdir(filename)} and \type {lfs.isfile(filename)}) and one extra
string field in its attributes table (\type {permissions}). There is an
additional function \type {lfs.shortname()} which takes a file name and returns
its short name on \type {win32} platforms. On other platforms, it just returns
the given argument. The file name is not tested for existence. Finally, for
non|-|\type {win32} platforms only, there is the new function \type
{lfs.readlink()} hat takes an existing symbolic link as argument and returns its
content. It returns an error on \type {win32}.

The \type {string} library has an extra function: \type {string.explode(s[,m])}.
This function returns an array containing the string argument \type {s} split
into sub-strings based on the value of the string argument \type {m}. The second
argument is a string that is either empty (this splits the string into
characters), a single character (this splits on each occurrence of that
character, possibly introducing empty strings), or a single character followed by
the plus sign \type {+} (this special version does not create empty sub-strings).
The default value for \type {m} is \quote {\type { +}} (multiple spaces). Note:
\type {m} is not hidden by surrounding braces as it would be if this function was
written in \TEX\ macros.

The \type {string} library also has six extra iterators that return strings
piecemeal:

\startitemize
\startitem
    \type {string.utfvalues(s)}: an integer value in the \UNICODE\ range
\stopitem
\startitem
    \type {string.utfcharacters(s)}: a string with a single \UTF-8 token in it
\stopitem
\startitem
    \type {string.characters(s)} \NC a string containing one byte
\stopitem
\startitem
    \type {string.characterpairs(s)} two strings each containing one byte or an
    empty second string if the string length was odd
\stopitem
\startitem
    \type {string.bytes(s)} a single byte value
\stopitem
\startitem
    \type {string.bytepairs(s)} two byte values or nil instead of a number as
    its second return value if the string length was odd
\stopitem
\stopitemize

The \type {string.characterpairs()} and \type {string.bytepairs()} iterators
are useful especially in the conversion of \UTF-16 encoded data into \UTF-8.

There is also a two|-|argument form of \type {string.dump()}. The second argument
is a boolean which, if true, strips the symbols from the dumped data. This
matches an extension made in \type {luajit}.

The \type {string} library functions \type {len}, \type {lower}, \type {sub}
etc.\ are not \UNICODE|-|aware. For strings in the \UTF8 encoding, i.e., strings
containing characters above code point 127, the corresponding functions from the
\type {slnunicode} library can be used, e.g., \type {unicode.utf8.len}, \type
{unicode.utf8.lower} etc. The exceptions are \type {unicode.utf8.find}, that
always returns byte positions in a string, and \type {unicode.utf8.match} and
\type {unicode.utf8.gmatch}. While the latter two functions in general {\it
are} \UNICODE|-|aware, they fall|-|back to non|-|\UNICODE|-|aware behavior when
using the empty capture \type {()} but other captures work as expected. For the
interpretation of character classes in \type {unicode.utf8} functions refer to
the library sources at \hyphenatedurl {http://luaforge.net/projects/sln}. Version
5.3 of \LUA\ will provide some native \UTF8 support.

\blank

The \type {os} library has a few extra functions and variables:

\startitemize

\startitem
    \type {os.selfdir} is a variable that holds the directory path of the
    actual executable. For example: \type {\directlua {tex.sprint(os.selfdir)}}.
\stopitem

\startitem
    \type {os.exec(commandline)} is a variation on \type {os.execute}. Here
    \type {commandline} can be either a single string or a single table.

    If the argument is a table: \LUATEX\ first checks if there is a value at
    integer index zero. If there is, this is the command to be executed.
    Otherwise, it will use the value at integer index one. (if neither are
    present, nothing at all happens).

    The set of consecutive values starting at integer~1 in the table are the
    arguments that are passed on to the command (the value at index~1 becomes
    \type {arg[0]}). The command is searched for in the execution path, so there
    is normally no need to pass on a fully qualified pathname.

    If the argument is a string, then it is automatically converted into a table
    by splitting on whitespace. In this case, it is impossible for the command
    and first argument to differ from each other.

    In the string argument format, whitespace can be protected by putting (part
    of) an argument inside single or double quotes. One layer of quotes is
    interpreted by \LUATEX, and all occurrences of \type {\"}, \type {\'} or \type
    {\\} within the quoted text are unescaped. In the table format, there is no
    string handling taking place.

    This function normally does not return control back to the \LUA\ script: the
    command will replace the current process. However, it will return the two
    values \type {nil} and \type {'error'} if there was a problem while
    attempting to execute the command.

    On \MSWINDOWS, the current process is actually kept in memory until after the
    execution of the command has finished. This prevents crashes in situations
    where \TEXLUA\ scripts are run inside integrated \TEX\ environments.

    The original reason for this command is that it cleans out the current
    process before starting the new one, making it especially useful for use in
    \TEXLUA.
\stopitem

\startitem
    \type {os.spawn(commandline)} is a returning version of \type {os.exec},
    with otherwise identical calling conventions.

    If the command ran ok, then the return value is the exit status of the
    command. Otherwise, it will return the two values \type {nil} and \type
    {'error'}.
\stopitem

\startitem
    \type {os.setenv('key','value')} sets a variable in the environment.
    Passing \type {nil} instead of a value string will remove the variable.
\stopitem

\startitem
    \type {os.env} is a hash table containing a dump of the variables and
    values in the process environment at the start of the run. It is writeable,
    but the actual environment is {\em not\/} updated automatically.
\stopitem

\startitem
    \type {os.gettimeofday()} returns the current \quote {\UNIX\ time}, but as a
    float. This function is not available on the \SUNOS\ platforms, so do not use
    this function for portable documents.
\stopitem

\startitem
    \type {os.times()}returns the current process times according to \ the
    \UNIX\ C library function \quote {times}. This function is not available on
    the \MSWINDOWS\ and \SUNOS\ platforms, so do not use this function for
    portable documents.
\stopitem

\startitem
    \type {os.tmpdir()} creates a directory in the \quote {current directory}
    with the name \type {luatex.XXXXXX} where the \type {X}-es are replaced by a
    unique string. The function also returns this string, so you can \type
    {lfs.chdir()} into it, or \type {nil} if it failed to create the directory.
    The user is responsible for cleaning up at the end of the run, it does not
    happen automatically.
\stopitem

\startitem
    \type {os.type} is a string that gives a global indication of the class of
    operating system. The possible values are currently \type {windows}, \type
    {unix}, and \type {msdos} (you are unlikely to find this value \quote {in the
    wild}).
\stopitem

\startitem
    \type {os.name} is a string that gives a more precise indication of the
    operating system. These possible values are not yet fixed, and for \type
    {os.type} values \type {windows} and \type {msdos}, the \type {os.name}
    values are simply \type {windows} and \type {msdos}

    The list for the type \type {unix} is more precise: \type {linux}, \type
    {freebsd}, \type {kfreebsd}, \type {cygwin}, \type {openbsd}, \type
    {solaris}, \type {sunos} (pre-solaris), \type {hpux}, \type {irix}, \type
    {macosx}, \type {gnu} (hurd), \type {bsd} (unknown, but \BSD|-|like), \type
    {sysv} (unknown, but \SYSV|-|like), \type {generic} (unknown).
\stopitem

\startitem
    \type {os.version} is planned as a future extension.
\stopitem

\startitem
    \type {os.uname()} returns a table with specific operating system
    information acquired at runtime. The keys in the returned table are all
    string valued, and their names are: \type {sysname}, \type {machine}, \type
    {release}, \type {version}, and \type {nodename}.
\stopitem

\stopitemize

In stock \LUA, many things depend on the current locale. In \LUATEX, we can't do
that, because it makes documents unportable.  While \LUATEX\ is running if
forces the following locale settings:

\starttyping
LC_CTYPE=C
LC_COLLATE=C
LC_NUMERIC=C
\stoptyping

\section {\LUA\ modules}

The implied use of the built|-|in Lua modules in this section is deprecated. If
you want to use one of these libraries, please start your source file with a
proper \type {require} line. At some point \LUATEX\ will switch to loading these
modules on demand.

Some modules that are normally external to \LUA\ are statically linked in with
\LUATEX, because they offer useful functionality:

\startitemize

\startitem
    \type {slnunicode}, from the \type {Selene} libraries, \hyphenatedurl
    {http://luaforge.net/projects/sln}. (version 1.1) This library has been
    slightly extended so that the \type {unicode.utf8.*} functions also accept the
    first 256 values of plane~18. This is the range \LUATEX\ uses for raw binary
    output, as explained above.
\stopitem

\startitem
    \type {luazip}, from the kepler project,
    \hyphenatedurl{http://www.keplerproject.org/luazip/}. (version 1.2.1, but
    patched for compilation with \LUA\ 5.2)
\stopitem

\startitem
    \type {luafilesystem}, also from the kepler project, \hyphenatedurl
    {http://www.keplerproject.org/luafilesystem/}. (version 1.5.0)
\stopitem

\startitem
    \type {lpeg}, by Roberto Ierusalimschy, \hyphenatedurl
    {http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html}. (version 0.10.2) This
    library is not \UNICODE|-|aware, but interprets strings on a
    byte|-|per|-|byte basis. This mainly means that \type {lpeg.S} cannot be
    used with \UTF\ characters encoded in more than two bytes, and thus \type
    {lpeg.S} will look for one of those two bytes when matching, not the
    combination of the two. The same is true for \type {lpeg.R}, although the
    latter will display an error message if used with multibyte characters.
    Therefore \type {lpeg.R('aä')} results in the message \type {bad argument
    #1 to 'R' (range must have two characters)}, since to \type {lpeg}, \type {ä}
    is two 'characters' (bytes), so \type {aä} totals three. In practice this is
    no real issue.
\stopitem

\startitem
    \type {lzlib}, by Tiago Dionizio, \hyphenatedurl
    {http://luaforge.net/projects/lzlib/}. (version 0.2)
\stopitem

\startitem
    \type {md5}, by Roberto Ierusalimschy \hyphenatedurl
    {http://www.inf.puc-rio.br/~roberto/md5/md5-5/md5.html}.
\stopitem

\startitem
    \type {luasocket}, by Diego Nehab \hyphenatedurl
    {http://w3.impa.br/~diego/software/luasocket/} (version 2.0.2). The \type
    {.lua} support modules from \type {luasocket} are also preloaded inside the
    executable, there are no external file dependencies.
\stopitem

\stopitemize

\stopchapter

\stopcomponent