beta-0.89.2

[luatex.git] / manual / luatex-languages.tex

\environment luatex-style
\environment luatex-logos

\startcomponent luatex-languages

\startchapter[reference=languages,title={Languages and characters, fonts and glyphs}]

\LUATEX's internal handling of the characters and glyphs that eventually become
typeset is quite different from the way \TEX82 handles those same objects. The
easiest way to explain the difference is to focus on unrestricted horizontal mode
(i.e.\ paragraphs) and hyphenation first. Later on, it will be easy to deal
with the differences that occur in horizontal and math modes.

In \TEX82, the characters you type are converted into \type {char_node} records
when they are encountered by the main control loop. \TEX\ attaches and processes
the font information while creating those records, so that the resulting \quote
{horizontal list} contains the final forms of ligatures and implicit kerning.
This packaging is needed because we may want to get the effective width of for
instance a horizontal box.

When it becomes necessary to hyphenate words in a paragraph, \TEX\ converts (one
word at time) the \type {char_node} records into a string array by replacing
ligatures with their components and ignoring the kerning. Then it runs the
hyphenation algorithm on this string, and converts the hyphenated result back
into a \quote {horizontal list} that is consecutively spliced back into the
paragraph stream. Keep in mind that the paragraph may contain unboxed horizontal
material, which then already contains ligatures and kerns and the words therein
are part of the hyphenation process.

The \type {char_node} records are somewhat misnamed, as they are glyph positions
in specific fonts, and therefore not really \quote {characters} in the linguistic
sense. There is no language information inside the \type {char_node} records.
Instead, language information is passed along using \type {language whatsit}
records inside the horizontal list.

In \LUATEX, the situation is quite different. The characters you type are always
converted into \type {glyph_node} records with a special subtype to identify them
as being intended as linguistic characters. \LUATEX\ stores the needed language
information in those records, but does not do any font|-|related processing at
the time of node creation. It only stores the index of the current font.

When it becomes necessary to typeset a paragraph, \LUATEX\ first inserts all
hyphenation points right into the whole node list. Next, it processes all the
font information in the whole list (creating ligatures and adjusting kerning),
and finally it adjusts all the subtype identifiers so that the records are \quote
{glyph nodes} from now on.

That was the broad overview. The rest of this chapter will deal with the minutiae
of the new process.

\section[charsandglyphs]{Characters and glyphs}

\TEX82 (including \PDFTEX) differentiates between \type {char_node}s and \type
{lig_node}s. The former are simple items that contained nothing but a \quote
{character} and a \quote {font} field, and they lived in the same memory as
tokens did. The latter also contained a list of components, and a subtype
indicating whether this ligature was the result of a word boundary, and it was
stored in the same place as other nodes like boxes and kerns and glues.

In \LUATEX, these two types are merged into one, somewhat larger structure called
a \type {glyph_node}. Besides having the old character, font, and component
fields, and the new special fields like \quote {attr}
(see~\in{section}[glyphnodes]), these nodes also contain:

\startitemize

\startitem A subtype, split into four main types:

    \startitemize
        \startitem
            \type {character}, for characters to be hyphenated: the lowest bit
            (bit 0) is set to 1.
        \stopitem
        \startitem
            \type {glyph}, for specific font glyphs: the lowest bit (bit 0) is
            not set.
        \stopitem
        \startitem
            \type {ligature}, for ligatures (bit 1 is set)
        \stopitem
        \startitem
            \type {ghost}, for \quote {ghost objects} (bit 2 is set)
        \stopitem
    \stopitemize

    The latter two make further use of two extra fields (bits 3 and 4):

    \startitemize
        \startitem
            \type {left}, for ligatures created from a left word boundary and for
            ghosts created from \type {\leftghost}
        \stopitem
        \startitem
            \type {right}, for ligatures created from a right word boundary and
            for ghosts created from \type {\rightghost}
        \stopitem
   \stopitemize

   For ligatures, both bits can be set at the same time (in case of a
   single|-|glyph word).

\stopitem

\startitem
    \type {glyph_node}s of type \quote {character} also contain language data,
    split into four items that were current when the node was created: the
    \type {\setlanguage} (15 bits), \type {\lefthyphenmin} (8 bits), \type
    {\righthyphenmin} (8 bits), and \type {\uchyph} (1 bit).
\stopitem

\stopitemize

Incidentally, \LUATEX\ allows 16383 separate languages, and words can be 256
characters long. The language is stored with each character. You can set
\type {\firstvalidlanguage} to for instance~1 and make thereby language~0
an ignored hyphenation language.

The new primitive \type {\hyphenationmin} can be used to signal the minimal length
of a word. This value stored with the (current) language.

Because the \type {\uchyph} value is saved in the actual nodes, its handling is
subtly different from \TEX82: changes to \type {\uchyph} become effective
immediately, not at the end of the current partial paragraph.

Typeset boxes now always have their language information embedded in the nodes
themselves, so there is no longer a possible dependency on the surrounding
language settings. In \TEX82, a mid-paragraph statement like \type {\unhbox0} would
process the box using the current paragraph language unless there was a
\type {\setlanguage} issued inside the box. In \LUATEX, all language variables are
already frozen.

In traditional \TEX\ the process of hyphenation is driven by so called lccodes.
In \LUATEX\ we made this dependency less strong. There are several strategies
possible. When you do nothing, the currently used lccodes are used, when loading
patterns, setting exceptions or hyphenating a list.

When you set \type {\savinghyphcodes} to a value larger than zero the current set of
lccodes will be saved with the language. In that case changing a lccode afterwards
has no effect. However, you can adapt the set with:

\starttyping
\hjcode`a=`a
\stoptyping

This change is global which makes sense if you keep in mind that the moment that
hyphenation happens is (normally) when the paragraph or a horizontal box is
constructed. When \type {\savinghyphcodes} was zero when the language got
initialized you start out with nothing, otherwise you already have a set.

Carrying all this information with each glyph would give too much overhead and
also make the definition more complex. A solution with hj codesets was considered
but rejected because in practice the current approach is sufficient and it would
not be compatible anyway.

Beware: the values are always saved in the format, independent of the setting
of \type {\savinghyphcodes} at the mnoment the format is dumped.

\section{The main control loop}

In \LUATEX's main loop, almost all input characters that are to be typeset are
converted into \type {glyph} node records with subtype \quote {character}, but
there are a few exceptions.

First, the \type {\accent} primitives creates nodes with subtype \quote {glyph}
instead of \quote {character}: one for the actual accent and one for the
accentee. The primary reason for this is that \type {\accent} in \TEX82 is
explicitly dependent on the current font encoding, so it would not make much
sense to attach a new meaning to the primitive's name, as that would invalidate
many old documents and macro packages. A secondary reason is that in \TEX82,
\type {\accent} prohibits hyphenation of the current word. Since in \LUATEX\
hyphenation only takes place on \quote {character} nodes, it is possible to
achieve the same effect.

This change of meaning did happen with \type {\char}, that now generates \quote
{glyph} nodes with a character subtype. In traditional \TEX\ there was a strong
relationship betwene the 8|-|bit input encoding, hyphenation and glyph staken
from a font. In \LUATEX\ we have \UTF\ input, and in most cases this maps
directly to a character in a font, apart from glyph replacement in the font
engine. If you want to access arbitrary glyphs in a font directly you can alwasy
use \LUA\ to do so, because fonts are available as \LUA\ table.

Second, all the results of processing in math mode eventually become nodes with
\quote {glyph} subtypes.

Third, the \ALEPH|-|derived commands \type {\leftghost} and \type {\rightghost}
create nodes of a third subtype: \quote {ghost}. These nodes are ignored
completely by all further processing until the stage where inter|-|glyph kerning
is added.

Fourth, automatic discretionaries are handled differently. \TEX82 inserts an
empty discretionary after sensing an input character that matches the \type
{\hyphenchar} in the current font. This test is wrong, in our opinion: whether or
not hyphenation takes place should not depend on the current font, it is a
language property.

In \LUATEX, it works like this: if \LUATEX\ senses a string of input characters
that matches the value of the new integer parameter \type {\exhyphenchar}, it will
insert an explicit discretionary after that series of nodes. Initex sets the \type
{\exhyphenchar=`\-}. Incidentally, this is a global parameter instead of a
language-specific one because it may be useful to change the value depending on
the document structure instead of the text language.

The insertion of discretionaries after a sequence of explicit hyphens happens at
the same time as the other hyphenation processing, {\it not\/} inside the main
control loop.

The only use \LUATEX\ has for \type {\hyphenchar} is at the check whether a word
should be considered for hyphenation at all. If the \type {\hyphenchar} of the font
attached to the first character node in a word is negative, then hyphenation of
that word is abandoned immediately. {\bf This behavior is added for backward
compatibility only, and the use of \type {\hyphenchar=-1} as a means of
preventing hyphenation should not be used in new \LUATEX\ documents.}

Fifth, \type {\setlanguage} no longer creates whatsits. The meaning of \type
{\setlanguage} is changed so that it is now an integer parameter like all others.
That integer parameter is used in \type {\glyph_node} creation to add language
information to the glyph nodes. In conjunction, the \type {\language} primitive is
extended so that it always also updates the value of \type {\setlanguage}.

Sixth, the \type {\noboundary} command (this command prohibits word boundary
processing where that would normally take place) now does create whatsits. These
whatsits are needed because the exact place of the \type {\noboundary} command in
the input stream has to be retained until after the ligature and font processing
stages.

Finally, there is no longer a \type {main_loop} label in the code. Remember that
\TEX82 did quite a lot of processing while adding \type {char_nodes} to the
horizontal list? For speed reasons, it handled that processing code outside of
the \quote {main control} loop, and only the first character of any \quote {word}
was handled by that \quote {main control} loop. In \LUATEX, there is no longer a
need for that (all hard work is done later), and the (now very small) bits of
character|-|handling code have been moved back inline. When \type
{\tracingcommands} is on, this is visible because the full word is reported,
instead of just the initial character.

\section[patternsexceptions]{Loading patterns and exceptions}

The hyphenation algorithm in \LUATEX\ is quite different from the one in \TEX82,
although it uses essentially the same user input.

After expansion, the argument for \type {\patterns} has to be proper \UTF8 with
individual patterns separated by spaces, no \type {\char} or \type {\chardef}d
commands are allowed. The current implementation is even more strict, and will
reject all non|-|\UNICODE\ characters, but that will be changed in the future.
For now, the generated errors are a valuable tool in discovering font-encoding
specific pattern files.

Likewise, the expanded argument for \type {\hyphenation} also has to be proper
\UTF8, but here a tiny little bit of extra syntax is provided:

\startitemize[n]
\startitem
    Three sets of arguments in curly braces (\type {{}{}{}}) indicates a desired
    complex discretionary, with arguments as in \type {\discretionary}'s command in
    normal document input.
\stopitem
\startitem
    A \type {-} indicates a desired simple discretionary, cf.\ \type {\-} and \type
    {\discretionary{-}{}{}} in normal document input.
\stopitem
\startitem
    Internal command names are ignored. This rule is provided especially for \type
    {\discretionary}, but it also helps to deal with \type {\relax} commands that
    may sneak in.
\stopitem
\startitem
    An \type {=} indicates a (non|-|discretionary) hyphen in the document input.
\stopitem
\stopitemize

The expanded argument is first converted back to a space-separated string while
dropping the internal command names. This string is then converted into a
dictionary by a routine that creates key|-|value pairs by converting the other
listed items. It is important to note that the keys in an exception dictionary
can always be generated from the values. Here are a few examples:

\starttabulate[|l|l|l|]
\NC \ssbf value \NC \ssbf implied key (input) \NC \ssbf effect \NC\NR
\NC \type {ta-ble} \NC table \NC \type {ta\-ble} ($=$ \type {ta\discretionary{-}{}{}ble}) \NC\NR
\NC \type {ba{k-}{}{c}ken} \NC backen \NC \type {ba\discretionary{k-}{}{c}ken} \NC\NR
\stoptabulate

The resultant patterns and exception dictionary will be stored under the language
code that is the present value of \type {\language}.

In the last line of the table, you see there is no \type {\discretionary} command
in the value: the command is optional in the \TEX-based input syntax. The
underlying reason for that is that it is conceivable that a whole dictionary of
words is stored as a plain text file and loaded into \LUATEX\ using one of the
functions in the \LUA\ \type {lang} library. This loading method is quite a bit
faster than going through the \TEX\ language primitives, but some (most?) of that
speed gain would be lost if it had to interpret command sequences while doing so.

It is possible to specify extra hyphenation points in compound words by using
\type {{-}{}{-}} for the explicit hyphen character (replace \type {-} by the
actual explicit hyphen character if needed). For example, this matches the word
\quote {multi|-|word|-|boundaries} and allows an extra break inbetweem \quote
{boun} and \quote {daries}:

\starttyping
\hyphenation{multi{-}{}{-}word{-}{}{-}boun-daries}
\stoptyping

The motivation behind the \ETEX\ extension \type {\savinghyphcodes} was that
hyphenation heavily depended on font encodings. This is no longer true in
\LUATEX, and the corresponding primitive is ignored pending complete removal. The
future semantics of \type {\uppercase} and \type {\lowercase} are still under
consideration, no changes have taken place yet.

\section{Applying hyphenation}

The internal structures \LUATEX\ uses for the insertion of discretionaries in
words is very different from the ones in \TEX82, and that means there are some
noticeable differences in handling as well.

First and foremost, there is no \quote {compressed trie} involved in hyphenation.
The algorithm still reads \PATGEN-generated pattern files, but \LUATEX\ uses a
finite state hash to match the patterns against the word to be hyphenated. This
algorithm is based on the \quote {libhnj} library used by \OPENOFFICE, which in
turn is inspired by \TEX. The memory allocation for this new implementation is
completely dynamic, so the \WEBC\ setting for \type {trie_size} is ignored.

Differences between \LUATEX\ and \TEX82 that are a direct result of that:

\startitemize
\startitem
    \LUATEX\ happily hyphenates the full \UNICODE\ character range.
\stopitem
\startitem
    Pattern and exception dictionary size is limited by the available memory
    only, all allocations are done dynamically. The trie|-|related settings in
    \type {texmf.cnf} are ignored.
\stopitem
\startitem
    Because there is no \quote {trie preparation} stage, language patterns never
    become frozen. This means that the primitive \type {\patterns} (and its \LUA\
    counterpart \type {lang.patterns}) can be used at any time, not only in
    ini\TEX.
\stopitem
\startitem
    Only the string representation of \type {\patterns} and \type {\hyphenation} is
    stored in the format file. At format load time, they are simply
    re|-|evaluated. It follows that there is no real reason to preload languages
    in the format file. In fact, it is usually not a good idea to do so. It is
    much smarter to load patterns no sooner than the first time they are actually
    needed.
\stopitem
\startitem
    \LUATEX\ uses the language-specific variables \type {\prehyphenchar} and \type
    {\posthyphenchar} in the creation of implicit discretionaries, instead of
    \TEX82's \type {\hyphenchar}, and the values of the language|-|specific variables
    \type {\preexhyphenchar} and \type {\postexhyphenchar} for explicit
    discretionaries (instead of \TEX82's empty discretionary).
\stopitem
\startitem
    The value of the two counters related to hyphenation, \type {hyphenpenalty}
    and \type {exhyphenpenalty}, are now stored in the discretionary nodes. This
    permits a local overload for explicit \type {\discretionary} commands. The
    value current when the hyphenation pass is applied is used. When no callbacks
    are used this is compatible with traditional \TEX. When you apply the \LUA\
    \type {lang.hyphenate} function the current values are used.
\stopitem
\stopitemize

Because we store penalties in the disc node the \type {\discretionary} command has
been extended to accept an optional penalty specification, so you can do the
following:

\startbuffer
\hsize1mm
1:foo{\hyphenpenalty 10000\discretionary{}{}{}}bar\par
2:foo\discretionary penalty 10000 {}{}{}bar\par
3:foo\discretionary{}{}{}bar\par
\stopbuffer

\typebuffer

This results in:

\blank \start \getbuffer \stop \blank

Inserted characters and ligatures inherit their attributes from the nearest glyph
node item (usually the preceding one, but the following one for the items
inserted at the left-hand side of a word).

Word boundaries are no longer implied by font switches, but by language switches.
One word can have two separate fonts and still be hyphenated correctly (but it
can not have two different languages, the \type {\setlanguage} command forces a
word boundary).

All languages start out with \type {\prehyphenchar=`\-}, \type {\posthyphenchar=0},
\type {\preexhyphenchar=0} and \type {\postexhyphenchar=0}. When you assign the
values of one of these four parameters, you are actually changing the settings
for the current \type {\language}, this behavior is compatible with \type {\patterns}
and \type {\hyphenation}.

\LUATEX\ also hyphenates the first word in a paragraph. Words can be up to 256
characters long (up from 64 in \TEX82). Longer words generate an error right now,
but eventually either the limitation will be removed or perhaps it will become
possible to silently ignore the excess characters (this is what happens in
\TEX82, but there the behavior cannot be controlled).

If you are using the \LUA\ function \type {lang.hyphenate}, you should be aware
that this function expects to receive a list of \quote {character} nodes. It will
not operate properly in the presence of \quote {glyph}, \quote {ligature}, or
\quote {ghost} nodes, nor does it know how to deal with kerning. In the near
future, it will be able to skip over \quote {ghost} nodes, and we may add a less
fuzzy function you can call as well.

The hyphenation exception dictionary is maintained as key|-|value hash, and that
is also dynamic, so the \type {hyph_size} setting is not used either.

\section{Applying ligatures and kerning}

After all possible hyphenation points have been inserted in the list, \LUATEX\
will process the list to convert the \quote {character} nodes into \quote {glyph}
and \quote {ligature} nodes. This is actually done in two stages: first all
ligatures are processed, then all kerning information is applied to the result
list. But those two stages are somewhat dependent on each other: If the used font
makes it possible to do so, the ligaturing stage adds virtual \quote {character}
nodes to the word boundaries in the list. While doing so, it removes and
interprets \type {noboundary} nodes. The kerning stage deletes those word
boundary items after it is done with them, and it does the same for \quote
{ghost} nodes. Finally, at the end of the kerning stage, all remaining \quote
{character} nodes are converted to \quote {glyph} nodes.

This work separation is worth mentioning because, if you overrule from \LUA\ only
one of the two callbacks related to font handling, then you have to make sure you
perform the tasks normally done by \LUATEX\ itself in order to make sure that the
other, non|-|overruled, routine continues to function properly.

Work in this area is not yet complete, but most of the possible cases are handled
by our rewritten ligaturing engine. We are working hard to make sure all of the
possible inputs will become supported soon.

For example, take the word \type {office}, hyphenated \type {of-fice}, using a
\quote {normal} font with all the \type {f}-\type {f} and \type {f}-\type {i}
type ligatures:

\starttabulate[|l|l|]
\NC Initial:               \NC \type {{o}{f}{f}{i}{c}{e}}             \NC\NR
\NC After hyphenation:     \NC \type {{o}{f}{{-},{},{}}{f}{i}{c}{e}}  \NC\NR
\NC First ligature stage:  \NC \type {{o}{{f-},{f},{<ff>}}{i}{c}{e}}  \NC\NR
\NC Final result:          \NC \type {{o}{{f-},{<fi>},{<ffi>}}{c}{e}} \NC\NR
\stoptabulate

That's bad enough, but let us assume that there is also a hyphenation point
between the \type {f} and the \type {i}, to create \type {of-f-ice}. Then the
final result should be:

\starttyping
{o}{{f-},
    {{f-},
     {i},
     {<fi>}},
    {{<ff>-},
     {i},
     {<ffi>}}}{c}{e}
\stoptyping

with discretionaries in the post-break text as well as in the replacement text of
the top-level discretionary that resulted from the first hyphenation point.

Here is that nested solution again, in a different representation:

\starttabulate[|l|l|l|l|]
\NC         \NC pre              \NC post          \NC replace           \NC \NR
\NC topdisc \NC \type {f-}$^1$   \NC sub1          \NC sub2              \NC \NR
\NC sub1    \NC \type {f-}$^2$   \NC \type {i}$^3$ \NC \type {<fi>}$^4$  \NC \NR
\NC sub2    \NC \type {<ff>-}$^5$\NC \type {i}$^6$ \NC \type {<ffi>}$^7$ \NC \NR
\stoptabulate

When line breaking is choosing its breakpoints, the following fields will
eventually be selected:

\starttabulate[|l|l|l|]
\NC \type {of-f-ice} \NC \type {f-}$^1$    \NC \NR
\NC                  \NC \type {f-}$^2$    \NC \NR
\NC                  \NC \type {i}$^3$     \NC \NR
\NC \type {of-fice}  \NC \type {f-}$^1$    \NC \NR
\NC                  \NC \type {<fi>}$^4$  \NC \NR
\NC \type {off-ice}  \NC \type {<ff>-}$^5$ \NC \NR
\NC                  \NC \type {i}$^6$     \NC \NR
\NC \type {office}   \NC \type {<ffi>}$^7$ \NC \NR
\stoptabulate

The current solution in \LUATEX\ is not able to handle nested discretionaries,
but it is in fact smart enough to handle this fictional \type {of-f-ice} example.
It does so by combining two sequential discretionary nodes as if they were a
single object (where the second discretionary node is treated as an extension of
the first node).

One can observe that the \type {of-f-ice} and \type {off-ice} cases both end with
the same actual post replacement list (\type {i}), and that this would be the
case even if that \type {i} was the first item of a potential following ligature
like \type {ic}. This allows \LUATEX\ to do away with one of the fields, and thus
make the whole stuff fit into just two discretionary nodes.

The mapping of the seven list fields to the six fields in this discretionary node
pair is as follows:

\starttabulate[|l|p|]
\NC \bf field            \NC \bf description \NC \NR
\NC \type {disc1.pre}     \NC \type {f-}$^1$  \NC \NR
\NC \type {disc1.post}    \NC \type {<fi>}$^4$  \NC \NR
\NC \type {disc1.replace} \NC \type {<ffi>}$^7$ \NC \NR
\NC \type {disc2.pre}     \NC \type {f-}$^2$  \NC \NR
\NC \type {disc2.post}    \NC \type {i}$^{3{,}6}$\NC \NR
\NC \type {disc2.replace} \NC \type {<ff>-}$^5$\NC \NR
\stoptabulate

What is actually generated after ligaturing has been applied is therefore:

\starttyping
{o}{{f-},
    {<fi>},
    {<ffi>}}
   {{f-},
    {i},
    {<ff>-}}{c}{e}
\stoptyping

The two discretionaries have different subtypes from a discretionary appearing on
its own: the first has subtype 4, and the second has subtype 5. The need for
these special subtypes stems from the fact that not all of the fields appear in
their \quote {normal} location. The second discretionary especially looks odd,
with things like the \type {<ff>-} appearing in \type {disc2.replace}. The fact
that some of the fields have different meanings (and different processing code
internally) is what makes it necessary to have different subtypes: this enables
\LUATEX\ to distinguish this sequence of two joined discretionary nodes from the
case of two standalone discretionaries appearing in a row.

Of course there is still that relationship with fonts: ligatures can be implemented by
mapping a sequence of glyphs onto one glyph, but also by selective replacement and
kerning. This means that the above examples are just representing the traditional
approach.

\section{Breaking paragraphs into lines}

This code is still almost unchanged, but because of the above|-|mentioned changes
with respect to discretionaries and ligatures, line breaking will potentially be
different from traditional \TEX. The actual line breaking code is still based on
the \TEX82 algorithms, and it does not expect there to be discretionaries inside
of discretionaries.

But that situation is now fairly common in \LUATEX, due to the changes to the
ligaturing mechanism. And also, the \LUATEX\ discretionary nodes are implemented
slightly different from the \TEX82 nodes: the \type {no_break} text is now
embedded inside the disc node, where previously these nodes kept their place in
the horizontal list (the discretionary node contained a counter indicating how
many nodes to skip).

The combined effect of these two differences is that \LUATEX\ does not always use
all of the potential breakpoints in a paragraph, especially when fonts with many
ligatures are used.

\stopchapter

\stopcomponent