Move multiple translations

[apertium.git] / apertium2-documentation-en / documentation.tex

\documentclass [12pt,a4paper]{book} 
%\ifx\pdfoutput\undefined
%\usepackage[dvips]{graphicx} 
%\else 
%\usepackage[pdftex]{graphicx}
%\usepackage{type1cm} 
%\fi 
\usepackage[dvips]{graphicx}
\usepackage{rotating} 
\usepackage{palatino,helvet}
\usepackage[english]{babel} 
\usepackage[latin1]{inputenc}
\usepackage{sectsty} 
\usepackage{alltt} 
\usepackage[small,bf]{caption}
\usepackage{url} 
\usepackage{rotating} 
\usepackage{longtable} 
% \usepackage{tocvsec2}

% \allsectionsfont{\sffamily}
% 


\setcounter{secnumdepth}{3} 
%\setcounter{tocdepth}{3} %%(so that index reaches the third level, more specific)

% Line break after \paragraph 
\makeatletter % so that '@' is recognized as a normal character
\renewcommand{\paragraph}{\@startsection{paragraph}{4}{\z@}{-3.25ex \@plus
-1ex \@minus -.2ex}{1.5ex \@plus .2ex}{\normalfont\normalsize\bfseries}}
\makeatother % so that '@' is again a special character



%  \newcommand{\nota}[1]{ \begin{small}
%   \begin{quote}
%   \begin{sf} 
%   [Nota: #1]
%   \end{sf}
%   \end{quote}
%   \end{small} 
% }

\newcommand{\nota}[1]{}


\newcommand{\notavisible}[1]{ 
  \begin{small}
    \begin{quote}
      \begin{sf}
        [#1]
      \end{sf}
    \end{quote}
  \end{small}
}


%% Project ``Open Source Machine Translation for the languages of Spain (FIT-340101-2004-3) \\[.5ex] 
\frontmatter

\title{\sffamily\bfseries Documentation of the Open-Source
Shallow-Transfer Machine Translation Platform \emph{Apertium}}
%%\date{28 June 2005}



 \author{\textbf{AUTHORS}:\\Mikel L. Forcada\\Boyan Ivanov
Bonev\\Sergio Ortiz Rojas\\ Juan Antonio Pérez Ortiz \\
Gema Ramírez Sánchez\\Felipe Sánchez
 Martínez\\ Carme Armentano-Oller\\ Marco A.\ Montava \\ Francis M.\ Tyers\\\\\textbf{EDITOR}:\\Mireia Ginestí
 Rosell\\[0.8cm]\\Departament de Llenguatges i Sistemes
 Informàtics\\Universitat d'Alacant} 
%% \textit{Eleka Ingeniaritza Linguistikoa} \\ 
%% \textit{Zelai Haundi Kalea, 3} \\ 
%% \textit{Osinalde Industrialdea} \\ 
%% \textit{20170 Usurbil}}



\begin{document} 
\pagestyle{headings} 
%\maxtocdepth{subsubsection}
%\maxtocdepth{paragraph}

%\settocdepth{subsubsection}

\maketitle


\newpage \thispagestyle{empty}

\bigskip
\begin{quote} Copyright \copyright 2007 Grup Transducens, Universitat
  d'Alacant.  Permission is granted to copy, distribute and/or modify
  this document under the terms of the GNU Free Documentation License,
  Version 1.2 or any later version published by the Free Software
  Foundation; with no Invariant Sections, no Front-Cover Texts, and no
  Back-Cover Texts. A copy of the license can be found in
  \url{http://www.gnu.org/copyleft/fdl.html}.  

  % The unofficial
%   translation of the license to Spanish can be found at
%   \url{http://gugs.sindominio.net/licencias/gfdl-1.2-es.html}, the
%   unofficial translation to Catalan can be found at
%   \url{http://www.softcatala.org/llicencies/fdl-ca.html}, and the
%   unofficial translation to Galician can be found at
%   \url{http://members.tripod.com.br/RamonFlores/GNU/gpl.html}.


\notavisible{Shouldn't we license this under GPL or another license that is free in Debian terms?

Make sure 1.2 is the right license.

Perhaps we don't want ``or any later version''.

Check the author list. We might have forgotten someone.}

\end{quote}


\bigskip
    

\tableofcontents

\newpage

\mainmatter
\chapter*{Introduction}\addcontentsline{toc}{chapter}{Introduction}


This documentation describes the Apertium platform, one of the
open-source machine translation systems which originated within the
project "Open-Source Machine Translation for the Languages of Spain"
("Traducción automática de código abierto para las lenguas del estado
español")\nota{Posem un resum de les dades del projecte en un apèndix
  (codi, durada, finançament, participants, etc.) i hi fem referència
  ací? - Mikel}. It is a shallow-transfer machine translation system,
initially designed for the translation between related language pairs,
although some of its components have been also used in the
deep-transfer architecture (\emph{Matxin}) that has been developed in
the same project for the pair Spanish-Basque. \emph{Apertium} can
translate at present between the pairs Spanish-Galician,
Spanish--Catalan\footnote{With the name \emph{Catalan} we refer also
  to the Valencian dialectal variant of this language.}
Catalan-Occitan, Catalan-French, and can be used to build translators
between other related language pairs, such as
Danish-Swedish,Czech--Slovak, etc.  \notavisible{Update the
  language-pair list!}  \notavisible{I think it is very important to say in this paragraph that the system has been extended}


\notavisible{The next paragraph needs updating or generalizing:}
Existing machine translation systems available at present for the
pairs \texttt{es}--\texttt{ca} and \texttt{es}--\texttt{gl} are mostly
commercial or use proprietary technologies, which makes them very hard
to adapt to new usages; furthermore, they use different technologies
across language pairs, which makes it very difficult to integrate them
in a single multilingual content management system.

One of the main novelties of the architecture described here is that
it has been released under open-source licenses (in most cases, GNU
GPL; some data still have a Creative Commons license) and is
distributed free of charge. This means that anyone having the
necessary computational and linguistic skills will be able to adapt or
enhance the platform or the language-pair data to create a new machine
translation system, even for other pairs of related languages. The
licenses chosen make these improvements immediately available to
everyone.  We therefore expect that the introduction of this of
open-source machine translation architecture will solve some of the
mentioned problems (having different technologies for different pairs,
closed-source architectures being hard to adapt to new uses, etc.) and
promote the exchange of existing linguistic data through the use of
the XML-based formats defined in this documentation. On the other
hand, we think that it will help shift the current business model from
a license-centred one to a services-centred one.

It is worth mentioning that "Open-Source Machine Translation for the
Languages of Spain" was the first large open-source machine
translation project funded by the central Spanish Government, although
the adoption of open-source software by the Spanish governments is not
new.

\notavisible{Don't forget about the other funding agencies supporting open source MT; this needs some contextualization, relating to funding, etc. Mention later funding and refer to the appropriate section.}

This documentation describes in detail the characteristics of the
Apertium platform, and is organized as follows:


\begin{itemize}
\item Chapter \ref{ss:descrarq}: \textbf{general description} of the
shallow-transfer machine translation system and of the modules that
make it up.

\item Chapter \ref{se:flujodatos}: description of the \textbf{format
of the data stream} that circulates from one module to the next one.

\item Chapter \ref{se:especificmodulos}: \textbf{specification of the
modules} of the system. For each module there is a description of: the
\textit{program} and its characteristics,  the \textit{format of the data}
that the module uses, and the \textit{compilers}  used for it.
This chapter is divided in the following sections:
  \begin{itemize}
  \item [-]Section \ref{ss:modproclex}: \emph{Lexical processing
    modules}, where the morphological analyser, the lexical transfer
    module, the morphological generator and the post-generator are
    described (Section \ref{ss:funcproclex}), along with the format of
    the dictionaries used by these modules (section
    \ref{ss:diccionarios}) and their compilers (section
    \ref{se:compiladoresdic})
  \item [-]Section \ref{ss:tagger}: \emph{Part-of-speech Tagger},
    which describes the tagger (Section \ref{functagger}) and the
    format of the linguistic data used by the tagger (section
    \ref{datostagger}.
% MLF 20060328 elimina % y el compilador % correspondiente (apartado
%\ref{ss:gentagger})

\nota{falta parlar del lextor, i afegir-ho a tot arreu on es parli
dels mòduls del sistema}
     
  \item [-]Section \ref{se:pretransfer}: \emph{Pre-transfer module},
    which describes the module that runs before the structural
    transfer module to perform some operations on multiword units
  \item [-]Section \ref{ss:transfer}: \emph{Structural transfer
    module}, where there is a description of the program (section
    \ref{functransfer}) and of the format of the structural transfer
    rules (Section \ref{formatotransfer}).
% MLF 20060328 % y el % compilador correspondiente (apartado
% \ref{gentransfer})
  \item [-]Section \ref{se:desformat}: \emph{De-formatter and
    Re-formatter}, which describes these modules (section
    \ref{ss:formato}), the rules for format processing (section
    \ref{ss:reglasformato}) and how these modules are generated
    (Section \ref{se:gendeformat})
 
  \end{itemize}



\item Chapter \ref{se:instalacion}: it describes the way to 
\textbf{install the system} and to \textbf{run the translator}.

\item Chapter \ref{se:datosling}: here you will find an explanation of
  how to \textbf{modify the linguistic data} used by the translator,
  that is, the dictionaries, the part-of-speech disambiguation data
  and the structural transfer rules created in this project for
  Spanish, Catalan and Galician. Furthermore, it contains a brief
  description of the characteristics of the
available data for these three language pairs.
\notavisible{I would try to be more general, and perhaps remove this section or update with some other pairs. Any ideas on how to do this?}


\nota{Es diuen a tot arreu els noms de programa i en quin paquet
estan?}


\end{itemize}


The files which this documentation refers to can be found at and
downloaded from the project web page in Sourceforge:
\url{http://apertium.sourceforge.net/}.  From this page you can
download the packages needed for installation, as well as view the
individual files in the SVN (main) and CVS (residual) repositories of
the project.  The machine translation systems for the different
language pairs can also be tested in Internet at
\url{http://xixona.dlsi.ua.es/apertium/}.

\notavisible{Shouldn't we mention the debugging interfaces?}
\notavisible{Should we define SVN and CVS?}

%El presente documento tiene algunas secciones que están incompletas o
%no han sido escritas todavía.


\paragraph*{Acknowledgements:} The present work has benefited from the
contribution of many people and institutions:
\begin{itemize}
\item The Spanish Ministry of Industry, Commerce and Tourism has
  funded the development of this toolbox through the projects
  ``Open-Source Machine Translation for the Languages of Spain'', code
  FIT-340101-2004-3, and its extension FIT-340001-2005-2, and
  ``EurOpenTrad: Open-Source Advanced Machine Translation for the
  European Integration of the Languages of Spain'', code
  FIT-350101-2006-5, all of them belonging to the PROFIT program.



\item Workers and scholars from other machine translation projects at
the Universitat d'Alacant: Míriam Antunes Scalco, Carme Armentano i
Oller, Raül Canals i Marote, Alicia Garrido Alenda, Patrícia Gilabert
i Zarco, Maribel Guardiola i Savall, Javier Herrero Vicente, Amaia
Iturraspe Bellver, Sandra Montserrat i Buendia, Hermínia Pastor Pina,
Antonio Pertusa Ibáñez, Francisco Javier Ramos Salas, Marcial Samper
Asensio and Miguel Sánchez Molina.
\item The companies and institutions that have funded these other
machine translation projects: Spanish Ministry of Science and
Technology, Caja de Ahorros del Mediterráneo, Universitat d'Alacant
and Portal Universia, S.A.
\item Iñaki Alegria, from the Ixa group of the Euskal Herriko
Unibertsitatea (University of the Basque Country), for his close
reading of previous versions of this document.
\end{itemize}

\vspace{12cm}




\chapter[The translation engine]{The shallow-transfer machine
translation engine }
\label{ss:descrarq}


This chapter describes briefly the structure of the shallow-transfer
machine translation engine, which is largely based on that of the
existing systems for Spanish--Catalan \textsf{interNOSTRUM}
\cite{canals01b,garridoalenda01p,garrido99j} and for
Spanish--Portuguese \textsf{Traductor Universia} \cite{garrido03p,
gilabert03j}, both developed by the Transducens group of the
Universitat d'Alacant.  It is a classical indirect translation system
that uses a partial syntactic transfer strategy similar to the one
used by some commercial MT systems for personal computers.


The design of the system makes it possible to produce MT systems that
are \emph{fast} (translating tens of thousands of words per second in
ordinary desktop computers) and that achieve results that are, in spite of
the errors, reasonably intelligible and easily correctable. In the
case of related languages such as the ones involved in the project
(Spanish, Galician, Catalan), a mechanical word-for-word translation
(with a fixed equivalent) would produce errors that, in most of the
cases, can be solved with a quite rudimentary analysis (a
morphological analysis followed by a superficial, local and partial
syntactic analysis) and with an appropriate treatment of lexical
ambiguities (mainly due to homography). The design of our system
follows this approach with very interesting results. The Apertium
architecture uses finite-state transducers for lexical processing,
hidden Markov models for part-of-speech tagging and finite-state-based
chunking for structural transfer.


The translation engine consists of an 8-module \emph{assembly line},
which is represented in Figure \ref{fg:modules}.  To ease diagnosis
and independent testing, modules communicate between them using text
streams.  This way, the input and output of the modules can be checked
at any moment and, when an error in the translation process is
detected, it is easy to test the output of each module separately to
track down the origin of the error. At the same time, communication
via text allows for some of the modules to be used in isolation,
independently form the rest of the MT system, for other
natural-language processing tasks, and enables the construction of
prototypes with modified or additional modules.

We decided to encode linguistic data files in
XML\footnote{\url{http://www.w3.org/XML/}}-based formats due to its
interoperability, its independence on the character set and the
availability of many tools and libraries that make easy the analysis
of data  in this format. As stated in \cite{ide00}, XML is the
emerging standard for data representation and exchange in
Internet. Technologies around XML include very powerful mechanisms for
accessing and editing XML documents, which will probably have a
significant impact on the development of tools for natural language
processing and annotated corpora.


The modules Apertium consists of are the following:

\begin{figure*} {\footnotesize \setlength{\tabcolsep}{0.5mm}
\begin{center}
\begin{tabular}{cccccccc} 
\\
\parbox{0.7cm}{SL text} \\ 
$\downarrow$ \\
\framebox{\parbox{1.4cm}{de\-formatter}} $\rightarrow$ &
\framebox{\parbox{0.8cm}{morph. anal.}}  $\rightarrow$ &
\framebox{\parbox{1.2cm}{PoS tagger}} $\rightarrow$ &
\framebox{\parbox{1.1cm}{struct.\ transf.}} $\rightarrow$ &
\framebox{\parbox{0.8cm}{morph. gen.}}  $\rightarrow$ &
\framebox{\parbox{1.0cm}{post\-genera\-tor}} $\rightarrow$ &
\framebox{\parbox{1.2cm}{re-format\-ter}} \\ & & & $\updownarrow$ & &
& $\downarrow$ \\ & & & \framebox{\parbox{1.0cm}{lex.\ transfer}} & &
& \parbox{0.7cm}{TL text}\\\\
\end{tabular}
\end{center} }
\caption{The eight modules that build the assembly line of the
shallow-transfer machine translation system.}
\label{fg:modules}
\label{pg:modules}
\end{figure*}



\begin{itemize}
\item The \emph{de-formatter}, which separates the text to be
translated from the format information (RTF, HTML, etc.); its
specification can be found in Section \ref{ss:formato}. Format
information is encapsulated so that the rest of the modules treat it
as blanks between words. For example, for the HTML text in Spanish:
\begin{alltt}
 es <em>una señal</em>
\end{alltt} 
("it is a sign") the de-formatter encapsulates in brackets
the HTML tags and gives the output:
\begin{alltt} 
es [<em>]una señal[</em>]
\end{alltt} 
The character sequences in brackets are treated by the
rest of the modules as simple blanks between words.
\item \label{pg:FSFL} The \emph{morphological analyser}, which
  tokenizes the text in \emph{surface forms} (SF) (lexical units as
  they appear in texts) and delivers, for each SF, one or more
  \emph{lexical forms} (LF) consisting of \emph{lemma} (the base form
  commonly used in classic dictionary entries), the \emph{lexical
  category} (noun, verb, preposition, etc.) and morphological
  inflection information (number, gender, person, tense,
  etc.). Tokenization of a text in SFs is not straightforward due to
  the existence, on the one hand, of contractions (in Spanish,
  \emph{del}, \emph{teniéndolo}, \emph{vámonos}; in English,
  \emph{didn't}, \emph{can't}) and, on the other hand, of lexical
  units made of more than one word (in Spanish, \emph{a pesar de},
  \emph{echó de menos}; in English, \emph{in front of}, \emph{taken
  into account}). The morphological analyser is able to analyse these
  complex SFs and treat them appropriately so that they can be
  processed by the next modules. In the case of contractions, the
  system reads a single surface form and gives as output a sequence of
  two or more lexical forms (for instance, the Spanish
  preposition-article contraction \emph{del} would be analysed into
  two lexical forms, one for the preposition \emph{de} and another one
  for the article \emph{el}). Lexical units made of more than one word
  (multiwords) are treated as single lexical forms and processed
  specifically according to its type.\footnote{For more information
  about the treatment of multiwords, please refer to page
  ~\pageref{ss:multipalabras}.}

Upon receiving as input the example text from the previous module, the
morphological analyser would deliver:
\begin{alltt} 
^es/ser<vbser><pri><p3><sg>\$[ <em>]
^una/un<det><ind><f><sg>/unir<vblex><prs><1><sg>/unir
<vblex><prs><3><sg>\$ 
^señal/señal<n><f><sg>\$[</em>]
\end{alltt}

where each surface form has been analysed into one or more lexical
forms: \emph{es} has been analysed as one SF with lemma \emph{ser}
("to be"), whereas \emph{una} receives three analyses: lemma \emph{un}
("one"), determiner, indefinite, feminine, singular; lemma \emph{unir}
("to join"), verb in subjunctive present, 1st person singular, and
lemma \emph{unir}, verb in subjunctive present, 3rd person singular.

This module is generated from a source language (SL) morphological
dictionary, the format of which is specified in section
\ref{ss:diccionarios}.
\item The \emph{part-of-speech tagger} chooses, using a statistical
model (hidden Markov model), one of the analyses of an ambiguous word
according to its context; in the previous example, the ambiguous word
would be the surface form \emph{una}, which can have three different
analyses. A sizeable fraction of surface forms (in Romance languages,
for instance, around one out of every three words) are ambiguous, that
is, they can be analysed into more than one lemma, more than one
part-of-speech or have more than one inflection analysis, and are
therefore an important source of translation errors when the wrong
equivalent is chosen. The statistical model is trained on
representative source-language text corpora.
 
  The result of processing the example text delivered by the
  morphological analyser with the part-of-speech tagger would be:

\begin{alltt}
^ser<vbser><pri><p3><sg>\$[ <em>]^un<det><ind><f><sg>\$ 
^señal<n><f><sg>\$[</em>]
\end{alltt}

where the correct lexical form (determiner) has been selected for the
word \emph{una}.


  The specification of the part-of-speech tagger is in section
  \ref{ss:tagger}.


\item The \emph{lexical transfer module}, that uses a bilingual
dictionary and is called by the structural transfer module, reads each
LF of the SL and delivers the corresponding target language (TL)
lexical form. The dictionary contains a single equivalent for each SL
lexical form; that is, no word-sense disambiguation is performed
\nota{now not true: lextor}. Multiwords are translated as a single unit.
The lexical forms in the running example would be translated into
Catalan as follows:

\begin{alltt}
ser<vbser> \(\longrightarrow\) ser<vbser>
un<det> \(\longrightarrow\) un<det>
señal<n><f> \(\longrightarrow\) senyal<n><m>
\end{alltt}

This module is generated from a bilingual dictionary, which is
described in Section \ref{ss:diccionarios}.

\item The \emph{structural transfer module}, which detects and
processes patterns of words (chunks or phrases) that need special
processing due to grammatical divergences between the two languages
(gender and number changes, word reorderings, changes in prepositions,
etc.). This module is generated from a file containing rules which
describe the action to be taken for each pattern.  In the running
example, the pattern formed by
\verb!^!\texttt{un<det><ind><f><sg>}\verb!$!
\verb!^!\texttt{señal<n><f><sg>}\verb!$! would be detected by a
determiner--noun rule, which in this case would change the gender of
the determiner so that it agrees with the noun; the result would be:

\begin{alltt}
^ser<vbser><pri><p3><sg>\$[ <em>]^un<det><ind><m><sg>\$
^senyal<n><m><sg>\$[</em>]
\end{alltt}

 The format of the structural transfer rules file, inspired in the one
 described in \cite{garridoalenda01p}, is specified in Section
 \ref{ss:transfer}.
\item The \emph{morphological generator}, that, from a lexical form in
the target language, generates a suitably inflected surface form. The
result for the example phrase would be:
\begin{alltt} 
és[ <em>]un senyal[</em>]
\end{alltt}

This module is generated from a morphological dictionary, which is
described in detail in Section \ref{ss:diccionarios}.
\item The \emph{post-generator}, that performs some orthographic
operations in the TL such as contractions and apostrophations, and
which is generated from a transformation rules file the format of
which is very similar to the format of the mentioned dictionaries. Its
format is specified in Section \ref{ss:diccionarios}. In the example
text there is no need to perform any contraction or apostrophation.
\item The \emph{re-formatter}, which restores the original format
information into the translated text; the result for the running
example would be the correct conversion of the text into HTML format:
\begin{alltt} 
és <em>un senyal</em>
\end{alltt}


The specification of the re-formatter is described in Section
\ref{ss:formato}.
\end{itemize}

The four lexical processing modules (morphological analyser, lexical
transfer module, morphological generator and post-generator) use a
single compiler, based on a class of \emph{finite-state transducers}
\cite{garrido99j}, in particular, letter transducers
\cite{roche97,ortiz05j}; its characteristics are described in Section
\ref{se:compiladoresdic}.



\chapter[Stream format specification]{Format specification of the
data stream between modules}
\label{se:flujodatos} \nota{Material duplicat en "formatadors i
reformatadors": declarar-ho, treure-ho? - feina Gema}

\section{Introduction}

The format of the data that circulate between the engine's modules has
to be specified so that document processing is more effective and
transparent. The proposed system design (see Section
\ref{ss:descrarq}) imposes the need to use three different data stream
types, as shown in Figure \ref{fig:fdatos}.

The stream format is text-based to facilitate, among other things, the
diagnosis of possible system errors, since it is easy to manipulate
the stream in order to reproduce the phenomena that are to be tested,
and change it to see the result. Other benefits of using text streams
are that it is possible to test independently the output of each
module, and that it allows for fast building of prototypes to test the
system's global performance, the validity of linguistic data, etc.



\begin{figure}[h]
\begin{center}
\includegraphics[width=14cm]{fdatos}
\end{center}
\caption{The different data stream types in the machine translation
system. See the text for its description.}
\label{fig:fdatos}
\end{figure}

The data stream types are:

\begin{itemize}
\item \textit{Data stream with format:} It is the text in its original
format, with no further marks: XML, ANSI text, RTF, HTML, etc. Since
it is the original format of the documents, nothing needs to be
specified about it except the name of the format.
\item \textit{Data stream without format:} It is the text with
\textit{superblanks}, that is, with special characters that
encapsulate the format (see Section \ref{ss:formato}); superblanks are
treated by the linguistic modules as blanks between words (with some
exceptions).  This is the format generated by the de-formatter and
used by the re-formatter when generating the final translated
document.
\item \textit{Segmented data stream:} In this format, apart from
superblanks, lexical units that are to be translated are delimited
also with special characters. These characters are put by the
morphological analyser and deleted by the generator, which delivers
the final surface forms.
\end{itemize}


We describe next the characteristics of the data stream used between
the modules of the translator, that is, the second and the third
stream types. In general terms, it is a plain text format marked with
characters that have a special meaning. This format is intended for
the processing in servers that translate large volumes of text.

Some of the formats that the engine can process may contain extensive
blocks of information in binary format ---RTF for instance, that may
include bitmap images---.  To enable an efficient processing of this
type of documents, we designed a way to extract this information and
restore it after translation has been performed; see Section
\ref{ss:formato} for a complete description.

\section{Data stream without format}

Data stream without format is output by the de-formatter and by the
generator \nota{no del tot: postgenerador}, and is used as input by
the morphological analyser, the post-generator and the re-formatter.

In the
subsection of this section you can find a description of the method to
delimit \textit{superblanks} and \textit{extensive superblanks}. As an example
we will use the HTML document in
Figure~\ref{fg:docorig}.

\begin{figure}[htbp]
\begin{small}
\begin{alltt}
<\textbf{html}> 
  <\textbf{head}> 
    <\textbf{title}>Title</\textbf{title}>
  </\textbf{head}>
  <\textbf{body}> 
    <\textbf{p}>Divided 
       sentence</\textbf{p}>
  </\textbf{body}> 
</\textbf{html}>
\end{alltt}
\end{small}
\caption{Example of HTML document}
\label{fg:docorig}
\end{figure}

The structural elements that must include this data stream type are
the following:

\begin{itemize}
\item \textit{Superblanks}.  Blocks that contain segments of format
information included in the documents, when these are short.
\item \textit{Extensive superblanks}.  Marks that are used to specify
external documents that include segments of format information for the
document being processed, when these segments are long.
\item \textit{Text}. The document text that can be translated.
\item \textit{Artificial sentence endings}. \label{finfrase} When the
format in the document suggests a sentence separation that is not
signalled by any punctuation mark (for instance, titles with no full
stop at the end, or the content of cells in a table), the format
processing must have a mechanism (invisible for the user) that enables
the marking of these sentence endings.
\item \textit{Special characters protection (for non-XML stream)}.
  Characters that must be protected to avoid conflict with the ones
  used in the data stream format.
\end{itemize}

% \subsection{XML format}

% En este tipo de flujo se usa el elemento \texttt{<\textbf{b}>} para definir los 
% superblancos y los superblancos extensos.  Para el caso de los 
% \textbf{superblancos} la sintaxis es la siguiente:

% \begin{small}
% \begin{alltt} % <\textbf{b}>\textit{contenido del bloque de formato}</\textbf{b}>
% \end{alltt}
% \end{small}

% Hay que resaltar que para los formatos basados en SGML, es necesario
% incluir el formato en bloques \texttt{<![CDATA[\ldots]]>} dentro de
% las marcas indicadas. \nota{millor dir com són: prendre text de EAMT
% '05 - Gema} Por su parte, los \textit{superblancos extensos} se deben 
% expresar, a modo de atributos, de la siguiente manera:

% \begin{small}
% \begin{alltt} % <\textbf{b} \textsl{filename}="\textit{nombre de fichero}"/>
% \end{alltt}
% \end{small}

% El \emph{texto} estará incluido entre los elementos \textbf{b} que se 
% acaban de explicar sin ninguna marca de estructura particular.

% Los \emph{finales de frase artificiales} se expresan mediante un punto y un 
% superblanco vacío inmediatamente a continuación.

% \begin{small}
% \begin{alltt} % .<\textbf{b}/>
% \end{alltt}
% \end{small}

% Resumiendo, el flujo de datos de un documento en cualquier formato de los que 
% trata el traductor se reduce a otro documento XML que debe cumplir la 
% siguiente DTD:

% \begin{small}
% \begin{alltt} % <!\textsl{ELEMENT} \textbf{document} (b|\textsl{#PCDATA})*> 
% <!\textsl{ELEMENT} \textbf{b} (\textsl{#PCDATA}?)> 
% <!\textsl{ATTLIST} b filename \textsl{CDATA} \textsl{#IMPLIED}>
% \end{alltt}
% \end{small}

% El resultado de encapsular el formato del fichero de la 
% figura~\ref{fg:docorig} en el flujo con formato XML se ve en la 
% figura~\ref{fg:docorigXML}.  Si hubiese algún superblanco que por su longitud
 % se convirtiese en un superblanco extenso, la forma de especificarlo sería como sigue:
% \begin{small}
% \begin{alltt} % <\textbf{b} \textsl{filename}="/tmp/ficherotemporal"/>
% \end{alltt}
% \end{small}donde \texttt{"/tmp/ficherotemporal"} es un fichero que 
% contiene el superblanco extenso para que pueda ser recuperado por el reformateador.

% \begin{figure}
% \begin{small}
% \begin{alltt} % <?\textbf{xml} \textsl{version}="1.0" \textsl{encoding}="iso-8859-15"?> 
% <\textbf{document}> 
% <\textbf{b}><![CDATA[<html> % <head>
 % <title>]]></\textbf{b}>Título.<\textbf{b}/><\textbf{b}><![CDATA[</title>
% </head> % <body> % <p>]]></\textbf{b}>Frase<\textbf{b}><![CDATA[ 
% ]]></\textbf{b}>dividida.<\textbf{b}/><\textbf{b}><![CDATA[ % </body>
% </html>]]></\textbf{b}> % </\textbf{document}>
% \end{alltt}
% \end{small}
% \caption{El documento de la figura \protect\ref{fg:docorig} con el
% formato encapsulado usando marcas en XML y segmentos
%\texttt{<![CDATA[\ldots]]>}}
% \label{fg:docorigXML}
% \end{figure}

%\subsection{Formato no XML}
\subsection{Stream format}
\label{se:noxml1} This format is based on the one used in the machine
translation systems \textsf{interNOSTRUM}
\cite{canals01b,garridoalenda01p,garrido99j} and \textsf{Traductor
Universia} \cite{garrido03p, gilabert03j}.

In this stream type, the characters \texttt{[} and \texttt{]} are used
to indicate \emph{superblanks}, as shown in the following example:

\begin{small}
\begin{alltt} 
[\textit{superblank content}]
\end{alltt}
\end{small}

In the case of \emph{extensive superblanks}, the file name is
specified using the at sign \texttt{@}:

\begin{small}
\begin{alltt} 
[@\textit{file name}]
\end{alltt}
\end{small}

The \emph{text} is outside the superblank marks.

\emph{Artificial sentence endings}
are expressed by a full stop and an empty superblank right after it.

\begin{small}
\begin{alltt} 
.[]
\end{alltt}
\end{small}

The following table shows the \textbf{protected characters}:

\begin{center}
\begin{tabular}{|l|c|c|l|} \hline Name & Character & Protected form&
Meaning \\ 
\hline 
At & \texttt{@} & \verb!\@! & External superblank\\
Slash & \texttt{/} & \verb!\/! & Divider of meanings\\ 
Backslash & \verb!\! & \verb!\\! & Protection character \\ 
Caret & \verb!^! & \verb!\^! & Beginning of LF\\ 
Opening square bracket & \texttt{[} & \verb!\[! & Beginning of blank\\ 
Closing square bracket & \texttt{]} & \verb!\]! & End of blank \\ 
Dollar & \verb!$! & \verb!\$! & End of LF\\ 
Greater than & \texttt{>} & \verb!\>! & Begin. of morph. symbol\\
Less than & \texttt{<} & \verb!\<! & End of morph. symbol \\ 
\hline
\end{tabular}
\end{center}


Figure ~\ref{fg:docorigtext} shows the document in Figure
~\ref{fg:docorig} after encapsulation.

\begin{figure}[here]
\begin{small}
\begin{alltt}
[<html> 
  <head> 
    <title>]Title.[][</title> 
  </head>
  <body> 
    <p>]Divided[
       ]sentence.[][</p>
  </body> 
<html>]
\end{alltt}
\end{small}
\caption{The document in Figure \protect\ref{fg:docorig} with format
encapsulated using square brackets}
\label{fg:docorigtext}
\end{figure}


\section{Segmented data stream}

Segmented data stream is the stream that circulates between the
modules that handle linguistic information in the translation engine.
In this stream, words are delimited and labelled.  There are two types
of segmented stream:

\begin{itemize}
\item \textit{Ambiguous segmented stream}. Its main characteristic is
that words have a surface form and potentially more than one lexical
form (lexical multiform).  This stream type is the format in which
the morphological analyser provides the input data for the
part-of-speech tagger (see diagram \ref{eq:formaanalizada} in page
~\pageref{formaanalizada} for a detailed description of ambiguous
segmented stream).
 
\item \textit{Unambiguous segmented stream}.  It has only one lexical
form for each word and it does not include the surface form.  This is
the format in which data circulate from the part-of-speech tagger to
the transfer module, and from this module to the generator (see
diagram \ref{eq:formaanalizada2} in page~\pageref{formaanalizada2} for
a detailed description of the format of unambiguous segmented stream).
\end{itemize}

Furthermore, besides the information already marked in the data stream
without format, the new stream has to enable marking of the following
information:

\begin{itemize}
\item \textit{Lexical units}.  A lexical unit is made of a surface
form (in the case of ambiguous segmented stream) plus one or more
lexical forms (the different possible analyses of the SF) with their
grammatical symbols.
\item \textit{Surface forms (ambiguous segmented stream)}.  The word
as it appears in the original text.
\item \textit{Lexical forms}.  The lemma of the word and its
grammatical symbols.
\item \textit{Grammatical symbols}.  They describe the morphological
and grammatical attributes of a surface form.
\end{itemize}

% \subsection{XML format}

% Las \textit{palabras} se etiquetan de la forma que se muestra a 
% continuación:

% \begin{small}
% \begin{alltt} % <\textbf{w}>\textit{información de la palabra}</\textbf{w}>
% \end{alltt}
% \end{small}

% Para el caso del \textit{flujo de datos segmentado ambiguo}, la 
% \textit{forma superficial} se indica en el interior de un elemento 
% \texttt{<\textbf{w}>} mediante el contenido de un único elemento 
%\texttt{<\textbf{sf}>}.  A continuación, se sitúan la forma o 
%\textit{formas léxicas} que sean necesarias:

% \begin{small}
% \begin{alltt} % <\textbf{w}> % <\textbf{sf}>\textit{forma superficial}</\textbf{sf}> 
% <\textbf{lf}>\textit{forma léxica 1}</\textbf{lf}> 
% <\textbf{lf}>\textit{forma léxica 2 (opcional)}</\textbf{lf}> 
% ...  % </\textbf{w}>
% \end{alltt}
% \end{small}

% Para el caso del flujo no ambiguo, sólo se especifica una única forma léxica.


% \begin{small}
% \begin{alltt} % <\textbf{w}> % <\textbf{lf}>\textit{forma léxica}</\textbf{lf}> % </\textbf{w}>
% \end{alltt}
% \end{small}

% %% \pagebreak

% La DTD de este flujo de datos para textos \textit{sin desambiguar} es la % que se muestra en la figura~\ref{fg:ambdtd} a continuación.

% \begin{figure}[here]
% \begin{small}
% \begin{alltt}
%   <!\textsl{ELEMENT} \textbf{document} (b|w|\textsl{#PCDATA})*>
%   <!-- atención, el #PCDATA anterior sigue siendo necesario para los 
%        carácteres no etiquetados y que no forman parte del formato -->
%   <!\textsl{ELEMENT} \textbf{b} (\textsl{#PCDATA}?)>
%   <!\textsl{ATTLIST} b filename \textsl{CDATA} \textsl{#IMPLIED}>
%   <!\textsl{ELEMENT} \textbf{w} (sf,lf+)>
%   <!\textsl{ELEMENT} \textbf{sf} (\textsl{#PCDATA})>
%   <!\textsl{ELEMENT} \textbf{lf} (\textsl{#PCDATA}|s)+>
%   <!\textsl{ELEMENT} \textbf{s} \textsl{EMPTY}>
%   <!\textsl{ATTLIST} s n \textsl{IDREF #REQUIRED}>
% \end{alltt}
% \end{small}
% \caption{DTD para textos no desambiguados con formato XML}
% \label{fg:ambdtd}
% \end{figure}




% Para los ya \textit{ desambiguados}, los textos deben cumplir la DTD de la figura~\ref{fg:desambdtd}.

% \begin{alltt}
%   <!\textsl{ELEMENT} \textbf{document} (b|w|\textsl{#PCDATA})*>
%   <!-- atención, el #PCDATA anterior sigue siendo necesario para los 
%        carácteres no etiquetados y que no forman parte del formato -->
%   <!\textsl{ELEMENT} \textbf{b} (\textsl{#PCDATA}?)>
%   <!\textsl{ATTLIST} b filename \textsl{CDATA} \textsl{#IMPLIED}>
%   <!\textsl{ELEMENT} \textbf{w} (lf)>
%   <!\textsl{ELEMENT} \textbf{lf} (\textsl{#PCDATA}|s)+>
%   <!\textsl{ELEMENT} \textbf{s} \textsl{EMPTY}>
%   <!\textsl{ATTLIST} s n \textsl{IDREF #REQUIRED}>
% \end{alltt}
% \end{small}
% \caption{DTD para textos desambiguados con formato XML}
% \label{fg:desambdtd}
% \end{figure}

% La figura~\ref{fg:docorigXML2} muestra un ejemplo de segmentación del flujo
% que incluye la forma de encapsular el formato y la información léxica.  Este
% ejemplo es para el caso de flujo segmentado ambiguo y corresponde al texto
% HTML original de la figura~\ref{fg:docorig}. 

% \begin{figure}[htbp]
% \begin{small}
% \begin{alltt}
% <?\textbf{xml} \textsl{version}="1.0" \textsl{encoding}="iso-8859-15"?>
% <document>
% <\textbf{b}><![CDATA[<html>
%   <head>
%     <title>]]></\textbf{b}>
% <\textbf{w}>
%   <\textbf{sf}>Título<\textbf{sf}>
%   <\textbf{lf}>Título<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{lf}>
% </\textbf{w}>
% <\textbf{w}>
%   <\textbf{sf}>.</\textbf{sf}>
%   <\textbf{lf}>.<s n="sent"/></\textbf{lf}>
% </\textbf{w}><\textbf{b}/>
% <\textbf{b}><![CDATA[</title>
%   </head>
%   <body>
%     <p>]]></\textbf{b}>
% <\textbf{w}>
%   <\textbf{sf}>Frase</\textbf{sf}>
%   <\textbf{lf}>Frase<s n="n"/><s n="f"/><s n="sg"/></\textbf{lf}>
% </\textbf{w}>
% <\textbf{b}><![CDATA[
%        ]]></\textbf{b}>
% <\textbf{w}>
%   <\textbf{sf}>dividida</\textbf{sf}>
%   <\textbf{lf}>dividir<s n="vblex"/><s n="pp"/><s n="f"/><s n="sg"/></\textbf{lf}>
% </\textbf{w}>
% <\textbf{w}>
%   <\textbf{sf}>.</\textbf{sf}>
%   <\textbf{lf}>.<s n="sent"/></\textbf{lf}>
% </\textbf{w}><\textbf{b}/>
% <\textbf{b}><![CDATA[
%   </body>
% <html>]]></\textbf{b}>
% </document>
% \end{alltt}
% \end{small}
% \caption{Ejemplo de flujo segmentado con el formato encapsulado en XML,
%   correspondiente al documento HTML de la figura~\ref{fg:docorig}.} 
% \label{fg:docorigXML2}
% \end{figure}
%\subsection{Formato no XML}
%\subsubsection{Formato de flujo}
\label{se:noxml2} The symbols '\verb!^!' for word beginning and
'\verb!$!' for word end are used to delimit \textit{words}, as shown
in this example:

\begin{small}
\begin{alltt} 
  \verb!^!\textit{word}\verb!$!
\end{alltt}
\end{small}

To separate the \textit{surface form} and the following
\textit{lexical forms}, the symbol \texttt{/} is used.  This separator
only has sense in the ambiguous segmented stream, since in the
unambiguous stream there is only the lexical form.  It is used as
follows:

\begin{small}
\begin{alltt} 
  \verb!^!\textit{surface form}/\textit{lexical form 1}/...\verb!$!
\end{alltt}
\end{small}

Lexical forms can include symbols (generally located at the end), as
shown in the example of Figure \ref{fg:docorigtext2}.


\begin{figure}
\begin{small}
\begin{alltt}
[<html> 
   <head> 
     <title>]^Title/Title<n><m><sg>\$^./.<sent>\$[][</title>
   </head>
   <body> 
     <p>]^Divided/Divide<vblex><pp>/Divided<vblex><past>\$[
        ]^sentence/sentence<n><sg>/sentence<vblex><inf>\$^./.\\<sent>\$[][</p>
  </body> 
<html>]
\end{alltt}
\end{small}
\caption{Example of segmented stream with format encapsulated in
non-XML format, corresponding to the HTML document in Figure
~\ref{fg:docorig}.}
\label{fg:docorigtext2}
\end{figure}





\chapter{Modules specification}
\label{se:especificmodulos}



\section{Lexical processing modules}
\label{ss:modproclex}

\subsection{Module description }
\label{ss:funcproclex}

One of the most efficient approaches to lexical processing is based on
the use of finite-state transducers (FST)
\cite{mohri97a,roche97b}. FST are a type of finite-state automata,
which may be used as one-pass morphological analysers and generators
and may be very efficiently implemented. In this project, we have used
a class of FST called letter-transducers
\cite{roche97b,garrido02a,garrido99j}; in fact, any finite-state
transducer may always be turned into a letter-transducer. Garrido and
collaborators \cite{garrido99j,garrido02a} give a formal definition of
the letter transducers used in this project; describing them
informally, a letter-transducer is an idealised machine consisting of:
\begin{enumerate}

\item A (finite) set of states, that is, of situations in which the
transducer can be while it is reading, from left to right, the input
letters or symbols. Among the states of the set, we can distinguish:

\begin{enumerate}
\item A single initial state: this is the state in which the
transducer is before processing the first letter or the first symbol
of the input.
\item One or more acceptance states, which are only reached after
having completely read a valid entry and, therefore, are used to
detect valid words.
\end{enumerate}
\item A set (also finite) of state transitions consisting of:
\begin{enumerate}
\item the origin state
\item the destination state
\item the input letter or symbol
\item the output letter or symbol
\end{enumerate} To make possible that input and output have different
lengths at any time, it is allowed that there is no input symbol, that
there is no output symbol or that there is neither input nor output
symbol. This case is generally represented using a special symbol (the
empty symbol).
\end{enumerate}

Every time the transducer reads an entry symbol, it creates a list of
\emph{live} or \emph{active} states, each one of which has an
associated output (a sequence of symbols). The way the letter
transducer works is different for each type of lexical processing
operation. For example, in the morphological analysis, the transducer
tries to read the longest entry recognised by the dictionary
(``left-to-right, longest-match'' mode).
\begin{enumerate}
\item Beginning: the set of live states is given a single live state:
the initial state, with the empty word ("") as output associated to
the state.
\item When from one of the states in the current set of live states it
is possible to reach other states through transitions that do not have
input symbol, these states are added to the set of live states, and
are associated to the output obtained when extending the associated
outputs with the output symbol found in the corresponding
transitions. This expansion operation of the set of live states
continues until it is not possible to add more states.
\item A symbol from the input word is read.
\item A new set of live states is created, made with the states
reached through transitions that have that symbol as input, and this
states are associated to the outputs extended by adding the
corresponding output symbols found in the transitions.
\item If the current set has any live state, the process continues on
step 2.
\item The sets of live states are read backwards until a set is found
which contains acceptance states. The morphological analyses will be
the outputs associated to these states, and the reading position is
set to the position immediately after this set (so that it can be
processed again by the transducer in the next pass).
\end{enumerate} Not all acceptance states have the same
characteristics, and this fact adds more conditions to the acceptance
process, in order to be able to deal with unknown words or with words
that are joined to other words, as will be explained later.

The transducer reads the input word only once on average, from right
to left and symbol by symbol, and keeps a tentative list of possible
partial outputs that is updated and pruned as the input is being
read. When letter transducers are used as morphological analysers or
as lemmatizers, they read a surface form and write the resulting
lexical form(s). In this case, input symbols are the letters of the
surface form, and output symbols are the letters needed to write the
lemmas, as well as the letters and special symbols needed to represent
the morphological analysis, such as in \texttt{<n>}, \texttt{<f>},
\texttt{<2p>}, etc.

The transducers work in a similar way for other lexical processing
tasks.

\nota{La noció de LRLM (left-to-right, longest-match) (o ODSCML,
"izquierda a derecha, recortando el segmento concordante más largo")
ha de quedar clara en el funcionamient del morfològic i del trànsfer
estructural. Afegir coses de l'article de EAMT 2005.}

\subsubsection{Letter case handling in dictionaries}
\label{mayusc}

The same input word in a lexical processing module can be written
differently regarding letter case.  The most frequent cases are:

\begin{itemize}
\item The whole word is in lower case.
\item The whole word is in upper case.
\item The first letter is capitalised and the rest is in lower case
(typical case for proper nouns).
\end{itemize}

The transductions in the dictionary can also be found in these three
states.  The way in which one word is written in the dictionary is
used to discard possible analysis of the word, according to the
following rules:

\begin{itemize}
\item If the input letter is upper case and in the current analysis
state there are concordant transitions in lower case, these
transductions are made.
\item If the input letter is lower case and in the current state there
are not concordant transitions in lower case, the transductions are
not made.
\end{itemize}

Thanks to this policy, a surface form that is not capitalised can not
be analysed as a proper noun.

The case of an input word will be maintained in the output of the
translator unless it is decided not to do so. The case can be changed
in the structural transfer module; this option is useful, for example,
when there is a reordering of words or when a word is added before a
capitalised word at the beginning of a sentence, such as in the
translation of the Catalan phrase \emph{Vindran} into
English: \emph{They will come}.


\subsection{Data format: the dictionaries}
\label{ss:diccionarios}
\subsubsection{General criteria for dictionary design}

The experience of the Transducens group at the Universitat d'Alacant
in the creation of machine translation systems between Romance
languages (\texttt{es}, \texttt{ca} and \texttt{pt}) already operative
and publicly accessible has inspired the main characteristics of the
whole shallow-transfer machine translation system described in this
document, as well as its application to the Romance languages of Spain
(\texttt{es}, \texttt{ca} and \texttt{gl}). In some sense, it could be
stated that in the present project the only work was to adapt (rewrite
in a standardised and interoperable format) the specifications and
programs used in already operative projects.

In particular, the design of the dictionaries has been based in an
architecture that pretends to separate, as far as possible, the source
language from the target language, even knowing that these
dictionaries are translation-oriented and, therefore, that it is not
advisable to elaborate them completely separately.  The chosen format
is used for the specification of both morphological dictionaries
(monolingual) and bilingual dictionaries.

The format for dictionaries, as well as for the rest of linguistic
data (definition file for part-of-speech tagger and structural
transfer rules) is XML\footnote{\url{http://www.w3.org/XML/}}, an
international standard used in numerous natural language processing
projects which, thanks to the availability of many utilities and
libraries, it is becoming a very powerful tool for linguistic data
representation and exchange (see article \cite{ide00}).



Dictionaries are designed so that they can be compiled into
\textit{letter transducers }, for efficiency reasons. For more
information on letter transducers as a particular case of finite-state
transducers, see Section \ref{ss:funcproclex} or the article
\cite{garrido02a}.

The letter transducers that are generated from the system dictionaries
(morphological, bilingual and post-generation dictionaries) process
input character strings to produce output strings. According to this,
dictionaries are made of entries consisting of string pairs that
correspond to the inputs and outputs of the transducer.


The most powerful tool in these dictionaries is the definition and use
of \emph{paradigms}. Since in Romance languages a lot of lemmas share
the same inflection pattern (there are regularities in their
inflection), it is useful and straightforward to group these
regularities in inflection paradigms to avoid having to write all the
forms of every word. Paradigms allow the representation of dictionary
entries compactly and help optimise the speed for building a
dictionary. Once the most frequent paradigms in a dictionary are
defined, the linguist does not need to bother, in most of the cases,
with the whole inflection of a new term, since entering an inflective
word is generally limited to writing the lemma and choosing one
inflection pattern among the previously defined paradigms.
Furthermore, the use of paradigms reduces the memory requisites,
facilitates the construction of efficient letter transducers and
speeds up the compilation process \cite{ortiz05j}. We did not use
paradigms in bilingual dictionaries (although it is possible to)
because most of the inflection information is processed implicitly in
these dictionaries, as explained in page~\pageref{ss:bil}.



\subsubsection{Dictionary types}
 
In our system there are three types of dictionaries: morphological
(monolingual) dictionaries for each of the languages involved
(Spanish, Catalan and Galician); bilingual dictionaries for the
different translation pairs (Spanish--Catalan and Spanish--Galician),
and post-generation dictionaries for each of the languages (a
post-generation dictionary is not a typical dictionary, with lemmas
and morphological information, but is like a little dictionary of the
orthographic transformations that may undergo words when they come
together).  The structure of the three dictionary types is specified
by the same DTD (\emph{Document Type Definition}), which can be found
in Appendix \ref{ss:dtd_dics}.


\textbf{Morphological dictionaries} are used both for building
morphological analysers ---the translation system module used to
obtain all the possible lexical forms for a certain surface form in
the source language --- and morphological generators
---the module that generates the surface form in the target language
from the lexical form of each word---.  These two modules are obtained
from a single morphological dictionary, depending on the direction in
which it is read by the system: read from left to right, we obtain the
analyser, and read from right to left, the generator.

The block structure typical for these dictionaries is the following:

\begin{itemize}
\item \textit{An alphabet definition}.  This definition is used
exclusively for building the morphological analyser; specifically, it
enables the morphological analyser to appropriately tokenize unknown
words and the ones in the conditional sections (see the description of
the element \texttt{<section>} in page \pageref{ss:section}); the
morphological generator does not need this definition.

\item \textit{A definition of symbols}.  It consists of a declaration
of the grammatical symbols that will be used in dictionary entries
(you can find in Appendix \ref{se:simbolosmorf} a list with the
grammatical symbols used in this project).
\item \textit{A definition of paradigms}.  Paradigms need to be
defined here in order to be used in the dictionary sections or in other
paradigms.
\item \textit{One or more dictionary sections with conditional
  tokenization}, type \texttt{standard}. To include most of the words
  of the dictionary.
\item \textit{One or more dictionary sections with unconditional
  tokenization}.  To include certain words that follow a regular
  pattern or that are tokenized regardless the text directly after
  them (see description of the element \texttt{<section>} in page
  \pageref{ss:section}). In the Catalan morphological dictionaries,
  words requiring an unconditional tokenization are distributed in two
  sections: one for the forms that require the introduction of a blank
  immediately after (due to processing requirements of the lexical
  forms), like the apostrophized forms \emph{l'} or \emph{d'}, and
  another one for punctuation marks, numbers and other signs.

\end{itemize}

\textbf{Bilingual dictionaries} represent in the system the lexical
transfer process, that is, the assignment of the TL lexical form that
corresponds to each SL lexical form. Two \emph{products} are obtained
from each bilingual dictionary, depending on the direction in which it
is read by the system: when the dictionary is read from left to right,
we obtain the lexical transfer module in one translation direction,
and when it is read from right to left, in the other direction. For
the bilingual dictionaries of our project, it has been established
that Spanish will be put always on the left side of the entries, and
the rest of the languages (Catalan and Galician), on the right
side. Thus, for example, the bilingual Spanish--Galician dictionary
will be read from left to right for the translation
\texttt{es}--\texttt{gl} and from right to left for the translation
\texttt{gl}--\texttt{es}.  In applications like the ones in this
project, these dictionaries do not have paradigms: they are build with
generic entries which almost always have no more information than
lemma and part of speech, and there is no inflection information.

The block structure used in the bilingual dictionaries of this project
is the following:

\begin{itemize}
\item \textit{A definition of symbols}.  It consists of a declaration
of the grammatical symbols that will be used in dictionary entries.
\item \textit{A single dictionary section}.  Where bilingual
correspondences are specified.
\end{itemize}

Since 2007, bilingual dictionaries allow the specification of more
than one TL translation, so that a lexical selection module (see
Section \ref{se:seleccio_lex}) can choose the most suitable equivalent
according to the context. To that end, an attribute has been added to
bilingual dictionaries. You can find its description in section
\ref{dic_lextor}.


\textbf{Post-generation dictionaries} are used to perform some
transformations (orthographic changes, contractions, apostrophation,
etc.) required after surface forms in the target language have been
generated and come into contact with each other.  Since this kind of
operations can be expressed as a translation of character strings, it
has been decided to use the same type of dictionaries. It is
implicitly assumed that the parts of the text whose processing has not
been specified are copied just as they arrive. In these dictionaries,
the definition of paradigms is useful to express systematic changes in
the word contact phenomena. Unlike the other dictionary types, these
do not include grammatical symbols, since they process surface forms.

The block structure of post-generation dictionaries is the following:
\begin{itemize}

\item \textit{A definition of paradigms}. To use in entries.
\item \textit{A dictionary section}.  Where the patterns for
post-generation operations are specified.
\end{itemize}


The following table contains an overview of the possible reading
directions of dictionaries and their application to the Romance
languages in this project:

\begin{center}
 \begin{tabular}{|l|l|l|} 
\hline 
Dictionary & Reading direction & Function \\ 
\hline 
Morphological & left--right & analysis for \texttt{es}, \texttt{ca} and \texttt{gl}\\ 
              & right--left & generation for \texttt{es}, \texttt{ca} and \texttt{gl}\\\hline 
Bilingual     & left--right & translation for \texttt{es-ca} and \texttt{es-gl}\\ 
              & right--left & translation for \texttt{ca-es} and \texttt{gl-es}\\\hline 
Post-generation & left--right & post-generation for \texttt{ca}, \texttt{es} and \texttt{gl}\\\hline

\end{tabular}
\end{center}



\subsubsection{Description of the dictionary format}
\label{formatodics} This section presents the main elements of the
format in which dictionaries are build. The formal definition (a DTD)
can be found in Appendix ~\ref{ss:dtd_dics}.  Section \ref{dic_lextor}
describes the characteristics of a bilingual dictionary that works in
an Apertium system with lexical selection module.  Finally, from pages
\pageref{ss:morfgen} to %\pageref{ss:bil} y 
\pageref{ss:postgen} there
is a description of the different particularities of entries for the
three dictionary types (morphological, bilingual and post-generation).



\paragraph{Element for dictionary \texttt{<dictionary>}}
 
This is the root element and includes the whole dictionary.  It
contains an alphabetic character definition, a definition of symbols
(which are the morphological tags for the words), a definition of
inflection paradigms and one or more dictionary sections, which
contain the entries for the lexical forms (consisting of pairs made of
surface form--lexical form). Figure \ref{fig:dictionary} shows the
basic block structure of a generic dictionary.

\begin{figure}
\begin{small}
\begin{alltt}
<?\textbf{xml} \textsl{version}="1.0" \textsl{encoding}="iso-8859-15"?>
<\textbf{dictionary}>
  <\textbf{alphabet}>abcdefghijk ... ABCDEFGH ... çñáéíóú</\textbf{alphabet}>
  <\textbf{sdefs}>
    <!-- ... -->
  </\textbf{sdefs}>
  <\textbf{pardefs}>
    <!-- ... -->
  </\textbf{pardefs}>
  <\textbf{section} ...>
    <!-- ... -->
  </\textbf{section}>
  <!-- ... -->
</\textbf{dictionary}>
\end{alltt}
\end{small}
\caption{Use of the elements \texttt{<\textbf{dictionary}>} and
\texttt{<\textbf{alphabet}>}}
\label{fig:dictionary}
\end{figure}


\paragraph{Element for alphabet \texttt{<alphabet>}}

It is used to specify a definition of alphabetic characters.  The
purpose of this specification is enabling the modules that process the
input by means of letter transducers to tokenize it in individual
words.\nota{Parlar dels mots desconeguts. Cita \ref{ss:section} -
Mikel?}

In the present design, the definition of an alphabet only has sense in
morphological dictionaries, since it is needed for the
analysis. Figure \ref{fig:dictionary} shows a use example for this
element.


\paragraph{Element for symbol definition section \texttt{<sdefs>}} It
groups all the symbol definitions in a dictionary
(\texttt{<\textbf{sdef}>}).  There is an example of its use in Figure
\ref{fig:sdefs}.

\paragraph{Element for symbol definition \texttt{<sdef>}}

It is an empty element (it does not delimit any content): it is used
to specify, through the values of the attribute \texttt{\textsl{n}},
the names of the grammatical symbols that are used in the dictionary
to morphologically label lexical forms. In Figure \ref{fig:sdefs} you
can find a use example for this element. Refer to Appendix
\ref{se:simbolosmorf} if you need a list with all the grammatical
symbols used in the dictionaries of this project.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{sdefs}>
  <\textbf{sdef} \textsl{n}="n"/>
  <\textbf{sdef} \textsl{n}="det"/>
  <\textbf{sdef} \textsl{n}="sg"/>
  <\textbf{sdef} \textsl{n}="pl"/>
  <!-- ... -->
</\textbf{sdefs}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{sdefs}>}}
\label{fig:sdefs}
\end{figure}

\paragraph{Element for dictionary section \texttt{<section>}}
\label{ss:section}

It contains the words that will be recognised by the dictionary.  The
reason to divide a dictionary in sections is that some forms ---for
example, the ones coming from the identification of certain regular
patterns, or some forms that pertain to a specific dialect--- may need
a different processing.

One of the problems that the definition of sections in a dictionary
helps to solve is the tokenization procedure during morphological
analysis.  Most of the forms are tokenized following a conditional
criterion: identifying if the character being processed is followed by
a non-alphabetic character ---that is, not defined in
\texttt{<\textbf{alphabet}>}---. However, there are other forms, like
the Catalan apostrophized words \emph{l'} or \emph{d'}, that need an
unconditional tokenization model: there is no need to analyse what
comes after them, since, if it is an alphabetic character, it will
belong to the \textit{next} word. The forms that require unconditional
tokenization are included in a specific section of the
dictionary. Other kinds of processing can also be solved through these
divisions.



The value of the attribute \texttt{\textsl{type}} is used to express
the kind of string tokenization applied in each dictionary section:
the possible values of this attribute are: \texttt{standard}, for
almost all the forms of the dictionary (conditional mode),
\texttt{postblank}, for the forms that require an unconditional
tokenization and the placing of a blank, and \texttt{inconditional}
for the rest of forms that require unconditional tokenization.

The attribute \texttt{\textsl{id}} is used to assign an identifier (a
name) to the dictionary sections.

\begin{figure}
\begin{small}
\begin{alltt} 
<\textbf{section} \textsl{id}="principal" \textsl{type}="standard">
<!-- ... -->
</\textbf{section}>
<\textbf{section} \textsl{id}="patterns" \textsl{type}="inconditional">
<!-- ... -->
</\textbf{section}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{section}>}}
\label{fig:section}
\end{figure}

\paragraph{Element for entries \texttt{<e>}}

An entry is the basic unit of a dictionary or of a paradigm
definition.  Entries consist of a concatenation in any order of string
pairs \texttt{<\textbf{p}>}, identity transductions
\texttt{<\textbf{i}>}, references to paradigm \texttt{<\textbf{par}>}
or regular expressions \texttt{<\textbf{re}>}.  The structure and
meaning of these elements is explained later in this section (in pages
~\pageref{ss:p}, \pageref{ss:i}, \pageref{ss:par} and \pageref{ss:re}
respectively).

\label{restric}Two optional attributes are used with this entry.  The
first one is \texttt{\textsl{r}} (for \textit{restriction}), which
specifies if the entry has to be considered only when reading the
dictionary from left to right (\texttt{LR}) or when reading it from
right to left (\texttt{RL}).  If nothing is specified, it is assumed
that the entry must be considered in both directions.

In morphological dictionaries, the restriction \texttt{LR} causes that
a LF is analysed but not generated (for example, when the LF belongs
to a dialectal variant that we wish to recognise but not to generate)
and the restriction \texttt{RL} causes that a word is generated but not
analysed (needed, for example, for forms with post-generator
activation mark, see page \pageref{ss:a} for more details).

In bilingual dictionaries, the restrictions \texttt{LR} and
\texttt{RL} cause that the translation is done only in one direction:
for example, in a bilingual \texttt{es}--\texttt{ca} dictionary,
\texttt{LR} indicates that the LF is only translated from Spanish to
Catalan, and \texttt{RL} only from Catalan to Spanish. Let's
illustrate it with an example: the Spanish adverbs \emph{aún} and
\emph{todavía} ("still") are translated into Catalan as the same word,
\emph{encara}. We can only translate the Catalan adverb \emph{encara}
as one of both words into Spanish (there is no difference in meaning);
we decide to translate it as \emph{todavía}. In this case, we have to
write two entries in the bilingual dictionary: the entry that matches
\emph{aún} with \emph{encara} needs to have the restriction
\texttt{LR} (translation only from \texttt{es} to \texttt{ca}) and the
one that matches \emph{todavía} with \emph{encara} does not need to
have any restriction (translation in both directions).

Direction restrictions are also necessary in bilingual dictionaries
when we have words with gender to be determined ("GD") or number to be
determined ("ND") (consult page ~\pageref{ss:bil} for more
information).

The other optional attribute in entries is the lemma name
\texttt{\textsl{lm}}. Due to the employment of paradigms to represent
the inflection regularities of lexical units, an entry in
morphological dictionaries contains the part of the lemma that is
common to all the inflected forms, that is, it contains the lemma cut
at the point in which the paradigm regularity begins (for example, the
Spanish adjectives \emph{distinto}, \emph{absoluto} and \emph{marino}
appear in entries as \emph{distint}, \emph{absolut} and \emph{marin},
since the rest of the inflected forms is common to all of them and
specified in a paradigm).  This fact can make the dictionary difficult
to understand.  Therefore entries have this attribute, which contains
the whole lemma of the lexical form, so that the dictionary becomes
more understandable and linguists can solve problems quickly.  In
bilingual dictionaries, which normally do not have references to
paradigms,\footnote{They could have references to paradigms, but we
did not judge it necessary for the languages involved \nota{atenció:
ex--, vice--?}.} this attribute is not used.


\paragraph{Element for string pair \texttt{<p>}}
\label{ss:p}

This basic element of dictionaries is used in any kind of entry to
indicate the correspondence between two strings; this
correspondence specifies a lexical transformation that will be carried
out by a state path in the resulting finite-state transducer
\cite{garrido99j}.

It is defined by a pair of internal elements: The left element
(\texttt{<\textbf{l}>}) and the right element (\texttt{<\textbf{r}>}).
Its structure is shown in Figure \ref{fig:p}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{p}>
  <\textbf{l}><!-- ... --></\textbf{l}>
  <\textbf{r}><!-- ... --></\textbf{r}>
</\textbf{p}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{p}>}}
\label{fig:p}
\end{figure}

A pair \texttt{<\textbf{p}>} must include these two parts although one
can be empty, which means deleting (or inserting) a string. The
elements \texttt{<\textbf{l}>} and \texttt{<\textbf{r}>} have the same
internal structure and the same requisites.  They can contain text and
references to grammatical symbols (which, for the languages of the
present project, inflected by suffixation, are usually placed at the
end in any amount).  Outside the tags \texttt{<\textbf{l}>} and
\texttt{<\textbf{r}>} of a string pair there is nothing.


\paragraph{Element for reference to symbol \texttt{<s>}}

References to symbols (or tags) are used to specify the morphological
information of a LF and are used in any place inside a string pair,
that is, inside the elements \texttt{<\textbf{l}>} and
\texttt{<\textbf{r}>}, as if they were individual characters; for the
languages of our project, however, they are put at the end of the
pairs and always in the same order for the same word type.  This order
is decided by the linguist according to how he/she wishes to
characterise morphologically the LF in the dictionaries, and must be
the same in all the dictionaries of a system if we want that the
lexical and structural transfer operations work correctly. So, for
example, in the Romance language dictionaries of this project, a noun
has in the first place the symbol for part of speech (\textit{n},
noun), then for gender (\textit{m}, masculine, \textit{f}, feminine,
\textit{mf}, masculine--feminine), and finally for number
(\textit{sg}, singular, \textit{pl}, plural, \textit{sp},
singular--plural).  The list in Appendix \ref{se:simbolosmorf}
contains all the grammatical symbols used in the dictionaries of this
project and shows the order which has been established for each type
of word.

In morphological dictionaries, references to symbols are used in
paradigms as well as in entries which do not include any reference to
a paradigm. In bilingual dictionaries, usually only the first symbol
of each LF is specified, since the rest is automatically copied from
the source language LF to the target language LF (in the case they are
identical in both languages).

To specify which symbol we are referring to, we use the (mandatory)
attribute \texttt{\textsl{n}}.  The symbol must be defined in the
symbol definition section (\texttt{<\textbf{sdefs}>}).




\paragraph{Element for identity transduction \texttt{<i>}}
\label{ss:i}

It is a way to write a string pair in which left side and right side
are identical.  For example, the two entries shown in Figure
\ref{fig:i} are completely equivalent. The advantage of writing
entries with this element is that the result is more compact and more
readable.

\begin{figure}
\begin{small}
\begin{alltt}
[1]

<\textbf{e} \textsl{lm}="perro">
  <\textbf{p}>
    <\textbf{l}>perr</\textbf{l}><\textbf{r}>perr</\textbf{r}>
  </\textbf{p}>
  <\textbf{par} \textsl{n}="abuel_o__n"/>
</\textbf{e}>

[2]

<\textbf{e} \textsl{lm}="perro">
  <\textbf{i}>perr</\textbf{i}>
  <\textbf{par} \textsl{n}="abuel_o__n"/>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{i}>} entries [1] and [2]
are equivalent}
\label{fig:i}
\end{figure}



\paragraph{Element for paradigm definition section \texttt{<pardefs>}}

This element includes all the paradigm definitions of a dictionary,
each definition in an element \texttt{<\textbf{pardef}>}, as shown in
Figure \ref{fig:pardefs}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{pardefs}>
  <\textbf{pardef} \textsl{n}="abuel_o__n"> 
    <!-- ... -->
  </\textbf{pardef}> 
  <!-- ... -->
</\textbf{pardefs}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{pardefs}>}}
\label{fig:pardefs}
\end{figure}



\paragraph{Element for paradigm definition \texttt{<pardef>}}


It defines an inflection paradigm in the dictionary.  A paradigm can
be understood as a small dictionary of alternative transformations
that can be concatenated to parts of words (or to entries of another
paradigm) to specify regularities in the lexical processing of the
dictionary entries, such as inflection regularities. To specify these
regularities, each paradigm is a list of entries \texttt{<\textbf{e}>}
like the ones in the dictionary, that is, it has the same structure as
a dictionary section \texttt{<\textbf{section}>}; therefore, paradigm
entries consist of a pair (\texttt{<\textbf{p}>}) with left side
(\texttt{<\textbf{l}>}) and right side (\texttt{<\textbf{r}>}). These
elements can contain text or grammatical symbols
\texttt{<\textbf{s}>}.


As in symbol definitions, paradigm definitions have an attribute
\texttt{\textsl{n}} which specifies the paradigm name, so that it can
be referred to inside dictionary entries. In a dictionary entry,
therefore, one only needs to indicate the corresponding paradigm name
in order that all its possible forms get specified.

The example of paradigm definition pointed out in Figure
\ref{fig:pardefs} appears developed in Figure \ref{fig:pardef}.  The
following table shows the information expressed by the paradigm:

\begin{center}
 \begin{tabular}{|l|c|l|} 
\hline 
Root (SF and LF) & Ending (SF) & Analysis (LF) \\ 
\hline 
\texttt{abuel} & \texttt{o} &\texttt{o<n><m><sg>}\\ 
\texttt{abuel} & \texttt{a} &\texttt{o<n><f><sg>}\\ 
\texttt{abuel} & \texttt{os} &\texttt{o<n><m><pl>}\\ 
\texttt{abuel} & \texttt{as} &\texttt{o<n><f><pl>}\\ 
\hline
\end{tabular}
\end{center}


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{pardef} \textsl{n}="abuel_o__n">
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>o</\textbf{l}>
      <\textbf{r}>o<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
    </\textbf{p}>
  </\textbf{e}>
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>a</\textbf{l}>
      <\textbf{r}>o<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
    </\textbf{p}>
  </\textbf{e}>
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>os</\textbf{l}>
      <\textbf{r}>o<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
    </\textbf{p}>
  </\textbf{e}>
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>as</\textbf{l}>            
      <\textbf{r}>o<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
    </\textbf{p}>
  </\textbf{e}>      
</\textbf{pardef}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{pardef}>} to define the
  inflective morphology of Spanish nouns with four endings, such as
  \emph{abuelo, -a, -os, -as} ("grandfather, grandmother") }
\label{fig:pardef}
\end{figure}

This paradigm is assigned to all Spanish nouns (\texttt{n}) that
inflect like \emph{abuelo}, such as \emph{alumno}, \emph{amigo} or
\emph{gato}, and is designed to be used as a \textit{suffix} in
dictionary entries.  In general, paradigms can be applied to any
position of a dictionary entry (if it makes sense, of course).  We can
think of paradigms as transducers that are inserted at the point where
they are specified.  Figure \ref{fig:pardef2} shows an example of paradigm
defined to be used as a prefix. It is the paradigm used to analyse and
generate Spanish words beginning with \emph{ex}, \emph{ex-}, etc.,
like \emph{ex-presidente}, \emph{exministro}, \emph{ex director},
etc., with all the orthographic variations (\emph{ex} with hyphen,
without hyphen and joined, without hyphen and with a blank
\texttt{<\textbf{b}/>}, see page~\ref{s3:b}); the output lemma simply
adds \emph{ex} without hyphen nor blank to the accompanying lemma. The
direction restrictions (\texttt{"LR"}) that appear in the example are
used to determine which form will the translator generate. The empty
identity transduction (\texttt{<\textbf{i}/>}) is necessary in this
case to analyse and generate the word without the prefix \emph{ex}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{pardef} \textsl{n}="ex">
  <\textbf{e} \textsl{r}="LR"><\textbf{p}><\textbf{l}>ex<\textbf{b}/></\textbf{l}><\textbf{r}>ex</\textbf{r}></\textbf{p}></\textbf{e}>
  <\textbf{e}><\textbf{i}>ex</\textbf{i}></\textbf{e}>
  <\textbf{e} \textsl{r}="LR"><\textbf{p}><\textbf{l}>ex-</\textbf{l}><\textbf{r}>ex</\textbf{r}></\textbf{p}></\textbf{e}>
  <\textbf{e}><\textbf{i}/></\textbf{e}>
</\textbf{pardef}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{pardef}>} in the paradigm
  for the prefix \emph{ex}.}
\label{fig:pardef2}
\end{figure}


Entries in a paradigm can contain references to other paradigms
provided that these have been defined upper in the file.  On the other
hand, for the moment a paradigm definition can not include itself
neither directly nor indirectly.

Paradigms are used in morphological dictionaries for the analysis and
generation of lexical forms. For the language pairs of this project,
there is no need to define paradigms in bilingual dictionaries (see
page~\pageref{ss:bil}).

From Apertium 2 on, there is a new type of paradigm, called
metaparadigm, that allows the definition of paradigms with variations
according to the value of an attribute specified in each entry that
refers to that paradigm. Section \ref{ss:metaparadigmas} describes the
characteristics and use of metaparadigms.



\paragraph{Element for reference to a paradigm \texttt{<par>}}
\label{ss:par}

It is used inside an entry to indicate which inflection paradigm,
among the ones defined in \texttt{<\textbf{pardefs}>}, follows the
entry. Thanks to the references to paradigms there is no need to write
all the inflected forms of a lemma in a morphological dictionary
entry.  The attribute \texttt{\textsl{n}} is used to specify the name
of the paradigm we want to refer to.

The result of inserting a reference to a paradigm in an entry is the
creation of so many string pairs as cases specified in the
paradigm. For example, the entry in Figure \ref{fig:par}, with a
reference to the paradigm "\texttt{abuel\_o\_\_n}" (defined in Figure
\ref{fig:pardef}), is equivalent to an entry where each string pair of
the paradigm is concatenated to the lemma (that is, an entry with
every inflected form of the lemma), as shown in Figure
\ref{fig:lema_par}. In this figure, you can see that the paradigm
delivers always in the right string (\texttt{<\textbf{r}>}) the lemma
(\emph{perro}) with the grammatical symbols that apply to the surface
form, since it is from the lemma that transfer operations are carried
out.


The appropriate use of paradigms, besides enabling the creation of
compact dictionaries, improves compilation speed and reduces memory
requirements during this process, since in compilation it is possible
to create a single data structure for each one of most paradigms
\cite{ortiz05j}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="perro">
  <\textbf{i}>perr</\textbf{i}>
  <\textbf{par} \textsl{n}="abuel_o__n"/>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{par}>}}
\label{fig:par}
\end{figure}

\begin{figure}
\begin{small}
\begin{alltt}
 <\textbf{e}>
   <\textbf{p}>
     <\textbf{l}>perro</\textbf{l}>
     <\textbf{r}>perro<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
   </\textbf{p}>
 </\textbf{e}>
 <\textbf{e}>
   <\textbf{p}>
     <\textbf{l}>perra</\textbf{l}>
     <\textbf{r}>perro<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
   </\textbf{p}>
 </\textbf{e}>
 <\textbf{e}>
   <\textbf{p}>
     <\textbf{l}>perros</\textbf{l}>
     <\textbf{r}>perro<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
   </\textbf{p}>
 </\textbf{e}>
 <\textbf{e}>
   <\textbf{p}>
     <\textbf{l}>perras</\textbf{l}>            
     <\textbf{r}>perro<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
   </\textbf{p}>
 </\textbf{e}>      
\end{alltt}
\end{small}
\caption{Entry equivalent to the one in Figure \ref{fig:par}, that
  shows the result of inserting the reference to paradigm
  \texttt{<\textbf{par}>} with the paradigm defined in Figure
  \ref{fig:pardef}.}
\label{fig:lema_par}
\end{figure}



\paragraph{Element for regular expression \texttt{<re>}}
\label{ss:re}

In natural languages too there are patterns that can be recognized as
regular expressions: for example, punctuation marks, numbers (Latin or
Roman), e-mail or web page addresses, or any kind of code identifiable
through these mechanisms.


For this cases we use the string contained in the tag
\texttt{<\textbf{re}>}.  The compiler reads the regular expression
definition and transforms it in a transducer that is inserted in the
rest of the dictionary and that translates all the strings that match
the expression into identical strings.

The syntax of the present implementation of these regular expressions
processes a subgroup of Unix regular expressions, which includes the
operators \texttt{*}, \texttt{?}, \texttt{|} and \texttt{+}, as well
as groupings through parentheses and optional character ranks, for
example \texttt{[a-zA-zñú]} or its negated versions, like
\verb![^a-z]!.

By analogy, they can be seen as \texttt{<\textbf{i}>} elements, with
the difference that they can identify strings which may be infinite
(like numbers).

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{re}>[0-9]+([.,][0-9]+)?(\%)?</\textbf{re}>
  <\textbf{p}><\textbf{l}/><\textbf{r}><\textbf{s} \textsl{n}="num"/></\textbf{r}></\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Us of the element \texttt{<\textbf{re}>} in an entry for the
detection of Arabic numbers.}
\label{fig:e}
\end{figure}

Figure \ref{fig:e} shows the way to tag quantities expressed as Arabic
numbers in the dictionary.


\paragraph{Element for blank block \texttt{<b>}}
\label{s3:b}

It is used to express the presence of blanks between the words of a
multiword (see page~\pageref{ss:multipalabras} for an explanation on
multiwords). It can be inserted in the \texttt{<\textbf{i}>},
\texttt{<\textbf{l}>} and \texttt{<\textbf{r}>} elements.  In Figure
\ref{fig:b} you can see the entry for the Spanish multiword expression
\emph{hoy en día} ("nowadays"): the blanks between words are expressed
as \texttt{<\textbf{b}/>} elements inside the left and right strings.
\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="hoy en día">
  <\textbf{p}>
    <\textbf{l}>hoy<\textbf{b}/>en<\textbf{b}/>día</\textbf{l}>
    <\textbf{r}>hoy<\textbf{b}/>en<\textbf{b}/>día<\textbf{s} \textsl{n}="adv"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{b}>}}
\label{fig:b}
\end{figure}

Blanks can consist of normal space characters or of document format
information blocks encapsulated by the de-formatter
(\textit{superblanks}, see Section \ref{ss:formato}).


\paragraph{Element for post-generator activation \texttt{<a>}}
\label{ss:a} The element \texttt{<\textbf{a}>} for the activation of
the post-generator is used to indicate that a word in target language
may undergo orthographic transformations due to the contact with other
words; for example, being apostrophized, contracted, written without
intermediate spaces, etc.  These transformations need be carried out
after the generation of the target language surface forms, as until
then words are isolated and it is not possible to know which words
will get in contact . Therefore, these operations must be carried out
by the module next to the generator, which is called
post-generator. In order to signal which words are to be processed by
the post-generator, this element is used in the surface form side of
these entries in the morphological dictionary.

The example in Figure \ref{fig:a} shows its use, in a Catalan
morphological dictionary, for the preposition \textit{de}, which, when
appearing before a singular or plural masculine definite article
(\textit{el, els}), forms a contraction (\textit{del, dels}).  The
presence of the tag \texttt{<\textbf{a}/>} causes the activation of
the post-generator, which checks whether the preposition is followed
by one of the words that cause it to contract and, if it is so, makes
the contraction (see page~\pageref{ss:postgen} for more details). The
restriction \texttt{RL} indicates that this is an only-generation
entry, since it does not make any sense for the analysis.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{r}="RL" \textsl{lm}="de">
   <\textbf{p}>
      <\textbf{l}><\textbf{a}/>de</\textbf{l}>
      <\textbf{r}>de<\textbf{s} \textsl{n}="pr"/></\textbf{r}>
   </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{a}>} in a morphological
dictionary}
\label{fig:a}
\end{figure}



\paragraph{Element for group marking \texttt{<g>}}

This element is used, inside the \texttt{<\textbf{l}>} and
\texttt{<\textbf{r}>} elements, to define groups that require a
special treatment beyond the normal word by word processing. It is
used in inflective multiwords to signal the beginning and the end of
the group of invariable lexical forms (one or more) that are adjacent to the
inflected word and that, together with it, build an inseparable
unit. In Section~\ref{ss:multipalabras} you will find a detailed
explanation of the different multiword types, and in Figure
\ref{fig:hacertilin} of that section you can see an example of its
use.



\paragraph{Element for joining of lexical forms \texttt{<j>}}
\label{ss:j}

This element is used only in the right side of an entry
(\texttt{<\textbf{r}>}) to indicate that the words that form a
multiword are treated as individual lexical forms and, therefore, have
a grammatical symbol each. This way, this multiword will be processed
as a unit by the analyser and by the tagger until it reaches the
auxiliary module \texttt{pretransfer} (see section
\ref{se:pretransfer}), which is responsible for separating the lexical
forms it is made of so that they reach the transfer module as
independent forms. If the linguist wants that these forms reach the
generator as joined forms, building again a multiword, it is necessary
to define a structural transfer rule that groups them in a multiword
(see Section \ref{formatotransfer}). If, on the contrary, these joined
forms must be only for the analysis, the entry must have the
restriction \texttt{LR}.

In Section~\ref{ss:multipalabras} you will find a more detailed
explanation of this element. An example of its use can be found in
Figure \ref{fig:cont} of the mentioned section.

\subsubsection{Modification of bilingual dictionaries for the new
lexical selection module}
\label{dic_lextor}

In 2007, a new module has been added to the Apertium system: the
lexical selection module, which is described in section
\ref{se:seleccio_lex}.

In order for them to work in a lexical selection system, bilingual
dictionaries must be slightly modified so that they allow the
specification of more than one translation in target language. The
only change is the addition of two new attributes to the element
\texttt{<e>}.  Although these new attributes can be used in all the
dictionaries of a system, they only make sense in a bilingual
dictionary entry.


In Appendix~\ref{dixdtd} there is the part of the DTD \texttt{dix.dtd}
\nota{MG: no caldria ajuntar les dues DTDs en una de sola?}  where the
element \texttt{e} used for dictionary entries is defined.  The new
attributes are:
\begin{description}
\item[slr (\emph{sense from left to right})] is used to specify the
\emph{translation mark} when there is more than one translation from
left to right for the lemma specified in the left side of an
entry. The attribute can receive any value; however, the recommended
action is to assign as value the lemma contained in the right part
\texttt{<r>} (the translation of the lemma).
\item[srl (\emph{sense from right to left})] is used to specify the
\emph{translation mark} when there is more than one translation from
right to left for the lemma specified in the right side of an entry.
As before, the attribute can receive any value, but the recommended
action is to assign as value the lemma contained in the left part
\texttt{<l>} (the translation of the lemma).
\end{description}

Furthermore, in both cases the value of the attribute can end in a
white space and the letter ``D'' to indicate that this is the default
translation, that is, the translation that will be chosen when there
is not enough information to make a decision. It is compulsory that,
for entries that have more than one equivalent in target language, one
of the equivalents, and only one, is marked with the letter ``D'' for
\emph{default}.

The following example shows how the new attributes are used.  We take
as example a bilingual English-Catalan dictionary, with the following
entries having more than one translation in the target language:
\begin{itemize}
\item \emph{look}: can be translated into Catalan as \emph{mirar}
(default) or as \emph{semblar} (according to the English senses
\emph{view/seem}),
\item \emph{floor}: can be translated into Catalan as \emph{pis}
(default) or as \emph{terra} (according to the English senses
\emph{level of building/ground}),
\item \emph{pis}: can be translated into English as \emph{flat}
(default) or as \emph{floor}.
\end{itemize}

This information is represented by means of the two attributes
described:\label{entrades_lextor}
\begin{alltt}
\begin{small}
<e srl="flat D">
   <p>
      <l>flat<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
    </p>
</e>

<e slr="pis D" srl="floor">
   <p>
      <l>floor<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
   </p>
</e>

<e slr="terra">
   <p>
      <l>floor<s n="n"/></l>
      <r>terra<s n="n"/><s n="m"/></r>
   </p>
</e>

<e slr="mirar D">
   <p>
      <l>look<s n="vblex"/></l>
      <r>mirar<s n="vblex"/></r>
   </p>
</e>

<e slr="semblar">
   <p>
      <l>look<s n="vblex"/></l>
      <r>semblar<s n="vblex"/></r>
   </p>
</e>
\end{small}
\end{alltt}


%\settocdepth{paragraph}

\subsubsection{Particularities of the different dictionary types}
\label{ss:morfgen}

Dictionary entries have different characteristics depending on the
dictionary type. Although some of these characteristics have been
presented in the previous sections, we are going to describe them here
more exhaustively.


\paragraph{Morphological dictionaries}

In these dictionaries, used to generate the system's morphological
analysers and generators, it is necessary to mark with
\texttt{<\textbf{a}/>} those surface forms which, once generated, may
need certain orthographic transformations due to the contact with
other words; these operations are carried out by the post-generator.
As these marks are only generated, the entries containing them must be
only for the generation, which means that need to have the restriction
\texttt{\textsl{r}=}\verb!"RL"! (from right to left).  Figure
\ref{fig:a} shows an entry containing this element.



\paragraph{Bilingual dictionaries}
\label{ss:bil}

As explained before, we have not used paradigms in the bilingual
dictionaries of our system; these dictionaries are built with generic
entries in which, almost always, only part of speech is specified, and
which do not have inflection information. For example, in the
\texttt{es-ca} dictionary, the entry for the Spanish words
\textit{pan}, \textit{panes} ("bread"), translated into Catalan as
\textit{pa}, \textit{pans}, would be as shown in Figure \ref{fg:pan}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>pan<\textbf{s} \textsl{n}="n"/></\textbf{l}>
    <\textbf{r}>pa<\textbf{s} \textsl{n}="n"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Bilingual dictionary entry for the translation \emph{pan}
(\texttt{es})--\emph{pa} (\texttt{ca})}
\label{fg:pan}
\end{figure}


As you can see in the figure, only the first grammatical symbol
\texttt{<\textbf{s} \textsl{n}="\ldots}\texttt{"}\texttt{/>} of each
word is specified, since the unspecified symbols that come after the
specified ones in the bilingual dictionary are copied from the source
lexical form to the target lexical form. This entry, therefore, works
both for \textit{pan} (singular) and for \textit{panes} (plural): the
morphological analyser delivers the lemma (\emph{pan}) followed by the
grammatical symbols that apply to the analysed surface form (\emph{n m
sg} or \emph{n m pl} as applicable), and the symbols that are not
specified in the bilingual entry (\emph{m sg} or \emph{m pl}) are
copied to the target language. This is valid for both translation
directions.  The idea is to specify the information indispensable to
differentiate the entries, and the rest is \textit{deduced}
(copied). It is important to bear this in mind, because, when there
are differences between the grammatical symbols of a lexical form from
SL to TL, these differences must be specified in the bilingual
dictionary.  For example, when between source word and translated word
there is a gender or number change, one has to specify the grammatical
symbols in order (the order in which these symbols appear in the
morphological dictionaries)\footnote{To know which grammatical symbols
have been used in the dictionaries and in which order, see Appendix
\ref{se:simbolosmorf}.} until the symbol that changes between SL and
TL is reached.

For example, to translate the Spanish word \textit{cama}, feminine
noun, into the Catalan word \textit{llit}, masculine noun, the entry
in the bilingual dictionary must be as shown in Figure
\ref{fg:cama}. The gender must be specified (\emph{f}, \emph{m})
because, if not, the symbols for gender and number would be copied
from the SL lexical form into de TL lexical form. Therefore, when
translating from \texttt{es} to \texttt{ca}, we would obtain the
lexical form \emph{llit} with the symbols \texttt{n f sg} or \texttt{n
f pl}. In both cases, the generator would receive as input a word that
is impossible to generate, since the Catalan morphological dictionary
does not contain any entry with lemma \emph{llit} and feminine gender.


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>cama<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/></\textbf{l}>
    <\textbf{r}>llit<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Bilingual dictionary entry for the translation \emph{cama}
(\texttt{es})--\emph{llit} (\texttt{ca})}
\label{fg:cama}
\end{figure}


In this example, the number symbols are not specified; therefore, it
works for the correspondence \textit{cama--llit} (singular) as well as
for \textit{camas--llits} (plural).  However, when there is a number
change, the only way is to specify also the gender if the order used
in all the dictionary for grammatical symbols is \emph{gender,
number}.



By means of a direction restriction \texttt{r} we can indicate which
translations are to be done only in one direction and not in the other
one (see the description of the restrictions \texttt{LR} and
\texttt{RL} in page \pageref{restric}).  This is necessary when the
correspondence between two lexical forms is not symmetrical; in such
case, in the bilingual dictionary two or more entries have to be
created and a direction restriction must be applied, like in the
example shown in Figure~\ref{fg:postre}. In this example, when
translating from Spanish to Catalan (\texttt{LR}), we must generate
only plural forms, since the word \textit{postres} ("dessert" ) in
Catalan does not have singular form.  But, on the other hand, we will
translate into Spanish only in plural form (although in Spanish the
word has singular and plural forms), since it is not possible to
determine, from the Catalan word, whether the number should be
singular or plural.

\begin{figure}[htbp]
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{r}="LR">
  <\textbf{p}>
    <\textbf{l}>postre<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{l}>
    <\textbf{r}>postres<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>

<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>postre<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{l}>
    <\textbf{r}>postres<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Entries in the Spanish-Catalan bilingual dictionary for the
correspondence \emph{postre}--\emph{postres} ("dessert")}
\label{fg:postre}
\end{figure}


\label{pg:GD} There is another problem due to grammatical divergences
between two languages that is resolved with the help of two special
symbols, \texttt{GD} (for \textit{gender to be determined}) and
\texttt{ND} (for \textit{number to be determined}), symbols which have
to be defined in the symbol section of the bilingual dictionary. This
problem arises when the grammatical information of a SL lexical form
is not enough to determine the gender (masculine or feminine) or the
number (singular or plural) of the TL lexical form.  Let's put an
example: the Spanish adjective \textit{común} ("common") is masculine
and feminine at the same time (and, therefore, masculine--feminine,
\texttt{mf}), but in Catalan the adjective has different forms for the
masculine, \textit{comú}/\textit{comuns}, and the feminine,
\textit{comuna}/\textit{comunes}.  In the bilingual dictionary, the
entry should be as shown in Figure~\ref{fg:comuna}: in the \texttt{LR}
direction (from Spanish to Catalan), the gender information is not
\texttt{m}, \texttt{f} nor \texttt{mf} but \texttt{GD}; this
\textit{gender to be determined} will be determined next by the
structural transfer module, by means of the application of the
suitable transfer rules (usually, rules for the agreement between the
lexical forms in a pattern; see Section \ref{ss:transfer} to obtain a
detailed description of transfer rules). In an analogous way, a
similar mechanism exists for singular--plural using the symbol
\texttt{ND} (for example, in Spanish \textit{análisis} ("analysis") is
singular and plural, whereas in Catalan the singular form is
\textit{anàlisi} and the plural form \textit{anàlisis}).


\begin{figure}[htbp]
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{r}="LR">
  <\textbf{p}>
    <\textbf{l}>común<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>comú<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="GD"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>

<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>común<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>comú<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>

<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>común<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>comú<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="f"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Entries in the Spanish--Catalan bilingual dictionary for the
  correspondence \emph{común}--\emph{comú} ("common"), the first one
  for the translation from Spanish to Catalan and the two others for
  the translation from Catalan to Spanish}
\label{fg:comuna}


\end{figure}



\paragraph{Post-generation dictionaries}
\label{ss:postgen}



In the morphological dictionary, the lexical forms which, once
generated, may undergo contraction, apostrophation or other
transformations, depending of which words are in contact with them in
the output text, must have the post-generator activation mark
(\texttt{<\textbf{a}/>}, see page \pageref{ss:a}) in the generation
entry (\texttt{RL} direction).  It is essential that the surface forms
marked with the post-generator activation mark are identical in the
morphological and the post-generation dictionaries of the same
translator. In the post-generation dictionary, all entries begin with
this activation mark.


In Figure~\ref{fg:postgen} there is an extract of the Spanish
post-generator; the example shows how the contraction for \textit{de}
and \textit{el} is done, to form the word \textit{del}.  The paradigm
\texttt{puntuación} not defined in the example contains the
non-alphabetic characters that can appear in a text. We can see in the
example that the entry for the preposition \emph{de} has the mark
\texttt{<\textbf{a}/>}. The paradigm assigned to this entry,
"\texttt{el}", is the one defined just above. According to this entry,
when the system receives as input the left string of the entry (the part
between \texttt{<\textbf{l}>}) concatenated to the left string of
the paradigm (that is, when the input is
\texttt{"}\texttt{<a/>\textbf{de}<b/>\textbf{el}<b/>"} or
\texttt{"}\texttt{<a/>\textbf{de}\\<b/>\textbf{el}[puntuación]}\texttt{"}),
the module delivers as output string (the part between \texttt{<r>}
elements) the string \texttt{"}\textbf{del}\texttt{"} followed by the
blanks represented with \texttt{<b/>} or by the symbols represented
with \texttt{[puntu\-a\-ción]}. Note that, in the module output, all
the marks \texttt{<\textbf{a}/>} have been removed.





\begin{figure}[htbp]
\begin{small}
\begin{alltt} 
<\textbf{dictionary}>
<\textbf{pardefs}>
  ... 
  <\textbf{pardef} \textsl{n}="el">
    <\textbf{e}>
      <\textbf{p}>
        <\textbf{l}>el<\textbf{b}/></\textbf{l}>
        <\textbf{r}>l<\textbf{b}/></\textbf{r}>
      </\textbf{p}>
    </\textbf{e}>
    <\textbf{e}>
      <\textbf{p}>
        <\textbf{l}>el</\textbf{l}>
        <\textbf{r}>l</\textbf{r}>
      </\textbf{p}>
      <\textbf{par} \textsl{n}="puntuación"/>
    </\textbf{e}>
  </\textbf{pardef}>
  ... 
</\textbf{pardefs}>
<\textbf{section} \textsl{id}="main" \textsl{type}="standard">
  ... 
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}><\textbf{a}/>de<\textbf{b}/></\textbf{l}>
      <\textbf{r}>de</\textbf{r}>
    </\textbf{p}>
    <\textbf{par} \textsl{n}="el"/>
  </\textbf{e}>
  ... 
</\textbf{section}/>
</\textbf{ditionary}>
\end{alltt}
\end{small}
\caption{Post-generation dictionary data to perform the contraction
  for Spanish \emph{de} + \emph{el} = \emph{del} .}
\label{fg:postgen}
\end{figure} \nota{en l'exemple, "el" no ha de portar la marca
d'activació oi? - l'he treta de l'exemple, treure-la dels diccionaris
(Mikel?)}


%\settocdepth{subsubsection}


\subsubsection{Multiword lexical units}
\label{ss:multipalabras}


The designed dictionary format allows the creation of
\textit{multiword lexical units} ---in short, \textit{multiwords}---
of different kinds, depending on the problem to be approached.

In this project we have considered three basic types of multiwords:
\begin{enumerate}
\item The most simple case are \textit{multiwords without inflection},
  which consist of only one lexical form: the lemma is made of two or
  more invariable orthographic words but it is tagged as a unit.
  Figure \ref{fig:msf} shows an example of invariable multiword (the
  Spanish expression \emph{hoy en día}, "nowadays"): It is made of
  three words separated by a blank (\texttt{<\textbf{b}/>}) and,
  although it actually consists of an adverb, a preposition and a
  noun, it is tagged as an adverb as a whole, since it acts as one.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="hoy en día">
  <\textbf{p}>
    <\textbf{l}>hoy<\textbf{b}/>en<\textbf{b}/>día</\textbf{l}>
    <\textbf{r}>hoy<\textbf{b}/>en<\textbf{b}/>día<\textbf{s} \textsl{n}="adv"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Example of multiword without inflection in the morphological
dictionary}
\label{fig:msf}
\end{figure}

\item A more complicated issue is the case of \textit{compound
  multiwords}, made of more than one lexical form, each one with its
  grammatical symbols. The words they are made of are considered not
  to build a semantic unit like in the previous case, but to appear
  together building a unit due to contact reasons (phonetic or
  orthographic reasons). In this category we include
  \textit{contractions} and \textit{enclitic pronouns} accompanying
  verbs.  To mark this phenomenon we use the tag \texttt{<\textbf{j}>}
  described in page~\pageref{ss:j}. You can see an example in
  Figure~\ref{fig:cont}, in which the analysis of \emph{del} delivers
  a lexical multiform made of two lexical forms: \emph{de},
  preposition, and \emph{el}, singular masculine definite determiner,
  linked with the \texttt{<\textbf{j}/>} element. The analyser and the
  part-of-speech tagger handle this multiwords as a unit; however,
  before entering the transfer module, they are processed by an
  auxiliary module called \texttt{pretransfer} (see section
  \ref{se:pretransfer}) which is responsible for separating the
  lexical forms they are made of. This way, they reach the transfer
  module as independent forms; the linguist has to decide whether they
  have to be joined again (which must be done in the structural
  transfer module) or they have to remain as independent forms through
  the next modules.


  In our system, the elements forming a contraction continue as
independent forms, and the post-generator is responsible for making
the contractions in the target language if it is necessary. On the
other hand, enclitic pronouns are joined again to the verb by means of
a structural transfer rule (see Section \ref{ss:transfer}), so the
verb plus its enclitic pronouns get into the generation module as a
single lexical multiform, its components joined with a
\texttt{<\textbf{j}/>}. Therefore, entries containing enclitic
pronouns must not have any direction restriction, as can be seen in
the example in Figure \ref{fig:encl}, which shows a part of the
paradigm for the Spanish verb "dar" ("to give"), specifically the
entry for the infinitive form joined to an enclitic pronoun.


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="del" \textsl{r}="LR">
  <\textbf{p}>
    <\textbf{l}>del</\textbf{l}>
    <\textbf{r}>de<\textbf{s} \textsl{n}="pr"/><\textbf{j}/>
     el<\textbf{s} \textsl{n}="det"/><\textbf{s} \textsl{n}="def"/>
     <\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{Entry in the morphological dictionary for the analysis of a
contraction (the Spanish contraction \emph{del})}
\label{fig:cont}
\end{figure}

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>ar</\textbf{l}>
    <\textbf{r}>ar<\textbf{s} \textsl{n}="vblex"/><\textbf{s} \textsl{n}="inf"/><\textbf{j}/></\textbf{r}>
  </\textbf{p}>
  <\textbf{par} \textsl{n}="S__cantar"/>
</\textbf{e}>
\end{alltt}
\end{small}
\caption{A fragment of the inflection paradigm for the Spanish verb
\emph{dar} ("to give"), which shows the entry for the infinitive form
followed by an enclitic pronoun. Enclitic pronouns are contained in
the paradigm \texttt{S\_\_cantar}. Note that, unlike in Figure
\ref{fig:cont}, this entry is both for analysis and generation.}
\label{fig:encl}
\end{figure}



\item The most complicated case in our system is the case of
  \textit{multiwords with inner inflection} inside the lemma (or
  "split lemma" forms), like the example shown in Figure
  \ref{fig:echardemenos}. The lemma of this kind of multiwords has one
  part with inflection (the \emph{lemma head}) followed by one
  invariable part (the \emph{lemma tail}).  The invariable part has to
  be put between \texttt{<\textbf{g}>} elements, so that it can be
  moved to the position immediately after the lemma head to obtain the
  whole lemma of the multiword. For example, the lemma of the Spanish
  multiwords \emph{echó de menos} ("he/she missed"), \emph{echándole
  de menos} ("missing him/her"), etc.  has to be \emph{echar de menos}
  ("to miss"), since this form will be the one searched in the
  bilingual dictionary to find its translation.  This means that the
  invariable lemma tail (\emph{de menos}) has to be moved after the
  uninflected lemma head (\emph{echar}). This moving backwards will be
  done by the auxiliary module \texttt{pretransfer} (see section
  \ref{se:pretransfer}) which runs before the structural transfer
  module.

  To understand the example in Figure \ref{fig:echardemenos}, you have
  to be aware that the paradigm defining the verb \emph{echar}
  includes, besides the verb inflection, the enclitic pronouns that
  can appear at the end of the inflected forms of the verb; in the
  output lexical multiform, this enclitic pronouns are joined using
  the empty element \texttt{<\textbf{j}/>}.



\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="echar de menos"> 
  <\textbf{i}>ech</\textbf{i}>
  <\textbf{par} \textsl{n}="aspir/ar__vblex"/> <!-it includes enclitic pronouns -->
  <\textbf{p}>
    <\textbf{l}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{l}>
    <\textbf{r}><\textbf{g}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{g}></\textbf{r}>
  </\textbf{p}> 
</\textbf{e}>
\end{alltt}
\end{small}
\caption{A morphological dictionary entry containing a
  \texttt{<\textbf{g}>} group.}
\label{fig:echardemenos}
\label{fig:hacertilin}
\end{figure}



When the translation is also a \emph{split lemma} (for example, the
translation of "to miss" in Catalan is \emph{trobar a faltar}, with
forms like \emph{trobem a faltar}, \emph{trobar-lo a faltar}, etc.),
it is necessary to place again the lemma tail in its original place,
after the inflected form plus the enclitic pronouns (if any), and
indicate the correspondence of these invariable parts of the lemma
(\emph{de menos}, \emph{a faltar}) at both sides of the
translation. So, in the example of Figure ~\ref{fig:echardemenos}, the
\texttt{<\textbf{g}>} element is used to mark the group
`\texttt{<b/>de<b/>menos}' in the morphological dictionary, whereas in
the bilingual dictionary (see Figure~\ref{fig:menosfaltar}), the
\texttt{<\textbf{g}>} element is used to establish the correspondence
between the groups ``\texttt{<b/>de<b/>menos}'' and
``\texttt{<b/>a<b/>faltar}''. \nota{I com serà el cas de ``dirección
general'' - ``direcciones generales''?}

If the translation is not a \emph{split lemma}, you do not need to
insert any \texttt{<\textbf{g}>} element in the target language
string.

\end{enumerate}

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>echar<\textbf{g}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{g}><\textbf{s} \textsl{n}="vblex"/></\textbf{l}>
    <\textbf{r}>trobar<\textbf{g}><\textbf{b}/>a<\textbf{b}/>faltar</\textbf{g}><\textbf{s} \textsl{n}="vblex"/></\textbf{r}>
  </\textbf{p}> 
</\textbf{e}>
\end{alltt}
\end{small}
\caption{A bilingual dictionary entry containing two corresponding
\texttt{<\textbf{g}>} groups.}
\label{fig:menosfaltar}
\end{figure}

\subsubsection{Metaparadigms}
\label{ss:metaparadigmas}


\nota{Marco diu: Especificar la DTD?}


When developing the dictionaries for the Occitan translator, we were
faced with a new need: we wanted to be able to specify paradigms for
verbs that had a same inflection pattern but whose root changed in the
different inflected forms.  With the existing paradigm system, a new
paradigm had to be created for each of these verbs, since it was only
possible to specify an inflection regularity pattern for a group of
verbs with invariable root. With metaparadigms, it is possible to
specify the inflection regularity as well as verb root variations.

At the same time, metaparadigms allow the specification, in a single
paradigm, of variations in the grammatical symbols of a lemma.  That
is, several lemmas can refer to a same metaparadigm even if they have
different grammatical symbols. Whereas for Occitan, metaparadigms have
allowed having a same paradigm for entries with root variations, for
English, these have allowed having a same paradigm for entries with
variations in their grammatical symbols.


Related with this, we created the concept of metadictionary: it is a
dictionary which contains metaparadigms as well as the normal
paradigms used so far. The name of a metadictionary is
\texttt{apertium-PAIR.}$L_1$\texttt{.metadix}
(for example, for the English monolingual dictionary in the
Apertium-en-ca system, \texttt{apertium-en-ca.en.metadix}).  When
linguistic data are compiled these dictionaries are pre-processed, so
that they have the appropriate format for the dictionary compiler.

\paragraph{Specification of metaparadigms}

Metaparadigms are defined in the \texttt{<\textbf{pardefs}>} section
of the monolingual dictionary, the same section where also the rest of
the dictionary paradigms are defined. A metaparadigm, just like a
paradigm, has a name specified in the attribute \texttt{n}.  This name
will have the same characteristics as in the other paradigms, with the
difference that the variable part of the lemma root will be in brackets and
in capital letters, as you can see in this example:

\begin{alltt} 
<\textbf{pardef} n="m/é[T]er\_\_vblex">
\end{alltt}

This is the definition of a verb paradigm, where the inflection
endings have a variable part in the root.  The inflection paradigms
specified inside this metaparadigm have to present inflection only in
the part at the right of the brackets, for example like the one
specified in the paradigm:

\begin{alltt} 
<\textbf{par} n="mét/er\_\_vblex"/>
\end{alltt}


In conclusion, a complete example of metaparadigm definition would be:


\begin{alltt}
<\textbf{pardef} n="m/é[T]er__vblex">
  <\textbf{e}> 
    <\textbf{p}> 
      <\textbf{l}>e</\textbf{l}>
      <\textbf{r}>é</\textbf{r}> 
    </\textbf{p}>
    <\textbf{i}><prm/></\textbf{i}> 
    <\textbf{par} n="sent/eria__vblex"/>
  </\textbf{e}> 
  <\textbf{e}> 
    <\textbf{i}>é<prm/></\textbf{i}>
    <\textbf{par} n="mét/er__vblex"/> 
  </\textbf{e}> 
</\textbf{pardef}>

\end{alltt}


The tag \texttt{<\textbf{prm}/>} is the marker that is used to place
the variable text part (the root variation) in the paradigm
definition.


Once a metaparadigm is defined, we may want that a verb uses it. To do
so, in the verb entry (inside a \texttt{<\textbf{e}>} element) we must
indicate the suitable metaparadigm and, through the attribute
\texttt{prm}, define with which letters we want to replace the
variable part specified in brackets. For example:

\begin{alltt} 
<\textbf{e} lm="acuélher"> 
  <\textbf{i}>acu</\textbf{i}>
  <\textbf{par} n="m/é[T]er__vblex" prm="lh"/> 
</\textbf{e}>

\end{alltt}

This entry defines the Occitan verb \emph{acuélher} ("to receive") and
specifies that its inflection paradigm is the one defined by the
metaparadigm \texttt{m/é[T]er\_\_vblex}, but replacing \texttt{T} with
\texttt{lh}; that is, the letters following \emph{acu} will be
\emph{élher} instead of \emph{éter}.



As mentioned before, metaparadigms can also be used for entries which
have some variation in their grammatical symbols. The way to specify
them is basically the same: the variable part must be specified in the
entry with the attribute \texttt{sa}, whereas in the paradigm the tag
\texttt{<\textbf{sa}>} has to be placed where the optional grammatical
symbol should appear.

For example, we have the following metaparadigm:

\begin{alltt}
<\textbf{pardef} n="house__n">
  <\textbf{e}> 
    <\textbf{p}> 
      <\textbf{l}/> 
      <\textbf{r}><\textbf{s} n="n"/><sa/><\textbf{s} n="sg"/></\textbf{r}> 
    </\textbf{p}>
  </\textbf{e}>
  <\textbf{e}> 
    <\textbf{p}> 
      <\textbf{l}>s</\textbf{l}>
      <\textbf{r}><\textbf{s} n="n"/><sa/><\textbf{s} n="pl"/></r>
    </\textbf{p}> 
  </\textbf{e}>
</\textbf{pardef}>

\end{alltt}


and the following entry:

\begin{alltt} 
<\textbf{e} lm="time"> 
  <\textbf{i}>time</\textbf{i}>
  <\textbf{par} n="house__n" sa="unc"/> 
</\textbf{e}>
\end{alltt}

where \emph{unc} means that the noun is uncountable.

In the metaparadigm, the tag \texttt{<\textbf{sa}>} shows the place
where the grammatical symbol is to be placed if an entry contains the
attribute \texttt{sa} with a value, as happens in the entry for
\emph{time}.


A dictionary which contains entries like the ones described here is
called metadictionary and must be pre-processed in order to generate a
dictionary that follows the DTD for Apertium 2, since the engine does
not allow the direct use of metaparadigms. The next section describes
how is this pre-processing like.




\paragraph{Pre-processing of the metadictionary}


A metadictionary is an XML file to which two XSLT style sheets are
applied, in order to pre-process the metaparadigms and obtain a
dictionary with all the paradigms derived from the metaparadigms.  The
first style sheet, \texttt{buscaPar.xsl}, produces the list of verbs
that use metaparadigms and deletes the possible repetitions of
metaparadigms to be expanded. This style sheet generates, in
combination with the sheet \texttt{principal.xsl}, a second style
sheet called \texttt{gen.xsl}, which processes the metadictionary with
the list of metaparadigms to be expanded and generates a dictionary in
Apertium 2 format. Basically, what this generated style sheet does is:

\begin{enumerate}


\item In verb entries, if a verb uses a metaparadigm, this
metaparadigm is replaced by the corresponding expanded and
deparametrized paradigm. Thus, the previous example entry:

\begin{alltt}
<\textbf{e} lm="acuélher">
  <\textbf{i}>acu</\textbf{i}>
  <\textbf{par} n="m/é[T]er__vblex" prm="lh"/>
</\textbf{e}>
\end{alltt}

        would be deparametrized and expanded into:

\begin{alltt} 
<\textbf{e} lm="acuélher">
  <\textbf{i}>acu</\textbf{i}>
  <\textbf{par} n="m/élher__vblex"/>
</\textbf{e}>
\end{alltt}


\item On the other hand, since from the first pass the system knows
which paradigms have to be created from metaparadigms, these are
created.  In the previous example, from the metaparadigm:

\begin{alltt}
<\textbf{pardef} n="m/é[T]er__vblex">
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>e</\textbf{l}>
      <\textbf{r}>é</\textbf{r}>
    </\textbf{p}>
    <\textbf{i}><prm/></\textbf{i}>
    <\textbf{par} n="sent/eria__vblex"/>
  </\textbf{e}>
  <\textbf{e}>
    <\textbf{i}>é<prm/></\textbf{i}>
    <\textbf{par} n="mét/er__vblex"/>
  </\textbf{e}>
</\textbf{pardef}>     
\end{alltt}

        the system would generate the paradigm
        \texttt{"m/élher\_\_vblex"} :

\begin{alltt}
<\textbf{pardef} n="m/élher__vblex">
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}>e</\textbf{l}>
      <\textbf{r}>é</\textbf{r}>
    </\textbf{p}>
    <\textbf{i}>lh/></\textbf{i}>
    <\textbf{par} n="sent/eria__vblex"/>
  </\textbf{e}>
  <\textbf{e}>
    <\textbf{i}>élh</\textbf{i}>
    <\textbf{par} n="mét/er__vblex"/>
  </\textbf{e}>
</\textbf{pardef}>  
\end{alltt}

\end{enumerate}

After the metadictionary has been processed according to these steps,
a .dix dictionary is generated which follows the DTD for Apertium 2
and which can already be compiled.

 
In the case of our second example, where the variable part was the
sequence of grammatical symbols in the paradigm, the style sheets
would be applied and, from the value \emph{unc} specified in the
attribute \texttt{sa}, the following paradigm would be generated:

\begin{alltt}
<\textbf{pardef} n="house__n__unc">
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}/>
      <\textbf{r}><\textbf{s} n="n"/><\textbf{s} n="unc"/><\textbf{s} n="sg"/></\textbf{r}>
    </\textbf{p}>
 </\textbf{e}>
 <\textbf{e}>
   <\textbf{p}>
     <\textbf{l}>s</\textbf{l}>
     <\textbf{r}><\textbf{s} n="n"/><\textbf{s} n="unc"/><\textbf{s} n="pl"/></r>
   </\textbf{p}>
 </\textbf{e}>
</\textbf{pardef}>

\end{alltt}

for nouns the morphological analysis of which should be (in data
stream format):


\begin{alltt} 
time<n><unc><sg>
\end{alltt}

In this case, metaparadigms allows the use of the same paradigm for
entries with the same inflection but with a slightly different
morphological analysis.

It is important to note that, when a dictionary uses metaparadigms
and, accordingly, its name has the extension \texttt{.metadix}, this
will be the file where dictionary changes have to be made (adding,
changing or deleting entries or paradigms), since the file
\texttt{.dix} is automatically generated from this one every time
linguistic data are compiled and, therefore, any changes made in the
latter will be overwritten during compilation.



\subsection[Automatic generation of the modules]{Automatic generation
of the lexical processing modules}
\label{se:compiladoresdic}


The four lexical processing modules (morphological analyser, lexical
transfer, morphological generator and post-generator) are compiled
from dictionaries by means of a single compiler based
on letter transducers \cite{roche97}. This compiler is much faster
than the ones used in the systems \textsf{interNOSTRUM}
\cite{canals01b,garridoalenda01p,garrido99j} and \textsf{Traductor
Universia} \cite{garrido03p, gilabert03j}, thanks to the use of new
compiler building strategies and the minimization of partial
transducers during the building process \cite{ortiz05j}.

The division of dictionary entries into lemma and paradigm enables the
effective construction of minimal letter transducers. The compiler
makes the most of the factorization allowed by paradigms in order to
speed up the construction. Taking into account that, in most European
languages, word variations occur at the end or the beginning of words,
we took advantage of this fact to improve the construction speed of
the minimal transducer.

Paradigms are also minimized before being inserted in the big
transducer in order to reduce the size of the big transducer before
its minimization.  Since, before minimizing, the paradigms of the
dictionaries for the languages we have dealt with usually have just a
few hundreds of states, the minimization of these paradigms is a very
fast process.

If we assume that an entry can have at any point a reference to a
paradigm, we could decide to copy at this point the transducer
calculated in the paradigm definition. The method used in
\emph{Apertium} is based on the idea that it is not always necessary
to copy, because in certain cases it is possible to reuse a paradigm
that was already copied.  In particular, two or more entries that
share a paradigm as a suffix can reuse the same copy of this paradigm;
the same can be said when it is as a prefix. However, generally it is
not possible to reuse paradigms if they are located in intermediate
positions of different entries, since new suffixes (or prefixes) can
be added to existing entries, which causes the information inserted in
the transducer not to be consistent with the dictionary, and therefore
the generated transducer would be incorrect (it would add string pairs
that are not present in the formal language defined by dictionaries).

Minimal letter transducers are built as explained next. From a string
transduction it is possible to build a \textit{sequence of letter
transductions} $S(s:t)$ with length $N = \max(|s|,|t|)$ which is
defined as follows for each element $1 \leq i \leq N$:


\begin{equation}
\label{eq:transletras} S_i(s:t)=\left\{
\begin{array}{ll} 
(s_i:\theta) & \textrm{if } i \leq |s| \wedge i > |t| \\ 
(\theta:t_i) & \textrm{if } i \leq |t| \wedge i > |s| \\
(s_i:t_i) & \textrm{in other cases}
\end{array}\right.
\label{e:montaje}
\end{equation}
 
It should be emphasized that the construction design forbids the
existence of a $(s:t)$ that is equal to $(\epsilon:\epsilon)$, which
is crucial for the consistence of the building method.

The building method uses two procedures: the \textit{assembly}
procedure inferred from equation \ref{e:montaje}, and the minimization
procedure, which is executed by a conventional minimization algorithm
\cite{vandesnepscheut93b} for deterministic finite state automata,
which consists of inverting, determining, inverting again and
determining again, taking as the alphabet of the automaton to be
minimized the Cartesian product of $L$ and as empty transition the
$\left(\theta:\theta\right)$.


\begin{figure}
\begin{center}
\includegraphics[width=10cm]{fig1}
\end{center}
\caption{Building of the dictionary as prefix acceptor and link to
paradigms through transitions $\left(\theta:\theta\right)$.}
\label{fig:construccion}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=8cm]{fig2}
\end{center}
\caption{Minimized paradigm "-es \textbf{n m}" used in Figure
\ref{fig:construccion}.}
\label{fig:paradigmapan}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=8cm]{fig3}
\end{center}
\caption{Minimized paradigm "z/-ces \textbf{n m}" used in Figure
\ref{fig:construccion}.}
\label{fig:paradigmavez}
\end{figure}



Figure \ref{fig:construccion} shows a simplified example of the
assembly process.  Transductions, composed as in the equation
\ref{e:montaje}, are inserted one by one in a transducer in the form
of a \textit{prefix acceptor} or \textit{trie}, that is, in a way that
there is only one node for each common prefix of the group of
transductions that form the dictionary.  With the suffixes of the
transductions (that are not shared) new states are created.  In the
point where there is a reference to a paradigm, a replica of this
paradigm is created and a link is created to the dictionary entry
which is being inserted in the transducer by means of a null
transduction $\left(\theta:\theta\right)$.

Each paradigm, as it can be seen as a little dictionary, has been
built according to this same procedure and been minimized to reduce
the size of the content when building the big dictionary. In Figures
\ref{fig:paradigmapan} and \ref{fig:paradigmavez} you can see the
state of the paradigms used in Figure \ref{fig:construccion} after its
minimization.



\section{Part-of-speech tagger}
\label{ss:tagger}

\subsection{Module description }
\label{functagger}


The part-of-speech tagger is based on first-order hidden Markov
models~\cite{rabiner89}, that is, on statistical data. The states of
the Markov model represent parts of speech, and the observable
parameters are ambiguity classes~\cite{cutting92a}, formed by groups
of parts of speech.

In spite of working with statistical information, the training and
behaviour of the tagger improve with the application of restrictions
that forbid certain sequences of parts of speech (in the first-order
models, these sequences can only include two parts of speech). For
example, in Spanish or Catalan a preposition can never be followed by
a verb in personal form; this restriction is of great help when the
word after a preposition is ambiguous and one of its possible analyses
is a verb in personal form (e.g., \emph{de trabajo}, \emph{en
libertad}, etc.).  Restrictions are explicitly declared in the tagger
definition file, sometimes in the form of \emph{prohibitions} and
sometimes of \emph{obligations}.

The morphological tags which the tagger works with are not the same as
the ones used in the morphological analyser. Usually, the information
delivered by the analyser is too detailed for the purposes of the
part-of-speech disambiguation (for example, for most purposes, it
suffices to group in the same category all common nouns, regardless of
their gender and number). The use of finer-grained tags does not improve the
results, whereas it increases the number of parameters to be estimated
and intensifies the problem of lack of linguistic resources such as
manually disambiguated texts. For this reason, in the tagger file one
has to specify how to group the \emph{fine-grained} tags delivered by the
morphological analyser into more general \emph{coarse} tags ---which
we will call \emph{categories}--- that will be used in the
part-of-speech disambiguation. Apart from coarse categories, one can
also define lexicalized tags. Basically there are two types of
lexicalizations described in bibliography: one type adds new
observables and the other one, in addition, adds new states to the
Markov model~\cite{pla04}; the tagger in Apertium uses the latter
lexicalization type.

It is important to note that, in spite of working with \emph{coarse}
categories, the tagger outputs fine-grained tags like the ones from the
morphological analyser. Sometimes it may occur that the morphological
analyser delivers, for a certain word, two or more fine-grained tags that can
be grouped under the same tagger category: e.g. in Spanish
\emph{cante} can be the 1st or the 3rd singular person of the
subjunctive present of the verb \emph{cantar} ("to sing"); both fine-grained
tags, \texttt{\emph{<vblex><prs><p1><sg>}} and
\texttt{\emph{<vblex><prs><p3><sg>}}, are grouped under the tagger
category \ \texttt{VLEXSUBJ} (\emph{subjunctive verb}). In this case,
one of both fine tags is discarded; in the tagger definition file it
is possible to define which fine-grained tag, among the ones that compose a
coarse tag, will be delivered after disambiguation.




\subsection{Data for the part-of-speech tagger}
\label{datostagger}
\subsubsection{Introduction}
\label{ss:introtagger} We describe next the format of the files that
specify how to group the \emph{fine-grained} tags delivered by the
morphological analyser into more general \emph{coarse} tags.  In this
files, moreover, one can specify \emph{restrictions} that help in the
estimation of the statistical model underlying the process of lexical
disambiguation, as well as preference rules to be applied when two
fine-grained tags belong to the same category.


The tagger assumes that, in the input stream, lexical forms will be
appropriately delimited, as described in the format specification for
the data stream between modules (Section \ref{se:flujodatos}). In
brief, the format of the data delivered by the morphological analyser
is the following:
\begin{equation}
\label{eq:formaanalizada}
  \begin{array}{rcl} 
  \mbox{analysedform}&\to& \mbox{lexicalmultiform}\; 
  [\; \mbox{lexicalmultiform} \; ]^* 
\\
  \mbox{lexicalmultiform}&\to& \mbox{lexicalform}\; [\;\mbox{lexicalform}\; ]^*\;\mbox{lemma-queue?} \\
  \mbox{lexicalform}&\to&\mbox{lemma}\;\mbox{finetag}\\
  \mbox{lemma-queue}&\to&\mbox{lemma}\\
  \mbox{finetag}&\to&\mbox{morphsymbol}\;[\;\mbox{morphsymbol}\;]^* \\
  \end{array}
\end{equation}
\label{formaanalizada}

where:


\begin{itemize}
\item \emph{analysedform} is all the information delivered for each
surface form in the output of the morphological analyser
\item \emph{lexicalmultiform} is a sequence of one or more lexical
forms followed, optionally, by an invariable queue as happens in some
multiwords (like the Spanish expression \emph{cántale las cuarenta}).
\item \emph{lexicalforms}\footnote{Separated from each other by a
delimiter which corresponds to the \texttt{<j/>} element (see page
\pageref{ss:j}).} are units made of one lemma and one or more
grammatical symbols (which compose the fine-grained tag) with the output
information of the analyser
\item \emph{lemma-queue} is made of one or more lemmas
  \footnote{Separated from each other by the \texttt{<b/>} element
  (see page~\pageref{s3:b}).} that are the invariable part of a
  multiword. The queue of a multiword is made of the lemma or lemmas
  with no inflection that follow the lemmas with inflection. For
  example, the Spanish multiword \emph{cantar las cuarenta} ("to
  lecture", "to reproach") can take the forms \emph{cántale las
  cuarenta}, \emph{(le) cantaré las cuarenta}, \emph{cantándole las
  cuarenta}, etc. In this case, the queue would be \emph{las cuarenta}
  (see page~\pageref{ss:multipalabras} for more information).

\item \emph{finetag} is made of one or more grammatical symbols
(\emph{símbologram}).
\end{itemize}

For example, the entry for the Spanish ambiguous surface form
\emph{correos} would have two lexical multiforms; the first lexical
multiform would have one single lexical form, with lemma \emph{correo}
("post office") and a fine tag made of the grammatical symbols
\emph{common noun}, \emph{masculine}, \emph{plural}; the second
lexical multiform would be a sequence of two lexical forms, one with
lemma \emph{correr} ("to move") and a fine tag made of the grammatical
symbols \emph{lexical verb}, \emph{imperative}, \emph{second person},
\emph{plural}, and the other one with lemma \emph{vosotros} ("you")
and fine tag made of the grammatical symbols \emph{pronoun},
\emph{enclitic}, \emph{second person}, \emph{masculine-feminine},
\emph{plural}.

\notavisible{An  explanation of how a word containing more than one
  lexical form is treated when no multilexical form is defined for it
  should be added}

\subsubsection{Format specification}
\label{formatotagger} The format of the file (encoded in XML) is
specified by the DTD that can be found in
Appendix~\ref{ss:DTD_desambiguador}.


The meaning of the different tags is the following:
\begin{description}
\item[\texttt{tagger}]: is the root element; its mandatory attribute
\texttt{name} is used to specify the name of the tagger generated from
the file.
\item[\texttt{tagset}]: defines the \emph{coarse} tagset or categories
with which the tagger works. Categories are defined by the fine-grained tags
output by the morphological analyser.
\item[\texttt{def-label}]: defines a category or coarse tag (whose
  name is specified in the mandatory attribute \texttt{name}) by means
  of a list of fine tags defined with one or more \texttt{tags-item}
  elements; an optional attribute \texttt{closed} indicates whether
  this is a closed category; if this is the case, it is assumed that
  an unknown word can never belong to this category.\footnote{Closed
  categories are those that do not grow when new words are created:
  prepositions, determiners, conjunctions, etc.}
  
  The more specific categories \emph{must} be defined before the more
  general ones.  When the definition of a general category implicitly
  includes that of a specific category defined before, it is
  understood that it refers to all cases \emph{except} the ones
  defined by the more specific category.
 
\item[\texttt{tags-item}]: is used to define a fine-grained tag by means of a
sequence of grammatical symbols. The sequence of grammatical symbols
that make up the fine tag is specified in the mandatory attribute
\texttt{tags}. In this sequence, symbols are separated by a dot, and
the asterisk ``\texttt{*}'' is used to express that any sequence of
symbols may appear in its place. It is also possible to define
lexicalized categories, specifying the lemma of the word in the
attribute \texttt{lemma}.
  
\item[\texttt{def-mult}]: defines special categories
(\emph{multicategories}) made of more than one category, in order to
deal with entries with more than one lexical form, like in the example
given in the previous section. Each category is defined as a set of
valid sequences (\texttt{sequence}) of previously defined categories
or of fine-grained tags.  It is designed for contractions, verbs with enclitic
pronouns, etc.

\item[\texttt{sequence}]: defines a sequence of elements, which can be
categories (\texttt{label-item}) or fine-grained tags
(\texttt{tags-item}). Using fine-grained tags directly is useful if one wishes
to use a sequence of grammatical symbols that is not part of any
previously defined fine tag \nota{MG: en comptes de 'fine tag' no es
refereix aquí a 'category'?} or that represents a greater
specialization of a defined fine tag \nota{ídem: category}.
  
\item[\texttt{label-item}]: is used to refer to a category or coarse
tag previously defined, to be specified in the mandatory attribute
\texttt{label}.
  
\item[\texttt{forbid}]: this (optional) section is aimed to define
restrictions as sequences of categories \texttt{label-sequence} that
can not occur in the language involved. In the current version, due to
the fact that the tagger is based on first-order hidden Markov models,
sequences can only be made of \emph{two} \texttt{label-items}.
  
\item[\texttt{label-sequence}]: defines a sequence of categories
(\texttt{label-item}).

\item[\texttt{enforce-rules}]: this (optional) section allows defining
restrictions in the form of obligations.
  
\item[\texttt{enforce-after}]: defines a restriction that forces that
a certain category can only be followed by the categories belonging to
the set of categories defined in \texttt{label-set}. Note that this
kind of restrictions is equivalent to defining several forbidden
(\texttt{forbid}) sequences (\texttt{label-sequence}) with the
category defined in the mandatory attribute \texttt{label} and the
rest of categories that do not belong to the set defined in
\texttt{label-set}. For this reason, this kind of restriction must be
used very cautiously.
  
\item[\texttt{label-set}]: defines a set of categories
(\texttt{label-items}).

\item[\texttt{preferences}]: used to define priorities in terms of
which fine-grained tag must be delivered in the tagger output when two or more
fine tags are assigned to the same category.
  
\item[\texttt{prefer}]: specifies that, in case of conflict between
different fine-grained tags assigned to the same category, the tagger must
output the tag specified in the mandatory attribute \texttt{tags}. If
a category contains more than one of the fine tags included in these
\texttt{prefer} elements, the tag defined in the first place will be
the selected one.
\end{description}

Figures~\ref{fg:exemple_desambiguador1}
and~\ref{fg:exemple_desambiguador2} contain an example with the most
significant parts of a tagger specification file defined by the DTD
just described.

% DTD moguda a Apèndix


\begin{figure}[htbp]
  \begin{small}
    \begin{alltt} 
<?\textsl{xml} \textsl{version}="1.0" \textsl{encoding}="iso-8859-1"?>
<!\textsl{DOCTYPE} \textbf{tagger} SYSTEM "tagger.dtd">
<\textbf{tagger} \emph{name}="es-ca">
<\textbf{tagset}>
   <\textbf{def-label} \textsl{name}="adv">
      <\textbf{tags-item} \textsl{tags}="adv"/>
   </\textbf{def-label}>
   <\textbf{def-label} \textsl{name}="detnt" \textsl{closed}="true">
      <\textbf{tags-item} \textsl{tags}="detnt"/>
   </\textbf{def-label}>
   <\textbf{def-label} \textsl{name}="detm" \textsl{closed}="true">
      <\textbf{tags-item} \textsl{tags}="det.*.m"/>
   </\textbf{def-label}>
   <\textbf{def-label} \textsl{name}="vlexpfci">
      <\textbf{tags-item} \textsl{tags}="vblex.pri"/>
      <\textbf{tags-item} \textsl{tags}="vblex.fti"/>
      <\textbf{tags-item} \textsl{tags}="vblex.cni"/>
   </\textbf{def-label}>   
   <\textbf{def-mult} \textsl{name}="infserprnenc" \textsl{closed}="true">
      <\textbf{sequence}>
         <\textbf{label-item} \textsl{label}="vserinf"/>
         <\textbf{label-item} \textsl{label}="prnenc"/>
      </\textbf{sequence}>
      <\textbf{sequence}>
         <\textbf{label-item} \textsl{label}="vserinf"/>
         <\textbf{label-item} \textsl{label}="prnenc"/>
         <\textbf{label-item} \textsl{label}="prnenc"/>
      </\textbf{sequence}>
   </\textbf{def-mult}>   
   <\textbf{def-mult} \textsl{name}="prepdet" \textsl{closed}="true">
      <\textbf{sequence}>
         <\textbf{label-item} \textsl{label}="prep"/>
         <\textbf{tags-item} \textsl{tags}="det.def.m.sg"/>
      </\textbf{sequence}>
   </\textbf{def-mult}>
</\textbf{tagset}>
<!-- ... -->
    \end{alltt}
  \end{small}
  \caption{Example of a tagger definition file (continues in
  Figure~\ref{fg:exemple_desambiguador2}).}
  \label{fg:exemple_desambiguador1}
\end{figure}


\begin{figure}[htbp]
  \begin{small}
    \begin{alltt} 
<!-- ... -->
<\textbf{forbid}>
   <\textbf{label-sequence}>
      <\textbf{label-item} \textsl{label}=="prep"/>
      <\textbf{label-item} \textsl{label}=="vlexpfci"/>
   </\textbf{label-sequence}>
   <!-- ... -->
</\textbf{forbid}>
<\textbf{enforce-rules}>
   <\textbf{enforce-after} \textsl{label}=="prnpro">
      <\textbf{label-set}>
         <\textbf{label-item} \textsl{label}=="prnpro"/>
         <\textbf{label-item} \textsl{label}=="vlexpfci"/>
         <!-- ... -->
      </\textbf{label-set}>
   </\textbf{enforce-after}>
   <!-- ... -->
</\textbf{enforce-rules}>
<\textbf{preferences}>
   <\textbf{prefer} \textsl{tags}="vblex.pii.p3.sg"/>
   <\textbf{prefer} \textsl{tags}="vbser.pii.p3.sg"/>
   <!-- ... -->
</\textbf{preferences}>
</\textbf{tagger}>
    \end{alltt}
  \end{small}
  \caption{Example of a tagger definition file (comes from
  Figure~\ref{fg:exemple_desambiguador1}).}
  \label{fg:exemple_desambiguador2}
\end{figure}

\subsection{Some questions about the training of the part-of-speech
tagger} The training of the part-of-speech tagger can be made both in
a supervised manner, using manually disambiguated texts, and a
unsupervised manner, using ambiguous texts.

When the training is made with ambiguous texts (unsupervised), the
format of the required text can be automatically obtained from a plain
text corpus in the chosen language using the system's morphological
analyser; in this case, the format of the text forms will be like the
one defined in the figure~\ref{eq:formaanalizada2} (its description
can be found in page~\pageref{formaanalizada}). As the chart shows,
each analysed surface form can have more than one analysis (an
\emph{analysedform} can give as a result more than one
\emph{lexicalmultiform}).


\begin{equation}
\label{eq:formaanalizada2}
  \begin{array}{rcl} \mbox{analysedform}&\to&
\mbox{lexicalmultiform}\; [\; \mbox{lexicalmultiform} \; ]^* \\
\mbox{lexicalmultiform}&\to& \mbox{lexicalform}\;
[\;\mbox{lexicalform}\; ]^*\;\mbox{lemma-queue?} \\
\mbox{lexicalform}&\to&\mbox{lemma}\;\mbox{finetag}\\
\mbox{lemma-queue}&\to&\mbox{lemma}\\
\mbox{finetag}&\to&\mbox{morphsymbol}\;[\;\mbox{morphsymbol}\;]^* \\
  \end{array}
\end{equation}
\label{formaanalizada2}

For the supervised training we need manually disambiguated text. The
format of the text forms in this case will be like the format
delivered by the morphological analyser (see
Section~\ref{se:flujodatos}) except that, being the text already
disambiguated, a surface form can never produce more than one lexical
form, as shown in Figure~\ref{eq:formadesambiguada} (a
\emph{disambiguatedform} will consist always of a single
\emph{lexicalmultiform}).
\begin{equation}
\label{eq:formadesambiguada}
  \begin{array}{rcl}
  \mbox{disambiguatedform}&\to&\mbox{lexicalmultiform}\\
  \mbox{lexicalmultiform}&\to&\mbox{lexicalform}\;[\;\mbox{lexicalform}\;]^*\;\mbox{lemma-queue?}\\
  \mbox{lexicalform}&\to&\mbox{lemma}\;\mbox{finetag}\\
  \mbox{lemma-queue}&\to&\mbox{lemma}\\
  \mbox{finetag}&\to&\mbox{morphsymbol}\;[\;\mbox{morphsymbol}\;]^* \\
  \end{array}
\end{equation}
 

Finally, we need also the dictionary of the involved language to train
the tagger. This dictionary is used to determine, in combination with
the tagset specification, the different ambiguity classes with which
the tagger will work.

Figure \ref{fig:dependencias} shows the dependency diagram for the
training and the use of the tagger.

\nota{Aquest esquema canviarà amb el nou tagger - Sergio}

\begin{figure}
\begin{center}
\includegraphics[width=15cm]{diagram}
\end{center}
\caption{Dependency diagram for the part-of-speech tagger.}
\label{fig:dependencias}
\end{figure}


\newpage

\section[Transfer pre-processing]{Auxiliary module: transfer
pre-processing module}
\label{se:pretransfer}
\subsection{Justification} The transfer pre-processing module
\texttt{pretransfer} is in charge of separating compound multiwords
(see page~\pageref{ss:multipalabras}) and shifting certain parts of
multiwords with inner inflection or \emph{split lemma} forms.  This
module processes the tagger output and generates an entry suitable for
the transfer module.  The processing performed by this module is
necessary for different reasons:

\begin{itemize}
\item So that the transfer module can process these units separately
in order to deal with, for example, the movement of clitic pronouns
when changing from enclitic to proclitic and vice versa.
\item So that the bilingual dictionary only has to store information
about the lemmas to be translated.  If the particles that make up a
multiword are included jointly in the bilingual dictionary, the
dictionary would have to store an entry for each of the different
combinations.  By separating compound multiwords and processing multiwords with
inner inflection, we can avoid having
entries including inflection variations in the bilingual dictionary.
\end{itemize}

\subsection{Behaviour and example}

The program replaces each \texttt{<j/>} in the dictionary, that is,
each \texttt{+} in the data stream, by a symbol for word end, a blank
and a symbol for word beginning.  Moreover, if the form is a multiword
with split lemma, the queue is moved to the position between the first
word of the multiword and its first grammatical symbol.

The task of generating an output which has the original order accepted
by the generator, is left to the rules of the transfer
module, which are also responsible for creating the compound
multiwords which may be required in the target language.  In general,
the generator works with the same multiwords as the morphological
analyser, and with the elements in the same order; that is the reason
why this task has to be done in the transfer module.

We show below the result of applying this process to the compound
multiword \textit{darlo} ("give it" in Spanish):

\begin{small}
\begin{alltt}
\$ pretransfer
^dar<vblex><inf>+lo<prn><enc><p3><m><sg>\$     \(\longleftarrow\) \textrm{input} 
^dar<vblex><inf>\$ ^lo<prn><enc><p3><m><sg>\$     \(\longleftarrow\) \textrm{output}
\end{alltt}
\end{small}

As can be seen, it consists only in dividing the lexical forms of a
compound multiword into individual lexical forms.

When the input is a multiword with split lemma, the process is as
shown in the following example for the Spanish multiword
\textit{echarte de menos} ("to miss you"):

\begin{small}
\begin{alltt} 
\$ pretransfer
^echar<vblex><inf>+te<prn><enc><p2><m><sg># de menos\$ 
^echar# de menos<vblex><inf>\$ ^te<prn><enc><p2><m><sg>\$
\end{alltt}
\end{small}

Here, besides dividing into lexical forms, the module moves the
invariable lemma queue into the mentioned position.  As you can see,
semantic units are maintained after the movement of the invariable
queue, since we can consider \textit{echar de menos} a verbal unit
with own meaning.




\section{Lexical selection module}
\label{se:seleccio_lex}


\subsection{Introduction}


When the Apertium system is used to translate between less related
languages than the ones dealt with in the first stages of the engine,
the question of lexical selection becomes significant, because there
are more cases, and more critical, in which a source language word can
have more than one different translation in the target language. For
this reason we created a new module, the lexical selection module,
which deals with this problem.

Before going into its characteristics, we will see how the problems of
\emph{multiple equivalence} (the fact of existing more than one
possible translation in target language for a source language lexical
form) are tackled in Apertium in two ways.

On the one hand, we have the situation where there is no big
difference in meaning between the multiple equivalents in the target
language, and the fact of choosing one or the other can not lead to
any translation error. We could say that between these equivalents
there is a synonymy or quasi-synonymy relation. In such a case, the
linguist chooses one of the lemmas as a translation (generally the
most frequent or usual), and adds a direction restriction to the other
lemmas (with the attributes \texttt{LR} or \texttt{RL}) so that they
are translated in the opposite direction but not in the direction
where there are multiple equivalents.


On the other hand, we have the case where there is a clear difference
in meaning between the multiple equivalents, which can lead to
translation errors if the inappropriate lemma is chosen. These are the
cases dealt with the new lexical selection module. The linguist has to
encode entries with the attributes \texttt{slr} or \texttt{srl}
described in the next section, thus identifying the different
translation options; then, the lexical selection module, by means of
statistical methods, chooses the translation which is most suitable in
a given context.



Sometimes it is not easy to decide whether a multiple equivalence
situation should be solved in one way or the other. For example, if
there is difference in the meaning of two or more lemmas in the target
language, but we think that the lexical selection module will not be
capable of choosing the right translation by means of the context, we
will follow the first method: choose a fixed translation (the most
general, the most suitable in the maximum number of situations) and
add a direction restriction to the rest of translations.  In the other
cases, we will encode the entries so that the decision is left to the
lexical selection module.


When we use an Apertium system without lexical selection module, the
only way to add entries with different possible translations is the
first one, that is, choosing an only translation and marking the other
equivalences with a direction restriction.  In the event that we use
bilingual dictionaries with multiple translations, encoded with the
attributes \texttt{slr} or \texttt{srl}, in a system that does not
have any lexical selection module, a style sheet will
convert these entries designed for a lexical selection module into
entries with direction restrictions \texttt{LR} or \texttt{RL}, so
that one of the multiple equivalents (the one chosen as default entry
by the linguist) becomes the fixed translation of the source language
lemma.



As examples of bilingual equivalencies that should have a direction
restriction, we can give the translation pairs \texttt{ca-es}
\emph{encara -- aún/todavía} ("still") and \emph{sobtat --
súbito/repentino} ("sudden"), the first one of which could be encoded
like this:
\begin{alltt}
\begin{small}

<e r="LR">
   <p>
      <l>aún<s n="adv"/></l>
      <r>encara<s n="adv"/></r>
   </p>
</e>
<e>
    <p>
      <l>todavía<s n="adv"/></l>
      <r>encara<s n="adv"/></r>
    </p>
</e>
\end{small}
\end{alltt}

As examples of the second case (multiple equivalents with big
difference in meaning) we have the pairs \texttt{es-ca} \emph{hoja --
full/fulla} ("sheet/leaf") and \emph{muñeca -- nina/canell}
("doll/wrist"), as well as the \texttt{en-ca} examples shown in page
\pageref{entrades_lextor}, where it is described how to specify these
multiple equivalents in the bilingual dictionary.




\begin{figure} {\footnotesize \setlength{\tabcolsep}{0.5mm}
\begin{center}
\begin{tabular}{ccccccccc} \\
\parbox{0.95cm}{source language text} \\ $\downarrow$ \\
\framebox{\parbox{1.0cm}{de-for\-matter}} $\rightarrow$ &
\framebox{\parbox{0.6cm}{morph. anal.}}  $\rightarrow$ &
\framebox{\parbox{1.0cm}{POS tagger}} $\rightarrow$ &
\framebox{\parbox{0.6cm}{lex. select.}} $\rightarrow$ &
\framebox{\parbox{0.85cm}{struct. transf.}} $\rightarrow$ &
\framebox{\parbox{0.6cm}{morph. gen.}}  $\rightarrow$ &
\framebox{\parbox{1.2cm}{post\-generator}} $\rightarrow$ &
\framebox{\parbox{1.0cm}{re-for\-matter}} \\ & & & & $\updownarrow$ &
& & $\downarrow$ \\ & & & & \framebox{\parbox{0.8cm}{lex. transf.}} &
& &
\parbox{0.95cm}{target language text} \\
\end{tabular}
\end{center} }
\caption{The nine modules that build the assembly line in the version
  2 of the machine translation system Apertium.}
\label{fig:moduls}
\end{figure}

Figure~\ref{fig:moduls} shows the new assembly line of the version 2
of Apertium.\footnote{This figure substitutes the figure
\ref{fg:modules} in page \pageref{pg:modules} which represents the
version 1 of Apertium.} \nota{MG: caldria canviar la figura de la
pàgina 6 per aquesta d'aquí?} The module in charge of the lexical
selection (lexical selector) runs after the part-of-speech tagger and
before the structural transfer module; therefore, this new module
works only with source language information.


Section~\ref{se:preprocessament} next describes the pre-processing
that must be done on a bilingual dictionary  containing more than
one translation per entry (whether the system uses a
lexical selector or not), and Section~\ref{se:lextor} describes
how the lexical selector works and how it has to be trained.


 
\subsection{Pre-processing of the bilingual dictionaries
}\label{se:preprocessament}

Bilingual dictionaries have been modified to allow the specification
of more than one translation per entry (refer to Section
\ref{dic_lextor} to learn how to write such dictionary entries); this
fact makes it necessary to pre-process these dictionaries, since the
Apertium engine works with compiled dictionaries in which there is
only one possible translation for each word.

The pre-processing of dictionaries is done automatically during
compilation, therefore the final user does not need to perform any
specific action.


\subsubsection{Pre-processing without lexical selection module}

When bilingual dictionaries with multiple equivalents are used in a
system where there is no lexical selection module, the pre-processing
is done by the application of the style sheet
\texttt{translate-to\--de\-fault\--e\-qui\-va\-lent.xsl}.  This style
sheet turns dictionaries with multiple translations per entry into
dictionaries with only one translation per entry; to do this, it
chooses as translation the entry marked as default, and adds a
direction restriction (\texttt{LR} or \texttt{RL} as applicable) to
the other entries, so that they are only translated in the translation
direction where there is no equivalent multiplicity.  The style sheet
is called from the \texttt{Makefile}.


To put an example, the result of applying the style sheet on the first
three entries shown in page \pageref{entrades_lextor} is the
following:

\begin{alltt}
\begin{small}
<e>
   <p>
      <l>flat<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
   </p>
</e>

<e r="LR">
   <p>
      <l>floor<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
   </p>
</e>

<e r="RL">
   <p>
      <l>floor<s n="n"/></l>
      <r>terra<s n="n"/><s n="m"/></r>
   </p>
</e>
\end{small}
\end{alltt}

\subsubsection{Preprocessing with lexical selection module}

If the Apertium system works with a lexical selection module, the
bilingual dictionary must be pre-processed in order to obtain:
\begin{itemize}
\item a monolingual dictionary that, for each source language word
(for example \emph{look}) delivers all the possible translation marks
or equivalents (\texttt{look\_\_mirar D} and
\texttt{look\_\_semblar}); this dictionary will be used by the lexical
selection module; and

\item a new bilingual dictionary that, given a word with the lexical
selection already done (for example \texttt{look\_\_semblar}) delivers
the translation (\emph{semblar}); this will be the bilingual
dictionary to be used in the lexical transfer.

\end{itemize}


This pre-processing is automatically done by means of the following
software during dictionary compilation:
\begin{itemize}
\item \texttt{apertium-gen-lextormono}, that receives three
parameters:
  \begin{itemize}
  \item the translation direction for which you want to generate the
  monolingual dictionary used in the lexical selection; \texttt{lr}
  for the translation left to right, and \texttt{rl} for the
  translation right to left;
  \item the monolingual dictionary to be pre-processed; and
  \item the file where the output monolingual dictionary has to be
  written.
  \end{itemize}

\item \texttt{apertium-gen-lextorbil}, that receives three parameters:
  \begin{itemize}
  \item the translation direction (\texttt{lr} or \texttt{rl}) for
    which you want to generate the bilingual dictionary to be used by
    the lexical transfer module;
  \item the bilingual dictionary to be pre-processed; and
  \item the file where the output bilingual dictionary has to be
  written.
  \end{itemize}
\end{itemize}

\subsection{Execution of the lexical selection
module}\label{se:lextor}

The module responsible for the lexical selection runs after the
part-of-speech tagger and before the structural transfer (see
Figure~\ref{fig:moduls} in page~\pageref{fig:moduls}); therefore, it
uses only information from the source language. However, during the
training of the module, target language information is also used.


\subsubsection{Training}\label{se:entrenament}

To train the lexical selection module, a corpus in the source language
and another one in the target language are required; they do not need
to be related. Both corpora must be pre-processed before the
training. This pre-processing, consisting in analysing the corpora and
performing the POS disambiguation, can be done  with
\texttt{apertium-prepro\-cess\--cor\-pus\--lex\-tor}.

The training of the module that performs the lexical selection
consists of the following tasks:\footnote{The training of the models
used for the lexical selection has been automated in all the packages
using it. Furthermore, all the software mentioned has its UNIX manual
page}



\begin{enumerate}
\item Obtain the list of words that will be ignored when performing
lexical selection (\emph{stopwords}). This list can be done manually
or using \texttt{apertium-gen-stopwords-lextor};
\item Obtain the list of (source language) words that have more than
one translation in the target language, using
\texttt{apertium-gen-wlist-lextor};
\item Translate to the target language all the words obtained in the
previous step, using \texttt{apertium-gen-wlist-lextor-translation};
\item Running \texttt{apertium-lextor --trainwrd} and using the target
language pre-processed corpus, train a word co-occurrence model for
the words obtained in the previous step;
\item Running \texttt{apertium-lextor --trainlch} and using the source
language pre-processed corpus, the dictionaries generated by the
programs mentioned in Section~\ref{se:preprocessament} and the word
co-occurrence models calculated in the previous step, train a
co-occurrence model for each of the translation marks of those words
that can have more than one translation in the target language.
\end{enumerate}

\subsubsection{Use}\label{se:us} 

The word co-occurrence models
calculated for each translation mark as described in the previous
section provide the information required to perform lexical selection
with information from the context.

Lexical selection is done by \texttt{apertium-lextor --lextor}; the
formats used to communicate with the rest of the modules of the
translation engine are:

\begin{description}
\item [Input:] text in the same format as the input for the structural
transfer module, that is, text analysed and disambiguated, with
invariable queues of multiwords moved before morphological tags.
\item [Output:] text in the same format, but with the translation mark
to be used when executing lexical transfer.
\end{description}


The following example illustrates the input/output formats used by the
lexical selector (we have assumed in the example that only the English
verb \emph{get} has more than one translation equivalent in the
dictionaries):
\begin{itemize}
\item Source language text (English): \emph{To get to the city centre}
\item Lexical selector input: \verb!^To<pr>$!
\verb!^get<vblex><inf>$! \verb!^to<pr>$! \verb!^the<det><def><sp>$!
\verb!^city<n><sg>$! \verb!^centre<n><sg>$!
\item Translation marks in the en-ca bilingual dictionary for the verb
\emph{get}: \texttt{rebre}, \texttt{agafar}, \texttt{arribar},
\texttt{aconseguir D}
\item Lexical selector output: \verb!^To<pr>$!
\verb!^get__arribar<vblex><inf>$! \verb!^to<pr>$!
\verb!^the<det><def><sp>$!  \verb!^city<n><sg>$!
\verb!^centre<n><sg>$!
\end{itemize}


\newpage
\section{Structural transfer module}
\label{ss:transfer}


\nota{Faena per fer (mlf):
  \begin{itemize} 
  \item Hi ha bastants vacil·lacions en la terminologia usada per a
  referir-se a conceptes i en els noms usats per als programes.
\item He intentat substituir en cada cas l'expressió \emph{per
defecte} per una altra més adequada; però caldrà distingir en quin cas
ens trobem en cada cas.
  \end{itemize}}

\subsection{Introduction}

In 2007, Apertium incorporated a more advanced structural transfer system than 
the one used until then; it became necessary when we started developing
 machine translators for less related language pairs in
comparison with the ones dealt with before, such as 
the \emph{English}--\emph{Catalan} translator.

This enhanced transfer system is made of three modules, the first one
of which can be used in isolation in order to run a
\textbf{shallow-transfer} system (which is the transfer system used so
far for related language pairs such as \emph{Spanish}--\emph{Catalan} or
\emph{Spanish}--\emph{Galician}).  When the system is used for less
related language pairs and, therefore, an 
\textbf{advanced transfer} becomes necessary, the three transfer modules will be executed.

The two transfer systems differ in the number of passes over the input
text.  The shallow-transfer system makes structural transformations
with a single pass of the rules, which detect sequences or
\emph{patterns} of lexical forms and perform on them the required
verifications and changes. On the other hand, the advanced transfer
system works with a new architecture that allows to detect
\emph{patterns of patterns} of lexical forms with three passes, done
by its three modules.

We describe next the characteristics of the structural transfer system.  Section
\ref{functransfer} describes the shallow-transfer system and Section
\ref{apertium2}, the advanced transfer system.  The description of the
shallow-transfer system is also applicable to the first module of the
advanced transfer system, with the differences mentioned in that
section.  Section \ref{formatotransfer} describes the format used to
create rules in both systems. In Section \ref{noutransfer} there is a
detailed description of how the three modules of the advanced transfer
system work, and finally, Section \ref{ss:preproceso_transfer}
describes the pre-processing required by the modules.


\subsection{Shallow-transfer}
\label{functransfer}


In this system, only the first of the three modules that compose the
advanced transfer system is used. This module is called
\emph{chunker}.

The design of the language and the compiler used to generate the
structural transfer module is largely based upon the MorphTrans
language described in \cite{garridoalenda01p} and used by the MT
systems \textsf{interNOSTRUM}
\cite{canals01b,garridoalenda01p,garrido99j} (Spanish--Catalan) and
\textsf{Traductor Universia} \cite{garrido03p, gilabert03j}
(Spanish--Portuguese), developed by the Transducens group at the
Universitat d'Alacant.


The transfer process is organized around patterns representing
fixed-length sequences of source language lexical forms (SLLFs) (see
page~\pageref{pg:FSFL} for a description of lexical form (LF)); a
sequence follows a certain pattern if it contains the sequence of lexical forms
of the pattern.  Patterns do not need to be constituents or
phrases in the syntactic sense: they are mere concatenations of
lexical forms that may need a conjoint processing additional to the
simple word-for-word translation, due to the grammatical divergences
between SL and TL (gender and number changes, reorderings,
prepositional changes, etc). The catalogue of patterns defined for a
certain language is selected with a view to covering the most common structural
transformations.  When source language and target language
are syntactically similar, as is the case between Spanish, Catalan and
Galician, simple rules based on sequences of lexical categories
achieve a reasonable translation quality.

The transfer module detects, in the SL, sequences of lexical forms
that match one of the patterns previously defined in the pattern
catalogue, and processes them applying the corresponding structural
transfer rule, doing at the same time the lexical transfer by reading
the bilingual dictionary.

The \emph{pattern detection} phase occurs as follows: if the transfer
module starts to process the $i$-th SLLF of the text, $l_i$, it tries
to match the sequence of SLLFs $l_i, l_{i+1}, \ldots$ with all of the
patterns in its pattern catalogue: the longest matching pattern is
chosen, the matching sequence is processed (see below), and processing
continues at SLLF $l_{i+k}$, where $k$ is the length of the pattern
just processed. If no pattern matches the sequence starting at SLLF
$l_i$, it is translated as an isolated word an processing restarts at
SLLF $l_{i+1}$ (when no patterns are applicable, the system resorts to
word-for-word translation). Note that each SLLF is processed only
once: patterns do not overlap; hence, processing occurs left to right
and in distinct "chunks".


In the \emph{pattern processing } phase, the system takes the detected
sequence of SLLFs and builds (using a program to consult the bilingual
dictionary) a sequence of TL lexical forms (TLLFs) obtained after the
application of the operations described in the rule associated to the
detected pattern (reordering, addition, replacement or deleting of
words, inflection changes, etc.). The information that does not change
is automatically copied from SL to TL. The resulting data, that is,
the lemmas with their associated morphological tags, are sent to the
generator, which creates the inflected forms.



For instance, the Spanish sequence \emph{una señal inequívoca} ("an
unmistakable signal"), that would go from the tagger to the transfer
module in the following format~\footnote{The example has been
presented in a way that it does not contain superblanks with format
information, so that the linguistic side of the transformation is
clearer. See Chapter \ref{se:flujodatos}.}:\\

\begin{alltt}
\begin{small} 
\textasciicircum\textbf{uno}<det><ind><f><sg>\$
\textasciicircum\textbf{señal}<n><f><sg>\$
\textasciicircum\textbf{inequívoco}<adj><f><sg>\$
\end{small}
\end{alltt}


\noindent{would be detected as a pattern by a rule for
determiner--noun--adjective.} The transfer module would consult the
bilingual dictionary to get the Catalan equivalents and, as it would
detect a gender change in the word \emph{señal} (its Catalan
translation \emph{senyal} is masculine), it would propagate this
change to the determiner and the adjective to deliver the output
sequence:\\

\begin{alltt}
\begin{small} 
\textasciicircum\textbf{un}<det><ind><m><sg>\$
\textasciicircum\textbf{senyal}<n><m><sg>\$
\textasciicircum\textbf{inequívoc}<adj><m><sg>\$
\end{small}
\end{alltt}

\noindent{which the generation module would turn into the Catalan
inflected sequence: \emph{un senyal inequívoc}.}

The task of most rules is to ensure gender and number agreement in
simple noun phrases (determi\-ner--noun, determiner--noun--adjective,
determiner--adjective--noun, determiner--adjective, etc.), provided
that there is agreement between the SLLFs of the detected
pattern. These rules are required either because the noun changes its
gender or number between SL and TL (as in the previous example) or
because gender or number in the TL have to be determined due to the
fact that it was ambiguous in SL for some of the words (for example,
the Catalan determiner \emph{cap} can be translated into Spanish as
\emph{ningún} (masc.) or \emph{ninguna} (fem.) depending on the
accompanying noun: \emph{cap cotxe} (\texttt{ca}) $\rightarrow$
\emph{ningún coche} (\texttt{es}) and \emph{cap casa} (\texttt{ca})
$\rightarrow$ \emph{ninguna casa} (\texttt{es})). Furthermore, there
other rules defined to solve frequent transfer problems between
Spanish, Catalan and Galician, such as, among others:

\begin{itemize}
  

\item rules to change prepositions in certain constructions: \emph{in
Barcelona} (\texttt{es}) $\rightarrow$ \emph{a Barcelona}
(\texttt{ca}); \emph{consiste en hacer} (\texttt{es}) $\rightarrow$
\emph{consisteix a fer} (\texttt{ca});

\item rules to add/remove the preposition \emph{a} in certain Galician
modal constructions with the verbs \emph{ir} and \emph{vir}: \emph{vai
comprar} (\texttt{gl}) $\rightarrow$ \emph{va a comprar}
(\texttt{es});

\item rules for articles before proper nouns: \emph{ve la Marta}
  (\texttt{ca}) $\rightarrow$ \emph{viene Marta} (\texttt{es});

\item lexical rules, for instance, to decide the correct translation
of the adverb \emph{molt} (\texttt{ca}) into Spanish (\emph{muy,
mucho}) or of the adjective \emph{primeiro} (\texttt{gl}) or
\emph{primer} (\texttt{ca}) into Spanish (\emph{primer, primero});

\item rules to displace atonic or clitic pronouns, whose position in
Galician is different to that in Spanish (proclitic in Galician and
enclitic in Spanish or vice versa): \emph{envioume} (\texttt{gl})
$\rightarrow$ \emph{me envió} (\texttt{es}); \emph{para nos dicir}
(\texttt{gl}) $\rightarrow$ \emph{para decirnos} (\texttt{es}).

\end{itemize}



\emph{Multiwords} (its different types are described in
page~\pageref{ss:multipalabras}) are processed in a special way in
this module:

\begin{itemize}
\item \emph{Multiwords without inflection}, made of only one lexical
form, do not need any special processing, since they are treated like
other LFs.
\item In the case of \emph{compound multiwords}, that is, multiwords
formed by more than one \emph{lexical form}, each one with its own
grammatical symbols and joined to each other with the element
\texttt{<j>} in the dictionary entry (which corresponds to the symbol
'+' in the data stream), the auxiliary module \texttt{pretransfer}
(see \ref{se:pretransfer}), located before this module, separates the
different lexical forms so that they reach the transfer module as
independent LFs. If we want to join them again so that they reach the
generator as multiwords (as is the case of enclitic pronouns in our
system), it has to be done by means of a transfer rule, using the
\texttt{<\textbf{mlu}>} element (described later, in section
\ref{ss:mlu}). In page~\pageref{regla_verbo2} you can find an example
of a rule for joining enclitic pronouns to the verb.
\item As for \emph{multiwords with inner inflection}, the
\texttt{pre\-trans\-fer} module moves the lemma queue (the invariable
part) to place it after the lemma head (the inflective form), thus
making possible to find the multiword in the bilingual
dictionary. This kind of multiwords must be processed by a structural
transfer rule which replaces the lemma queue in its proper
position. This is done by using, in the output of the rule, the attributes
\texttt{lemh} ``lemma head'' and \texttt{lemq} ``lemma queue'') of the
\texttt{<\textbf{clip}>} element. See page~\pageref{ss:lu} for a more
detailed description of the use of this element, and page
\pageref{regla_verbo1} to see two rules where these attributes are
used.
\end{itemize}


\subsection{Advanced transfer}
\label{apertium2}

The shallow-transfer architecture described in the previous section is
based, as we have seen, in the automatic handling of word
co-occurrence patterns by means of rules defined by the user. This
model considers two levels from the point of view of the nature of
data: a basic level we call \textit{lexical level}, which handles
words and the tasks of consulting and changing its characteristics
(lemma and tags), besides translating individual lemmas by asking the
bilingual dictionary; and another level we call \textit{word pattern
level}, which is in charge of doing, when applicable, reorderings of
the words that build these patterns, as well as changes in the
properties of words that depend on the specific pattern that has been
detected. All this process of detection and manipulation of words and
patterns is carried out in a single pass.

In contrast, the new advanced transfer architecture is defined as a
transfer system in three levels and three passes. The first two
levels, lexical and pattern level, are the same ones of the
shallow-transfer system. The new added level is a level of
\emph{patterns of patterns} of words. The aim of this new processing
level is to allow the handling and interaction of patterns of words in
a similar way as words are handled in the patterns of the shallow
system. With this new structure we intend to achieve a more
appropriate handling of all transformations that may be required when
translating from one language to another. We want to emphasize that
the definition of word patterns in the shallow-transfer system does
not need to be the same as the definition of word patterns in the
advanced system: we pretend that, in the latter, patterns have a
\textit{spirit} of phrases that does not exist in the previous
system. Therefore we will use the term \textit{chunk} to refer to word
sequences in the advanced transfer system.

The advanced transfer system is organized in three passes. According
to the Apertium processing mode, these three passes are carried out by
three different modules (programs):

\begin{itemize}
\item \texttt{chunker}: identifies chunks, translates word for word,
and carries out required reorderings and morphosyntactic data
propagation inside the chunk (for example, to maintain
agreement). Besides, it creates the chunks that will be processed by
the next module.  The \texttt{chunker} has the option of running as a
single module in a shallow-transfer system.  This is controlled by an
attribute in the \texttt{<transfer>} element.


\item \texttt{interchunk}: this module receives the chunks generated
by the \texttt{chunker} and is able to reorder them, modify the
``syntactic information'' associated to each chunk and, finally,
output the chunks in the new order and with the new properties,
creating new chunks if needed.
\item \texttt{postchunk}: it receives the chunks modified by the
interchunk and carries out final tasks concerning modification of the
words contained in each chunk and printing of the text contained in
chunks in the format accepted by the generator.
\end{itemize}


In the following lines we specify the format of the chunks that
circulate between the modules of the transfer system (Section
\ref{sec:format}) and the letter case handling in chunks (Section
\ref{ss:majuscules}), which is different from case handling of
individual lexical forms in a shallow-transfer system.


The following section, \ref{formatotransfer}, describes the format of
transfer rules, which is the same for the three modules and the two
transfer modes, with little differences.  Finally, after this
description, in \ref{noutransfer} you will find a more detailed
explanation of the three modules that make up an advanced transfer
system.




\subsubsection{Chunk format}
\label{sec:format}


Communication between \texttt{chunker} and \texttt{interchunk}, as
well as between \texttt{interchunk} and \texttt{postchunk}, is
performed through sequences of chunks. We define $C$ as a
\emph{sequence of chunks}, that has the form:
$$
C=b_{0}c_{1}b_{1}c_{2}b_{2} \ldots b_{k-1}c_{k}b_{k}
$$

where each $b_i$ is a \textit{superblank}, and each $c$ is a
\emph{chunk}. A chunk $c$ is defined as a string
\verb!^!$F$\verb!{!$W$\verb!}$! that contains the following
information:

\begin{itemize}
\item $F$ is the \emph{lexical pseudoform}\nota{help: pseudoforma
lèxica = lexical pseudoform or pseudolexical form}; it is a string
that has the form $fE$, where $f$ is the \textit{pseudolemma} of the
chunk, and $E=e_{1}e_{2} \ldots$ is a sequence of grammatical symbols
called \emph{chunk symbols}. Changing these symbols will cause the
changing of the morphological information of words in the chunk, if
this information is linked to these parameters.
\item $W=b_{0}w_{1}b_{1}w_{2}b_{2} \ldots w_{k}b_{k}$ is the sequence
of words $w_i$ sent by the chunker with the intermediate
\textit{superblanks} $b_i$. These words have the same format in both
transfer systems, that is, an individual word
$w_i=$\verb!^!$l_{i}E_{i}$\verb!$!  contains lemma $l_i$ and
  grammatical symbols $E_i$, some of which can be \emph{references or links
  to the symbols} of the chunk and are identified with natural numbers
  \texttt{<1>}, \texttt{<2>}, \texttt{<3>}, etc. These references to
  symbols correspond, in the specified order, to the symbols of $E$.
\end{itemize}

The following is a use example of the described format, with the text
\emph{el gat} ("the cat"):

\begin{small}
\begin{alltt} 
\verb!^!det_nom<SN><m><pl>\verb!{^!el<det><def><2><3>$[
<a href="http://www.ua.es">]^gat<n><2><3>$\verb!}$![</a>]
\end{alltt}
\end{small}

The characters \verb!{! and \verb!}!, if present in the original text,
must be escaped with a backslash \verb!\!.

\subsubsection{Letter case handling}
\label{ss:majuscules}

For each chunk, the case of words is determined by the case of the
pseudolemma of the chunk, taking into account the following rules:

\begin{itemize}

\item When all the letters of the pseudolemma are in lower case: the
case state of words is not modified.
\item When the first letter of the pseudolemma is in upper case and
the rest are in lower case: in the module \texttt{postchunk}, when
words are printed, the letter that is the first of the chunk after all
the possible word reorderings will be put in upper case \nota{MG: and
the rest will be put in lower case except proper nouns? is this
correct?}.
 \item When all the letters of the pseudolemma are in upper case: all
 the words will remain upper case.
\end{itemize}


It is required that the words in the chunk are not capitalized unless
they are proper nouns, so as to avoid the postchunk module having to
look for the word that has to lose capitalization, if this is the
case\nota{MG: I am not sure I understand this}. This task belongs to
the \texttt{chunker} module and is done with a macro or similar
mechanism.


%\settocdepth{subsection}
\subsection{Format specification for structural transfer rules}
\label{formatotransfer}


This section describes the format in which structural transfer rules
are written. In the Appendix, in sections~\ref{ss:dtdtransfer},
\ref{ss:dtdinterchunk} and \ref{ss:dtdpostchunk}, there is the formal
definition (DTD).

Structural transfer rules files have two well-differentiated parts:
one for the declaration of the elements to be used in rules, and
another one for the rules themselves.\\


In the \textbf{declaration} part we find:

\begin{itemize}

\item A series of declarations of \emph{lexical categories}, which
specify those lexical forms that will be treated as a particular
category and will be detected by patterns.  The linguist may include any data about the lexical form
to define a category; categories can be very generic (i.e. all the
nouns) or very specific (i.e. only those determiners that are
demonstrative feminine plural).
\item A series of declarations of the \emph{attributes} we want to
detect in lexical forms (like \emph{gender}, \emph{number},
\emph{person} or \emph{tense}), to perform with them the required
transformation operations and send the resulting data in the output of
the rules. The declaration of an attribute contains the name of the
attribute and the possible values it can take in a lexical form (in
general they correspond to the morphological attributes that
characterize the form): for example, the attribute \emph{number} can
take the values \emph{singular}, \emph{plural}, \emph{singular-plural}
(for invariable lexical forms, like \emph{crisis} in Spanish) and
\emph{number to be determined} (for TL lexical forms with different
forms for \emph{singular}--\emph{plural}, but whose number can not be
determined in the translation due to the fact that the SL lexical form
is invariable in number, see explanation in page \pageref{pg:GD}). If
inside the rule, outside of the pattern, one wishes to refer to any of
the lexical categories defined in the previous point (to perform tests
or actions on them), it will be also necessary to define attributes
for them.

\item A series of declarations of \emph{global variables}, which are
used to transfer values of active attributes inside a rule, or from
one rule to the ones applied subsequently.

\item A section for the \textit{definition of string lists}, generally
lists of lemmas, which will be used to make searches on them for a certain value
to perform a specific transformation.

\item A series of declarations of \emph{macro-instructions};
macro-instructions contain sequences of frequently used instructions,
and can be included in different rules (for example, a
macro-instruction to ensure gender and number agreement between two
lexical forms of a pattern).

\end{itemize}

In the \textbf{structural transfer rules} we find:

\begin{itemize}
\item The definition of the pattern that will be detected, specified
as a sequence of lexical categories as they have been defined in the
declaration part. It must be noted that, if a sequence of lexical
forms matches two different rules, firstly, the longest is chosen, and
secondly, for rules of the same length, the one defined before is
chosen.

\item The process part of the rules, where actions to be performed on
SLLF are specified, and the TL pattern is built.

\end{itemize} \nota{Assegurem-nos que totes les sigles estan
definides}

In the following pages we describe in detail the characteristics of
all the elements used in rules.


\subsubsection{Element \texttt{<transfer>}}

(\textit{Only in the chunker module})

This is the root element of the \texttt{chunker} module and contains
all the rest of the elements of the structural transfer rules file of
this module.

Its attribute \texttt{default} can take two values:
\begin{itemize}

\item \texttt{lu}: it means that it will run in shallow mode, that is,
as only transfer module in a shallow-transfer system and, therefore,
no special action will be done on words not detected by any pattern

\item \texttt{chunk}: it means that it will run in advanced mode and,
therefore, when a word is not recognized by any rule, a chunk will be
created to encapsulate it, so that it can be processed by the next
transfer modules of an advanced transfer system.

\end{itemize}

The default value is \texttt{lu}.

\subsubsection{Element \texttt{<interchunk>}}

(\textit{Only in interchunk})

This is the root element of the \texttt{interchunk} module and
contains all the rest of the elements of the structural transfer rules
file of this module.


\subsubsection{Element \texttt{<postchunk>}}


(\textit{Only in postchunk})

This is the root element of the \texttt{postchunk} module and contains
all the rest of the elements of the structural transfer rules file of
this module.



\subsubsection{Element for category definition section
\\\texttt{<section-def-cats>}} \nota{Atenció a l'ús polisèmic del mot
\emph{categoria} en el document}

This section contains the definition of the lexical categories that
will be used to create the patterns used in rules. Each definition is
made with a \texttt{<\textbf{def-cat}>}.



\subsubsection{Element for category definition \texttt{<def-cat>}}

Each category definition has a mandatory name \texttt{n}
(e.g. \texttt{det}, \texttt{adv}, \texttt{prep}, etc.) and a list of
categories (\texttt{<\textbf{cat-item}>}) that define it. The name of
the category can not contain accents.


\subsubsection{Element for category \texttt{<cat-item>}}


This element has two well-differentiated uses depending on the module
it is used in.

\paragraph{Use in chunker (shallow transfer and advanced transfer)}


This element defines the lexical categories that will be used in
patterns, that is, that the linguist wishes to detect in the source
text. These categories are defined by a subsequence of the fine tags
(see definition in page~\pageref{ss:introtagger}) that deliver both
the morphological analyser and the tagger\footnote{Please note that
throughout the different linguistic modules, different lexical
categorizations are used: in morphological dictionaries, lemmas are
accompanied by a fine tag (for instance, \texttt{\emph{<n><m><pl>}}
for plural masculine nouns); the POS tagger groups these fine tags in
more general tags (for instance, the category \texttt{NOUN} for all
the nouns), although its output is again the whole fine tag of each
LF; finally, in the transfer module, the fine tags of LFs are grouped
again in more general categories (although it is also possible to
define particularized categories) depending on the type of lexical
forms that one wants to detect in patterns.}.

Each \texttt{<\textbf{cat-item}>} element has a mandatory attribute
\texttt{tags} whose value is a sequence of grammatical symbols
separated by a dot; this sequence is a subsequence of the fine tag,
that is, of the sequence of grammatical symbols that defines every
possible lexical form delivered by the tagger.  According to this, a
category represents a certain set of lexical forms.  We must define as
many different categories as kinds of lexical forms we want to detect
in patterns. Thus, if we want to detect all the nouns to perform
certain actions on them, we will create a category defined with the
grammatical symbol \texttt{n}. On the other hand, if we want to detect
all the plural feminine nouns, we will have to define a category using
the symbols \texttt{n} \texttt{f} and \texttt{pl}.



When, for the set of lemmas we want to include in a category, a
grammatical symbol used to define the category is followed by other
grammatical symbols, the character \texttt{"*"} is used. For example,
\texttt{tags}=\texttt{"n.*"} covers all the lexical forms that contain
this symbol, such as the Spanish nouns \texttt{casa<n><f><pl>} or
\texttt{coche<n><m><sg>}. On the other hand, when after the used
symbol there can not be any other symbol, the asterisk is not
included: for example, \texttt{tags}=\texttt{"}\texttt{adv"} will
cover all adverbs, since in our system they are characterized with
only one grammatical symbol. The asterisk can also be used to signal
the existence of preceding symbols: \texttt{tags}=\texttt{"*.f.*"}
includes all feminine lexical forms, whichever category they
are. Furthermore, an optional attribute, \texttt{lemma}, can be used
to define lexical forms on the basis of its lemma (see Figure
\ref{fig:cat-item}).



\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{def-cat} \textsl{n}="nom"/> 
  <\textbf{cat-item} \textsl{tags}="n.*"/>
</\textbf{def-cat}>

<\textbf{def-cat} \textsl{n}="que"/> 
  <\textbf{cat-item} \textsl{lemma}="que" \textsl{tags}="cnjsub"/> 
  <\textbf{cat-item} \textsl{lemma}="que" \textsl{tags}="rel.an.mf.sp"/>
</\textbf{def-cat}>
\end{alltt}
\end{small}
\caption{Use of the \texttt{<\textbf{cat-item}>} element to define two
  categories, one for nouns without lemma specification (\emph{nom}),
  which includes all lexical forms whose first grammatical symbol is
  \emph{n}, and another one with associated lemma (\emph{que}), which
  has two subsequences of fine tags, to include the \emph{que}
  conjunction and the \emph{que} relative pronoun.}
\label{fig:cat-item}
\end{figure}


\paragraph{Use in interchunk}


It is used like in the \texttt{chunker} module, but here, instead of
being defined with the grammatical symbols of lexical forms, it is
defined with the symbols of the chunks delivered by the
\texttt{chunker}. For example, in the case that we want to define a
category to detect all the determined noun phrases, we will define it
with the symbols \texttt{NP} and \texttt{DET} if this is how we tagged
these chunks by means of the \texttt{<tag>} instructions contained in
the \texttt{<chunk>} element (see \ref{ss:chunker}). You can also use
the optional attribute \texttt{lemma} to refer to the
\emph{pseudolemma} of the chunk. So, its formal characteristics are
the same in the modules \texttt{chunker} and \texttt{interchunk}, with
the difference that in the former they are used to detect lexical
forms, and in the latter, to detect chunks.


\paragraph{Use in postchunk}

In this module, this element only has the mandatory attribute
\texttt{name}, which refers to the name of the chunk,

\nota{MG: abans deia 'al nom de la regla', comentari mlf: De la regla
o del patró?}  without tags, since in the \texttt{postchunk} module
only the pseudolemma (name of the chunk) is used for detection.  Case
is ignored in detection, because the pseudolemma is used to convey
information about the case of the chunk. (See Figure
\ref{fig:cat-item-postchunk}).

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{def-cat} \textsl{n}="det-nom"/> 
  <\textbf{cat-item} \textsl{name}="det-nom"/>
</\textbf{def-cat}>
\end{alltt}
\end{small}
\caption{Use of the \texttt{<\textbf{cat-item}>} element in the
postchunk to detect chunks of determiner-noun.}
\label{fig:cat-item-postchunk}
\end{figure}



\subsubsection{Element for category attribute definition section
\\\texttt{<section-def-attrs>}}


This section is to describe the attributes that will be extracted
from the categories detected by the pattern and that will be used in
the action part of the rules. Each attribute is defined by a
\texttt{<\textbf{def-attr}>} tag.

\nota{De vegades les etiquetes aprareixen en el text en negretes i de
vegades sense negretes. Decidim-nos per una tipografia i usem-la en
tot el document.}


\subsubsection{Element for category attribute definition
\\\texttt{<def-attr>}}

Each \texttt{<\textbf{def-attr}>} defines an attribute regarding
morphological information (both inflection information --gender,
number, person, etc.--, and categorial --verb, adjective, etc--) by
specifying a list of category attribute
(\texttt{<\textbf{attr-item}>}) elements, and has a mandatory unique
name \texttt{n}. Therefore, an attribute is defined on the basis of
the grammatical symbols that can be found in a given lexical
form. Each attribute extracts, from the lexical forms of the pattern,
the symbols that these contain among the set of possible values
defined.

\subsubsection{Element for category attribute \texttt{<attr-item>}}

Each category attribute element represents one of the possible values
the attribute can take. For example, the attribute for number
\texttt{nbr} can take the values singular \texttt{sg}, plural
\texttt{pl}, singular--plural \texttt{sp} and number to be determined
\texttt{ND}. These values are a subsequence of the morphological tags
that characterize each lexical form, and are specified in the
\texttt{tags} attribute of the element, separated by a dot if there is
more than one. In Figure \ref{fig:attr-item} you can find an example
for the attributes for \emph{number} and \emph{noun}.  \nota{Potser
s'hauria d'explicar per què s'ha triat el nom \emph{a\_nom} en la
figura}

Compare the definition of the attribute for number in this figure
(with all possible values and without asterisks) with the definition
of the category for noun in Figure \ref{fig:cat-item}.



\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{def-attr} \textsl{n}="nbr"/>
  <\textbf{attr-item} \textsl{tags}="sg"/>
  <\textbf{attr-item} \textsl{tags}="pl"/>
  <\textbf{attr-item} \textsl{tags}="sp"/>
  <\textbf{attr-item} \textsl{tags}="ND"/>
</\textbf{def-attr}>

<\textbf{def-attr} \textsl{n}="a_nom"/>
  <\textbf{attr-item} \textsl{tags}="n"/>
  <\textbf{attr-item} \textsl{tags}="n.acr"/>
</\textbf{def-attr}>

\end{alltt}
\end{small}
\caption{Definition of the category attribute \texttt{nbr} for
  \emph{number}, which can take the values \emph{singular},
  \emph{plural}, \emph{singular-plural} or
 \emph{number to be determined}, and the category attribute
\texttt{a\_nom} for \emph{noun}, which can take the values of the
symbols \emph{n} or \emph{n acr}.}
\label{fig:attr-item}
\end{figure}


\subsubsection{Element for variable definition section
\\\texttt{<section-def-vars>}}

In this section, \texttt{<\textbf{def-var}>} tags are used to define
global string variables, that will be used to transfer information
inside the rule and from one rule to another one (for example, to
transmit information on gender or number between two patterns)


\nota{Que quede clar que aquesta transferència d'una regla a altra es
fa només d'una aplicació d'una regla a l'aplicació d'altra regla en un
moment posterior, o d'esquerra a dreta}

\subsubsection{Element for variable definition \texttt{<def-var>}}
\label{ss:defvar} The definition of a global string variable has a
mandatory unique name \texttt{n} that will be used to refer to it
inside a rule.  Variables contain strings that describe state
information, such as the existence of agreement between two elements,
the detection of a question mark in SL that should be deleted in TL,
etc.


\subsubsection{Element for string lists definition section
\\\texttt{<section-def-lists>}} In this section, lists are defined
(with \texttt{<\textbf{def-list}>} tags) that will be used to do
string searches.  These lists can be used to group word lemmas that
have a common feature (i.e. verbs expressing movement, adjectives
expressing emotions, etc.).  This section is optional.

\subsubsection{Element for string lists definition
\texttt{<def-list>}} This element is used to name the string list,
with the attribute \texttt{n}, and to encapsulate the list defined by
one or more \texttt{<\textbf{list-item}>} elements. An example of its
use can be found in Figure \ref{fig:deflist}.

\subsubsection{Element for string list item \texttt{<list-item>}} It
 defines, with the value of the attribute \texttt{v}, the specific
 string that is included in the definition of the list.  An example of
 its use can be found in Figure \ref{fig:deflist}.




\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{def-list} n="verbos_est">
  <\textbf{list-item} v="actuar"/>
  <\textbf{list-item} v="buscar"/>
  <\textbf{list-item} v="estudiar"/>
  <\textbf{list-item} v="existir"/>
  <\textbf{list-item} v="ingressar"/>
  <\textbf{list-item} v="introduir"/>
  <\textbf{list-item} v="penetrar"/>
  <\textbf{list-item} v="publicar"/>
  <\textbf{list-item} v="treballar"/>
  <\textbf{list-item} v="viure"/>
<\textbf{/def-list}>
\end{alltt}
\end{small}
\caption{Definition of a list of Catalan lemmas. These lemmas are used
in the rule in Figure \ref{fig:in}.}
\label{fig:deflist}
\end{figure}


\subsubsection{Element for macro-instruction definition section
\\\texttt{<section-def-macros>}}

This section is for the definition of macro-instructions that contain
pieces of code used frequently in the action part of the rules.

\subsubsection{Element for macro-instruction definition
\texttt{<def-macro>}}

Each macro-instruction definition has a mandatory name (the value of
the attribute \texttt{n}), the number of arguments it receives
(attribute \texttt{npar}) and a body with instructions.


\subsubsection{Element for rules section \texttt{<section-rules>}}

This section contains the structural transfer rules, each one in a
\texttt{<\textbf{rule}>} element.

\subsubsection{Element for rule \texttt{<rule>}}

Each rule has a pattern (\texttt{<\textbf{pattern}>}) and the
associated action (\texttt{<\textbf{action}>}) performed when the
pattern is matched.

The rule can have an optional attribute \texttt{comment} with a
comment on, usually, the function of the rule.

\subsubsection{Element for pattern \texttt{<pattern>}}

A pattern is specified using pattern items
(\texttt{<\textbf{pattern-\\item}>}), each one of which corresponds to
a lexical form in the matched pattern, in order of appearance.

\subsubsection{Element for pattern constituent
\texttt{<pattern-item>}}

Each pattern item specifies, in the attribute with mandatory name
\texttt{n}, which kind of lexical form is to be matched.  To do that,
one has to use the categories defined in
\texttt{<\textbf{section-def-cats}>} (see in Figure \ref{fig:regla}
the definition of a pattern for determiner--noun ).


\subsubsection{Element for action \texttt{<action>}}

This element contains the ``instructions'' that have to be executed to
process as desired each matched pattern.

The processing part for matched patterns is a block of zero or more
instructions of the kind: \texttt{<\textbf{choose}>} (conditional
processing), \texttt{<\textbf{let}>} (value assignment),
\texttt{<\textbf{out}>} (print TL lexical forms),
\texttt{<\textbf{modify-case}>} (modify case state of a lexical form),
\texttt{<\textbf{call-macro}>} (call a macro-instruction) and
\texttt{<\textbf{append}>} (concatenate strings).


Through the processing step, depending on whether a series of
conditional options are met or not, different operations are carried
out, such as creating agreement between pattern components, necessary
when these undergo gender or number changes in the lexical transfer
process. To do this, in spite of working with TLLF, also the SL
information is taken into account, since, for example, if pattern
components do not agree in SL, maybe they do not have to agree in TL
either. As a consequence of the application of the different
operations in a pattern, values are assigned to pattern attributes
and, if applicable, to global or state variables, and the information
on the resulting TL pattern is sent to the next module (the
morphological generator in a shallow-transfer system, or the next
transfer module in an advanced transfer system).


\subsubsection{Element for macro-instruction call
\texttt{<call-macro>}}

In a rule it is possible to call any of the macro-instructions defined
in \texttt{<\textbf{section-def-macros}>}. To do this, one has to
specify the name of the macro-instruction in the \texttt{n} attribute,
and one or more arguments in the parameter element
\texttt{<\textbf{with-param}>} (see next).

\subsubsection{Element for parameters \texttt{<with-param>}}

This element is used inside a macro-instruction call
\texttt{<\textbf{call-macro}>}. The \texttt{pos} attribute of an
argument is used to refer to a lexical form of the rule from where the
macro-instruction is called. For example, if a macro-instruction with
2 parameters has been defined, to make agreement operations between
noun--adjective, it can be used with arguments 1 and 2 in a rule for
noun--adjective, with arguments 2 and 3 in a rule for
determiner--noun--adjective, with arguments 1 and 3 in a rule for
noun--adverb--adjective and with arguments 2 and 1 in a rule for
adjective--noun. You can see an example of macro-instruction call in
Figure \ref{fig:macro}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{call-macro} n="f_concord2"> 
  <\textbf{with-param} pos="3"/>
  <\textbf{with-param} pos="1"/>
<\textbf{/call-macro}>
\end{alltt}
\end{small}
\caption{Call of the macro-instruction \texttt{f-concord2} designed to
create agreement between two elements in a pattern such as
determiner--adverb--noun. Propagation of gender and number is done
from one of the components, in this case, from the noun which is the
third element of the pattern (3). Therefore, the position of the noun
is the first parameter given, and the other parameters come
next. Since the adverb (in position 2) does not need agreement
information, only the position of the determiner is specified (1).}
\label{fig:macro}
\end{figure}



\subsubsection{Element for selection \texttt{<choose>}}
\label{choose}

The selection instruction consists of one or more conditional options
(\texttt{<\textbf{when}>}) and an alternative option
\texttt{<\textbf{otherwise}>}, which is optional.


\subsubsection{Element for condition \texttt{<when>}}

This element describes a conditional option (see Section
\ref{choose}).  It contains the condition to be tested
\texttt{<\textbf{test}>} and one block of zero or more instructions of
the kind \texttt{<\textbf{choose}>}, \texttt{<\textbf{let}>},
\texttt{<\textbf{out}>}, \texttt{<\textbf{modify-case}>},
\texttt{<\textbf{call-macro}>} or \texttt{<\textbf{append}>}, \nota{OK
append?} which will be executed if the above condition is met.

\subsubsection{Element for alternative option \texttt{<otherwise>}}

The element \texttt{<\textbf{otherwise}>} contains one block of one or
more instructions (of the kind \texttt{<\textbf{choose}>},
\texttt{<\textbf{let}>}, \texttt{<\textbf{out}>},
\texttt{<\textbf{modify-case}>}, \texttt{<\textbf{call-macro}>} and
\texttt{<\textbf{append}>}) that must be executed if none of the
conditions described in the \texttt{<\textbf{when}>} elements of a
\texttt{<\textbf{choose}>} is met.

\subsubsection{Element for evaluation \texttt{<test>}}

The test element \texttt{<\textbf{test}>} in a condition element
\texttt{<\textbf{when}>} can contain a conjunction
(\texttt{<\textbf{and}>}), a disjunction (\texttt{<\textbf{or}>}) or a
negation (\texttt{<\textbf{not}>}) of conditions to be tested, as well
as a simple condition of string equality (\texttt{<\textbf{equal}>}),
string beginning (\texttt{<\textbf{begins-with}>}), string end
(\texttt{<\textbf{ends-with}>}), substring
(\texttt{<\textbf{contains-substring}>}) or inclusion in a set
(\texttt{<\textbf{in}>}).

\nota{Segur que es pot millorar la redacció de l'últim paràgraf,
canviat per mlf perquè hi estiguen totes les condicions booleanes
simples.}

\subsubsection{Elements for conditional or boolean operators:
\texttt{<equal>}, \texttt{<and>}, \texttt{<or>}, \texttt{<not>},
\texttt{<in>}}

\nota{To be completed: add \texttt{contains-substring},
\texttt{ends-with}, \texttt{begins-with}, etc.}

\begin{itemize}
  
\item The conjunction element \texttt{<\textbf{and}>} represents a
condition, consisting of two or more conditions, that is met when all
included conditions are true. An example of its use can be found in
Figure \ref{fig:regla}.

\item The disjunction element \texttt{<\textbf{or}>} represents a
condition, consisting of two or more conditions, that is met when at
least one of the included conditions is true. Figure \ref{fig:ornot}
displays an example of this condition type used when testing gender
agreement in a SL pattern.
  
\item The negation element \texttt{<\textbf{not}>} represents a
condition that is met when the included condition is not met, and vice
versa. An example of negation of an equality can be found in Figure
\ref{fig:ornot}.

\item The conditional equality operator \texttt{<\textbf{equal}>} is
an instruction that evaluates if two arguments (two strings) are
identical or not. See examples of its use in Figures \ref{fig:clip}
and \ref{fig:lit-tag}.  In addition, this operator can have the
attribute \texttt{caseless}, which, when its value is \texttt{yes},
causes the comparison of strings to be made ignoring case.  \nota{All
string conditional tests have the attribute \texttt{caseless}; also
\texttt{in} described below}

\item The "search in lists" operator \texttt{<\textbf{in}>} is used to
search for any value (specified as the first parameter of the condition)
in a list referred to by the \texttt{n} attribute of the
\texttt{<\textbf{list}>} element; this list must be defined in the
appropriate section (\texttt{<\textbf{section-def-lists}}).  The
search result is true if the value is found in the list.  This
comparison can also use the attribute \texttt{caseless}: if its value
is \texttt{yes}, the search is done ignoring case. Figure \ref{fig:in}
shows an example of its use.
  
\end{itemize}

\nota{Cal unificar tota la discussió anterior, traient factor comú.}

\nota{Cal descriure la resta d'elements condicionals que no hi són.}


\subsubsection{Element \texttt{<clip>}}
\label{ss:clip}


The \texttt{<\textbf{clip}>} element represents a substring of a SL or
TL lexical form, defined by the value of its different attributes (see an
example in Figure \ref{fig:clip}):

\begin{itemize}
\item \texttt{pos} is an index (1, 2, 3, etc.) used to select a
lexical form inside a rule: it refers to the place the lexical form
occupies in the pattern. In the \textit{postchunk} module there is
also the index ``0'', which refers to the pseudolemma of the chunk
\nota{MG: is it not "lexical pseudoform"?}, which is treated as a word
by itself in order to be able to consult its information and make
decisions from this.

\item \texttt{side} \textit{(only in the \texttt{chunker} module)}
specifies if the selected \emph{clip} is from the source language
(\texttt{sl}) or from the target language (\texttt{tl}).
  
\item \texttt{part} indicates which part of the lexical form is
processed; generally its value is one of the attributes defined in
\texttt{<\textbf{section-def-\\attrs}>} (\texttt{gen}, \texttt{nbr},
etc.), although it can also take four predefined values: \texttt{lem}
(refers to the lemma of the lexical form), \texttt{lemh} (the first
part of a split lemma), \texttt{lemq} (the queue of a split lemma),
and \texttt{whole} (the whole lexical form, including lemma and all
grammatical symbols, which may have been modified in the preceding
part of the rule).

\item \texttt{link-to} \textit{(only in the \texttt{chunker} module in
  advanced mode)} replaces the value that would result from consulting
  the rest of the attributes of the clip, by the value specified in
  this attribute, which must be a natural number ($>0$). \nota{MG:
  explain the new characteristics - Sergio?} This number indicates to
  which \texttt{<\textbf{tag}>} of the \texttt{<\textbf{chunk}>} is
  linked the clip content, the number being the order this tag
  occupies inside the element \texttt{<\textbf{tags}>}. The other
  attributes of the clip remain only for informational purposes, since
  they are overwritten by the value of the linked tag. An example of
  its use can be found in Figure \ref{fig:chunkintrachunk}.
   
\end{itemize}


\begin{figure}
\begin{small}
\begin{alltt}
    <\textbf{test}>
      <\textbf{not}>
        <\textbf{equal}> 
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="gen"/> 
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="sl" \textsl{part}="gen"/>
        <\textbf{/equal}> 
      <\textbf{/not}>
    <\textbf{/test}>
\end{alltt}
\end{small}
\caption{Extract from a rule where it is tested whether the TL
(\texttt{tl}) gender (\texttt{gen}) of the second lexical unit
identified in a pattern is different from the gender of the same
lexical unit in the SL (\texttt{sl})}.
\label{fig:clip}
\end{figure}



\subsubsection{Element for literal string \texttt{<lit>}} This element
is used to specify the value of a literal string by means of the
attribute \texttt{v}. For example, \texttt{<\textbf{lit}
v=\texttt{"}andar\texttt{"}/>} represents the string \emph{andar}.


\subsubsection{Element for tag value \texttt{<lit-tag>}} It is similar
to the \texttt{<\textbf{lit}>} element, with the difference that it
does not specify the value of a literal string but the value of a
grammatical symbol or tag, by means of the attribute \texttt{v}. An
example of its use can be found in Figure \ref{fig:lit-tag}.


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{equal}>
  <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="nbr"/>
  <\textbf{lit-tag} \textsl{v}="ND"/>
<\textbf{/equal}>
\end{alltt}
\end{small}
\caption{Use of the element \texttt{<\textbf{lit-tag}>}: it is tested
  whether the number (\texttt{nbr}) symbol of the second
  lexical unit in the TL (\texttt{tl}) is \texttt{ND} (number to be
  determined)}
\label{fig:lit-tag}
\end{figure}
 
\begin{figure}
\begin{small}
\begin{alltt}
   <\textbf{test}>
    <\textbf{or}>
      <\textbf{not}> 
        <\textbf{equal}>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="sl" \textsl{part}="gen"/>
          <\textbf{clip} \textsl{pos}="3" \textsl{side}="sl" \textsl{part}="gen"/>
        <\textbf{/equal}>
      <\textbf{/not}>
      <\textbf{not}>
        <\textbf{equal}>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="sl" \textsl{part}="gen"/>
          <\textbf{clip} \textsl{pos}="3" \textsl{side}="sl" \textsl{part}="gen"/>
        <\textbf{/equal}>
      <\textbf{/not}>
    <\textbf{/or}>
  <\textbf{/test}>
\end{alltt}
\end{small}
\caption{Extract from a rule where it is tested whether the SL gender
  of the first or the second lexical unit matched in a pattern (it
  could be, for example, determiner--adjective--noun) is different
  from the gender of the third lexical unit also in the SL.}
\label{fig:ornot}
\end{figure}



\subsubsection{Element for variable \texttt{<var>}}


Each \texttt{<\textbf{var}>} is a variable identifier: the mandatory
attribute \texttt{n} specifies its name as has been defined in
\texttt{<\textbf{section-def-vars}>}. When it appears in an
\texttt{<\textbf{out}>}, a \texttt{<\textbf{test}>}, or the right part
of a \texttt{<\textbf{let}>}, it represents the value of the variable;
when it appears on the left side of a \texttt{<\textbf{let}>}, in an
\texttt{<\textbf{append}>} or in a \texttt{<\textbf{modify-case}>}, it
represents the reference of the variable and its value can be changed.

\subsubsection{Element for reference to string list \texttt{<list>}}

This element is only used as the second parameter of a
\texttt{<\textbf{in}>} search.  The \texttt{n} attribute refers to the
specific list defined in the string lists definition section
\texttt{<\textbf{section-def-lists}>}. An example of its use can be found in
Figure \ref{fig:in}.


\begin{figure}
\begin{small}
\begin{alltt}
    <\textbf{rule}>
      <\textbf{pattern}> 
        <\textbf{pattern-item} \textsl{n}="verb"/>
        <\textbf{pattern-item} \textsl{n}="a"/>
      <\textbf{/pattern}> 
      <\textbf{action}>
      <\textbf{choose}>
        <\textbf{when}>
          <\textbf{test}>
            <\textbf{in} \textsl{caseless}="yes"/> 
              <\textbf{clip} \textsl{pos}="1" \textsl{side}="sl" \textsl{part}="lem"/> 
              <\textbf{list} \textsl{n}="verbos_est"/>
            <\textbf{/in}> 
          <\textbf{/test}>
          <\textbf{let}> 
            <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="lem"/> 
            <\textbf{lit} \textsl{v}="en"/>
         <\textbf{/let}> 
      <\textbf{/when}> 
      <!-- ... -->
\end{alltt}
\end{small}
\caption{Extract of a rule that detects a pattern made of a verb and
  the preposition \emph{a}, and then testes whether the verb (the
  lemma indicated in \texttt{lem}) of the source language
  (\texttt{sl}) is one of the lemmas included in the list of state
  verbs (defined in Figure \ref{fig:deflist}). If that be the case,
  the lemma of the second word in target language (\texttt{tl}) is
  changed to \emph{en}.}
\label{fig:in}
\end{figure}


\subsubsection{Element for case application \texttt{<get-case-from>}}

The \texttt{<\textbf{get-case-from}>} element represents the string
obtained after applying the letter case state of the lemma of a SL
lexical unit to a string (\emph{clip}, \emph{lit} or \emph{var}).  To
refer to the lexical unit from where the information is taken, the
attribute \texttt{pos} is used, which indicates the position of that
unit in the SL. This element is useful when the lexical units in a
pattern are reordered, or when a lexical unit is added or deleted. You
can see an example of its use in Figure \ref{fig:case}, which displays
a rule to transform the simple perfect preterite tense in Spanish
(\emph{dije}, "I said") into the compound form in Catalan (\emph{vaig
dir}). In this rule, a LF with lemma \emph{anar} and grammatical
symbol \emph{vaux} ("auxiliary verb") is added; it has to take the
case information from the Spanish verb (which has position "1" in the
pattern), so that the system translates \emph{Dije} as \emph{Vaig
dir}, \emph{dije} as \emph{vaig dir} and \emph{DIJE} as \emph{VAIG
DIR}.


\subsubsection{Element for case pattern query \texttt{<case-of>}}

It is used to get the case pattern of a string, that is, one of the
values "\texttt{aa}", "\texttt{Aa"} or "\texttt{AA}". It works like the
\texttt{<\textbf{clip}>} element, since it has the same attributes:
\texttt{pos}, the position of the word in the matched pattern;
\texttt{part}, the specific attribute that we refer to (normally the
lemma), which has the predefined attributes described in Section
\ref{ss:clip}, and finally, only in the \texttt{chunker} module, the
attribute \texttt{side}, referring to the translation side,
\texttt{sl} or \texttt{tl}. In Figure \ref{fig:case} you can see this
element in use, and you can find a more detailed description of this
example in the following Section (description of
\texttt{<\textbf{modify-case}>}).


\subsubsection{Element for case modification \texttt{<modify-case>}}

This instructions is used to modify the case of the first parameter
(usually a lemma) by means of the second parameter (a literal or a
variable). The first parameter can be a \texttt{<\textbf{var}>}, a
\texttt{<\textbf{clip}>} or a \texttt{<\textbf{case-of}>}, whereas the
second one can be anything that delivers a value, but in principle it
will be a \texttt{<\textbf{var}>} or a \texttt{<\textbf{lit}>}.  The
values that this value can take are usually ``\texttt{Aa}'', to
express that the ``left part'' of this case modification must have the
first letter in upper case and the rest in lower case, ``\texttt{aa}''
to put all in lower case, and ``\texttt{AA}'' to put all in upper
case.

Figure \ref{fig:case} shows a rule where this element is used. It
modifies in this rule the case of the TL lemma in position "1", which
corresponds to \emph{dir}, because, although in the rule output this
verb is the second lexical form (\emph{vaig dir}), it is actually the
translation of the LF which has position 1 in the SL, and, therefore,
it retains the same assigned position in the TL. This lemma is
assigned the value ``\texttt{aa}'' in the case that the SL lemma has
the state ``\texttt{Aa}''. There is nothing to specify for the rest of
the cases, since the case state of the LF in position 1 will be the
same in the SL and in the TL and, therefore, will be automatically
transferred (see Section~\pageref{mayusc} to obtain more information
on letter case handling in dictionaries ).


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{rule}>
  <\textbf{pattern}>
    <\textbf{pattern-item} n="pretind"/>
  <\textbf{/pattern}>
  <\textbf{action}>
    <\textbf{out}>
      <\textbf{lu}>
         <\textbf{get-case-from} pos ="1">
           <\textbf{lit} v="anar"/>
         <\textbf{/get-case-from}>
         <\textbf{lit-tag} v="vaux"/>
         <\textbf{clip} pos="1" side="sl" part="persona"/>
         <\textbf{clip} pos="1" side="sl" part="nbr"/>
       <\textbf{/lu}>
       <\textbf{b/}>
     <\textbf{/out}>
     <\textbf{choose}>
       <\textbf{when}>
         <\textbf{test}>
           <\textbf{equal}>
              <\textbf{case-of} pos="1" side="sl" part="lemh"/>
              <\textbf{lit} v="Aa"/>
           <\textbf{/equal}>
         <\textbf{/test}>
         <\textbf{modify-case}>
             <\textbf{case-of} pos="1" side="tl" part="lemh"/>
             <\textbf{lit} v="aa"/>
         <\textbf{/modify-case}>
       <\textbf{/when}>
     <\textbf{/choose}>
     <\textbf{out}>
       <\textbf{lu}>
          <\textbf{clip} pos="1" side="tl" part="lemh"/> 
          <\textbf{clip} pos="1" side="tl" part="a_verb"/>
          <\textbf{lit-tag} v="inf"/>
          <\textbf{clip} pos="1" side="tl" part="lemq"/>
       <\textbf{/lu}>
    <\textbf{/out}>
  <\textbf{/action}>
<\textbf{/rule}>
\end{alltt}
\end{small}
\caption{Rule for the translation from Spanish into Catalan, which
  turns the verbs in simple perfect preterite tense (\emph{dije}) into
  the
  compound perfect preterite tense usual in Catalan (\emph{vaig dir}),
    and at the same time assigns the appropriate case state
  to the two resulting words.}
\label{fig:case}
\end{figure}



\subsubsection{Element for assignment \texttt{<let>}}

The assignment instruction \texttt{<\textbf{let}>} assigns the value
of the right part of the assignment (a literal string, a
\texttt{clip}, a variable, etc.) to the left part (a \texttt{clip}, a
variable, etc.). An example of its use can be found in Figure
\ref{fig:regla}.



\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{rule}>
  <\textbf{pattern}>
    <\textbf{pattern-item} n="det"/>
    <\textbf{pattern-item} n="nom"/>
  <\textbf{/pattern}>
  <\textbf{action}>
      <\textbf{choose}>
        <\textbf{when}>
          <\textbf{test}>
            <\textbf{and}>
              <\textbf{not}>
                <\textbf{equal}>
                  <\textbf{clip} pos="2" side="tl" part="gen"/>
                  <\textbf{clip} pos="2" side="sl" part="gen"/>
                <\textbf{/equal}>
              <\textbf{/not}>
              <\textbf{not}>
                <\textbf{equal}>
                  <\textbf{clip} pos="2" side="tl" part="gen"/>
                  <\textbf{lit-tag} v="mf"/>
                <\textbf{/equa}l>
              <\textbf{/not}>
              <\textbf{not}>
                <\textbf{equal}>
                  <\textbf{clip} pos="2" side="tl" part="gen"/>
                  <\textbf{lit-tag} v="GD"/>
                <\textbf{/equal}>
              <\textbf{/not}>
            <\textbf{/and}>
          <\textbf{/test}>
          <\textbf{let}>
            <\textbf{clip} pos="1" side="tl" part="gen"/>
            <\textbf{clip} pos="2" side="tl" part="gen"/> 
          <\textbf{/let}>
        <\textbf{/when}>
      <\textbf{/choose}>
      <!-- Other gender and number agreement actions -->
\end{alltt}
\end{small}
\caption{Extract from a rule for the pattern \texttt{determiner--noun}
  (continues in Fig. \ref{fig:regla2}): in this part of the rule, the
  gender of the noun is assigned to the determiner in the case that
  the gender of the noun changes from the SL (\texttt{sl}) to the TL
  (\texttt{tl}) during the lexical transfer process between both
  languages.}
\label{fig:regla}
\end{figure}

\subsubsection{Element for string concatenation \texttt{<concat>}}

This element is used to concatenate strings in order to assign them to
a variable. It is used in combination with \texttt{<\textbf{let}>},
and the previous value of the variable is lost with the assignment of
\texttt{<\textbf{concat}>}.

It does not have any attribute. It can contain any instruction that
delivers a string, such as \texttt{<\textbf{lit}>},
\texttt{<\textbf{lit-tag}>} or \texttt{<\textbf{clip}>}.

Figure \ref{fig:concat} shows an example of its use.


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{let}>    
  <\textbf{var} n="palabra"/>
    <\textbf{concat}>
       <\textbf{clip} pos="3" side="tl" part="lem"/>  
       <\textbf{lit-tag} v="adj"/>
    <\textbf{/concat}>
<\textbf{/let}>   
\end{alltt}
\end{small}
\caption{In this example, the variable \texttt{palabra} is assigned
the value of the concatenation of a \texttt{clip} (the lemma in
position 3) and the \emph{adj} tag.}
\label{fig:concat}
\end{figure}




\subsubsection{Element for string concatenation \texttt{<append>}}

The \texttt{<\textbf{append}>} instruction can be used to save the
output of an action before printing it in the corresponding
\texttt{<\textbf{out}>}, if required by the designer of the transfer
rules.

The mandatory attribute \texttt{n} specifies the name of the variable
used. After applying the instruction, the previous content of the
referred variable will be the prefix of the new content, that is, the
new content inserted in the \texttt{<\textbf{append}>} will be
concatenated to the pre-existing content of the variable specified in
\texttt{n}.

The content of this instruction can be one or more of the following
tags: \texttt{<\textbf{b}>}, \texttt{<\textbf{clip}>},
\texttt{<\textbf{lit}>}, \texttt{<\textbf{lit-tag}>},
\texttt{<\textbf{var}>}, \texttt{<\textbf{get-case-from}>},
\texttt{<\textbf{case-of}>} or \texttt{<\textbf{concat}>}. There is an
example of its use in Figure \ref{fig:append}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{append} n="temporal"> 
  <clip pos="3" part="gen" side="tl"/>
<\textbf{/append}>
\end{alltt}
\end{small}
\caption{In this example, the variable \texttt{temporal} is assigned
the value of the gender, in the TL, of the third word matched by the
rule.}
\label{fig:append}
\end{figure}




\subsubsection{Element for output \texttt{<out>}}

\label{ss:out} The output instruction is used to specify the lexical
forms that are sent at the output of the module after having been
applied the required structural transfer operations. Its use is
different according to the module. On the one hand, its use in the
\texttt{chunker} module when it runs as only module (shallow-transfer)
and its use in the \texttt{postchunk} module are similar, since in
both cases, the output must be the input for the generator.  The
\texttt{chunker} in Apertium 2 and the \texttt{interchunk} have
different use modes: the former to create the chunks, and the latter
to modify the chunks without modifying its internal part.

\begin{enumerate}

\item \textbf{Use in \texttt{chunker} in shallow-transfer mode, and in
\texttt{postchunk}}

  The instruction sends each lexical form inside a
  \texttt{<\textbf{lu}>} set, which in turn can be contained inside a
  \texttt{<\textbf{mlu}>} element when the output is a multiword made
  of two or more LF. Besides, also the blanks or superblanks
  (\texttt{<\textbf{b}>}) between LF and LF are sent. You can find an
  example of its use in Figures \ref{fig:case} and \ref{fig:regla2}.

\begin{figure}
\begin{small}
\begin{alltt} 
    <!-- ... -->
    <\textbf{out}>
      <\textbf{lu}>
         <\textbf{clip} pos="1" side="tl" part="whole"/> 
      <\textbf{/lu}>
      <\textbf{lu}>
         <\textbf{clip} pos="2" side="tl" part="whole"/>
      <\textbf{/lu}>
    <\textbf{/out}>
  <\textbf{/process}>
 <\textbf{/action}>
<\textbf{/rule}>
\end{alltt}
\end{small}
\caption{Extract from a rule (comes from Fig. \ref{fig:regla}). At the
  end of the rule, and after different actions, the resulting data are
  sent by means of the attribute \texttt{whole}, which contains the
  lemma and the grammatical symbols of each LF (positions 1 and 2 in
  the pattern).}
\label{fig:regla2}
\end{figure}


\item \textbf{Use in \texttt{chunker} in advanced mode}

  The output of this module is expected to be a sequence of one or
  more chunks (sent inside a \texttt{<\textbf{chunk}>} element)
  separated by blanks \texttt{<\textbf{b}>}. Lexical forms and
  multiforms, as well as the blanks between them, are sent inside
  chunks. You can see in Figure \ref{fig:chunkintrachunk} an example
  of use.


\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{out}>
  <\textbf{chunk} name="pr" case="caseFirstWord">
    <\textbf{tags}> 
      <\textbf{tag}><\textbf{lit-tag} v="PREP"/><\textbf{/tag}>
    <\textbf{/tags}>
    <\textbf{lu}> 
      <\textbf{clip} pos="1" side="tl" part="whole"/>
    <\textbf{/lu}> 
  <\textbf{/chunk}> 
  <\textbf{b} pos="1"/>
  <\textbf{chunk} name="probj" case="caseOtherWord">
    <\textbf{tags}> 
      <\textbf{tag}><\textbf{lit-tag} v="NP"/><\textbf{/tag}> 
      <\textbf{tag}><\textbf{lit-tag} v="tn"/><\textbf{/tag}> 
      <\textbf{tag}><\textbf{clip} pos="2" side="tl" part="pers"/><\textbf{/tag}>
      <\textbf{tag}><\textbf{clip} pos="2" side="tl" part="gen"/><\textbf{/tag}> 
      <\textbf{tag}><\textbf{clip} pos="2" side="tl" part="nbr"/><\textbf{/tag}>
    <\textbf{/tags}>
    <\textbf{lu}> 
      <\textbf{clip} pos="2" side="tl" part="lem"/>
      <\textbf{lit-tag} v="prn"/> 
      <\textbf{lit-tag} v="2"/>
      <\textbf{clip} pos="2" side="tl" part="pers"/> 
      <\textbf{clip} pos="2" side="tl" part="gen" link-to="4"/> 
      <\textbf{clip} pos="2" side="tl" part="nbr" link-to="5"/>
    <\textbf{/lu}> 
  <\textbf{/chunk}>
<\textbf{/out}>
\end{alltt}
\end{small}
\caption{Output instruction that sends two chunks separated by a
  blank. The printed sequence is a preposition followed by a noun
  phrase ("NP"). The tags that are linked from the second chunk to the outside are
  pronoun type ("tn"), gender and number of the noun phrase
  (pronoun). The \texttt{<\textbf{tag}>} elements are used to specify
  the tags of the chunk, and the value of the attributes \texttt{name}
  and \texttt{case} is used to specify the pseudolemma of the chunk.}
\label{fig:chunkintrachunk}
\end{figure}


\item \textbf{Use in \texttt{interchunk}}

  In this module, lexical forms (words) are inaccessible, since it is
  only possible to operate with chunks and, therefore, inside an
  \texttt{<\textbf{out}>} element you can only put
  \texttt{<\textbf{chunk}>} elements or blanks \texttt{<\textbf{b}>}.
  The information on lemma and tags specified here in a \texttt{<\textbf{chunk}>}
  element refers exclusively to the lemma (pseudolemma) and the tags of
  the chunk.

An example of its use can be found in Figure
\ref{fig:chunkinterchunk}.

\begin{figure}
\begin{small}
\begin{alltt}
<\textbf{out}> 
  <\textbf{b} pos="1"/>
  <\textbf{chunk}> 
    <\textbf{clip} pos="2" part="lem"/> 
    <\textbf{clip} pos="2" part="tags"/> 
    <\textbf{clip} pos="2" part="chcontent"/>
  <\textbf{/chunk}> 
<\textbf{/out}>
\end{alltt}

\end{small}
\caption{The aim of this rule output is to discard the first chunk of
  the matched pattern (pronoun drop). The three
  \texttt{<\textbf{clip}>} elements have been included here for
  illustrative purposes, since they could have been replaced by the
  \texttt{part="whole"} which would group them in a single
  \texttt{<\textbf{clip}>} .}
\label{fig:chunkinterchunk}
\end{figure}



\end{enumerate}




\subsubsection{Element for lexical unit \texttt{<lu>}}

\label{ss:lu} This is the element by means of which each TLLF is sent out at the
end of a rule, inside an \texttt{<\textbf{out}>} element.
With this element, one can send the whole lexical form, using the
attribute \texttt{whole} of a \texttt{<\textbf{clip}>}, or, if
required, specify its parts separately (lemma plus tags, indicated by
means of \texttt{<\textbf{clip}>} strings, literal strings
\texttt{<\textbf{lit}>}, tags \texttt{<\texttt{\textbf{lit-tag}}>},
variables \texttt{<\texttt{\textbf{var}}>}, besides case information
[\texttt{<\textbf{get-case-from}>}, \texttt{<\textbf{case-of}>}]).



Please note that, as has been explained before, in the case of
multiwords with \emph{split lemma} it is necessary to replace the
lemma queue \emph{after} the grammatical symbols of the inflected word
(or lemma head), because the \texttt{pretransfer} module has moved the
queue to put it before the grammatical symbols of the head.  This
replacement is done here, inside the \texttt{<\textbf{lu}>} element,
using the values \texttt{lemh} and \texttt{lemq} of the attribute
\texttt{part} in a \texttt{<\textbf{clip}>}. The \texttt{lemh}
attribute refers to the lemma head, and \texttt{lemq} to the lemma
queue. As can be seen in the example \ref{fig:case}, the \texttt{lemq}
part of a \texttt{<\textbf{clip}>} is placed after the lemma head and
all the grammatical symbols that follow it.  This rule would be
suitable, for example, for the Spanish form \emph{eché de menos} ("I
missed"), which has to be translated into Catalan as \emph{vaig trobar
a faltar}. The attribute \texttt{a\_verb} which comes after
\texttt{lemh} contains the grammatical symbol that describes the verb
category (\emph{vblex}, \emph{vbser}, \emph{vbhaver} or \emph{vbmod}
as applicable). Therefore, the last lexical form sent by this rule, in
the case of \emph{vaig trobar a faltar}, would be, in the data stream:
\begin{alltt} ^trobar<vblex><inf># a faltar\$ \end{alltt}

\noindent The number sign \texttt{\#} in the data stream corresponds
to the \texttt{<\textbf{g}>} element in dictionaries, used to signal
the position of the invariable part in a split lemma multiword.

It is important to note that the attributes included in
\texttt{<\textbf{lu}>} may be empty. So, a verb matched by the rule in
Fig. \ref{fig:case} which is not a split lemma multiword, will be sent
with an empty \texttt{lemq} attribute, since the verb does not have
lemma queue. This way it is not necessary to define different rules
for lexical forms with and without queue. You can find another example
of this in page \pageref{regla_verbo1}, where the rule for verb sends
in a \texttt{<\textbf{lu}>} the attributes \texttt{gen}
(\emph{gender}) and \texttt{nbr} (\emph{number}). This way, it
includes participles (with gender and number) and the rest of verb
forms (which will have these attributes empty).

In the same page you can see a rule for a verb followed by an enclitic
pronoun. Here, the lemma queue is placed after the enclitic pronoun;
so, for a split lemma multiword joined to an enclitic pronoun, such as
\emph{echándote de menos}, the output in the data stream would be,
when translating into Catalan:

\begin{alltt} ^trobar<vblex><ger>+et<prn><enc><p2><mf><sg># a faltar\$
\end{alltt}

Of course, this rule works also for verbs which do not belong to this
multiword type; so, the form \emph{explicándote} ("explaining to you")
would be output, when translating from Spanish to Catalan:


\begin{alltt} ^explicar<vblex><ger>+et<prn><enc><p2><mf><sg>\$
\end{alltt}

As for the attribute \texttt{whole} of a \texttt{<\textbf{clip}>}, it
must be taken into account that it can be used to send the whole
lexical form only in the case that the sent word can not be a
multiword, that is, can not contain a split lemma.  Compare figures
\ref{fig:case} and \ref{fig:regla2}. The \texttt{whole} attribute can
be used in the second example because it contains the lemma
\texttt{lem} plus all the morphological tags of the lexical forms in
position 1 and 2 (determiner and noun). \nota{but nouns can also be mw now!}Contrarily, in the first
example, the lexical form in \texttt{<\textbf{lu}>} is sent in parts,
with a \texttt{lemh} (lemma head) and a \texttt{lemq} (lemma queue),
since it may occur that the verb matched in the pattern is a multiword
with split lemma. In practice, in our system this means that the
\texttt{whole} attribute can be used to send any kind of lexical form
except verbs and nouns, because we defined multiwords with inner
inflection only for verbs and nouns.

\subsubsection{Element for lexical unit \texttt{<mlu>}}
\label{ss:mlu}

Its name derives from \emph{multilexical unit}; it is used inside the
\texttt{<\textbf{out}>} element to output multiwords consisting of
more than one lexical form. Each lexical form in a
\texttt{<\textbf{mlu}>} is sent inside a \texttt{<\textbf{lu}>}
element. On the output of the module, lexical forms contained in this
element will be joined to each other by the symbol '+' in the data
stream. This means that they will become a multiword made of different
lexical forms, which will be treated as a single unit by the
subsequent modules; therefore, the generation dictionary will have to
contain an entry for this multiword in order for it to be generated.

In our system, this element is used to join enclitic pronouns to
conjugated verbs.

\subsubsection{Element for chunk encapsulation \texttt{<chunk>}}

This is the element in which chunks are sent, in an
\texttt{<\textbf{out}>} element, on the output of the module.  It is
only used in the \texttt{chunker} module in advanced mode, and in the
\texttt{interchunk} module.  It is not used in the \texttt{postchunk}
module because its output does not contain any chunk. Neither it is
used in the \texttt{chunker} module in shallow-transfer mode, because
its output does not contain chunks but individual lexical units and
blanks.

\begin{enumerate}

\item \textbf{Use in \texttt{chunker} in advanced mode}


In this mode, the \texttt{<\textbf{chunk}>} element must have an
attribute \texttt{name}, which is the lemma of the chunk, or an
attribute \texttt{namefrom} which refers to a variable previously
defined, whose value will be used as the lemma of the chunk. Besides,
it can include the attribute \texttt{case} to specify which variable
is the case policy taken from (for example, a value obtained with the
instruction \texttt{<\textbf{case-from}>}).

An example of its use can be found in Figure
\ref{fig:chunkintrachunk}.


\item \textbf{Use in \texttt{interchunk}}

  In this module, the \texttt{<\textbf{chunk}>} element does not
specify any attribute; it is used just as the \texttt{<\textbf{lu}>}
element is used in the shallow-transfer or in the \texttt{postchunk}
to delimit the lexical forms.  The elements it sends are (generally in
a \texttt{<\textbf{clip}>} instruction): the lemma of the chunk
(\texttt{lem}), its tags (\texttt{tags}) and the chunk content
(\texttt{chcontent}, contains LF plus blanks), which is an invariable
part since it can not be accessed from the \texttt{interchunk} module.
The invariable part of the chunk is sent at the end.  You can also use
the \texttt{whole} attribute to send the whole chunk (lemma, tags and
invariable content).

  An example of its use can be found in Figure
  \ref{fig:chunkinterchunk}.

\end{enumerate}

\subsubsection{Element for tag links section \texttt{<tags>}}

\textit{Only in chunker in advanced mode}.

This element is used to specify a list of tags, or
\texttt{<\textbf{tag}>} elements, which will become the pseudotags of
the chunk. It does not have attributes, and must be included as first
item inside the \texttt{<\textbf{chunk}>} element. See Figure
\ref{fig:chunkintrachunk}.


\subsubsection{Element for tag link \texttt{<tag>}}

\textit{Only in chunker in advanced mode}.

The \texttt{<\textbf{tag}>} element must contain a morphological tag,
which can be specified by means of a \texttt{<\textbf{clip}>}
instruction or a literal tag \texttt{<\textbf{lit-tag}>}. It does not
have attributes.

The tag or tags specified this way in a chunk will become the
grammatical symbols of the chunk; the next module,
\texttt{interchunk}, will be able to use them to test and modify the
characteristics of the chunks.


\subsubsection{Element for blank \texttt{<b>}}

The \texttt{<\textbf{b}>} element refers to [super]blanks and is
indexed by the attribute \texttt{pos}. For example, a
\texttt{<\textbf{b}>} with \texttt{pos="2"} refers to the
[super]blanks (including format data encapsulated by the de-formatter)
between the 2nd SLLF and the 3rd SLLF. The explicit management of
[super]blanks enables the correct placement of format when the result
of the structural transfer has more or less elements than the
original, or when it has been reordered in some way.


\subsection{Specification of the three modules that build an advanced
transfer system}
\label{noutransfer}

In the following lines we describe the differences between the rule
format in the three modules of an advanced transfer system. When
Apertium works as a shallow-transfer system, the only module to be run
is the first one, called \texttt{chunker}, which communicates directly
with the generation module.


\subsubsection{\texttt{Chunker} module}
\label{ss:chunker}


This module can be used alone as a shallow-transfer system, or in
combination with the other two transfer modules to build an advanced
transfer system. An attribute of the \texttt{<transfer>} element
controls its run mode.

\paragraph{Input/output}

\begin{itemize}
\item Input: data in the \texttt{pretransfer} output format, that is,
with invariable queues of multiwords moved to the position right
before the first grammatical symbol.

\item Output:
\begin{itemize}
\item[-] in advanced mode (in an advanced transfer system): chunks,
that will be detected and processed by the next module
\item[-] in shallow-transfer mode (in a shallow-transfer system):
lexical forms, that will be the input of the generation module.
\end{itemize}

\end{itemize}


\paragraph{Data files}

\nota{Explicar millor això de l'únic fitxer de configuració}

This program uses a single configuration file and a precompiled file
for pattern detection calculated from the former. The name of the
pattern file (the configuration file) will have the extension
\texttt{.t1x}.  Since the \texttt{chunker} is the program that looks
up the bilingual dictionary, this dictionary (compiled) also has to be
provided to the program.

\nota{Potser seria bona idea esmentar en quina secció s'explica el
compilador a què es fa referència}

The DTD of this data file is specified in Appendix
\ref{ss:dtdtransfer}, and the elements used to create the rules in the
file are described in Section \ref{formatotransfer}.

\paragraph{Pattern matching}

The rule matching system in this module will be the one described in
\ref{functransfer}, since it is the same in advanced transfer mode and
in shallow-transfer mode. The \texttt{a\-per\-tium-pre\-trans\-fer}
program \nota{Vacil·lació terminològica \texttt{pretransfer}.}  is
needed to adapt the tagger output format to the input format required
by the transfer module.  There is the possibility that, in later
versions of Apertium, the \textit{part-of-speech tagger} is modified
so that it does the work of \texttt{apertium-pretransfer}.
\nota{També hem d'unificar la terminologia d'altres mòduls:
\emph{desambiguador categorial}, \emph{etiquetador}; tal com està
redactat el paràgraf es podria pensar que són dues coses diferents.}


\paragraph{How it works}

The module works similarly in shallow-transfer mode and in advanced
mode, with these differences:

\begin{itemize}
\item If we want that the module works as the first module in an
advanced transfer system, we must specify the value \texttt{chunk} in
the optional attribute \texttt{default} of the root element
\texttt{<transfer>}. The default value is \texttt{lu}, which implies
that the \texttt{chunker} works in shallow-transfer mode (as a single
module).

\item Chunk generation in the output: the \texttt{<chunk>} tag is an
element one level higher than \texttt{<lu>} (\textit{lexical unit}),
which generates chunks with the characteristics described in
\ref{sec:format}; it has the following attributes:

  \begin{itemize}
  \item \texttt{name} (optional): pseudolemma of the chunk. It
  contains a string that is identified as the pseudolemma of the
  chunk.

  \item \texttt{namefrom} (optional): pseudolemma of the chunk,
  obtained from a variable. It is compulsory to specify whether
  \texttt{name} or \texttt{namefrom}.

  \item \texttt{case} (optional): variable that is used to obtain the
  information on case from it and apply it to the lemma specified in
  \texttt{name} or in \texttt{namefrom}.
  \end{itemize}

\item Each chunk begins with a \texttt{<tags>} instruction, which does
not allow any attribute, and which contains one or more individual
instructions \texttt{<tag>}.
\item Instructions \texttt{<tag>} do not have attributes. They can
include any instruction that returns a string as a value:
\texttt{<lit>}, \texttt{<var>} \nota{clip, lit-tag}.
\item Instructions \texttt{<clip>} have an optional attribute:
\texttt{link-to}, which is used to specify a tag \verb!<!\textit{value
of link-to}\verb!>! that replaces \nota{Spanish: ``una etiqueta en
lugar de'' (instead of) or ``additionally''?. Explain new aspects of
link-to} the information specified by the \texttt{<clip>} in the rest
of its attributes.\nota{No s'entén gaire bé - Not understandable} This
information is dispensable but can be useful as information on the
origin of the linguistic decision.
\end{itemize}

The following is a use example of the \texttt{<chunk>} element :

\begin{alltt}
<out>
  <chunk name="adj-noun" case="variableCase"> 
    <tags> 
      <tag><lit-tag v="NP"/></tag> 
      <tag><clip pos="2" side="tl" part="gen"/></tag>
      <tag><clip pos="2" side="tl" part="nbr"/></tag> 
    </tags> 
    <lu> 
      <clip pos="2" side="tl" part="lemh"/> 
      <clip pos="2" side="tl" part="a_noun"/> 
      <clip pos="2" side="tl" part="gen" link-to="2"/>
      <clip pos="2" side="tl" part="nbr" link-to="3"/> 
    </lu> 
    <b pos="1"/> 
    <lu> 
      <var n="adjectiu"/> 
      <clip pos="1" side="tl" part="lem"/> 
      <clip pos="1" side="tl" part="a_adj"/> 
      <clip pos="2" side="tl" part="gen" link-to="2"/> 
      <clip pos="2" side="tl" part="nbr" link-to="3"/> 
    </lu> 
  </chunk>
</out>
\end{alltt}


\paragraph{Default action}

Isolated \textit{superblanks} which are not detected by any pattern in
this module, are written in the same order in which they arrive.

The default action for words not matched by any pattern
is different depending on the transfer mode (that is, on the value of the
optional attribute \texttt{default} of the root element \texttt{<transfer>}):

\begin{itemize}
\item if the value is \texttt{chunk} (i.e. the module works in advanced
  mode): it will generate trivial chunks with the words not matched by
  any rule, so that in the output there are no words not included in a
  chunk.  The new chunk will be created with the translation of the
  word by the bilingual dictionary.  The fixed lemma of these
  implicitly created chunks is \texttt{default}.
\item if the value is \texttt{lu} (default value; i.e. the module works as single
module in a shallow-transfer system): it will not create chunks for
words not matched by rules, they will just be translated using the
bilingual dictionary.

\end{itemize}

The following is an automatically generated chunk for a lexical form
not matched by any rule in the \texttt{chunker} module when the
\texttt{default} attribute has the value \texttt{chunk}:


\begin{alltt} 
^default\verb!{!^that<cnjsub>$\verb!}!$
\end{alltt}

\nota{Va sense etiquetes entre \texttt{default} i \texttt{\{}? No
caldria dir-ho explícitament?}


\subsubsection{\texttt{Interchunk} module}
\label{ss:interchunk}


\nota{\texttt{apertium-interchunk} or simply \texttt{interchunk}?}

The \texttt{interchunk} module processes chunks; it may reorder them
and change its morphosyntactic information. This is done by detecting
patterns of chunks (sequences of chunks).  The instructions that
control how it works are, with little differences, the same used by
the \texttt{chunker} module; they are written, however, in a different
file. Chunks are processed here in a similar way as words are
processed in the \texttt{chunker} of Apertium.  \nota{Comprovar la
denominació dels programes}

\paragraph{Input/output}

\begin{itemize}
\item Input: chunks from the \texttt{chunker}.
\item Output: chunks possibly reordered and with the data on its
pseudolemmas (lexical pseudoforms) possibly changed.
\end{itemize}

\paragraph{Data files}

This module uses two data files. A specification file of the
\texttt{in\-ter\-chunk} program, with extension \texttt{.t2x} by
analogy with the previous module, and a file of precalculated patterns
to accelerate the analysis of the input.  The binary file of the
bilingual dictionary is not included because it is not used.
\nota{Citar el compilador?}

The syntax of the specification file is very similar to that of the
\texttt{chunker}. Its DTD is specified in Appendix
\ref{ss:dtdinterchunk}, and the elements used to create the rules in
the file are described in Section \ref{formatotransfer}.


\paragraph{Pattern matching}

Rules detect patterns defined by sequences of lexical
pseudoforms. These lexical pseudoforms have a format based on the
format of lexical forms for words. In practice, a lexical pseudoform
is seen equivalently as \nota{mlforcada: La alternança
\emph{pseudolema} i \emph{pseudoparaula} s'ha de resoldre. MG: ho he
traduit tot com a 'lexical pseudoform', crec que era aquest el
sentit.} lexical forms are seen in the \texttt{chunker} regarding
pattern matching.  This way, pattern matching will be based on
attributes defined for lexical pseudoforms, not for lexical forms
(words) of the original pattern.

\paragraph{How it works}

With regard to the set of instructions used in \texttt{chunker}, the
changes on the set of instructions for this module are the following:

\begin{itemize}
\item The root element is called \texttt{<interchunk>} and does not
have any attribute.
\item The attribute \texttt{side} disappears: This module does not use
bilingual dictionaries; therefore, the attribute used to indicate
whether the consulted side is SL or TL looses sense. This attribute
was basically used in the \texttt{<out>} instructions.
\item The \texttt{<chunk>} tag is used here without attributes, simply
inside an \texttt{<out>} to delimit the output of chunks.
\item The predefined attribute \texttt{lem} refers to the pseudolemma
of the chunk. In the same way, the predefined attribute \texttt{tags}
refers to the grammatical symbols or tags of the chunk. The chunk
content becomes something like a queue which can be printed with the
implicit attribute \texttt{chcontent}.\nota{Només imprimir o s'hi pot
fer referència també?}  \nota{Dir de quin element són aquests
atributs}
\item All the values of \texttt{part}, except \texttt{chname}, access
the pseudolemma and the tags of the chunk (not of individual words).
\item Unlike what happens in the \texttt{chunker} module, in the rules
of this module it is not allowed to print anything else than
\texttt{<chunk>}s in the \texttt{<out>} instructions, in no case
isolated words.\nota{MG: and blanks too, right?}
\end{itemize}


\paragraph{Default action}

Like in the previous module, a default action has been defined, which
writes without modifications the chunks not matched by any pattern of
the specification file. This default action writes exactly what it
reads, be it chunks or blanks.  \nota{Atenció a la vacil·lació
\emph{regla}/\emph{acció} en la resta del document. Sempre havia
cregut que era \emph{regla}=\emph{patró}+\emph{acció}.}


\subsubsection{\texttt{Postchunk} module}
\label{ss:postchunk}

The \texttt{postchunk} module detects single chunks and, for each of
them, performs the specified actions. Detection is based on the lemma
of the chunk, and not in patterns (not in tags); this causes detection
in this module to be done specific for each ``name'' of
chunk.\nota{Quan fixem bé la terminologia hem d'assegurar-nos que la
redacció d'aquesta part és l'adequada.}


On the other hand, detection and processing in rules is based on the
fact that references to parameters are solved right after detection,
that is, the tags \texttt{<1>}, \texttt{<2>}, etc. are automatically
replaced by the value of the parameters before the processing
begins. Positions (attribute \texttt{pos}) specified in instructions
such as \texttt{<clip>}, refer to the position of the words inside the
chunk.

Also the case policy is automatically applied (see Section
\ref{ss:majuscules}) from the pseudolemma of the chunk to the words
inside the chunk.



\paragraph{Input/output}

\begin{itemize}
\item Input: chunks from the \texttt{in\-ter\-chunk}.
\item Output: valid input for the morphological generator of Apertium.
\end{itemize}

\paragraph{Data files}

This program has its own specification file, which will have the
extension \texttt{.t3x}. Its syntax is based as well on the
\texttt{chunker} and the \texttt{in\-ter\-chunk}.  \nota{Explicar que
no ha de llegir cap fitxer compilat de patrons perquè usa noms i no
patrons?}

\paragraph{Pattern matching}

Chunk matching is based on the name of the chunk. Unmatched chunks
receive the default processing.

\paragraph{How it works}

The differences with regard to the \texttt{in\-ter\-chunk} module are
the following:

\begin{itemize}
\item It is not allowed to write chunks (\texttt{<chunk>}) in the
  output: only lexical units (\texttt{<lu>} or \texttt{<mlu>}) and
  blanks can be written.  \nota{Comprovar aquest ítem perquè era
  incomplet i l'ha completat mlf}
\item New detection attribute \texttt{name} in \texttt{<cat-item>},
which is used in the \texttt{<pattern>} part of rules isolatedly, to
force pattern detection basing on its name.  \nota{mlf: Què vol dir
``de manera aïllada''? Sembla que vulga dir ``de tant en tant''. MG:
the attribute 'name' is used in the pattern part of rules? is this
correct?}
\item Also the attribute \texttt{side} is not used here, as in the
\texttt{in\-ter\-chunk}, for the same reason: the bilingual dictionary
is not looked up.  \nota{MG: però llavors això no és una diferència
respecte de \texttt{interchunk} no?}
\end{itemize}

\paragraph{Default action}


In this module, the default action is to write the words contained in
the chunks, replacing the references with the parameters of the
chunk. It will be applied to most chunks, since it is foreseen that
this module performs non-default actions only for specific cases
requiring some special processing.

Also the case policy is applied by default (see Section
\ref{ss:majuscules}).

In any case, blanks outside chunks are copied in the same order as are
read, since chunk matching is done individually (this module does
not group chunks).




\subsection{Preprocessing of the structural transfer module}
\label{ss:preproceso_transfer}

Specification files for the structural transfer modules, also called
\emph{transfer rules files}, are pre-processed by the program
\textit{apertium-preprocess-transfer}, which calculates the patterns
to match rules preconditions, and indexes the rules to speed up its
processsing during execution time.  This information is saved in a
binary file which is read together with the bilingual dictionary and
the rules file itself, because the structural transfer and lexical
transfer modules are executed together.


\section{De-formatter and re-formatter}
\label{se:desformat}


\subsection{Format processing}
\label{ss:formato}

This section describes how the de-formatter and re-formatter process
the format of the documents. These two modules are created from a set
of format specification rules in XML, which are described in Section
\ref{ss:reglasformato}.


Apertium can process documents in XML, HTML, RTF and plain text. For
all these document types, format is \textit{encapsulated} as explained
in the following lines.

Text strings that are identified as part of the format ---from now on
referred to as \textit{blocks of format} or \textit{superblanks}---
are encapsulated between delimiters that depend on the specification
of the data flow between modules (which is described in detail in
Section~\ref{se:flujodatos}); so, in the flow format (sections
\ref{se:noxml1} and \ref{se:noxml2}), \emph{superblanks} are put
between brackets '\texttt{[}' and '\texttt{]}'.  Each of these
encapsulated strings will be treated as it were a blank
\texttt{<\textbf{b}/>} (page~\pageref{s3:b}) ---that is why they are
called \textit{superblanks}--- and will be restored in the correct
order in the translator's output.

As has been explained in Section \ref{se:flujodatos}, when the blocks
of format are large (as is sometimes the case in HTML with Javascript
code fragments, or in RTF with bitmap images), these blocks will be
saved as temporary files so that they can be removed from the data
flow of the translation.

Sometimes, the format in a document can implicitly indicate the
division of the text into sentences (see page \pageref{finfrase} in
Section \ref{se:flujodatos}). For example, section or document titles
can be a sentence without full stop.  If we know that a format mark is
indicating this division, we have to take advantage of this
information in order to do a better translation.  Some examples of
format that give us data about the end of a sentence are: two
consecutive line breaks in plain text format, a \texttt{</h1>} tag in
HTML, etc. The de-formatter generates in such cases a mark of sentence
end that is equivalent to a full stop.

\subsubsection{Format encapsulation method}

The types of blocks of format or \emph{superblanks} that can be
generated as a result of the format processing are the following:

\begin{itemize}
\item \textit{Non-empty blocks of format or superblanks}.  They
contain exclusively format marks of the source document. In the data
flow described in Section~\ref{se:flujodatos} , they begin with a left
square bracket '\texttt{[}' and end with a right square bracket
'\texttt{]}'.
\item \textit{Blocks of format with reference to an external file} or
\textit{extensive superblanks}.  They encapsulate long format fragments
in a way that improves the translator's performance. In the data flow
described in Section~\ref{se:flujodatos}, they begin with the
characters '\texttt{[@}', then there is the name of the file where the
format fragment extracted from the source text is saved, and finally
they end with a right square bracket '\texttt{]}'.
\item \textit{Empty blocks of format}. They contain artificial
information on text division obtained from the format data.  Before
the empty block of format, the system places the appropriate
artificial punctuation mark.  When the original format is restored in
the document at the end of the process, the presence of a block of
format like this will cause the deletion of the character right before
the block in the data flow.
\end{itemize}

%% [movido al apéndice]
%% Dentro de los bloques de formato, los caracteres '\texttt{[}', '\texttt{]}',
%% '\texttt{@}' y '\verb!\!' se escapan mediante las secuencias de escape
%% '\verb!\[!', '\verb!\]!', '\verb!\@!' y '\verb!\\!', respectivamente.  Esto
%% hay que tenerlo en cuenta para encapsular y desencapsular.  En el exterior de
%% los bloques de formato es necesario también escapar los corchetes de apertura
%% y cierre. 

The general criteria applied to the creation of blocks of format are
the following:

\label{pg:criteri}
\begin{itemize}
\item Everything that is considered not to be part of the text to be
translated, has to be encapsulated in blocks of format.
\item There can not be two or more strictly consecutive non-empty
blocks of format.  Two consecutive blocks of format must be merged
into a single block.
\item Empty blocks of format must precede a non-empty block of format
or the end of the file.
\end{itemize}

Figure~\ref{fg:ejemplopelado} shows an example document the format of
which must be processed before translation; the encapsulation
corresponds to the flow format not based on
XML. Figure~\ref{fg:ejemploencapsulado} displays the result of
processing the mentioned document.



\begin{figure}[htbp]
\begin{small}
\begin{alltt} 
<html> 
<head> 
<title>This is the title</title> 
<script>
<!-- ... (an extensive code block) --> 
</script> 
</head> 
<body>
<p>This 
is a paragraph in two lines</p> 
</body> 
</html>
\end{alltt}
\end{small}
\caption{Example of HTML document}
\label{fg:ejemplopelado}
\end{figure}

\begin{figure}[htbp]
\begin{small}
\begin{alltt} 
\textbf{[<html> 
<head> 
<title>]}This is the title\textbf{.[][@/tmp/temp35345]}This\textbf{[ 
]}is a paragraph in two lines\textbf{.[][</p> 
</body> 
</html>]}
\end{alltt}
\end{small}
\caption{Example of HTML document where the blocks of format have been
  encapsulated by the de-formatter}\nota{repeteix coses capítol format
  -revisar -Gema}
\label{fg:ejemploencapsulado}
\end{figure}

 We would like to emphasize the following from this example:
\begin{itemize}
\item The system does not generate consecutive blocks of format with
content (non-empty).
\item Tags like \texttt{</\textbf{title}>} or \texttt{</\textbf{p}>}
cause the insertion of an artificial punctuation mark; this insertion
is done systematically, even when it is not necessary, because it does
not interfere and is efficient.
\item Extensive superblanks are literally removed from the translation
process. In this case, the temporary file \texttt{temp35345} contains
the tags from \texttt{</\textbf{title}>} to \texttt{<\textbf{p}>}
\item Simple blanks between words are not encapsulated.  But the
system does encapsulate multiple blanks (two or more consecutive
blanks), tabs, etc. Also line breaks are encapsulated.
\end{itemize}






\subsection{Data: format specification rules}
\label{ss:reglasformato} This section describes how the de-formatter
and re-formatter are generated from a format specification in XML.


Rules for format, like linguistic data, are specified in XML, and they
contain regular expressions with \texttt{flex} syntax.  The
specification is divided in three parts (see its DTD in the Appendix
\ref{ss:dtd_formato}):

\begin{itemize}
\item \textbf{Configuration options}. Here one specifies the value for
the maximum length of a non-extensive superblank, the input and output
encodings, whether case must be considered, and the regular expressions for
escape characters and space characters.

\item \textbf{Format rules}. Describes the set of tags belonging to a
specific format which have to be included in a block of format by the
de-formatter. These tags may, optionally, indicate a sentence end, in which case
the de-formatter will insert an artificial punctuation mark (followed
by an empty block of format, as explained in the previous
section). One has to specify the priority of application of the rules,
although, when this is not relevant, it is possible to give the same
priority to all the rules by assigning them the same value (any
number).
  
  Everything that is not specified as format will be left without
  encapsulation and, therefore, will be considered as translatable
  text.

\item \textbf{Replacement rules}. Allow to replace special characters
in the text. A regular expression will recognize \nota{MG: HELP: in
Spanish, "recogerá", I don't know how to translate this:
include/detect/group/recognize???} a set of special characters, and
will replace it with the specified characters.  For example, in HTML,
the characters specified in hexadecimal have to be replaced with the
corresponding entity or ASCII character. For example,
\texttt{cami\&oacute;n} corresponds to \texttt{camión}.
\end{itemize}

Rules are described in more detail next.
\begin{itemize}
\item Root of the specification file. The attribute \texttt{name}
contains the name of the format.
\begin{small}
\begin{alltt} 
<?xml version="1.0" encoding="ISO-8859-1"?> 
<format name="html"> 
  <options> 
  ...  
  </options>
  
  <rules> 
  ...  
  </rules> 
</format>
\end{alltt}
\end{small}

\end{itemize}

It has to include the options and rules, an example of which is
presented next:

\begin{itemize}

\item Options.
\begin{small}
\begin{alltt} 
  <options> 
    <largeblocks size="8192"/>
    <input encoding="ISO-8859-1"/>
    <output encoding="ISO-8859-1"/>
    <escape-chars regexp='[\verb!\![\verb!\!]^\$\verb!\!\verb!\!]'/>
    <space-chars regexp='[ \verb!\!n\verb!\!t\verb!\!r]'/>
    <case-sensitive value="no"/>
  </options>
\end{alltt}
\end{small}

\end{itemize}

The element \texttt{<largeblocks>} specifies the maximum length of a
non-extensive superblank, through the value of the attribute
\texttt{size}.  The elements \texttt{<input>} and \texttt{<output>}
specify the input and output encoding of the text, through the
attribute \texttt{encoding}.

The element \texttt{escape-chars} specifies, by means of a regular
expression declared in the value of the attribute \texttt{regexp},
which characters must be escaped with a backslash.  The element
\texttt{<space-chars>} specifies the set of characters that must be
considered as blanks.

Finally, the element \texttt{case-sensitive} specifies if case is
relevant in the specifications of format attributes in which regular
expressions are contained.


\begin{itemize}
\item Rules. There are format rules and replacement rules.
\begin{small}
\begin{alltt}
  <rules>
    <format-rule ... >
      ...
    </format-rule>
    ...
    
    <replacement-rule>
      ...
    </replacement-rule>
    ...
  </rules>
\end{alltt}
\end{small} The two types are described in the following points.

\item Format rules. The de-formatter will encapsulate in blocks of
format the tags indicated by these rules in the field
\texttt{regexp}. If they are begin and end tags, and everything
delimited by them is format, one has to specify a \texttt{regexp} both
for \texttt{begin} and for \texttt{end}:
\begin{small}
\begin{alltt} 
    <format-rule eos="no" priority="1">
      <begin regexp='"\verb!\!\&lt;!--"'/>
      <end regexp='"--\verb!\!\&gt;"'/>
    </format-rule>
\end{alltt}
\end{small} Otherwise only one \texttt{begin-end} element is used:
\begin{small}
\begin{alltt} 
    <format-rule eos="yes" priority="3">
      <begin-end regexp='"\&lt;"[/]?"li"[^\&gt;]*"\&gt;"'/>
    </format-rule>
\end{alltt}
\end{small}


Besides, in \texttt{priority} you have to specify a priority to tell
the system in which order the format rules must be applied (the
absolute value is not relevant, only the order resulting from the
values). In ``\texttt{eos}'' you indicate, with \texttt{yes} or
\texttt{no}, whether the block of format that contains the detected
pattern must be preceded by an artificial punctuation mark or
not.\footnote{In all these cases, the typical entities \texttt{\&lt;}
and \texttt{\&gt;} are used to represent the characters \texttt{<} and
\texttt{>} respectively.}

\item Replacement rules. Are used to replace special characters in the
text. The regular expression in the attribute \texttt{regexp} will
recognize \nota{idem: help in translation of "recogerá"} a set of
special characters and will replace them with the specified characters
in the text to be translated.  The correspondence between original and
replacement characters is stated in the attributes \texttt{source} and
\texttt{target} of the \texttt{replace} elements, which can be
multiple:
\begin{small}
\begin{alltt} 
    <replacement-rule regexp='"\&amp;"[^;]+;'>
      <replace source="\&amp;Agrave;" target="À"/>
      <replace source="\&amp;#192;" target="À"/>
      <replace source="\&amp;#xC0;" target="À"/>
      <replace source="\&amp;#xc0;" target="À"/>
      <replace source="\&amp;Aacute;" target="Á"/>
      <replace source="\&amp;#193;" target="Á"/>
      <replace source="\&amp;#xC1;" target="Á"/>
      <replace source="\&amp;#xc1;" target="Á"/>
      ...
    </replacement-rule>  
\end{alltt}
\end{small}
\item Regular expressions of \texttt{regexp} attributes. They have the
syntax used in \texttt{flex} \cite{lesk75tr}.
    
\end{itemize}

% DTD moguda a Apèndix


As example of a format specification, we will give that for HTML. The
explanation given in the following paragraphs can be followed looking
at Figure \ref{fg:formato-html}.


In the first place, we find the format rule that specifies in a
general way all the HTML tags: it considers as HTML tag everything
that begins with the sign \textbf{\texttt{<}} and ends with the sign
\textbf{\texttt{>}}. This rule has the lowest priority (4) so that the
more specific rules are applied preferentially.  But before
considering a tag in a general way by applying this rule, some of the
higher priority rules will be applied. In the case of HTML, the
highest priority is for comments \texttt{<!-- ... -->}.  The marks for
beginning and end \texttt{<script> </script>} and \texttt{<style>
</style>}, where everything included by them is considered to be
format, has priority 2.  Priority 3 is for tags that indicate end of
sentence (artificial punctuation), which are \texttt{</br>},
\texttt{</hr>}, \texttt{</p>}, etc.

Last of all are the replacement rules, which replace all the codes
that begin with \texttt{\&}, as specified in the regular
expression. Then, each one of the replacements is defined:
\texttt{\&Agrave}, as well as \texttt{\&\#192}, \texttt{\&\#xC0} and
\texttt{\&\#xc0} are replaced with \texttt{À}. The remaining special
characters are declared in the same way.



\begin{figure}[htbp]
\begin{small}
\begin{alltt} 
 <?xml version="1.0" encoding="ISO-8859-1"?>
 <format name="html">
   <options>
     <largeblocks size="8192"/>
     <input encoding="ISO-8859-1"/>
     <output encoding="ISO-8859-1"/>
     <escape-chars regexp='[\verb!\![\verb!\!]^\$\verb!\!\verb!\!]'/>
     <space-chars regexp='[ \verb!\! n\verb!\! t\verb!\! r]'/>
     <case-sensitive value="no"/>
   </options>
 
   <rules>
    <format-rule eos="no" priority="1">
       <begin regexp='"\&lt;!--"'/>
      <end regexp='"--\&gt;"'/>
    </format-rule>

    <format-rule eos="no" priority="2">
      <begin regexp='"\&lt;script"[^\&gt;]*"\&gt;"'/>
      <end regexp='"\&lt;/script"[^\&gt;]*"\&gt;"'/>
    </format-rule>
    <format-rule eos="no" priority="2">
      <begin regexp='"\&lt;style"[^\&gt;]*"\&gt;"'/>
      <end regexp='"\&lt;/style"[^\&gt;]*"\&gt;"'/>
    </format-rule>

    <format-rule eos="yes" priority="3">
      <begin-end regexp='"\&lt;"[/]?"br"[^\&gt;]*"\&gt;"'/>
    </format-rule>
    <!-- Here come more declarations of format-rule eos="yes"-->
    <!-- ...                                                -->

    <format-rule eos="no" priority="4">
      <begin-end regexp='"\&lt;"[a-zA-Z][^\&gt;]*"\&gt;"'/>
    </format-rule>

    <replacement-rule regexp='"\&amp;"[^;]+;'>
      <replace source="\&amp;Agrave;" target="À"/>
      <replace source="\&amp;#192;" target="À"/>
      <replace source="\&amp;#xC0;" target="À"/>
      <replace source="\&amp;#xc0;" target="À"/>
      <!-- Here come more replace elements                -->    
      <!-- ...                                              -->
    </replacement-rule>
  </rules>
</format>
\end{alltt}
\end{small}
\caption{Part of the rules definition for HTML format}
\label{fg:formato-html}
\end{figure}


\subsection{Generation of de-formatters and re-formatters}
\label{se:gendeformat}

To generate the de-formatter and re-formatter for a given format, the
XML rules that declare the format are applied a style sheet that
carries out the generation. This XSLT transformation produces a
\texttt{lex} \cite{lesk75tr} file that, once compiled, is the
executable of the de-formatter and the re-formatter for the specified
format.

Thanks to the general specification of formats described in this
chapter, it has been possible to define specifications for HTML, RTF
and plain text.  These specifications are in the \texttt{apertium}
package, in the respective files \texttt{html-format.xml},
\texttt{rtf-format.xml}, \texttt{txt-format.xml}.  In particular, it
is quite simple to define de-formatters and re-formatters for any XML
format.

\chapter{Installing and running the system}
\label{se:instalacion}


\section{System requirements}

The system where you want to install and run Apertium must have the
following programs installed:

\begin{itemize}
\item \texttt{libxml2} version 2.6.17 or later (on Ubuntu you may need
to install \texttt{libxml2-dev} too)

\item \texttt{xmllint} tool (usually comes with \texttt{libxml2}, but
may be an independent package on your system, i.e. Debian GNU-Linux)

\item \texttt{xsltproc} tool (non-PowerPC users); also comes with
\texttt{libxml2} but may also be an independent package in your
system, as happens with the \texttt{xmllint} tool

\item \texttt{sabcmd} tool (PowerPC users), provided by package
\texttt{sablotron}

\item flex 2.5.4 or earlier (in some distributions, flex-old package)
\item GNU \texttt{make}, \texttt{gcc} (\texttt{g++}), \texttt{bash}
shell

\end{itemize}

\section{Installing program packages}

To install the Apertium machine translation system programs and
libraries first you need to download (from
\url{http://sourceforge.net/projects/apertium}), compile and install
the latest version of the following packages, in the specified order:

\begin{enumerate}
\item \texttt{lttoolbox}
\item \texttt{apertium}
\end{enumerate}

The simplest way to compile each package is:

\begin{enumerate}
\item Go to the directory containing the package's source code and
type \texttt{./configure} to configure the package for your system.
If you're using csh on an old version of System V, you might need to
type \texttt{sh ./configure} instead to prevent \texttt{csh} (the
default shell in old System V) from trying to execute
\texttt{configure} itself. Running \texttt{configure} takes a
while. While running, it prints some messages telling which features
it is checking for.

\item Type \texttt{make} to compile the package

\item Type \texttt{make install} (possibly with root privileges) to
install the programs and any data files and documentation.

\item You can remove the program binaries and object files from the
  source code directory by typing \texttt{make clean}. To remove also
  the files that \texttt{configure} created (so you can compile the
  package for a different kind of computer), type \texttt{make
  distclean}. There is also a\\ \texttt{maintainer-clean} option in
  the Makefile, but that is intended mainly for the package's
  developers. If you use it, you may have to get all sorts of other
  programs in order to regenerate files that came with the
  distribution.
\end{enumerate}

If you don't have root privileges to install the programs in your
system, you can use the \texttt{-prefix} flag with the configure
script to install them at your user account. For example:

\begin{small}
\begin{alltt} 
  \verb!$! pwd 
  /home/me/lttoolbox-0.9.1 
  \verb!$! ./configure --prefix=/home/me/myinstall
\end{alltt}
\end{small}

Libraries will be installed in the \texttt{LIBDIR=\$prefix/lib}
directory. If no \texttt{-prefix} flag is specified with configure
script, LIBDIR will be \texttt{/usr/local/lib}.


If you find some error to link against installed libraries in a given
directory \texttt{LIBDIR}, you must either use libtool, and specify
the full pathname of the library, or use the \texttt{LIBDIR} flag
during linking and do at least one of the following:

\begin{itemize}

\item add \verb!LIBDIR! to the \verb!LD_LIBRARY_PATH! environment
variable during execution

\item add \verb!LIBDIR! to the \verb!LD_RUN_PATH! environment variable
during linking

\item use the \texttt{-Wl}, \texttt{--rpath -Wl}, \texttt{LIBDIR}
linker flag

\item have your system administrator add \texttt{LIBDIR} to
\texttt{/etc/ld.so.conf} and run \texttt{ldconfig}

\end{itemize}

See any operating system documentation about shared libraries for more
information, such as the \texttt{ld(1)} and \texttt{ld.so(8)} manual
pages.

\section{Installing data packages}

To install the linguistic data packages, follow these steps:

\begin{enumerate}

\item Download a data package
(\texttt{apertium-}$LANG_1$\texttt{-}$LANG_2$\texttt{-}$VERSION$\texttt{.tar.gz})
from Apertium's website in Sourceforge
(\url{http://apertium.sourceforge.net/}). For example, to get version
0.9 of the linguistic data for the Spanish--Catalan translator, you
need to download the package \texttt{apertium-es-ca-0.9.tar.gz}.

\item Unpack the tarball in any directory, go to this directory and
type \texttt{make} in the terminal. Wait while linguistic data are
compiled.


\end{enumerate}


\section{Using the translator}

There are Apertium versions that work both in Linux systems (always
more up-to-date) and in Windows systems.  The information in this
section is intended for Linux users.


To run the translator, you have to use the
\texttt{apertium-translator} tool referring to the directory where
linguistic data are saved, and specifying the translation direction
(\texttt{es-ca}, \texttt{ca-es}, \texttt{es-gl}, etc.), the file
format (\texttt{txt}, \texttt{html}, \texttt{rtf}), the name of the
file to be translated and the name of the output file. So, the command
structure is as follows:


\begin{small}
\begin{alltt} 
\$ apertium-translator <directory> <translation> <format> \\ 
                           < input_file > output_file
\end{alltt}
\end{small}


For example, if your directory is \texttt{/home/maria/apertium-es-ca},
you have to type the following to translate a file in \texttt{txt}
format from Spanish to Catalan:

\begin{small}
\begin{alltt} 
\$ apertium-translator /home/maria/apertium-es-ca es-ca \\txt <file_sp >file_ca
\end{alltt}
\end{small}

It is recommended to go to the directory where linguistic data are
saved, because this way you only need to type a dot to refer to the
current directory:

\begin{small}
\begin{alltt} 
\$ apertium-translator . es-ca txt <file_sp >file_ca
\end{alltt}
\end{small}

If no format is specified, the default format is \texttt{txt}. When
working with the \texttt{txt}, \texttt{html} and \texttt{rtf} formats,
unknown words are marked with an asterisk (*) and errors with a symbol
(@, \# or /); if you wish that neither unknown words nor errors are
marked, you have to add a \texttt{u} to the format name. Therefore,
the format options are the following:

\begin{itemize}
\item \texttt{txt} : Default option, text with marks for unknown words
and errors

\item \texttt{txtu} : text without marks for unknown words and errors

\item \texttt{html} : HTML with marks for unknown words and errors

\item \texttt{htmlu} : HTML without marks for unknown words and errors

\item \texttt{rtf} : RTF with marks for unknown words and errors

\item \texttt{rtfu} : RTF without marks for unknown words and errors

\end{itemize}

If you do not wish to translate a file but just a sentence or a
paragraph in the screen, you can run the \texttt{apertium-translator}
tool without specifying any file name. The command, if you are in the
directory where linguistic data are saved, would be the following:

\begin{small}
\begin{alltt} 
\$ apertium-translator . es-ca
\end{alltt}
\end{small}

Then, you have to type or paste the text you wish to translate (it can
contain line breaks). To get the translated version, press Ctrl +
D. The translation will be displayed on the screen.

A third way of translating with Apertium is using the \texttt{echo}
command to send text through the translator:

\begin{small}
\begin{alltt} 
\$ echo "text to be translated" | apertium-translator . es-ca
\end{alltt}
\end{small}



\chapter{Maintaining linguistic data}
\label{se:datosling}

\notavisible{Perhaps one could integrate material from Fran Tyers' howto as found in Apertium Wiki}
\section[Description of current data]{Description of linguistic data
currently available}

 At present, Apertium has linguistic data for three language pairs
 \nota{MG: This is old, needs UPDATING}: Spanish--Catalan and
 Spanish--Galician. The files containing the linguistic data are saved
 in a single directory: \texttt{apertium-es-ca} for the pair
 Spanish--Catalan and \texttt{apertium-es-gl} for the pair
 Spanish--Galician. The names of the files in this directory have the
 following structure:

\begin{itemize}\setlength{\itemsep}{-\parsep}
    \item \texttt{apertium-PAIR.LANG.dix} : monolingual dictionary for
    LANG.
    \item \texttt{apertium-PAIR.LANG1-LANG2.dix} :
    \texttt{LANG1-LANG2} bilingual dictionary.
    \item \texttt{apertium-PAIR.trules-LANG1-LANG2.xml} : structural
    transfer rules for the translation from \texttt{LANG1} to
    \texttt{LANG2} .
    \item \texttt{apertium-PAIR.LANG.tsx} : tagger definition file for
    \texttt{LANG}.
    \item \texttt{apertium-PAIR.post-LANG.dix} : Post-generation
    dictionary for \texttt{LANG} (applies when translating into
    \texttt{LANG}).
    \item directory \texttt{LANG-tagger-data} : contains data needed
    for the \texttt{LANG} tagger (corpora, etc.)

\end{itemize}

\texttt{apertium-PAIR} refers to the linguistic combination of the
translator. Its two possible values at the moment are
\texttt{apertium-es-ca} and \\ \texttt{apertium-es-gl}. According to
this structure, the Catalan monolingual dictionary is called
\texttt{apertium-es-ca.ca.dix}, the Spanish--Galician bilingual
dictionary is called \texttt{apertium-es-gl.es-gl.dix} and the
structural transfer rules file for the translation from Catalan into
Spanish is called \texttt{apertium-es-ca.trules-ca-es.xml}.


The linguistic data available (by January 2006) for the different
language pairs are summarized in the following table.
\begin{small}
\begin{center}
\begin{tabular}{|p{8cm}|p{5cm}|} \hline
\multicolumn{2}{|c|}{\textbf{Translator Apertium-es-ca}} \\ \hline
Spanish monolingual dictionary & 11.800 entries \\ Catalan monolingual
dictionary & 11.800 entries \\ Spanish--Catalan bilingual dictionary &
12.800 entries (correspondences \texttt{es-ca})\\ Structural transfer
rules from Spanish into Catalan & 44 rules \\ Structural transfer
rules from Catalan into Spanish & 58 rules \\ Spanish post-generation
dictionary & 25 entries and 5 paradigms\\ Catalan post-generation
dictionary & 16 entries and 57 paradigms\\ \hline
\multicolumn{2}{|c|}{\textbf{Translator Apertium-es-gl}} \\ \hline
Spanish monolingual dictionary & 9.000 entries \\ Galician monolingual
dictionary & 8.600 entries \\ Spanish--Galician bilingual dictionary &
8.500 entries (correspondences \texttt{es-gl})\\ Structural transfer
rules from Spanish into Galician & 46 rules \\ Structural transfer
rules from Galician into Spanish & 38 rules \\ Spanish post-generation
dictionary & 36 entries and 12 paradigms\\ Galician post-generation
dictionary & 74 entries and 48 paradigms\\ \hline
\end{tabular}
\end{center}
\end{small}


\section[Adding words to dictionaries]{Adding words to monolingual and
bilingual dictionaries}


When extending or adapting Apertium, the most likely operation that
will be performed will be to extend its dictionaries. In fact, it will
be far more common than adding transfer or post-generation rules.

We describe next the most important things one has to take into
account when adding new words to the translator. This information is
more general than the data provided in the section describing
dictionaries (chapter \ref{ss:diccionarios}), although we give here
some practical information that might be very useful to the users who
decide to make changes in the translator. 

IMPORTANT: Every time a set
of modifications is made to any of the dictionaries, the modules have
to be recompiled. Type \emph{make} in the directory where the linguistic data
are saved (apertium-es-ca, apertium-es-gl or what may be applicable)
so that the system generates the new binary files.

If you want to add a new word to Apertium, you need to add three
entries in the dictionaries. Suppose you are working with the
Spanish-Catalan pair.  In this case, you have to add:

\begin{enumerate}
\item an entry in the Spanish monolingual dictionary: so that the
translator can analyze ("understand") the word when it finds it in a
text, and generate it when translating this word into Spanish.

\item an entry in the bilingual dictionary: so that you can tell
Apertium how to translate this word from one language to the other.

\item an entry in the Catalan monolingual dictionary: so that the
translator can analyze ("understand") the word when it finds it in a
text, and generate it when translating this word into Catalan.
\end{enumerate}

You will need to go to the directory containing the XML dictionaries
(for the Spanish-Catalan pair, this is \texttt{apertium-es-ca}) and
open with a text editor or a specialized XML editor the three
dictionary files mentioned: \texttt{apertium-es-ca.es.dix},
\texttt{apertium-es-ca.es-ca.dix} and
\texttt{apertium-es-ca.ca.dix}. The entries you need to create in
these three dictionaries share a common structure.  \\

\textbf{Monolingual dictionary (Spanish)}


You may want, for example, to add the Spanish adjective
\emph{cósmico}, whose equivalent in Catalan is \emph{còsmic}. The
first step is to add this word to the Spanish monolingual dictionary.

You will see that a monolingual dictionary has basically two types of
data: \textbf{paradigms} (in the "\texttt{<pardefs>}" section of the
dictionary, each paradigm inside a \texttt{<pardef>} element) and
\textbf{word entries} (in the main (\texttt{<section>} of the
dictionary, each one inside an \texttt{<e>} element). Word entries
consist of a lemma (that is, the word as you would find it in a
typical paper dictionary) plus grammatical information; paradigms
contain the inflection data of all lemmas in the dictionary. You can
search a particular word by searching the string \texttt{lm="word"}
(\texttt{lm} meaning \emph{lemma}).  Bear in mind, however, that the
element \texttt{lm} is optional and some other dictionaries may not
contain it.

Look at the word entries in the Spanish monolingual dictionary, for
example at the entry for the adjective \emph{bonito}. You can find it
by searching \texttt{lm="bonito"}:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="bonito"> 
  <\textbf{i}>bonit</\textbf{i}>
  <\textbf{par} \textsl{n}="absolut/o__adj"/> 
</\textbf{e}>
\end{alltt}
\end{small}

To add a word, you will have to create an entry with the same
structure. The part between \texttt{<i>} and \texttt{</i>} contains
the prefix of the word that is common to all inflected forms, and the
element \texttt{<par>} refers to the inflection paradigm of this
word. Therefore, this entry means that the adjective \emph{bonito}
inflects like the adjective \emph{absoluto} and has the same
morphological analysis: the forms \emph{bonit\textbf{o}},
\emph{bonit\textbf{a}}, \emph{bonit\textbf{os}},
\emph{bonit\textbf{as}} are equivalent to the forms
\emph{absolut\textbf{o}}, \emph{absolut\textbf{a}},
\emph{absolut\textbf{os}}, \emph{absolut\textbf{as}} and have the
morphological analysis: \texttt{adj m sg}, \texttt{adj f sg},
\texttt{adj m pl} and \texttt{adj f pl} respectively.

Now, you have to decide which is the lexical category of the word you
want to add: the word \emph{cósmico} is an adjective, like
\emph{bonito}. Next, you have to find the appropriate paradigm for
this adjective. Is it the same as the one for \emph{bonito} and
\emph{absoluto}?  ¿Can you say \emph{cósmic\textbf{o}},
\emph{cósmic\textbf{a}}, \emph{cósmic\textbf{os}},
\emph{cósmic\textbf{as}}? The answer is yes, and, with all this
information, you can now create the correct entry:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="cósmico">
    <\textbf{i}>cósmic</\textbf{i}>
  <\textbf{par} \textsl{n}="absolut/o__adj"/>
</\textbf{e}>
\end{alltt}
\end{small}


If the word you want to add has a different paradigm, you have to find
it in the dictionary and assign it to the entry. You have two ways to
find the appropriate paradigm: looking in the \texttt{<pardefs>}
section of the dictionary, where all the paradigms are defined inside
a \texttt{<pardef>} element, or finding another word that you think
may already exist in the dictionary and that has the same inflection
paradigm as the one to be added. For example, if you want to add the
word \emph{genoma}, you need to find an appropriate paradigm for a
\textbf{noun} whose gender is masculine and forms the plural with the
addition of an \textbf{-s}. This will be the paradigm
"\texttt{abismo\_\_n}" in our present dictionaries. Therefore, the
entry for this new word would be:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="genoma">
    <\textbf{i}>genoma</\textbf{i}>
  <\textbf{par} \textsl{n}="abismo__n"/>
</\textbf{e}>
\end{alltt}
\end{small}

In exceptional cases you will need to create a new paradigm for a
certain word. You can look at the structure of other paradigms and
create one accordingly. For a more detailed description of paradigms
and word entries in the dictionaries, refer to section
\ref{ss:diccionarios}.  \\

\textbf{Monolingual dictionary (Catalan)}

Once you have added the word to one monolingual dictionary, you have
to do the same to the other monolingual dictionary of the translation
pair (in our example, the Catalan monolingual dictionary) using the
same structure. The result would be:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="còsmic">
    <\textbf{i}>còsmi</\textbf{i}>
  <\textbf{par} \textsl{n}="acadèmi/c__adj"/>
</\textbf{e}>
\end{alltt}
\end{small}

\textbf{Monolingual dictionary (Galician)}

In the case you are trying to improve the XML dictionaries for the
Spanish-Galician pair, you will need to go to the directory
\texttt{apertium-es-gl} and open with a text editor or a specialized
XML editor the three dictionary files \texttt{apertium-es-gl.es.dix},
\texttt{apertium-es-gl.es-gl.dix} and
\texttt{apertium-es-gl.gl.dix}. In that case, once you have added the
new Spanish word \emph{genoma} to the Spanish monolingual dictionary
(\texttt{apertium-es-gl.es.dix}), you have to add the equivalent
Galician word \emph{xenoma} to the Galician monolingual dictionary
(\texttt{apertium-es-gl.gl.dix}), that is:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="xenoma">
    <\textbf{i}>xenoma</\textbf{i}>
  <\textbf{par} \textsl{n}="Xulio__n"/>
</\textbf{e}>
\end{alltt}
\end{small}

\textbf{Bilingual dictionary}

The last step is to add the translation to the bilingual dictionary.

A bilingual dictionary does not usually have paradigms, only
lemmas. An entry contains only the lemma in both languages and the
first grammatical symbol (the lexical category) of each one. Entries
have a left side (\texttt{<l>}) and a right side (\texttt{<r>}), and
each language has always to be in the same position: in our system, it
has been agreed that Spanish occupies the left side, and Catalan,
Galician and Portuguese the right side.


With the addition of the lemma of both words, the system will
translate all their inflected forms (the grammatical symbols are
copied from the source language word to the target language
word). This will only work if the source language word and the target
language word are grammatically equivalent, that is, if they share
exactly the same grammatical symbols for all of their inflected
forms. This is the case with our example; therefore, the entry you
have to add to the bilingual dictionary is:


\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>cósmico<\textbf{s} \textsl{n}="adj"/></\textbf{l}>
    <\textbf{r}>còsmic<\textbf{s} \textsl{n}="adj"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

This entry will translate all the inflected forms, that is,
\texttt{adj m sg}, \texttt{adj f sg}, \texttt{adj m pl} and
\texttt{adj f pl}. It works for the translation in both directions:
from Spanish to Catalan and from Catalan to Spanish.

In the case of the Spanish-Galician pair, the following bilingual
entry in the Spanish-Galician bilingual dictionary
(\texttt{apertium-es-gl.es-gl.dix}) will translate all the inflected
forms for the equivalent words \emph{genoma}/\emph{xenoma} in both
directions:

\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>genoma<\textbf{s} \textsl{n}="n"/></\textbf{l}>
    <\textbf{r}>xenoma<\textbf{s} \textsl{n}="n"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

What to do if the word pair is not equivalent grammatically (their
grammatical symbols are not exactly the same)? In that case, you need
to specify all the grammatical symbols (in the same order as they are
specified in the monolingual dictionaries) until you reach the symbol
that differs between the source language word and the target language
word. For example, the Spanish noun \emph{limón} has masculine gender
and its equivalent in Catalan, \emph{llimona}, has feminine
gender. The entry in the bilingual dictionary must be as follows:

\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>limón<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/></\textbf{l}>
    <\textbf{r}>llimona<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}


A more difficult problem arises when two words have different
grammatical symbols and the grammatical information of the source
language word is not enough to determine the gender (masculine or
feminine) or the number (singular or plural) of the target language
word. Take for example the Spanish adjective \emph{canadiense}. Its
gender is masculine--feminine since it is invariable in gender, that
is, it can go both with masculine and feminine nouns (\emph{hombre
canadiense}, \emph{mujer canadiense}). In Catalan, on the other hand,
the adjective has a different inflection for the masculine and the
feminine (\emph{home canadenc}, \emph{dona canadenca}). Therefore,
when translating from Spanish to Catalan it is not possible to know,
without looking at the accompanying noun, whether the Spanish
adjective (\emph{mf}) has to be translated as a feminine or a
masculine adjective in Catalan. In that case, the symbol \texttt{GD}
(for "gender to be determined") is used instead of the gender
symbol. \label{GDND} The word's gender will be determined by the
structural transfer module, by means of a transfer rule (a rule that
detects the gender of the preceding noun in this particular
case). Therefore, \texttt{GD} must be used only when translating from
Spanish to Catalan, but not when translating from Catalan to Spanish,
as in Spanish the gender will always be \texttt{mf} regardless of the
gender of the original word.  In the bilingual dictionary you will
need to add, in this case, more than one entry with direction
indications, as you must specify in which translation direction the
gender remains undetermined. The entries for this adjective should be
as follows:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{r}="LR">
  <\textbf{p}>
    <\textbf{l}>canadiense<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>canadenc<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="GD"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>canadiense<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>canadenc<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="f"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>canadiense<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="mf"/></\textbf{l}>
    <\textbf{r}>canadenc<\textbf{s} \textsl{n}="adj"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

"\texttt{LR}" means \emph{left to right} and "\texttt{RL}",
\emph{right to left}. Since Spanish is on the left and Catalan on the
right, the adjective will be \texttt{GD} only when translating from
Spanish to Catalan (\texttt{LR}). For the translation \texttt{RL} you
need to create two entries, one for the adjective in feminine and
another one for the adjective in masculine.\footnote{You could also
group them using a small paradigm}

The same principle applies when it is not possible to determine the
number of the target word for the same reasons mentioned above. For
example, the Spanish noun \emph{rascacielos} ("skyscraper") is
invariable in number, that is, it can be singular as well as plural
(\emph{un rascacielos}, \emph{dos rascacielos}). In Catalan, on the
other hand, the noun has a different inflection for the singular and
for the plural (\emph{un gratacel}, \emph{dos gratacels}).  In this
case the symbol used is "\texttt{ND}" ("number to be determined") and
the entries should be like this:


\begin{small}
\begin{alltt}
<\textbf{e} \textsl{r}="LR">
  <\textbf{p}>
    <\textbf{l}>rascacielos<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sp"/></\textbf{l}>
    <\textbf{r}>gratacel<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="ND"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>rascacielos<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sp"/></\textbf{l}>
    <\textbf{r}>gratacel<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="pl"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>rascacielos<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sp"/></\textbf{l}>
    <\textbf{r}>gratacel<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/><\textbf{s} \textsl{n}="sg"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

For a more detailed description of this kind of entries, refer to
section~\pageref{ss:bil}.



\subsection{Adding direction restrictions}

In the previous example we have already seen the use of direction
restrictions for entries with undetermined gender or number
(\texttt{GD} or \texttt{ND}). These restrictions can also be used in
other cases.

It is important to note that the current version of Apertium can give
only a single equivalent for each source-language lexical form
\nota{NEEDS UPDATING, reference to lextor} (a lexical form is the
lemma plus its grammatical information), that is, no word-sense
disambiguation is performed.\footnote{The system performs only
part-of-speech disambiguation for homograph words, that is, for
ambiguous words that can be analyzed as more than one lexical form,
like \emph{vino} in Spanish, that can mean both "wine" and "he/she
came". This type of disambiguation is performed by the tagger.} When a
lexical form can be translated in two or more different ways, one has
to be chosen (the most general, the most frequent, etc.).  You can
tell Apertium that a certain word has to be analyzed ("understood")
but not generated, as it is not the translation of any word in the
other language.

Let's see this with an example. The Spanish noun \emph{muñeca} can be
translated in two different ways in Catalan depending on its meaning:
\emph{canell} ("wrist") or \emph{nina} ("doll"). The context decides
which translation is the correct one, but in its present state
Apertium can not make such a decision .\footnote{See Section
\ref{multi} on multiword units for ways to circumvent this problem.}
Therefore, you have to decide which word you want as an equivalent
when translating from Spanish to Catalan.  From Catalan to Spanish,
both words can be translated as \emph{muñeca} without any problem. You
have to specify all these circumstances in the dictionary entries
using direction restrictions (\texttt{LR} meaning "left to right",
that is, \texttt{es}--\texttt{ca}, and \texttt{RL} meaning "right to
left", that is, \texttt{ca}--\texttt{es}). If you decide to translate
\emph{muñeca} as \emph{canell} in all cases, the entries in the
bilingual dictionary shall be:


\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>muñeca<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/></\textbf{l}>
    <\textbf{r}>canell<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>

<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>muñeca<\textbf{s} \textsl{n}="n"/></\textbf{l}>
    <\textbf{r}>nina<\textbf{s} \textsl{n}="n"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

This means that translation directions will be:
\begin{small}
\begin{alltt} 
    muñeca --> canell
    muñeca <-- canell
    muñeca <-- nina 
\end{alltt}
\end{small}

(Note that that there is also a gender change in the case of
\emph{muñeca} (feminine) and \emph{canell} (masculine)).

It should be emphasized that a lemma can not have two translations in
the target language, because the system would give an error when
translating that lemma (see Section \ref{errores} "Detecting errors"
to see how to find and correct these and other types of errors). When
a word can be translated in two different ways in the target language
in all contexts, you need to choose one as the translation equivalent
and leave the other one as a lemma that can be analyzed but not
generated, using direction restrictions like in the previous
example. For example, the Catalan lemmas \emph{mot} and \emph{paraula}
can be both translated into Spanish as \emph{palabra} ("word") and the
entries in the bilingual dictionary should look like this:

\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>palabra<\textbf{s} \textsl{n}="n"/></\textbf{l}>
    <\textbf{r}>paraula<\textbf{s} \textsl{n}="n"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>

<\textbf{e} \textsl{r}="RL">
  <\textbf{p}>
    <\textbf{l}>palabra<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/></\textbf{l}>
    <\textbf{r}>mot<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

Therefore, for this lemmas the translation directions will be:
\begin{small}
\begin{alltt} 
    palabra --> paraula
    palabra <-- paraula
    palabra <-- mot 
\end{alltt}
\end{small}

One may have to specify restrictions regarding translation direction
also in monolingual dictionaries. For example, both Spanish forms
\emph{cantaran} and \emph{cantasen} should be analyzed as lemma
\emph{cantar}, verb, subjunctive imperfect, 3rd person plural, but
when generating Spanish text, one has to decide which one will be
generated. Monolingual dictionaries are read in two directions
depending on its purpose: for the analysis, the reading direction is
left to right; for the generation, right to left. Therefore, a word
that must be analyzed but not generated must have the restriction
\texttt{LR}, and a word that must be generated but not analyzed must
have the restriction \texttt{RL}.


The case of \emph{cantaran} or \emph{cantasen} must have already been
taken care of in inflection paradigms and it is unlikely to be a
problem for most people extending a dictionary. In some other cases it
can be necessary to introduce a restriction in the word entries of
monolingual dictionaries.

\subsection{Adding multiwords}
\label{multi}

It is possible to create entries consisting of two ore more words, if
these words are considered to build a single "translation unit".
These multiword units can also be useful when it comes to select the
correct equivalent for a word inside a fixed expression. For example,
the Spanish word \emph{dirección} may be translated into two Catalan
words: \emph{direcció} ("direction, management, directorate,
steering", etc.) and \emph{adreça} ("address"); including, for
example, frequent multiword units such as \emph{dirección general}
\(\to\) \emph{direcció general} ("general directorate") and
\emph{dirección postal} \(\to\) \emph{adreça postal} ("postal
address") may help get improved translations in some situations.

Multiword units can be classified basically into two categories:
multiwords with inner inflection and multiwords without inner
inflection.

\subsubsection{Multiwords without inner inflection}

They are just like the normal one-word entries, with the only
difference that you need to insert the element \texttt{<b>} (which
represents a blank) between the individual words that make up the
unit. Therefore, if you want to add, for example, the Spanish
multiword \emph{hoy en día} ("nowadays"), whose equivalent in Catalan
is \emph{avui dia}, the entries you need to add to the different
dictionaries are:

\begin{itemize}

\item Spanish monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="hoy en día">
  <\textbf{i}>hoy<\textbf{b}/>en<\textbf{b}/>día</\textbf{i}>
  <\textbf{par} \textsl{n}="ahora__adv"/> 
</\textbf{e}>
\end{alltt}
\end{small}

\item Catalan monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="avui dia">
  <\textbf{i}>avui<\textbf{b}/>dia</\textbf{i}> 
  <\textbf{par} \textsl{n}="ahir__adv"/> 
</\textbf{e}>
\end{alltt}
\end{small}

\item Spanish-Catalan bilingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>hoy<\textbf{b}/>en<\textbf{b}/>día<\textbf{s} \textsl{n}="adv"/></\textbf{l}>
    <\textbf{r}>avui<\textbf{b}/>dia<\textbf{s} \textsl{n}="adv"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\end{itemize}

For Spanish-Galician pair, if you want to add, for example, the
Spanish multiword \emph{manga por hombro} ("disarranged"), whose
equivalent in Galician is \emph{sen xeito nin modo}, the entries you
need to add are:

\begin{itemize}

\item Spanish monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="manga por hombro">
  <\textbf{i}>manga<\textbf{b}/>por<\textbf{b}/>hombro</\textbf{i}>
  <\textbf{par} \textsl{n}="ahora__adv"/> 
</\textbf{e}>
\end{alltt}
\end{small}

\item Galician monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="sen xeito nin modo">
  <\textbf{i}>sen<\textbf{b}/>xeito<\textbf{b}/>nin<\textbf{b}/>modo</\textbf{i}>
  <\textbf{par} \textsl{n}="Deo_gratias__adv"/> 
</\textbf{e}>
\end{alltt}
\end{small}

\item Spanish-Galician bilingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>manga<\textbf{b}/>por<\textbf{b}/>hombro<\textbf{s} \textsl{n}="adv"/></\textbf{l}>
    <\textbf{r}>sen<\textbf{b}/>xeito<\textbf{b}/>nin<\textbf{b}/>modo<\textbf{s} \textsl{n}="adv"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\end{itemize}

\subsubsection{Brief introduction to paradigms}

The paradigms of the previous examples, as adverbs do not inflect,
contain only the grammatical symbol of the lexical form, as you see in
this example:

\begin{small}
\begin{alltt}
<\textbf{pardef} \textsl{n}="ahora__adv">
  <\textbf{e}>
    <\textbf{p}>
      <\textbf{l}/>
      <\textbf{r}><\textbf{s} \textsl{n}="adv"/></\textbf{r}>
    </\textbf{p}>
  </\textbf{e}>
</\textbf{pardef}>
\end{alltt}
\end{small}

Paradigms are build like a lexical entry. We have seen so far lexical
entries where the common part of the lemma is put between \texttt{<i>}
\texttt{</i>}:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="cósmico"> 
  <\textbf{i}>cósmic</\textbf{i}>
  <\textbf{par} \textsl{n}="absolut/o__adj"/> 
</\textbf{e}>
\end{alltt}
\end{small}


But you can also express the same with a pair of strings: a left
string \texttt{<l>} and a right string \texttt{<r>} inside a
\texttt{<p>} element:

\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="cósmico">
  <\textbf{p}>
    <\textbf{l}>cósmic</\textbf{l}>
    <\textbf{r}>cósmic</\textbf{r}>
  </\textbf{p}>
  <\textbf{par} \textsl{n}="absolut/o__adj"/>
</\textbf{e}>
\end{alltt}
\end{small}


These two entries are equivalent. The use of the \texttt{<i>} element
helps get more simple and compact entries, and you can use it when the
left side and the right side of the string pair are identical. As has
been explained before, monolingual dictionaries are read \texttt{LR}
for the analysis of a text and \texttt{RL} for the
generation. Therefore, when there is some difference between the
analysed string and the generated string (not very usual) the entry
can not be written using the \texttt{<i>} element. This is what
happens in paradigms, where the left and right strings are never
identical, since the right side must contain the grammatical symbols
that will go through all the modules of the system.

\subsubsection{Multiwords with inner inflection}


They consist of a word that can inflect (typically a verb) followed by
one or more invariable words. For these entries you need to specify
the inflection paradigm just after the word that inflects. The
invariable part must be marked with the element \texttt{<g>} (for
\emph{group}) in the right side. The blanks between words are
indicated, like in the previous case, with the element
\texttt{<b>}. Look at the following example for the Spanish multiword
\emph{echar de menos} (to miss), translated into Catalan as
\emph{trobar a faltar}:

\begin{itemize}

\item Spanish monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="echar de menos">
    <\textbf{i}>ech</\textbf{i}>
    <\textbf{par} \textsl{n}="aspir/ar__vblex"/>
    <\textbf{p}>
      <\textbf{l}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{l}>
      <\textbf{r}><\textbf{g}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{g}></\textbf{r}>
    </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\item Catalan monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="trobar a faltar">
    <\textbf{i}>trob</\textbf{i}>
    <\textbf{par} \textsl{n}="abander/ar__vblex"/>
    <\textbf{p}>
      <\textbf{l}><\textbf{b}/>a<\textbf{b}/>faltar</\textbf{l}>
      <\textbf{r}><\textbf{g}><\textbf{b}/>a<\textbf{b}/>faltar</\textbf{g}></\textbf{r}>
    </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\item Spanish-Catalan bilingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>echar<\textbf{g}><\textbf{b}/>de<\textbf{b}/>menos</\textbf{g}><\textbf{s} \textsl{n}="vblex"/></\textbf{l}>
    <\textbf{r}>trobar<\textbf{g}><\textbf{b}/>a<\textbf{b}/>faltar</\textbf{g}><\textbf{s} \textsl{n}="vblex"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\end{itemize}


Note that the grammatical symbol is appended at the end, after the
group marked with the \texttt{<g>}.

It can be the case that a lemma is a multiword of this kind in one
language and a single word in the other language. In that case, in the
bilingual dictionary, the multiword will contain the \texttt{<g>}
element and the single word will not. In the monolingual dictionaries,
each entry will be created according to its type.  Look at the
following example for the Spanish multiword \emph{darse cuenta} (to
realize), translated into Catalan as the verb
\emph{adonar-se}:\footnote{The verb \emph{adonar-se} is considered a
simple word, since the incorporation of enclitic pronouns (such as
"-se") is treated inside the inflection paradigms of verbs (for all
the Romance languages of \emph{Apertium}); therefore, it is not
necessary to specify them in lexical entries. The correct placement of
clitic pronouns is one of the main reasons for using the
\texttt{<g>}... \texttt{</g>} labels around the invariable part of
multi-word verbs.}

\begin{itemize}

\item Spanish monolingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e} \textsl{lm}="darse cuenta">
    <\textbf{i}>d</\textbf{i}>
    <\textbf{par} \textsl{n}="d/ar__vblex"/>
    <\textbf{p}>
      <\textbf{l}><\textbf{b}/>cuenta</\textbf{l}>
      <\textbf{r}><\textbf{g}><\textbf{b}/>cuenta</\textbf{g}></\textbf{r}>
    </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\item Catalan monolingual dictionary:
\begin{small}
\begin{alltt} 
<\textbf{e} \textsl{lm}="adonar-se">
    <\textbf{i}>adon</\textbf{i}>
    <\textbf{par} \textsl{n}="abander/ar__vblex"/>
</\textbf{e}>
\end{alltt}
\end{small}

\item Spanish-Catalan bilingual dictionary:
\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>dar<\textbf{g}><\textbf{b}/>cuenta</\textbf{g}><\textbf{s} \textsl{n}="vblex"/></\textbf{l}>
    <\textbf{r}>adonar<\textbf{s} \textsl{n}="vblex"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\end{itemize}

The same principles and actions described for basic entries (gender
and number change, direction restrictions, etc.) apply to all kinds of
multiwords. For a more detailed description of multiword units, refer
to section~\ref{ss:multipalabras}.

\subsection{Consider contributing your improved lexical data}

If you have successfully added general-purpose lexical data to any of
the Apertium language pairs, please consider contributing it to the
project so that we can offer a better toolbox to the community.  You
can e-mail your data (in three XML files, one for each monolingual
dictionary and another one for the bilingual dictionary) to the
following addresses: \\

\begin{tabular}{ll} 
Spanish-Catalan data & Mireia Ginestí: \texttt{mginesti@dlsi.ua.es}\\ 
Spanish-Portuguese data & Carme Armentano: \texttt{carmentano@dlsi.ua.es}\footnote{The group at the
Universitat d'Alacant has also developed data for this language pair
outside the present project.}\\ 
Spanish-Galician data & Xavier Gómez-Guinovart: \texttt{xgg@uvigo.es}\\\\

\end{tabular}


If you believe you are going to contribute more heavily to the
project, you can join the development team through
www.sourceforge.net. If you do not have a Sourceforge account, please
create one; then write to Mikel L. Forcada (\texttt{mlf@ua.es}) or
Sergio Ortiz (\texttt{sortiz@dlsi.ua.es}), or to Xavier Gómez
Guinovart if you are interested in the Spanish-Galician language pair,
explaining briefly your motivations and background to join the
project.  The usual way to contribute is to use CVS; as a project
member, you will be able to commit your changes to dictionaries
directly.

The addition of simple lexical contributions will soon be made simpler
by means of web forms in
\url{http://xixona.dlsi.ua.es/prototype/webform/}, so that
contributors do not have to deal directly with XML.


You should be aware that the data you contribute to the project, once
added, will be freely distributed under the current license (GNU
General Public License or Creative Commons 2.5
attribution-sharealike-noncommercial, as indicated). Make sure the
data you contribute is not affected by any kind of license which may
be incompatible with the licenses used in this project. No kind of
agreement or contract is created between you and the developers. If
you have any doubt, or you plan to make a massive contribution,
contact Mikel L. Forcada.


\section[Adding structural transfer rules]{Adding structural transfer
(grammar) rules}

The content in this chapter partially repeats information already
presented in the chapter describing the structural transfer module
(Section \ref{ss:transfer}), although rules are described here in a
more general and practical way, aimed at those who wish a first
approach to them.

Structural transfer rules carry out transformations to the analysed
and disambiguated text, which are needed because of grammatical,
syntactical and lexical divergences between the two languages involved
(gender and number changes to ensure agreement in the target language,
word reorderings, changes in prepositions, etc.). The rules detect
patterns (sequences) of source text lexical forms and apply to them
the corresponding transformations.  The module detects the patterns in
a left-to-right, longest-match way; for example, the phrase \emph{the
big cat} will be detected and processed by the rule for
\emph{determiner}--\emph{adjective}--\emph{noun} and not by the rule
for \emph{determiner}--\emph{adjective}, since the first pattern is
longer. If two patterns have the same length, the rule that applies is
the one defined in the first place.

The structural transfer module (generated from the structural transfer
rules file) calls the lexical transfer module (generated from the
bilingual dictionary) all through the process to determine the target
language equivalents of the source language lexical forms.

The structural transfer rules are contained in a XML file, one for
each translation direction (for example, for the translation from
Spanish to Catalan, the file is
\texttt{apertium-es-ca.trules-es-ca.xml}). You need to edit this file
if you want to add or change transfer rules.

Rules have a \textbf{pattern} and an \textbf{action} part. The pattern
specifies which sequences of lexical forms have to be detected and
processed. The action describes the verifications and transformations
that need to be done on its constituents. Usual transformation
operations (such as gender and number agreement) are defined inside a
macroinstruction which is called inside the rule.  At the end of the
action part of the rule, the resulting lexical forms in the target
language are sent out so that they are processed by the next modules
in the translation system.

A transfer rules file contains four sections with definitions of
elements used in the rules, and a fifth section where the actual rules
are defined. The sections are the following:

\begin{itemize}

\item \texttt{<section-def-cats>}: This section contains the
  definition of the categories which are to be used in the rule
  patterns (that is, the type of lexical forms that will be detected
  by a certain rule). For the rule presented below, the categories
  \texttt{det} and \texttt{nom} (determiner and noun) need to be
  defined here. Categories are defined specifying the grammatical
  symbols that the lexical forms have. An asterisk indicates that one
  or more grammatical symbols follow the ones specified. The following
  is the definition of the category \texttt{det}, which groups
  determiners and predeterminers\footnote{such as in Spanish
  \emph{todo}, \emph{toda}, \emph{todos}, \emph{todas}} in the same
  category since they play the same role for transfer purposes:

\begin{small}
\begin{alltt} 
<\textbf{def-cat} \textsl{n}="det">
    <\textbf{cat-item} \textsl{tags}="det.*"/>
    <\textbf{cat-item} \textsl{tags}="predet.*"/>
</\textbf{def-cat}>
\end{alltt}
\end{small}

It is also possible to define as a category a certain lemma, like the
following for the preposition \texttt{en}:

\begin{small}
\begin{alltt} 
<\textbf{def-cat} \textsl{n}="en">
    <\textbf{cat-item} \textsl{lemma}="en" \textsl{tags}="pr"/>
</\textbf{def-cat}>
\end{alltt}
\end{small}


\item \texttt{<section-def-attrs>}: This section contains the
definition of the attributes that will be used inside of the rules, in
the action part. You need attributes for all the categories defined in
the previous section, if they are to be used in the action part of the
rule (to make verifications on them or to send them out at the end of
the rule), as well as for other attributes needed in the rule (such as
gender or number). Attributes have to be defined using their
corresponding grammatical symbols and can not have asterisks; its name
must be unique. The following are the definitions for the attributes
\texttt{a\_det} (for determiners) and \texttt{gen} (gender):

\begin{small}
\begin{alltt} 
<\textbf{def-attr} \textsl{n}="a_det">
    <\textbf{attr-item} \textsl{tags}="det.def"/>
    <\textbf{attr-item} \textsl{tags}="det.ind"/>
    <\textbf{attr-item} \textsl{tags}="det.dem"/>
    <\textbf{attr-item} \textsl{tags}="det.pos"/>
    <\textbf{attr-item} \textsl{tags}="predet"/>
</\textbf{def-attr}>

<\textbf{def-attr} \textsl{n}="gen">
    <\textbf{attr-item} \textsl{tags}="m"/>
    <\textbf{attr-item} \textsl{tags}="f"/>
    <\textbf{attr-item} \textsl{tags}="mf"/>
    <\textbf{attr-item} \textsl{tags}="nt"/>
    <\textbf{attr-item} \textsl{tags}="GD"/>
</\textbf{def-attr}>

\end{alltt}
\end{small}

\item \texttt{<section-def-vars>}: This section contains the
definition of the variables used in the rules.

\begin{small}
\begin{alltt} 
  <\textbf{def-var} \textsl{n}="interrogativa"/>
\end{alltt}
\end{small}

\item \texttt{<section-def-macros>}: Here the macroinstructions are
defined, which contain sequences of code that are frequently used in
the rules; this way, linguists do not need to write the same actions
repeatedly. There are, for example, macroinstructions for gender and
number agreement operations.

\item \texttt{<section-def-rules>}: This is the section where the
structural transfer rules are written.

\end{itemize}

The following is an example of a rule which detects the sequence
\emph{determiner--noun}:

\begin{small}
\begin{alltt}
<\textbf{rule}>
  <\textbf{pattern}>
    <\textbf{pattern-item} \textsl{n}="det"/>
    <\textbf{pattern-item} \textsl{n}="nom"/>
  <\textbf{/pattern}>
  <\textbf{action}>
    <\textbf{call-macro} \textsl{n}="f_concord2">
      <\textbf{with-param} \textsl{pos}="2"/>
      <\textbf{with-param} \textsl{pos}="1"/>
    </\textbf{call-macro}>
    <\textbf{out}>
      <\textbf{lu}>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="whole"/>
      </\textbf{lu}>
      <\textbf{b} \textsl{pos}="1"/>
      <\textbf{lu}>
        <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="whole"/>
      </\textbf{lu}>
    </\textbf{out}>
  </\textbf{action}>
</\textbf{rule}>
\end{alltt}
\end{small}

Part of the action performed on this pattern is specified inside the
macroinstruction \texttt{f\_concord2}, which is defined in the
\texttt{<section-def-macros>}. It performs gender and number agreement
operations: if there is a gender or number change between the source
language and the target language (in the noun), the determiner changes
its gender or number accordingly; furthermore, if gender or number are
undetermined (\texttt{GD} or \texttt{ND}\footnote{See pages
\pageref{pg:GD} or \pageref{GDND}}), the noun receives the correct
gender or number values from the preceding determiner. In the Apertium
es--ca, es--gl and es--pt systems, there are agreement
macroinstructions defined for one, two, three or four lexical units
(\texttt{f\_concord1}, \texttt{f\_concord2}, \texttt{f\_concord3},
\texttt{f\_concord4}). When calling the macroinstructions in a rule,
it must be specified which is the main lexical unit (the one which
most heavily determines the gender or number of the other lexical
units) and which other lexical units of the pattern have to be
included in the agreement operations, in order of importance. This is
done with the \texttt{<with-param pos=""/>} element. In the presented
rule, the main lexical unit is the noun (position "2" in the pattern)
and the second one is the determiner (positions "1" in the pattern).

After the pertinent actions, the resulting lexical forms are sent out,
inside the \texttt{<out>} element. Each lexical unit is defined with a
\texttt{<clip>}. Its attributes mean the following:

\begin{itemize}

\item [-]\texttt{pos}: refers to the position of the lexical form in
the pattern; \texttt{1} is the first lexical form (the determiner) and
\texttt{2} the second one (the noun).

\item [-]\texttt{side}: indicates if the lexical form is in the source
language (\texttt{sl}) or in the target language (\texttt{tl}).  Of
course, words are sent out always in the target language; source
language lexical forms may be needed inside of a rule, when testing
its attributes or characteristics.

\item [-]\texttt{part}: indicates which part of the lexical form is
referred to in the \texttt{clip}. You can use some predefined values:

\begin{itemize}

\item [-]\texttt{whole}: the whole lexical form (lemma and grammatical
symbols). Used only when sending out the lexical unit (inside an
\texttt{<out>} element).

\item [-]\texttt{lem}: the lemma of the lexical unit

\item [-]\texttt{lemh}: the head of the lemma of a multiword with
inner inflection (see Section \ref{multi} in this chapter, or
Section~\ref{ss:multipalabras} if you wish a more detailed
description)

\item [-]\texttt{lemq}: the queue of a lemma of a multiword with inner
inflection


\end{itemize}

Apart from these predefined values, you can use any of the attributes
defined in \texttt{<section-def-attrs>} (for example \texttt{gen} or
\texttt{a\_det}).

The values \texttt{lemh} and \texttt{lemq} are used when sending out
multiwords with inner inflection in order to place the head and the
queue of the lemma in the right position, since the previous module
moved the queue just after the lemma head for various reasons. In
practice, in our system, this means that you must use these values
instead of \texttt{whole} when sending out verbs. This is because, in
our dictionaries, multiwords with inner inflection are always verbs
\nota{NEEDS UPDATING}and, if you use the value \texttt{whole} when
sending them out, the multiword would not be well formed (the head and
the queue of the lemma would not have the correct position and the
multiword could not be generated by the generator).

\end{itemize}


Therefore, a rule that has a verb in its pattern must send the lexical
forms like in the following two examples:

\label{regla_verbo1}
\begin{small}
\begin{alltt}
<\textbf{rule}>
  <\textbf{pattern}>
    <\textbf{pattern-item} \textsl{n}="verb"/>
  <\textbf{/pattern}>
  <\textbf{action}>
    <\textbf{out}>
      <\textbf{lu}>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="lemh"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="a_verb"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="temps"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="persona"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="gen"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="nbr"/>
        <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="lemq"/>
      </\textbf{lu}>
    </\textbf{out}>
  </\textbf{action}>
</\textbf{rule}>
\end{alltt}
\end{small}


\label{regla_verbo2}
\begin{small}
\begin{alltt}
<\textbf{rule}>
  <\textbf{pattern}>
    <\textbf{pattern-item} \textsl{n}="verb"/>
    <\textbf{pattern-item} \textsl{n}="prnenc"/>
  <\textbf{/pattern}>
  <\textbf{action}>
    <\textbf{out}>
      <\textbf{mlu}>
        <\textbf{lu}>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="lemh"/>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="a_verb"/>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="temps"/>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="persona"/>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="nbr"/>
        </\textbf{lu}>
        <\textbf{lu}>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="lem"/>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="a_prnenc"/>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="persona"/>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="gen"/>
          <\textbf{clip} \textsl{pos}="2" \textsl{side}="tl" \textsl{part}="nbr"/>
          <\textbf{clip} \textsl{pos}="1" \textsl{side}="tl" \textsl{part}="lemq"/>
        </\textbf{lu}>
      </\textbf{mlu}>
    </\textbf{out}>
  </\textbf{action}>
</\textbf{rule}>
\end{alltt}
\end{small}


The first rule detects a verb and places the queue in the correct
place, after all the grammatical symbols. The lexical unit is sent
specifying the attributes separately: lemma head, lexical category
(verb), tense, person, gender (for the participles), number and lemma
queue.

The second rule detects a verb followed by an enclitic pronoun and
sends the two lexical forms specifying also the attributes separately;
the first lexical unit consists of: lemma head, lexical category
(verb), tense, person and number; the second lexical unit consists of:
lemma, lexical category (enclitic pronoun), person, gender, number and
lemma queue (of the first lexical form). This way, the queue of the
lemma is placed after the enclitic pronoun. The two lexical units
(verb and enclitic pronoun) are sent inside a \texttt{<mlu>} element,
since they have to reach the morphological generator as a multilexical
unit (multiword).


Taking into account what we have explained here, if you want to
\textbf{add a new transfer rule} you have to follow these steps:

\begin{enumerate}

\item Specify which pattern you want to detect. Bear in mind that
words are processed only once by a rule, and that rules are applied
left to right and choosing the longest match.  For example, imagine
you have in your transfer rules file only two rules, one for the
pattern \emph{determiner--noun} and one for the pattern
\emph{noun--adjective}.  The Spanish phrase \emph{el valle verde}
("the green valley") would be detected and processed by the first one,
not by the second. You will need to add a rule for the pattern
\emph{determiner - noun - adjective} if you wish that the three
lexical units are processed in the same pattern.

\item Describe the operations you want to perform on the pattern. In
the Apertium \texttt{es-ca}, \texttt{es-gl} and \texttt{es-pt}
systems, simple agreement operations (gender and number agreement) are
easy to perform in a rule by means of a macroinstruction. To perform
other operations, you will need to use more complicated elements; for
a more detailed description of the language used to create rules,
refer to the section \ref{formatotransfer}.

\item Send the lexical units of the pattern in the target language
inside an \texttt{<out>} element. Each lexical unit must be included
in a \texttt{<lu>} element. If two or more lexical units must be
generated as a multilexical unit (only for enclitic pronouns in the
present language pairs) , they must be grouped inside a \texttt{<mlu>}
element.

All the words that are detected by a rule (that are part of a pattern)
must be sent out at the end of the rule so that the next module (the
generator) receives them. If a lexical unit is detected by a pattern
and is not included in the \texttt{<out>} element, it will not be
generated.


\end{enumerate}


\section[Adding data for the part-of-speech tagger]{Adding data for
the lexical categorial disambiguator (part-of-speech tagger)}

The lexical categorial disambiguator takes the linguistic information
needed to disambiguate a text basically from two sources: a tagset
definition file and corpora. The tagset definition file is contained
in the linguistic data directory and its name has the structure
\texttt{apertium-PAIR.LANG.tsx}, whereas corpora information is
contained in the \texttt{LANG-tagger-data} directory included in the
previous directory.

The \emph{tagset definition file} contains the definition of the
coarse tags (or categories) used by the tagger when being trained and
when disambiguating a text, as well as tag co-occurrence restrictions
that help obtain better tag probabilities. In Section \ref{ss:tagger}
you can find a detailed description of its characteristics.

The \emph{corpora} that need to be in the \texttt{LANG-tagger-data}
directory are different depending on whether the tagger is trained in
a supervised way (with manually disambiguated text) or unsupervised
(without manually disambiguated text):

\begin{itemize}

\item to train the tagger in a supervised way you need the files
(examples from es-tagger-data): \texttt{es.tagged.txt},
\texttt{es.untagged}, \texttt{es.tagged}, \texttt{es.dic}.

\item to train the tagger in an unsupervised way you need the files
(examples from es-tagger-data): \texttt{es.crp.txt}, \texttt{es.crp},
\texttt{es.dic}

\end{itemize}

These files have the following characteristics:

\begin{itemize}

\item \texttt{es.tagged.txt}: A Spanish corpus in plain text format.
\item \texttt{es.untagged}: The corpus \texttt{es.tagged.txt}
morphologically analysed, which means, processed by the de-formatter
and the morphological analyser (automatically generated corpus).
\item \texttt{es.tagged}: The preceding corpus manually disambiguated.
\item \texttt{es.crp.txt}: A large corpus (hundreds of thousands of
words) used when training the tagger in an unsupervised way with
Baum-Welch reestimation.
\item \texttt{es.crp}: The preceding corpus processed consecutively by
the de-formatter and the morphological analyser (automatically
generated corpus).
\item \texttt{es.dic}: File created from the Spanish monolingual
dictionary \texttt{*.es.dix}, by means of the \texttt{lt-expand} and
\texttt{aper\-tium\--fil\-ter\--am\-biguity} tools, which expand the
dictionary and filter the ambiguity classes, so that the file contains
all the forms identified as different ambiguity classes by the tagger
defined with \texttt{*.es.tsx}; that is, which lexical categories can
be homographs (automatically generated corpus).
\end{itemize}

When downloading Apertium from Sourceforge
(\url{http://apertium.sourceforge.net/}), if the tagger has been
trained in a supervised way, it is probable that you get the files
needed for this kind of training, \texttt{es.tagged} and
\texttt{es.tagged.txt} (for Spanish). The other required files are
automatically generated when running the training.  If the tagger has
been trained in an unsupervised way, you will not get any corpus in
the download since the files required for this kind of training are
huge. If you wish to train the tagger with this method, you will need
to collect a large corpus and name it \texttt{es.crp.txt}. The other
required files are automatically generated when running the training.

Anyway, the Apertium translator comes with all the data required for a
good performance of the tagger. You don't need to train the tagger in
order to use Apertium. A retraining might be required in the case that
you have made really extensive changes to the dictionaries or you have
modified the tagset definition file.

Therefore, the tagger data can be modified in two ways:

\begin{enumerate}

\item Change the tagset definition file. You can add, change or delete
the coarse tags used by the tagger, if you think that a new category
could be useful for the disambiguation or that a certain category
should be modified to obtain better results. You can also add
restrictions (for example, you can forbid the sequence
determiner--determiner if this is an impossible combination in a given
language and can help in the disambiguation of certain homograph
words).

\item Modify the corpora used to train the tagger.  You can modify the
manually disambiguated text (\texttt{es.tagged} for Spanish) if you
think that certain tags have been wrongly selected. You can also add
sentences to this text (and to \texttt{es.tagged.txt}, used to automatically
generate the corpus \texttt{es.untagged}) in order to
add information to the tagger, since it is possible that certain
combinations are incorrectly disambiguated because the tagger has not
found them in the training corpora.


\end{enumerate}

There are two commands to run the training:

\begin{itemize}

\item to train in a supervised way, type, in the directory containing
the linguistic data (example for \emph{es}--\emph{ca}): \texttt{make
-f es-ca-supervised.make}


\item to train in an unsupervised way, type, in the directory
containing the linguistic data (example for \emph{es}--\emph{ca}):
\texttt{make -f es-ca-unsupervised.make}


\end{itemize}

In both cases, planned files will be automatically generated.


\section{Detecting errors}
\label{errores}


It is easy to make errors when adding new words or transfer rules to
the Apertium system.

On the one hand, it is possible that, when compiling the new files,
the system displays an error message. In this case, this is a formal
error (a missing XML tag, a tag that is not allowed in a certain
context, etc.).  You just have to go to the line number indicated by
the error message, correct the error and compile again. On the other
hand, there are other types of errors not detected when compiling, but
which can make the system mistranslate a word or give an
incomprehensible text string.  These are linguistic errors, which can
be detected and corrected with the tips given in this chapter. The
following information is for Linux users, since Apertium works for the
moment only in this operating system.\footnote{There are in
\url{http://apertium.org} experimental packages for Windows with fixed
linguistic data (non-modifiable binary files).}

\subsection{Adjusting error symbols}
\label{subsec:marcaserror}

When the system encounters a problem to translate any word of a source
language text, in the default mode the system outputs the problematic
word together with a symbol that indicates that an error has occurred.
The meaning of the different symbols is the following:



\begin{itemize}


\item '\verb!@!': The problem is in the lexical transfer module, which
can not translate the lexical form (the bilingual dictionary does not
contain it)

\item '\verb!#!': The problem has occurred in the generator, which can
not generate the surface form from the input lexical form (the
morphological dictionary does not contain it in the generation
direction)

\item '\verb!/!': This symbol separates two or more surface forms
delivered by the generator. The problem, therefore, is in the target
language monolingual dictionary, which has, in the generation
direction, two surface forms for a single lexical form, when it should
have only one.


\end{itemize}


The generation module has three modes, which enable us to decide how
errors will be displayed in the final output.  The three possible
parameters are:

\begin{itemize}

\item -n : error symbols and the unknown-word symbol will NOT be
displayed, and neither will any grammatical symbols

\item -g : error symbols and the unknown-word symbol will be displayed
(default mode)

\item -d : error symbols and the unknown-word symbol will be
displayed, as well as the grammatical symbols of the lexical forms
producing the error.


\end{itemize}


The preferable mode depends on the type of user and on the translation
purpose. The first option is the most suitable when the user does not
want that external signs interfere in the reading of the
translation. The second option is useful when the user wants the
system to show where there has been a problem in the translation
(errors or unknown words) in order to be able to post-edit it
easily. The third option is ideal for linguistic developers of
Apertium, since it displays all the linguistic information of the
forms that produced an error.

Taking advantage of the error symbols output by the system, it is
possible to carry out a thorough test of the dictionaries of a certain
language pair. This will enable you to detect and correct all its
errors. To learn how to do it, see Section \ref{integridad}.

\subsection{Output of the different Apertium modules}

Sometimes it is difficult to find the origin of an error. In such
cases, it is useful to see the output of each of the modules.  As all
the data processed by the system, from the original text to the
translated text, circulate between the eight modules of the system in
text format, it is possible to stop the text stream at any point to
know what is the input or the output of a certain module.

Using a pipeline structure and the \texttt{echo} or \texttt{cat}
commands, you can send a text through one or more modules to analyse
their output and detect the origin of the error. We describe next how
to do it. You have to move to the directory where the linguistic data
are saved and type the described commands.



\subsubsection{The morphological analyser output}

To know how a word is analyzed by the translator, type the following
in the terminal (example for the Catalan word \emph{sabates}):


\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin
\end{alltt}
\end{small}

You can replace \texttt{ca-es} with the translation direction you want
to test.

The output in Apertium should be:
\begin{small}
\begin{alltt} 
^sabates/sabata<n><f><pl>\$^./.<sent>\$[][]
\end{alltt}
\end{small}

The string structure is
\verb!^!\texttt{word/lemma<}\textsl{morphological
analysis}\texttt{>}\verb!$!. The \texttt{<sent>} tag is the analysis
of the full stop, as every sentence end is represented as a full stop
by the system, whether or not explicitly indicated in the sentence.

The analysis of an unknown word is (ignoring the full stop info):

\begin{small}
\begin{alltt}
^genoma/*genoma\$
\end{alltt}
\end{small}

\noindent and the analysis of an ambiguous word:

\begin{small}
\begin{alltt}
^casa/casa<n><f><sg>/casar<vblex><pri><p3><sg>/casar<vblex><imp><p2><sg>\$
\end{alltt}
\end{small}

Each lexical form (lemma plus morphological analysis) is presented as
a possible analysis of the word \emph{casa}.

\subsubsection{The tagger output}


To know the output of the tagger for a source language text, type the
following in the terminal (example for the Catalan-Spanish direction):

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob
\end{alltt}
\end{small}

The output will be:
\begin{small}
\begin{alltt} 
^sabata<n><f><pl>\$^./.<sent>\$[][]
\end{alltt}
\end{small}

The output for an ambiguous word will be like the one above, since the
tagger chooses one lexical form among all the
possibilities. Therefore, the output for \emph{casa} in Catalan will
be, for example (depending on the context):

\begin{small}
\begin{alltt} 
^casa<n><f><sg>\$^.<sent>\$[][]
\end{alltt}
\end{small}

\subsubsection{The \texttt{pretransfer} output}

This module applies some changes to multiwords (move the lemma queue
of a multiword with inner inflection just after the lemma head). To
know its output, type:

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob | apertium-pretransfer
\end{alltt}
\end{small}

Since \emph{sabates} is not a multiword, this module does not alter
its input.

\subsubsection{The structural and lexical transfer output}

To know how a word, phrase or sentence is translated into the target
language and processed by structural transfer rules, type the
following in the terminal:
\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob | apertium-pretransfer \\| ./ca-es.transfer ca-es.autobil.bin
\end{alltt}
\end{small}

The output for this word will be:

\begin{small}
\begin{alltt} 
^zapato<n><m><pl>\$^.<sent>\$[][]
\end{alltt}
\end{small}


Analysing how a word or phrase is output by this module can help you
detect errors in the bilingual dictionary or in the structural
transfer rules. Typical bilingual dictionary errors are: two
equivalents for the same source language lexical form, or wrong
assignment of grammatical symbols. Errors due to structural transfer
rules vary a lot depending on the actions performed by the rules.


\subsubsection{The morphological generator output}

To know how a word is generated by the system, type the following in
the terminal:

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob | apertium-pretransfer \\| ./ca-es.transfer ca-es.autobil.bin | ltproc -g ca-es.autogen.bin
\end{alltt}
\end{small}

With this command you can detect generation errors due to an incorrect
entry in the target language monolingual dictionary or to a divergence
between the output of the bilingual dictionary (the output of the
previous module) and the entry in the monolingual dictionary.

The correct output for the input \emph{sabates} would be:

\begin{small}
\begin{alltt} 
zapatos.[][]
\end{alltt}
\end{small}

There are in this step no grammatical symbols, and the word appears
inflected.

\subsubsection{The post-generator output}

It is not very usual to have errors due to the post-generator, because
of its generally small size and the fact that it is seldom changed
after adding usual combinations, but you can also test how a source
language text comes out of this module, by typing:

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob | apertium-pretransfer \\| ./ca-es.transfer ca-es.autobil.bin | ltproc -g ca-es.autogen.bin \\| ltproc -p es-ca.autopgen.bin
\end{alltt}
\end{small}

\subsubsection{The Apertium output}

You can put all the modules of the system in the pipeline structure
and see how a source language text goes through all the modules and
gets translated into the target language. You just have to add the
re-formatter to the previous command:

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-destxt | lt-proc ca-es.automorf.bin \\|apertium-tagger -g ca-es.prob | apertium-pretransfer \\| ./ca-es.transfer ca-es.autobil.bin | ltproc -g ca-es.autogen.bin \\| ltproc -p es-ca.autopgen.bin | apertium-retxt
\end{alltt}
\end{small}

This is the same as using the \texttt{apertium-translator} shell
script provided by the Apertium package:

\begin{small}
\begin{alltt} 
echo "sabates" | apertium-translator . ca-es
\end{alltt}
\end{small}

\noindent (The dot indicates the directory where the linguistic data
are saved, in this case the current directory).

Of course, instead of typing all the presented commands every time you
need to test a translation, you can create shell scripts for every
action and use them to test the output of each module.




\subsection{Error examples}


1) We can get the following kind of output in a translation:

\begin{small}
\begin{alltt} 
\$ echo "nord" | apertium-translator . ca-es 
\$ #norte<n><m><sg>
\end{alltt}
\end{small}

This means that the word was correctly translated by the bilingual
dictionary but that the system does not find it in the Spanish
morphological dictionary to generate it. The problem can be in the
morphological dictionary but can also be caused by an incorrect
bilingual entry, in which the grammatical symbols that the translated
word is assigned do not correspond with the grammatical symbols that
this word has in the morphological dictionary.

2) The following \texttt{es-ca} bilingual entry does not take into
account the gender change between \emph{adhesiu} (masculine) and
\emph{pegatina} (feminine), causing the translator to give an error:

\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>pegatina<\textbf{s} \textsl{n}="n"/></\textbf{l}>
    <\textbf{r}>adhesiu<\textbf{s} \textsl{n}="n"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

\begin{small}
\begin{alltt} 
\$ echo "adhesiu" | apertium-translator . ca-es 
\$ #pegatina<n><m><sg>
\end{alltt}
\end{small}

The correct entry should be:

\begin{small}
\begin{alltt}
<\textbf{e}>
  <\textbf{p}>
    <\textbf{l}>pegatina<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="f"/></\textbf{l}>
    <\textbf{r}>adhesiu<\textbf{s} \textsl{n}="n"/><\textbf{s} \textsl{n}="m"/></\textbf{r}>
  </\textbf{p}>
</\textbf{e}>
\end{alltt}
\end{small}

3) The following error is given when the source language lexical form
can not be found in the bilingual dictionary, either because there is not an entry for this
lemma or because the entry does not correspond with the grammatical
symbols received from the analyser:


\begin{small}
\begin{alltt} 
\$ echo "illot" | apertium-translator . ca-es 
\$ @illot<n><m><sg>
\end{alltt}
\end{small}

4) When a source language lexical form has two correspondences in the
bilingual dictionary, the translator output is like the following one:

\begin{small}
\begin{alltt} 
\$ echo "llavor" | apertium-translator . ca-es 
\$ #pepita<n>/semilla<n><m><sg>
\end{alltt}
\end{small}

The solution is to put a direction restriction in one of the bilingual
entries.


Some errors can be due to structural transfer rules. The way to solve
a problem whose origin we don't know, is to test the output of the
different modules to detect where the problem arises.

\subsection{Testing the integrity of the
dictionaries}\label{integridad}

It is highly advisable to test the integrity of our dictionaries from
time to time, especially if we changed them significantly --or if we
changed the transfer rules, because some errors can be due to its
application.

The test is carried out in one translation direction. For this reason,
for a given language pair, you will have to perform two tests, one in
each direction.

The steps you have to follow to perform the test are:

\begin{itemize}

\item expand the source language monolingual dictionary, using the
\texttt{lt-expand} tool, to obtain all the lexical forms (which are
the forms that appear on the right of the colon in the output file);

\item send these lexical forms (except those that are only generation
forms, which \texttt{lt-expand} will have marked with the symbol
'\texttt{<}' ) through all the system modules from pretransfer to the
generator;

\item Search in the result, the lexical forms marked with the symbols
'\texttt{\#}' , '\texttt{@}' or '\texttt{/}', which will be the error
forms (see Section~\ref{subsec:marcaserror}).


\end{itemize}




\section{Generating a new Apertium system from modified data}

If you make changes to any of the linguistic data files of Apertium
(dictionaries, transfer rules or tagger definition file), the changes
will not be applied until you recompile the modules. To do this, type
\texttt{make} in the directory where the linguistic data are saved so
that the system generates the new binary files.

If changes were made to the tagger definition file or to the corpora
used to train the tagger, you will need also to retrain the tagger: in
the same linguistic data directory, you have to type (example for the
Spanish tagger in the es-ca translator) \texttt{make -f
es-ca-unsupervised.make} for unsupervised training or \texttt{make -f
es-ca-supervised.make} for supervised training.

After compilation, \texttt{apertium-translator} will already use the
new data.

\newpage


\chapter{Data insertion web forms}



This chapter describes the dictionary maintaining system in Apertium
2.  It is organized in two sections.  Section \ref{ss:formadmin} gives
the necessary information to install and adjust the web application
for word insertion.  Section \ref{ss:formus} describes how to use the
tool to add linguistic data.


\section{Introduction}

Adding lemmas to the dictionaries of the different languages in
Apertium is a slow task if you do it by manually editing the XML
dictionaries; for this reason web forms have been created, which make
the word insertion task considerably easier and, furthermore, allow
the users to do it remotely from any computer with Internet access.

The tool consists of a set of forms written in \texttt{php} which can
be used from any Internet navigator, either locally in the same
computer where dictionaries are saved, or remotely.

\section{Installing and managing}
\label{ss:formadmin}

\subsection{Installing the tool}

The installation must be done in a Unix machine which has an Apache
web server with \texttt{php} installed. So, you will first need to
install the \texttt{php} server if it is not installed, and then
proceed to install the form tool.


To install the tool, download the package
\textit{`apertium-lexical-webform-0.9'} from the Apertium web page in
Sourceforge (\url{http://apertium.sourceforge.net/}) and unpack it in
the directory where you want to leave the tool.


\begin{alltt}
   # cd /path/to the /forms tar -xvzf
   # /path/apertium-lexical-webform-0.9.tar.gz
\end{alltt}

You must take into account that Apache only serves the pages that are
in the root directory that we configured. Therefore, the directory
where you place the forms must be a subdirectory inside the root
directory of the Apache server.

Next, you have to edit the configuration file, which you can find in
\textit{private/config.php}, and give the appropriate values to the
configuration variables:

\begin{itemize}
\item \texttt{\$anmor}: entire path of the morphological analyser
\texttt{lt-proc}.
\item \texttt{\$dicos\_path}: path to the directory where the final
dictionaries and the compiled binaries of each dictionary are
saved. This directory must contain a subdirectory for each dictionary
with which the form can work. The subdirectory name must have the
following structure: \texttt{paradigmes-ll-rr} , where \textit{ll} and
\textit{rr} are the initials of the language pair involved. Each
directory must contain the final dictionaries used by the machine
translation system and the corresponding compiled binaries.  These
directories can be replaced with symbolic links in the case that they
are located in a different place.
\item \texttt{\$usuaris\_professionals}: a list of the professional
users in the system that have permission to insert words in the form
dictionaries and to validate entries pending confirmation.
  
\item \texttt{\$mail}: E-mail address of the administrator of the
forms. When someone wants to register as a user, an e-mail will be
sent to this address.
\end{itemize}

Once the parameters of this file have been configured, the forms
server is already in use.


\subsection{Directory structure}

All the files required by the application are structured as follows:

\begin{itemize}
\item \texttt{/index.php:} displays the initial insertion form.  It
has a section for each language pair, where the user inserts the SL
lemma and the TL lemma and chooses the appropriate part of
speech. After pressing the \textit{'Go on'} button, the next page is
displayed, where the user has to select the appropriate inflection
paradigms for the SL lemma and the TL lemma.
\item \texttt{/dics:} directory that contains the dictionaries with
the entries inserted from the forms. It contains the files with the
entries from non-professional users (pending validation) and the
dictionaries with the \texttt{XML} entries from professional users.
\item \texttt{/private:} most modules used in the forms are saved
here. It contains also the directories with the definition of
paradigms for all the languages of the forms; these directories have
the name \texttt{paradigmes-ll-rr}, where \textit{ll} and \textit{rr}
are the initials of a given language pair. The order chosen for the
two languages, first \textit{ll} and then \textit{rr}, depends on the
order defined for entries in the bilingual dictionary. This directory
contains also the files that carry out the whole processing of the
words being inserted.  These files are:
\begin{itemize}
\item \texttt{resultado.php: } This \textit{php} is called when two
  words for any language pair are inserted from the module
  \textit{index.php}. Basically, what it does is to establish the
  language pair involved (\textit{\$LR} and \textit{\$RL}) and the
  part of speech of the words being inserted (\textit{\$tipus}). It is
  included in the \textit{selec.php} module, that is the next one
  called in the insertion process. In the case that the \textit{tipus}
  (\textit{type}) of the word being inserted is a multiword unit
  (\textit{Multi Word Verb}), then \textit{multip.php} is the module
  included and called instead of \textit{selec.php}. The \textit{Multi
  Word Verb} elements consist of a verb that can inflect followed by
  an invariable queue of one or more words (see Section
  \ref{ss:multipalabras} for a detailed description).
\item \texttt{selecc.php: } This module is in charge of the selection
of paradigms for the pair of words, the SL word and the TL word. It
displays a list of paradigms to be chosen from, which depends on the
part of speech of the entry being inserted. When a new paradigm is
selected for a lemma, it displays some examples of inflected forms of
the lemma according to the chosen paradigm. If the user accepts the
chosen paradigms, the module calls \textit{insertarPro.php} or
\textit{insertar.php} depending on whether the user is professional or
non-professional respectively.
\item \texttt{multip.php: } It has the same function as the
\textit{selecc.php} module but for multiword units. It uses the same
variables and performs the same actions, but in the examples
displayed, the verb is inflected and the words of the queue are added
after it. It works in an analogous way as the \textit{selecc.php}
module, whose detailed description can be found in Section
\ref{ss:fitxersphp}.
\item \texttt{valida.php: } This module is called when a professional
  user wants to validate words that are in the queue of entries
  pending validation. It consults the file of words to be validated
  reading them one by one; it takes the data of the entry in turn
  (\textit{LRlem, RLlem, paradigmaLR, paradigmaRL, LR, RL}, etc.) and
  calls \textit{selecc.php} to continue with the insertion process of
  that specific entry.
\item \texttt{insertarPro.php: } This module is called when the
paradigms for the SL word and the TL word have already been selected
(which was done in \textit{selecc.php}), and displays what the
resulting \texttt{XML} entries will look like for the three
dictionaries (SL monolingual, bilingual and TL monolingual) . From
this screen it is possible to directly modify the code, and finally to
accept the new entry or to cancel the operation.
\item \texttt{ins\_multip.php: } It has the same function as
\textit{insertarPro.php} but it is designed for multiword entries,
therefore, the entry is treated differently so that the inserted
\texttt{XML} code is correct.
\item \texttt{insertar.php: } This module is equivalent to
\textit{insertarPro.php} but for non-professional users. The actions
it performs are much more simple, since the module just adds the
lemmas and the paradigms selected by the non-professional user to the
file of words to be validated; they remain in this file until a
professional user validates them.
\item \texttt{verSemi.php: } This module displays the file of entries
inserted by non-professional users which are waiting for
validation. It is useful for professional users who, before starting
validating words, want to see which words are in the queue waiting for
validation. It can be called from a link displayed in the form
generated by \textit{selec.php}.
\item \texttt{paradigmas.xsl:} Style sheet used to generate the
paradigm files that are used by the form modules. It is used with the
specification of paradigms of a language written in \texttt{XML}
format. This question will be explained in more detail in Section \ref{paradigm}
\textit{Paradigm files}.
\item \texttt{creaparadigma.awk:} \texttt{awk} file used also to
generate the mentioned paradigm files.
\item \texttt{gen\_paradig.sh:} Script that can be used if we want to
generate automatically the paradigm files for all the language pairs
installed in our system.
\end{itemize}
\end{itemize}

In the next sections you will find a detailed description of the tasks
of each module.

\subsection{Php files}

\subsubsection{resultado.php}

Depending on the value of the variable \texttt{\$nomtrad} updated by
\textit{index.php}, the module assigns the appropriate values to
\texttt{\$LR} and \texttt{\$RL} (source language and target language
respectively). Then, according to the part of speech of the word being
inserted, the variable \$tipo is assigned the appropriate value, and
then \textit{selec.php} or \textit{multip.php} are called depending on
whether the word is a simple unit or a multiword unit.  \nota{MG:
``asignamos'' i ``llamamos'' no seria més aviat ``se asigna'' y ``se
llama''?}

\subsubsection{selecc.php}
\label{ss:fitxersphp}

The function of this module is the selection of a paradigm for the
words being inserted. The user will have to select a paradigm for the
SL word and another one for the TL word.

There are a group of variables which, depending on the part of speech
of the word, are assigned certain values that will be used at the end
\nota{MG: "que darrerament s'utilitzaran" vol dir 'que s'utilitzaran
al final'?}; these variables are:
\begin{itemize}
\item \texttt{cadFich:} part of speech of the lemma.
\item \texttt{show:} string displayed in the form that indicates the
part of speech of the word being inserted.
\item \texttt{tag:} string with the \texttt{XML} tag output by the
morphological analyser for this part of speech.
\item \texttt{tagout:} string with the \texttt{XML} code that shows
the part of speech of the word. This string will be used when building
the final \texttt{XML} entry that will be inserted in the dictionary.
\item \texttt{nota:} string with possible comments to be inserted in
the \texttt{XML} code of the entry.
\end{itemize} Forms work with 4 kinds of dictionaries:
\begin{itemize}
\item \textit{Semi-professional dictionaries}: They contain the words
inserted from the form by non-professional users and which are pending
validation. Their extension is "\textit{semi.dic}"
\item \textit{Form dictionaries}: They contain the words inserted from
the form by professional users, and also the ones that have been
validated from the semi-professional dictionaries. Their extension is
"\textit{webform}".
\item \textit{Final dictionaries}: The files with all the entries
written in \texttt{XML} code. These are the files finally used by the
translator after being compiled. Their extension is "\textit{dix}".
\item \textit{Final compiled dictionaries}: These are the compiled
final dictionaries, which can already be used by the binaries of the
translator. Their extension is "\textit{bin}"
\end{itemize}

All these dictionaries are used by the forms; there are variables that
contain the paths to them. Values are also assigned to variables that
manage the paths to the auxiliary and the configuration files:
\begin{itemize}
\item \texttt{path:} path to the temporary dictionaries.
\item \texttt{fich\_LR:} source language dictionary with the words
inserted from the form that are not yet in the final dictionary nor in
the compiled dictionary.
\item \texttt{fich\_RL:} target language dictionary with the words
  inserted from the form that are not yet in the final dictionary nor
  in the compiled dictionary.  \nota{MG: I don't like speaking of SL and
    TL dictionaries, entries are for both directions, I think this is
    confusing. It should be changed in the whole chapter.}
\item \texttt{fich\_LRRL:} bilingual dictionary with the words
inserted from the form that are not yet in the final dictionary nor in
the compiled dictionary.
\item \texttt{fich-semi:} entries inserted from the form by
non-professional users and which are pending validation.
\item \texttt{path\_paradigmasLR:} path to the files that contain the
inflection paradigms of the source language.
\item \texttt{path\_paradigmasRL:} path to the files that contain the
inflection paradigms of the target language.
\item \texttt{anmor:} path to the morphological analyser.
\item \texttt{aut\_LRRL:} path to the bilingual binary from source
language to target language.\nota{MG: the original said "binario
morfológico", I think it's an error, I wrote 'bilingual binary'}
\item \texttt{aut\_RLLR:} path to the bilingual binary from target
language to source language.\nota{MG: ídem ("bilingual").}
\end{itemize}

Then the html code is inserted with the operations to be performed
depending on the selected action. The actions performed by the module
are the following, in sequential order:

\begin{itemize}
\item Tests that the source language lemma being inserted is not
already in the dictionaries containing the words inserted from the
form. If \texttt{selecc.php} has been called from the word validation
screen (\texttt{valida.php}), then the module tests that the lemma is
not already in the file of words inserted by non-professional
users. It tests this also in the final dictionary.
\item Performs the same test for the target language.
\item Code is written to select translation direction restrictions.
\item A series of functions are defined, which will be used when
generating the examples for the lemmas after the selection of the
appropriate paradigm. These are:
  \begin{itemize}
  \item \texttt{esVocalFuerte}
  \item \texttt{esVocalDebil}
  \item \texttt{esVocal}
  \item \texttt{PosicioVocalTall}
  \end{itemize} These functions are described later in section
  \ref{insertarpro}.
\item The paradigm file is opened to display a drop-down box with the
paradigms that can be selected for the source language lemma. To do
this, the program has to test sequentially the paradigms defined for
the part of speech of the lemma, checking whether the paradigm can be
applied to the lemma in question.
\item Then the same is done with the paradigms for the target language
lemma.
\item After the lemmas and the corresponding paradigms have been
  selected, examples must be generated to show how these lemmas would
  be inflected according to the selected paradigms. To do this, we
  need the root of the lemma (\texttt{raiz\_LR and raiz\_RL}), as well
  as the example endings for the selected paradigm
  (\texttt{paradigma\_LR and paradigma\_RL}); these endings are
  obtained from the paradigm file. Finally, a string is build
  containing the generated examples (\texttt{ejemplos\_LR and
  ejemplos\_RL}), and these are displayed.
\item If we arrived to this screen because we were validating words
(\texttt{va\-li\-da=1}), then a button is added to the form, which
allows us to delete the current entry if we decide not to validate it.
\item If the user that arrived to this screen is a professional user,
then a button is added to the form, which allows the user to select
the option for the validation of words entered by non-professional
users.
\item Finally, after one of the action buttons located at the bottom
of the form is pressed, the applicable actions are performed. If the
chosen action is \textit{"Delete"}, which can only be the case if the
user is validating entries, the current entry is deleted from the file
of entries made by non-professional users.  If the chosen action is a
confirmation (\textit{"Go on"} button), the module
\texttt{insertarPro.php} or \texttt{insertar.php} is called, depending
on whether the user is professional or non-professional respectively.
These modules are in charge of inserting the words in the
dictionaries.
\end{itemize} After the entry has been inserted, the page
\texttt{va\-li\-dar.php} or the page \texttt{selecc.php} are displayed
again, depending on whether the user was doing a validation process
(and then \textit{valida=1}) or a normal insertion.

\subsubsection{multip.php}

The code and behaviour of this module is the same as
\textit{selecc.php}.  The only difference is that this module is
designed for managing multiword units, whereas \textit{selec.php}
manages the rest of units. Therefore, the main difference is the
existence of the variables \texttt{\$LRcua} and \texttt{\$RLcua},
which contain the invariable queue that comes after the variable part
of a multiword. When the examples are displayed, besides showing the
variable part inflected according to the selected paradigm, also and
editable text box is displayed with the invariable queue.

When the button to continue with the insertion of the entry in the
dictionaries is pressed, the module \textit{ins\_multip.php} is called
instead of \textit{insertarPro.php}.


\subsubsection{valida.php}

This module is called when a professional user presses the button
"\textit{validate pairs}". It reads the dictionary of entries pending
validation (\$fichSemi) for the applicable language pair. Then, the
module enters a loop that goes through this file and reads the entries
one by one. With the information of a given entry, it assigns values
to a set of variables that will be used in the modules that will
perform the subsequent actions. These variables are, for example:
\begin{center} 
% use packages: array
\begin{tabular}{ll} 
\$LRlem & \$RLlem \\ 
\$paradigmaLR & \$paradigmaRL \\ 
\$direccions & \$tipo \\ 
\$comentarios & \$user \\ 
\$geneLR & \$geneRL \\ 
\$numLR & \$numRL \\ 
\$LR & \$RL
\end{tabular}
\end{center}

Once the appropriate values for these variables have been established,
the module \textit{selec.php} comes into action and treats the entries
as if they were made by a professional user. After inserting the
entries in the dictionaries by means of \textit{insertarPro.php}, the
flow returns to \textit{valida.php}, which proceeds to the next entry
to be validated.

\subsubsection{insertarPro.php}
\label{insertarpro}

After the lemmas have been entered and their paradigms selected in
\textit{selec.php}, this is the module that generates the
corresponding \texttt{XML} entries and inserts them in the monolingual
dictionaries and the bilingual dictionary.

It performs many operations similar to those performed in
\textit{selec.php}, such as generating the examples for the inflected
word. Thus, firstly, it gives values to \texttt{cadFich, show, tag,
tagout, nota} depending on the part of speech (\texttt{\$tipus}) of
the word being inserted.  It assigns paths to the file location
variables and defines some required functions as occurred in
\textit{selec.php}.
\begin{itemize}
\item \texttt{esVocalFuerte}: Returns \textit{true} if the vowel is
strong, that is, \textit{a, e, o}.
\item \texttt{esVocalDebil}: Returns \textit{true} if the vowel is
weak, that is \textit{i, u}.
\item \texttt{esVocal}: Returns \textit{true} if the character passed
as an argument is a vowel.
\item \texttt{diptongo}: Returns \textit{true} if the two letters
passed as an argument make a diphthong. This will be the case when at
least one of the two vowels is not strong.
\item \texttt{acentuar}: It receives a text string and accentuates it
according to the Spanish accentuation rules, depending on the
parameter \textit{\$siguienteletra}. \nota{MG: only for Spanish?}
\item \texttt{esMayuscula}: Returns \textit{true} if the character is
in upper case.
\item \texttt{TieneAcento}: Returns \textit{true} if the string has an
accent.
\item \texttt{acentua}: Accentuates the last accentuable vowel of a
word with an open or closed accent, depending on the direction
specified in the parameter \$sentit.\nota{MG: then not only for
Spanish but also for Catalan or Occitan?}
\item \texttt{PonQuitaAcento}: Inserts or removes the accent of the
first string passed as an argument depending on whether the second
string passed as an argument has an accent or not.
\item \texttt{PosicioVocalTall}: Returns the position in the lemma
(\$lema) for the vowel (\$vocal) that separates the root from the
ending. The vowel is searched from the end to the beginning and the
first occurrence of \$vocal is returned.
\end{itemize}

Now, the same operations as in \textit{selec.php} are
performed. Firstly, it makes sure that the entry is not yet in the
dictionaries, and then generates the examples of the word inflected
according to the paradigm previously selected. After this, it builds
the string with the \texttt{XML} code that is going to be inserted in
the source language monolingual dictionary. With the information on
the lemmas entered in \textit{selec.php}, a text string is generated
(\texttt{\$cad\_LR}) that contains the \texttt{XML} code for the
monolingual dictionary. This string is displayed in a text box that
can be manually edited. The same process is done to generate the
string for the target language monolingual dictionary
(\texttt{\$cad\_RL}) and for the bilingual dictionary
(\texttt{\$cad\_bil}). Then, the
possible comments and the name of the user making the entry are
concatenated to these variables, if applicable.  Finally, the form
screen is completed adding the buttons for accepting, deleting and going
back.  The code to process each one of the possible actions is at the
end of the file:
\begin{itemize}
\item \texttt{Insert: } In this case, it makes some character
  replacements so that the entry has the right format in the
  dictionaries, and inserts the strings \texttt{\$cad\_LR, \$cad\_bil,
  \$cad\_RL} in the source monolingual, bilingual and target
  monolingual dictionaries respectively (\texttt{\$fich\_LR,
  \$fich\_LRRL, \$fich\_RL}). If some error occurs when inserting the
  entry, a warning message is displayed. If \textit{insertarPro.php}
  was called from a word validation process (\textit{\$valida=1}),
  then the button "\textit{Continue}" is inserted to continue with the
  validation. If this is not the case, then a button to close the
  window is inserted, to allow the user to end the process.
\item \texttt{Delete: } It deletes the entry from the file of entries
pending validation.
\end{itemize}

\subsubsection{ins\_multip.php}

It performs the same actions as \textit{insertarPro.php} but it is
intended for multiword units. The main difference is the existence of
two additional variables, \texttt{\$LRcua} and \texttt{\$RLcua}, that
contain the invariable part of a multiword. When the entry is added to
the dictionaries, this queue has to be inserted in the right place and
the blanks have to be turned into \texttt{<b/>} tags.

\subsubsection{insertar.php}

The function of this module is very simple. It builds a text string
with the information provided by \textit{selec.php} separated by
tabs. This string contains all the required information to generate a
dictionary entry:

\texttt{\$LRlem.\$RLlem.\$paradigmaLR.\$direccion.\$paradigmaRL.}


\texttt{\$tipo.\$comentarios.\$user.\$geneLR.\$geneRL.}

 

This entry is saved in a file (\$fichSemi) that contains the queue
with the entries waiting for validation inserted by non-professional
users. When a professional user wishes to validate pending entries,
the \textit{valida.php} module will read from this file.


\subsubsection{verSemi.php}

It displays the file of entries waiting for validation, in this way:
it reads the file containing the entries (\textit{\$fichSemi}) and
enters a loop that reads all the entries of the file. For each entry,
it displays a line with the following information:

\texttt{\$LRlem
\$paradigmaLR
\$direccion
\$RLlem}

\texttt{\$paradigmaRL
\$tipo
\$comentarios}

\subsection{Dictionary files}

The files containing the entries inserted from the form are saved in
\texttt{/dics}. There are here two kinds of files:

\begin{itemize}
\item \texttt{apertium-ll-rr.xx.webform}: This is the file that
contains the entries in \texttt{XML} code, ready to be copied to the
final dictionaries. The name of the file has the presented structure,
where \texttt{ll-rr} are the initials of the language pair of the
translator and \texttt{xx} the initials of the language of the
monolingual dictionary or the languages of the bilingual dictionary
referred to, as applicable. For example, the initials of the
Spanish-Catalan translator are \texttt{es-ca}. For this translator, we
have the Spanish monolingual (\texttt{es}), the Catalan monolingual
(\texttt{ca}) and the bilingual (\texttt{es-ca})
dictionaries. Therefore, this directory will contain the following
files for the Spanish-Catalan translator:
\begin{center} \texttt{apertium-es-ca.es.webform
apertium-es-ca.ca.webform apertium-es-ca.es-ca.webform}
\end{center}


\item \texttt{oo-mm.semi.dic}: This is the file containing the entries
pending validation for a given language pair. \texttt{oo-mm} are the
initials of the pair. For example, for the Spanish-Catalan translator
this file would be: \texttt{es-ca.semi.dic}


\end{itemize}

\subsection{Paradigm files}
\label{paradigm}

The paradigms used for each language pair are specified in two
\texttt{XML} files named \texttt{paradig.ll-rr.xx.xml}, where
\texttt{xx} are the initials of the language and \texttt{ll-rr} the
initials of the language pair. These files consist of a set of entries
describing the paradigms or inflection models for the words of a given
language. The \texttt{XML} file has the following parts:
\begin{itemize}
\item Head/root of the specification file.\\
\begin{alltt} 
<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="paradigmas.xsl"?>
<!DOCTYPE form SYSTEM "form.dtd">
<form lang="oc" langpair="oc-ca">
\end{alltt} 
The \textit{lang} attribute states the initials of the
language for which paradigms are specified, and the \textit{langpair}
attribute states the initials of the language pair of the translator
for which the specification is made. It is required that the same
directory containing the paradigm files contains the \texttt{form.dtd}
file, which is the DTD specifying these files. You can find this DTD
in the Appendix \ref{ss:dtdparadigmes}.
\item A set of elements that define the paradigms. To explain its
format, we reproduce the following example: \\
\begin{alltt}
<entry PoS="adj" nbr="sg_pl" gen="mf">
        <endings>
                <stem>amable</stem> 
                <ending/>
                <ending>s</ending>  
        </endings>
        <paradigms howmany="1">
                <par n="amable\_\_adj"/>
        </paradigms>
</entry>
\end{alltt} 
Each paradigm is specified in a \texttt{<entry>} element.
This element can have three attributes:
  \begin{itemize}
  \item \textit{PoS}: the part of speech of the paradigm. It can take
  the values: acr, adj, adv, noun, pname, pr, verbo. \nota{also
  cnjadv?} It is mandatory for any part of speech.
  \item \textit{nbr}: the numbers admitted by the paradigm. It can
  take the values: sg, pl, sg\_pl, sp.
  \item \textit{gen}: the genders admitted by the paradigm. It can
  take the values: m, f, m f, mf.
  \end {itemize} It has two more elements:
  \begin{itemize}
  \item \texttt{endings}: the root and the endings used to select the
  paradigm in the form and display the inflection examples.
    
  \item \texttt{paradigms}: specification of the paradigm/s that
  define the inflection of an entry.  It requires the attribute
  \textit{howmany} , which specifies the number of paradigms used by
  an entry. Each used paradigm is indicated in a line, where the name
  of the paradigm in the dictionary is inserted according to this
  format:
    \begin{center}
\begin{alltt} 
<par n="long\_\_adj"/>
\end{alltt}
    \end{center}
  \end{itemize}
\end{itemize}

From the \texttt{XML} paradigm file, it is necessary to generate the
files directly used by the modules of the forms. Running the script
\texttt{/private/gen\_paradig.sh}, the process is automatically done
for all the available language pairs:
\begin{alltt}
   #  cd private 
   # ./gen\_paradig.sh
\end{alltt} 
To add a new paradigm to the forms, an appropriate entry
has to be added to the \texttt{XML} paradigm file, and then run the
previous script to update the working files.

The automatic process can also be done manually if we do not want to
update the files for all the installed language pairs. The manual
generation of the working files has to be done with a \texttt{XSL}
style sheet using the following command:
\begin{alltt}
   # xsltproc paradigmas.xsl paradigm\_file.xml
                                     | ./creaparadig.awk
\end{alltt}

This action generates a working file for each part of speech. The
generated files are saved in the directories
\texttt{/private/paradigmas.ll-rr}.  These directories contain the
files with the paradigms that can be used for each language pair
\texttt{ll-rr} and for each part of speech.  Each one of these
directories contain the following files:
\begin{itemize}
\item \texttt{paradigacr\_xx}: paradigms for acronyms in the language
\texttt{xx}.
\item \texttt{paradigadj\_xx}: paradigms for adjectives in the
language \texttt{xx}.
\item \texttt{paradigadv\_xx}: paradigms for adverbs in the language
\texttt{xx}.
\item \texttt{paradigcnjadv\_xx}: paradigms for adverbial conjunctions
in the language \texttt{xx}.
\item \texttt{paradigcnjcoo\_xx}: paradigms for copulative
conjunctions in the language \texttt{xx}.\nota{MG: aquesta no està en
la pàgina web del formulari}
\item \texttt{paradigcnjsub\_xx}: paradigms for subordinating
conjunctions in the language \texttt{xx}.\nota{ídem}
\item \texttt{paradignoun\_xx}: paradigms for nouns in the language
\texttt{xx}.
\item \texttt{paradigpname\_xx}: paradigms for proper nouns in the
language \texttt{xx}.
\item \texttt{paradigpr\_xx}: paradigms for prepositions in the
language \texttt{xx}.
\item \texttt{paradigverb\_xx}: paradigms for verbs in the language
\texttt{xx}.
\end{itemize}

The files consist of one entry per line. Each entry contains the
following information:

\begin{center} % use packages: array
\begin{tabular}{lllll} 
\textit{examples} & \textit{number of paradigms} & \textit{model\_paradigms} & \textit{(numbers)} &
\textit{(genders)}
\end{tabular}
\end{center}


The separator used for the different parts of an entry is the tab.
\begin{itemize}
\item \textit{Examples}: the endings that will be used to generate the
examples when the user chooses this paradigm as a model for the word
being inserted.
\item \textit{Number of paradigms}: the number of paradigms that are
used in the dictionary to inflect this inflection model.
\item \textit{Model paradigms}: the name that have in the dictionary
the paradigm/s that will be used to inflect a new entry.
\item \textit{(Numbers)}: Only completed for names, adjectives and
acronyms.  Refers to the grammatical number in the paradigm.
\item \textit{(Genders)}: Only completed for names, adjectives and
acronyms.  Refers to the grammatical gender in the paradigm.
\end{itemize}

So, therefore, for the Spanish-Catalan translator we would have the
directory \texttt{/private\-/paradigmas.es-ca} that would contain two
\texttt{XML} files: \texttt{paradig.es-ca.es.xml} and
\texttt{paradig.es-ca.ca.xml}, specifying the paradigms used in each
language. From these files, you may generate all the paradigm files
for the language pair using the command:
\begin{alltt}
   #  cd private/paradigmas.es-ca
   #  xsltproc ../paradigmas.xsl paradig.es-ca.es.xml 
                                    | ../creaparadig.awk
   #  xsltproc ../paradigmas.xsl paradig.es-ca.ca.xml 
                                    | ../creaparadig.awk
\end{alltt}


Or you can automatically generate them for all the language pairs,
using:
\begin{alltt}
   #  ./private/gen\_paradig.sh
\end{alltt}

Among the generated working files, one would be, for example, a file
called \texttt{paradigverb\_ca} that would contain the possible verb
paradigms for Catalan, where a possible line might be:

\begin{center} 
\texttt{abra/çar /ço /ci        1       abalan/çar\_\_vblex}
\end{center}

that is generated from the \texttt{XML} entry:

\begin{alltt}
<entry PoS="verb">
        <endings>
                <stem>abra</stem> 
                <ending>çar</ending>  
                <ending>ço</ending> 
                <ending>ci</ending>
        </endings>
        <paradigms howmany="1">
                <par n="abalan/çar\_\_vblex"/>
        </paradigms>
</entry>
\end{alltt}



\section{Using the forms}
\label{ss:formus}
\subsection{Introduction}


 When a user wants to insert new entries in a
dictionary, he/she has to use a web navigator to connect to the
address where the form server has been installed; for example:
\begin{center} \texttt{http://xixona.dlsi.ua.es/forms}
\end{center} A web page will be displayed with the portal of access to
\texttt{Opentrad\- Apertium\- Insertion\- Form}. The left margin
contains links to get more \textit{information} , \textit{download}
the programs and \textit{contact} the administrator of the forms to
request registration as a system user. To register as a user you will
have to send an e-mail to the administrator.

\nota{Canviar a tot arreu \emph{registrar} per \emph{inscribir}.}  To
insert new words, you will have to introduce the required data in the
form and press the \textit{'Go On'} button; at this point you will
have to identify yourself as a registered user, or else you will not
be able to continue. There are two user registration types: you can be
registered as a \emph{professional} or as a \emph{non-professional}
user. Each mode has different functionalities, that are explained in
the following section.

\subsection{Insertion of entries}
\label{insertion}

\subsubsection{Professional mode}

If you want to add a new entry to the dictionaries, you have to go to
the section of the language pair you want to improve. There, you have
to enter the source language lemma and the target language lemma, and
select their part of speech. Press the \textit{Go on} button to
continue.

A new window is displayed, with the lemmas and some parameters used to
define the entries. If the entry already exists in one of the
dictionaries, a warning message is displayed and the system automatically
selects one-way translation (from left to right or vice versa). If
none of the dictionaries contain the entry, the entry will be entered
for both directions.

In this window you can do three actions:

\notavisible{Cal repassar la primera oració del paràgraf següent;
sembla que hi ha algun material que hauria de ser esborrat; Una altra
cosa, els formularis en l'actualitat no tenen suport per a traduccions
múltiples, segons sembla. Caldria fer constar aquesta circumstància en
algun lloc.}
\begin{itemize}
\item Choose the paradigm for the SL and the TL lemmas (this is
  mandatory, the remaining actions are not).\footnote{Choosing the
  paradigm has to be done very carefully. You have to choose the
  paradigm that describes exactly the grammatical and inflection
  characteristics of the inserted word. In the case of adjectives,
  nouns and acronyms, you have to select a paradigm that fits the
  inflection of the word and the genders it may present. For example,
  in the case of acronyms you have to consider the gender and the
  number admitted by each possible paradigm; the paradigm BBC, for
  example, is for feminine singular acronyms, whereas SA is for
  feminine acronyms that may have plural form. In the case of proper
  nouns, you have to choose a different paradigm depending on whether
  the word is a proper noun of a thing (e.g. a newspaper), a person or
  a place.}
\item Select the translation direction of the entry if it is different
from the automatically suggested.
\item Add comments to the entry, that will be included in the final
dictionary.
\end{itemize}

Once the required actions have been done, you have to press
\textit{'Go on'} if you want to confirm the entry or \textit{'Close'}
if you want to cancel the insertion operation.

The following and last screen displays the three generated
\texttt{XML} entries for the SL monolingual, TL monolingual and
bilingual dictionaries. These entries are displayed in three text
boxes that can be edited if you want to do any change. Once you
checked the entries, press the \textit{'Insert'} button to finally
insert them in the corresponding dictionaries. You can also press the
\textit{'Go back'} button to return to the previous step.

\subsubsection{Non-professional mode}

When a user enters the insertion system as a non-professional user,
the word insertion mechanism is the same as for the professional user,
with the difference that the entries will not be saved in the
dictionaries generated by the forms, but will be entered in a queue of
entries pending validation. The words in this queue will not be
inserted in the dictionaries until a professional user validates them.

\subsection{Validating entries}

Professional users have two additional links in the screen for 
paradigm selection:
\begin{itemize}
\item \textit{See pairs to be validated}: Selecting this option will
open a screen that displays the content of the file of entries pending
validation; these are the entries inserted by non-professional
users. This is a merely informative screen, which can be closed
pressing the \textit{'Close'} button.
\item \textit{Validate pairs}: This option allows a professional user
to validate one by one the entries waiting for validation. Selecting
this button will open the screen for the selection of paradigms
already described in section \ref{insertion}. This screen will show
the data selected by the user for the added entry.  Now, the
professional user can modify the lemmas, delete the entry or continue
with the insertion process. If the user decides to proceed with the
insertion, the process is the same as for a normal insertion; only at
the end, when the entry is finally added to the dictionaries of the
form, the control returns to the following entry of the queue pending
validation and displays it.

This process is repeated until all the words of the queue are
validated or until the process is finished by selecting
\textit{'Close'}.

\end{itemize}



\newpage
\appendix

\chapter[XML DTDs]{Document Type Definitions (DTD) in XML}
\label{DTDs}

\section{DTD for the format of dictionaries}
\label{ss:dtd_dics}


Document type definition for the format of morphological, bilingual
and post-generation dictionaries in XML; this definition is provided
with the \texttt{apertium} package (last version) which can be
downloaded from \url{http://www.sourceforge.net}.

The description of its elements can be found in Section
\ref{formatodics}.



\begin{small}
\begin{alltt}
<!\textsl{ELEMENT} \textbf{dictionary} (alphabet?, sdefs?,
                      pardefs?, section+)>

<!\textsl{ELEMENT} \textbf{alphabet} (\textsl{#PCDATA})>
        
<!\textsl{ELEMENT} \textbf{sdefs} (sdef+)>
        
<!\textsl{ELEMENT} \textbf{sdef} \textsl{EMPTY}>
<!\textsl{ATTLIST} sdef n ID \textsl{#REQUIRED}>
        
<!\textsl{ELEMENT} \textbf{pardefs} (pardef+)>
        
<!\textsl{ELEMENT} \textbf{pardef} (e+)>
<!\textsl{ATTLIST} pardef n CDATA \textsl{#REQUIRED}>
        
<!\textsl{ELEMENT} \textbf{section} (e+)>

<!\textsl{ATTLIST} section id ID \textsl{#REQUIRED}
                  type (standard|inconditional|postblank) \textsl{#REQUIRED}>
        
<!\textsl{ELEMENT} \textbf{e} (i | p | par | re)+>
<!\textsl{ATTLIST} e r (LR|RL) \textsl{#IMPLIED}
            lm CDATA \textsl{#IMPLIED}
            a CDATA \textsl{#IMPLIED}
            c CDATA \textsl{#IMPLIED}
        
<!\textsl{ELEMENT} \textbf{par} \textsl{EMPTY}>
<!\textsl{ATTLIST} par n CDATA \textsl{#REQUIRED}>
        
<!\textsl{ELEMENT} \textbf{i} (\textsl{#PCDATA} | b | s | g | j | a)*>
        
<!\textsl{ELEMENT} \textbf{re} (\textsl{#PCDATA})>
        
<!\textsl{ELEMENT} \textbf{p} (l, r)>
        
<!\textsl{ELEMENT} \textbf{l} (\textsl{#PCDATA} | a | b | g | j | s)*>
        
<!\textsl{ELEMENT} \textbf{r} (\textsl{#PCDATA} | a | b | g | j | s)*>
        
<!\textsl{ELEMENT} \textbf{a} \textsl{EMPTY}>
        
<!\textsl{ELEMENT} \textbf{b} \textsl{EMPTY}>
        
<!\textsl{ELEMENT} \textbf{g} (\textsl{#PCDATA} | a | b | j | s)*>
<!\textsl{ATTLIST} g i CDATA \textsl{#IMPLIED}>
        
<!\textsl{ELEMENT} \textbf{j} \textsl{EMPTY}>
        
<!\textsl{ELEMENT} \textbf{s} \textsl{EMPTY}>

<!\textsl{ATTLIST} s n \textsl{IDREF} \textsl{#REQUIRED}>

\end{alltt}
\end{small}


\subsection{Modification of the DTD of dictionaries for lexical
selection}
\label{dixdtd}

The DTD for the format of dictionaries has been slightly modified so
that dictionaries can be used in a system that has a lexical selection
module. The change only affects the \texttt{<e>} element and is
displayed next.



\begin{small}
\begin{alltt}

...
<!\textsl{ATTLIST} e  
        r (LR|RL) \textsl{#IMPLIED}   
        lm \textsl{CDATA #IMPLIED}
        a \textsl{CDATA #IMPLIED}
        c \textsl{CDATA #IMPLIED}>
        i CDATA \textsl{#IMPLIED}
        slr CDATA \textsl{#IMPLIED}
        srl CDATA \textsl{#IMPLIED}>

  <!-- r: restriction LR: left-to-right,
                      RL: right-to-left -->
  <!-- lm: lemma -->
  <!-- a: author -->
  <!-- c: comment -->
  <!-- i: ignore ('yes') means ignore, otherwise it is not ignored) -->
  <!-- slr: translation sense when translating from left to right -->
  <!-- srl: translation sense when translating from right to left --> 
...

\end{alltt}
\end{small}




\section[DTD for the tagger file]{DTD for the format of the tagger
file}
\label{ss:DTD_desambiguador}

DTD that defines the format of the tagger specification file.  This
definition is provided with the \texttt{apertium} package (last
version) which can be downloaded from
\url{http://www.sourceforge.net}.

The description of its elements can be found in
Section~\ref{formatotagger}.

  \begin{small}
  \begin{alltt} 
<!\textsl{ELEMENT} \textbf{tagger} (tagset,forbid?,enforce-rules?,preferences?)>
<!\textsl{ATTLIST} tagger name \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{tagset} (def-label+,def-mult*)>

<!\textsl{ELEMENT} \textbf{def-label} (tags-item+)>
<!\textsl{ATTLIST} def-label name \textsl{CDATA} \textsl{#REQUIRED}
                    closed \textsl{CDATA} \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{tags-item} \textsl{#EMPTY}>
<!\textsl{ATTLIST} tags-item tags \textsl{CDATA} \textsl{#REQUIRED}
                    lemma \textsl{CDATA} \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{def-mult} (sequence+)>
<!\textsl{ATTLIST} def-mult name \textsl{CDATA} \textsl{#REQUIRED}
                   closed \textsl{CDATA} \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{sequence} ((tags-item|label-item)+)>

<!\textsl{ELEMENT} \textbf{label-item} \textsl{#EMPTY}>
<!\textsl{ATTLIST} label-item label \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{forbid} (label-sequence+)>

<!\textsl{ELEMENT} \textbf{label-sequence} (label-item+)>

<!\textsl{ELEMENT} \textbf{enforce-rules} (enforce-after+)>

<!\textsl{ELEMENT} \textbf{enforce-after} (label-set)>
<!\textsl{ATTLIST} enforce-after label \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{label-set} (label-item+)>

<!\textsl{ELEMENT} \textbf{preferences} (prefer+)>

<!\textsl{ELEMENT} \textbf{prefer} \textsl{EMPTY}>
<!\textsl{ATTLIST} prefer tags \textsl{CDATA} \textsl{#REQUIRED}>
  \end{alltt}
\end{small}



\section[DTD of the chunker module]{DTD of the structural transfer
module (chunker)}
\label{ss:dtdtransfer}

DTD for the format of the structural transfer rules in the
\texttt{chunker} module.  This definition is provided with the
\texttt{apertium} package (version 2.0) which can be downloaded from
\url{http://www.sourceforge.net}.

Its elements are described in Section \ref{formatotransfer}.


\begin{small}
\begin{alltt}
<!\textsl{ENTITY} \% condition "(and|or|not|equal|begins-with|
                       ends-with|contains-substring|in)">
<!\textsl{ENTITY} \% container "(var|clip)">
<!\textsl{ENTITY} \% sentence "(let|out|choose|modify-case|
                      call-macro|append)">
<!\textsl{ENTITY} \% value "(b|clip|lit|lit-tag|var|get-case-from|
                   case-of|concat)">
<!\textsl{ENTITY} \% stringvalue "(clip|lit|var|get-case-from|
                         case-of)">

<!\textsl{ELEMENT} \textbf{transfer} (section-def-cats, 
                    section-def-attrs, 
                    section-def-vars, 
                    section-def-lists?, 
                    section-def-macros?, 
                    section-rules)>

<!\textsl{ATTLIST} transfer default (lu|chunk) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{section-def-cats} (def-cat+)>

<!\textsl{ELEMENT} \textbf{def-cat} (cat-item+)>
<!\textsl{ATTLIST} def-cat n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{cat-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} cat-item lemma CDATA \textsl{#IMPLIED}
                   tags CDATA \textsl{#REQUIRED} >

<!\textsl{ELEMENT} \textbf{section-def-attrs} (def-attr+)>

<!\textsl{ELEMENT} \textbf{def-attr} (attr-item+)>
<!\textsl{ATTLIST} def-attr n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{attr-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} attr-item tags CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{section-def-vars} (def-var+)>

<!\textsl{ELEMENT} \textbf{def-var} \textsl{EMPTY}>
<!\textsl{ATTLIST} def-var n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-lists} (def-list)+>

<!\textsl{ELEMENT} \textbf{def-list} (list-item+)>
<!\textsl{ATTLIST} def-list n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{list-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} list-item v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-macros} (def-macro)+>

<!\textsl{ELEMENT} \textbf{def-macro} (\%sentence;)+>
<!\textsl{ATTLIST} def-macro n ID \textsl{#REQUIRED}>
<!\textsl{ATTLIST} def-macro npar CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-rules} (rule+)>

<!\textsl{ELEMENT} \textbf{rule} (pattern, action)>
<!\textsl{ATTLIST} rule comment CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{pattern} (pattern-item+)>

<!\textsl{ELEMENT} \textbf{pattern-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} pattern-item n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{action} (\%sentence;)*>

<!\textsl{ELEMENT} \textbf{choose} (when+,otherwise?)>

<!\textsl{ELEMENT} \textbf{when} (test,(\%sentence;)*)>

<!\textsl{ELEMENT} \textbf{otherwise} (\%sentence;)+>

<!\textsl{ELEMENT} \textbf{test} (\%condition;)+>

<!\textsl{ELEMENT} \textbf{and} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{or} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{not} (\%condition;)>

<!\textsl{ELEMENT} \textbf{equal} (\%value;,\%value;)>
<!\textsl{ATTLIST} equal caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{begins-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} begins-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{ends-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} ends-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{contains-substring} (\%value;,\%value;)>
<!\textsl{ATTLIST} contains-substring caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{in} (\%value;, list)>
<!\textsl{ATTLIST} in caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{list} \textsl{EMPTY}>
<!\textsl{ATTLIST} list n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{let} (\%container;, \%value;)>

<!\textsl{ELEMENT} \textbf{append} (\%value;)+>
<!\textsl{ATTLIST} append n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{out} (mlu|lu|b|chunk)+>

<!\textsl{ELEMENT} \textbf{modify-case} (\%container;, \%stringvalue;)>

<!\textsl{ELEMENT} \textbf{call-macro} (with-param)*>
<!\textsl{ATTLIST} call-macro n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{with-param} \textsl{EMPTY}>
<!\textsl{ATTLIST} with-param pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{clip} \textsl{EMPTY}>
<!\textsl{ATTLIST} clip pos CDATA \textsl{#REQUIRED}
               side (sl|tl) \textsl{#REQUIRED}
               part CDATA \textsl{#REQUIRED}
               queue CDATA \textsl{#IMPLIED}
               link-to CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{lit} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{lit-tag} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit-tag v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{var} \textsl{EMPTY}>
<!\textsl{ATTLIST} var n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{get-case-from} (clip|lit|var)>
<!\textsl{ATTLIST} get-case-from pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{case-of} \textsl{EMPTY}>
<!\textsl{ATTLIST} case-of pos CDATA \textsl{#REQUIRED}
                  side (sl|tl) \textsl{#REQUIRED}
                  part CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{concat} (\%value;)+>

<!\textsl{ELEMENT} \textbf{mlu} (lu+)>

<!\textsl{ELEMENT} \textbf{lu} (\%value;)+>

<!\textsl{ELEMENT} \textbf{chunk} (tags,(mlu|lu|b)+)>
<!\textsl{ATTLIST} chunk name CDATA \textsl{#IMPLIED}
                namefrom CDATA \textsl{#IMPLIED}
                case CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{tags} (tag+)>
<!\textsl{ELEMENT} \textbf{tag} (\%value;)>

<!\textsl{ELEMENT} \textbf{b} \textsl{EMPTY}>
<!\textsl{ATTLIST} b pos CDATA \textsl{#IMPLIED}>

\end{alltt}
\end{small}



\newpage
\section{DTD of the interchunk module}
\label{ss:dtdinterchunk}

DTD for the format of the structural transfer rules in the
\texttt{interchunk} module. This definition is provided with the
\texttt{apertium} package (version 2.0) which can be downloaded from
\url{http://www.sourceforge.net}.

Its elements are described in Section \ref{formatotransfer}.


\begin{small}
\begin{alltt}

<!\textsl{ENTITY} \% condition "(and|or|not|equal|begins-with|
                       ends-with|contains-substring|in)">
<!\textsl{ENTITY} \% container "(var|clip)">
<!\textsl{ENTITY} \% sentence "(let|out|choose|modify-case|
                      call-macro|append)">
<!\textsl{ENTITY} \% value "(b|clip|lit|lit-tag|var|get-case-from|
                   case-of|concat)">
<!\textsl{ENTITY} \% stringvalue "(clip|lit|var|get-case-from|
                         case-of)">

<!\textsl{ELEMENT} \textbf{interchunk} (section-def-cats, 
                      section-def-attrs, 
                      section-def-vars, 
                      section-def-lists?, 
                      section-def-macros?, 
                      section-rules)>

<!\textsl{ELEMENT} \textbf{section-def-cats} (def-cat+)>

<!\textsl{ELEMENT} \textbf{def-cat} (cat-item+)>
<!\textsl{ATTLIST} def-cat n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{cat-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} cat-item lemma CDATA \textsl{#IMPLIED}
                    tags CDATA \textsl{#REQUIRED} >

<!\textsl{ELEMENT} \textbf{section-def-attrs} (def-attr+)>

<!\textsl{ELEMENT} \textbf{def-attr} (attr-item+)>
<!\textsl{ATTLIST} def-attr n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{attr-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} attr-item tags CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{section-def-vars} (def-var+)>

<!\textsl{ELEMENT} \textbf{def-var} \textsl{EMPTY}>
<!\textsl{ATTLIST} def-var n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-lists} (def-list)+>

<!\textsl{ELEMENT} \textbf{def-list} (list-item+)>
<!\textsl{ATTLIST} def-list n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{list-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} list-item v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-macros} (def-macro)+>

<!\textsl{ELEMENT} \textbf{def-macro} (\%sentence;)+>
<!\textsl{ATTLIST} def-macro n ID \textsl{#REQUIRED}>
<!\textsl{ATTLIST} def-macro npar CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-rules} (rule+)>

<!\textsl{ELEMENT} \textbf{rule} (pattern, action)>
<!\textsl{ATTLIST} rule comment CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{pattern} (pattern-item+)>

<!\textsl{ELEMENT} \textbf{pattern-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} pattern-item n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{action} (\%sentence;)*>

<!\textsl{ELEMENT} \textbf{choose} (when+,otherwise?)>

<!\textsl{ELEMENT} \textbf{when} (test,(\%sentence;)*)>

<!\textsl{ELEMENT} \textbf{otherwise} (\%sentence;)+>

<!\textsl{ELEMENT} \textbf{test} (\%condition;)+>

<!\textsl{ELEMENT} \textbf{and} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{or} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{not} (\%condition;)>

<!\textsl{ELEMENT} \textbf{equal} (\%value;,\%value;)>
<!\textsl{ATTLIST} equal caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{begins-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} begins-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{ends-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} ends-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{contains-substring} (\%value;,\%value;)>
<!\textsl{ATTLIST} contains-substring caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{in} (\%value;, list)>
<!\textsl{ATTLIST} in caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{list} \textsl{EMPTY}>
<!\textsl{ATTLIST} list n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{let} (\%container;, \%value;)>

<!\textsl{ELEMENT} \textbf{append} (\%value;)+>
<!\textsl{ATTLIST} append n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{out} (b|chunk)+>

<!\textsl{ELEMENT} \textbf{modify-case} (\%container;, \%stringvalue;)>

<!\textsl{ELEMENT} \textbf{call-macro} (with-param)*>
<!\textsl{ATTLIST} call-macro n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{with-param} \textsl{EMPTY}>
<!\textsl{ATTLIST} with-param pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{clip} \textsl{EMPTY}>
<!\textsl{ATTLIST} clip pos CDATA \textsl{#REQUIRED}
               part CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{lit} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{lit-tag} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit-tag v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{var} \textsl{EMPTY}>
<!\textsl{ATTLIST} var n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{get-case-from} (clip|lit|var)>
<!\textsl{ATTLIST} get-case-from pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{case-of} \textsl{EMPTY}>
<!\textsl{ATTLIST} case-of pos CDATA \textsl{#REQUIRED}
                  part CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{concat} (\%value;)+>

<!\textsl{ELEMENT} \textbf{chunk} (\%value;)+>

<!\textsl{ELEMENT} \textbf{pseudolemma} (\%value;)>

<!\textsl{ELEMENT} \textbf{b} \textsl{EMPTY}>
<!\textsl{ATTLIST} b pos CDATA \textsl{#IMPLIED}>

\end{alltt}
\end{small}

\newpage

\section{DTD of the postchunk module}
\label{ss:dtdpostchunk}

DTD for the format of the structural transfer rules in the
\texttt{postchunk} module. This definition is provided with the
\texttt{apertium} package (version 2.0) which can be downloaded from
\url{http://www.sourceforge.net}.

Its elements are described in Section \ref{formatotransfer}.



\begin{small}
\begin{alltt}
<!\textsl{ENTITY} \% condition "(and|or|not|equal|begins-with|
                       ends-with|contains-substring|in)">
<!\textsl{ENTITY} \% container "(var|clip)">
<!\textsl{ENTITY} \% sentence "(let|out|choose|modify-case|
                      call-macro|append)">
<!\textsl{ENTITY} \% value "(b|clip|lit|lit-tag|var|get-case-from|
                   case-of|concat)">
<!\textsl{ENTITY} \% stringvalue "(clip|lit|var|get-case-from|
                         case-of)">

<!\textsl{ELEMENT} \textbf{postchunk} (section-def-cats, 
                      section-def-attrs, 
                      section-def-vars, 
                      section-def-lists?, 
                      section-def-macros?, 
                      section-rules)>

<!\textsl{ELEMENT} \textbf{section-def-cats} (def-cat+)>

<!\textsl{ELEMENT} \textbf{def-cat} (cat-item+)>
<!\textsl{ATTLIST} def-cat n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{cat-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} cat-item name CDATA \textsl{#REQUIRED}>
 
<!\textsl{ELEMENT} \textbf{section-def-attrs} (def-attr+)>

<!\textsl{ELEMENT} \textbf{def-attr} (attr-item+)>
<!\textsl{ATTLIST} def-attr n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{attr-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} attr-item tags CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{section-def-vars} (def-var+)>

<!\textsl{ELEMENT} \textbf{def-var} \textsl{EMPTY}>
<!\textsl{ATTLIST} def-var n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-lists} (def-list)+>

<!\textsl{ELEMENT} \textbf{def-list} (list-item+)>
<!\textsl{ATTLIST} def-list n ID \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{list-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} list-item v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-def-macros} (def-macro)+>

<!\textsl{ELEMENT} \textbf{def-macro} (\%sentence;)+>
<!\textsl{ATTLIST} def-macro n ID \textsl{#REQUIRED}>
<!\textsl{ATTLIST} def-macro npar CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{section-rules} (rule+)>

<!\textsl{ELEMENT} \textbf{rule} (pattern, action)>
<!\textsl{ATTLIST} rule comment CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{pattern} (pattern-item+)>

<!\textsl{ELEMENT} \textbf{pattern-item} \textsl{EMPTY}>
<!\textsl{ATTLIST} pattern-item n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{action} (\%sentence;)*>

<!\textsl{ELEMENT} \textbf{choose} (when+,otherwise?)>

<!\textsl{ELEMENT} \textbf{when} (test,(\%sentence;)*)>

<!\textsl{ELEMENT} \textbf{otherwise} (\%sentence;)+>

<!\textsl{ELEMENT} \textbf{test} (\%condition;)+>

<!\textsl{ELEMENT} \textbf{and} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{or} ((\%condition;),(\%condition;)+)>

<!\textsl{ELEMENT} \textbf{not} (\%condition;)>

<!\textsl{ELEMENT} \textbf{equal} (\%value;,\%value;)>
<!\textsl{ATTLIST} equal caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{begins-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} begins-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{ends-with} (\%value;,\%value;)>
<!\textsl{ATTLIST} ends-with caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{contains-substring} (\%value;,\%value;)>
<!\textsl{ATTLIST} contains-substring caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{in} (\%value;, list)>
<!\textsl{ATTLIST} in caseless (no|yes) \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{list} \textsl{EMPTY}>
<!\textsl{ATTLIST} list n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{let} (\%container;, \%value;)>

<!\textsl{ELEMENT} \textbf{append} (\%value;)+>
<!\textsl{ATTLIST} append n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{out} (b|lu|mlu)+>

<!\textsl{ELEMENT} \textbf{modify-case} (\%container;, \%stringvalue;)>

<!\textsl{ELEMENT} \textbf{call-macro} (with-param)*>
<!\textsl{ATTLIST} call-macro n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{with-param} \textsl{EMPTY}>
<!\textsl{ATTLIST} with-param pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{clip} \textsl{EMPTY}>
<!\textsl{ATTLIST} clip pos CDATA \textsl{#REQUIRED}
               part CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{lit} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{lit-tag} \textsl{EMPTY}>
<!\textsl{ATTLIST} lit-tag v CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{var} \textsl{EMPTY}>
<!\textsl{ATTLIST} var n \textsl{IDREF} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{get-case-from} (clip|lit|var)>
<!\textsl{ATTLIST} get-case-from pos CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{case-of} \textsl{EMPTY}>
<!\textsl{ATTLIST} case-of pos CDATA \textsl{#REQUIRED}
                  part CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{concat} (\%value;)+>

<!\textsl{ELEMENT} \textbf{mlu} (lu+)>

<!\textsl{ELEMENT} \textbf{lu} (\%value;)+>

<!\textsl{ELEMENT} \textbf{b} \textsl{EMPTY}>
<!\textsl{ATTLIST} b pos CDATA \textsl{#IMPLIED}>

\end{alltt}
\end{small}

\newpage


\section[DTD for the format rules]{DTD for the format specification
rules}
\label{ss:dtd_formato}

DTD for the format specification rules. This definition can be
downloaded from the web page
\url{http://cvs.sourceforge.net/viewcvs.py/apertium/apertium/apertium/format.dtd}. \nota{needs
updating}


Its elements are described in Section \ref{ss:reglasformato}.

\begin{small}
\begin{alltt} 
<!\textsl{ELEMENT} \textbf{format} (options,rules)>
<!\textsl{ATTLIST} format name \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{options} (largeblocks, input, output, 
                   escape-chars, space-chars, case-sensitive)>

<!\textsl{ELEMENT} \textbf{largeblocks} \textsl{EMPTY}>
<!\textsl{ATTLIST} largeblocks size \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{input} \textsl{EMPTY}>
<!\textsl{ATTLIST} input zip-path \textsl{CDATA} \textsl{#IMPLIED}
                encoding \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{output} \textsl{EMPTY}>
<!\textsl{ATTLIST} output zip-path \textsl{CDATA} \textsl{#IMPLIED}
                 encoding \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{escape-chars} \textsl{EMPTY}>
<!\textsl{ATTLIST} escape-chars regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{space-chars} \textsl{EMPTY}>
<!\textsl{ATTLIST} space-chars regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{case-sensitive} \textsl{EMPTY}>
<!\textsl{ATTLIST} case-sensitive value (yes|no) \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{rules} (format-rule|replacement-rule)+>

<!\textsl{ELEMENT} \textbf{format-rule} (begin-end|(begin,end))>
<!\textsl{ATTLIST} format-rule eos (yes|no) \textsl{#IMPLIED}
                      priority \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{begin-end} \textsl{EMPTY}>
<!\textsl{ATTLIST} begin-end regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{begin} \textsl{EMPTY}>
<!\textsl{ATTLIST} begin regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{end} \textsl{EMPTY}>
<!\textsl{ATTLIST} end regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{replacement-rule} (replace+)>
<!\textsl{ATTLIST} replacement-rule regexp \textsl{CDATA} \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{replace} \textsl{EMPTY}>
<!\textsl{ATTLIST} replace source \textsl{CDATA} \textsl{#REQUIRED}
                  target \textsl{CDATA} \textsl{#REQUIRED}
                  prefer (yes|no) \textsl{#IMPLIED}>

\end{alltt}
\end{small}

\newpage
\section{DTD for the form paradigms}
\label{ss:dtdparadigmes}

DTD for the format of the paradigm files used in the forms. This
definition is included in the package
\texttt{apertium-lexical-webform}.

\begin{small}
\begin{alltt}


<!\textsl{ELEMENT} \textbf{form} (entry)+>

<!\textsl{ATTLIST} \textbf{form}
        lang CDATA \textsl{#REQUIRED}
        langpair CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{entry} (endings, paradigms)+>

<!\textsl{ATTLIST} \textbf{entry}
        PoS CDATA \textsl{#REQUIRED}
        nbr CDATA \textsl{#IMPLIED}
        gen CDATA \textsl{#IMPLIED}>

<!\textsl{ELEMENT} \textbf{endings} (stem, ending+)>

<!\textsl{ELEMENT} \textbf{stem} (\textsl{#PCDATA})>

<!\textsl{ELEMENT} \textbf{ending} (\textsl{#PCDATA})>

<!\textsl{ELEMENT} \textbf{paradigms} (par+)>

<!\textsl{ATTLIST} \textbf{paradigms} howmany CDATA \textsl{#REQUIRED}>

<!\textsl{ELEMENT} \textbf{par} \textsl{EMPTY}>

<!\textsl{ATTLIST} \textbf{par} n CDATA \textsl{#REQUIRED}>


\end{alltt}
\end{small}



\chapter[Grammatical symbols]{Grammatical symbols used in the modules}
\label{se:simbolosmorf}



\section[Dictionary symbols]{Grammatical symbols used in dictionaries}

\subsection{List of symbols}


\begin{tabular}{ll}

\textbf{aa} & adjective-adjective (function of relative pronoun) \\
 \textbf{acr} & acronym \\ \textbf{al} & others (for proper nouns) \\
 \textbf{an} & adjective-noun (function of relative pronoun) \\
 \textbf{ant} & antroponym \\ \textbf{cni} & conditional \\
 \textbf{cnjadv} & adverbial conjunction\\ \textbf{cnjcoo} &
 co-ordinating conjunction\\ \textbf{cnjsub} & subordinating
 conjunction\\ \textbf{def} & definite \\ \textbf{dem} & demonstrative
 \\ \textbf{det} & determiner\\ \textbf{detnt} & neuter determiner \\
 \textbf{enc} & enclitic\\ \textbf{f} & feminine\\ \textbf{fti} &
 future indicative\\ \textbf{fts} & future subjunctive\\ \textbf{ger}
 & gerund\\ \textbf{ifi} & perfect preterite\\ \textbf{ij} &
 interjection\\ \textbf{imp} & imperative\\ \textbf{ind} &
 indefinite\\ \textbf{inf} & infinitive\\

\end{tabular} \newpage

\begin{tabular}{ll} \textbf{itg} & interrogative\\
\textbf{loc} & locative\\
\textbf{lpar} & ([\\ \textbf{lquest} & ¿\\ \textbf{m} & masculine\\
\textbf{mf} & masculine-feminine\\ \textbf{n} & noun\\ \textbf{nn} &
noun-noun (function of relative pronoun)\\ \textbf{np} & proper noun\\
\textbf{nt} & neuter\\ \textbf{num} & numeral - number\\ \textbf{p1} &
first person\\ \textbf{p2} & second person\\ \textbf{p3} & third
person\\ \textbf{pii} & imperfect preterite indicative \\ \textbf{pis}
& imperfect preterite subjunctive \\ \textbf{pl} & plural \\
\textbf{pos} & possessive\\ \textbf{pp} & participle\\ \textbf{pr} &
preposition\\ \textbf{preadv} & preadverb\\ \textbf{predet} &
predeterminer\\ \textbf{pri} & present indicative\\ \textbf{prn} &
pronoun\\ \textbf{pro} & proclitic\\ \textbf{prs} & present
subjunctive\\ \textbf{ref} & reflexive\\ \textbf{rel} & relative\\
\textbf{rpar} & )]\\ \textbf{sent} & . ? ; : ! \\ \textbf{sg} &
singular\\ \textbf{sp} & singular-plural\\ \textbf{sup} &
superlative\\ \textbf{tn} & tonic\\ \textbf{vaux} & auxiliary verb\\
\textbf{vbhaver}& verb \emph{to have}\\ \textbf{vblex} & lexical
verb\\ \textbf{vbmod} & modal verb\\ \textbf{vbser} & verb \emph{to
be}

\end{tabular}

\newpage
\subsection{Specification of lexical forms}

Order for the placement of grammatical symbols in the morphological
dictionaries of this system (from left to right in the table). The
examples in brackets are from Spanish. \\ \\

\begin{footnotesize}
\begin{tabular}{|l|llllll|}

\hline \textbf{Common adjectives} & \textbf{PoS} & \textbf{Gender} &
\textbf{Number} &&& \\\cline{2-7}
(difícil, rojo) & adj & m & sg &&& \\ & & f & pl &&& \\ & & mf & sp
                  &&& \\ \hline \textbf{Interrogative, possessive,} &
                  \textbf{PoS} & \textbf{Type} & \textbf{Gender}
                  &\textbf{Number}&& \\\cline{2-7}
\textbf{indetermined and superlative} & adj & itg & m & sg &&\\
 \textbf{adjectives} & & pos & f & pl &&\\ (qué, tus, otra, buenísimo)
  & & ind & mf & sp &&\\ & & sup & & &&\\\hline


\textbf{Adverbs} & \textbf{PoS} &&&&&\\\cline{2-7} (siempre, mañana)&
adv &&&&&\\\hline

\textbf{Preadverbs} & \textbf{PoS} &&&&&\\\cline{2-7} (muy, tan)&
preadv &&&&&\\\hline

\textbf{Interrogative adverbs} & \textbf{PoS}
&\textbf{Type}&&&&\\\cline{2-7} (dónde) & adv & itg &&&&\\\hline

\textbf{Adverbial conjunctions} & \textbf{PoS} &&&&&\\\cline{2-7}
(que, así como) & cnjadv &&&&&\\ & cnjcoo &&&&&\\ & cnjsub
&&&&&\\\hline


\textbf{Determiners} & \textbf{PoS} & \textbf{Type} & \textbf{Gender}
&\textbf{Number}&& \\\cline{2-7} (el, uno, este, mi) & det & def & m &
sg &&\\ & & ind & f & pl &&\\ & & dem & mf & sp &&\\ & & pos & &
&&\\\hline

\textbf{Neuter determiners} & \textbf{PoS} &&&&&\\\cline{2-7} (lo)&
detnt &&&&&\\\hline

\textbf{Predeterminers} & \textbf{PoS} & \textbf{Gender} &
\textbf{Number} &&& \\\cline{2-7} (todos) & predet & m & sg &&&\\ & &
f & pl &&&\\ & & nt & sp &&&\\\hline \textbf{Interjections} &
\textbf{PoS} &&&&&\\\cline{2-7} (hola) & ij &&&&&\\\hline




\textbf{Common nouns}& \textbf{PoS} & \textbf{Gender} &
\textbf{Number} &&& \\\cline{2-7} (casa, perro) & n & m & sg &&&\\ & n
& f & pl &&&\\ & n & mf & sp &&&\\\hline

\textbf{Proper nouns}& \textbf{PoS} &\textbf{Type}&&&&\\\cline{2-7}
(Pedro, Londres) & np & ant &&&&\\ & & loc &&&&\\ & & al &&&&\\\hline

\end{tabular} \newpage
\begin{tabular}{|l|llllll|} \hline

\textbf{Acronyms} & \textbf{PoS} & \textbf{Type} & \textbf{Gender} &
\textbf{Number} && \\\cline{2-7} (IRPF, INEM) & n & acr & m & sg &&\\
& & & f & pl &&\\ & & & mf & sp &&\\\hline


\textbf{Numerals} & \textbf{PoS} & \textbf{Gender} & \textbf{Number}
&&& \\\cline{2-7} (tres) & num & m & sg &&& \\ & & f & pl &&& \\ & &
mf & sp &&& \\\hline

\textbf{Prepositions} & \textbf{PoS} &&&&&\\\cline{2-7} (de, por) & pr
&&&&&\\\hline

\textbf{Interrogative pronouns} & \textbf{PoS} & \textbf{Type} &
\textbf{Gender} &\textbf{Number}&& \\\cline{2-7} (quién, qué) & prn &
itg & m & sg &&\\ & & & f & pl &&\\\hline


\textbf{Enclitic, proclitic and} & \textbf{PoS} & \textbf{Type} &
\textbf{Person}& \textbf{Gender} &\textbf{Number}& \\\cline{2-7}
\textbf{tonic personal} & prn & enc & p1 & m & sg &\\
   \textbf{pronouns} & & pro & p2 & f & pl &\\ (yo, vosotros,
   ayudar\textbf{te}, & & tn & p3 & mf & sp & \\ \textbf{te} ayudo) &
   & & & nt && \\ & & & & & & \\\cline{2-7}
\textbf{Procl. reflexive pron.} (se): & prn & pro & ref & p3 & mf &
sp\\\cline{2-7} \textbf{Tonic reflex. pron.} (si): & prn & tn & ref &
p3 & mf & sp\\\hline



\textbf{Tonic possessive pron.} & \textbf{PoS} & \textbf{Type} &
\textbf{Subtype}& \textbf{Gender} &\textbf{Number}& \\\cline{2-7}
(mío, suyo) & prn & tn & pos & m & sg &\\ & & & & f & pl &\\\hline


\textbf{Other tonic pronouns} & \textbf{PoS} & \textbf{Type} &
\textbf{Gender} &\textbf{Number}&& \\\cline{2-7} (aquella, nadie,
otro) & prn & tn & m & sg &&\\ & & & f & pl &&\\ & & & mf & sp && \\ &
& & nt &&& \\\hline


\textbf{Pronominal and adjectival} & \textbf{PoS} & \textbf{Type} &
\textbf{Gender} & \textbf{Number} && \\\cline{2-7} \textbf{relatives}
& rel & nn & m & sg &&\\ (que, cuyo) & & an & f & pl &&\\ & & aa & f &
pl &&\\\hline

\textbf{Adverbial relatives} & \textbf{PoS} & \textbf{Type} & & &&
\\\cline{2-7} (como, donde) & rel & adv & & &&\\\hline


\textbf{Verbs} & \textbf{Type} & \textbf{Tense} & \textbf{Person}
&\textbf{Number}&& \\ \textbf{(personal forms)} & & \textbf{and mode}
& & && \\\cline{2-7} (subo, vamos) & vblex & cni & p1 & sg &&\\ &
vbser & fti & p2 & pl &&\\ & vbhaver & fts & p3 & &&\\ & vbmod & ifi &
& &&\\ & & imp & & &&\\ & & pii & & &&\\ & & pis & & &&\\ & & pri & &
&&\\ & & prs & & &&\\\hline


\textbf{Verbs} & \textbf{Type} & \textbf{Form} & & && \\\cline{2-7}
\textbf{(infinitive and gerund)} & vblex & inf & & &&\\ (cantar,
buscando) & vbser & ger & & &&\\ & vbhaver & & & &&\\ & vbmod & & &
&&\\\hline



\textbf{Verbs} & \textbf{Type} & \textbf{Form} &\textbf{Gender}
&\textbf{Number} && \\\cline{2-7} \textbf{(participle)} & vblex & pp &
m & sg &&\\ (dormido, cansadas) & vbser & & f & pl &&\\ & vbhaver & &
& &&\\ & vbmod & & & &&\\\hline
   


\end{tabular}
\end{footnotesize}


\newpage
\section{Categories used in the part-of-speech tagger}
\subsection{Spanish tagger}

These are the categories or coarse tags used by the Spanish
part-of-speech tagger.


\begin{footnotesize}
\begin{longtable}{l|l|c|l} \hline \bf{Tag} & \bf{Description} &
\bf{Closed} & \bf{Examples} \\ \hline \hline
\endhead \multicolumn{4}{c}{\bf{Simple tags}} \\ \hline \hline PARAPR
& Lexicalization of \emph{para} as a preposition & Yes & \\ \hline
PARAVBPRI & Lexicalization of \emph{para} as a lexical verb & & \\ &
in present indicative & Yes& \\ \hline PARAVBIMP & Lexicalization of
\emph{para} as a lexical verb & & \\ & in imperative & Yes& \\ \hline
QUECNJ & Lexicalization of \emph{que} as a conjunction & Yes& \\
\hline QUEREL & Lexicalization of \emph{que} as a relative pronoun &
Yes& \\ \hline COMOPR\footnote{The morphological analyser considers
that \emph{como} can be a preposition since it can be replaced with
\emph{en calidad de} in some contexts (e.g.- \emph{'Os hablo como
director de la película'}).} & Lexicalization of \emph{como} as a
       preposition& Yes& \\
%!!!!!!!!!!Explicar esto de la preposición porque no es muy estándar que digamos 
\hline COMOREL & Lexicalization of \emph{como} as a
relative pronoun & Yes& \\ \hline COMOVB & Lexicalization of
\emph{como} as a lexical verb & & \\ & in present indicative & Yes& \\
\hline MASADV & Lexicalization of \emph{más}/\emph{menos} as an adverb
& Yes& \\ \hline MASADJ & Lexicalization of \emph{más}/\emph{menos} as
an adjective & Yes& \\ \hline MASNP & Lexicalization of \emph{Más} as
a proper noun & Yes& \\ \hline ALGOADV & Lexicalization of \emph{algo}
as an adverb & Yes& \\ \hline ACRONIMOM & Acronym & No& BCH\\ \hline
ACRONIMOF & Acronym & No& ONU\\ \hline ACRONIMOMF & Acronym & No&
ATS\\ \hline INTNOM & Interrogative pronoun & Yes& quién, cuál\\
\hline ADJINT & Interrogative adjective & Yes& cuánto, qué\\ \hline
INTADV & Interrogative adverb & Yes& cuándo, dónde\\ \hline PREADV &
Adverb that can precede another & &\\& adverb or an adjective & Yes&
muy, bien, mal\\ \hline ADV & Adverb & No& nunca, ahí\\ \hline CNJSUBS
& Subordinating conjunction & Yes& que\\ 
%!!!!!!! No hay más conjunciones subordinadas a parte de que?????  
\hline CNJCOORD &
Co-ordinating conjunction & Yes& y, pero\\ \hline CNJADV & Adverbial
conjunction & No& si\\ \hline DETNT & Neuter determiner & Yes& lo\\
\hline DETM & Determiner & Yes& el, un\\ \hline DETF & Determiner &
Yes& la, una\\ \hline DETMF & Determiner & Yes& cada\\ \hline INTERJ &
Interjection & No& ojalá, hola\\ \hline NOM & Noun & No& casa, coche\\
\hline ANTROPONIM & Proper noun for person & No& Fernando\\ \hline
TOPONIM & Proper noun for place & No& Alicante\\ \hline NPALTRES &
Other proper nouns & No& Linux, Seat\\ \hline NUM & Numeral & Yes&
tres, cuatro\\ \hline PREDETNT & Neuter predeterminer & Yes& todo\\
\hline PREDET & Predeterminer & Yes& toda\\ \hline PREP & Preposition
& Yes& ante, desde\\ \hline PRNTNNT & Neuter tonic pronoun & Yes&
algo, esto\\ \hline PRNTN & Tonic pronoun & Yes& ambos, nadie\\ \hline
PRNENCREF & Reflexive enclitic pronoun & Yes& se \\ \hline PRNPROREF &
Reflexive proclitic pronoun & Yes& se \\ \hline PRNENC & Enclitic
pronoun & Yes& me, nos\\ \hline PRNPRO & Proclitic pronoun & Yes& le,
te\\ \hline VLEXINF & Lexical verb in infinitive & No& cantar, reír\\
\hline VLEXGER & Lexical verb in gerund & No& hablando\\ \hline
VLEXPARTPI & Lexical verb in participle & No& dicho, cantado\\ \hline
VLEXPFCI & Lexical verb in present, future or & & \\ & conditional
indicative & No& digo, diré, diría\\ \hline VLEXIPI & Lexical verb in
imperfect preferite or & & \\ & perfect preterite indicative & No&
cantaba, dijo\\ \hline VLEXSUBJ & Lexical verb in subjunctive & No&
hablase, dijeramos\\ \hline VLEXIMP & Lexical verb in imperative & No&
canta, comed\\ \hline VSERINF & Verb \emph{to be} in infinitive & Yes&
ser\\ \hline VSERGER & Verb \emph{to be} in gerund & Yes& siendo\\
\hline VSERPARTPI & Verb \emph{to be} in participle & Yes& sido\\
\hline VSERPFCI & Verb \emph{to be} in present, future or & & \\ &
conditional indicative & Yes& soy, seré, sería\\ \hline VSERIPI & Verb
\emph{to be} in imperfect preterite or & & \\ & perfect preterite
indicative & Yes& era, fui\\ \hline VSERSUBJ & Verb \emph{to be} in
subjunctive & Yes& fueras\\ \hline VSERIMP & Verb \emph{to be} in
imperative & Yes& sé\\ \hline VHABERINF & Verb \emph{to have} in
infinitive & Yes& haber\\ \hline VHABERGER & Verb \emph{to have} in
gerund & Yes& habiendo\\ \hline VHABERPARTPI & Verb \emph{to have} in
participle & Yes& habido\\ \hline VHABERPFCI & Verb \emph{to have} in
present, future or & & \\ & conditional indicative & Yes& hay, habrán,
habría\\ \hline VHABERIPI & Verb \emph{to have} in imperfect preterite
or & & \\ & perfect preterite indicative & Yes& había, hubo\\ \hline
VHABERSUBJ & Verb \emph{to have} in subjunctive & Yes& hubieran\\
\hline VMODALINF & Modal verb in infinitive & Yes& deber, poder\\
\hline VMODALGER & Modal verb in gerund & Yes& debiendo\\ \hline
VMODALPARTPI & Modal verb in participle & Yes& podido\\ \hline
VMODALPFCI & Modal verb in present, future or & & \\ & conditional
indicative & Yes& puede, deberá, podría\\ \hline VMODALIPI & Modal
verb in imperfect preterite or & & \\ & perfect preterite indicative &
Yes& podía, debió\\ \hline VMODALSUBJ & Modal verb in subjunctive &
Yes& pudiese, debiéramos\\ \hline VMODALIMP & Modal verb in imperative
& Yes& poded, debed\\ \hline ADJM & Adjective & No& gracioso\\ \hline
ADJF & Adjective & No& graciosa\\ \hline ADJMF & Adjective & No&
inteligente\\ \hline ADJPOS & Possessive adjective & Yes& mío\\ \hline
REL & Relative pronoun & Yes& quien, cuya\\ \hline RELADV & Adverbial
relative & Yes& cuando, donde\\ \hline \hline
\multicolumn{4}{c}{\bf{Compound tags}} \\ \hline \hline PREPDET &
Contraction of preposition and determiner & Yes& del, al\\ \hline
PRCNJ & Multiword made of preposition and & & \\ &conjunction & Yes& a
que\\ \hline PRREL & Multiword made of preposition and & & \\
&relative & Yes& en que\\ \hline INFLEXPRNENC & Lexical verb in
infinitive with enclitics & No& dármelo, cantarlo\\ \hline
GERLEXPRNENC & Lexical verb in gerund with enclitics & No&
cantándosela\\ \hline IMPLEXPRNENC & Lexical verb in imperative with
enclitics & No& dímelo\\ \hline INFSERPRNENC & Verb \emph{to be} in
infinitive with enclitics & Yes& serlo\\ \hline GERSERPRNENC & Verb
\emph{to be} in gerund with enclitics & Yes& siéndolo\\ \hline
IMPSERPRNENC & Verb \emph{to be} in imperative with enclitics & Yes&
sedlo\\ \hline INFHABPRNENC & Verb \emph{to have} in infinitive with
enclitics & Yes& habérsela\\ \hline GERHABPRNENC & Verb \emph{to have}
in gerund with enclitics & Yes& habiéndole\\ \hline INFMODPRNENC &
Modal verb in infinitive with enclitics & Yes& poderla, deberlo\\
\hline GERMODPRNENC & Modal verb in gerund with enclitics& Yes&
debiéndosela\\ \hline IMPMODPRNENC & Modal verb in imperative with
enclitics& Sí& debédmela\\ \hline \hline \multicolumn{4}{c}{\bf{Other
tags}} \\ \hline \hline LQUEST & Opening question mark & & ¿ \\ \hline
LPAR & Opening parenthesis or square bracket & & (, [ \\ \hline RPAR &
Closing parenthesis or square bracket & & ), ] \\ \hline CM & Comma &
& , \\ \hline SENT & Sentence end character & & ., :, ;, ?, !\\ \hline
\hline \multicolumn{4}{l}{}\\ %p{0.50\textwidth}

\end{longtable}
\end{footnotesize}

\subsection{Catalan tagger}

Due to the similarity of the Catalan tagger categories and the Spanish
ones, we list here only the tags that are new or different in the
Catalan tagger.

\begin{footnotesize}
\begin{longtable}{l|l|c|l} \hline \bf{Tag} & \bf{Description} &
\bf{Closed} & \bf{Examples} \\ \hline \hline
\endhead \multicolumn{4}{c}{\bf{Simple tags}} \\ \hline \hline MOLTADV
& Lexicalization of \emph{molt}/\emph{gaire} as an adverb & Yes & \\
\hline MOTLPREADV & Lexicalization of \emph{molt}/\emph{gaire} as an
adverb & Yes& \\ \hline VOLERMOD & Lexicalization of \emph{voler} as a
modal verb & Yes& \\ \hline VOLERLEX & Lexicalization of \emph{voler}
as a lexical verb & Yes& \\ \hline VA & Lexicalization of \emph{va} as
a form of the verb \emph{anar} & Yes& \\ \hline \multicolumn{4}{l}{}\\
%p{0.50\textwidth}
\end{longtable}
\end{footnotesize}

\subsection{Galician tagger}


Due to the similarity of the Galician tagger categories and the
Spanish ones, we list here only the tags that are new or different in
the Galician tagger.


\begin{footnotesize}
\begin{longtable}{l|l|c|l} \hline \bf{Tag} & \bf{Description} &
\bf{Closed} & \bf{Examples} \\ \hline \hline
\endhead \multicolumn{4}{c}{\bf{Simple tags}} \\ \hline \hline VBIRNPS
& Lexicalization of \emph{to go} in infinitive & & \\ & and gerund &
Yes & \\ \hline VBIRPARTPI & Lexicalization of \emph{to go} in
participle & Yes& \\ \hline VBIRPS & Lexicalization of \emph{to go} in
the personal forms & & \\ & of indicative & & \\ & and subjunctive &
Yes& \\ \hline VBIRIMP & Lexicalization of \emph{to go} in imperative
& Yes& \\ \hline VHABERNPS & Lexicalization of \emph{to have} in
infinitive & & \\ & and gerund & Yes & \\ \hline VHABERPARTPI &
Lexicalization of \emph{to have} in participle & Yes& \\ \hline
VHABERPS & Lexicalization of \emph{to have} in the personal forms & &
\\ & of indicative & & \\ & and subjunctive & Yes& \\ \hline VHABERIMP
& Lexicalization of \emph{to have} in imperative & Yes& \\ \hline
APREP & Lexicalization of \emph{a} as a preposition & Yes& \\ \hline
VLEXNPS & Lexical verb: infinitive and gerund & No& achegar,
achegándomos\\ \hline VLEXPS & Lexical verb: personal forms & & \\ &
in indicative & No& achegue, achegaré\\ \hline VSERNPS & Verb \emph{to
be}: infinitive and gerund & Yes& ser, seres\\ \hline VSERPS & Verb
\emph{to be}: personal forms & & \\ & in indicative& Yes& fosen, es\\
\hline \hline \multicolumn{4}{c}{\bf{Compound tags}} \\ \hline \hline
PREPDETM &
Contraction of preposition and & & \\ & masculine determiner & Yes&
do, ao\\ \hline PREPDETF & Contraction of preposition and & & \\ &
feminine determiner & Yes& da, ás\\ \hline PREPDETN & Contraction of
preposition and & & \\ & neuter determiner & Yes& do\\ \hline
PREPDETDET & Contraction of preposition and & & \\ & two determiners &
Yes& destoutro\\ \hline PREPPRTNNT & Contraction of preposition and &
& \\ &neuter tonic pronoun & Yes& daquilo\\ \hline PREPPRNTN &
Contraction of preposition and & & \\ &tonic pronoun & Yes&
daqueloutra\\ \hline PREPTNTN & Contraction of preposition and & & \\
& two tonic pronouns & Yes& nestoutra\\ \hline PREPNUM & Contraction
of preposition and & & \\ & numeral & Yes& dunha\\ \hline PREDETDET &
Contraction of predeterminer and & & \\ & determiner & Yes& tódalas\\
\hline INTADVDET & Contraction of adverbial interrogative and & & \\
& determiner & Yes& u-la\\ \hline DETDETM & Contraction of two masculine
determiners & Yes& ámbolos\\ \hline DETDETF & Contraction of two
feminine determiners& Yes& ámbalas\\ \hline PRNPRN & Contraction of
two tonic pronouns & Yes& esoutra\\ \hline PRNPRN & Contraction of two
proclitic pronouns & Yes& chas\\ \hline CNJCDET & Contraction of
co-ordinating conjunction and & & \\ & determiner & Yes& maila\\
\hline CNJSUB & Contraction of subordinating conjunction and & &
\\ & determiner & Yes& cás\\ \hline \hline \multicolumn{4}{l}{}\\
%p{0.50\textwidth}
\end{longtable}
\end{footnotesize}


\newpage
\chapter{Abbreviations used in the text}
\label{se:apendiceabrev}
\begin{description}
\item[ANSI] American National Standards Institute; when used
informally in the expression \emph{ANSI text}, it refers to a text
encoded in any of the encodings of one byte per character defined in
the standard ISO-8859 \cite{Unicode}.
\item[ca] ISO 639 two-letter code\footnote{See
  \texttt{\url{http://www.w3.org/WAI/ER/IG/ert/iso639.htm}}} for
  Catalan
\item[DTD] Document type definition in XML
\item[es] ISO 639 two-letter code for Spanish
\item[eu] ISO 639 two-letter code for Basque
\item[LF] Lexical form (see page~\pageref{pg:FSFL})
\item[TLLF] Target language lexical form
\item[SLLF] Source language lexical form
\item[SF] Surface form (see page~\pageref{pg:FSFL})
\item[gl] ISO 639 two-letter code for Galician
\item[pt] ISO 639 two-letter code for Portuguese
\item[HTML] Hypertext markup language
\item[TL] Target language
\item[SL] Source language
\item[RTF] Rich text format
\item[MT] Machine translation
\item[XML] Extensible markup language
\item[POS] Part of speech
\nota{order alfabetically}
\end{description}

\newpage \nota{Afegir article de l'EAMT 2005 i citar-lo}
\bibliography{documentation} \bibliographystyle{plain}
\addcontentsline{toc}{chapter}{Bibliografía}
\end{document}