Move manual to doc/.

[libidn.git] / doc / libidn.texi

\input texinfo   @c -*-texinfo-*-
@c This file is part of the GNU Libidn Manual.
@c Copyright (C) 2002, 2003 Simon Josefsson
@c See below for copying conditions.

@setfilename libidn.info
@include version.texi
@settitle GNU Libidn @value{VERSION}

@syncodeindex pg cp

@copying
This manual is for GNU Libidn version @value{VERSION},
@value{UPDATED}.

Copyright @copyright{} 2002, 2003 Simon Josefsson.

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.1 or
any later version published by the Free Software Foundation; with no
Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,''
and with the Back-Cover Texts as in (a) below.  A copy of the
license is included in the section entitled ``GNU Free Documentation
License.''

(a) The FSF's Back-Cover Text is: ``You have freedom to copy and modify
this GNU Manual, like GNU software.  Copies published by the Free
Software Foundation raise funds for GNU development.''
@end quotation
@end copying

@dircategory GNU Libraries
@direntry
* libidn: (libidn).     Internationalized string processing library.
@end direntry

@dircategory GNU utilities
@direntry
* idn: (libidn)Invoking idn.            Command line interface to GNU Libidn.
@end direntry

@dircategory Emacs
@direntry
* IDN Library: (libidn)Emacs API.       Emacs API for IDN functions.
@end direntry

@titlepage
@title GNU Libidn
@subtitle for version @value{VERSION}, @value{UPDATED}
@author Simon Josefsson (@email{bug-libidn@@gnu.org})
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top GNU Libidn

@insertcopying
@end ifnottex

@menu
* Introduction::                How to use this manual.
* Preparation::                 What you should do before using the library.
* Stringprep Functions::        Stringprep functions.
* Punycode Functions::          Punycode functions.
* IDNA Functions::              IDNA functions.
* Examples::                    Demonstrate how to use the library.
* Invoking idn::                Command line interface to the library.
* Emacs API::                   Emacs Lisp API for Libidn.
* Acknowledgements::            Whom to blame.

Indices

* Concept Index::
* Function and Variable Index::

Appendices

* Library Copying::             How you can copy and share GNU Libidn.
* Copying This Manual::         How you can copy and share this manual.

@end menu


@node Introduction
@chapter Introduction

GNU Libidn is an implementation of the Stringprep, Punycode and IDNA
specifications defined by the IETF Internationalized Domain Names
(IDN) working group, used for internationalized domain names.  The
package is available under the GNU Lesser General Public License.

The library contains a generic Stringprep implementation that does
Unicode 3.2 NFKC normalization, mapping and prohibitation of
characters, and bidirectional character handling.  Profiles for iSCSI,
Kerberos 5, Nameprep, SASL and XMPP are included.  Punycode and ASCII
Compatible Encoding (ACE) via IDNA are supported.

The Stringprep API consists of two main functions, one for converting
data from the system's native representation into UTF-8, and one
function to perform the Stringprep processing.  Adding a new
Stringprep profile for your application within the API is
straightforward.  The Punycode API consists of one encoding function
and one decoding function.  The IDNA API consists of the ToASCII and
ToUnicode functions, as well as an high-level interface for converting
entire domain names to and from the ACE encoded form.

The library is used by, e.g., GNU SASL and Shishi to process user
names and passwords.  Libidn can be built into GNU Libc to enable a
new system-wide getaddrinfo() flag for IDN processing.

Libidn is developed for the GNU/Linux system, but runs on over 20 Unix
platforms (including Solaris, IRIX, AIX, and Tru64) and Windows.
Libidn is written in C and (parts of) the API is accessible from C,
C++, Emacs Lisp, Python and Java.

@menu
* Getting Started::
* Features::
* Supported Platforms::
* Bug Reports::
@end menu

@node Getting Started
@section Getting Started

This manual documents the library programming interface.  All
functions and data types provided by the library are explained.

The reader is assumed to possess basic familiarity with
internationalization concepts and network programming in C or C++.

This manual can be used in several ways.  If read from the beginning
to the end, it gives a good introduction into the library and how it
can be used in an application.  Forward references are included where
necessary.  Later on, the manual can be used as a reference manual to
get just the information needed about any particular interface of the
library.  Experienced programmers might want to start looking at the
examples at the end of the manual (@pxref{Examples}), and then only
read up those parts of the interface which are unclear.

@node Features
@section Features

This library might have a couple of advantages over other libraries
doing a similar job.

@table @asis
@item It's Free Software
Anybody can use, modify, and redistribute it under the terms of the
GNU Lesser General Public License.

@item It's thread-safe
No global state is kept in the library.

@item It's portable
It should work on all Unix like operating systems, including Windows.

@end table

@node Supported Platforms
@section Supported Platforms

Libidn has at some point in time been tested on the following
platforms.

@enumerate

@item Debian GNU/Linux 3.0 (Woody)
@cindex Debian

GCC 2.95.4 and GNU Make. This is the main development platform.
@code{alphaev67-unknown-linux-gnu}, @code{alphaev6-unknown-linux-gnu},
@code{arm-unknown-linux-gnu}, @code{hppa-unknown-linux-gnu},
@code{hppa64-unknown-linux-gnu}, @code{i686-pc-linux-gnu},
@code{ia64-unknown-linux-gnu}, @code{m68k-unknown-linux-gnu},
@code{mips-unknown-linux-gnu}, @code{mipsel-unknown-linux-gnu},
@code{powerpc-unknown-linux-gnu}, @code{s390-ibm-linux-gnu},
@code{sparc-unknown-linux-gnu}.

@item Debian GNU/Linux 2.1
@cindex Debian

GCC 2.95.1 and GNU Make. @code{armv4l-unknown-linux-gnu}.

@item Tru64 UNIX
@cindex Tru64

Tru64 UNIX C compiler and Tru64 Make. @code{alphaev67-dec-osf5.1},
@code{alphaev68-dec-osf5.1}.

@item SuSE Linux 7.1
@cindex SuSE

GCC 2.96 and GNU Make. @code{alphaev6-unknown-linux-gnu},
@code{alphaev67-unknown-linux-gnu}.

@item SuSE Linux 7.2a
@cindex SuSE Linux

GCC 3.0 and GNU Make. @code{ia64-unknown-linux-gnu}.

@item RedHat Linux 7.2
@cindex RedHat

GCC 2.96 and GNU Make. @code{alphaev6-unknown-linux-gnu},
@code{alphaev67-unknown-linux-gnu}, @code{ia64-unknown-linux-gnu}.

@item RedHat Linux 8.0
@cindex RedHat

GCC 3.2 and GNU Make. @code{i686-pc-linux-gnu}.

@item RedHat Advanced Server 2.1
@cindex RedHat Advanced Server

GCC 2.96 and GNU Make. @code{i686-pc-linux-gnu}.

@item Slackware Linux 8.0.01
@cindex RedHat

GCC 2.95.3 and GNU Make. @code{i686-pc-linux-gnu}.

@item Mandrake Linux 9.0
@cindex Mandrake

GCC 3.2 and GNU Make. @code{i686-pc-linux-gnu}.

@item IRIX 6.5
@cindex IRIX

MIPS C compiler, IRIX Make. @code{mips-sgi-irix6.5}.

@item AIX 4.3.2
@cindex AIX

IBM C for AIX compiler, AIX Make.  @code{rs6000-ibm-aix4.3.2.0}.

@item Microsoft Windows 2000 (Cygwin)
@cindex Windows

GCC 3.2, GNU make. @code{i686-pc-cygwin}.

@item HP-UX 11
@cindex HP-UX

HP-UX C compiler and HP Make. @code{ia64-hp-hpux11.22},
@code{hppa2.0w-hp-hpux11.11}.

@item SUN Solaris 2.8
@cindex Solaris

Sun WorkShop Compiler C 6.0 and SUN Make. @code{sparc-sun-solaris2.8}.

@item NetBSD 1.6
@cindex NetBSD

GCC 2.95.3 and GNU Make. @code{alpha-unknown-netbsd1.6},
@code{i386-unknown-netbsdelf1.6}.

@item OpenBSD 3.1 and 3.2
@cindex OpenBSD

GCC 2.95.3 and GNU Make. @code{alpha-unknown-openbsd3.1},
@code{i386-unknown-openbsd3.1}.

@item FreeBSD 4.7
@cindex FreeBSD

GCC 2.95.4 and GNU Make. @code{alpha-unknown-freebsd4.7},
@code{i386-unknown-freebsd4.7}.

@end enumerate

If you use Libidn on, or port Libidn to, a new platform please report
it to the author.

@node Bug Reports
@section Bug Reports
@cindex Reporting Bugs

If you think you have found a bug in Libidn, please investigate it and
report it.

@itemize @bullet

@item Please make sure that the bug is really in Libidn, and
preferably also check that it hasn't already been fixed in the latest
version.

@item You have to send us a test case that makes it possible for us to
reproduce the bug.

@item You also have to explain what is wrong; if you get a crash, or
if the results printed are not good and in that case, in what way.
Make sure that the bug report includes all information you would need
to fix this kind of bug for someone else.

@end itemize

Please make an effort to produce a self-contained report, with
something definite that can be tested or debugged.  Vague queries or
piecemeal messages are difficult to act on and don't help the
development effort.

If your bug report is good, we will do our best to help you to get a
corrected version of the software; if the bug report is poor, we won't
do anything about it (apart from asking you to send better bug
reports).

If you think something in this manual is unclear, or downright
incorrect, or if the language needs to be improved, please also send a
note.

Send your bug report to:

@center @samp{bug-libidn@@gnu.org}


@c **********************************************************
@c *******************  Preparation  ************************
@c **********************************************************
@node Preparation
@chapter Preparation

To use `Libidn', you have to perform some changes to your sources and
the build system.  The necessary changes are small and explained in
the following sections.  At the end of this chapter, it is described
how the library is initialized, and how the requirements of the
library are verified.

A faster way to find out how to adapt your application for use with
`Libidn' may be to look at the examples at the end of this manual
(@pxref{Examples}).

@menu
* Header::
* Initialization::
* Version Check::
* Building the source::
@end menu

@node Header
@section Header

The library contains a few independent parts, and each part export the
interfaces (data types and functions) in a header file.  You must
include the appropriate header files in all programs using the
library, either directly or through some other header file, like this:

@example
#include <stringprep.h>
@end example

The header files and the functions they define are categorized as
follows:

@table @asis
@item stringprep.h

The low-level stringprep API entry point.  For IDN applications, this
is usually invoked via IDNA. Some applications, specifically non-IDN
ones, may want to prepare strings directly though, and should include
this header file.

The name space of the stringprep part of Libidn is @code{stringprep*}
for function names, @code{Stringprep*} for data types and
@code{STRINGPREP_*} for other symbols.  In addition the same name
prefixes with one prepended underscore are reserved for internal use
and should never be used by an application.

@item punycode.h

The entry point to Punycode encoding and decoding functions.  Normally
punycode is used via the idna.h interface, but some application may
want to perform raw punycode operations.

The name space of the punycode part of Libidn is @code{punycode_*} for
function names, @code{Punycode*} for data types and @code{PUNYCODE_*}
for other symbols.  In addition the same name prefixes with one
prepended underscore are reserved for internal use and should never be
used by an application.

@item idna.h

The entry point to the IDNA functions.  This is the normal entry point
for applications that need IDN functionality.

The name space of the IDNA part of Libidn is @code{idna_*} for
function names, @code{Idna*} for data types and @code{IDNA_*} for
other symbols.  In addition the same name prefixes with one prepended
underscore are reserved for internal use and should never be used by
an application.

@end table

@node Initialization
@section Initialization

Libidn is stateless and does not need any initialization.

@node Version Check
@section Version Check

It is often desirable to check that the version of `Libidn' used is
indeed one which fits all requirements.  Even with binary
compatibility new features may have been introduced but due to problem
with the dynamic linker an old version is actually used.  So you may
want to check that the version is okay right after program startup.

@deftypefun {const char *} stringprep_check_version (const char * @var{req_version})

@var{req_version}:  Required version number, or NULL.

Check that the the version of the library is at minimum the requested one
and return the version string; return NULL if the condition is not
satisfied.  If a NULL is passed to this function, no check is done,
but the version string is simply returned.

See @var{STRINGPREP_VERSION} for a suitable @code{req_version} string.

 Version string of run-time library, or NULL if the
run-time library does not meet the required version number.

@end deftypefun

The normal way to use the function is to put something similar to the
following first in your @code{main()}:

@example
  if (!stringprep_check_version (STRINGPREP_VERSION))
    @{
      printf ("stringprep_check_version() failed:\n"
              "Header file incompatible with shared library.\n");
      exit(1);
    @}
@end example

@node Building the source
@section Building the source
@cindex Compiling your application

If you want to compile a source file including e.g. the `idna.h' header
file, you must make sure that the compiler can find it in the
directory hierarchy.  This is accomplished by adding the path to the
directory in which the header file is located to the compilers include
file search path (via the @option{-I} option).

However, the path to the include file is determined at the time the
source is configured.  To solve this problem, `Libidn' uses the
external package @command{pkg-config} that knows the path to the
include file and other configuration options.  The options that need
to be added to the compiler invocation at compile time are output by
the @option{--cflags} option to @command{pkg-config libidn}.  The
following example shows how it can be used at the command line:

@example
gcc -c foo.c `pkg-config libidn --cflags`
@end example

Adding the output of @samp{pkg-config libidn --cflags} to the
compilers command line will ensure that the compiler can find e.g. the
idna.h header file.

A similar problem occurs when linking the program with the library.
Again, the compiler has to find the library files.  For this to work,
the path to the library files has to be added to the library search
path (via the @option{-L} option).  For this, the option
@option{--libs} to @command{pkg-config libidn} can be used.  For
convenience, this option also outputs all other options that are
required to link the program with the `libidn' libarary.  The example
shows how to link @file{foo.o} with the `libidn' library to a program
@command{foo}.

@example
gcc -o foo foo.o `pkg-config libidn --libs`
@end example

Of course you can also combine both examples to a single command by
specifying both options to @command{pkg-config}:

@example
gcc -o foo foo.c `pkg-config libidn --cflags --libs`
@end example

@c **********************************************************
@c ******************  Stringprep Functions *****************
@c **********************************************************
@node Stringprep Functions
@chapter Stringprep Functions
@cindex Stringprep Functions

Stringprep describes a framework for preparing Unicode text strings in
order to increase the likelihood that string input and string
comparison work in ways that make sense for typical users throughout
the world. The stringprep protocol is useful for protocol identifier
values, company and personal names, internationalized domain names,
and other text strings.

@defcv {Enumerated type} Stringprep_profile_flags STRINGPREP_NO_NFKC
STRINGPREP_NO_NFKC disables the NFKC normalization, as well as
selecting the non-NFKC case folding tables.  Usually the profile
specifies BIDI and NFKC settings.
@end defcv

@defcv {Enumerated type} Stringprep_profile_flags STRINGPREP_NO_BIDI
STRINGPREP_NO_BIDI disables the BIDI step.  Usually the profile
specifies BIDI and NFKC settings.
@end defcv

@defcv {Enumerated type} Stringprep_profile_flags STRINGPREP_NO_UNASSIGNED
STRINGPREP_NO_UNASSIGNED causes stringprep() abort with an error if
string contains unassigned characters according to profile.
@end defcv

@deftypefun {int} stringprep (char * @var{in}, size_t @var{maxlen}, int @var{flags}, Stringprep_profile * @var{profile})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

@var{flags}:  optional stringprep profile flags.

@var{profile}:  pointer to stringprep profile to use.

Prepare the input UTF-8 string according to the stringprep profile.
Normally application programmers use stringprep profile macros such
as @code{stringprep_nameprep()}, @code{stringprep_kerberos5()} etc instead of
calling this function directly.

Since the stringprep operation can expand the string, @code{maxlen}
indicate how large the buffer holding the string is.  The @code{flags}
are one of Stringprep_profile_flags, or 0.  The profile indicates
processing details, see the profile header files, such as
stringprep_generic.h and stringprep_nameprep.h for two examples.
Your application can define new profiles, possibly re-using the
generic stringprep tables that always will be part of the library.
Note that you must convert strings entered in the systems locale
into UTF-8 before using this function.

 Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_profile (char * @var{in}, char ** @var{out}, char * @var{profile}, int @var{flags})

@var{in}:  input/ouput array with string to prepare.

@var{out}:  output variable with newly allocate string.

@var{profile}:  name of stringprep profile to use.

@var{flags}:  optional stringprep profile flags.

Prepare the input UTF-8 string according to the stringprep profile.
Normally application programmers use stringprep profile macros such
as @code{stringprep_nameprep()}, @code{stringprep_kerberos5()} etc instead of
calling this function directly.

Note that you must convert strings entered in the systems locale
into UTF-8 before using this function.

The output @code{out} variable must be deallocated by the caller.

 Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {uint32_t} stringprep_utf8_to_unichar (const char * @var{p})

@var{p}:  a pointer to Unicode character encoded as UTF-8

Converts a sequence of bytes encoded as UTF-8 to a Unicode character.
If @code{p} does not point to a valid UTF-8 encoded character, results are
undefined.

 the resulting character

@end deftypefun

@deftypefun {int} stringprep_unichar_to_utf8 (uint32_t @var{c}, char * @var{outbuf})

@var{c}:  a ISO10646 character code

@var{outbuf}:  output buffer, must have at least 6 bytes of space.
If @var{NULL}, the length will be computed and returned
and nothing will be written to @code{outbuf}.

Converts a single character to UTF-8.

 number of bytes written

@end deftypefun

@deftypefun {uint32_t *} stringprep_utf8_to_ucs4 (const char * @var{str}, ssize_t @var{len}, size_t * @var{items_written})

@var{str}:  a UTF-8 encoded string

@var{len}:  the maximum length of @code{str} to use. If @code{len} < 0, then
the string is nul-terminated.

@var{items_written}:  location to store the number of characters in the
result, or @var{NULL}.

Convert a string from UTF-8 to a 32-bit fixed width
representation as UCS-4, assuming valid UTF-8 input.
This function does no error checking on the input.

 a pointer to a newly allocated UCS-4 string.
This value must be freed with @code{free()}.

@end deftypefun

@deftypefun {char *} stringprep_ucs4_to_utf8 (const uint32_t * @var{str}, ssize_t @var{len}, size_t * @var{items_read}, size_t * @var{items_written})

@var{str}:  a UCS-4 encoded string

@var{len}:  the maximum length of @code{str} to use. If @code{len} < 0, then
the string is terminated with a 0 character.

@var{items_read}:  location to store number of characters read read, or @var{NULL}.

@var{items_written}:  location to store number of bytes written or @var{NULL}.
The value here stored does not include the trailing 0
byte.

Convert a string from a 32-bit fixed width representation as UCS-4.
to UTF-8. The result will be terminated with a 0 byte.

 a pointer to a newly allocated UTF-8 string.
This value must be freed with @code{free()}. If an
error occurs, @var{NULL} will be returned and
@code{error} set.

@end deftypefun

@deftypefun {char *} stringprep_utf8_nfkc_normalize (const char * @var{str}, ssize_t @var{len})

@var{str}:  a UTF-8 encoded string.

@var{len}:  length of @code{str}, in bytes, or -1 if @code{str} is nul-terminated.

Converts a string into canonical form, standardizing
such issues as whether a character with an accent
is represented as a base character and combining
accent or as a single precomposed character. You
should generally call @code{g_utf8_normalize()} before
comparing two Unicode strings.

The normalization mode is NFKC (ALL COMPOSE).  It standardizes
differences that do not affect the text content, such as the
above-mentioned accent representation. It standardizes the
"compatibility" characters in Unicode, such as SUPERSCRIPT THREE to
the standard forms (in this case DIGIT THREE). Formatting
information may be lost but for most text operations such
characters should be considered the same. It returns a result with
composed forms rather than a maximally decomposed form.

 a newly allocated string, that is the
NFKC normalized form of @code{str}.

@end deftypefun

@deftypefun {uint32_t *} stringprep_ucs4_nfkc_normalize (uint32_t * @var{str}, ssize_t @var{len})

@var{str}:  a Unicode string.

@var{len}:  length of @code{str} array, or -1 if @code{str} is nul-terminated.

Converts UCS4 string into UTF-8 and runs
@code{stringprep_utf8_nfkc_normalize()}.

 a newly allocated Unicode string, that is the NFKC
normalized form of @code{str}.

@end deftypefun

@deftypefun {const char *} stringprep_locale_charset ( @var{void})


 Return the character set used by the system locale.
It will never return NULL, but use "ASCII" as a fallback.

@end deftypefun

@deftypefun {char *} stringprep_convert (const char * @var{str}, const char * @var{to_codeset}, const char * @var{from_codeset})

@var{str}:  input zero-terminated string.

@var{to_codeset}:  name of destination character set.

@var{from_codeset}:  name of origin character set, as used by @code{str}.

Convert the string from one character set to another using the
system's @code{iconv()} function.

 Returns newly allocated zero-terminated string which
is @code{str} transcoded into to_codeset.

@end deftypefun

@deftypefun {char *} stringprep_locale_to_utf8 (const char * @var{str})

@var{str}:  input zero terminated string.

Convert string encoded in the locale's character set into UTF-8 by
using @code{stringprep_convert()}.

 Returns newly allocated zero-terminated string which
is @code{str} transcoded into UTF-8.

@end deftypefun

@deftypefun {char *} stringprep_utf8_to_locale (const char * @var{str})

@var{str}:  input zero terminated string.

Convert string encoded in UTF-8 into the locale's character set by
using @code{stringprep_convert()}.

 Returns newly allocated zero-terminated string which
is @code{str} transcoded into the locale's character set.

@end deftypefun

@deftypefun {int} stringprep_nameprep_no_unassigned (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the nameprep profile.
The AllowUnassigned flag is false, use @code{stringprep_nameprep()} for
true AllowUnassigned.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_iscsi (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the draft iSCSI
stringprep profile.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_kerberos5 (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the draft Kerberos5
stringprep profile.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_plain (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the draft SASL
ANONYMOUS profile.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_xmpp_nodeprep (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the draft XMPP node
identifier profile.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_xmpp_resourceprep (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to the draft XMPP resource
identifier profile.  Returns 0 iff successful, or an error code.

@end deftypefun

@deftypefun {int} stringprep_generic (char * @var{in}, int @var{maxlen})

@var{in}:  input/ouput array with string to prepare.

@var{maxlen}:  maximum length of input/output array.

Prepare the input UTF-8 string according to a hypotetical "generic"
stringprep profile. This is mostly used for debugging or when
constructing new stringprep profiles. Returns 0 iff successful, or
an error code.

@end deftypefun

@c **********************************************************
@c *******************  Punycode Functions ******************
@c **********************************************************
@node Punycode Functions
@chapter Punycode Functions
@cindex Punycode Functions

Punycode is a simple and efficient transfer encoding syntax designed
for use with Internationalized Domain Names in Applications. It
uniquely and reversibly transforms a Unicode string into an ASCII
string. ASCII characters in the Unicode string are represented
literally, and non-ASCII characters are represented by ASCII
characters that are allowed in host name labels (letters, digits, and
hyphens). This document defines a general algorithm called Bootstring
that allows a string of basic code points to uniquely represent any
string of code points drawn from a larger set. Punycode is an instance
of Bootstring that uses particular parameter values specified by this
document, appropriate for IDNA.

@deftypefun {enum punycode_status} punycode_encode (size_t @var{input_length}, const uint32_t @var{input[]}, const unsigned char @var{case_flags[]}, size_t * @var{output_length}, char @var{output[]})

@var{input_length}:  The input_length is the number of code points in the input.

@var{output_length}:  The output_length is an in/out argument: the caller
passes in the maximum number of code points that it
can receive, and on successful return it will
contain the number of code points actually output.

Converts Unicode to Punycode.

 The return value can be any of the punycode_status
values defined above except punycode_bad_input; if
not punycode_success, then output_size and output
might contain garbage.

@end deftypefun

@deftypefun {enum punycode_status} punycode_decode (size_t @var{input_length}, const char @var{input[]}, size_t * @var{output_length}, uint32_t @var{output[]}, unsigned char @var{case_flags[]})

@var{input_length}:  The input_length is the number of code points in the input.

@var{output_length}:  The output_length is an in/out argument: the caller
passes in the maximum number of code points that it
can receive, and on successful return it will
contain the actual number of code points output.

Converts Punycode to Unicode.

 The return value can be any of the punycode_status
values defined above; if not punycode_success, then
output_length, output, and case_flags might contain
garbage.  On success, the decoder will never need to
write an output_length greater than input_length,
because of how the encoding is defined.

@end deftypefun

@c **********************************************************
@c ********************* IDNA Functions *********************
@c **********************************************************
@node IDNA Functions
@chapter IDNA Functions
@cindex IDNA Functions

Until now, there has been no standard method for domain names to use
characters outside the ASCII repertoire. The IDNA document defines
internationalized domain names (IDNs) and a mechanism called IDNA for
handling them in a standard fashion. IDNs use characters drawn from a
large repertoire (Unicode), but IDNA allows the non-ASCII characters
to be represented using only the ASCII characters already allowed in
so-called host names today. This backward-compatible representation is
required in existing protocols like DNS, so that IDNs can be
introduced with no changes to the existing infrastructure. IDNA is
only meant for processing domain names, not free text.

@deftypefun {int} idna_to_ascii_4i (const uint32_t * @var{in}, size_t @var{inlen}, char * @var{out}, int @var{flags})

@var{in}:  input array with unicode code points.

@var{inlen}:  length of input array with unicode code points.

@var{out}:  output zero terminated string that must have room for at
least 63 characters plus the terminating zero.

@var{flags}:  IDNA flags, e.g. IDNA_ALLOW_UNASSIGNED or IDNA_USE_STD3_ASCII_RULES.

The ToASCII operation takes a sequence of Unicode code points that make
up one label and transforms it into a sequence of code points in the
ASCII range (0..7F). If ToASCII succeeds, the original sequence and the
resulting sequence are equivalent labels.

It is important to note that the ToASCII operation can fail. ToASCII
fails if any step of it fails. If any step of the ToASCII operation
fails on any label in a domain name, that domain name MUST NOT be used
as an internationalized domain name. The method for deadling with this
failure is application-specific.

The inputs to ToASCII are a sequence of code points, the AllowUnassigned
flag, and the UseSTD3ASCIIRules flag. The output of ToASCII is either a
sequence of ASCII code points or a failure condition.

ToASCII never alters a sequence of code points that are all in the ASCII
range to begin with (although it could fail). Applying the ToASCII
operation multiple times has exactly the same effect as applying it just
once.

 Returns 0 on success, or an error code.

@end deftypefun

@deftypefun {int} idna_to_ascii (const unsigned long * @var{in}, size_t @var{inlen}, char * @var{out}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{in}:  input array with unicode code points.

@var{inlen}:  length of input array with unicode code points.

@var{out}:  output zero terminated string that must have room for at
least 63 characters plus the terminating zero.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

The ToASCII operation takes a sequence of Unicode code points that make
up one label and transforms it into a sequence of code points in the
ASCII range (0..7F). If ToASCII succeeds, the original sequence and the
resulting sequence are equivalent labels.

It is important to note that the ToASCII operation can fail. ToASCII
fails if any step of it fails. If any step of the ToASCII operation
fails on any label in a domain name, that domain name MUST NOT be used
as an internationalized domain name. The method for deadling with this
failure is application-specific.

The inputs to ToASCII are a sequence of code points, the AllowUnassigned
flag, and the UseSTD3ASCIIRules flag. The output of ToASCII is either a
sequence of ASCII code points or a failure condition.

ToASCII never alters a sequence of code points that are all in the ASCII
range to begin with (although it could fail). Applying the ToASCII
operation multiple times has exactly the same effect as applying it just
once.

 Returns 0 on success, or an error code.

@end deftypefun

@deftypefun {int} idna_to_unicode (const unsigned long * @var{in}, size_t @var{inlen}, unsigned long * @var{out}, size_t * @var{outlen}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{in}:  input array with unicode code points.

@var{inlen}:  length of input array with unicode code points.

@var{out}:  output array with unicode code points.

@var{outlen}:  on input, maximum size of output array with unicode code points,
on exit, actual size of output array with unicode code points.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

The ToUnicode operation takes a sequence of Unicode code points
that make up one label and returns a sequence of Unicode code
points. If the input sequence is a label in ACE form, then the
result is an equivalent internationalized label that is not in ACE
form, otherwise the original sequence is returned unaltered.

ToUnicode never fails. If any step fails, then the original input
sequence is returned immediately in that step.

The ToUnicode output never contains more code points than its
input.  Note that the number of octets needed to represent a
sequence of code points depends on the particular character
encoding used.

The inputs to ToUnicode are a sequence of code points, the
AllowUnassigned flag, and the UseSTD3ASCIIRules flag. The output of
ToUnicode is always a sequence of Unicode code points.

 Returns error condition, but it must only be used for
debugging purposes.  The output buffer is always
guaranteed to contain the correct data according to
the specification (sans malloc induced errors).  NB!
This means that you normally ignore the return code
from this function, as checking it means breaking the
standard.

@end deftypefun

@deftypefun {int} idna_to_ascii_from_ucs4 (const unsigned long * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero terminated input Unicode string.

@var{output}:  pointer to newly allocated output string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert UCS-4 domain name to ASCII string.  The domain name may
contain several labels, separated by dots.  The output buffer must
be deallocated by the caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_ascii_from_utf8 (const char * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero terminated input UTF-8 string.

@var{output}:  pointer to newly allocated output string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert UTF-8 domain name to ASCII string.  The domain name may
contain several labels, separated by dots.  The output buffer must
be deallocated by the caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_ascii_from_locale (const char * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero terminated input UTF-8 string.

@var{output}:  pointer to newly allocated output string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert domain name in the locale's encoding to ASCII string.  The
domain name may contain several labels, separated by dots.  The
output buffer must be deallocated by the caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_unicode_ucs4_from_ucs4 (const unsigned long * @var{input}, unsigned long ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero-terminated Unicode string.

@var{output}:  pointer to newly allocated output Unicode string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert possibly ACE encoded domain name in UCS-4 format into a
UCS-4 string.  The domain name may contain several labels,
separated by dots.  The output buffer must be deallocated by the
caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_unicode_ucs4_from_utf8 (const char * @var{input}, unsigned long ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero-terminated UTF-8 string.

@var{output}:  pointer to newly allocated output Unicode string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert possibly ACE encoded domain name in UTF-8 format into a
UCS-4 string.  The domain name may contain several labels,
separated by dots.  The output buffer must be deallocated by the
caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_unicode_utf8_from_utf8 (const char * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero-terminated UTF-8 string.

@var{output}:  pointer to newly allocated output UTF-8 string.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert possibly ACE encoded domain name in UTF-8 format into a
UTF-8 string.  The domain name may contain several labels,
separated by dots.  The output buffer must be deallocated by the
caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_unicode_locale_from_utf8 (const char * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero-terminated UTF-8 string.

@var{output}:  pointer to newly allocated output string encoded in the
current locale's character set.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert possibly ACE encoded domain name in UTF-8 format into a
string encoded in the current locale's character set.  The
The domain name may contain several labels, separated by dots.  The
output buffer must be deallocated by the caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@deftypefun {int} idna_to_unicode_locale_from_locale (const char * @var{input}, char ** @var{output}, int @var{allowunassigned}, int @var{usestd3asciirules})

@var{input}:  zero-terminated string encoded in the current locale's
character set.

@var{output}:  pointer to newly allocated output string encoded in the
current locale's character set.

@var{allowunassigned}:  whether to allow unassigned code points.

@var{usestd3asciirules}:  whether to check input for STD3 compliance.

Convert possibly ACE encoded domain name in the locale's character
set into a string encoded in the current locale's character set.
The domain name may contain several labels, separated by dots.  The
output buffer must be deallocated by the caller.

 Returns IDNA_SUCCESS on success, or error code.

@end deftypefun

@c **********************************************************
@c ***********************  Examples  ***********************
@c **********************************************************
@node Examples
@chapter Examples
@cindex Examples

This chapter contains example code which illustrate how `Libidn' can
be used when writing your own application.

@menu
* Example 1::           Example using stringprep.
* Example 2::           Example using punycode.
* Example 3::           Example using IDNA ToASCII.
* Example 4::           Example using IDNA ToUnicode.
@end menu

@node Example 1
@section Example 1

This example demonstrates how the stringprep functions are used.

@example
@include example.c.texi
@end example


@node Example 2
@section Example 2

This example demonstrates how the punycode functions are used.

@example
@include example2.c.texi
@end example


@node Example 3
@section Example 3

This example demonstrates how the library is used to convert
internationalized domain names into ASCII compatible names.

@example
@include example3.c.texi
@end example


@node Example 4
@section Example 4

This example demonstrates how the library is used to convert ASCII
compatible names to internationalized domain names.

@example
@include example4.c.texi
@end example

@c **********************************************************
@c *********************  Invoking idn  *********************
@c **********************************************************
@node Invoking idn
@chapter Invoking idn

@pindex idn
@cindex invoking @command{idn}
@cindex command line

@majorheading Name

GNU Libidn (idn) -- Internationalized Domain Names command line tool

@majorheading Description
@code{idn} is a utility part of GNU Libidn.  It allows preparation of
strings, encoding and decoding of punycode data, and IDNA
ToASCII/ToUnicode operations to be performed on the command line,
without the need to write a program that uses libidn.

Data is read, line by line, from the standard input, and one of the
operations indicated by command parameters are performed and the
output is printed to standard output.  If any errors are encountered,
the execution of the applications is aborted.

@majorheading Options
@code{idn} recognizes these commands:

@verbatim
       -h  --help
              Print help and exit

       -V  --version
              Print version and exit

       -s --stringprep
              Prepare string according to nameprep profile

       -e  --punycode-encode
              Encode UTF-8 to Punycode

       -d  --punycode-decode
              Decode Punycode to UTF-8

       -a  --idna-to-ascii
              Convert UTF-8 to ACE according to IDNA

       -u  --idna-to-unicode
              Convert ACE to UTF-8 according to IDNA

       --allow-unassigned
              Toggle IDNA AllowUnassigned flag (default=off)

       --usestd3asciirules
              Toggle IDNA UseSTD3ASCIIRules flag (default=off)

       -pSTRING   --profile=STRING
              Use specified stringprep profile instead

              Valid stringprep profiles are 'generic', 'Nameprep',
              'KRBprep', 'Nodeprep', 'Resourceprep', 'plain',
              'SASLprep', and 'ISCSIprep'.

       --debug
              Print debugging information (default=off)

       --quiet
              Don't print the welcome greeting (default=off)
@end verbatim

@majorheading Environment Variables

The @var{CHARSET} environment variable can be used to override what
character set to be used for decoding incoming data on the standard
input, and to encode data to the standard output.  If your system is
set up correctly, the application will guess which character set is
used automatically.  Example usage:

@verbatim
$ CHARSET=ISO-8859-1 idn --punycode-encode
...
@end verbatim

@node Emacs API
@chapter Emacs API

Included in Libidn are @file{punycode.el} and @file{idna.el} that
provides an Emacs Lisp API to (a limited set of) the Libidn API.  This
section describes the API.

@defvar punycode-program
Name of the GNU Libidn @file{idn} application.  The default is
@samp{idn}.  This variable can be customized.
@end defvar

@defvar punycode-environment
List of environment variable definitions prepended to
@samp{process-environment}.  The default is @samp{("CHARSET=UTF-8")}.
This variable can be customized.
@end defvar

@defvar punycode-encode-parameters
List of parameters passed to @var{punycode-program} to invoke punycode
encoding mode.  The default is @samp{("--quiet" "--punycode-encode")}.
This variable can be customized.
@end defvar

@defvar punycode-decode-parameters
Parameters passed to @var{punycode-program} to invoke punycode
decoding mode.  The default is @samp{("--quiet" "--punycode-decode")}.
This variable can be customized.
@end defvar

@defun punycode-encode string
Returns a Punycode encoding of the @var{string}, after converting the
input into UTF-8.
@end defun

@defun punycode-decode string
Returns a possibly multibyte string which is the decoding of the
@var{string} which is a punycode encoded string.
@end defun

@defvar idna-program
Name of the GNU Libidn @file{idn} application.  The default is
@samp{idn}.  This variable can be customized.
@end defvar

@defvar idna-environment
List of environment variable definitions prepended to
@samp{process-environment}.  The default is @samp{("CHARSET=UTF-8")}.
This variable can be customized.
@end defvar

@defvar idna-to-ascii-parameters
List of parameters passed to @var{idna-program} to invoke IDNA ToASCII
mode.  The default is @samp{("--quiet" "--idna-to-ascii")}.  This
variable can be customized.
@end defvar

@defvar idna-to-unicode-parameters
Parameters passed @var{idna-program} to invoke IDNA ToUnicode mode.
The default is @samp{("--quiet" "--idna-to-unicode")}.  This variable
can be customized.
@end defvar

@defun idna-to-ascii string
Returns an ASCII Compatible Encoding (ACE) of the string computed by
the IDNA ToASCII operation on the input @var{string}, after converting
the input to UTF-8.
@end defun

@defun idna-to-unicode string
Returns a possibly multibyte string which is the output of the IDNA
ToUnicode operation computed on the input @var{string}.
@end defun

@c **********************************************************
@c *******************  Acknowledgements  *******************
@c **********************************************************
@node Acknowledgements
@chapter Acknowledgements

The punycode code was taken from the IETF IDN Punycode specification,
by Adam M. Costello.

Some functions (see nfkc.c and toutf8.c) has been borrowed from GLib
downloaded from www.gtk.org.

Several people reported bugs, sent patches or suggested improvements,
see the file THANKS.

@node Concept Index
@unnumbered Concept Index

@printindex cp

@node Function and Variable Index
@unnumbered Function and Variable Index

@printindex fn

@include lgpl.texi

@node Copying This Manual
@appendix Copying This Manual

@menu
* GNU Free Documentation License::  License for copying this manual.
@end menu

@include fdl.texi

@bye