1 \input texinfo @c -*- mode: texinfo; coding: us-ascii; -*-
2 @c This file is part of GNU Libidn.
3 @c See below for copyright and license.
5 @setfilename libidn.info
13 This manual is last updated @value{UPDATED} for version
14 @value{VERSION} of GNU Libidn.
16 Copyright @copyright{} 2002, 2003, 2004, 2005, 2006 Simon Josefsson.
19 Permission is granted to copy, distribute and/or modify this document
20 under the terms of the GNU Free Documentation License, Version 1.2 or
21 any later version published by the Free Software Foundation; with the
22 Invariant Sections being ``Commercial Support'', no Front-Cover Texts,
23 and no Back-Cover Texts. A copy of the license is included in the
24 section entitled ``GNU Free Documentation License''.
28 @dircategory GNU Libraries
30 * libidn: (libidn). Internationalized string processing library.
33 @dircategory GNU utilities
35 * idn: (libidn)Invoking idn. Command line interface to GNU Libidn.
40 * IDN Library: (libidn)Emacs API. Emacs API for IDN functions.
45 @subtitle Internationalized string processing for the GNU system
46 @subtitle for version @value{VERSION}, @value{UPDATED}
47 @author Simon Josefsson
49 @vskip 0pt plus 1filll
63 * Introduction:: How to use this manual.
64 * Preparation:: What you should do before using the library.
65 * Utility Functions:: Unicode transformation utility functions.
66 * Stringprep Functions:: Stringprep functions.
67 * Punycode Functions:: Punycode functions.
68 * IDNA Functions:: IDNA functions.
69 * TLD Functions:: TLD functions.
70 * PR29 Functions:: Detect strings non-idempotent under NFKC.
71 * Examples:: Demonstrate how to use the library.
72 * Invoking idn:: Command line interface to the library.
73 * Emacs API:: Emacs Lisp API for Libidn.
74 * Java API:: Notes on the Java port of Libidn.
75 * C# API:: Notes on the C# port of Libidn.
76 * Acknowledgements:: Whom to blame.
77 * Milestones:: Rough outline of development history.
82 * Function and Variable Index::
86 * PR29 discussion:: Implementation aspects of the PR29 flaw.
87 * Library Copying:: How you can copy and share GNU Libidn.
88 * Copying This Manual:: How you can copy and share this manual.
96 GNU Libidn is an implementation of the Stringprep, Punycode and IDNA
97 specifications defined by the IETF Internationalized Domain Names
98 (IDN) working group, used for internationalized domain names. The
99 package is available under the GNU Lesser General Public License.
101 The library contains a generic Stringprep implementation that does
102 Unicode 3.2 NFKC normalization, mapping and prohibitation of
103 characters, and bidirectional character handling. Profiles for
104 Nameprep, iSCSI, SASL and XMPP are included. Punycode and ASCII
105 Compatible Encoding (ACE) via IDNA are supported. A mechanism to
106 define Top-Level Domain (TLD) specific validation tables, and to
107 compare strings against those tables, is included. Default tables for
108 some TLDs are also included.
110 The Stringprep API consists of two main functions, one for converting
111 data from the system's native representation into UTF-8, and one
112 function to perform the Stringprep processing. Adding a new
113 Stringprep profile for your application within the API is
114 straightforward. The Punycode API consists of one encoding function
115 and one decoding function. The IDNA API consists of the ToASCII and
116 ToUnicode functions, as well as an high-level interface for converting
117 entire domain names to and from the ACE encoded form. The TLD API
118 consists of one set of functions to extract the TLD name from a domain
119 string, one set of functions to locate the proper TLD table to use
120 based on the TLD name, and core functions to validate a string against
121 a TLD table, and some utility wrappers to perform all the steps in one
124 The library is used by, e.g., GNU SASL and Shishi to process user
125 names and passwords. Libidn can be built into GNU Libc to enable a
126 new system-wide getaddrinfo flag for IDN processing.
128 Libidn is developed for the GNU/Linux system, but runs on over 20 Unix
129 platforms (including Solaris, IRIX, AIX, and Tru64) and Windows.
130 Libidn is written in C and (parts of) the API is accessible from C,
131 C++, Emacs Lisp, Python and Java. A native Java and C# port is also
138 * Supported Platforms::
140 * Commercial Support::
141 * Downloading and Installing::
146 @node Getting Started
147 @section Getting Started
149 This manual documents the library programming interface. All
150 functions and data types provided by the library are explained.
151 Included are also examples, and documentation for the command line
152 tool @file{idn} that provide a quick interface to the library. The
153 Emacs Lisp bindings for the library is also discussed.
155 The reader is assumed to possess basic familiarity with
156 internationalization concepts and network programming in C or C++.
158 This manual can be used in several ways. If read from the beginning
159 to the end, it gives a good introduction into the library and how it
160 can be used in an application. Forward references are included where
161 necessary. Later on, the manual can be used as a reference manual to
162 get just the information needed about any particular interface of the
163 library. Experienced programmers might want to start looking at the
164 examples at the end of the manual (@pxref{Examples}), and then only
165 read up those parts of the interface which are unclear.
170 This library might have a couple of advantages over other libraries
174 @item It's Free Software
175 Anybody can use, modify, and redistribute it under the terms of the
176 GNU Lesser General Public License.
178 @item It's thread-safe
179 No global state is kept in the library. All functions are reentrant.
182 The code is intended to be written in pure ANSI C89. It has been
183 tested on many Unix like operating systems, and Windows.
185 @item It's modularized
186 The library is composed of several modules, and the only interaction
187 between modules is through each modules' public API. If you only need
188 one piece of functionality, it is possible to take the files you need
189 and incorporate them into your own project.
191 @item It's not bloated
192 The design of the library is based on the smallest API necessary to
193 implement the basic functionality. It has been carefully extended
194 with a small number of high-level wrappers to make it comfortable to
195 use the library. However, it does not implement additional
196 functionality just for the sake of completeness.
198 @item It's documented
199 Sadly, not all software comes with documentation these days. This one
204 @node Library Overview
205 @section Library Overview
207 The following illustration show the components that make up Libidn,
208 and how your application relates to the library. In the illustration,
209 various components are shown as boxes. You see the generic StringPrep
210 component, the various StringPrep profiles including Nameprep, the
211 Punycode component, the IDNA component, and the TLD component. The
212 arrows indicate aggregation, e.g., IDNA uses Punycode and Nameprep,
213 and in turn Nameprep uses the generic StringPrep interface. The
214 interfaces to all components are available for applications, no
215 component within the library is hidden from the application.
219 @node Supported Platforms
220 @section Supported Platforms
222 Libidn has at some point in time been tested on the following
227 @item Debian GNU/Linux 3.0 (Woody)
230 GCC 2.95.4 and GNU Make. This is the main development platform.
231 @code{alphaev67-unknown-linux-gnu}, @code{alphaev6-unknown-linux-gnu},
232 @code{arm-unknown-linux-gnu}, @code{armv4l-unknown-linux-gnu},
233 @code{hppa-unknown-linux-gnu}, @code{hppa64-unknown-linux-gnu},
234 @code{i686-pc-linux-gnu}, @code{ia64-unknown-linux-gnu},
235 @code{m68k-unknown-linux-gnu}, @code{mips-unknown-linux-gnu},
236 @code{mipsel-unknown-linux-gnu}, @code{powerpc-unknown-linux-gnu},
237 @code{s390-ibm-linux-gnu}, @code{sparc-unknown-linux-gnu},
238 @code{sparc64-unknown-linux-gnu}.
240 @item Debian GNU/Linux 2.1
243 GCC 2.95.1 and GNU Make. @code{armv4l-unknown-linux-gnu}.
248 Tru64 UNIX C compiler and Tru64 Make. @code{alphaev67-dec-osf5.1},
249 @code{alphaev68-dec-osf5.1}.
254 GCC 2.96 and GNU Make. @code{alphaev6-unknown-linux-gnu},
255 @code{alphaev67-unknown-linux-gnu}.
257 @item SuSE Linux 7.2a
260 GCC 3.0 and GNU Make. @code{ia64-unknown-linux-gnu}.
265 GCC 3.2.2 and GNU Make. @code{x86_64-unknown-linux-gnu} (AMD64
268 @item SuSE Enterprise Server 9 on IBM OpenPower 720
270 @cindex OpenPower 720
272 GCC 3.3.3 and GNU Make. @code{powerpc64-unknown-linux-gnu}.
274 @item RedHat Linux 7.2
277 GCC 2.96 and GNU Make. @code{alphaev6-unknown-linux-gnu},
278 @code{alphaev67-unknown-linux-gnu}, @code{ia64-unknown-linux-gnu}.
280 @item RedHat Linux 8.0
283 GCC 3.2 and GNU Make. @code{i686-pc-linux-gnu}.
285 @item RedHat Advanced Server 2.1
286 @cindex RedHat Advanced Server
288 GCC 2.96 and GNU Make. @code{i686-pc-linux-gnu}.
290 @item Slackware Linux 8.0.01
293 GCC 2.95.3 and GNU Make. @code{i686-pc-linux-gnu}.
295 @item Mandrake Linux 9.0
298 GCC 3.2 and GNU Make. @code{i686-pc-linux-gnu}.
303 MIPS C compiler, IRIX Make. @code{mips-sgi-irix6.5}.
308 IBM C for AIX compiler, AIX Make. @code{rs6000-ibm-aix4.3.2.0}.
310 @item Microsoft Windows 2000 (Cygwin)
313 GCC 3.2, GNU make. @code{i686-pc-cygwin}.
318 HP-UX C compiler and HP Make. @code{ia64-hp-hpux11.22},
319 @code{hppa2.0w-hp-hpux11.11}.
321 @item SUN Solaris 2.7
324 GCC 3.0.4 and GNU Make. @code{sparc-sun-solaris2.7}.
326 @item SUN Solaris 2.8
329 Sun WorkShop Compiler C 6.0 and SUN Make. @code{sparc-sun-solaris2.8}.
331 @item SUN Solaris 2.9
334 Sun Forte Developer 7 C compiler and GNU
335 Make. @code{sparc-sun-solaris2.9}.
340 GCC 2.95.3 and GNU Make. @code{alpha-unknown-netbsd1.6},
341 @code{i386-unknown-netbsdelf1.6}.
343 @item OpenBSD 3.1 and 3.2
346 GCC 2.95.3 and GNU Make. @code{alpha-unknown-openbsd3.1},
347 @code{i386-unknown-openbsd3.1}.
349 @item FreeBSD 4.7 and 4.8
352 GCC 2.95.4 and GNU Make. @code{alpha-unknown-freebsd4.7},
353 @code{alpha-unknown-freebsd4.8}, @code{i386-unknown-freebsd4.7},
354 @code{i386-unknown-freebsd4.8}.
356 @item MacOS X 10.2 Server Edition
359 GCC 3.1 and GNU Make. @code{powerpc-apple-darwin6.5}.
361 @item MacOS X 10.4 ``Tiger'' with Xcode 2.0
364 GCC 4.0 and GNU Make. @code{powerpc-apple-darwin8.0}.
366 @item Cross compiled to uClinux/uClibc on Motorola Coldfire
367 @cindex Motorola Coldfire
371 GCC 3.4 and GNU Make @code{m68k-uclinux-elf}.
373 @item Cross compiled to ARM using Glibc
376 GCC 2.95 and GNU Make @code{arm-linux}.
378 @item Cross compiled to Mingw32.
383 GCC 3.4.4 and GNU Make @code{i586-mingw32msvc}.
387 If you use Libidn on, or port Libidn to, a new platform please report
391 @section Getting help
393 A mailing list where users of Libidn may help each other exists, and
394 you can reach it by sending e-mail to @email{help-libidn@@gnu.org}.
395 Archives of the mailing list discussions, and an interface to manage
396 subscriptions, is available through the World Wide Web at
397 @url{http://lists.gnu.org/mailman/listinfo/help-libidn}.
399 @node Commercial Support
400 @section Commercial Support
402 Commercial support is available for users of GNU Libidn. The kind of
403 support that can be purchased may include:
407 @item Implement new features.
408 Such as country code specific profiling to support a restricted subset
411 @item Port Libidn to new platforms.
412 This could include porting Libidn to an embedded platforms that may
413 need memory or size optimization.
415 @item Integrating IDN support in your existing project.
417 @item System design of components related to IDN.
421 If you are interested, please write to:
424 Simon Josefsson Datakonsult
429 E-mail: simon@josefsson.org
432 If your company provide support related to GNU Libidn and would like
433 to be mentioned here, contact the author (@pxref{Bug Reports}).
435 @node Downloading and Installing
436 @section Downloading and Installing
440 The package can be downloaded from several places, including:
442 @url{http://josefsson.org/libidn/releases/}
444 The latest version is stored in a file, e.g.,
445 @samp{gsasl-@value{VERSION}.tar.gz} where the @samp{@value{VERSION}}
446 value is the highest version number in the directory.
448 The package is then extracted, configured and built like many other
449 packages that use Autoconf. For detailed information on configuring
450 and building it, refer to the @file{INSTALL} file that is part of the
451 distribution archive.
453 Here is an example terminal session that download, configure, build
454 and install the package. You will need a few basic tools, such as
455 @samp{sh}, @samp{make} and @samp{cc}.
458 $ wget -q http://josefsson.org/libidn/releases/libidn-@value{VERSION}.tar.gz
459 $ tar xfz libidn-@value{VERSION}.tar.gz
460 $ cd libidn-@value{VERSION}/
469 After that Libidn should be properly installed and ready for use.
471 A few @code{configure} options may be relevant, summarized in the
477 Build the Java port into a *.JAR file. @xref{Java API}, for more
481 Disable the TLD module. This would typically only be useful if you
482 are building on a memory restricted platforms. @xref{TLD Functions},
483 for more information.
485 @item --enable-csharp[=IMPL]
486 Build the C3 port into a *.DLL file. @xref{C# API}, for more
487 information. Here, @code{IMPL} is @code{pnet} or @code{mono},
488 indicating whether the PNET @command{cscc} compiler or the Mono
489 @command{mcs} compiler should be used, respectively.
493 For the complete list, refer to the output from @code{configure
498 @cindex Reporting Bugs
500 If you think you have found a bug in Libidn, please investigate it and
505 @item Please make sure that the bug is really in Libidn, and
506 preferably also check that it hasn't already been fixed in the latest
509 @item You have to send us a test case that makes it possible for us to
512 @item You also have to explain what is wrong; if you get a crash, or
513 if the results printed are not good and in that case, in what way.
514 Make sure that the bug report includes all information you would need
515 to fix this kind of bug for someone else.
519 Please make an effort to produce a self-contained report, with
520 something definite that can be tested or debugged. Vague queries or
521 piecemeal messages are difficult to act on and don't help the
524 If your bug report is good, we will do our best to help you to get a
525 corrected version of the software; if the bug report is poor, we won't
526 do anything about it (apart from asking you to send better bug
529 If you think something in this manual is unclear, or downright
530 incorrect, or if the language needs to be improved, please also send a
533 Send your bug report to:
535 @center @samp{bug-libidn@@gnu.org}
539 @section Contributing
543 If you want to submit a patch for inclusion -- from solve a typo you
544 discovered, up to adding support for a new feature -- you should
545 submit it as a bug report (@pxref{Bug Reports}). There are some
546 things that you can do to increase the chances for it to be included
547 in the official package.
549 Unless your patch is very small (say, under 10 lines) we require that
550 you assign the copyright of your work to the Free Software Foundation.
551 This is to protect the freedom of the project. If you have not
552 already signed papers, we will send you the necessary information when
553 you submit your contribution.
555 For contributions that doesn't consist of actual programming code, the
556 only guidelines are common sense. Use it.
558 For code contributions, a number of style guides will help you:
563 Follow the GNU Standards document (@pxref{top, GNU Coding Standards,,
566 If you normally code using another coding standard, there is no
567 problem, but you should use @samp{indent} to reformat the code
568 (@pxref{top, GNU Indent,, indent}) before submitting your work.
570 @item Use the unified diff format @samp{diff -u}.
573 No reason whatsoever should abort the execution of the library. Even
574 memory allocation errors, e.g. when malloc return NULL, should work
575 although result in an error code.
577 @item Design with thread safety in mind.
578 Don't use global variables and the like.
580 @item Avoid using the C math library.
581 It causes problems for embedded implementations, and in most
582 situations it is very easy to avoid using it.
584 @item Document your functions.
585 Use comments before each function headers, that, if properly
586 formatted, are extracted into GTK-DOC web pages. Don't forget to
587 update the Texinfo manual as well.
589 @item Supply a ChangeLog and NEWS entries, where appropriate.
593 @c **********************************************************
594 @c ******************* Preparation ************************
595 @c **********************************************************
599 To use `Libidn', you have to perform some changes to your sources and
600 the build system. The necessary changes are small and explained in
601 the following sections. At the end of this chapter, it is described
602 how the library is initialized, and how the requirements of the
603 library are verified.
605 A faster way to find out how to adapt your application for use with
606 `Libidn' may be to look at the examples at the end of this manual
613 * Building the source::
620 The library contains a few independent parts, and each part export the
621 interfaces (data types and functions) in a header file. You must
622 include the appropriate header files in all programs using the
623 library, either directly or through some other header file, like this:
626 #include <stringprep.h>
629 The header files and the functions they define are categorized as
635 The low-level stringprep API entry point. For IDN applications, this
636 is usually invoked via IDNA. Some applications, specifically non-IDN
637 ones, may want to prepare strings directly though, and should include
640 The name space of the stringprep part of Libidn is @code{stringprep*}
641 for function names, @code{Stringprep*} for data types and
642 @code{STRINGPREP_*} for other symbols. In addition,
643 @code{_stringprep*} is reserved for internal use and should never be
644 used by applications.
648 The entry point to Punycode encoding and decoding functions. Normally
649 punycode is used via the idna.h interface, but some application may
650 want to perform raw punycode operations.
652 The name space of the punycode part of Libidn is @code{punycode_*} for
653 function names, @code{Punycode*} for data types and @code{PUNYCODE_*}
654 for other symbols. In addition, @code{_punycode*} is reserved for
655 internal use and should never be used by applications.
658 The entry point to the IDNA functions. This is the normal entry point
659 for applications that need IDN functionality.
661 The name space of the IDNA part of Libidn is @code{idna_*} for
662 function names, @code{Idna*} for data types and @code{IDNA_*} for
663 other symbols. In addition, @code{_idna*} is reserved for internal
664 use and should never be used by applications.
668 The entry point to the TLD functions. Normal applications are not
669 expected to need this functionality, but it is present for
670 applications that are used by TLDs to validate customer input.
672 The name space of the TLD part of Libidn is @code{tld_*} for function
673 names, @code{Tld_*} for data types and @code{TLD_*} for other symbols.
674 In addition, @code{_tld*} is reserved for internal use and should
675 never be used by applications.
679 The entry point to the PR29 functions. These functions are used to
680 detect ``problem sequences'' (@pxref{PR29 Functions}), mostly for use
681 in security critical applications.
683 The name space of the PR29 part of Libidn is @code{pr29_*} for
684 function names, @code{Pr29_*} for data types and @code{PR29_*} for
685 other symbols. In addition, @code{_pr29*} is reserved for internal
686 use and should never be used by applications.
691 @section Initialization
693 Libidn is stateless and does not need any initialization.
696 @section Version Check
698 It is often desirable to check that the version of `Libidn' used is
699 indeed one which fits all requirements. Even with binary
700 compatibility new features may have been introduced but due to problem
701 with the dynamic linker an old version is actually used. So you may
702 want to check that the version is okay right after program startup.
704 @include texi/stringprep_check_version.texi
706 The normal way to use the function is to put something similar to the
707 following first in your @code{main}:
710 if (!stringprep_check_version (STRINGPREP_VERSION))
712 printf ("stringprep_check_version() failed:\n"
713 "Header file incompatible with shared library.\n");
718 @node Building the source
719 @section Building the source
720 @cindex Compiling your application
722 If you want to compile a source file including e.g. the `idna.h' header
723 file, you must make sure that the compiler can find it in the
724 directory hierarchy. This is accomplished by adding the path to the
725 directory in which the header file is located to the compilers include
726 file search path (via the @option{-I} option).
728 However, the path to the include file is determined at the time the
729 source is configured. To solve this problem, `Libidn' uses the
730 external package @command{pkg-config} that knows the path to the
731 include file and other configuration options. The options that need
732 to be added to the compiler invocation at compile time are output by
733 the @option{--cflags} option to @command{pkg-config libidn}. The
734 following example shows how it can be used at the command line:
737 gcc -c foo.c `pkg-config libidn --cflags`
740 Adding the output of @samp{pkg-config libidn --cflags} to the
741 compilers command line will ensure that the compiler can find e.g. the
744 A similar problem occurs when linking the program with the library.
745 Again, the compiler has to find the library files. For this to work,
746 the path to the library files has to be added to the library search
747 path (via the @option{-L} option). For this, the option
748 @option{--libs} to @command{pkg-config libidn} can be used. For
749 convenience, this option also outputs all other options that are
750 required to link the program with the `libidn' libarary. The example
751 shows how to link @file{foo.o} with the `libidn' library to a program
755 gcc -o foo foo.o `pkg-config libidn --libs`
758 Of course you can also combine both examples to a single command by
759 specifying both options to @command{pkg-config}:
762 gcc -o foo foo.c `pkg-config libidn --cflags --libs`
766 @section Autoconf tests
767 @cindex Autoconf tests
768 @cindex Configure tests
770 If your project uses Autoconf (@pxref{top, GNU Autoconf,, autoconf})
771 to check for installed libraries, you might find the following snippet
772 illustrative. It add a new @file{configure} parameter
773 @code{--with-libidn}, and check for @file{idna.h} and @samp{-lidn}
774 (possibly below the directory specified as the optional argument to
775 @code{--with-libidn}), and define the @acronym{CPP} symbol
776 @code{LIBIDN} if the library is found. The default behaviour is to
777 search for the library and enable the functionality (that is, define
778 the symbol) when the library is found, but if you wish to make the
779 default behaviour of your package be that Libidn is not used (even if
780 it is installed on the system), change @samp{libidn=yes} to
781 @samp{libidn=no} on the third line.
784 AC_ARG_WITH(libidn, AC_HELP_STRING([--with-libidn=[DIR]],
785 [Support IDN (needs GNU Libidn)]),
786 libidn=$withval, libidn=yes)
787 if test "$libidn" != "no"; then
788 if test "$libidn" != "yes"; then
789 LDFLAGS="$@{LDFLAGS@} -L$libidn/lib"
790 CPPFLAGS="$@{CPPFLAGS@} -I$libidn/include"
792 AC_CHECK_HEADER(idna.h,
793 AC_CHECK_LIB(idn, stringprep_check_version,
794 [libidn=yes LIBS="$@{LIBS@} -lidn"], libidn=no),
797 if test "$libidn" != "no" ; then
798 AC_DEFINE(LIBIDN, 1, [Define to 1 if you want IDN support.])
800 AC_MSG_WARN([Libidn not found])
802 AC_MSG_CHECKING([if Libidn should be used])
803 AC_MSG_RESULT($libidn)
806 If you require that your users have installed @code{pkg-config} (which
807 I cannot recommend generally), the above can be done more easily as
811 AC_ARG_WITH(libidn, AC_HELP_STRING([--with-libidn=[DIR]],
812 [Support IDN (needs GNU Libidn)]),
813 libidn=$withval, libidn=yes)
814 if test "$libidn" != "no" ; then
815 PKG_CHECK_MODULES(LIBIDN, libidn >= 0.0.0, [libidn=yes], [libidn=no])
816 if test "$libidn" != "yes" ; then
818 AC_MSG_WARN([Libidn not found])
821 AC_DEFINE(LIBIDN, 1, [Define to 1 if you want Libidn.])
824 AC_MSG_CHECKING([if Libidn should be used])
825 AC_MSG_RESULT($libidn)
828 @c **********************************************************
829 @c ******************** Utility Functions ******************
830 @c **********************************************************
831 @node Utility Functions
832 @chapter Utility Functions
833 @cindex Utility Functions
835 The rest of this library makes extensive use of Unicode characters.
836 In order to interface this library with the outside world, your
837 application may need to make various Unicode transformations.
839 @section Header file @code{stringprep.h}
841 To use the functions explained in this chapter, you need to include
842 the file @file{stringprep.h} using:
845 #include <stringprep.h>
848 @section Unicode Encoding Transformation
850 @include texi/stringprep_unichar_to_utf8.texi
851 @include texi/stringprep_utf8_to_unichar.texi
852 @include texi/stringprep_ucs4_to_utf8.texi
853 @include texi/stringprep_utf8_to_ucs4.texi
855 @section Unicode Normalization
857 @include texi/stringprep_ucs4_nfkc_normalize.texi
858 @include texi/stringprep_utf8_nfkc_normalize.texi
860 @section Character Set Conversion
862 @include texi/stringprep_locale_charset.texi
863 @include texi/stringprep_convert.texi
864 @include texi/stringprep_locale_to_utf8.texi
865 @include texi/stringprep_utf8_to_locale.texi
868 @c **********************************************************
869 @c ****************** Stringprep Functions *****************
870 @c **********************************************************
871 @node Stringprep Functions
872 @chapter Stringprep Functions
873 @cindex Stringprep Functions
875 Stringprep describes a framework for preparing Unicode text strings in
876 order to increase the likelihood that string input and string
877 comparison work in ways that make sense for typical users throughout
878 the world. The stringprep protocol is useful for protocol identifier
879 values, company and personal names, internationalized domain names,
880 and other text strings.
882 @section Header file @code{stringprep.h}
884 To use the functions explained in this chapter, you need to include
885 the file @file{stringprep.h} using:
888 #include <stringprep.h>
891 @section Defining A Stringprep Profile
893 Further types and structures are defined for applications that want to
894 specify their own stringprep profile. As these are fairly obscure,
895 and by necessity tied to the implementation, we do not document them
896 here. Look into the @file{stringprep.h} header file, and the
897 @file{profiles.c} source code for the details.
899 @section Control Flags
901 @deftypevr {Stringprep flags} {Stringprep_profile_flags} {STRINGPREP_NO_NFKC}
902 Disable the NFKC normalization, as well as selecting the non-NFKC case
903 folding tables. Usually the profile specifies BIDI and NFKC settings,
904 and applications should not override it unless in special situations.
907 @deftypevr {Stringprep flags} {Stringprep_profile_flags} {STRINGPREP_NO_BIDI}
908 Disable the BIDI step. Usually the profile specifies BIDI and NFKC
909 settings, and applications should not override it unless in special
913 @deftypevr {Stringprep flags} {Stringprep_profile_flags} {STRINGPREP_NO_UNASSIGNED}
914 Make the library return with an error if string contains unassigned
915 characters according to profile.
918 @section Core Functions
920 @include texi/stringprep_4i.texi
921 @include texi/stringprep_4zi.texi
922 @include texi/stringprep.texi
923 @include texi/stringprep_profile.texi
925 @section Error Handling
927 @include texi/stringprep_strerror.texi
929 @section Stringprep Profile Macros
931 @deftypefun {int} stringprep_nameprep_no_unassigned (char * @var{in}, int @var{maxlen})
933 @var{in}: input/ouput array with string to prepare.
935 @var{maxlen}: maximum length of input/output array.
937 Prepare the input UTF-8 string according to the nameprep profile. The
938 AllowUnassigned flag is false, use @code{stringprep_nameprep} for
939 true AllowUnassigned. Returns 0 iff successful, or an error code.
942 @deftypefun {int} stringprep_iscsi (char * @var{in}, int @var{maxlen})
944 @var{in}: input/ouput array with string to prepare.
946 @var{maxlen}: maximum length of input/output array.
948 Prepare the input UTF-8 string according to the draft iSCSI stringprep
949 profile. Returns 0 iff successful, or an error code.
952 @deftypefun {int} stringprep_plain (char * @var{in}, int @var{maxlen})
954 @var{in}: input/ouput array with string to prepare.
956 @var{maxlen}: maximum length of input/output array.
958 Prepare the input UTF-8 string according to the draft SASL ANONYMOUS
959 profile. Returns 0 iff successful, or an error code.
962 @deftypefun {int} stringprep_xmpp_nodeprep (char * @var{in}, int @var{maxlen})
964 @var{in}: input/ouput array with string to prepare.
966 @var{maxlen}: maximum length of input/output array.
968 Prepare the input UTF-8 string according to the draft XMPP node
969 identifier profile. Returns 0 iff successful, or an error code.
972 @deftypefun {int} stringprep_xmpp_resourceprep (char * @var{in}, int @var{maxlen})
974 @var{in}: input/ouput array with string to prepare.
976 @var{maxlen}: maximum length of input/output array.
978 Prepare the input UTF-8 string according to the draft XMPP resource
979 identifier profile. Returns 0 iff successful, or an error code.
982 @c **********************************************************
983 @c ******************* Punycode Functions ******************
984 @c **********************************************************
985 @node Punycode Functions
986 @chapter Punycode Functions
987 @cindex Punycode Functions
989 Punycode is a simple and efficient transfer encoding syntax designed
990 for use with Internationalized Domain Names in Applications. It
991 uniquely and reversibly transforms a Unicode string into an ASCII
992 string. ASCII characters in the Unicode string are represented
993 literally, and non-ASCII characters are represented by ASCII
994 characters that are allowed in host name labels (letters, digits, and
995 hyphens). A general algorithm called Bootstring allows a string of
996 basic code points to uniquely represent any string of code points
997 drawn from a larger set. Punycode is an instance of Bootstring that
998 uses particular parameter values, appropriate for IDNA.
1000 @section Header file @code{punycode.h}
1002 To use the functions explained in this chapter, you need to include
1003 the file @file{punycode.h} using:
1006 #include <punycode.h>
1009 @section Unicode Code Point Data Type
1011 The punycode function uses a special type to denote Unicode code
1012 points. It is guaranteed to always be a 32 bit unsigned integer.
1014 @deftypevr {Punycode Unicode code point} uint32_t punycode_uint
1015 A unsigned integer that hold Unicode code points.
1018 @section Core Functions
1020 Note that the current implementation will fail if the
1021 @code{input_length} exceed 4294967295 (the size of
1022 @code{punycode_uint}). This restriction may be removed in the future.
1023 Meanwhile applications are encouraged to not depend on this problem,
1024 and use @code{sizeof} to initialize @code{input_length} and
1025 @code{output_length}.
1027 The functions provided are the following two entry points:
1029 @include texi/punycode_encode.texi
1030 @include texi/punycode_decode.texi
1032 @section Error Handling
1034 @include texi/punycode_strerror.texi
1036 @c **********************************************************
1037 @c ********************* IDNA Functions *********************
1038 @c **********************************************************
1039 @node IDNA Functions
1040 @chapter IDNA Functions
1041 @cindex IDNA Functions
1043 Until now, there has been no standard method for domain names to use
1044 characters outside the ASCII repertoire. The IDNA document defines
1045 internationalized domain names (IDNs) and a mechanism called IDNA for
1046 handling them in a standard fashion. IDNs use characters drawn from a
1047 large repertoire (Unicode), but IDNA allows the non-ASCII characters
1048 to be represented using only the ASCII characters already allowed in
1049 so-called host names today. This backward-compatible representation is
1050 required in existing protocols like DNS, so that IDNs can be
1051 introduced with no changes to the existing infrastructure. IDNA is
1052 only meant for processing domain names, not free text.
1054 @section Header file @code{idna.h}
1056 To use the functions explained in this chapter, you need to include
1057 the file @file{idna.h} using:
1063 @section Control Flags
1065 The IDNA @code{flags} parameter can take on the following values, or a
1066 bit-wise inclusive or of any subset of the parameters:
1068 @deftypevr {Return code} {Idna_flags} IDNA_ALLOW_UNASSIGNED
1069 Allow unassigned Unicode code points.
1072 @deftypevr {Return code} {Idna_flags} IDNA_USE_STD3_ASCII_RULES
1073 Check output to make sure it is a STD3 conforming host name.
1076 @section Prefix String
1078 @deftypevr {Macro} {#define} IDNA_ACE_PREFIX
1079 String with the official IDNA prefix, @code{xn--}.
1082 @section Core Functions
1084 The idea behind the IDNA function names are as follows: the
1085 @code{idna_to_ascii_4i} and @code{idna_to_unicode_44i} functions are
1086 the core IDNA primitives. The @code{4} indicate that the function
1087 takes UCS-4 strings (i.e., Unicode code points encoded in a 32-bit
1088 unsigned integer type) of the specified length. The @code{i} indicate
1089 that the data is written ``inline'' into the buffer. This means the
1090 caller is responsible for allocating (and deallocating) the string,
1091 and providing the library with the allocated length of the string.
1092 The output length is written in the output length variable. The
1093 remaining functions all contain the @code{z} indicator, which means
1094 the strings are zero terminated. All output strings are allocated by
1095 the library, and must be deallocated by the caller. The @code{4}
1096 indicator again means that the string is UCS-4, the @code{8} means the
1097 strings are UTF-8 and the @code{l} indicator means the strings are
1098 encoded in the encoding used by the current locale.
1100 The functions provided are the following entry points:
1102 @include texi/idna_to_ascii_4i.texi
1103 @include texi/idna_to_unicode_44i.texi
1105 @section Simplified ToASCII Interface
1107 @include texi/idna_to_ascii_4z.texi
1108 @include texi/idna_to_ascii_8z.texi
1109 @include texi/idna_to_ascii_lz.texi
1111 @section Simplified ToUnicode Interface
1113 @include texi/idna_to_unicode_4z4z.texi
1114 @include texi/idna_to_unicode_8z4z.texi
1115 @include texi/idna_to_unicode_8z8z.texi
1116 @include texi/idna_to_unicode_8zlz.texi
1117 @include texi/idna_to_unicode_lzlz.texi
1119 @section Error Handling
1121 @include texi/idna_strerror.texi
1123 @c **********************************************************
1124 @c ********************** TLD Functions *********************
1125 @c **********************************************************
1127 @chapter TLD Functions
1128 @cindex TLD Functions
1130 Organizations that manage some Top Level Domains (@acronym{TLD}s) have
1131 published tables with characters they accept within the domain. The
1132 reason may be to reduce complexity that come from using the full
1133 Unicode range, and to protect themselves from future (backwards
1134 incompatible) changes in the IDN or Unicode specifications. Libidn
1135 implement an infrastructure for defining and checking strings against
1136 such tables. Libidn also ship some tables from @acronym{TLD}s that we
1137 have managed to get permission to use them from. Because these tables
1138 are even less static than Unicode or StringPrep tables, it is likely
1139 that they will be updated from time to time (even in backwards
1140 incompatibe ways). The Libidn interface provide a ``version'' field
1141 for each @acronym{TLD} table, which can be compared for equality to
1142 guarantee the same operation over time.
1144 From a design point of view, you can regard the @acronym{TLD} tables
1145 for IDN as the ``localization'' step that come after the
1146 ``internationalization'' step provided by the IETF standards.
1148 The TLD functionality rely on up-to-date tables. The latest version
1149 of Libidn aim to provide these, but tables with unclear copying
1150 conditions, or generally experimental tables, are not included. Some
1151 such tables can be found at @url{http://tldchk.berlios.de}.
1153 @section Header file @code{tld.h}
1155 To use the functions explained in this chapter, you need to include
1156 the file @file{tld.h} using:
1162 @c @section Data Types
1164 @c @deftp {Data type} {Tld_table_element} @var{start} @var{end}
1166 @c /* Interval of valid code points in the TLD. */
1167 @c struct Tld_table_element
1169 @c uint32_t start; /* Start of range. */
1170 @c uint32_t end; /* End of range, end == start if single. */
1172 @c typedef struct Tld_table_element Tld_table_element;
1174 @c This @code{struct} contain the @var{start} and @var{end} positions
1175 @c (inclusive) of a range. If the range is a single (i.e., starts and
1176 @c ends in the same character), then set @var{end} to the same as
1177 @c @var{start}. This structure is normally used as an array.
1180 @c @deftp {Data type} {Tld_table} @var{name} @var{version} @var{nvalid} @var{valid}
1182 @c /* List valid code points in a TLD. */
1185 @c char *name; /* TLD name, e.g., "no". */
1186 @c char *version; /* Version string from TLD file. */
1187 @c size_t nvalid; /* Number of entries in data. */
1188 @c Tld_table_element *valid[]; /* Sorted array of valid code points. */
1190 @c typedef struct Tld_table Tld_table;
1192 @c In this @code{struct}, the @var{name} field is a string (@samp{char*})
1193 @c indicating the TLD name (e.g., ``no''). The @var{version} field is a
1194 @c string (@samp{char*}) containing a free form humanly readable string
1195 @c that can be used for equality comparison to compare different versions
1196 @c of the table. The @var{nvalid} field indicate how many entries there
1197 @c are in @var{valid}, which brings us finally to @var{valid} that
1198 @c contain the actual code points that are valid for this TLD (see
1199 @c @code{Tld_table_element} above).
1202 @section Core Functions
1204 @include texi/tld_check_4t.texi
1205 @include texi/tld_check_4tz.texi
1207 @section Utility Functions
1209 @include texi/tld_get_4.texi
1210 @include texi/tld_get_4z.texi
1211 @include texi/tld_get_z.texi
1212 @include texi/tld_get_table.texi
1213 @include texi/tld_default_table.texi
1215 @section High-Level Wrapper Functions
1217 @include texi/tld_check_4.texi
1218 @include texi/tld_check_4z.texi
1219 @include texi/tld_check_8z.texi
1220 @include texi/tld_check_lz.texi
1222 @section Error Handling
1224 @include texi/tld_strerror.texi
1226 @c **********************************************************
1227 @c ********************** PR29 Functions ********************
1228 @c **********************************************************
1229 @node PR29 Functions
1230 @chapter PR29 Functions
1231 @cindex PR29 Functions
1233 A deficiency in the specification of Unicode Normalization Forms has
1234 been found. The consequence is that some strings can be normalized
1235 into different strings by different implementations. In other words,
1236 two different implementations may return different output for the same
1237 input (because the interpretation of the specification is
1238 ambiguous). Further, an implementation invoked again on the one of the
1239 output strings may return a different string (because one of the
1240 interpretation of the ambiguous specification make normalization
1241 non-idempotent). Fortunately, only a select few character sequence
1242 exhibit this problem, and none of them are expected to occur in
1243 natural languages (due to different linguistic uses of the involved
1246 A full discussion of the problem may be found at:
1248 @url{http://www.unicode.org/review/pr-29.html}
1250 The PR29 functions below allow you to detect the problem sequence. So
1251 when would you want to use these functions? For most applications,
1252 such as those using Nameprep for IDN, this is likely only to be an
1253 interoperability problem. Thus, you may not want to care about it, as
1254 the character sequences will rarely occur naturally. However, if you
1255 are using a profile, such as SASLPrep, to process authentication
1256 tokens; authorization tokens; or passwords, there is a real danger
1257 that attackers may try to use the peculiarities in these strings to
1258 attack parts of your system. As only a small number of strings, and
1259 no naturally occurring strings, exhibit this problem, the conservative
1260 approach of rejecting the strings is recommended. If this approach is
1261 not used, you should instead verify that all parts of your system,
1262 that process the tokens and passwords, use a NFKC implementation that
1263 produce the same output for the same input.
1265 Technically inclined readers may be interested in knowing more about
1266 the implementation aspects of the PR29 flaw. @xref{PR29 discussion}.
1268 @section Header file @code{pr29.h}
1270 To use the functions explained in this chapter, you need to include
1271 the file @file{pr29.h} using:
1277 @section Core Functions
1279 @include texi/pr29_4.texi
1281 @section Utility Functions
1283 @include texi/pr29_4z.texi
1284 @include texi/pr29_8z.texi
1286 @section Error Handling
1288 @include texi/pr29_strerror.texi
1290 @c **********************************************************
1291 @c *********************** Examples ***********************
1292 @c **********************************************************
1297 This chapter contains example code which illustrate how `Libidn' can
1298 be used when writing your own application.
1301 * Example 1:: Example using stringprep.
1302 * Example 2:: Example using punycode.
1303 * Example 3:: Example using IDNA ToASCII.
1304 * Example 4:: Example using IDNA ToUnicode.
1305 * Example 5:: Example using TLD checking.
1311 This example demonstrates how the stringprep functions are used.
1313 @verbatiminclude ../examples/example.c
1318 This example demonstrates how the punycode functions are used.
1320 @verbatiminclude ../examples/example2.c
1325 This example demonstrates how the library is used to convert
1326 internationalized domain names into ASCII compatible names.
1328 @verbatiminclude ../examples/example3.c
1333 This example demonstrates how the library is used to convert ASCII
1334 compatible names to internationalized domain names.
1336 @verbatiminclude ../examples/example4.c
1341 This example demonstrates how the library is used to check a string
1342 for invalid characters within a specific TLD.
1344 @verbatiminclude ../examples/example5.c
1346 @c **********************************************************
1347 @c ********************* Invoking idn *********************
1348 @c **********************************************************
1350 @chapter Invoking idn
1353 @cindex invoking @command{idn}
1354 @cindex command line
1358 GNU Libidn (idn) -- Internationalized Domain Names command line tool
1360 @section Description
1361 @code{idn} allows internationalized string preparation
1362 (@samp{stringprep}), encoding and decoding of punycode data, and IDNA
1363 ToASCII/ToUnicode operations to be performed on the command line.
1365 If strings are specified on the command line, they are used as input
1366 and the computed output is printed to standard output @code{stdout}.
1367 If no strings are specified on the command line, the program read
1368 data, line by line, from the standard input @code{stdin}, and print
1369 the computed output to standard output. What processing is performed
1370 (e.g., ToASCII, or Punycode encode) is indicated by options. If any
1371 errors are encountered, the execution of the applications is aborted.
1373 All strings are expected to be encoded in the preferred charset used
1374 by your locale. Use @code{--debug} to find out what this charset is.
1375 You can override the charset used by setting environment variable
1378 To process a string that starts with @code{-}, for example
1379 @code{-foo}, use @code{--} to signal the end of parameters, as in
1380 @code{idn --quiet -a -- -foo}.
1383 @code{idn} recognizes these commands:
1386 -h, --help Print help and exit
1388 -V, --version Print version and exit
1390 -s, --stringprep Prepare string according to nameprep profile
1392 -d, --punycode-decode Decode Punycode
1394 -e, --punycode-encode Encode Punycode
1396 -a, --idna-to-ascii Convert to ACE according to IDNA (default)
1398 -u, --idna-to-unicode Convert from ACE according to IDNA
1400 --allow-unassigned Toggle IDNA AllowUnassigned flag (default=off)
1402 --usestd3asciirules Toggle IDNA UseSTD3ASCIIRules flag (default=off)
1404 -t, --tld Check string for TLD specific rules
1405 Only for --idna-to-ascii and --idna-to-unicode
1408 -p, --profile=STRING Use specified stringprep profile instead
1409 Valid stringprep profiles are `Nameprep',
1410 `iSCSI', `Nodeprep', `Resourceprep', `trace', and
1413 --debug Print debugging information (default=off)
1415 --quiet Silent operation (default=off)
1418 @section Environment Variables
1420 The @var{CHARSET} environment variable can be used to override what
1421 character set to be used for decoding incoming data (i.e., on the
1422 command line or on the standard input stream), and to encode data to
1423 the standard output. If your system is set up correctly, however, the
1424 application will guess which character set is used automatically.
1428 $ CHARSET=ISO-8859-1 idn --punycode-encode
1434 Standard usage, reading input from standard input:
1439 Copyright 2002, 2003 Simon Josefsson.
1440 GNU Libidn comes with NO WARRANTY, to the extent permitted by law.
1441 You may redistribute copies of GNU Libidn under the terms of
1442 the GNU Lesser General Public License. For more information
1443 about these matters, see the file named COPYING.LIB.
1444 Type each input string on a line by itself, terminated by a newline character.
1445 r@"aksm@"org@aa{}s.se
1446 xn--rksmrgs-5wao1o.se
1450 Reading input from command line, and disabling copyright and license
1454 jas@@latte:~$ idn --quiet r@"aksm@"org@aa{}s.se bl@aa{}b@ae{}rgr@o{}d.no
1455 xn--rksmrgs-5wao1o.se
1456 xn--blbrgrd-fxak7p.no
1460 Accessing a specific StringPrep profile directly:
1463 jas@@latte:~$ idn --quiet --profile=SASLprep --stringprep te@ss{}t@ordf{}
1468 @section Troubleshooting
1470 Getting character data encoded right, and making sure Libidn use the
1471 same encoding, can be difficult. The reason for this is that most
1472 systems encode character data in more than one character encoding,
1473 i.e., using @code{UTF-8} together with @code{ISO-8859-1} or
1474 @code{ISO-2022-JP}. This problem is likely to continue to exist until
1475 only one character encoding come out as the evolutionary winner, or
1476 (more likely, at least to some extents) forever.
1478 The first step to troubleshooting character encoding problems with
1479 Libidn is to use the @samp{--debug} parameter to find out which
1480 character set encoding @samp{idn} believe your locale uses.
1483 jas@@latte:~$ idn --debug --quiet ""
1484 system locale uses charset `UTF-8'.
1489 If it prints @code{ANSI_X3.4-1968} (i.e., @code{US-ASCII}), this
1490 indicate you have not configured your locale properly. To configure
1491 the locale, you can, for example, use @samp{LANG=sv_SE.UTF-8; export
1492 LANG} at a @code{/bin/sh} prompt, to set up your locale for a Swedish
1493 environment using @code{UTF-8} as the encoding.
1495 Sometimes @samp{idn} appear to be unable to translate from your system
1496 locale into @code{UTF-8} (which is used internally), and you get an
1497 error like the following:
1500 jas@@latte:~$ idn --quiet foo
1501 idn: could not convert from ISO-8859-1 to UTF-8.
1505 The simplest explanation is that you haven't installed the
1506 @samp{iconv} conversion tools. You can find it as a standalone
1507 library in @acronym{GNU} Libiconv
1508 (@uref{http://www.gnu.org/software/libiconv/}). On many
1509 @acronym{GNU}/Linux systems, this library is part of the system, but
1510 you may have to install additional packages (e.g., @samp{glibc-locale}
1511 for Debian) to be able to use it.
1513 Another explanation is that the error is correct and you are feeding
1514 @samp{idn} invalid data. This can happen inadvertently if you are not
1515 careful with the character set encodings you use. For example, if
1516 your shell run in a @code{ISO-8859-1} environment, and you invoke
1517 @samp{idn} with the @samp{CHARSET} environment variable as follows,
1518 you will feed it @code{ISO-8859-1} characters but force it to believe
1519 they are @code{UTF-8}. Naturally this will lead to an error, unless
1520 the byte sequences happen to be parsable as @code{UTF-8}. Note that
1521 even if you don't get an error, the output may be incorrect in this
1522 situation, because @code{ISO-8859-1} and @code{UTF-8} does not in
1523 general encode the same characters as the same byte sequences.
1526 jas@@latte:~$ idn --quiet --debug ""
1527 system locale uses charset `ISO-8859-1'.
1529 jas@@latte:~$ CHARSET=UTF-8 idn --quiet --debug r@"aksm@"org@aa{}s
1530 system locale uses charset `UTF-8'.
1559 The sense moral here is to forget about @samp{CHARSET} (configure your
1560 locales properly instead) unless you know what you are doing, and if
1561 you want to use it, do it carefully, after verifying with
1562 @samp{--debug} that you get the desired results.
1567 Included in Libidn are @file{punycode.el} and @file{idna.el} that
1568 provides an Emacs Lisp API to (a limited set of) the Libidn API. This
1569 section describes the API. Currently the IDNA API always set the
1570 @code{UseSTD3ASCIIRules} flag and clear the @code{AllowUnassigned}
1571 flag, in the future there may be functionality to specify these flags
1574 @section Punycode Emacs API
1576 @defvar punycode-program
1577 Name of the GNU Libidn @file{idn} application. The default is
1578 @samp{idn}. This variable can be customized.
1581 @defvar punycode-environment
1582 List of environment variable definitions prepended to
1583 @samp{process-environment}. The default is @samp{("CHARSET=UTF-8")}.
1584 This variable can be customized.
1587 @defvar punycode-encode-parameters
1588 List of parameters passed to @var{punycode-program} to invoke punycode
1589 encoding mode. The default is @samp{("--quiet" "--punycode-encode")}.
1590 This variable can be customized.
1593 @defvar punycode-decode-parameters
1594 Parameters passed to @var{punycode-program} to invoke punycode
1595 decoding mode. The default is @samp{("--quiet" "--punycode-decode")}.
1596 This variable can be customized.
1599 @defun punycode-encode string
1600 Returns a Punycode encoding of the @var{string}, after converting the
1604 @defun punycode-decode string
1605 Returns a possibly multibyte string which is the decoding of the
1606 @var{string} which is a punycode encoded string.
1609 @section IDNA Emacs API
1611 @defvar idna-program
1612 Name of the GNU Libidn @file{idn} application. The default is
1613 @samp{idn}. This variable can be customized.
1616 @defvar idna-environment
1617 List of environment variable definitions prepended to
1618 @samp{process-environment}. The default is @samp{("CHARSET=UTF-8")}.
1619 This variable can be customized.
1622 @defvar idna-to-ascii-parameters
1623 List of parameters passed to @var{idna-program} to invoke IDNA ToASCII
1624 mode. The default is @samp{("--quiet" "--idna-to-ascii"
1625 "--usestd3asciirules")}. This variable can be customized.
1628 @defvar idna-to-unicode-parameters
1629 Parameters passed @var{idna-program} to invoke IDNA ToUnicode mode.
1630 The default is @samp{("--quiet" "--idna-to-unicode"
1631 "--usestd3asciirules")}. This variable can be customized.
1634 @defun idna-to-ascii string
1635 Returns an ASCII Compatible Encoding (ACE) of the string computed by
1636 the IDNA ToASCII operation on the input @var{string}, after converting
1640 @defun idna-to-unicode string
1641 Returns a possibly multibyte string which is the output of the IDNA
1642 ToUnicode operation computed on the input @var{string}.
1648 Libidn has been ported to the Java programming language, and as a
1649 consequence most of the API is available to native Java applications.
1650 This section contain notes on this support, complete documentation is
1653 The Java library, if Libidn has been built with Java support
1654 (@pxref{Downloading and Installing}), will be placed in
1655 @file{java/libidn.jar}. The source code is located in
1656 @file{java/gnu/inet/encoding/}.
1660 This package provides a Java implementation of the Internationalized
1661 Domain Names in Applications (IDNA) standard. It is written entirely
1662 in Java and does not require any additional libraries to be set up.
1664 The gnu.inet.encoding.IDNA class offers two public functions, toASCII
1665 and toUnicode which can be used as follows:
1668 gnu.inet.encoding.IDNA.toASCII("bl@"ods.z@"ug");
1669 gnu.inet.encoding.IDNA.toUnicode("xn--blds-6qa.xn--zg-xka");
1672 @section Miscellaneous Programs
1674 The @file{misc/} directory contains several programs that are related
1675 to the Java part of GNU Libidn, but that don't need to be included in
1676 the main source tree.
1678 @subsection GenerateRFC3454
1680 This program parses RFC3454 and creates the RFC3454.java program that
1681 is required during the StringPrep phase.
1683 The RFC can be found at various locations, for example at
1684 @url{http://www.ietf.org/rfc/rfc3454.txt}.
1686 Invoke the program as follows:
1689 $ java GenerateRFC3454
1690 Creating RFC3454.java... Ok.
1693 @subsection GenerateNFKC
1695 The GenerateNFKC program parses the Unicode character database file
1696 and generates all the tables required for NFKC. This program requires
1697 the two files UnicodeData.txt and CompositionExclusions.txt of version
1698 3.2 of the Unicode files. Note that RFC3454 (Stringprep) defines that
1699 Unicode version 3.2 is to be used, not the latest version.
1701 The Unicode data files can be found at
1702 @url{http://www.unicode.org/Public/}.
1704 Invoke the program as follows:
1708 Creating CombiningClass.java... Ok.
1709 Creating DecompositionKeys.java... Ok.
1710 Creating DecompositionMappings.java... Ok.
1711 Creating Composition.java... Ok.
1714 @subsection TestIDNA
1716 The TestIDNA program allows to test the IDNA implementation manually
1717 or against Simon Josefsson's test vectors.
1719 The test vectors can be found at the Libidn homepage,
1720 @url{http://www.gnu.org/software/libidn/}.
1722 To test the tranformation manually, use:
1725 $ java -cp .:../libidn.jar TestIDNA -a <string to test>
1726 Input: <string to test>
1727 Output: <toASCII(string to test)>
1728 $ java -cp .:../libidn.jar TestIDNA -u <string to test>
1729 Input: <string to test>
1730 Output: <toUnicode(string to test)>
1733 To test against draft-josefsson-idn-test-vectors.html, use:
1736 $ java -cp .:../libidn.jar TestIDNA -t
1740 @subsection TestNFKC
1742 The TestNFKC program allows to test the NFKC implementation manually
1743 or against the NormalizationTest.txt file from the Unicode data files.
1745 To test the normalization manually, use:
1748 $ java -cp .:../libidn.jar TestNFKC <string to test>
1749 Input: <string to test>
1750 Output: <nfkc version of the string to test>
1753 To test against NormalizationTest.txt:
1756 $ java -cp .:../libidn.jar TestNFKC
1760 @section Possible Problems
1762 Beware of Bugs: This Java API needs a lot more testing, especially
1763 with "exotic" character sets. While it works for me, it may not work
1766 Encoding of your Java sources: If you are using non-ASCII characters
1767 in your Java source code, make sure javac compiles your programs with
1768 the correct encoding. If necessary specify the encoding using the
1769 -encoding parameter.
1771 Java Unicode handling: Java 1.4 only handles 16-bit Unicode code
1772 points (i.e. characters in the Basic Multilingual Plane), this
1773 implementation therefore ignores all references to so-called
1774 Supplementary Characters (U+10000 to U+10FFFF). Starting from Java
1775 1.5, these characters will also be supported by Java, but this will
1776 require changes to this library. See also the next section.
1778 @section A Note on Java and Unicode
1780 This library uses Java's builtin 'char' datatype. Up to Java 1.4, this
1781 datatype only supports 16-bit Unicode code points, also called the
1782 Basic Multilingual Plane. For this reason, this library doesn't work
1783 for Supplementary Characters (i.e. characters from U+10000 to
1784 U+10FFFF). All references to such characters are silently ignored.
1786 Starting from Java 1.5, also Supplementary Characters will be
1787 supported. However, this will require changes in the present version
1788 of the library. Java 1.5 is currently in beta status.
1790 For more information refer to the documentation of java.lang.Character
1796 The Libidn library has been ported to the C# language. The port
1797 reside in the top-level @file{csharp/} directory. Currently, no
1798 further documentation about the implementation or the API is
1799 available. However, the C# port was based on the Java port, and the
1800 API is exactly the same as in the Java version. The help files for
1801 the Java API may thus be useful.
1803 @c **********************************************************
1804 @c ******************* Acknowledgements *******************
1805 @c **********************************************************
1806 @node Acknowledgements
1807 @chapter Acknowledgements
1809 The punycode implementation was taken from the IETF IDN Punycode
1810 specification, by Adam M. Costello. The TLD code was contributed by
1811 Thomas Jacob. The Java implementation was contributed by Oliver Hitz.
1812 The C# implementation was contributed by Alexander Gnauck. The
1813 Unicode tables were provided by Unicode, Inc. Some functions for
1814 dealing with Unicode (see nfkc.c and toutf8.c) were borrowed from
1815 GLib, downloaded from @url{http://www.gtk.org/}. The manual borrowed
1816 text from Libgcrypt by Werner Koch.
1818 Inspiration for many things that, consciously or not, have gone into
1819 this package is due to a number of free software package that the
1820 author has been exposed to. The author wishes to acknowledge the free
1821 software community in general, for giving an example on what role
1822 software development can play in the modern society.
1824 Several people reported bugs, sent patches or suggested improvements,
1825 see the file THANKS in the top-level directory of the source code.
1827 @c **********************************************************
1828 @c ********************** Milestones **********************
1829 @c **********************************************************
1833 The complete history of user visible changes is stored in the file
1834 @file{NEWS} in the top-level directory of the source code tree. The
1835 complete history of modifications to each file is stored in the file
1836 @file{ChangeLog} in the same directory. This section contain a
1837 condensed version of that information, in the form of ``milestones''
1841 @item Stringprep implementation.
1842 Version 0.0.0 released on 2002-11-05.
1844 @item IDNA and Punycode implementations, part of the GNU project.
1845 Version 0.1.0 released on 2003-01-05.
1847 @item Uses official IDNA ACE prefix 'xn--'.
1848 Version 0.1.7 released on 2003-02-12.
1850 @item Command line interface.
1851 Version 0.1.11 released on 2003-02-26.
1853 @item GNU Libc add-on proposed.
1854 Version 0.1.12 released on 2003-03-06.
1856 @item Interoperability testing during IDNConnect.
1857 Version 0.3.1 released on 2003-10-02.
1859 @item TLD restriction testing.
1860 Version 0.4.0 released on 2004-02-28.
1862 @item GNU Libc add-on integrated.
1863 Version 0.4.1 released on 2004-03-08.
1865 @item Native Java implementation.
1866 Version 0.4.2-0.4.9 released between 2004-03-20 and 2004-06-11.
1868 @item PR-29 functions for ``problem sequences''.
1869 Version 0.5.0 released on 2004-06-26.
1871 @item Many small portability fixes and wider use.
1872 Version 0.5.1 through 0.5.20, released between 2004-07-09 and
1875 @item Native C# implementation.
1876 Version 0.6.0 released on 2005-12-03.
1881 @unnumbered Concept Index
1885 @node Function and Variable Index
1886 @unnumbered Function and Variable Index
1890 @node PR29 discussion
1891 @appendix PR29 discussion
1893 If you wish to experiment with a modified Unicode NFKC implementation
1894 according to the PR29 proposal, you may find the following bug report
1895 useful. However, I have not verified that the suggested modifications
1896 are correct. For reference, I'm including my response to the report
1900 From: Rick McGowan <rick@unicode.org>
1901 Subject: Possible bug and status of PR 29 change(s)
1902 To: bug-libidn@gnu.org
1903 Date: Wed, 27 Oct 2004 14:49:17 -0700
1905 Hello. On behalf of the Unicode Consortium editorial committee, I would
1906 like to find out more information about the PR 29 fixes, if any, and
1907 functions in Libidn. Your implementation was listed in the text of PR29 as
1908 needing investigation, so I am following up on several implementations.
1910 The UTC has accepted the proposed fix to D2 as outlined in PR29, and a new
1911 draft of UAX #15 has been issued.
1913 I have looked at Libidn 0.5.8 (today), and there may still be a possible
1914 bug in NFKC.java and nfkc.c.
1916 ------------------------------------------------------
1918 1. In NFKC.java, this line in canonicalOrdering():
1920 if (i > 0 && (last_cc == 0 || last_cc != cc)) {
1922 should perhaps be changed to:
1924 if (i > 0 && (last_cc == 0 || last_cc < cc)) {
1926 but I'm not sure of the sense of this comparison.
1928 ------------------------------------------------------
1930 2. In nfkc.c, function _g_utf8_normalize_wc() has this code:
1933 (last_cc == 0 || last_cc != cc) &&
1934 combine (wc_buffer[last_start], wc_buffer[i],
1935 &wc_buffer[last_start]))
1938 This appears to have the same bug as the current Python implementation (in
1939 Python 2.3.4). The code should be checking, as per new rule D2 UAX #15
1940 update, that the next combining character is the same or HIGHER than the
1941 current one. It now checks to see if it's non-zero and not equal.
1943 The above line(s) should perhaps be changed to:
1946 (last_cc == 0 || last_cc < cc) &&
1947 combine (wc_buffer[last_start], wc_buffer[i],
1948 &wc_buffer[last_start]))
1951 but I'm not sure of the sense of the comparison (< or > or <=?) here.
1953 In the text of PR29, I will be marking Libidn as "needs change" and adding
1954 the version number that I checked. If any further change is made, please
1955 let me know the release version, and I'll update again.
1962 From: Simon Josefsson <jas@extundo.com>
1963 Subject: Re: Possible bug and status of PR 29 change(s)
1964 To: Rick McGowan <rick@unicode.org>
1965 Cc: bug-libidn@gnu.org
1966 Date: Thu, 28 Oct 2004 09:47:47 +0200
1968 Rick McGowan <rick@unicode.org> writes:
1970 > Hello. On behalf of the Unicode Consortium editorial committee, I would
1971 > like to find out more information about the PR 29 fixes, if any, and
1972 > functions in Libidn. Your implementation was listed in the text of PR29 as
1973 > needing investigation, so I am following up on several implementations.
1975 > The UTC has accepted the proposed fix to D2 as outlined in PR29, and a new
1976 > draft of UAX #15 has been issued.
1978 > I have looked at Libidn 0.5.8 (today), and there may still be a possible
1979 > bug in NFKC.java and nfkc.c.
1983 I believe the current behavior is intentional. Libidn do not aim to
1984 implement latest-and-greatest NFKC, it aim to implement the NFKC
1985 functionality required for StringPrep and IDN. As you may know,
1986 StringPrep/IDN reference Unicode 3.2.0, and explicitly says any later
1987 changes (which I consider PR29 as) do not apply.
1989 In fact, I believe that would I incorporate the changes suggested in
1990 PR29, I would in fact be violating the IDN specifications.
1992 Thanks for looking into the code and finding the place where the
1993 change could be made. I'll see if I can mention this in the manual
1994 somewhere, for technically interested readers.
2002 @node Copying This Manual
2003 @appendix Copying This Manual
2006 * GNU Free Documentation License:: License for copying this manual.