1 .\" Copyright (C) Markus Kuhn, 1995, 2001
3 .\" %%%LICENSE_START(GPLv2+_DOC_FULL)
4 .\" This is free documentation; you can redistribute it and/or
5 .\" modify it under the terms of the GNU General Public License as
6 .\" published by the Free Software Foundation; either version 2 of
7 .\" the License, or (at your option) any later version.
9 .\" The GNU General Public License's references to "object code"
10 .\" and "executables" are to be interpreted as the output of any
11 .\" document formatting or typesetting system, including
12 .\" intermediate and printed output.
14 .\" This manual is distributed in the hope that it will be useful,
15 .\" but WITHOUT ANY WARRANTY; without even the implied warranty of
16 .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 .\" GNU General Public License for more details.
19 .\" You should have received a copy of the GNU General Public
20 .\" License along with this manual; if not, see
21 .\" <http://www.gnu.org/licenses/>.
24 .\" 1995-11-26 Markus Kuhn <mskuhn@cip.informatik.uni-erlangen.de>
25 .\" First version written
26 .\" 2001-05-11 Markus Kuhn <mgk25@cl.cam.ac.uk>
29 .TH UNICODE 7 2016-03-15 "GNU" "Linux Programmer's Manual"
31 unicode \- universal character set
33 The international standard ISO 10646 defines the
34 Universal Character Set (UCS).
35 UCS contains all characters of all other character set standards.
36 It also guarantees "round-trip compatibility";
38 conversion tables can be built such that no information is lost
39 when a string is converted from any other encoding to UCS and back.
41 UCS contains the characters required to represent practically all
43 This includes not only the Latin, Greek, Cyrillic,
44 Hebrew, Arabic, Armenian, and Georgian scripts, but also Chinese,
45 Japanese and Korean Han ideographs as well as scripts such as
46 Hiragana, Katakana, Hangul, Devanagari, Bengali, Gurmukhi, Gujarati,
47 Oriya, Tamil, Telugu, Kannada, Malayalam, Thai, Lao, Khmer, Bopomofo,
48 Tibetan, Runic, Ethiopic, Canadian Syllabics, Cherokee, Mongolian,
49 Ogham, Myanmar, Sinhala, Thaana, Yi, and others.
51 covered, research on how to best encode them for computer usage is
52 still going on and they will be added eventually.
54 eventually include not only Hieroglyphs and various historic
55 Indo-European languages, but even some selected artistic scripts such
56 as Tengwar, Cirth, and Klingon.
57 UCS also covers a large number of
58 graphical, typographical, mathematical, and scientific symbols,
59 including those provided by TeX, Postscript, APL, MS-DOS, MS-Windows,
60 Macintosh, OCR fonts, as well as many word processing and publishing
61 systems, and more are being added.
63 The UCS standard (ISO 10646) describes a
64 31-bit character set architecture
65 consisting of 128 24-bit
67 each divided into 256 16-bit
73 positions, one for each character.
74 Part 1 of the standard (ISO 10646-1)
75 defines the first 65534 code positions (0x0000 to 0xfffd), which form
77 .IR "Basic Multilingual Plane"
78 (BMP), that is plane 0 in group 0.
79 Part 2 of the standard (ISO 10646-2)
80 adds characters to group 0 outside the BMP in several
81 .I "supplementary planes"
82 in the range 0x10000 to 0x10ffff.
83 There are no plans to add characters
84 beyond 0x10ffff to the standard, therefore of the entire code space,
85 only a small fraction of group 0 will ever be actually used in the
87 The BMP contains all characters found in the
88 commonly used other character sets.
89 The supplemental planes added by
90 ISO 10646-2 cover only more exotic characters for special scientific,
91 dictionary printing, publishing industry, higher-level protocol and
94 The representation of each UCS character as a 2-byte word is referred
95 to as the UCS-2 form (only for BMP characters),
96 whereas UCS-4 is the representation of each character by a 4-byte word.
97 In addition, there exist two encoding forms UTF-8
98 for backward compatibility with ASCII processing software and UTF-16
99 for the backward-compatible handling of non-BMP characters up to
100 0x10ffff by UCS-2 software.
102 The UCS characters 0x0000 to 0x007f are identical to those of the
104 character set and the characters in the range 0x0000 to 0x00ff
105 are identical to those in
106 ISO 8859-1 (Latin-1).
107 .SS Combining characters
108 Some code points in UCS
109 have been assigned to
110 .IR "combining characters" .
111 These are similar to the nonspacing accent keys on a typewriter.
112 A combining character just adds an accent to the previous character.
113 The most important accented characters have codes of their own in UCS,
114 however, the combining character mechanism allows us to add accents
115 and other diacritical marks to any character.
116 The combining characters
117 always follow the character which they modify.
118 For example, the German
119 character Umlaut-A ("Latin capital letter A with diaeresis") can
120 either be represented by the precomposed UCS code 0x00c4, or
121 alternatively as the combination of a normal "Latin capital letter A"
122 followed by a "combining diaeresis": 0x0041 0x0308.
124 Combining characters are essential for instance for encoding the Thai
125 script or for mathematical typesetting and users of the International
127 .SS Implementation levels
128 As not all systems are expected to support advanced mechanisms like
129 combining characters, ISO 10646-1 specifies the following three
130 .I implementation levels
134 Combining characters and Hangul Jamo
135 (a variant encoding of the Korean script, where a Hangul syllable
136 glyph is coded as a triplet or pair of vowel/consonant codes) are not
140 In addition to level 1, combining characters are now allowed for some
141 languages where they are essential (e.g., Thai, Lao, Hebrew,
142 Arabic, Devanagari, Malayalam).
145 All UCS characters are supported.
147 The Unicode 3.0 Standard
148 published by the Unicode Consortium
149 contains exactly the UCS Basic Multilingual Plane
150 at implementation level 3, as described in ISO 10646-1:2000.
151 Unicode 3.1 added the supplemental planes of ISO 10646-2.
152 The Unicode standard and
153 technical reports published by the Unicode Consortium provide much
154 additional information on the semantics and recommended usages of
156 They provide guidelines and algorithms for
157 editing, sorting, comparing, normalizing, converting, and displaying
159 .SS Unicode under Linux
160 Under GNU/Linux, the C type
162 is a signed 32-bit integer type.
163 Its values are always interpreted
164 by the C library as UCS
165 code values (in all locales), a convention that is signaled by the GNU
166 C library to applications by defining the constant
167 .B __STDC_ISO_10646__
168 as specified in the ISO C99 standard.
170 UCS/Unicode can be used just like ASCII in input/output streams,
171 terminal communication, plaintext files, filenames, and environment
172 variables in the ASCII compatible UTF-8 multibyte encoding.
173 To signal the use of UTF-8 as the character
174 encoding to all applications, a suitable
176 has to be selected via environment variables (e.g.,
180 .B nl_langinfo(CODESET)
181 function returns the name of the selected encoding.
182 Library functions such as
186 can be used to transform the internal
188 characters and strings into the system character encoding and back
191 tells, how many positions (0\(en2) the cursor is advanced by the
192 output of a character.
194 .SS Private Use Areas (PUA)
195 In the Basic Multilingual Plane,
196 the range 0xe000 to 0xf8ff will never be assigned to any characters by
197 the standard and is reserved for private usage.
199 community, this private area has been subdivided further into the
200 range 0xe000 to 0xefff which can be used individually by any end-user
201 and the Linux zone in the range 0xf000 to 0xf8ff where extensions are
202 coordinated among all Linux users.
203 The registry of the characters
204 assigned to the Linux zone is maintained by LANANA and the registry
206 .I Documentation/unicode.txt
207 in the Linux kernel sources.
209 Two other planes are reserved for private usage, plane 15
210 (Supplementary Private Use Area-A, range 0xf0000 to 0xffffd)
211 and plane 16 (Supplementary Private Use Area-B, range
212 0x100000 to 0x10fffd).
215 Information technology \(em Universal Multiple-Octet Coded Character
216 Set (UCS) \(em Part 1: Architecture and Basic Multilingual Plane.
217 International Standard ISO/IEC 10646-1, International Organization
218 for Standardization, Geneva, 2000.
220 This is the official specification of UCS .
222 .UR http://www.iso.ch/
225 The Unicode Standard, Version 3.0.
226 The Unicode Consortium, Addison-Wesley,
227 Reading, MA, 2000, ISBN 0-201-61633-5.
229 S. Harbison, G. Steele. C: A Reference Manual. Fourth edition,
230 Prentice Hall, Englewood Cliffs, 1995, ISBN 0-13-326224-3.
232 A good reference book about the C programming language.
234 edition covers the 1994 Amendment 1 to the ISO C90 standard, which
235 adds a large number of new C library functions for handling wide and
236 multibyte character encodings, but it does not yet cover ISO C99,
237 which improved wide and multibyte character support even further.
239 Unicode Technical Reports.
241 .UR http://www.unicode.org\:/reports/
245 Markus Kuhn: UTF-8 and Unicode FAQ for UNIX/Linux.
247 .UR http://www.cl.cam.ac.uk\:/~mgk25\:/unicode.html
251 Bruno Haible: Unicode HOWTO.
253 .UR http://www.tldp.org\:/HOWTO\:/Unicode\-HOWTO.html
257 .\" Markus Kuhn <mgk25@cl.cam.ac.uk>