1 <?xml version="1.0" encoding="iso-8859-1"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
4 <!ENTITY % global_entities SYSTEM '../entities/global.entities'>
12 <firstname>TAKAHASHI</firstname><surname>Motonobu</surname>
14 <address><email>monyo@home.monyo.com</email></address>
16 <contrib>Japanese character support</contrib>
18 <pubdate>25 March 2003</pubdate>
21 <title>Unicode/Charsets</title>
24 <title>Features and Benefits</title>
27 Every industry eventually matures. One of the great areas of maturation is in
28 the focus that has been given over the past decade to make it possible for anyone
29 anywhere to use a computer. It has not always been that way, in fact, not so long
30 ago it was common for software to be written for exclusive use in the country of
35 Of all the effort that has been brought to bear on providing native
36 language support for all computer users, the efforts of the
37 <ulink url="http://www.openi18n.org/">Openi18n organization</ulink>
38 is deserving of special mention.
42 Samba-2.x supported a single locale through a mechanism called
43 <emphasis>codepages</emphasis>. Samba-3 is destined to become a truly trans-global
44 file and printer-sharing platform.
50 <title>What Are Charsets and Unicode?</title>
53 Computers communicate in numbers. In texts, each number will be
54 translated to a corresponding letter. The meaning that will be assigned
55 to a certain number depends on the <emphasis>character set (charset)
56 </emphasis> that is used.
60 A charset can be seen as a table that is used to translate numbers to
61 letters. Not all computers use the same charset (there are charsets
62 with German umlauts, Japanese characters, and so on). The American Standard Code
63 for Information Interchange (ASCII) encoding system has been the normative character
64 encoding scheme used by computers to date. This employs a charset that contains
65 256 characters. Using this mode of encoding each character takes exactly one byte.
69 There are also charsets that support extended characters, but those need at least
70 twice as much storage space as does ASCII encoding. Such charsets can contain
71 <command>256 * 256 = 65536</command> characters, which is more than all possible
72 characters one could think of. They are called multi-byte charsets because they use
73 more then one byte to store one character.
77 One standardized multi-byte charset encoding scheme is known as
78 <ulink url="http://www.unicode.org/">unicode</ulink>. A big advantage of using a
79 multi-byte charset is that you only need one. There is no need to make sure two
80 computers use the same charset when they are communicating.
83 <para>Old Windows clients use single-byte charsets, named
84 <parameter>codepages</parameter>, by Microsoft. However, there is no support for
85 negotiating the charset to be used in the SMB/CIFS protocol. Thus, you
86 have to make sure you are using the same charset when talking to an older client.
87 Newer clients (Windows NT, 200x, XP) talk unicode over the wire.
92 <title>Samba and Charsets</title>
95 As of Samba-3, Samba can (and will) talk unicode over the wire. Internally,
96 Samba knows of three kinds of character sets:
101 <term><smbconfoption><name>unix charset</name></smbconfoption></term>
103 This is the charset used internally by your operating system.
104 The default is <constant>UTF-8</constant>, which is fine for most
105 systems, which covers all characters in all languages. The default
106 in previous Samba releases was <constant>ASCII</constant>.
111 <term><smbconfoption><name>display charset</name></smbconfoption></term>
112 <listitem><para>This is the charset Samba will use to print messages
113 on your screen. It should generally be the same as the <parameter>unix charset</parameter>.
118 <term><smbconfoption><name>dos charset</name></smbconfoption></term>
119 <listitem><para>This is the charset Samba uses when communicating with
120 DOS and Windows 9x/Me clients. It will talk unicode to all newer clients.
121 The default depends on the charsets you have installed on your system.
122 Run <command>testparm -v | grep <quote>dos charset</quote></command> to see
123 what the default is on your system.
131 <title>Conversion from Old Names</title>
133 <para>Because previous Samba versions did not do any charset conversion,
134 characters in filenames are usually not correct in the UNIX charset but only
135 for the local charset used by the DOS/Windows clients.</para>
137 <para>Bjoern Jacke has written a utility named <ulink url="http://j3e.de/linux/convmv/">convmv</ulink>
138 that can convert whole directory structures to different charsets with one single command.
144 <title>Japanese Charsets</title>
147 Setting up Japanese charsets is quite difficult. This is mainly because:
151 <listitem><para>The Windows character set is extended from the original legacy Japanese
152 standard (JIS X 0208) and is not standardized. This means that the strictly
153 standardized implementation cannot support the full Windows character set.
156 <listitem><para> Mainly for historical reasons, there are several encoding methods in
157 Japanese, which are not fully compatible with each other. There are
158 two major encoding methods. One is the Shift_JIS series, it is used in Windows
159 and some UNIX's. The other is the EUC-JP series, used in most UNIX's
160 and Linux. Moreover, Samba previously also offered several unique encoding
161 methods, named CAP and HEX, to keep interoperability with CAP/NetAtalk and
162 UNIX's which can't use Japanese filenames. Some implementations of the
163 EUC-JP series can't support the full Windows character set.
166 <listitem><para>There are some code conversion tables between Unicode and legacy
167 Japanese character sets. One is compatible with Windows, another one
168 is based on the reference of the Unicode consortium and others are
169 a mixed implementation. The Unicode consortium does not officially
170 define any conversion tables between Unicode and legacy character
171 sets so there cannot be standard one.
174 <listitem><para>The character set and conversion tables available in iconv() depends
175 on the iconv library that is available. Next to that, the Japanese locale
176 names may be different on different systems. This means that the value of
177 the charset parameters depends on the implementation of iconv() you are using.
180 <para>Though 2 byte fixed UCS-2 encoding is used in Windows internally,
181 Shift_JIS series encoding is usually used in Japanese environments
182 as ASCII encoding is in English environments.
186 <sect2><title>Basic Parameter Setting</title>
189 <smbconfoption><name>dos charset</name></smbconfoption> and
190 <smbconfoption><name>display charset</name></smbconfoption>
191 should be set to the locale compatible with the character set
192 and encoding method used on Windows. This is usually CP932
193 but sometimes has a different name.
197 <smbconfoption><name>unix charset</name></smbconfoption> can be either Shift_JIS series,
198 EUC-JP series and UTF-8. UTF-8 is always available but the availability of other locales
199 and its name itself depends on the system.
203 Additionally, you can consider to use the Shift_JIS series as the
204 value of the <smbconfoption><name>unix charset</name></smbconfoption>
205 parameter by using the vfs_cap module, which does the same thing as
206 setting <quote>coding system = CAP</quote> in the Samba 2.2 series.
210 Where to set <smbconfoption><name>unix charset</name></smbconfoption>
211 to is a difficult question. Here is a list of details, advantages and
212 disadvantages of using a certain value.
216 <varlistentry><term>Shift_JIS series</term>
218 Shift_JIS series means a locale which is equivalent to <constant>Shift_JIS</constant>,
219 used as a standard on Japanese Windows. In the case of <constant>Shift_JIS</constant>,
220 for example if a Japanese file name consist of 0x8ba4 and 0x974c
221 (a 4 bytes Japanese character string meaning <quote>share</quote>) and <quote>.txt</quote>
222 is written from Windows on Samba, the file name on UNIX becomes
223 0x8ba4, 0x974c, <quote>.txt</quote> (a 8 bytes BINARY string), same as Windows.
226 <para>Since Shift_JIS series is usually used on some commercial based
227 UNIX's; hp-ux and AIX as Japanese locale (however, it is also possible
228 to use the EUC-JP series), To use Shift_JIS series on these platforms,
229 Japanese file names created from Windows can be referred to also on
233 If your UNIX is already working with Shift_JIS and there is a user
234 who needs to use Japanese file names written from Windows, the
235 Shift_JIS series is the best choice. However, broken file names
236 may be displayed and some commands which cannot handle non-ASCII
237 filenames may be aborted during parsing filenames. especially there
238 may be <quote>\ (0x5c)</quote> in file names, which need to be handled carefully.
239 So you had better not touch file names written from Windows on UNIX.
243 Note that most Japanized free software actually works with EUC-JP
244 only. You had better verify if the Japanized free software can work
250 <varlistentry><term>EUC-JP series</term>
252 EUC-JP series means a locale which is equivalent to the industry
253 standard called EUC-JP, widely used in Japanese UNIX (although EUC
254 contains specifications for languages other than Japanese, such as
255 EUC-KR). In the case of EUC-JP series, for example if a Japanese
256 file name consist of 0x8ba4 and 0x974c and <quote>.txt</quote> is written from
257 Windows on Samba, the file name on UNIX becomes 0xb6a6, 0xcdad,
258 <quote>.txt</quote> (a 8 bytes BINARY string).
262 Since EUC-JP is usually used on Open source UNIX, Linux and FreeBSD,
263 and on commercial based UNIX, Solaris, IRIX and Tru64 UNIX as
264 Japanese locale (however, it is also possible on Solaris to use
265 Shift_JIS and UTF-8, on Tru64 UNIX to use Shift_JIS). To use EUC-JP
266 series, most Japanese file names created from Windows can be
267 referred to also on UNIX. Also, most Japanized free software work
268 mainly with EUC-JP only.
272 It is recommended to choose EUC-JP series when using Japanese file
277 Although there is no character which needs to be carefully treated
278 like <quote>\ (0x5c)</quote>, broken file names may be displayed and some
279 commands which cannot handle non-ASCII filenames may be aborted
280 during parsing filenames.
284 Moreover, if you built Samba using differently installed libiconv,
285 eucJP-ms locale included in libiconv and EUC-JP series locale
286 included in OS may not be compatible. In this case, you may need to
287 avoid using incompatible characters for file names.
292 <varlistentry><term>UTF-8</term>
294 UTF-8 means a locale which is equivalent to UTF-8, the international
295 standard defined by Unicode consortium. In UTF-8, a <parameter>character</parameter> is
296 expressed using 1-3 bytes. In case of Japanese, most characters
297 are expressed using 3 bytes. Since on Windows Shift_JIS, where a
298 character is expressed with 1 or 2 bytes, is used to express
299 Japanese, basically a byte length of a UTF-8 string grows 1.5 times
300 the length of a original Shift_JIS string. In the case of UTF-8,
301 for example if a Japanese file name consist of 0x8ba4 and 0x974c and
302 <quote>.txt</quote> is written from Windows on Samba, the file name on UNIX
303 becomes 0xe585, 0xb1e6, 0x9c89, <quote>.txt</quote> (a 10 bytes BINARY string).
307 For systems where iconv() is not available or where iconv()'s locales
308 are not compatible with Windows, UTF-8 is the only locale available.
312 There are no systems that use UTF-8 as default locale for Japanese.
316 Some broken file names may be displayed and some commands which
317 cannot handle non-ASCII filenames may be aborted during parsing
318 filenames. especially there may be <quote>\ (0x5c)</quote> in file names, which
319 need to be handled carefully. So you had better not touch file names
320 written from Windows on UNIX.
324 In addition, although it is not directly concerned with Samba, since
325 there is a delicate difference between iconv() function, which is
326 generally used on UNIX and the functions used on other platforms,
327 such as Windows and Java about the conversion table between
328 Shift_JIS and Unicode, you should be carefully to handle UTF-8.
332 Although Mac OS X uses UTF-8 as its encoding method for filenames,
333 it uses an extended UTF-8 specification that Samba cannot handle so
334 UTF-8 locale is not available for Mac OS X.
339 <varlistentry><term>Shift_JIS series + vfs_cap (CAP encoding)</term>
341 CAP encoding means a specification using in CAP and NetAtalk, file
342 server software for Macintosh. In the case of CAP encoding, for
343 example if a Japanese file name consist of 0x8ba4 and 0x974c and
344 <quote>.txt</quote> is written from Windows on Samba, the file name on UNIX
345 becomes <quote>:8b:a4:97L.txt</quote> (a 14 bytes ASCII string).
349 For CAP encoding a byte which cannot be expressed as an ASCII
350 character (0x80 or above) is encoded as <quote>:xx</quote> form. You need to take
351 care of containing a <quote>\(0x5c)</quote> in a filename but filenames are not
352 broken in a system which cannot handle non-ASCII filenames.
356 The greatest merit of CAP encoding is the compatibility of encoding
357 filenames with CAP or NetAtalk, file server software of Macintosh.
358 Since they usually write a file name on UNIX with CAP encoding, if a
359 directory is shared with both Samba and NetAtalk, you need to use
360 CAP encoding to avoid non-ASCII filenames are broken.
364 However, recently there are some systems where NetAtalk has been
365 patched to write filenames with EUC-JP (i.e. Japanese original Vine Linux).
366 Here you need to choose EUC-JP series instead of CAP encoding.
370 vfs_cap itself is available for non Shift_JIS series locales for
371 systems which cannot handle non-ASCII characters or systems which
372 shares files with NetAtalk.
376 To use CAP encoding on Samba-3, you should use the unix charset parameter and VFS
380 <smbconfexample><title>VFS CAP</title>
381 <smbconfsection>[global]</smbconfsection>
382 <smbconfoption><name>dos charset</name><value>CP932<footnote><para>the locale name "CP932" may be different name</para></footnote></value></smbconfoption>
383 <smbconfoption><name>unix charset</name><value>CP932</value></smbconfoption>
385 <member><para>...</para></member>
387 <smbconfsection>[cap-share]</smbconfsection>
388 <smbconfoption><name>vfs option</name><value>cap</value></smbconfoption>
392 You should set CP932 if using GNU libiconv for unix charset. Setting this,
393 filenames in the <quote>cap-share</quote> share are written with CAP encoding.
401 <sect2><title>Individual Implementations</title>
404 Here is some additional information regarding individual implementations:
408 <varlistentry><term>GNU libiconv</term>
410 To handle Japanese correctly, you should apply the patch
411 <ulink url="http://www2d.biglobe.ne.jp/~msyk/software/libiconv-patch.html">libiconv-1.8-cp932-patch.diff.gz</ulink>
416 Using the patched libiconv-1.8, these settings are available:
420 <!-- FIXME: Convert to diagram ? -->
423 unix charset = CP932 / eucJP-ms / UTF-8
427 display charset = CP932
431 Other Japanese locales (for example Shift_JIS and EUC-JP) should not
432 be used for the lack of the compatibility with Windows.
437 <varlistentry><term>GNU glibc</term>
439 To handle Japanese correctly, you should apply a <ulink url="http://www2d.biglobe.ne.jp/~msyk/software/glibc/">patch</ulink>
440 to glibc-2.2.5/2.3.1/2.3.2 or should use the patch-merged versions, glibc-2.3.3 or later.
444 Using the above glibc, these setting are available:
448 <smbconfoption><name>dos charset</name><value>CP932</value></smbconfoption>
449 <smbconfoption><name>unix charset</name><value>CP932 / eucJP-ms / UTF-8</value></smbconfoption>
450 <smbconfoption><name>display charset</name><value>CP932</value></smbconfoption>
454 Other Japanese locales (for example Shift_JIS and EUC-JP) should not
455 be used for the lack of the compatibility with Windows.
464 <title>Migration from Samba-2.2 Series</title>
467 Prior to Samba-2.2 series <quote>coding system</quote> parameter is used as
468 <smbconfoption><name>unix charset</name></smbconfoption> parameter of the Samba-3 series.
469 <link linkend="japancharsets">Next table</link> shows the mapping table when migrating from the Samba-2.2 series to Samba-3.
472 <table frame="all" id="japancharsets">
473 <title>Japanese Character Sets in Samba-2.2 and Samba-3</title>
475 <tgroup cols="2" align="center">
476 <colspec align="center"/>
477 <colspec align="center"/>
479 <row><entry>Samba-2.2 Coding System</entry><entry>Samba-3 unix charset</entry></row>
482 <row><entry>SJIS</entry><entry>Shift_JIS series</entry></row>
483 <row><entry>EUC</entry><entry>EUC-JP series</entry></row>
484 <row><entry>EUC3<footnote><para>Only exists in Japanese Samba version</para></footnote></entry><entry>EUC-JP series</entry></row>
485 <row><entry>CAP</entry><entry>Shift_JIS series + VFS</entry></row>
486 <row><entry>HEX</entry><entry>currently none</entry></row>
487 <row><entry>UTF8</entry><entry>UTF-8</entry></row>
488 <row><entry>UTF8-Mac<footnote><para>Only exists in Japanese Samba version</para></footnote></entry><entry>currently none</entry></row>
489 <row><entry>others</entry><entry>none</entry></row>
499 <title>Common Errors</title>
502 <title>CP850.so Can't Be Found</title>
504 <para><quote>Samba is complaining about a missing <filename>CP850.so</filename> file.</quote></para>
506 <para><emphasis>Answer:</emphasis> CP850 is the default <smbconfoption><name>dos charset</name></smbconfoption>.
507 The <smbconfoption><name>dos charset</name></smbconfoption> is used to convert data to the codepage used by your dos clients.
508 If you do not have any dos clients, you can safely ignore this message. </para>
510 <para>CP850 should be supported by your local iconv implementation. Make sure you have all the required packages installed.
511 If you compiled Samba from source, make sure to configure found iconv.</para>