libstdc++-v3/docs/html/22_locale/codecvt.html

   1 <?xml version="1.0" encoding="ISO-8859-1"?>
   2 <!DOCTYPE html
   3           PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   4           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   5
   6 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   7 <head>
   8    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
   9    <meta name="AUTHOR" content="bkoz@redhat.com (Benjamin Kosnik)" />
  10    <meta name="KEYWORDS" content="HOWTO, libstdc++, GCC, g++, libg++, STL" />
  11    <meta name="DESCRIPTION" content="Notes on the codecvt implementation." />
  12    <title>Notes on the codecvt implementation.</title>
  13 <link rel="StyleSheet" href="../lib3styles.css" />
  14 </head>
  15 <body>
  16   <h1>
  17   Notes on the codecvt implementation.
  18   </h1>
  19 <p>
  20 <em>
  21 prepared by Benjamin Kosnik (bkoz@redhat.com) on August 28, 2000
  22 </em>
  23 </p>
  24
  25 <h2>
  26 1. Abstract
  27 </h2>
  28 <p>
  29 The standard class codecvt attempts to address conversions between
  30 different character encoding schemes. In particular, the standard
  31 attempts to detail conversions between the implementation-defined wide
  32 characters (hereafter referred to as wchar_t) and the standard type
  33 char that is so beloved in classic &quot;C&quot; (which can now be referred to
  34 as narrow characters.)  This document attempts to describe how the GNU
  35 libstdc++-v3 implementation deals with the conversion between wide and
  36 narrow characters, and also presents a framework for dealing with the
  37 huge number of other encodings that iconv can convert, including
  38 Unicode and UTF8. Design issues and requirements are addressed, and
  39 examples of correct usage for both the required specializations for
  40 wide and narrow characters and the implementation-provided extended
  41 functionality are given.
  42 </p>
  43
  44 <h2>
  45 2. What the standard says
  46 </h2>
  47 Around page 425 of the C++ Standard, this charming heading comes into view:
  48
  49 <blockquote>
  50 22.2.1.5 - Template class codecvt [lib.locale.codecvt]
  51 </blockquote>
  52
  53 The text around the codecvt definition gives some clues:
  54
  55 <blockquote>
  56 <em>
  57 -1- The class codecvt&lt;internT,externT,stateT&gt; is for use when
  58 converting from one codeset to another, such as from wide characters
  59 to multibyte characters, between wide character encodings such as
  60 Unicode and EUC.
  61 </em>
  62 </blockquote>
  63
  64 <p>
  65 Hmm. So, in some unspecified way, Unicode encodings and
  66 translations between other character sets should be handled by this
  67 class.
  68 </p>
  69
  70 <blockquote>
  71 <em>
  72 -2- The stateT argument selects the pair of codesets being mapped between.
  73 </em>
  74 </blockquote>
  75
  76 <p>
  77 Ah ha! Another clue...
  78 </p>
  79
  80 <blockquote>
  81 <em>
  82 -3- The instantiations required in the Table ??
  83 (lib.locale.category), namely codecvt&lt;wchar_t,char,mbstate_t&gt; and
  84 codecvt&lt;char,char,mbstate_t&gt;, convert the implementation-defined
  85 native character set. codecvt&lt;char,char,mbstate_t&gt; implements a
  86 degenerate conversion; it does not convert at
  87 all. codecvt&lt;wchar_t,char,mbstate_t&gt; converts between the native
  88 character sets for tiny and wide characters. Instantiations on
  89 mbstate_t perform conversion between encodings known to the library
  90 implementor.  Other encodings can be converted by specializing on a
  91 user-defined stateT type. The stateT object can contain any state that
  92 is useful to communicate to or from the specialized do_convert member.
  93 </em>
  94 </blockquote>
  95
  96 <p>
  97 At this point, a couple points become clear:
  98 </p>
  99
 100 <p>
 101 One: The standard clearly implies that attempts to add non-required
 102 (yet useful and widely used) conversions need to do so through the
 103 third template parameter, stateT.</p>
 104
 105 <p>
 106 Two: The required conversions, by specifying mbstate_t as the third
 107 template parameter, imply an implementation strategy that is mostly
 108 (or wholly) based on the underlying C library, and the functions
 109 mcsrtombs and wcsrtombs in particular.</p>
 110
 111 <h2>
 112 3. Some thoughts on what would be useful
 113 </h2>
 114 Probably the most frequently asked question about code conversion is:
 115 &quot;So dudes, what's the deal with Unicode strings?&quot; The dude part is
 116 optional, but apparently the usefulness of Unicode strings is pretty
 117 widely appreciated. Sadly, this specific encoding (And other useful
 118 encodings like UTF8, UCS4, ISO 8859-10, etc etc etc) are not mentioned
 119 in the C++ standard.
 120
 121 <p>
 122 In particular, the simple implementation detail of wchar_t's size
 123 seems to repeatedly confound people. Many systems use a two byte,
 124 unsigned integral type to represent wide characters, and use an
 125 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, Java,
 126 others.) Other systems, use a four byte, unsigned integral type to
 127 represent wide characters, and use an internal encoding of
 128 UCS4. (GNU/Linux systems using glibc, in particular.) The C
 129 programming language (and thus C++) does not specify a specific size
 130 for the type wchar_t.
 131 </p>
 132
 133 <p>
 134 Thus, portable C++ code cannot assume a byte size (or endianness) either.
 135 </p>
 136
 137 <p>
 138 Getting back to the frequently asked question: What about Unicode strings?
 139 </p>
 140
 141 <p>
 142 What magic spell will do this conversion?
 143 </p>
 144
 145 <p>
 146 A couple of comments:
 147 </p>
 148
 149 <p>
 150 The thought that all one needs to convert between two arbitrary
 151 codesets is two types and some kind of state argument is
 152 unfortunate. In particular, encodings may be stateless. The naming of
 153 the third parameter as stateT is unfortunate, as what is really needed
 154 is some kind of generalized type that accounts for the issues that
 155 abstract encodings will need. The minimum information that is required
 156 includes:
 157 </p>
 158
 159 <ul>
 160 <li>
 161 <p>
 162  Identifiers for each of the codesets involved in the conversion. For
 163 example, using the iconv family of functions from the Single Unix
 164 Specification (what used to be called X/Open) hosted on the GNU/Linux
 165 operating system allows bi-directional mapping between far more than
 166 the following tantalizing possibilities:
 167 </p>
 168
 169 <p>
 170 (An edited list taken from <code>`iconv --list`</code> on a Red Hat 6.2/Intel system:
 171 </p>
 172
 173 <blockquote>
 174 <pre>
 175 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
 176 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
 177 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
 178 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
 179 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
 180 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
 181 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
 182 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
 183 UTF-16, UTF8, UTF16).
 184 </pre>
 185 </blockquote>
 186
 187 <p>
 188 For iconv-based implementations, string literals for each of the
 189 encodings (ie. &quot;UCS-2&quot; and &quot;UTF-8&quot;) are necessary,
 190 although for other,
 191 non-iconv implementations a table of enumerated values or some other
 192 mechanism may be required.
 193 </p>
 194 </li>
 195
 196 <li>
 197  Maximum length of the identifying string literal.
 198 </li>
 199
 200 <li>
 201  Some encodings are require explicit endian-ness. As such, some kind
 202   of endian marker or other byte-order marker will be necessary. See
 203   &quot;Footnotes for C/C++ developers&quot; in Haible for more information on
 204   UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
 205   however implementations, most notably Microsoft, vary.)
 206 </li>
 207
 208 <li>
 209  Types representing the conversion state, for conversions involving
 210   the machinery in the &quot;C&quot; library, or the conversion descriptor, for
 211   conversions using iconv (such as the type iconv_t.)  Note that the
 212   conversion descriptor encodes more information than a simple encoding
 213   state type.
 214 </li>
 215
 216 <li>
 217  Conversion descriptors for both directions of encoding. (ie, both
 218   UCS-2 to UTF-8 and UTF-8 to UCS-2.)
 219 </li>
 220
 221 <li>
 222  Something to indicate if the conversion requested if valid.
 223 </li>
 224
 225 <li>
 226  Something to represent if the conversion descriptors are valid.
 227 </li>
 228
 229 <li>
 230  Some way to enforce strict type checking on the internal and
 231   external types. As part of this, the size of the internal and
 232   external types will need to be known.
 233 </li>
 234 </ul>
 235
 236 <h2>
 237 4. Problems with &quot;C&quot; code conversions : thread safety, global
 238 locales, termination.
 239 </h2>
 240
 241 In addition, multi-threaded and multi-locale environments also impact
 242 the design and requirements for code conversions. In particular, they
 243 affect the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt;
 244 when implemented using standard &quot;C&quot; functions.
 245
 246 <p>
 247 Three problems arise, one big, one of medium importance, and one small.
 248 </p>
 249
 250 <p>
 251 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
 252 on all systems required by the GNU tools. For GNU/Linux and glibc,
 253 this is not an issue.
 254 </p>
 255
 256 <p>
 257 Of medium concern, in the grand scope of things, is that the functions
 258 used to implement this specialization work on null-terminated
 259 strings. Buffers, especially file buffers, may not be null-terminated,
 260 thus giving conversions that end prematurely or are otherwise
 261 incorrect. Yikes!
 262 </p>
 263
 264 <p>
 265 The last, and fundamental problem, is the assumption of a global
 266 locale for all the &quot;C&quot; functions referenced above. For something like
 267 C++ iostreams (where codecvt is explicitly used) the notion of
 268 multiple locales is fundamental. In practice, most users may not run
 269 into this limitation. However, as a quality of implementation issue,
 270 the GNU C++ library would like to offer a solution that allows
 271 multiple locales and or simultaneous usage with computationally
 272 correct results. In short, libstdc++-v3 is trying to offer, as an
 273 option, a high-quality implementation, damn the additional complexity!
 274 </p>
 275
 276 <p>
 277 For the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt; ,
 278 conversions are made between the internal character set (always UCS4
 279 on GNU/Linux) and whatever the currently selected locale for the
 280 LC_CTYPE category implements.
 281 </p>
 282
 283 <h2>
 284 5. Design
 285 </h2>
 286 The two required specializations are implemented as follows:
 287
 288 <p>
 289 <code>
 290 codecvt&lt;char, char, mbstate_t&gt;
 291 </code>
 292 </p>
 293 <p>
 294 This is a degenerate (ie, does nothing) specialization. Implementing
 295 this was a piece of cake.
 296 </p>
 297
 298 <p>
 299 <code>
 300 codecvt&lt;char, wchar_t, mbstate_t&gt;
 301 </code>
 302 </p>
 303 <p>
 304 This specialization, by specifying all the template parameters, pretty
 305 much ties the hands of implementors. As such, the implementation is
 306 straightforward, involving mcsrtombs for the conversions between char
 307 to wchar_t and wcsrtombs for conversions between wchar_t and char.
 308 </p>
 309
 310 <p>
 311 Neither of these two required specializations deals with Unicode
 312 characters. As such, libstdc++-v3 implements a partial specialization
 313 of the codecvt class with and iconv wrapper class, __enc_traits as the
 314 third template parameter.
 315 </p>
 316
 317 <p>
 318 This implementation should be standards conformant. First of all, the
 319 standard explicitly points out that instantiations on the third
 320 template parameter, stateT, are the proper way to implement
 321 non-required conversions. Second of all, the standard says (in Chapter
 322 17) that partial specializations of required classes are a-ok. Third
 323 of all, the requirements for the stateT type elsewhere in the standard
 324 (see 21.1.2 traits typedefs) only indicate that this type be copy
 325 constructible.
 326 </p>
 327
 328 <p>
 329 As such, the type __enc_traits is defined as a non-templatized, POD
 330 type to be used as the third type of a codecvt instantiation. This
 331 type is just a wrapper class for iconv, and provides an easy interface
 332 to iconv functionality.
 333 </p>
 334
 335 <p>
 336 There are two constructors for __enc_traits:
 337 </p>
 338
 339 <p>
 340 <code>
 341 __enc_traits() : __in_desc(0), __out_desc(0)
 342 </code>
 343 </p>
 344 <p>
 345 This default constructor sets the internal encoding to some default
 346 (currently UCS4) and the external encoding to whatever is returned by
 347 nl_langinfo(CODESET).
 348 </p>
 349
 350 <p>
 351 <code>
 352 __enc_traits(const char* __int, const char* __ext)
 353 </code>
 354 </p>
 355 <p>
 356 This constructor takes as parameters string literals that indicate the
 357 desired internal and external encoding. There are no defaults for
 358 either argument.
 359 </p>
 360
 361 <p>
 362 One of the issues with iconv is that the string literals identifying
 363 conversions are not standardized. Because of this, the thought of
 364 mandating and or enforcing some set of pre-determined valid
 365 identifiers seems iffy: thus, a more practical (and non-migraine
 366 inducing) strategy was implemented: end-users can specify any string
 367 (subject to a pre-determined length qualifier, currently 32 bytes) for
 368 encodings. It is up to the user to make sure that these strings are
 369 valid on the target system.
 370 </p>
 371
 372 <p>
 373 <code>
 374 void
 375 _M_init()
 376 </code>
 377 </p>
 378 <p>
 379 Strangely enough, this member function attempts to open conversion
 380 descriptors for a given __enc_traits object. If the conversion
 381 descriptors are not valid, the conversion descriptors returned will
 382 not be valid and the resulting calls to the codecvt conversion
 383 functions will return error.
 384 </p>
 385
 386 <p>
 387 <code>
 388 bool
 389 _M_good()
 390 </code>
 391 </p>
 392 <p>
 393 Provides a way to see if the given __enc_traits object has been
 394 properly initialized. If the string literals describing the desired
 395 internal and external encoding are not valid, initialization will
 396 fail, and this will return false. If the internal and external
 397 encodings are valid, but iconv_open could not allocate conversion
 398 descriptors, this will also return false. Otherwise, the object is
 399 ready to convert and will return true.
 400 </p>
 401
 402 <p>
 403 <code>
 404 __enc_traits(const __enc_traits&amp;)
 405 </code>
 406 </p>
 407 <p>
 408 As iconv allocates memory and sets up conversion descriptors, the copy
 409 constructor can only copy the member data pertaining to the internal
 410 and external code conversions, and not the conversion descriptors
 411 themselves.
 412 </p>
 413
 414 <p>
 415 Definitions for all the required codecvt member functions are provided
 416 for this specialization, and usage of codecvt&lt;internal character type,
 417 external character type, __enc_traits&gt; is consistent with other
 418 codecvt usage.
 419 </p>
 420
 421 <h2>
 422 6.  Examples
 423 </h2>
 424
 425 <ul>
 426         <li>
 427         a. conversions involving string literals
 428
 429 <pre>
 430   typedef codecvt_base::result                  result;
 431   typedef unsigned short                        unicode_t;
 432   typedef unicode_t                             int_type;
 433   typedef char                                  ext_type;
 434   typedef __enc_traits                          enc_type;
 435   typedef codecvt&lt;int_type, ext_type, enc_type&gt; unicode_codecvt;
 436
 437   const ext_type*       e_lit = "black pearl jasmine tea";
 438   int                   size = strlen(e_lit);
 439   int_type              i_lit_base[24] =
 440   { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
 441     27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
 442     25856, 24832, 2560
 443   };
 444   const int_type*       i_lit = i_lit_base;
 445   const ext_type*       efrom_next;
 446   const int_type*       ifrom_next;
 447   ext_type*             e_arr = new ext_type[size + 1];
 448   ext_type*             eto_next;
 449   int_type*             i_arr = new int_type[size + 1];
 450   int_type*             ito_next;
 451
 452   // construct a locale object with the specialized facet.
 453   locale                loc(locale::classic(), new unicode_codecvt);
 454   // sanity check the constructed locale has the specialized facet.
 455   VERIFY( has_facet&lt;unicode_codecvt&gt;(loc) );
 456   const unicode_codecvt&amp; cvt = use_facet&lt;unicode_codecvt&gt;(loc);
 457   // convert between const char* and unicode strings
 458   unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
 459   initialize_state(state01);
 460   result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
 461                      i_arr, i_arr + size, ito_next);
 462   VERIFY( r1 == codecvt_base::ok );
 463   VERIFY( !int_traits::compare(i_arr, i_lit, size) );
 464   VERIFY( efrom_next == e_lit + size );
 465   VERIFY( ito_next == i_arr + size );
 466 </pre>
 467         </li>
 468         <li>
 469         b. conversions involving std::string
 470         </li>
 471         <li>
 472         c. conversions involving std::filebuf and std::ostream
 473         </li>
 474 </ul>
 475
 476 More information can be found in the following testcases:
 477 <ul>
 478 <li> testsuite/22_locale/codecvt_char_char.cc       </li>
 479 <li> testsuite/22_locale/codecvt_unicode_wchar_t.cc </li>
 480 <li> testsuite/22_locale/codecvt_unicode_char.cc    </li>
 481 <li> testsuite/22_locale/codecvt_wchar_t_char.cc    </li>
 482 </ul>
 483
 484 <h2>
 485 7.  Unresolved Issues
 486 </h2>
 487 <ul>
 488 <li>
 489    a. things that are sketchy, or remain unimplemented:
 490       do_encoding, max_length and length member functions
 491       are only weakly implemented. I have no idea how to do
 492       this correctly, and in a generic manner.  Nathan?
 493 </li>
 494
 495 <li>
 496    b. conversions involving std::string
 497
 498    <ul>
 499       <li>
 500       how should operators != and == work for string of
 501       different/same encoding?
 502       </li>
 503
 504       <li>
 505       what is equal? A byte by byte comparison or an
 506       encoding then byte comparison?
 507       </li>
 508
 509       <li>
 510       conversions between narrow, wide, and unicode strings
 511       </li>
 512    </ul>
 513 </li>
 514 <li>
 515    c. conversions involving std::filebuf and std::ostream
 516    <ul>
 517       <li>
 518       how to initialize the state object in a
 519       standards-conformant manner?
 520       </li>
 521
 522                 <li>
 523       how to synchronize the &quot;C&quot; and &quot;C++&quot;
 524       conversion information?
 525       </li>
 526
 527                 <li>
 528       wchar_t/char internal buffers and conversions between
 529       internal/external buffers?
 530       </li>
 531    </ul>
 532 </li>
 533 </ul>
 534
 535 <h2>
 536 8. Acknowledgments
 537 </h2>
 538 Ulrich Drepper for the iconv suggestions and patient answering of
 539 late-night questions, Jason Merrill for the template partial
 540 specialization hints, language clarification, and wchar_t fixes.
 541
 542 <h2>
 543 9. Bibliography / Referenced Documents
 544 </h2>
 545
 546 Drepper, Ulrich, GNU libc (glibc) 2.2 manual. In particular, Chapters &quot;6. Character Set Handling&quot; and &quot;7 Locales and Internationalization&quot;
 547
 548 <p>
 549 Drepper, Ulrich, Numerous, late-night email correspondence
 550 </p>
 551
 552 <p>
 553 Feather, Clive, &quot;A brief description of Normative Addendum 1,&quot; in particular the parts on Extended Character Sets
 554 http://www.lysator.liu.se/c/na1.html
 555 </p>
 556
 557 <p>
 558 Haible, Bruno, &quot;The Unicode HOWTO&quot; v0.18, 4 August 2000
 559 ftp://ftp.ilog.fr/pub/Users/haible/utf8/Unicode-HOWTO.html
 560 </p>
 561
 562 <p>
 563 ISO/IEC 14882:1998 Programming languages - C++
 564 </p>
 565
 566 <p>
 567 ISO/IEC 9899:1999 Programming languages - C
 568 </p>
 569
 570 <p>
 571 Khun, Markus, &quot;UTF-8 and Unicode FAQ for Unix/Linux&quot;
 572 http://www.cl.cam.ac.uk/~mgk25/unicode.html
 573 </p>
 574
 575 <p>
 576 Langer, Angelika and Klaus Kreft, Standard C++ IOStreams and Locales, Advanced Programmer's Guide and Reference, Addison Wesley Longman, Inc. 2000
 577 </p>
 578
 579 <p>
 580 Stroustrup, Bjarne, Appendix D, The C++ Programming Language, Special Edition, Addison Wesley, Inc. 2000
 581 </p>
 582
 583 <p>
 584 System Interface Definitions, Issue 6 (IEEE Std. 1003.1-200x)
 585 The Open Group/The Institute of Electrical and Electronics Engineers, Inc.
 586 http://www.opennc.org/austin/docreg.html
 587 </p>
 588
 589 </body>
 590 </html>