gcc-4_8-branch/libstdc++-v3/doc/html/manual/facets.html

   1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Facets</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.78.1" /><meta name="keywords" content="ISO C++, library" /><meta name="keywords" content="ISO C++, runtime, library" /><link rel="home" href="../index.html" title="The GNU C++ Library" /><link rel="up" href="localization.html" title="Chapter 8.  Localization" /><link rel="prev" href="localization.html" title="Chapter 8.  Localization" /><link rel="next" href="containers.html" title="Chapter 9.  Containers" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Facets</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="localization.html">Prev</a> </td><th width="60%" align="center">Chapter 8.
   3   Localization
   4
   5 </th><td width="20%" align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="std.localization.facet"></a>Facets</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.ctype"></a>ctype</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.impl"></a>Implementation</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="idm269999753024"></a>Specializations</h5></div></div></div><p>
   6 For the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt; ,
   7 conversions are made between the internal character set (always UCS4
   8 on GNU/Linux) and whatever the currently selected locale for the
   9 LC_CTYPE category implements.
  10 </p><p>
  11 The two required specializations are implemented as follows:
  12 </p><p>
  13 <code class="code">
  14 ctype&lt;char&gt;
  15 </code>
  16 </p><p>
  17 This is simple specialization. Implementing this was a piece of cake.
  18 </p><p>
  19 <code class="code">
  20 ctype&lt;wchar_t&gt;
  21 </code>
  22 </p><p>
  23 This specialization, by specifying all the template parameters, pretty
  24 much ties the hands of implementors. As such, the implementation is
  25 straightforward, involving mcsrtombs for the conversions between char
  26 to wchar_t and wcsrtombs for conversions between wchar_t and char.
  27 </p><p>
  28 Neither of these two required specializations deals with Unicode
  29 characters.
  30 </p></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
  31    How to deal with the global locale issue?
  32    </p></li><li class="listitem"><p>
  33    How to deal with different types than char, wchar_t? </p></li><li class="listitem"><p>
  34    Overlap between codecvt/ctype: narrow/widen
  35    </p></li><li class="listitem"><p>
  36        Mask typedef in codecvt_base, argument types in codecvt.  what
  37        is know about this type?
  38    </p></li><li class="listitem"><p>
  39    Why mask* argument in codecvt?
  40    </p></li><li class="listitem"><p>
  41        Can this be made (more) generic? is there a simple way to
  42        straighten out the configure-time mess that is a by-product of
  43        this class?
  44    </p></li><li class="listitem"><p>
  45        Get the ctype&lt;wchar_t&gt;::mask stuff under control. Need to
  46        make some kind of static table, and not do lookup every time
  47        somebody hits the do_is... functions. Too bad we can't just
  48        redefine mask for ctype&lt;wchar_t&gt;
  49    </p></li><li class="listitem"><p>
  50        Rename abstract base class. See if just smash-overriding is a
  51        better approach. Clarify, add sanity to naming.
  52      </p></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999738032"></a><p><span class="citetitle"><em class="citetitle">
  53       The GNU C Library
  54     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6  Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a id="idm269999733280"></a><p><span class="citetitle"><em class="citetitle">
  55       Correspondence
  56     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="idm269999730192"></a><p><span class="citetitle"><em class="citetitle">
  57       ISO/IEC 14882:1998 Programming languages - C++
  58     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999727904"></a><p><span class="citetitle"><em class="citetitle">
  59       ISO/IEC 9899:1999 Programming languages - C
  60     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999725632"></a><p><span class="title"><em>
  61         <a class="link" href="http://www.unix.org/version3/ieee_std.html" target="_top">
  62         The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)
  63         </a>
  64       </em>. </span><span class="copyright">Copyright © 1999
  65       The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span></p></div><div class="biblioentry"><a id="idm269999722384"></a><p><span class="citetitle"><em class="citetitle">
  66       The C++ Programming Language, Special Edition
  67     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
  68         Addison Wesley
  69       . </span></span></p></div><div class="biblioentry"><a id="idm269999717760"></a><p><span class="citetitle"><em class="citetitle">
  70       Standard C++ IOStreams and Locales
  71     </em>. </span><span class="subtitle">
  72       Advanced Programmer's Guide and Reference
  73     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
  74         Addison Wesley Longman
  75       . </span></span></p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.codecvt"></a>codecvt</h3></div></div></div><p>
  76 The standard class codecvt attempts to address conversions between
  77 different character encoding schemes. In particular, the standard
  78 attempts to detail conversions between the implementation-defined wide
  79 characters (hereafter referred to as wchar_t) and the standard type
  80 char that is so beloved in classic <span class="quote">“<span class="quote">C</span>”</span> (which can now be
  81 referred to as narrow characters.)  This document attempts to describe
  82 how the GNU libstdc++ implementation deals with the conversion between
  83 wide and narrow characters, and also presents a framework for dealing
  84 with the huge number of other encodings that iconv can convert,
  85 including Unicode and UTF8. Design issues and requirements are
  86 addressed, and examples of correct usage for both the required
  87 specializations for wide and narrow characters and the
  88 implementation-provided extended functionality are given.
  89 </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.req"></a>Requirements</h4></div></div></div><p>
  90 Around page 425 of the C++ Standard, this charming heading comes into view:
  91 </p><div class="blockquote"><blockquote class="blockquote"><p>
  92 22.2.1.5 - Template class codecvt
  93 </p></blockquote></div><p>
  94 The text around the codecvt definition gives some clues:
  95 </p><div class="blockquote"><blockquote class="blockquote"><p>
  96 <span class="emphasis"><em>
  97 -1- The class codecvt&lt;internT,externT,stateT&gt; is for use when
  98 converting from one codeset to another, such as from wide characters
  99 to multibyte characters, between wide character encodings such as
 100 Unicode and EUC.
 101 </em></span>
 102 </p></blockquote></div><p>
 103 Hmm. So, in some unspecified way, Unicode encodings and
 104 translations between other character sets should be handled by this
 105 class.
 106 </p><div class="blockquote"><blockquote class="blockquote"><p>
 107 <span class="emphasis"><em>
 108 -2- The stateT argument selects the pair of codesets being mapped between.
 109 </em></span>
 110 </p></blockquote></div><p>
 111 Ah ha! Another clue...
 112 </p><div class="blockquote"><blockquote class="blockquote"><p>
 113 <span class="emphasis"><em>
 114 -3- The instantiations required in the Table ??
 115 (lib.locale.category), namely codecvt&lt;wchar_t,char,mbstate_t&gt; and
 116 codecvt&lt;char,char,mbstate_t&gt;, convert the implementation-defined
 117 native character set. codecvt&lt;char,char,mbstate_t&gt; implements a
 118 degenerate conversion; it does not convert at
 119 all. codecvt&lt;wchar_t,char,mbstate_t&gt; converts between the native
 120 character sets for tiny and wide characters. Instantiations on
 121 mbstate_t perform conversion between encodings known to the library
 122 implementor.  Other encodings can be converted by specializing on a
 123 user-defined stateT type. The stateT object can contain any state that
 124 is useful to communicate to or from the specialized do_convert member.
 125 </em></span>
 126 </p></blockquote></div><p>
 127 At this point, a couple points become clear:
 128 </p><p>
 129 One: The standard clearly implies that attempts to add non-required
 130 (yet useful and widely used) conversions need to do so through the
 131 third template parameter, stateT.</p><p>
 132 Two: The required conversions, by specifying mbstate_t as the third
 133 template parameter, imply an implementation strategy that is mostly
 134 (or wholly) based on the underlying C library, and the functions
 135 mcsrtombs and wcsrtombs in particular.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.design"></a>Design</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.wchar_t_size"></a><span class="type">wchar_t</span> Size</h5></div></div></div><p>
 136       The simple implementation detail of wchar_t's size seems to
 137       repeatedly confound people. Many systems use a two byte,
 138       unsigned integral type to represent wide characters, and use an
 139       internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
 140       Java, others.) Other systems, use a four byte, unsigned integral
 141       type to represent wide characters, and use an internal encoding
 142       of UCS4. (GNU/Linux systems using glibc, in particular.) The C
 143       programming language (and thus C++) does not specify a specific
 144       size for the type wchar_t.
 145     </p><p>
 146       Thus, portable C++ code cannot assume a byte size (or endianness) either.
 147     </p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.unicode"></a>Support for Unicode</h5></div></div></div><p>
 148     Probably the most frequently asked question about code conversion
 149     is: "So dudes, what's the deal with Unicode strings?"
 150     The dude part is optional, but apparently the usefulness of
 151     Unicode strings is pretty widely appreciated. Sadly, this specific
 152     encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
 153     etc etc etc) are not mentioned in the C++ standard.
 154   </p><p>
 155     A couple of comments:
 156   </p><p>
 157     The thought that all one needs to convert between two arbitrary
 158     codesets is two types and some kind of state argument is
 159     unfortunate. In particular, encodings may be stateless. The naming
 160     of the third parameter as stateT is unfortunate, as what is really
 161     needed is some kind of generalized type that accounts for the
 162     issues that abstract encodings will need. The minimum information
 163     that is required includes:
 164   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 165         Identifiers for each of the codesets involved in the
 166         conversion. For example, using the iconv family of functions
 167         from the Single Unix Specification (what used to be called
 168         X/Open) hosted on the GNU/Linux operating system allows
 169         bi-directional mapping between far more than the following
 170         tantalizing possibilities:
 171       </p><p>
 172         (An edited list taken from <code class="code">`iconv --list`</code> on a
 173         Red Hat 6.2/Intel system:
 174       </p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting">
 175 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
 176 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
 177 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
 178 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
 179 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
 180 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
 181 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
 182 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
 183 UTF-16, UTF8, UTF16).
 184 </pre></blockquote></div><p>
 185 For iconv-based implementations, string literals for each of the
 186 encodings (i.e. "UCS-2" and "UTF-8") are necessary,
 187 although for other,
 188 non-iconv implementations a table of enumerated values or some other
 189 mechanism may be required.
 190 </p></li><li class="listitem"><p>
 191  Maximum length of the identifying string literal.
 192 </p></li><li class="listitem"><p>
 193  Some encodings require explicit endian-ness. As such, some kind
 194   of endian marker or other byte-order marker will be necessary. See
 195   "Footnotes for C/C++ developers" in Haible for more information on
 196   UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
 197   however implementations, most notably Microsoft, vary.)
 198 </p></li><li class="listitem"><p>
 199  Types representing the conversion state, for conversions involving
 200   the machinery in the "C" library, or the conversion descriptor, for
 201   conversions using iconv (such as the type iconv_t.)  Note that the
 202   conversion descriptor encodes more information than a simple encoding
 203   state type.
 204 </p></li><li class="listitem"><p>
 205  Conversion descriptors for both directions of encoding. (i.e., both
 206   UCS-2 to UTF-8 and UTF-8 to UCS-2.)
 207 </p></li><li class="listitem"><p>
 208  Something to indicate if the conversion requested if valid.
 209 </p></li><li class="listitem"><p>
 210  Something to represent if the conversion descriptors are valid.
 211 </p></li><li class="listitem"><p>
 212  Some way to enforce strict type checking on the internal and
 213   external types. As part of this, the size of the internal and
 214   external types will need to be known.
 215 </p></li></ul></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.issues"></a>Other Issues</h5></div></div></div><p>
 216 In addition, multi-threaded and multi-locale environments also impact
 217 the design and requirements for code conversions. In particular, they
 218 affect the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt;
 219 when implemented using standard "C" functions.
 220 </p><p>
 221 Three problems arise, one big, one of medium importance, and one small.
 222 </p><p>
 223 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
 224 on all systems required by the GNU tools. For GNU/Linux and glibc,
 225 this is not an issue.
 226 </p><p>
 227 Of medium concern, in the grand scope of things, is that the functions
 228 used to implement this specialization work on null-terminated
 229 strings. Buffers, especially file buffers, may not be null-terminated,
 230 thus giving conversions that end prematurely or are otherwise
 231 incorrect. Yikes!
 232 </p><p>
 233 The last, and fundamental problem, is the assumption of a global
 234 locale for all the "C" functions referenced above. For something like
 235 C++ iostreams (where codecvt is explicitly used) the notion of
 236 multiple locales is fundamental. In practice, most users may not run
 237 into this limitation. However, as a quality of implementation issue,
 238 the GNU C++ library would like to offer a solution that allows
 239 multiple locales and or simultaneous usage with computationally
 240 correct results. In short, libstdc++ is trying to offer, as an
 241 option, a high-quality implementation, damn the additional complexity!
 242 </p><p>
 243 For the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt; ,
 244 conversions are made between the internal character set (always UCS4
 245 on GNU/Linux) and whatever the currently selected locale for the
 246 LC_CTYPE category implements.
 247 </p></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.impl"></a>Implementation</h4></div></div></div><p>
 248 The two required specializations are implemented as follows:
 249 </p><p>
 250 <code class="code">
 251 codecvt&lt;char, char, mbstate_t&gt;
 252 </code>
 253 </p><p>
 254 This is a degenerate (i.e., does nothing) specialization. Implementing
 255 this was a piece of cake.
 256 </p><p>
 257 <code class="code">
 258 codecvt&lt;char, wchar_t, mbstate_t&gt;
 259 </code>
 260 </p><p>
 261 This specialization, by specifying all the template parameters, pretty
 262 much ties the hands of implementors. As such, the implementation is
 263 straightforward, involving mcsrtombs for the conversions between char
 264 to wchar_t and wcsrtombs for conversions between wchar_t and char.
 265 </p><p>
 266 Neither of these two required specializations deals with Unicode
 267 characters. As such, libstdc++ implements a partial specialization
 268 of the codecvt class with and iconv wrapper class, encoding_state as the
 269 third template parameter.
 270 </p><p>
 271 This implementation should be standards conformant. First of all, the
 272 standard explicitly points out that instantiations on the third
 273 template parameter, stateT, are the proper way to implement
 274 non-required conversions. Second of all, the standard says (in Chapter
 275 17) that partial specializations of required classes are a-ok. Third
 276 of all, the requirements for the stateT type elsewhere in the standard
 277 (see 21.1.2 traits typedefs) only indicate that this type be copy
 278 constructible.
 279 </p><p>
 280 As such, the type encoding_state is defined as a non-templatized, POD
 281 type to be used as the third type of a codecvt instantiation. This
 282 type is just a wrapper class for iconv, and provides an easy interface
 283 to iconv functionality.
 284 </p><p>
 285 There are two constructors for encoding_state:
 286 </p><p>
 287 <code class="code">
 288 encoding_state() : __in_desc(0), __out_desc(0)
 289 </code>
 290 </p><p>
 291 This default constructor sets the internal encoding to some default
 292 (currently UCS4) and the external encoding to whatever is returned by
 293 nl_langinfo(CODESET).
 294 </p><p>
 295 <code class="code">
 296 encoding_state(const char* __int, const char* __ext)
 297 </code>
 298 </p><p>
 299 This constructor takes as parameters string literals that indicate the
 300 desired internal and external encoding. There are no defaults for
 301 either argument.
 302 </p><p>
 303 One of the issues with iconv is that the string literals identifying
 304 conversions are not standardized. Because of this, the thought of
 305 mandating and or enforcing some set of pre-determined valid
 306 identifiers seems iffy: thus, a more practical (and non-migraine
 307 inducing) strategy was implemented: end-users can specify any string
 308 (subject to a pre-determined length qualifier, currently 32 bytes) for
 309 encodings. It is up to the user to make sure that these strings are
 310 valid on the target system.
 311 </p><p>
 312 <code class="code">
 313 void
 314 _M_init()
 315 </code>
 316 </p><p>
 317 Strangely enough, this member function attempts to open conversion
 318 descriptors for a given encoding_state object. If the conversion
 319 descriptors are not valid, the conversion descriptors returned will
 320 not be valid and the resulting calls to the codecvt conversion
 321 functions will return error.
 322 </p><p>
 323 <code class="code">
 324 bool
 325 _M_good()
 326 </code>
 327 </p><p>
 328 Provides a way to see if the given encoding_state object has been
 329 properly initialized. If the string literals describing the desired
 330 internal and external encoding are not valid, initialization will
 331 fail, and this will return false. If the internal and external
 332 encodings are valid, but iconv_open could not allocate conversion
 333 descriptors, this will also return false. Otherwise, the object is
 334 ready to convert and will return true.
 335 </p><p>
 336 <code class="code">
 337 encoding_state(const encoding_state&amp;)
 338 </code>
 339 </p><p>
 340 As iconv allocates memory and sets up conversion descriptors, the copy
 341 constructor can only copy the member data pertaining to the internal
 342 and external code conversions, and not the conversion descriptors
 343 themselves.
 344 </p><p>
 345 Definitions for all the required codecvt member functions are provided
 346 for this specialization, and usage of codecvt&lt;internal character type,
 347 external character type, encoding_state&gt; is consistent with other
 348 codecvt usage.
 349 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.use"></a>Use</h4></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting">
 350   typedef codecvt_base::result                  result;
 351   typedef unsigned short                        unicode_t;
 352   typedef unicode_t                             int_type;
 353   typedef char                                  ext_type;
 354   typedef encoding_state                          state_type;
 355   typedef codecvt&lt;int_type, ext_type, state_type&gt; unicode_codecvt;
 356
 357   const ext_type*       e_lit = "black pearl jasmine tea";
 358   int                   size = strlen(e_lit);
 359   int_type              i_lit_base[24] =
 360   { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
 361     27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
 362     25856, 24832, 2560
 363   };
 364   const int_type*       i_lit = i_lit_base;
 365   const ext_type*       efrom_next;
 366   const int_type*       ifrom_next;
 367   ext_type*             e_arr = new ext_type[size + 1];
 368   ext_type*             eto_next;
 369   int_type*             i_arr = new int_type[size + 1];
 370   int_type*             ito_next;
 371
 372   // construct a locale object with the specialized facet.
 373   locale                loc(locale::classic(), new unicode_codecvt);
 374   // sanity check the constructed locale has the specialized facet.
 375   VERIFY( has_facet&lt;unicode_codecvt&gt;(loc) );
 376   const unicode_codecvt&amp; cvt = use_facet&lt;unicode_codecvt&gt;(loc);
 377   // convert between const char* and unicode strings
 378   unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
 379   initialize_state(state01);
 380   result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
 381                      i_arr, i_arr + size, ito_next);
 382   VERIFY( r1 == codecvt_base::ok );
 383   VERIFY( !int_traits::compare(i_arr, i_lit, size) );
 384   VERIFY( efrom_next == e_lit + size );
 385   VERIFY( ito_next == i_arr + size );
 386 </pre></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 387    a. things that are sketchy, or remain unimplemented:
 388       do_encoding, max_length and length member functions
 389       are only weakly implemented. I have no idea how to do
 390       this correctly, and in a generic manner.  Nathan?
 391 </p></li><li class="listitem"><p>
 392    b. conversions involving std::string
 393   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>
 394       how should operators != and == work for string of
 395       different/same encoding?
 396       </p></li><li class="listitem"><p>
 397       what is equal? A byte by byte comparison or an
 398       encoding then byte comparison?
 399       </p></li><li class="listitem"><p>
 400       conversions between narrow, wide, and unicode strings
 401       </p></li></ul></div></li><li class="listitem"><p>
 402    c. conversions involving std::filebuf and std::ostream
 403 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>
 404       how to initialize the state object in a
 405       standards-conformant manner?
 406       </p></li><li class="listitem"><p>
 407       how to synchronize the "C" and "C++"
 408       conversion information?
 409       </p></li><li class="listitem"><p>
 410       wchar_t/char internal buffers and conversions between
 411       internal/external buffers?
 412       </p></li></ul></div></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999640080"></a><p><span class="citetitle"><em class="citetitle">
 413       The GNU C Library
 414     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">
 415       Chapters 6 Character Set Handling and 7 Locales and Internationalization
 416     . </span></p></div><div class="biblioentry"><a id="idm269999635328"></a><p><span class="citetitle"><em class="citetitle">
 417       Correspondence
 418     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="idm269999632240"></a><p><span class="citetitle"><em class="citetitle">
 419       ISO/IEC 14882:1998 Programming languages - C++
 420     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999629952"></a><p><span class="citetitle"><em class="citetitle">
 421       ISO/IEC 9899:1999 Programming languages - C
 422     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999627680"></a><p><span class="title"><em>
 423         <a class="link" href="http://www.opengroup.org/austin/" target="_top">
 424       System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
 425         </a>
 426       </em>. </span><span class="copyright">Copyright © 2008
 427         The Open Group/The Institute of Electrical and Electronics
 428         Engineers, Inc.
 429       . </span></p></div><div class="biblioentry"><a id="idm269999624448"></a><p><span class="citetitle"><em class="citetitle">
 430       The C++ Programming Language, Special Edition
 431     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
 432         Addison Wesley
 433       . </span></span></p></div><div class="biblioentry"><a id="idm269999619824"></a><p><span class="citetitle"><em class="citetitle">
 434       Standard C++ IOStreams and Locales
 435     </em>. </span><span class="subtitle">
 436       Advanced Programmer's Guide and Reference
 437     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
 438         Addison Wesley Longman
 439       . </span></span></p></div><div class="biblioentry"><a id="idm269999614160"></a><p><span class="title"><em>
 440         <a class="link" href="http://www.lysator.liu.se/c/na1.html" target="_top">
 441       A brief description of Normative Addendum 1
 442         </a>
 443       </em>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span></p></div><div class="biblioentry"><a id="idm269999610912"></a><p><span class="title"><em>
 444         <a class="link" href="http://tldp.org/HOWTO/Unicode-HOWTO.html" target="_top">
 445           The Unicode HOWTO
 446         </a>
 447       </em>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span></p></div><div class="biblioentry"><a id="idm269999608128"></a><p><span class="title"><em>
 448         <a class="link" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html" target="_top">
 449       UTF-8 and Unicode FAQ for Unix/Linux
 450         </a>
 451       </em>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span></p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="manual.localization.facet.messages"></a>messages</h3></div></div></div><p>
 452 The std::messages facet implements message retrieval functionality
 453 equivalent to Java's java.text.MessageFormat .using either GNU gettext
 454 or IEEE 1003.1-200 functions.
 455 </p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.req"></a>Requirements</h4></div></div></div><p>
 456 The std::messages facet is probably the most vaguely defined facet in
 457 the standard library. It's assumed that this facility was built into
 458 the standard library in order to convert string literals from one
 459 locale to the other. For instance, converting the "C" locale's
 460 <code class="code">const char* c = "please"</code> to a German-localized <code class="code">"bitte"</code>
 461 during program execution.
 462 </p><div class="blockquote"><blockquote class="blockquote"><p>
 463 22.2.7.1 - Template class messages [lib.locale.messages]
 464 </p></blockquote></div><p>
 465 This class has three public member functions, which directly
 466 correspond to three protected virtual member functions.
 467 </p><p>
 468 The public member functions are:
 469 </p><p>
 470 <code class="code">catalog open(const string&amp;, const locale&amp;) const</code>
 471 </p><p>
 472 <code class="code">string_type get(catalog, int, int, const string_type&amp;) const</code>
 473 </p><p>
 474 <code class="code">void close(catalog) const</code>
 475 </p><p>
 476 While the virtual functions are:
 477 </p><p>
 478 <code class="code">catalog do_open(const string&amp;, const locale&amp;) const</code>
 479 </p><div class="blockquote"><blockquote class="blockquote"><p>
 480 <span class="emphasis"><em>
 481 -1- Returns: A value that may be passed to get() to retrieve a
 482 message, from the message catalog identified by the string name
 483 according to an implementation-defined mapping. The result can be used
 484 until it is passed to close().  Returns a value less than 0 if no such
 485 catalog can be opened.
 486 </em></span>
 487 </p></blockquote></div><p>
 488 <code class="code">string_type do_get(catalog, int, int, const string_type&amp;) const</code>
 489 </p><div class="blockquote"><blockquote class="blockquote"><p>
 490 <span class="emphasis"><em>
 491 -3- Requires: A catalog cat obtained from open() and not yet closed.
 492 -4- Returns: A message identified by arguments set, msgid, and dfault,
 493 according to an implementation-defined mapping. If no such message can
 494 be found, returns dfault.
 495 </em></span>
 496 </p></blockquote></div><p>
 497 <code class="code">void do_close(catalog) const</code>
 498 </p><div class="blockquote"><blockquote class="blockquote"><p>
 499 <span class="emphasis"><em>
 500 -5- Requires: A catalog cat obtained from open() and not yet closed.
 501 -6- Effects: Releases unspecified resources associated with cat.
 502 -7- Notes: The limit on such resources, if any, is implementation-defined.
 503 </em></span>
 504 </p></blockquote></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.design"></a>Design</h4></div></div></div><p>
 505 A couple of notes on the standard.
 506 </p><p>
 507 First, why is <code class="code">messages_base::catalog</code> specified as a typedef
 508 to int? This makes sense for implementations that use
 509 <code class="code">catopen</code> and define <code class="code">nl_catd</code> as int, but not for
 510 others. Fortunately, it's not heavily used and so only a minor irritant.
 511 This has been reported as a possible defect in the standard (LWG 2028).
 512 </p><p>
 513 Second, by making the member functions <code class="code">const</code>, it is
 514 impossible to save state in them. Thus, storing away information used
 515 in the 'open' member function for use in 'get' is impossible. This is
 516 unfortunate.
 517 </p><p>
 518 The 'open' member function in particular seems to be oddly
 519 designed. The signature seems quite peculiar. Why specify a <code class="code">const
 520 string&amp; </code> argument, for instance, instead of just <code class="code">const
 521 char*</code>? Or, why specify a <code class="code">const locale&amp;</code> argument that is
 522 to be used in the 'get' member function? How, exactly, is this locale
 523 argument useful? What was the intent? It might make sense if a locale
 524 argument was associated with a given default message string in the
 525 'open' member function, for instance. Quite murky and unclear, on
 526 reflection.
 527 </p><p>
 528 Lastly, it seems odd that messages, which explicitly require code
 529 conversion, don't use the codecvt facet. Because the messages facet
 530 has only one template parameter, it is assumed that ctype, and not
 531 codecvt, is to be used to convert between character sets.
 532 </p><p>
 533 It is implicitly assumed that the locale for the default message
 534 string in 'get' is in the "C" locale. Thus, all source code is assumed
 535 to be written in English, so translations are always from "en_US" to
 536 other, explicitly named locales.
 537 </p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.impl"></a>Implementation</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.models"></a>Models</h5></div></div></div><p>
 538     This is a relatively simple class, on the face of it. The standard
 539     specifies very little in concrete terms, so generic
 540     implementations that are conforming yet do very little are the
 541     norm. Adding functionality that would be useful to programmers and
 542     comparable to Java's java.text.MessageFormat takes a bit of work,
 543     and is highly dependent on the capabilities of the underlying
 544     operating system.
 545   </p><p>
 546     Three different mechanisms have been provided, selectable via
 547     configure flags:
 548   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 549        generic
 550      </p><p>
 551        This model does very little, and is what is used by default.
 552      </p></li><li class="listitem"><p>
 553        gnu
 554      </p><p>
 555        The gnu model is complete and fully tested. It's based on the
 556        GNU gettext package, which is part of glibc. It uses the
 557        functions <code class="code">textdomain, bindtextdomain, gettext</code> to
 558        implement full functionality. Creating message catalogs is a
 559        relatively straight-forward process and is lightly documented
 560        below, and fully documented in gettext's distributed
 561        documentation.
 562      </p></li><li class="listitem"><p>
 563        ieee_1003.1-200x
 564      </p><p>
 565        This is a complete, though untested, implementation based on
 566        the IEEE standard. The functions <code class="code">catopen, catgets,
 567        catclose</code> are used to retrieve locale-specific messages
 568        given the appropriate message catalogs that have been
 569        constructed for their use. Note, the script <code class="code">
 570        po2msg.sed</code> that is part of the gettext distribution can
 571        convert gettext catalogs into catalogs that
 572        <code class="code">catopen</code> can use.
 573    </p></li></ul></div><p>
 574 A new, standards-conformant non-virtual member function signature was
 575 added for 'open' so that a directory could be specified with a given
 576 message catalog. This simplifies calling conventions for the gnu
 577 model.
 578 </p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.gnu"></a>The GNU Model</h5></div></div></div><p>
 579     The messages facet, because it is retrieving and converting
 580     between characters sets, depends on the ctype and perhaps the
 581     codecvt facet in a given locale. In addition, underlying "C"
 582     library locale support is necessary for more than just the
 583     <code class="code">LC_MESSAGES</code> mask: <code class="code">LC_CTYPE</code> is also
 584     necessary. To avoid any unpleasantness, all bits of the "C" mask
 585     (i.e. <code class="code">LC_ALL</code>) are set before retrieving messages.
 586   </p><p>
 587     Making the message catalogs can be initially tricky, but become
 588     quite simple with practice. For complete info, see the gettext
 589     documentation. Here's an idea of what is required:
 590   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 591        Make a source file with the required string literals that need
 592        to be translated. See <code class="code">intl/string_literals.cc</code> for
 593        an example.
 594      </p></li><li class="listitem"><p>
 595        Make initial catalog (see "4 Making the PO Template File" from
 596        the gettext docs).</p><p>
 597    <code class="code"> xgettext --c++ --debug string_literals.cc -o libstdc++.pot </code>
 598    </p></li><li class="listitem"><p>Make language and country-specific locale catalogs.</p><p>
 599    <code class="code">cp libstdc++.pot fr_FR.po</code>
 600    </p><p>
 601    <code class="code">cp libstdc++.pot de_DE.po</code>
 602    </p></li><li class="listitem"><p>
 603        Edit localized catalogs in emacs so that strings are
 604        translated.
 605      </p><p>
 606    <code class="code">emacs fr_FR.po</code>
 607    </p></li><li class="listitem"><p>Make the binary mo files.</p><p>
 608    <code class="code">msgfmt fr_FR.po -o fr_FR.mo</code>
 609    </p><p>
 610    <code class="code">msgfmt de_DE.po -o de_DE.mo</code>
 611    </p></li><li class="listitem"><p>Copy the binary files into the correct directory structure.</p><p>
 612    <code class="code">cp fr_FR.mo (dir)/fr_FR/LC_MESSAGES/libstdc++.mo</code>
 613    </p><p>
 614    <code class="code">cp de_DE.mo (dir)/de_DE/LC_MESSAGES/libstdc++.mo</code>
 615    </p></li><li class="listitem"><p>Use the new message catalogs.</p><p>
 616    <code class="code">locale loc_de("de_DE");</code>
 617    </p><p>
 618    <code class="code">
 619    use_facet&lt;messages&lt;char&gt; &gt;(loc_de).open("libstdc++", locale(), dir);
 620    </code>
 621    </p></li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.use"></a>Use</h4></div></div></div><p>
 622    A simple example using the GNU model of message conversion.
 623  </p><pre class="programlisting">
 624 #include &lt;iostream&gt;
 625 #include &lt;locale&gt;
 626 using namespace std;
 627
 628 void test01()
 629 {
 630   typedef messages&lt;char&gt;::catalog catalog;
 631   const char* dir =
 632   "/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale";
 633   const locale loc_de("de_DE");
 634   const messages&lt;char&gt;&amp; mssg_de = use_facet&lt;messages&lt;char&gt; &gt;(loc_de);
 635
 636   catalog cat_de = mssg_de.open("libstdc++", loc_de, dir);
 637   string s01 = mssg_de.get(cat_de, 0, 0, "please");
 638   string s02 = mssg_de.get(cat_de, 0, 0, "thank you");
 639   cout &lt;&lt; "please in german:" &lt;&lt; s01 &lt;&lt; '\n';
 640   cout &lt;&lt; "thank you in german:" &lt;&lt; s02 &lt;&lt; '\n';
 641   mssg_de.close(cat_de);
 642 }
 643 </pre></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
 644     Things that are sketchy, or remain unimplemented:
 645   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p>
 646           _M_convert_from_char, _M_convert_to_char are in flux,
 647           depending on how the library ends up doing character set
 648           conversions. It might not be possible to do a real character
 649           set based conversion, due to the fact that the template
 650           parameter for messages is not enough to instantiate the
 651           codecvt facet (1 supplied, need at least 2 but would prefer
 652           3).
 653         </p></li><li class="listitem"><p>
 654           There are issues with gettext needing the global locale set
 655           to extract a message. This dependence on the global locale
 656           makes the current "gnu" model non MT-safe. Future versions
 657           of glibc, i.e. glibc 2.3.x will fix this, and the C++ library
 658           bits are already in place.
 659         </p></li></ul></div></li><li class="listitem"><p>
 660     Development versions of the GNU "C" library, glibc 2.3 will allow
 661     a more efficient, MT implementation of std::messages, and will
 662     allow the removal of the _M_name_messages data member. If this is
 663     done, it will change the library ABI. The C++ parts to support
 664     glibc 2.3 have already been coded, but are not in use: once this
 665     version of the "C" library is released, the marked parts of the
 666     messages implementation can be switched over to the new "C"
 667     library functionality.
 668   </p></li><li class="listitem"><p>
 669     At some point in the near future, std::numpunct will probably use
 670     std::messages facilities to implement truename/falsename
 671     correctly. This is currently not done, but entries in
 672     libstdc++.pot have already been made for "true" and "false" string
 673     literals, so all that remains is the std::numpunct coding and the
 674     configure/make hassles to make the installed library search its
 675     own catalog. Currently the libstdc++.mo catalog is only searched
 676     for the testsuite cases involving messages members.
 677   </p></li><li class="listitem"><p> The following member functions:</p><p>
 678    <code class="code">
 679         catalog
 680         open(const basic_string&lt;char&gt;&amp; __s, const locale&amp; __loc) const
 681    </code>
 682    </p><p>
 683    <code class="code">
 684    catalog
 685    open(const basic_string&lt;char&gt;&amp;, const locale&amp;, const char*) const;
 686    </code>
 687    </p><p>
 688    Don't actually return a "value less than 0 if no such catalog
 689    can be opened" as required by the standard in the "gnu"
 690    model. As of this writing, it is unknown how to query to see
 691    if a specified message catalog exists using the gettext
 692    package.
 693    </p></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999527584"></a><p><span class="citetitle"><em class="citetitle">
 694       The GNU C Library
 695     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling, and 7 Locales and Internationalization
 696     . </span></p></div><div class="biblioentry"><a id="idm269999522832"></a><p><span class="citetitle"><em class="citetitle">
 697       Correspondence
 698     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="idm269999519744"></a><p><span class="citetitle"><em class="citetitle">
 699       ISO/IEC 14882:1998 Programming languages - C++
 700     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999517456"></a><p><span class="citetitle"><em class="citetitle">
 701       ISO/IEC 9899:1999 Programming languages - C
 702     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999515184"></a><p><span class="title"><em>
 703         <a class="link" href="http://www.opengroup.org/austin/" target="_top">
 704       System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
 705         </a>
 706       </em>. </span><span class="copyright">Copyright © 2008
 707         The Open Group/The Institute of Electrical and Electronics
 708         Engineers, Inc.
 709       . </span></p></div><div class="biblioentry"><a id="idm269999511952"></a><p><span class="citetitle"><em class="citetitle">
 710       The C++ Programming Language, Special Edition
 711     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
 712         Addison Wesley
 713       . </span></span></p></div><div class="biblioentry"><a id="idm269999507328"></a><p><span class="citetitle"><em class="citetitle">
 714       Standard C++ IOStreams and Locales
 715     </em>. </span><span class="subtitle">
 716       Advanced Programmer's Guide and Reference
 717     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
 718         Addison Wesley Longman
 719       . </span></span></p></div><div class="biblioentry"><a id="idm269999501664"></a><p><span class="title"><em>
 720         <a class="link" href="http://www.oracle.com/technetwork/java/api/index.html" target="_top">
 721         API Specifications, Java Platform
 722         </a>
 723       </em>. </span><span class="pagenums">java.util.Properties, java.text.MessageFormat,
 724 java.util.Locale, java.util.ResourceBundle
 725     . </span></p></div><div class="biblioentry"><a id="idm269999499328"></a><p><span class="title"><em>
 726         <a class="link" href="https://www.gnu.org/software/gettext/" target="_top">
 727       GNU gettext tools, version 0.10.38, Native Language Support
 728       Library and Tools.
 729         </a>
 730       </em>. </span></p></div></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="localization.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="localization.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 8.
 731   Localization
 732
 733  </td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 9.
 734   Containers
 735
 736 </td></tr></table></div></body></html>