mingw/html/ext/Unicode/Normalize/Normalize.html

   1 <?xml version="1.0" ?>
   2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4 <head>
   5 <title>Unicode::Normalize - Unicode Normalization Forms</title>
   6 <meta http-equiv="content-type" content="text/html; charset=utf-8" />
   7 <link rev="made" href="mailto:" />
   8 </head>
   9
  10 <body style="background-color: white">
  11 <table border="0" width="100%" cellspacing="0" cellpadding="3">
  12 <tr><td class="block" style="background-color: #cccccc" valign="middle">
  13 <big><strong><span class="block">&nbsp;Unicode::Normalize - Unicode Normalization Forms</span></strong></big>
  14 </td></tr>
  15 </table>
  16
  17 <p><a name="__index__"></a></p>
  18 <!-- INDEX BEGIN -->
  19
  20 <ul>
  21
  22         <li><a href="#name">NAME</a></li>
  23         <li><a href="#synopsis">SYNOPSIS</a></li>
  24         <li><a href="#description">DESCRIPTION</a></li>
  25         <ul>
  26
  27                 <li><a href="#normalization_forms">Normalization Forms</a></li>
  28                 <li><a href="#decomposition_and_composition">Decomposition and Composition</a></li>
  29                 <li><a href="#quick_check">Quick Check</a></li>
  30                 <li><a href="#character_data">Character Data</a></li>
  31         </ul>
  32
  33         <li><a href="#export">EXPORT</a></li>
  34         <li><a href="#caveats">CAVEATS</a></li>
  35         <li><a href="#author">AUTHOR</a></li>
  36         <li><a href="#see_also">SEE ALSO</a></li>
  37 </ul>
  38 <!-- INDEX END -->
  39
  40 <hr />
  41 <p>
  42 </p>
  43 <h1><a name="name">NAME</a></h1>
  44 <p>Unicode::Normalize - Unicode Normalization Forms</p>
  45 <p>
  46 </p>
  47 <hr />
  48 <h1><a name="synopsis">SYNOPSIS</a></h1>
  49 <p>(1) using function names exported by default:</p>
  50 <pre>
  51   use Unicode::Normalize;</pre>
  52 <pre>
  53   $NFD_string  = NFD($string);  # Normalization Form D
  54   $NFC_string  = NFC($string);  # Normalization Form C
  55   $NFKD_string = NFKD($string); # Normalization Form KD
  56   $NFKC_string = NFKC($string); # Normalization Form KC</pre>
  57 <p>(2) using function names exported on request:</p>
  58 <pre>
  59   use Unicode::Normalize 'normalize';</pre>
  60 <pre>
  61   $NFD_string  = normalize('D',  $string);  # Normalization Form D
  62   $NFC_string  = normalize('C',  $string);  # Normalization Form C
  63   $NFKD_string = normalize('KD', $string);  # Normalization Form KD
  64   $NFKC_string = normalize('KC', $string);  # Normalization Form KC</pre>
  65 <p>
  66 </p>
  67 <hr />
  68 <h1><a name="description">DESCRIPTION</a></h1>
  69 <p>Parameters:</p>
  70 <p><code>$string</code> is used as a string under character semantics
  71 (see <em>perlunicode</em>).</p>
  72 <p><code>$codepoint</code> should be an unsigned integer
  73 representing a Unicode code point.</p>
  74 <p>Note: Between XSUB and pure Perl, there is an incompatibility
  75 about the interpretation of <code>$codepoint</code> as a decimal number.
  76 XSUB converts <code>$codepoint</code> to an unsigned integer, but pure Perl does not.
  77 Do not use a floating point nor a negative sign in <code>$codepoint</code>.</p>
  78 <p>
  79 </p>
  80 <h2><a name="normalization_forms">Normalization Forms</a></h2>
  81 <dl>
  82 <dt><strong><a name="item_nfd"><code>$NFD_string = NFD($string)</code></a></strong>
  83
  84 <dd>
  85 <p>returns the Normalization Form D (formed by canonical decomposition).</p>
  86 </dd>
  87 </li>
  88 <dt><strong><a name="item_nfc"><code>$NFC_string = NFC($string)</code></a></strong>
  89
  90 <dd>
  91 <p>returns the Normalization Form C (formed by canonical decomposition
  92 followed by canonical composition).</p>
  93 </dd>
  94 </li>
  95 <dt><strong><a name="item_nfkd"><code>$NFKD_string = NFKD($string)</code></a></strong>
  96
  97 <dd>
  98 <p>returns the Normalization Form KD (formed by compatibility decomposition).</p>
  99 </dd>
 100 </li>
 101 <dt><strong><a name="item_nfkc"><code>$NFKC_string = NFKC($string)</code></a></strong>
 102
 103 <dd>
 104 <p>returns the Normalization Form KC (formed by compatibility decomposition
 105 followed by <strong>canonical</strong> composition).</p>
 106 </dd>
 107 </li>
 108 <dt><strong><a name="item_fcd"><code>$FCD_string = FCD($string)</code></a></strong>
 109
 110 <dd>
 111 <p>If the given string is in FCD (``Fast C or D'' form; cf. UTN #5),
 112 returns it without modification; otherwise returns an FCD string.</p>
 113 </dd>
 114 <dd>
 115 <p>Note: FCD is not always unique, then plural forms may be equivalent
 116 each other. <a href="#item_fcd"><code>FCD()</code></a> will return one of these equivalent forms.</p>
 117 </dd>
 118 </li>
 119 <dt><strong><a name="item_fcc"><code>$FCC_string = FCC($string)</code></a></strong>
 120
 121 <dd>
 122 <p>returns the FCC form (``Fast C Contiguous''; cf. UTN #5).</p>
 123 </dd>
 124 <dd>
 125 <p>Note: FCC is unique, as well as four normalization forms (NF*).</p>
 126 </dd>
 127 </li>
 128 <dt><strong><a name="item_normalize"><code>$normalized_string = normalize($form_name, $string)</code></a></strong>
 129
 130 <dd>
 131 <p>As <code>$form_name</code>, one of the following names must be given.</p>
 132 </dd>
 133 <dd>
 134 <pre>
 135   'C'  or 'NFC'  for Normalization Form C  (UAX #15)
 136   'D'  or 'NFD'  for Normalization Form D  (UAX #15)
 137   'KC' or 'NFKC' for Normalization Form KC (UAX #15)
 138   'KD' or 'NFKD' for Normalization Form KD (UAX #15)</pre>
 139 </dd>
 140 <dd>
 141 <pre>
 142   'FCD'          for &quot;Fast C or D&quot; Form  (UTN #5)
 143   'FCC'          for &quot;Fast C Contiguous&quot; (UTN #5)</pre>
 144 </dd>
 145 </li>
 146 </dl>
 147 <p>
 148 </p>
 149 <h2><a name="decomposition_and_composition">Decomposition and Composition</a></h2>
 150 <dl>
 151 <dt><strong><a name="item_decompose"><code>$decomposed_string = decompose($string)</code></a></strong>
 152
 153 <dt><strong><code>$decomposed_string = decompose($string, $useCompatMapping)</code></strong>
 154
 155 <dd>
 156 <p>Decomposes the specified string and returns the result.</p>
 157 </dd>
 158 <dd>
 159 <p>If the second parameter (a boolean) is omitted or false, decomposes it
 160 using the Canonical Decomposition Mapping.
 161 If true, decomposes it using the Compatibility Decomposition Mapping.</p>
 162 </dd>
 163 <dd>
 164 <p>The string returned is not always in NFD/NFKD.
 165 Reordering may be required.</p>
 166 </dd>
 167 <dd>
 168 <pre>
 169     $NFD_string  = reorder(decompose($string));       # eq. to NFD()
 170     $NFKD_string = reorder(decompose($string, TRUE)); # eq. to NFKD()</pre>
 171 </dd>
 172 </li>
 173 <dt><strong><a name="item_reorder"><code>$reordered_string  = reorder($string)</code></a></strong>
 174
 175 <dd>
 176 <p>Reorders the combining characters and the like in the canonical ordering
 177 and returns the result.</p>
 178 </dd>
 179 <dd>
 180 <p>E.g., when you have a list of NFD/NFKD strings,
 181 you can get the concatenated NFD/NFKD string from them, saying</p>
 182 </dd>
 183 <dd>
 184 <pre>
 185     $concat_NFD  = reorder(join '', @NFD_strings);
 186     $concat_NFKD = reorder(join '', @NFKD_strings);</pre>
 187 </dd>
 188 </li>
 189 <dt><strong><a name="item_compose"><code>$composed_string   = compose($string)</code></a></strong>
 190
 191 <dd>
 192 <p>Returns the string where composable pairs are composed.</p>
 193 </dd>
 194 <dd>
 195 <p>E.g., when you have a NFD/NFKD string,
 196 you can get its NFC/NFKC string, saying</p>
 197 </dd>
 198 <dd>
 199 <pre>
 200     $NFC_string  = compose($NFD_string);
 201     $NFKC_string = compose($NFKD_string);</pre>
 202 </dd>
 203 </li>
 204 </dl>
 205 <p>
 206 </p>
 207 <h2><a name="quick_check">Quick Check</a></h2>
 208 <p>(see Annex 8, UAX #15; and <em>DerivedNormalizationProps.txt</em>)</p>
 209 <p>The following functions check whether the string is in that normalization form.</p>
 210 <p>The result returned will be:</p>
 211 <pre>
 212     YES     The string is in that normalization form.
 213     NO      The string is not in that normalization form.
 214     MAYBE   Dubious. Maybe yes, maybe no.</pre>
 215 <dl>
 216 <dt><strong><a name="item_checknfd"><code>$result = checkNFD($string)</code></a></strong>
 217
 218 <dd>
 219 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>.</p>
 220 </dd>
 221 </li>
 222 <dt><strong><a name="item_checknfc"><code>$result = checkNFC($string)</code></a></strong>
 223
 224 <dd>
 225 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>;
 226 <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a> if <code>MAYBE</code>.</p>
 227 </dd>
 228 </li>
 229 <dt><strong><a name="item_checknfkd"><code>$result = checkNFKD($string)</code></a></strong>
 230
 231 <dd>
 232 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>.</p>
 233 </dd>
 234 </li>
 235 <dt><strong><a name="item_checknfkc"><code>$result = checkNFKC($string)</code></a></strong>
 236
 237 <dd>
 238 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>;
 239 <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a> if <code>MAYBE</code>.</p>
 240 </dd>
 241 </li>
 242 <dt><strong><a name="item_checkfcd"><code>$result = checkFCD($string)</code></a></strong>
 243
 244 <dd>
 245 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>.</p>
 246 </dd>
 247 </li>
 248 <dt><strong><a name="item_checkfcc"><code>$result = checkFCC($string)</code></a></strong>
 249
 250 <dd>
 251 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>;
 252 <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a> if <code>MAYBE</code>.</p>
 253 </dd>
 254 <dd>
 255 <p>If a string is not in FCD, it must not be in FCC.
 256 So <a href="#item_checkfcc"><code>checkFCC($not_FCD_string)</code></a> should return <code>NO</code>.</p>
 257 </dd>
 258 </li>
 259 <dt><strong><a name="item_check"><code>$result = check($form_name, $string)</code></a></strong>
 260
 261 <dd>
 262 <p>returns true (<code>1</code>) if <code>YES</code>; false (<code>empty string</code>) if <code>NO</code>;
 263 <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a> if <code>MAYBE</code>.</p>
 264 </dd>
 265 <dd>
 266 <p>As <code>$form_name</code>, one of the following names must be given.</p>
 267 </dd>
 268 <dd>
 269 <pre>
 270   'C'  or 'NFC'  for Normalization Form C  (UAX #15)
 271   'D'  or 'NFD'  for Normalization Form D  (UAX #15)
 272   'KC' or 'NFKC' for Normalization Form KC (UAX #15)
 273   'KD' or 'NFKD' for Normalization Form KD (UAX #15)</pre>
 274 </dd>
 275 <dd>
 276 <pre>
 277   'FCD'          for &quot;Fast C or D&quot; Form  (UTN #5)
 278   'FCC'          for &quot;Fast C Contiguous&quot; (UTN #5)</pre>
 279 </dd>
 280 </li>
 281 </dl>
 282 <p><strong>Note</strong></p>
 283 <p>In the cases of NFD, NFKD, and FCD, the answer must be
 284 either <code>YES</code> or <code>NO</code>. The answer <code>MAYBE</code> may be returned
 285 in the cases of NFC, NFKC, and FCC.</p>
 286 <p>A <code>MAYBE</code> string should contain at least one combining character
 287 or the like. For example, <code>COMBINING ACUTE ACCENT</code> has
 288 the MAYBE_NFC/MAYBE_NFKC property.</p>
 289 <p>Both <a href="#item_checknfc"><code>checkNFC(&quot;A\N{COMBINING ACUTE ACCENT}&quot;)</code></a>
 290 and <a href="#item_checknfc"><code>checkNFC(&quot;B\N{COMBINING ACUTE ACCENT}&quot;)</code></a> will return <code>MAYBE</code>.
 291 <code>&quot;A\N{COMBINING ACUTE ACCENT}&quot;</code> is not in NFC
 292 (its NFC is <code>&quot;\N{LATIN CAPITAL LETTER A WITH ACUTE}&quot;</code>),
 293 while <code>&quot;B\N{COMBINING ACUTE ACCENT}&quot;</code> is in NFC.</p>
 294 <p>If you want to check exactly, compare the string with its NFC/NFKC/FCC.</p>
 295 <pre>
 296     if ($string eq NFC($string)) {
 297         # $string is exactly normalized in NFC;
 298     } else {
 299         # $string is not normalized in NFC;
 300     }</pre>
 301 <pre>
 302     if ($string eq NFKC($string)) {
 303         # $string is exactly normalized in NFKC;
 304     } else {
 305         # $string is not normalized in NFKC;
 306     }</pre>
 307 <p>
 308 </p>
 309 <h2><a name="character_data">Character Data</a></h2>
 310 <p>These functions are interface of character data used internally.
 311 If you want only to get Unicode normalization forms, you don't need
 312 call them yourself.</p>
 313 <dl>
 314 <dt><strong><a name="item_getcanon"><code>$canonical_decomposed = getCanon($codepoint)</code></a></strong>
 315
 316 <dd>
 317 <p>If the character of the specified codepoint is canonically
 318 decomposable (including Hangul Syllables),
 319 returns the <strong>completely decomposed</strong> string canonically equivalent to it.</p>
 320 </dd>
 321 <dd>
 322 <p>If it is not decomposable, returns <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a>.</p>
 323 </dd>
 324 </li>
 325 <dt><strong><a name="item_getcompat"><code>$compatibility_decomposed = getCompat($codepoint)</code></a></strong>
 326
 327 <dd>
 328 <p>If the character of the specified codepoint is compatibility
 329 decomposable (including Hangul Syllables),
 330 returns the <strong>completely decomposed</strong> string compatibility equivalent to it.</p>
 331 </dd>
 332 <dd>
 333 <p>If it is not decomposable, returns <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a>.</p>
 334 </dd>
 335 </li>
 336 <dt><strong><a name="item_getcomposite"><code>$codepoint_composite = getComposite($codepoint_here, $codepoint_next)</code></a></strong>
 337
 338 <dd>
 339 <p>If two characters here and next (as codepoints) are composable
 340 (including Hangul Jamo/Syllables and Composition Exclusions),
 341 returns the codepoint of the composite.</p>
 342 </dd>
 343 <dd>
 344 <p>If they are not composable, returns <a href="file://C|\msysgit\mingw\html/pod/perlfunc.html#item_undef"><code>undef</code></a>.</p>
 345 </dd>
 346 </li>
 347 <dt><strong><a name="item_getcombinclass"><code>$combining_class = getCombinClass($codepoint)</code></a></strong>
 348
 349 <dd>
 350 <p>Returns the combining class of the character as an integer.</p>
 351 </dd>
 352 </li>
 353 <dt><strong><a name="item_isexclusion"><code>$is_exclusion = isExclusion($codepoint)</code></a></strong>
 354
 355 <dd>
 356 <p>Returns a boolean whether the character of the specified codepoint
 357 is a composition exclusion.</p>
 358 </dd>
 359 </li>
 360 <dt><strong><a name="item_issingleton"><code>$is_singleton = isSingleton($codepoint)</code></a></strong>
 361
 362 <dd>
 363 <p>Returns a boolean whether the character of the specified codepoint is
 364 a singleton.</p>
 365 </dd>
 366 </li>
 367 <dt><strong><a name="item_isnonstdecomp"><code>$is_non_starter_decomposition = isNonStDecomp($codepoint)</code></a></strong>
 368
 369 <dd>
 370 <p>Returns a boolean whether the canonical decomposition
 371 of the character of the specified codepoint
 372 is a Non-Starter Decomposition.</p>
 373 </dd>
 374 </li>
 375 <dt><strong><a name="item_iscomp2nd"><code>$may_be_composed_with_prev_char = isComp2nd($codepoint)</code></a></strong>
 376
 377 <dd>
 378 <p>Returns a boolean whether the character of the specified codepoint
 379 may be composed with the previous one in a certain composition
 380 (including Hangul Compositions, but excluding
 381 Composition Exclusions and Non-Starter Decompositions).</p>
 382 </dd>
 383 </li>
 384 </dl>
 385 <p>
 386 </p>
 387 <hr />
 388 <h1><a name="export">EXPORT</a></h1>
 389 <p><a href="#item_nfc"><code>NFC</code></a>, <a href="#item_nfd"><code>NFD</code></a>, <a href="#item_nfkc"><code>NFKC</code></a>, <a href="#item_nfkd"><code>NFKD</code></a>: by default.</p>
 390 <p><a href="#item_normalize"><code>normalize</code></a> and other some functions: on request.</p>
 391 <p>
 392 </p>
 393 <hr />
 394 <h1><a name="caveats">CAVEATS</a></h1>
 395 <dl>
 396 <dt><strong><a name="item_perl_27s_version_vs_2e_unicode_version">Perl's version vs. Unicode version</a></strong>
 397
 398 <dd>
 399 <p>Since this module refers to perl core's Unicode database in the directory
 400 <em>/lib/unicore</em> (or formerly <em>/lib/unicode</em>), the Unicode version of
 401 normalization implemented by this module depends on your perl's version.</p>
 402 </dd>
 403 <dd>
 404 <pre>
 405     perl's version         implemented Unicode version
 406        5.6.1                  3.0.1
 407        5.7.2                  3.1.0
 408        5.7.3                  3.1.1 (same normalized form as that of 3.1.0)
 409        5.8.0                  3.2.0
 410      5.8.1-5.8.3              4.0.0
 411      5.8.4-5.8.6 (latest)     4.0.1 (same normalized form as that of 4.0.0)</pre>
 412 </dd>
 413 </li>
 414 <dt><strong><a name="item_correction_of_decomposition_mapping">Correction of decomposition mapping</a></strong>
 415
 416 <dd>
 417 <p>In older Unicode versions, a small number of characters (all of which are
 418 CJK compatibility ideographs as far as they have been found) may have
 419 an erroneous decomposition mapping (see <em>NormalizationCorrections.txt</em>).
 420 Anyhow, this module will neither refer to <em>NormalizationCorrections.txt</em>
 421 nor provide any specific version of normalization. Therefore this module
 422 running on an older perl with an older Unicode database may use
 423 the erroneous decomposition mapping blindly conforming to the Unicode database.</p>
 424 </dd>
 425 </li>
 426 <dt><strong><a name="item_revised_definition_of_canonical_composition">Revised definition of canonical composition</a></strong>
 427
 428 <dd>
 429 <p>In Unicode 4.1.0, the definition D2 of canonical composition (which
 430 affects NFC and NFKC) has been changed (see Public Review Issue #29
 431 and recent UAX #15). This module has used the newer definition
 432 since the version 0.07 (Oct 31, 2001).
 433 This module does not support normalization according to the older
 434 definition, even if the Unicode version implemented by perl is
 435 lower than 4.1.0.</p>
 436 </dd>
 437 </li>
 438 </dl>
 439 <p>
 440 </p>
 441 <hr />
 442 <h1><a name="author">AUTHOR</a></h1>
 443 <p>SADAHIRO Tomoyuki &lt;<a href="mailto:SADAHIRO@cpan.org">SADAHIRO@cpan.org</a>&gt;</p>
 444 <p><code>Copyright(C)</code> 2001-2005, SADAHIRO Tomoyuki. Japan. All rights reserved.</p>
 445 <p>This module is free software; you can redistribute it
 446 and/or modify it under the same terms as Perl itself.</p>
 447 <p>
 448 </p>
 449 <hr />
 450 <h1><a name="see_also">SEE ALSO</a></h1>
 451 <dl>
 452 <dt><strong><a name="item_http_3a_2f_2fwww_2eunicode_2eorg_2freports_2ftr15_"><a href="http://www.unicode.org/reports/tr15/">http://www.unicode.org/reports/tr15/</a></a></strong>
 453
 454 <dd>
 455 <p>Unicode Normalization Forms - UAX #15</p>
 456 </dd>
 457 </li>
 458 <dt><strong><a name="item_http_3a_2f_2fwww_2eunicode_2eorg_2fpublic_2funidat"><a href="http://www.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt">http://www.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt</a></a></strong>
 459
 460 <dd>
 461 <p>Derived Normalization Properties</p>
 462 </dd>
 463 </li>
 464 <dt><strong><a href="http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt">http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt</a></strong>
 465
 466 <dd>
 467 <p>Normalization Corrections</p>
 468 </dd>
 469 </li>
 470 <dt><strong><a name="item_http_3a_2f_2fwww_2eunicode_2eorg_2freview_2fpr_2d2"><a href="http://www.unicode.org/review/pr-29.html">http://www.unicode.org/review/pr-29.html</a></a></strong>
 471
 472 <dd>
 473 <p>Public Review Issue #29: Normalization Issue</p>
 474 </dd>
 475 </li>
 476 <dt><strong><a name="item_http_3a_2f_2fwww_2eunicode_2eorg_2fnotes_2ftn5_2f"><a href="http://www.unicode.org/notes/tn5/">http://www.unicode.org/notes/tn5/</a></a></strong>
 477
 478 <dd>
 479 <p>Canonical Equivalence in Applications - UTN #5</p>
 480 </dd>
 481 </li>
 482 </dl>
 483 <table border="0" width="100%" cellspacing="0" cellpadding="3">
 484 <tr><td class="block" style="background-color: #cccccc" valign="middle">
 485 <big><strong><span class="block">&nbsp;Unicode::Normalize - Unicode Normalization Forms</span></strong></big>
 486 </td></tr>
 487 </table>
 488
 489 </body>
 490
 491 </html>