1 @node Strings and Characters
2 @chapter Strings and Characters
4 @c Copyright (C) 2009--2024 Free Software Foundation, Inc.
6 @c Permission is granted to copy, distribute and/or modify this document
7 @c under the terms of the GNU Free Documentation License, Version 1.3 or
8 @c any later version published by the Free Software Foundation; with no
9 @c Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
10 @c copy of the license is at <https://www.gnu.org/licenses/fdl-1.3.en.html>.
12 @c Written by Bruno Haible.
14 This chapter describes the APIs for strings and characters, provided by Gnulib.
24 Several possible representations exist for the representation of strings
25 in memory of a running C program.
29 * Iterating through strings::
30 * Strings with NUL characters::
31 * String Functions in C Locale::
32 * Comparison of string APIs::
36 @subsection The C string representation
38 The classical representation of a string in C is a sequence of
39 characters, where each character takes up one or more bytes, followed by
40 a terminating NUL byte. This representation is used for strings that
41 are passed by the operating system (in the @code{argv} argument of
42 @code{main}, for example) and for strings that are passed to the
43 operating system (in system calls such as @code{open}). The C type to
44 hold such strings is @samp{char *} or, in places where the string shall
45 not be modified, @samp{const char *}. There are many C library
46 functions, standardized by ISO C and POSIX, that assume this
47 representation of strings.
49 A @emph{character encoding}, or @emph{encoding} for short, describes
50 how the elements of a character set are represented as a sequence of
51 bytes. For example, in the @code{ASCII} encoding, the UNDERSCORE
52 character is represented by a single byte, with value 0x5F. As another
53 example, the COPYRIGHT SIGN character is represented:
56 in the @code{ISO-8859-1} encoding, by the single byte 0xA9,
58 in the @code{UTF-8} encoding, by the two bytes 0xC2 0xA9,
60 in the @code{GB18030} encoding, by the four bytes 0x81 0x30 0x84 0x38.
64 Note: The @samp{char} type may be signed or unsigned, depending on the
65 platform. When we talk about the "byte 0xA9" we actually mean the
66 @code{char} object whose value is @code{(char) 0xA9}; we omit the cast
67 to @code{char} in this documentation, for brevity.
69 In POSIX, the character encoding is determined by the locale. The
70 locale is some environmental attribute that the user can choose.
72 Depending on the encoding, in general, every character is represented by
73 one or more bytes (up to 4 bytes in practice -- but
74 use @code{MB_LEN_MAX} instead of the number 4 in the code).
75 @cindex unibyte locale
76 @cindex multibyte locale
77 When every character is represented by only 1 byte, we speak of an
78 ``unibyte locale'', otherwise of a ``multibyte locale''.
80 It is important to realize that the majority of Unix installations
81 nowadays use UTF-8 as locale encoding; therefore, the
82 majority of users are using multibyte locales.
84 Three important facts to remember are:
87 @emph{A @samp{char} is a byte, not a character.}
93 The @posixheader{ctype.h} API, that was designed only with unibyte
94 encodings in mind, is useless nowadays for general text processing; it
95 does not work in multibyte locales.
97 The @posixfunc{strlen} function does not return the number of characters
98 in a string. Nor does it return the number of screen columns occupied
99 by a string after it is output. It merely returns the number of
100 @emph{bytes} occupied by a string.
102 Truncating a string, for example, with @posixfunc{strncpy}, can have the
103 effect of truncating it in the middle of a multibyte character. Such
104 a string will, when output, have a garbled character at its end, often
105 represented by a hollow box.
109 @emph{Multibyte does not imply UTF-8 encoding.}
112 While UTF-8 is the most common multibyte encoding, GB18030 is also a
113 supported locale encoding on GNU systems (mostly because it is a Chinese
114 government standard, last revised in 2022).
117 @emph{Searching for a character in a string is not the same as searching
118 for a byte in the string.}
121 Take the above example of COPYRIGHT SIGN in the @code{GB18030} encoding:
122 A byte search will find the bytes @code{'0'} and @code{'8'} in this
123 string. But a search for the @emph{character} "0" or "8" in the string
124 "@copyright{}" must, of course, report ``not found''.
129 @posixfunc{strchr} and @posixfunc{strrchr} do not work with multibyte
130 strings if the locale encoding is GB18030 and the character to be
133 @posixfunc{strstr} does not work with multibyte strings if the locale
134 encoding is different from UTF-8.
136 @posixfunc{strcspn}, @posixfunc{strpbrk}, @posixfunc{strspn} cannot work
137 correctly in multibyte locales: they assume the second argument is a
138 list of single-byte characters. Even in this simple case, they do not
139 work with multibyte strings if the locale encoding is GB18030 and one of
140 the characters to be searched is a digit.
142 @posixfunc{strsep} and @posixfunc{strtok_r} do not work with multibyte
143 strings unless all of the delimiter characters are ASCII characters
146 The @posixfunc{strcasecmp}, @posixfunc{strncasecmp}, and
147 @posixfunc{strcasestr} functions do not work with multibyte strings.
150 Workarounds can be found in Gnulib, in the form of @code{mbs*} API
154 Gnulib has functions @func{mbslen} and @func{mbswidth} that can be used
155 instead of @posixfunc{strlen} when the number of characters or the
156 number of screen columns of a string is requested.
158 Gnulib has functions @func{mbschr} and @func{mbsrrchr} that are like
159 @posixfunc{strchr} and @posixfunc{strrchr}, but work in multibyte
162 Gnulib has a function @func{mbsstr} that is like @posixfunc{strstr}, but
163 works in multibyte locales.
165 Gnulib has functions @func{mbscspn}, @func{mbspbrk}, @func{mbsspn} that
166 are like @posixfunc{strcspn}, @posixfunc{strpbrk}, @posixfunc{strspn},
167 but work in multibyte locales.
169 Gnulib has functions @func{mbssep} and @func{mbstok_r} that are like
170 @posixfunc{strsep} and @posixfunc{strtok_r} but work in multibyte
173 Gnulib has functions @func{mbscasecmp}, @func{mbsncasecmp},
174 @func{mbspcasecmp}, and @func{mbscasestr} that are like
175 @posixfunc{strcasecmp}, @posixfunc{strncasecmp}, and
176 @posixfunc{strcasestr}, but work in multibyte locales. Still, the
177 function @code{ulc_casecmp} is preferable to these functions.
181 @emph{A C string can contain encoding errors.}
184 Not every NUL-terminated byte sequence represents a valid multibyte
185 string. Byte sequences can contain encoding errors, that is, bytes or
186 byte sequences that are invalid and do not represent characters.
188 String functions like @code{mbscasecmp} and @code{strcoll} whose
189 behavior depends on encoding have unspecified behavior on strings
190 containing encoding errors, unless the behavior is specifically
191 documented. If an application needs a particular behavior on these
192 strings it can iterate through them itself, as described in the next
195 @node Iterating through strings
196 @subsection Iterating through strings
198 For complex string processing, string functions may not be enough, and
199 you need to iterate through a string while processing each (possibly
200 multibyte) character or encoding error in turn. Gnulib has several
201 modules for iterating forward through a string in this way. Backward
202 iteration, that is, from the string's end to start, is not provided,
203 as it is too hairy in general.
207 The @code{mbiter} module iterates through a string whose length
208 is already known. The string can contain NULs and encoding errors.
210 The @code{mbiterf} module is like @code{mbiter}
211 except it is more complex and typically faster.
213 The @code{mbuiter} module iterates through a C string whose length
214 is not a-priori known. The string can contain encoding errors and is
215 terminated by the first NUL.
217 The @code{mbuiterf} module is like @code{mbuiter}
218 except it is more complex and typically faster.
220 The @code{mcel} module is simpler than @code{mbiter} and @code{mbuiter}
221 and can be faster than even @code{mbiterf} and @code{mbuiterf}.
222 It can iterate through either strings whose length is known, or
223 C strings, or strings terminated by other ASCII characters < 0x30.
225 The @code{mcel-prefer} module is like @code{mcel} except that it
226 causes some other modules to be based on @code{mcel} instead of
227 on the @code{mbiter} family.
230 The choice of modules depends on the application's needs. The
231 @code{mbiter} module family is more suitable for applications that
232 treat some sequences of two or more bytes as a single encoding error,
233 and for applications that need to support obsolescent encodings on
234 non-GNU platforms, such as CP864, EBCDIC, Johab, and Shift JIS.
235 In this module family, @code{mbuiter} and @code{mbuiterf} are more
236 suitable than @code{mbiter} and @code{mbiterf} when arguments are C strings,
237 lengths are not already known, and it is highly likely that only the
238 first few multibyte characters need to be inspected.
240 The @code{mcel} module is simpler and can be faster than the
241 @code{mbiter} family, and is more suitable for applications that do
242 not need the @code{mbiter} family's special features.
244 The @code{mcel-prefer} module is like @code{mcel} except that it also
245 causes some other modules, such as @code{mbscasecmp}, to use
246 @code{mcel} rather than the @code{mbiter} family. This can be simpler
247 and faster. However, it does not support the obsolescent encodings,
248 and it may behave differently on data containing encoding errors where
249 behavior is unspecified or undefined, because in @code{mcel} each
250 encoding error is a single byte whereas in the @code{mbiter} family a
251 single encoding error can contain two or more bytes.
253 If a package uses @code{mcel-prefer}, it may also want to give
254 @command{gnulib-tool} one or more of the options
255 @option{--avoid=mbiter}, @option{--avoid=mbiterf},
256 @option{--avoid=mbuiter} and @option{--avoid=mbuiterf},
257 to avoid packaging modules that are not needed.
259 @node Strings with NUL characters
260 @subsection Strings with NUL characters
262 The GNU Coding Standards, section
264 @ref{Semantics,,Writing Robust Programs,standards},
267 @url{https://www.gnu.org/prep/standards/html_node/Semantics.html},
271 Utilities reading files should not drop NUL characters, or any other
272 nonprinting characters.
275 When it is a requirement to store NUL characters in strings, a variant
276 of the C strings is needed. Gnulib offers a ``string descriptor'' type
277 for this purpose. See @ref{Handling strings with NUL characters}.
279 All remarks regarding encodings and multibyte characters in the previous
280 section apply to string descriptors as well.
282 @include c-locale.texi
284 @node Comparison of string APIs
285 @subsection Comparison of string APIs
287 This table summarizes the API functions available for strings, in POSIX
290 @multitable @columnfractions .17 .17 .17 .17 .16 .16
291 @headitem unibyte strings only
293 @tab multibyte strings
294 @tab multibyte strings with NULs
295 @tab wide character strings
296 @tab 32-bit wide character strings
301 @tab @code{string_desc_length}
303 @tab @code{u32_strlen}
310 @tab @code{u32_strnlen}, @code{u32_mbsnlen}
315 @tab @code{string_desc_cmp}
317 @tab @code{u32_strcmp}
324 @tab @code{u32_strncmp}
326 @item @code{strcasecmp}
327 @tab @code{strcasecmp}
328 @tab @code{mbscasecmp}
330 @tab @code{wcscasecmp}
331 @tab @code{u32_casecmp}
333 @item @code{strncasecmp}
334 @tab @code{strncasecmp}
335 @tab @code{mbsncasecmp}, @code{mbspcasecmp}
337 @tab @code{wcsncasecmp}
338 @tab @code{u32_casecmp}
345 @tab @code{u32_strcoll}
357 @tab @code{string_desc_index}
359 @tab @code{u32_strchr}
364 @tab @code{string_desc_last_index}
366 @tab @code{u32_strrchr}
371 @tab @code{string_desc_contains}
373 @tab @code{u32_strstr}
375 @item @code{strcasestr}
376 @tab @code{strcasestr}
377 @tab @code{mbscasestr}
387 @tab @code{u32_strspn}
394 @tab @code{u32_strcspn}
401 @tab @code{u32_strpbrk}
403 @item @code{strtok_r}
408 @tab @code{u32_strtok}
420 @tab @code{string_desc_copy}
422 @tab @code{u32_strcpy}
429 @tab @code{u32_stpcpy}
436 @tab @code{u32_strncpy}
443 @tab @code{u32_stpncpy}
448 @tab @code{string_desc_concat}
450 @tab @code{u32_strcat}
457 @tab @code{u32_strncat}
462 @tab @code{string_desc_free}
469 @tab @code{string_desc_copy}
471 @tab @code{u32_strdup}
480 @item @code{mbswidth}
485 @tab @code{c32swidth}, @code{u32_strwidth}
508 @item @code{strtoull}
515 @item @code{strtoimax}
516 @tab @code{strtoimax}
517 @tab @code{strtoimax}
519 @tab @code{wcstoimax}
522 @item @code{strtoumax}
523 @tab @code{strtoumax}
524 @tab @code{strtoumax}
526 @tab @code{wcstoumax}
544 @tab @code{c_strtold}
550 @item @code{strfromf}
557 @item @code{strfromd}
564 @item @code{strfroml}
576 @tab @code{mbstoc32s}
582 @tab @code{mbsrtowcs}
583 @tab @code{mbsrtoc32s}
589 @tab @code{mbsnrtowcs}
590 @tab @code{mbsnrtoc32s}
597 @tab @code{c32stombs}
603 @tab @code{wcsrtombs}
604 @tab @code{c32srtombs}
610 @tab @code{wcsnrtombs}
611 @tab @code{c32snrtombs}
618 A @emph{character} is the elementary unit that strings are made of.
620 What is a character? ``A character is an element of a character set''
621 is sort of a circular definition, but it highlights the fact that it is
622 not merely a number. Although many characters are visually represented
623 by a single glyph, there are characters that, for example, have a
624 different glyph when used at the end of a word than when used inside a
625 word. A character is also not the minimal rendered text processing
626 unit; that is a grapheme cluster and in general consists of one or more
627 characters. If you want to know more about the concept of character and
628 various concepts associated with characters, refer to the Unicode
631 For the representation in memory of a character, various types have been
632 in use, and some of them were failures: @code{char} and @code{wchar_t}
633 were invented for this purpose, but are not the right types.
634 @code{char32_t} is the right type (successor of @code{wchar_t}); and
635 @code{mbchar_t} (defined by Gnulib) is an alternative for specific kinds
641 * The char32_t type::
642 * The mbchar_t type::
643 * Comparison of character APIs::
647 @subsection The @code{char} type
649 The @code{char} type is in the C language since the beginning in the
650 1970ies, but -- due to its limitation of 256 possible values -- is no
651 longer the adequate type for storing a character.
653 Technically, it is still adequate in unibyte locales. But since most
654 locales nowadays are multibyte locales, it makes no sense to write a
655 program that runs only in unibyte locales.
657 ISO C and POSIX standardized an API for characters of type @code{char},
658 in @code{<ctype.h>}. This API is nowadays useless and obsolete, when it
659 comes to general text processing.
661 The important lessons to remember are:
664 @emph{A @samp{char} is just the elementary storage unit for a string,
669 @emph{Never use @code{<ctype.h>}!}
672 @node The wchar_t type
673 @subsection The @code{wchar_t} type
675 The ISO C and POSIX standard creators made an attempt to overcome the
676 dead end regarding the @code{char} type. They introduced
679 a type @samp{wchar_t}, designed to encapsulate a character,
681 a ``wide string'' type @samp{wchar_t *}, with some API functions
682 declared in @posixheader{wchar.h}, and
684 functions declared in @posixheader{wctype.h} that were meant to supplant
685 the ones in @posixheader{ctype.h}.
688 Unfortunately, this API and its implementation has numerous problems:
692 On Windows platforms and on AIX in 32-bit mode, @code{wchar_t} is a
693 16-bit type. This means that it can never accommodate an entire Unicode
694 character. Either the @code{wchar_t *} strings are limited to
695 characters in UCS-2 (the ``Basic Multilingual Plane'' of Unicode), or
696 -- if @code{wchar_t *} strings are encoded in UTF-16 -- a
697 @code{wchar_t} represents only half of a character in the worst case,
698 making the @posixheader{wctype.h} functions pointless.
701 On Solaris and FreeBSD, the @code{wchar_t} encoding is locale dependent
702 and undocumented. This means, if you want to know any property of a
703 @code{wchar_t} character, other than the properties defined by
704 @posixheader{wctype.h} -- such as whether it's a dash, currency symbol,
705 paragraph separator, or similar --, you have to convert it to
706 @code{char *} encoding first, by use of the function @posixfunc{wctomb}.
709 When you read a stream of wide characters, through the functions
710 @posixfunc{fgetwc} and @posixfunc{fgetws}, and when the input
711 stream/file is not in the expected encoding, you have no way to
712 determine the invalid byte sequence and do some corrective action. If
713 you use these functions, your program becomes ``garbage in - more
714 garbage out'' or ``garbage in - abort''.
717 As a consequence, it is better to use multibyte strings. Such multibyte
718 strings can bypass limitations of the @code{wchar_t} type, if you use
719 functions defined in Gnulib and GNU libunistring for text processing.
720 They can also faithfully transport malformed characters that were
721 present in the input, without requiring the program to produce garbage
724 @node The char32_t type
725 @subsection The @code{char32_t} type
727 The ISO C and POSIX standard creators then introduced the
728 @code{char32_t} type. In ISO C 11, it was conceptually a ``32-bit wide
729 character'' type. In ISO C 23, its semantics has been further
730 specified: A @code{char32_t} value is a Unicode code point.
732 Thus, the @code{char32_t} type is not affected the problems that plague
733 the @code{wchar_t} type.
735 The @code{char32_t} type and its API are defined in the @code{<uchar.h>}
738 ISO C and POSIX specify only the basic functions for the @code{char32_t}
739 type, namely conversion of a single character (@func{mbrtoc32} and
740 @func{c32rtomb}). For convenience, Gnulib adds API for classification
741 and case conversion of characters.
743 GNU libunistring can also be used on @code{char32_t} values. Since
744 @code{char32_t} is the same as @code{uint32_t}, all @code{u32_*}
745 functions of GNU libunistring are applicable to arrays of
746 @code{char32_t} values.
748 On glibc systems, use of the 32-bit wide strings (@code{char32_t[]}) is
749 exactly as efficient as the use of the older wide strings
750 (@code{wchar_t[]}). This is possible because on glibc, @code{wchar_t}
751 values already always were 32-bit and Unicode code points.
752 @code{mbrtoc32} is just an alias of @code{mbrtowc}. The Gnulib
753 @code{*c32*} functions are optimized so that on glibc systems they
754 immediately redirect to the corresponding @code{*wc*} functions.
756 Gnulib implements the ISO C 23 semantics of @code{char32_t} when you
757 import the @samp{uchar-c23} module. Without this module, it implements
758 only the ISO C 11 semantics; the effect is that on some platforms
759 (macOS, FreeBSD, NetBSD, Solaris) a @code{char32_t} value is the same
760 as a @code{wchar_t} value, not a Unicode code point. Thus, when you
761 want to pass @code{char32_t} values to GNU libunistring or to some Unicode
762 centric Gnulib functions, you need the @samp{uchar-c23} module in order
763 to do so without portability problems.
765 @node The mbchar_t type
766 @subsection The @code{mbchar_t} type
768 Gnulib defines an alternate way to encode a multibyte character:
769 @code{mbchar_t}. Its main feature is the ability to process a string or
770 stream with some malformed characters without reporting an error.
772 The type @code{mbchar_t}, defined in @code{"mbchar.h"}, holds a
773 character in both the multibyte and the 32-bit wide character
774 representation. In case of a malformed character only the multibyte
775 representation is used.
778 * Reading multibyte strings::
781 @node Reading multibyte strings
782 @subsubsection Reading multibyte strings
784 If you want to process (possibly multibyte) characters while reading
785 them from a @code{FILE *} stream, without reading them into a string
786 first, the @code{mbfile} module is made for this purpose.
788 @node Comparison of character APIs
789 @subsection Comparison of character APIs
791 This table summarizes the API functions available for characters, in
794 @multitable @columnfractions .2 .2 .2 .2 .2
795 @headitem unibyte character
798 @tab 32-bit wide character
799 @tab mbchar_t character
814 @tab @code{c_isalnum}
816 @tab @code{c32isalnum}
817 @tab @code{mb_isalnum}
820 @tab @code{c_isalpha}
822 @tab @code{c32isalpha}
823 @tab @code{mb_isalpha}
826 @tab @code{c_isblank}
828 @tab @code{c32isblank}
829 @tab @code{mb_isblank}
832 @tab @code{c_iscntrl}
834 @tab @code{c32iscntrl}
835 @tab @code{mb_iscntrl}
838 @tab @code{c_isdigit}
840 @tab @code{c32isdigit}
841 @tab @code{mb_isdigit}
844 @tab @code{c_isgraph}
846 @tab @code{c32isgraph}
847 @tab @code{mb_isgraph}
850 @tab @code{c_islower}
852 @tab @code{c32islower}
853 @tab @code{mb_islower}
856 @tab @code{c_isprint}
858 @tab @code{c32isprint}
859 @tab @code{mb_isprint}
862 @tab @code{c_ispunct}
864 @tab @code{c32ispunct}
865 @tab @code{mb_ispunct}
868 @tab @code{c_isspace}
870 @tab @code{c32isspace}
871 @tab @code{mb_isspace}
874 @tab @code{c_isupper}
876 @tab @code{c32isupper}
877 @tab @code{mb_isupper}
879 @item @code{isxdigit}
880 @tab @code{c_isxdigit}
881 @tab @code{iswxdigit}
882 @tab @code{c32isxdigit}
883 @tab @code{mb_isxdigit}
888 @tab @code{c32_get_type_test}
894 @tab @code{c32_apply_type_test}
898 @tab @code{c_tolower}
900 @tab @code{c32tolower}
904 @tab @code{c_toupper}
906 @tab @code{c32toupper}
912 @tab @code{c32_get_mapping}
917 @tab @code{towctrans}
918 @tab @code{c32_apply_mapping}