2 @c This is part of the GNU Emacs Lisp Reference Manual.
3 @c Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004,
4 @c 2005, 2006, 2007, 2008, 2009, 2010 Free Software Foundation, Inc.
5 @c See the file elisp.texi for copying conditions.
6 @setfilename ../../info/characters
7 @node Non-ASCII Characters, Searching and Matching, Text, Top
8 @chapter Non-@acronym{ASCII} Characters
9 @cindex multibyte characters
10 @cindex characters, multi-byte
11 @cindex non-@acronym{ASCII} characters
13 This chapter covers the special issues relating to characters and
14 how they are stored in strings and buffers.
17 * Text Representations:: How Emacs represents text.
18 * Converting Representations:: Converting unibyte to multibyte and vice versa.
19 * Selecting a Representation:: Treating a byte sequence as unibyte or multi.
20 * Character Codes:: How unibyte and multibyte relate to
21 codes of individual characters.
22 * Character Properties:: Character attributes that define their
23 behavior and handling.
24 * Character Sets:: The space of possible character codes
25 is divided into various character sets.
26 * Scanning Charsets:: Which character sets are used in a buffer?
27 * Translation of Characters:: Translation tables are used for conversion.
28 * Coding Systems:: Coding systems are conversions for saving files.
29 * Input Methods:: Input methods allow users to enter various
30 non-ASCII characters without special keyboards.
31 * Locales:: Interacting with the POSIX locale.
34 @node Text Representations
35 @section Text Representations
36 @cindex text representation
38 Emacs buffers and strings support a large repertoire of characters
39 from many different scripts, allowing users to type and display text
40 in almost any known written language.
42 @cindex character codepoint
45 To support this multitude of characters and scripts, Emacs closely
46 follows the @dfn{Unicode Standard}. The Unicode Standard assigns a
47 unique number, called a @dfn{codepoint}, to each and every character.
48 The range of codepoints defined by Unicode, or the Unicode
49 @dfn{codespace}, is @code{0..#x10FFFF} (in hexadecimal notation),
50 inclusive. Emacs extends this range with codepoints in the range
51 @code{#x110000..#x3FFFFF}, which it uses for representing characters
52 that are not unified with Unicode and @dfn{raw 8-bit bytes} that
53 cannot be interpreted as characters. Thus, a character codepoint in
54 Emacs is a 22-bit integer number.
56 @cindex internal representation of characters
57 @cindex characters, representation in buffers and strings
58 @cindex multibyte text
59 To conserve memory, Emacs does not hold fixed-length 22-bit numbers
60 that are codepoints of text characters within buffers and strings.
61 Rather, Emacs uses a variable-length internal representation of
62 characters, that stores each character as a sequence of 1 to 5 8-bit
63 bytes, depending on the magnitude of its codepoint@footnote{
64 This internal representation is based on one of the encodings defined
65 by the Unicode Standard, called @dfn{UTF-8}, for representing any
66 Unicode codepoint, but Emacs extends UTF-8 to represent the additional
67 codepoints it uses for raw 8-bit bytes and characters not unified with
68 Unicode.}. For example, any @acronym{ASCII} character takes up only 1
69 byte, a Latin-1 character takes up 2 bytes, etc. We call this
70 representation of text @dfn{multibyte}.
72 Outside Emacs, characters can be represented in many different
73 encodings, such as ISO-8859-1, GB-2312, Big-5, etc. Emacs converts
74 between these external encodings and its internal representation, as
75 appropriate, when it reads text into a buffer or a string, or when it
76 writes text to a disk file or passes it to some other process.
78 Occasionally, Emacs needs to hold and manipulate encoded text or
79 binary non-text data in its buffers or strings. For example, when
80 Emacs visits a file, it first reads the file's text verbatim into a
81 buffer, and only then converts it to the internal representation.
82 Before the conversion, the buffer holds encoded text.
85 Encoded text is not really text, as far as Emacs is concerned, but
86 rather a sequence of raw 8-bit bytes. We call buffers and strings
87 that hold encoded text @dfn{unibyte} buffers and strings, because
88 Emacs treats them as a sequence of individual bytes. Usually, Emacs
89 displays unibyte buffers and strings as octal codes such as
90 @code{\237}. We recommend that you never use unibyte buffers and
91 strings except for manipulating encoded text or binary non-text data.
93 In a buffer, the buffer-local value of the variable
94 @code{enable-multibyte-characters} specifies the representation used.
95 The representation for a string is determined and recorded in the string
96 when the string is constructed.
98 @defvar enable-multibyte-characters
99 This variable specifies the current buffer's text representation.
100 If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
101 it contains unibyte encoded text or binary non-text data.
103 You cannot set this variable directly; instead, use the function
104 @code{set-buffer-multibyte} to change a buffer's representation.
107 @defun position-bytes position
108 Buffer positions are measured in character units. This function
109 returns the byte-position corresponding to buffer position
110 @var{position} in the current buffer. This is 1 at the start of the
111 buffer, and counts upward in bytes. If @var{position} is out of
112 range, the value is @code{nil}.
115 @defun byte-to-position byte-position
116 Return the buffer position, in character units, corresponding to given
117 @var{byte-position} in the current buffer. If @var{byte-position} is
118 out of range, the value is @code{nil}. In a multibyte buffer, an
119 arbitrary value of @var{byte-position} can be not at character
120 boundary, but inside a multibyte sequence representing a single
121 character; in this case, this function returns the buffer position of
122 the character whose multibyte sequence includes @var{byte-position}.
123 In other words, the value does not change for all byte positions that
124 belong to the same character.
127 @defun multibyte-string-p string
128 Return @code{t} if @var{string} is a multibyte string, @code{nil}
132 @defun string-bytes string
133 @cindex string, number of bytes
134 This function returns the number of bytes in @var{string}.
135 If @var{string} is a multibyte string, this can be greater than
136 @code{(length @var{string})}.
139 @defun unibyte-string &rest bytes
140 This function concatenates all its argument @var{bytes} and makes the
141 result a unibyte string.
144 @node Converting Representations
145 @section Converting Text Representations
147 Emacs can convert unibyte text to multibyte; it can also convert
148 multibyte text to unibyte, provided that the multibyte text contains
149 only @acronym{ASCII} and 8-bit raw bytes. In general, these
150 conversions happen when inserting text into a buffer, or when putting
151 text from several strings together in one string. You can also
152 explicitly convert a string's contents to either representation.
154 Emacs chooses the representation for a string based on the text from
155 which it is constructed. The general rule is to convert unibyte text
156 to multibyte text when combining it with other multibyte text, because
157 the multibyte representation is more general and can hold whatever
158 characters the unibyte text has.
160 When inserting text into a buffer, Emacs converts the text to the
161 buffer's representation, as specified by
162 @code{enable-multibyte-characters} in that buffer. In particular, when
163 you insert multibyte text into a unibyte buffer, Emacs converts the text
164 to unibyte, even though this conversion cannot in general preserve all
165 the characters that might be in the multibyte text. The other natural
166 alternative, to convert the buffer contents to multibyte, is not
167 acceptable because the buffer's representation is a choice made by the
168 user that cannot be overridden automatically.
170 Converting unibyte text to multibyte text leaves @acronym{ASCII}
171 characters unchanged, and converts bytes with codes 128 through 159 to
172 the multibyte representation of raw eight-bit bytes.
174 Converting multibyte text to unibyte converts all @acronym{ASCII}
175 and eight-bit characters to their single-byte form, but loses
176 information for non-@acronym{ASCII} characters by discarding all but
177 the low 8 bits of each character's codepoint. Converting unibyte text
178 to multibyte and back to unibyte reproduces the original unibyte text.
180 The next two functions either return the argument @var{string}, or a
181 newly created string with no text properties.
183 @defun string-to-multibyte string
184 This function returns a multibyte string containing the same sequence
185 of characters as @var{string}. If @var{string} is a multibyte string,
186 it is returned unchanged. The function assumes that @var{string}
187 includes only @acronym{ASCII} characters and raw 8-bit bytes; the
188 latter are converted to their multibyte representation corresponding
189 to the codepoints @code{#x3FFF80} through @code{#x3FFFFF}, inclusive
190 (@pxref{Text Representations, codepoints}).
193 @defun string-to-unibyte string
194 This function returns a unibyte string containing the same sequence of
195 characters as @var{string}. It signals an error if @var{string}
196 contains a non-@acronym{ASCII} character. If @var{string} is a
197 unibyte string, it is returned unchanged. Use this function for
198 @var{string} arguments that contain only @acronym{ASCII} and eight-bit
202 @defun multibyte-char-to-unibyte char
203 This converts the multibyte character @var{char} to a unibyte
204 character, and returns that character. If @var{char} is neither
205 @acronym{ASCII} nor eight-bit, the function returns -1.
208 @defun unibyte-char-to-multibyte char
209 This convert the unibyte character @var{char} to a multibyte
210 character, assuming @var{char} is either @acronym{ASCII} or raw 8-bit
214 @node Selecting a Representation
215 @section Selecting a Representation
217 Sometimes it is useful to examine an existing buffer or string as
218 multibyte when it was unibyte, or vice versa.
220 @defun set-buffer-multibyte multibyte
221 Set the representation type of the current buffer. If @var{multibyte}
222 is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
223 is @code{nil}, the buffer becomes unibyte.
225 This function leaves the buffer contents unchanged when viewed as a
226 sequence of bytes. As a consequence, it can change the contents
227 viewed as characters; for instance, a sequence of three bytes which is
228 treated as one character in multibyte representation will count as
229 three characters in unibyte representation. Eight-bit characters
230 representing raw bytes are an exception. They are represented by one
231 byte in a unibyte buffer, but when the buffer is set to multibyte,
232 they are converted to two-byte sequences, and vice versa.
234 This function sets @code{enable-multibyte-characters} to record which
235 representation is in use. It also adjusts various data in the buffer
236 (including overlays, text properties and markers) so that they cover the
237 same text as they did before.
239 You cannot use @code{set-buffer-multibyte} on an indirect buffer,
240 because indirect buffers always inherit the representation of the
244 @defun string-as-unibyte string
245 If @var{string} is already a unibyte string, this function returns
246 @var{string} itself. Otherwise, it returns a new string with the same
247 bytes as @var{string}, but treating each byte as a separate character
248 (so that the value may have more characters than @var{string}); as an
249 exception, each eight-bit character representing a raw byte is
250 converted into a single byte. The newly-created string contains no
254 @defun string-as-multibyte string
255 If @var{string} is a multibyte string, this function returns
256 @var{string} itself. Otherwise, it returns a new string with the same
257 bytes as @var{string}, but treating each multibyte sequence as one
258 character. This means that the value may have fewer characters than
259 @var{string} has. If a byte sequence in @var{string} is invalid as a
260 multibyte representation of a single character, each byte in the
261 sequence is treated as a raw 8-bit byte. The newly-created string
262 contains no text properties.
265 @node Character Codes
266 @section Character Codes
267 @cindex character codes
269 The unibyte and multibyte text representations use different
270 character codes. The valid character codes for unibyte representation
271 range from 0 to @code{#xFF} (255)---the values that can fit in one
272 byte. The valid character codes for multibyte representation range
273 from 0 to @code{#x3FFFFF}. In this code space, values 0 through
274 @code{#x7F} (127) are for @acronym{ASCII} characters, and values
275 @code{#x80} (128) through @code{#x3FFF7F} (4194175) are for
276 non-@acronym{ASCII} characters.
278 Emacs character codes are a superset of the Unicode standard.
279 Values 0 through @code{#x10FFFF} (1114111) correspond to Unicode
280 characters of the same codepoint; values @code{#x110000} (1114112)
281 through @code{#x3FFF7F} (4194175) represent characters that are not
282 unified with Unicode; and values @code{#x3FFF80} (4194176) through
283 @code{#x3FFFFF} (4194303) represent eight-bit raw bytes.
285 @defun characterp charcode
286 This returns @code{t} if @var{charcode} is a valid character, and
287 @code{nil} otherwise.
305 @cindex maximum value of character codepoint
306 @cindex codepoint, largest value
308 This function returns the largest value that a valid character
313 (characterp (max-char))
317 (characterp (1+ (max-char)))
323 @defun get-byte &optional pos string
324 This function returns the byte at character position @var{pos} in the
325 current buffer. If the current buffer is unibyte, this is literally
326 the byte at that position. If the buffer is multibyte, byte values of
327 @acronym{ASCII} characters are the same as character codepoints,
328 whereas eight-bit raw bytes are converted to their 8-bit codes. The
329 function signals an error if the character at @var{pos} is
332 The optional argument @var{string} means to get a byte value from that
333 string instead of the current buffer.
336 @node Character Properties
337 @section Character Properties
338 @cindex character properties
339 A @dfn{character property} is a named attribute of a character that
340 specifies how the character behaves and how it should be handled
341 during text processing and display. Thus, character properties are an
342 important part of specifying the character's semantics.
344 On the whole, Emacs follows the Unicode Standard in its implementation
345 of character properties. In particular, Emacs supports the
346 @uref{http://www.unicode.org/reports/tr23/, Unicode Character Property
347 Model}, and the Emacs character property database is derived from the
348 Unicode Character Database (@acronym{UCD}). See the
349 @uref{http://www.unicode.org/versions/Unicode5.0.0/ch04.pdf, Character
350 Properties chapter of the Unicode Standard}, for a detailed
351 description of Unicode character properties and their meaning. This
352 section assumes you are already familiar with that chapter of the
353 Unicode Standard, and want to apply that knowledge to Emacs Lisp
356 In Emacs, each property has a name, which is a symbol, and a set of
357 possible values, whose types depend on the property; if a character
358 does not have a certain property, the value is @code{nil}. As a
359 general rule, the names of character properties in Emacs are produced
360 from the corresponding Unicode properties by downcasing them and
361 replacing each @samp{_} character with a dash @samp{-}. For example,
362 @code{Canonical_Combining_Class} becomes
363 @code{canonical-combining-class}. However, sometimes we shorten the
364 names to make their use easier.
366 Here is the full list of value types for all the character
367 properties that Emacs knows about:
371 This property corresponds to the Unicode @code{Name} property. The
372 value is a string consisting of upper-case Latin letters A to Z,
373 digits, spaces, and hyphen @samp{-} characters.
375 @cindex unicode general category
376 @item general-category
377 This property corresponds to the Unicode @code{General_Category}
378 property. The value is a symbol whose name is a 2-letter abbreviation
379 of the character's classification.
381 @item canonical-combining-class
382 Corresponds to the Unicode @code{Canonical_Combining_Class} property.
383 The value is an integer number.
386 Corresponds to the Unicode @code{Bidi_Class} property. The value is a
387 symbol whose name is the Unicode @dfn{directional type} of the
391 Corresponds to the Unicode @code{Decomposition_Type} and
392 @code{Decomposition_Value} properties. The value is a list, whose
393 first element may be a symbol representing a compatibility formatting
394 tag, such as @code{small}@footnote{
395 Note that the Unicode spec writes these tag names inside
396 @samp{<..>} brackets. The tag names in Emacs do not include the
397 brackets; e.g., Unicode specifies @samp{<small>} where Emacs uses
399 }; the other elements are characters that give the compatibility
400 decomposition sequence of this character.
402 @item decimal-digit-value
403 Corresponds to the Unicode @code{Numeric_Value} property for
404 characters whose @code{Numeric_Type} is @samp{Digit}. The value is an
408 Corresponds to the Unicode @code{Numeric_Value} property for
409 characters whose @code{Numeric_Type} is @samp{Decimal}. The value is
410 an integer number. Examples of such characters include compatibility
411 subscript and superscript digits, for which the value is the
412 corresponding number.
415 Corresponds to the Unicode @code{Numeric_Value} property for
416 characters whose @code{Numeric_Type} is @samp{Numeric}. The value of
417 this property is an integer or a floating-point number. Examples of
418 characters that have this property include fractions, subscripts,
419 superscripts, Roman numerals, currency numerators, and encircled
420 numbers. For example, the value of this property for the character
421 @code{U+2155} (@sc{vulgar fraction one fifth}) is @code{0.2}.
424 Corresponds to the Unicode @code{Bidi_Mirrored} property. The value
425 of this property is a symbol, either @code{Y} or @code{N}.
428 Corresponds to the Unicode @code{Unicode_1_Name} property. The value
431 @item iso-10646-comment
432 Corresponds to the Unicode @code{ISO_Comment} property. The value is
436 Corresponds to the Unicode @code{Simple_Uppercase_Mapping} property.
437 The value of this property is a single character.
440 Corresponds to the Unicode @code{Simple_Lowercase_Mapping} property.
441 The value of this property is a single character.
444 Corresponds to the Unicode @code{Simple_Titlecase_Mapping} property.
445 @dfn{Title case} is a special form of a character used when the first
446 character of a word needs to be capitalized. The value of this
447 property is a single character.
450 @defun get-char-code-property char propname
451 This function returns the value of @var{char}'s @var{propname} property.
455 (get-char-code-property ? 'general-category)
459 (get-char-code-property ?1 'general-category)
463 (get-char-code-property ?\u2084 'digit-value) ; subscript 4
467 (get-char-code-property ?\u2155 'numeric-value) ; one fifth
471 (get-char-code-property ?\u2163 'numeric-value) ; Roman IV
477 @defun char-code-property-description prop value
478 This function returns the description string of property @var{prop}'s
479 @var{value}, or @code{nil} if @var{value} has no description.
483 (char-code-property-description 'general-category 'Zs)
484 @result{} "Separator, Space"
487 (char-code-property-description 'general-category 'Nd)
488 @result{} "Number, Decimal Digit"
491 (char-code-property-description 'numeric-value '1/5)
497 @defun put-char-code-property char propname value
498 This function stores @var{value} as the value of the property
499 @var{propname} for the character @var{char}.
502 @defvar unicode-category-table
503 The value of this variable is a char-table (@pxref{Char-Tables}) that
504 specifies, for each character, its Unicode @code{General_Category}
505 property as a symbol.
508 @defvar char-script-table
509 The value of this variable is a char-table that specifies, for each
510 character, a symbol whose name is the script to which the character
511 belongs, according to the Unicode Standard classification of the
512 Unicode code space into script-specific blocks. This char-table has a
513 single extra slot whose value is the list of all script symbols.
516 @defvar char-width-table
517 The value of this variable is a char-table that specifies the width of
518 each character in columns that it will occupy on the screen.
521 @defvar printable-chars
522 The value of this variable is a char-table that specifies, for each
523 character, whether it is printable or not. That is, if evaluating
524 @code{(aref printable-chars char)} results in @code{t}, the character
525 is printable, and if it results in @code{nil}, it is not.
529 @section Character Sets
530 @cindex character sets
533 @cindex coded character set
534 An Emacs @dfn{character set}, or @dfn{charset}, is a set of characters
535 in which each character is assigned a numeric code point. (The
536 Unicode Standard calls this a @dfn{coded character set}.) Each Emacs
537 charset has a name which is a symbol. A single character can belong
538 to any number of different character sets, but it will generally have
539 a different code point in each charset. Examples of character sets
540 include @code{ascii}, @code{iso-8859-1}, @code{greek-iso8859-7}, and
541 @code{windows-1255}. The code point assigned to a character in a
542 charset is usually different from its code point used in Emacs buffers
545 @cindex @code{emacs}, a charset
546 @cindex @code{unicode}, a charset
547 @cindex @code{eight-bit}, a charset
548 Emacs defines several special character sets. The character set
549 @code{unicode} includes all the characters whose Emacs code points are
550 in the range @code{0..#x10FFFF}. The character set @code{emacs}
551 includes all @acronym{ASCII} and non-@acronym{ASCII} characters.
552 Finally, the @code{eight-bit} charset includes the 8-bit raw bytes;
553 Emacs uses it to represent raw bytes encountered in text.
555 @defun charsetp object
556 Returns @code{t} if @var{object} is a symbol that names a character set,
557 @code{nil} otherwise.
561 The value is a list of all defined character set names.
564 @defun charset-priority-list &optional highestp
565 This functions returns a list of all defined character sets ordered by
566 their priority. If @var{highestp} is non-@code{nil}, the function
567 returns a single character set of the highest priority.
570 @defun set-charset-priority &rest charsets
571 This function makes @var{charsets} the highest priority character sets.
574 @defun char-charset character &optional restriction
575 This function returns the name of the character set of highest
576 priority that @var{character} belongs to. @acronym{ASCII} characters
577 are an exception: for them, this function always returns @code{ascii}.
579 If @var{restriction} is non-@code{nil}, it should be a list of
580 charsets to search. Alternatively, it can be a coding system, in
581 which case the returned charset must be supported by that coding
582 system (@pxref{Coding Systems}).
585 @defun charset-plist charset
586 This function returns the property list of the character set
587 @var{charset}. Although @var{charset} is a symbol, this is not the
588 same as the property list of that symbol. Charset properties include
589 important information about the charset, such as its documentation
590 string, short name, etc.
593 @defun put-charset-property charset propname value
594 This function sets the @var{propname} property of @var{charset} to the
598 @defun get-charset-property charset propname
599 This function returns the value of @var{charset}s property
603 @deffn Command list-charset-chars charset
604 This command displays a list of characters in the character set
608 Emacs can convert between its internal representation of a character
609 and the character's codepoint in a specific charset. The following
610 two functions support these conversions.
612 @c FIXME: decode-char and encode-char accept and ignore an additional
613 @c argument @var{restriction}. When that argument actually makes a
614 @c difference, it should be documented here.
615 @defun decode-char charset code-point
616 This function decodes a character that is assigned a @var{code-point}
617 in @var{charset}, to the corresponding Emacs character, and returns
618 it. If @var{charset} doesn't contain a character of that code point,
619 the value is @code{nil}. If @var{code-point} doesn't fit in a Lisp
620 integer (@pxref{Integer Basics, most-positive-fixnum}), it can be
621 specified as a cons cell @code{(@var{high} . @var{low})}, where
622 @var{low} are the lower 16 bits of the value and @var{high} are the
626 @defun encode-char char charset
627 This function returns the code point assigned to the character
628 @var{char} in @var{charset}. If the result does not fit in a Lisp
629 integer, it is returned as a cons cell @code{(@var{high} . @var{low})}
630 that fits the second argument of @code{decode-char} above. If
631 @var{charset} doesn't have a codepoint for @var{char}, the value is
635 The following function comes in handy for applying a certain
636 function to all or part of the characters in a charset:
638 @defun map-charset-chars function charset &optional arg from-code to-code
639 Call @var{function} for characters in @var{charset}. @var{function}
640 is called with two arguments. The first one is a cons cell
641 @code{(@var{from} . @var{to})}, where @var{from} and @var{to}
642 indicate a range of characters contained in charset. The second
643 argument passed to @var{function} is @var{arg}.
645 By default, the range of codepoints passed to @var{function} includes
646 all the characters in @var{charset}, but optional arguments
647 @var{from-code} and @var{to-code} limit that to the range of
648 characters between these two codepoints of @var{charset}. If either
649 of them is @code{nil}, it defaults to the first or last codepoint of
650 @var{charset}, respectively.
653 @node Scanning Charsets
654 @section Scanning for Character Sets
656 Sometimes it is useful to find out which character set a particular
657 character belongs to. One use for this is in determining which coding
658 systems (@pxref{Coding Systems}) are capable of representing all of
659 the text in question; another is to determine the font(s) for
660 displaying that text.
662 @defun charset-after &optional pos
663 This function returns the charset of highest priority containing the
664 character at position @var{pos} in the current buffer. If @var{pos}
665 is omitted or @code{nil}, it defaults to the current value of point.
666 If @var{pos} is out of range, the value is @code{nil}.
669 @defun find-charset-region beg end &optional translation
670 This function returns a list of the character sets of highest priority
671 that contain characters in the current buffer between positions
672 @var{beg} and @var{end}.
674 The optional argument @var{translation} specifies a translation table
675 to use for scanning the text (@pxref{Translation of Characters}). If
676 it is non-@code{nil}, then each character in the region is translated
677 through this table, and the value returned describes the translated
678 characters instead of the characters actually in the buffer.
681 @defun find-charset-string string &optional translation
682 This function returns a list of character sets of highest priority
683 that contain characters in @var{string}. It is just like
684 @code{find-charset-region}, except that it applies to the contents of
685 @var{string} instead of part of the current buffer.
688 @node Translation of Characters
689 @section Translation of Characters
690 @cindex character translation tables
691 @cindex translation tables
693 A @dfn{translation table} is a char-table (@pxref{Char-Tables}) that
694 specifies a mapping of characters into characters. These tables are
695 used in encoding and decoding, and for other purposes. Some coding
696 systems specify their own particular translation tables; there are
697 also default translation tables which apply to all other coding
700 A translation table has two extra slots. The first is either
701 @code{nil} or a translation table that performs the reverse
702 translation; the second is the maximum number of characters to look up
703 for translating sequences of characters (see the description of
704 @code{make-translation-table-from-alist} below).
706 @defun make-translation-table &rest translations
707 This function returns a translation table based on the argument
708 @var{translations}. Each element of @var{translations} should be a
709 list of elements of the form @code{(@var{from} . @var{to})}; this says
710 to translate the character @var{from} into @var{to}.
712 The arguments and the forms in each argument are processed in order,
713 and if a previous form already translates @var{to} to some other
714 character, say @var{to-alt}, @var{from} is also translated to
718 During decoding, the translation table's translations are applied to
719 the characters that result from ordinary decoding. If a coding system
720 has the property @code{:decode-translation-table}, that specifies the
721 translation table to use, or a list of translation tables to apply in
722 sequence. (This is a property of the coding system, as returned by
723 @code{coding-system-get}, not a property of the symbol that is the
724 coding system's name. @xref{Coding System Basics,, Basic Concepts of
725 Coding Systems}.) Finally, if
726 @code{standard-translation-table-for-decode} is non-@code{nil}, the
727 resulting characters are translated by that table.
729 During encoding, the translation table's translations are applied to
730 the characters in the buffer, and the result of translation is
731 actually encoded. If a coding system has property
732 @code{:encode-translation-table}, that specifies the translation table
733 to use, or a list of translation tables to apply in sequence. In
734 addition, if the variable @code{standard-translation-table-for-encode}
735 is non-@code{nil}, it specifies the translation table to use for
736 translating the result.
738 @defvar standard-translation-table-for-decode
739 This is the default translation table for decoding. If a coding
740 systems specifies its own translation tables, the table that is the
741 value of this variable, if non-@code{nil}, is applied after them.
744 @defvar standard-translation-table-for-encode
745 This is the default translation table for encoding. If a coding
746 systems specifies its own translation tables, the table that is the
747 value of this variable, if non-@code{nil}, is applied after them.
750 @defvar translation-table-for-input
751 Self-inserting characters are translated through this translation
752 table before they are inserted. Search commands also translate their
753 input through this table, so they can compare more reliably with
754 what's in the buffer.
756 This variable automatically becomes buffer-local when set.
759 @defun make-translation-table-from-vector vec
760 This function returns a translation table made from @var{vec} that is
761 an array of 256 elements to map bytes (values 0 through #xFF) to
762 characters. Elements may be @code{nil} for untranslated bytes. The
763 returned table has a translation table for reverse mapping in the
764 first extra slot, and the value @code{1} in the second extra slot.
766 This function provides an easy way to make a private coding system
767 that maps each byte to a specific character. You can specify the
768 returned table and the reverse translation table using the properties
769 @code{:decode-translation-table} and @code{:encode-translation-table}
770 respectively in the @var{props} argument to
771 @code{define-coding-system}.
774 @defun make-translation-table-from-alist alist
775 This function is similar to @code{make-translation-table} but returns
776 a complex translation table rather than a simple one-to-one mapping.
777 Each element of @var{alist} is of the form @code{(@var{from}
778 . @var{to})}, where @var{from} and @var{to} are either characters or
779 vectors specifying a sequence of characters. If @var{from} is a
780 character, that character is translated to @var{to} (i.e.@: to a
781 character or a character sequence). If @var{from} is a vector of
782 characters, that sequence is translated to @var{to}. The returned
783 table has a translation table for reverse mapping in the first extra
784 slot, and the maximum length of all the @var{from} character sequences
785 in the second extra slot.
789 @section Coding Systems
791 @cindex coding system
792 When Emacs reads or writes a file, and when Emacs sends text to a
793 subprocess or receives text from a subprocess, it normally performs
794 character code conversion and end-of-line conversion as specified
795 by a particular @dfn{coding system}.
797 How to define a coding system is an arcane matter, and is not
801 * Coding System Basics:: Basic concepts.
802 * Encoding and I/O:: How file I/O functions handle coding systems.
803 * Lisp and Coding Systems:: Functions to operate on coding system names.
804 * User-Chosen Coding Systems:: Asking the user to choose a coding system.
805 * Default Coding Systems:: Controlling the default choices.
806 * Specifying Coding Systems:: Requesting a particular coding system
807 for a single file operation.
808 * Explicit Encoding:: Encoding or decoding text without doing I/O.
809 * Terminal I/O Encoding:: Use of encoding for terminal I/O.
810 * MS-DOS File Types:: How DOS "text" and "binary" files
811 relate to coding systems.
814 @node Coding System Basics
815 @subsection Basic Concepts of Coding Systems
817 @cindex character code conversion
818 @dfn{Character code conversion} involves conversion between the
819 internal representation of characters used inside Emacs and some other
820 encoding. Emacs supports many different encodings, in that it can
821 convert to and from them. For example, it can convert text to or from
822 encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and
823 several variants of ISO 2022. In some cases, Emacs supports several
824 alternative encodings for the same characters; for example, there are
825 three coding systems for the Cyrillic (Russian) alphabet: ISO,
826 Alternativnyj, and KOI8.
828 Every coding system specifies a particular set of character code
829 conversions, but the coding system @code{undecided} is special: it
830 leaves the choice unspecified, to be chosen heuristically for each
831 file, based on the file's data.
833 In general, a coding system doesn't guarantee roundtrip identity:
834 decoding a byte sequence using coding system, then encoding the
835 resulting text in the same coding system, can produce a different byte
836 sequence. But some coding systems do guarantee that the byte sequence
837 will be the same as what you originally decoded. Here are a few
841 iso-8859-1, utf-8, big5, shift_jis, euc-jp
844 Encoding buffer text and then decoding the result can also fail to
845 reproduce the original text. For instance, if you encode a character
846 with a coding system which does not support that character, the result
847 is unpredictable, and thus decoding it using the same coding system
848 may produce a different text. Currently, Emacs can't report errors
849 that result from encoding unsupported characters.
851 @cindex EOL conversion
852 @cindex end-of-line conversion
853 @cindex line end conversion
854 @dfn{End of line conversion} handles three different conventions
855 used on various systems for representing end of line in files. The
856 Unix convention, used on GNU and Unix systems, is to use the linefeed
857 character (also called newline). The DOS convention, used on
858 MS-Windows and MS-DOS systems, is to use a carriage-return and a
859 linefeed at the end of a line. The Mac convention is to use just
862 @cindex base coding system
863 @cindex variant coding system
864 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
865 conversion unspecified, to be chosen based on the data. @dfn{Variant
866 coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
867 @code{latin-1-mac} specify the end-of-line conversion explicitly as
868 well. Most base coding systems have three corresponding variants whose
869 names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
871 @vindex raw-text@r{ coding system}
872 The coding system @code{raw-text} is special in that it prevents
873 character code conversion, and causes the buffer visited with this
874 coding system to be a unibyte buffer. For historical reasons, you can
875 save both unibyte and multibyte text with this coding system. When
876 you use @code{raw-text} to encode multibyte text, it does perform one
877 character code conversion: it converts eight-bit characters to their
878 single-byte external representation. @code{raw-text} does not specify
879 the end-of-line conversion, allowing that to be determined as usual by
880 the data, and has the usual three variants which specify the
881 end-of-line conversion.
883 @vindex no-conversion@r{ coding system}
884 @vindex binary@r{ coding system}
885 @code{no-conversion} (and its alias @code{binary}) is equivalent to
886 @code{raw-text-unix}: it specifies no conversion of either character
887 codes or end-of-line.
889 @vindex emacs-internal@r{ coding system}
890 @vindex utf-8-emacs@r{ coding system}
891 The coding system @code{utf-8-emacs} specifies that the data is
892 represented in the internal Emacs encoding (@pxref{Text
893 Representations}). This is like @code{raw-text} in that no code
894 conversion happens, but different in that the result is multibyte
895 data. The name @code{emacs-internal} is an alias for
898 @defun coding-system-get coding-system property
899 This function returns the specified property of the coding system
900 @var{coding-system}. Most coding system properties exist for internal
901 purposes, but one that you might find useful is @code{:mime-charset}.
902 That property's value is the name used in MIME for the character coding
903 which this coding system can read and write. Examples:
906 (coding-system-get 'iso-latin-1 :mime-charset)
908 (coding-system-get 'iso-2022-cn :mime-charset)
909 @result{} iso-2022-cn
910 (coding-system-get 'cyrillic-koi8 :mime-charset)
914 The value of the @code{:mime-charset} property is also defined
915 as an alias for the coding system.
918 @defun coding-system-aliases coding-system
919 This function returns the list of aliases of @var{coding-system}.
922 @node Encoding and I/O
923 @subsection Encoding and I/O
925 The principal purpose of coding systems is for use in reading and
926 writing files. The function @code{insert-file-contents} uses a coding
927 system to decode the file data, and @code{write-region} uses one to
928 encode the buffer contents.
930 You can specify the coding system to use either explicitly
931 (@pxref{Specifying Coding Systems}), or implicitly using a default
932 mechanism (@pxref{Default Coding Systems}). But these methods may not
933 completely specify what to do. For example, they may choose a coding
934 system such as @code{undefined} which leaves the character code
935 conversion to be determined from the data. In these cases, the I/O
936 operation finishes the job of choosing a coding system. Very often
937 you will want to find out afterwards which coding system was chosen.
939 @defvar buffer-file-coding-system
940 This buffer-local variable records the coding system used for saving the
941 buffer and for writing part of the buffer with @code{write-region}. If
942 the text to be written cannot be safely encoded using the coding system
943 specified by this variable, these operations select an alternative
944 encoding by calling the function @code{select-safe-coding-system}
945 (@pxref{User-Chosen Coding Systems}). If selecting a different encoding
946 requires to ask the user to specify a coding system,
947 @code{buffer-file-coding-system} is updated to the newly selected coding
950 @code{buffer-file-coding-system} does @emph{not} affect sending text
954 @defvar save-buffer-coding-system
955 This variable specifies the coding system for saving the buffer (by
956 overriding @code{buffer-file-coding-system}). Note that it is not used
957 for @code{write-region}.
959 When a command to save the buffer starts out to use
960 @code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
961 and that coding system cannot handle
962 the actual text in the buffer, the command asks the user to choose
963 another coding system (by calling @code{select-safe-coding-system}).
964 After that happens, the command also updates
965 @code{buffer-file-coding-system} to represent the coding system that
969 @defvar last-coding-system-used
970 I/O operations for files and subprocesses set this variable to the
971 coding system name that was used. The explicit encoding and decoding
972 functions (@pxref{Explicit Encoding}) set it too.
974 @strong{Warning:} Since receiving subprocess output sets this variable,
975 it can change whenever Emacs waits; therefore, you should copy the
976 value shortly after the function call that stores the value you are
980 The variable @code{selection-coding-system} specifies how to encode
981 selections for the window system. @xref{Window System Selections}.
983 @defvar file-name-coding-system
984 The variable @code{file-name-coding-system} specifies the coding
985 system to use for encoding file names. Emacs encodes file names using
986 that coding system for all file operations. If
987 @code{file-name-coding-system} is @code{nil}, Emacs uses a default
988 coding system determined by the selected language environment. In the
989 default language environment, any non-@acronym{ASCII} characters in
990 file names are not encoded specially; they appear in the file system
991 using the internal Emacs representation.
994 @strong{Warning:} if you change @code{file-name-coding-system} (or
995 the language environment) in the middle of an Emacs session, problems
996 can result if you have already visited files whose names were encoded
997 using the earlier coding system and are handled differently under the
998 new coding system. If you try to save one of these buffers under the
999 visited file name, saving may use the wrong file name, or it may get
1000 an error. If such a problem happens, use @kbd{C-x C-w} to specify a
1001 new file name for that buffer.
1003 @node Lisp and Coding Systems
1004 @subsection Coding Systems in Lisp
1006 Here are the Lisp facilities for working with coding systems:
1008 @cindex list all coding systems
1009 @defun coding-system-list &optional base-only
1010 This function returns a list of all coding system names (symbols). If
1011 @var{base-only} is non-@code{nil}, the value includes only the
1012 base coding systems. Otherwise, it includes alias and variant coding
1016 @defun coding-system-p object
1017 This function returns @code{t} if @var{object} is a coding system
1021 @cindex validity of coding system
1022 @cindex coding system, validity check
1023 @defun check-coding-system coding-system
1024 This function checks the validity of @var{coding-system}. If that is
1025 valid, it returns @var{coding-system}. If @var{coding-system} is
1026 @code{nil}, the function return @code{nil}. For any other values, it
1027 signals an error whose @code{error-symbol} is @code{coding-system-error}
1028 (@pxref{Signaling Errors, signal}).
1031 @cindex eol type of coding system
1032 @defun coding-system-eol-type coding-system
1033 This function returns the type of end-of-line (a.k.a.@: @dfn{eol})
1034 conversion used by @var{coding-system}. If @var{coding-system}
1035 specifies a certain eol conversion, the return value is an integer 0,
1036 1, or 2, standing for @code{unix}, @code{dos}, and @code{mac},
1037 respectively. If @var{coding-system} doesn't specify eol conversion
1038 explicitly, the return value is a vector of coding systems, each one
1039 with one of the possible eol conversion types, like this:
1042 (coding-system-eol-type 'latin-1)
1043 @result{} [latin-1-unix latin-1-dos latin-1-mac]
1047 If this function returns a vector, Emacs will decide, as part of the
1048 text encoding or decoding process, what eol conversion to use. For
1049 decoding, the end-of-line format of the text is auto-detected, and the
1050 eol conversion is set to match it (e.g., DOS-style CRLF format will
1051 imply @code{dos} eol conversion). For encoding, the eol conversion is
1052 taken from the appropriate default coding system (e.g.,
1053 default value of @code{buffer-file-coding-system} for
1054 @code{buffer-file-coding-system}), or from the default eol conversion
1055 appropriate for the underlying platform.
1058 @cindex eol conversion of coding system
1059 @defun coding-system-change-eol-conversion coding-system eol-type
1060 This function returns a coding system which is like @var{coding-system}
1061 except for its eol conversion, which is specified by @code{eol-type}.
1062 @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
1063 @code{nil}. If it is @code{nil}, the returned coding system determines
1064 the end-of-line conversion from the data.
1066 @var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
1067 @code{dos} and @code{mac}, respectively.
1070 @cindex text conversion of coding system
1071 @defun coding-system-change-text-conversion eol-coding text-coding
1072 This function returns a coding system which uses the end-of-line
1073 conversion of @var{eol-coding}, and the text conversion of
1074 @var{text-coding}. If @var{text-coding} is @code{nil}, it returns
1075 @code{undecided}, or one of its variants according to @var{eol-coding}.
1078 @cindex safely encode region
1079 @cindex coding systems for encoding region
1080 @defun find-coding-systems-region from to
1081 This function returns a list of coding systems that could be used to
1082 encode a text between @var{from} and @var{to}. All coding systems in
1083 the list can safely encode any multibyte characters in that portion of
1086 If the text contains no multibyte characters, the function returns the
1087 list @code{(undecided)}.
1090 @cindex safely encode a string
1091 @cindex coding systems for encoding a string
1092 @defun find-coding-systems-string string
1093 This function returns a list of coding systems that could be used to
1094 encode the text of @var{string}. All coding systems in the list can
1095 safely encode any multibyte characters in @var{string}. If the text
1096 contains no multibyte characters, this returns the list
1100 @cindex charset, coding systems to encode
1101 @cindex safely encode characters in a charset
1102 @defun find-coding-systems-for-charsets charsets
1103 This function returns a list of coding systems that could be used to
1104 encode all the character sets in the list @var{charsets}.
1107 @defun check-coding-systems-region start end coding-system-list
1108 This function checks whether coding systems in the list
1109 @code{coding-system-list} can encode all the characters in the region
1110 between @var{start} and @var{end}. If all of the coding systems in
1111 the list can encode the specified text, the function returns
1112 @code{nil}. If some coding systems cannot encode some of the
1113 characters, the value is an alist, each element of which has the form
1114 @code{(@var{coding-system1} @var{pos1} @var{pos2} @dots{})}, meaning
1115 that @var{coding-system1} cannot encode characters at buffer positions
1116 @var{pos1}, @var{pos2}, @enddots{}.
1118 @var{start} may be a string, in which case @var{end} is ignored and
1119 the returned value references string indices instead of buffer
1123 @defun detect-coding-region start end &optional highest
1124 This function chooses a plausible coding system for decoding the text
1125 from @var{start} to @var{end}. This text should be a byte sequence,
1126 i.e.@: unibyte text or multibyte text with only @acronym{ASCII} and
1127 eight-bit characters (@pxref{Explicit Encoding}).
1129 Normally this function returns a list of coding systems that could
1130 handle decoding the text that was scanned. They are listed in order of
1131 decreasing priority. But if @var{highest} is non-@code{nil}, then the
1132 return value is just one coding system, the one that is highest in
1135 If the region contains only @acronym{ASCII} characters except for such
1136 ISO-2022 control characters ISO-2022 as @code{ESC}, the value is
1137 @code{undecided} or @code{(undecided)}, or a variant specifying
1138 end-of-line conversion, if that can be deduced from the text.
1140 If the region contains null bytes, the value is @code{no-conversion},
1141 even if the region contains text encoded in some coding system.
1144 @defun detect-coding-string string &optional highest
1145 This function is like @code{detect-coding-region} except that it
1146 operates on the contents of @var{string} instead of bytes in the buffer.
1149 @cindex null bytes, and decoding text
1150 @defvar inhibit-null-byte-detection
1151 If this variable has a non-@code{nil} value, null bytes are ignored
1152 when detecting the encoding of a region or a string. This allows to
1153 correctly detect the encoding of text that contains null bytes, such
1154 as Info files with Index nodes.
1157 @defvar inhibit-iso-escape-detection
1158 If this variable has a non-@code{nil} value, ISO-2022 escape sequences
1159 are ignored when detecting the encoding of a region or a string. The
1160 result is that no text is ever detected as encoded in some ISO-2022
1161 encoding, and all escape sequences become visible in a buffer.
1162 @strong{Warning:} @emph{Use this variable with extreme caution,
1163 because many files in the Emacs distribution use ISO-2022 encoding.}
1166 @cindex charsets supported by a coding system
1167 @defun coding-system-charset-list coding-system
1168 This function returns the list of character sets (@pxref{Character
1169 Sets}) supported by @var{coding-system}. Some coding systems that
1170 support too many character sets to list them all yield special values:
1173 If @var{coding-system} supports all the ISO-2022 charsets, the value
1176 If @var{coding-system} supports all Emacs characters, the value is
1179 If @var{coding-system} supports all emacs-mule characters, the value
1180 is @code{emacs-mule}.
1182 If @var{coding-system} supports all Unicode characters, the value is
1187 @xref{Coding systems for a subprocess,, Process Information}, in
1188 particular the description of the functions
1189 @code{process-coding-system} and @code{set-process-coding-system}, for
1190 how to examine or set the coding systems used for I/O to a subprocess.
1192 @node User-Chosen Coding Systems
1193 @subsection User-Chosen Coding Systems
1195 @cindex select safe coding system
1196 @defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
1197 This function selects a coding system for encoding specified text,
1198 asking the user to choose if necessary. Normally the specified text
1199 is the text in the current buffer between @var{from} and @var{to}. If
1200 @var{from} is a string, the string specifies the text to encode, and
1201 @var{to} is ignored.
1203 If the specified text includes raw bytes (@pxref{Text
1204 Representations}), @code{select-safe-coding-system} suggests
1205 @code{raw-text} for its encoding.
1207 If @var{default-coding-system} is non-@code{nil}, that is the first
1208 coding system to try; if that can handle the text,
1209 @code{select-safe-coding-system} returns that coding system. It can
1210 also be a list of coding systems; then the function tries each of them
1211 one by one. After trying all of them, it next tries the current
1212 buffer's value of @code{buffer-file-coding-system} (if it is not
1213 @code{undecided}), then the default value of
1214 @code{buffer-file-coding-system} and finally the user's most
1215 preferred coding system, which the user can set using the command
1216 @code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
1217 Coding Systems, emacs, The GNU Emacs Manual}).
1219 If one of those coding systems can safely encode all the specified
1220 text, @code{select-safe-coding-system} chooses it and returns it.
1221 Otherwise, it asks the user to choose from a list of coding systems
1222 which can encode all the text, and returns the user's choice.
1224 @var{default-coding-system} can also be a list whose first element is
1225 t and whose other elements are coding systems. Then, if no coding
1226 system in the list can handle the text, @code{select-safe-coding-system}
1227 queries the user immediately, without trying any of the three
1228 alternatives described above.
1230 The optional argument @var{accept-default-p}, if non-@code{nil},
1231 should be a function to determine whether a coding system selected
1232 without user interaction is acceptable. @code{select-safe-coding-system}
1233 calls this function with one argument, the base coding system of the
1234 selected coding system. If @var{accept-default-p} returns @code{nil},
1235 @code{select-safe-coding-system} rejects the silently selected coding
1236 system, and asks the user to select a coding system from a list of
1237 possible candidates.
1239 @vindex select-safe-coding-system-accept-default-p
1240 If the variable @code{select-safe-coding-system-accept-default-p} is
1241 non-@code{nil}, it should be a function taking a single argument.
1242 It is used in place of @var{accept-default-p}, overriding any
1243 value supplied for this argument.
1245 As a final step, before returning the chosen coding system,
1246 @code{select-safe-coding-system} checks whether that coding system is
1247 consistent with what would be selected if the contents of the region
1248 were read from a file. (If not, this could lead to data corruption in
1249 a file subsequently re-visited and edited.) Normally,
1250 @code{select-safe-coding-system} uses @code{buffer-file-name} as the
1251 file for this purpose, but if @var{file} is non-@code{nil}, it uses
1252 that file instead (this can be relevant for @code{write-region} and
1253 similar functions). If it detects an apparent inconsistency,
1254 @code{select-safe-coding-system} queries the user before selecting the
1258 Here are two functions you can use to let the user specify a coding
1259 system, with completion. @xref{Completion}.
1261 @defun read-coding-system prompt &optional default
1262 This function reads a coding system using the minibuffer, prompting with
1263 string @var{prompt}, and returns the coding system name as a symbol. If
1264 the user enters null input, @var{default} specifies which coding system
1265 to return. It should be a symbol or a string.
1268 @defun read-non-nil-coding-system prompt
1269 This function reads a coding system using the minibuffer, prompting with
1270 string @var{prompt}, and returns the coding system name as a symbol. If
1271 the user tries to enter null input, it asks the user to try again.
1272 @xref{Coding Systems}.
1275 @node Default Coding Systems
1276 @subsection Default Coding Systems
1277 @cindex default coding system
1278 @cindex coding system, automatically determined
1280 This section describes variables that specify the default coding
1281 system for certain files or when running certain subprograms, and the
1282 function that I/O operations use to access them.
1284 The idea of these variables is that you set them once and for all to the
1285 defaults you want, and then do not change them again. To specify a
1286 particular coding system for a particular operation in a Lisp program,
1287 don't change these variables; instead, override them using
1288 @code{coding-system-for-read} and @code{coding-system-for-write}
1289 (@pxref{Specifying Coding Systems}).
1291 @cindex file contents, and default coding system
1292 @defopt auto-coding-regexp-alist
1293 This variable is an alist of text patterns and corresponding coding
1294 systems. Each element has the form @code{(@var{regexp}
1295 . @var{coding-system})}; a file whose first few kilobytes match
1296 @var{regexp} is decoded with @var{coding-system} when its contents are
1297 read into a buffer. The settings in this alist take priority over
1298 @code{coding:} tags in the files and the contents of
1299 @code{file-coding-system-alist} (see below). The default value is set
1300 so that Emacs automatically recognizes mail files in Babyl format and
1301 reads them with no code conversions.
1304 @cindex file name, and default coding system
1305 @defopt file-coding-system-alist
1306 This variable is an alist that specifies the coding systems to use for
1307 reading and writing particular files. Each element has the form
1308 @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
1309 expression that matches certain file names. The element applies to file
1310 names that match @var{pattern}.
1312 The @sc{cdr} of the element, @var{coding}, should be either a coding
1313 system, a cons cell containing two coding systems, or a function name (a
1314 symbol with a function definition). If @var{coding} is a coding system,
1315 that coding system is used for both reading the file and writing it. If
1316 @var{coding} is a cons cell containing two coding systems, its @sc{car}
1317 specifies the coding system for decoding, and its @sc{cdr} specifies the
1318 coding system for encoding.
1320 If @var{coding} is a function name, the function should take one
1321 argument, a list of all arguments passed to
1322 @code{find-operation-coding-system}. It must return a coding system
1323 or a cons cell containing two coding systems. This value has the same
1324 meaning as described above.
1326 If @var{coding} (or what returned by the above function) is
1327 @code{undecided}, the normal code-detection is performed.
1330 @defopt auto-coding-alist
1331 This variable is an alist that specifies the coding systems to use for
1332 reading and writing particular files. Its form is like that of
1333 @code{file-coding-system-alist}, but, unlike the latter, this variable
1334 takes priority over any @code{coding:} tags in the file.
1337 @cindex program name, and default coding system
1338 @defvar process-coding-system-alist
1339 This variable is an alist specifying which coding systems to use for a
1340 subprocess, depending on which program is running in the subprocess. It
1341 works like @code{file-coding-system-alist}, except that @var{pattern} is
1342 matched against the program name used to start the subprocess. The coding
1343 system or systems specified in this alist are used to initialize the
1344 coding systems used for I/O to the subprocess, but you can specify
1345 other coding systems later using @code{set-process-coding-system}.
1348 @strong{Warning:} Coding systems such as @code{undecided}, which
1349 determine the coding system from the data, do not work entirely reliably
1350 with asynchronous subprocess output. This is because Emacs handles
1351 asynchronous subprocess output in batches, as it arrives. If the coding
1352 system leaves the character code conversion unspecified, or leaves the
1353 end-of-line conversion unspecified, Emacs must try to detect the proper
1354 conversion from one batch at a time, and this does not always work.
1356 Therefore, with an asynchronous subprocess, if at all possible, use a
1357 coding system which determines both the character code conversion and
1358 the end of line conversion---that is, one like @code{latin-1-unix},
1359 rather than @code{undecided} or @code{latin-1}.
1361 @cindex port number, and default coding system
1362 @cindex network service name, and default coding system
1363 @defvar network-coding-system-alist
1364 This variable is an alist that specifies the coding system to use for
1365 network streams. It works much like @code{file-coding-system-alist},
1366 with the difference that the @var{pattern} in an element may be either a
1367 port number or a regular expression. If it is a regular expression, it
1368 is matched against the network service name used to open the network
1372 @defvar default-process-coding-system
1373 This variable specifies the coding systems to use for subprocess (and
1374 network stream) input and output, when nothing else specifies what to
1377 The value should be a cons cell of the form @code{(@var{input-coding}
1378 . @var{output-coding})}. Here @var{input-coding} applies to input from
1379 the subprocess, and @var{output-coding} applies to output to it.
1382 @cindex default coding system, functions to determine
1383 @defopt auto-coding-functions
1384 This variable holds a list of functions that try to determine a
1385 coding system for a file based on its undecoded contents.
1387 Each function in this list should be written to look at text in the
1388 current buffer, but should not modify it in any way. The buffer will
1389 contain undecoded text of parts of the file. Each function should
1390 take one argument, @var{size}, which tells it how many characters to
1391 look at, starting from point. If the function succeeds in determining
1392 a coding system for the file, it should return that coding system.
1393 Otherwise, it should return @code{nil}.
1395 If a file has a @samp{coding:} tag, that takes precedence, so these
1396 functions won't be called.
1399 @defun find-auto-coding filename size
1400 This function tries to determine a suitable coding system for
1401 @var{filename}. It examines the buffer visiting the named file, using
1402 the variables documented above in sequence, until it finds a match for
1403 one of the rules specified by these variables. It then returns a cons
1404 cell of the form @code{(@var{coding} . @var{source})}, where
1405 @var{coding} is the coding system to use and @var{source} is a symbol,
1406 one of @code{auto-coding-alist}, @code{auto-coding-regexp-alist},
1407 @code{:coding}, or @code{auto-coding-functions}, indicating which one
1408 supplied the matching rule. The value @code{:coding} means the coding
1409 system was specified by the @code{coding:} tag in the file
1410 (@pxref{Specify Coding,, coding tag, emacs, The GNU Emacs Manual}).
1411 The order of looking for a matching rule is @code{auto-coding-alist}
1412 first, then @code{auto-coding-regexp-alist}, then the @code{coding:}
1413 tag, and lastly @code{auto-coding-functions}. If no matching rule was
1414 found, the function returns @code{nil}.
1416 The second argument @var{size} is the size of text, in characters,
1417 following point. The function examines text only within @var{size}
1418 characters after point. Normally, the buffer should be positioned at
1419 the beginning when this function is called, because one of the places
1420 for the @code{coding:} tag is the first one or two lines of the file;
1421 in that case, @var{size} should be the size of the buffer.
1424 @defun set-auto-coding filename size
1425 This function returns a suitable coding system for file
1426 @var{filename}. It uses @code{find-auto-coding} to find the coding
1427 system. If no coding system could be determined, the function returns
1428 @code{nil}. The meaning of the argument @var{size} is like in
1429 @code{find-auto-coding}.
1432 @defun find-operation-coding-system operation &rest arguments
1433 This function returns the coding system to use (by default) for
1434 performing @var{operation} with @var{arguments}. The value has this
1438 (@var{decoding-system} . @var{encoding-system})
1441 The first element, @var{decoding-system}, is the coding system to use
1442 for decoding (in case @var{operation} does decoding), and
1443 @var{encoding-system} is the coding system for encoding (in case
1444 @var{operation} does encoding).
1446 The argument @var{operation} is a symbol, one of @code{write-region},
1447 @code{start-process}, @code{call-process}, @code{call-process-region},
1448 @code{insert-file-contents}, or @code{open-network-stream}. These are
1449 the names of the Emacs I/O primitives that can do character code and
1452 The remaining arguments should be the same arguments that might be given
1453 to the corresponding I/O primitive. Depending on the primitive, one
1454 of those arguments is selected as the @dfn{target}. For example, if
1455 @var{operation} does file I/O, whichever argument specifies the file
1456 name is the target. For subprocess primitives, the process name is the
1457 target. For @code{open-network-stream}, the target is the service name
1460 Depending on @var{operation}, this function looks up the target in
1461 @code{file-coding-system-alist}, @code{process-coding-system-alist},
1462 or @code{network-coding-system-alist}. If the target is found in the
1463 alist, @code{find-operation-coding-system} returns its association in
1464 the alist; otherwise it returns @code{nil}.
1466 If @var{operation} is @code{insert-file-contents}, the argument
1467 corresponding to the target may be a cons cell of the form
1468 @code{(@var{filename} . @var{buffer})}). In that case, @var{filename}
1469 is a file name to look up in @code{file-coding-system-alist}, and
1470 @var{buffer} is a buffer that contains the file's contents (not yet
1471 decoded). If @code{file-coding-system-alist} specifies a function to
1472 call for this file, and that function needs to examine the file's
1473 contents (as it usually does), it should examine the contents of
1474 @var{buffer} instead of reading the file.
1477 @node Specifying Coding Systems
1478 @subsection Specifying a Coding System for One Operation
1480 You can specify the coding system for a specific operation by binding
1481 the variables @code{coding-system-for-read} and/or
1482 @code{coding-system-for-write}.
1484 @defvar coding-system-for-read
1485 If this variable is non-@code{nil}, it specifies the coding system to
1486 use for reading a file, or for input from a synchronous subprocess.
1488 It also applies to any asynchronous subprocess or network stream, but in
1489 a different way: the value of @code{coding-system-for-read} when you
1490 start the subprocess or open the network stream specifies the input
1491 decoding method for that subprocess or network stream. It remains in
1492 use for that subprocess or network stream unless and until overridden.
1494 The right way to use this variable is to bind it with @code{let} for a
1495 specific I/O operation. Its global value is normally @code{nil}, and
1496 you should not globally set it to any other value. Here is an example
1497 of the right way to use the variable:
1500 ;; @r{Read the file with no character code conversion.}
1501 ;; @r{Assume @acronym{crlf} represents end-of-line.}
1502 (let ((coding-system-for-read 'emacs-mule-dos))
1503 (insert-file-contents filename))
1506 When its value is non-@code{nil}, this variable takes precedence over
1507 all other methods of specifying a coding system to use for input,
1508 including @code{file-coding-system-alist},
1509 @code{process-coding-system-alist} and
1510 @code{network-coding-system-alist}.
1513 @defvar coding-system-for-write
1514 This works much like @code{coding-system-for-read}, except that it
1515 applies to output rather than input. It affects writing to files,
1516 as well as sending output to subprocesses and net connections.
1518 When a single operation does both input and output, as do
1519 @code{call-process-region} and @code{start-process}, both
1520 @code{coding-system-for-read} and @code{coding-system-for-write}
1524 @defopt inhibit-eol-conversion
1525 When this variable is non-@code{nil}, no end-of-line conversion is done,
1526 no matter which coding system is specified. This applies to all the
1527 Emacs I/O and subprocess primitives, and to the explicit encoding and
1528 decoding functions (@pxref{Explicit Encoding}).
1531 @cindex priority order of coding systems
1532 @cindex coding systems, priority
1533 Sometimes, you need to prefer several coding systems for some
1534 operation, rather than fix a single one. Emacs lets you specify a
1535 priority order for using coding systems. This ordering affects the
1536 sorting of lists of coding sysems returned by functions such as
1537 @code{find-coding-systems-region} (@pxref{Lisp and Coding Systems}).
1539 @defun coding-system-priority-list &optional highestp
1540 This function returns the list of coding systems in the order of their
1541 current priorities. Optional argument @var{highestp}, if
1542 non-@code{nil}, means return only the highest priority coding system.
1545 @defun set-coding-system-priority &rest coding-systems
1546 This function puts @var{coding-systems} at the beginning of the
1547 priority list for coding systems, thus making their priority higher
1551 @defmac with-coding-priority coding-systems &rest body@dots{}
1552 This macro execute @var{body}, like @code{progn} does
1553 (@pxref{Sequencing, progn}), with @var{coding-systems} at the front of
1554 the priority list for coding systems. @var{coding-systems} should be
1555 a list of coding systems to prefer during execution of @var{body}.
1558 @node Explicit Encoding
1559 @subsection Explicit Encoding and Decoding
1560 @cindex encoding in coding systems
1561 @cindex decoding in coding systems
1563 All the operations that transfer text in and out of Emacs have the
1564 ability to use a coding system to encode or decode the text.
1565 You can also explicitly encode and decode text using the functions
1568 The result of encoding, and the input to decoding, are not ordinary
1569 text. They logically consist of a series of byte values; that is, a
1570 series of @acronym{ASCII} and eight-bit characters. In unibyte
1571 buffers and strings, these characters have codes in the range 0
1572 through #xFF (255). In a multibyte buffer or string, eight-bit
1573 characters have character codes higher than #xFF (@pxref{Text
1574 Representations}), but Emacs transparently converts them to their
1575 single-byte values when you encode or decode such text.
1577 The usual way to read a file into a buffer as a sequence of bytes, so
1578 you can decode the contents explicitly, is with
1579 @code{insert-file-contents-literally} (@pxref{Reading from Files});
1580 alternatively, specify a non-@code{nil} @var{rawfile} argument when
1581 visiting a file with @code{find-file-noselect}. These methods result in
1584 The usual way to use the byte sequence that results from explicitly
1585 encoding text is to copy it to a file or process---for example, to write
1586 it with @code{write-region} (@pxref{Writing to Files}), and suppress
1587 encoding by binding @code{coding-system-for-write} to
1588 @code{no-conversion}.
1590 Here are the functions to perform explicit encoding or decoding. The
1591 encoding functions produce sequences of bytes; the decoding functions
1592 are meant to operate on sequences of bytes. All of these functions
1593 discard text properties. They also set @code{last-coding-system-used}
1594 to the precise coding system they used.
1596 @deffn Command encode-coding-region start end coding-system &optional destination
1597 This command encodes the text from @var{start} to @var{end} according
1598 to coding system @var{coding-system}. Normally, the encoded text
1599 replaces the original text in the buffer, but the optional argument
1600 @var{destination} can change that. If @var{destination} is a buffer,
1601 the encoded text is inserted in that buffer after point (point does
1602 not move); if it is @code{t}, the command returns the encoded text as
1603 a unibyte string without inserting it.
1605 If encoded text is inserted in some buffer, this command returns the
1606 length of the encoded text.
1608 The result of encoding is logically a sequence of bytes, but the
1609 buffer remains multibyte if it was multibyte before, and any 8-bit
1610 bytes are converted to their multibyte representation (@pxref{Text
1613 @cindex @code{undecided} coding-system, when encoding
1614 Do @emph{not} use @code{undecided} for @var{coding-system} when
1615 encoding text, since that may lead to unexpected results. Instead,
1616 use @code{select-safe-coding-system} (@pxref{User-Chosen Coding
1617 Systems, select-safe-coding-system}) to suggest a suitable encoding,
1618 if there's no obvious pertinent value for @var{coding-system}.
1621 @defun encode-coding-string string coding-system &optional nocopy buffer
1622 This function encodes the text in @var{string} according to coding
1623 system @var{coding-system}. It returns a new string containing the
1624 encoded text, except when @var{nocopy} is non-@code{nil}, in which
1625 case the function may return @var{string} itself if the encoding
1626 operation is trivial. The result of encoding is a unibyte string.
1629 @deffn Command decode-coding-region start end coding-system &optional destination
1630 This command decodes the text from @var{start} to @var{end} according
1631 to coding system @var{coding-system}. To make explicit decoding
1632 useful, the text before decoding ought to be a sequence of byte
1633 values, but both multibyte and unibyte buffers are acceptable (in the
1634 multibyte case, the raw byte values should be represented as eight-bit
1635 characters). Normally, the decoded text replaces the original text in
1636 the buffer, but the optional argument @var{destination} can change
1637 that. If @var{destination} is a buffer, the decoded text is inserted
1638 in that buffer after point (point does not move); if it is @code{t},
1639 the command returns the decoded text as a multibyte string without
1642 If decoded text is inserted in some buffer, this command returns the
1643 length of the decoded text.
1645 This command puts a @code{charset} text property on the decoded text.
1646 The value of the property states the character set used to decode the
1650 @defun decode-coding-string string coding-system &optional nocopy buffer
1651 This function decodes the text in @var{string} according to
1652 @var{coding-system}. It returns a new string containing the decoded
1653 text, except when @var{nocopy} is non-@code{nil}, in which case the
1654 function may return @var{string} itself if the decoding operation is
1655 trivial. To make explicit decoding useful, the contents of
1656 @var{string} ought to be a unibyte string with a sequence of byte
1657 values, but a multibyte string is also acceptable (assuming it
1658 contains 8-bit bytes in their multibyte form).
1660 If optional argument @var{buffer} specifies a buffer, the decoded text
1661 is inserted in that buffer after point (point does not move). In this
1662 case, the return value is the length of the decoded text.
1664 @cindex @code{charset}, text property
1665 This function puts a @code{charset} text property on the decoded text.
1666 The value of the property states the character set used to decode the
1671 (decode-coding-string "Gr\374ss Gott" 'latin-1)
1672 @result{} #("Gr@"uss Gott" 0 9 (charset iso-8859-1))
1677 @defun decode-coding-inserted-region from to filename &optional visit beg end replace
1678 This function decodes the text from @var{from} to @var{to} as if
1679 it were being read from file @var{filename} using @code{insert-file-contents}
1680 using the rest of the arguments provided.
1682 The normal way to use this function is after reading text from a file
1683 without decoding, if you decide you would rather have decoded it.
1684 Instead of deleting the text and reading it again, this time with
1685 decoding, you can call this function.
1688 @node Terminal I/O Encoding
1689 @subsection Terminal I/O Encoding
1691 Emacs can decode keyboard input using a coding system, and encode
1692 terminal output. This is useful for terminals that transmit or
1693 display text using a particular encoding such as Latin-1. Emacs does
1694 not set @code{last-coding-system-used} for encoding or decoding of
1697 @defun keyboard-coding-system &optional terminal
1698 This function returns the coding system that is in use for decoding
1699 keyboard input from @var{terminal}---or @code{nil} if no coding system
1700 is to be used for that terminal. If @var{terminal} is omitted or
1701 @code{nil}, it means the selected frame's terminal. @xref{Multiple
1705 @deffn Command set-keyboard-coding-system coding-system &optional terminal
1706 This command specifies @var{coding-system} as the coding system to use
1707 for decoding keyboard input from @var{terminal}. If
1708 @var{coding-system} is @code{nil}, that means do not decode keyboard
1709 input. If @var{terminal} is a frame, it means that frame's terminal;
1710 if it is @code{nil}, that means the currently selected frame's
1711 terminal. @xref{Multiple Terminals}.
1714 @defun terminal-coding-system &optional terminal
1715 This function returns the coding system that is in use for encoding
1716 terminal output from @var{terminal}---or @code{nil} if the output is
1717 not encoded. If @var{terminal} is a frame, it means that frame's
1718 terminal; if it is @code{nil}, that means the currently selected
1722 @deffn Command set-terminal-coding-system coding-system &optional terminal
1723 This command specifies @var{coding-system} as the coding system to use
1724 for encoding terminal output from @var{terminal}. If
1725 @var{coding-system} is @code{nil}, terminal output is not encoded. If
1726 @var{terminal} is a frame, it means that frame's terminal; if it is
1727 @code{nil}, that means the currently selected frame's terminal.
1730 @node MS-DOS File Types
1731 @subsection MS-DOS File Types
1732 @cindex DOS file types
1733 @cindex MS-DOS file types
1734 @cindex Windows file types
1735 @cindex file types on MS-DOS and Windows
1736 @cindex text files and binary files
1737 @cindex binary files and text files
1739 On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
1740 end-of-line conversion for a file by looking at the file's name. This
1741 feature classifies files as @dfn{text files} and @dfn{binary files}. By
1742 ``binary file'' we mean a file of literal byte values that are not
1743 necessarily meant to be characters; Emacs does no end-of-line conversion
1744 and no character code conversion for them. On the other hand, the bytes
1745 in a text file are intended to represent characters; when you create a
1746 new file whose name implies that it is a text file, Emacs uses DOS
1747 end-of-line conversion.
1749 @defvar buffer-file-type
1750 This variable, automatically buffer-local in each buffer, records the
1751 file type of the buffer's visited file. When a buffer does not specify
1752 a coding system with @code{buffer-file-coding-system}, this variable is
1753 used to determine which coding system to use when writing the contents
1754 of the buffer. It should be @code{nil} for text, @code{t} for binary.
1755 If it is @code{t}, the coding system is @code{no-conversion}.
1756 Otherwise, @code{undecided-dos} is used.
1758 Normally this variable is set by visiting a file; it is set to
1759 @code{nil} if the file was visited without any actual conversion.
1761 Its default value is used to decide how to handle files for which
1762 @code{file-name-buffer-file-type-alist} says nothing about the type:
1763 If the default value is non-@code{nil}, then these files are treated as
1764 binary: the coding system @code{no-conversion} is used. Otherwise,
1765 nothing special is done for them---the coding system is deduced solely
1766 from the file contents, in the usual Emacs fashion.
1769 @defopt file-name-buffer-file-type-alist
1770 This variable holds an alist for recognizing text and binary files.
1771 Each element has the form (@var{regexp} . @var{type}), where
1772 @var{regexp} is matched against the file name, and @var{type} may be
1773 @code{nil} for text, @code{t} for binary, or a function to call to
1774 compute which. If it is a function, then it is called with a single
1775 argument (the file name) and should return @code{t} or @code{nil}.
1777 When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
1778 which coding system to use when reading a file. For a text file,
1779 @code{undecided-dos} is used. For a binary file, @code{no-conversion}
1782 If no element in this alist matches a given file name, then
1783 the default value of @code{buffer-file-type} says how to treat the file.
1787 @section Input Methods
1788 @cindex input methods
1790 @dfn{Input methods} provide convenient ways of entering non-@acronym{ASCII}
1791 characters from the keyboard. Unlike coding systems, which translate
1792 non-@acronym{ASCII} characters to and from encodings meant to be read by
1793 programs, input methods provide human-friendly commands. (@xref{Input
1794 Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1795 use input methods to enter text.) How to define input methods is not
1796 yet documented in this manual, but here we describe how to use them.
1798 Each input method has a name, which is currently a string;
1799 in the future, symbols may also be usable as input method names.
1801 @defvar current-input-method
1802 This variable holds the name of the input method now active in the
1803 current buffer. (It automatically becomes local in each buffer when set
1804 in any fashion.) It is @code{nil} if no input method is active in the
1808 @defopt default-input-method
1809 This variable holds the default input method for commands that choose an
1810 input method. Unlike @code{current-input-method}, this variable is
1814 @deffn Command set-input-method input-method
1815 This command activates input method @var{input-method} for the current
1816 buffer. It also sets @code{default-input-method} to @var{input-method}.
1817 If @var{input-method} is @code{nil}, this command deactivates any input
1818 method for the current buffer.
1821 @defun read-input-method-name prompt &optional default inhibit-null
1822 This function reads an input method name with the minibuffer, prompting
1823 with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1824 by default, if the user enters empty input. However, if
1825 @var{inhibit-null} is non-@code{nil}, empty input signals an error.
1827 The returned value is a string.
1830 @defvar input-method-alist
1831 This variable defines all the supported input methods.
1832 Each element defines one input method, and should have the form:
1835 (@var{input-method} @var{language-env} @var{activate-func}
1836 @var{title} @var{description} @var{args}...)
1839 Here @var{input-method} is the input method name, a string;
1840 @var{language-env} is another string, the name of the language
1841 environment this input method is recommended for. (That serves only for
1842 documentation purposes.)
1844 @var{activate-func} is a function to call to activate this method. The
1845 @var{args}, if any, are passed as arguments to @var{activate-func}. All
1846 told, the arguments to @var{activate-func} are @var{input-method} and
1849 @var{title} is a string to display in the mode line while this method is
1850 active. @var{description} is a string describing this method and what
1854 The fundamental interface to input methods is through the
1855 variable @code{input-method-function}. @xref{Reading One Event},
1856 and @ref{Invoking the Input Method}.
1862 POSIX defines a concept of ``locales'' which control which language
1863 to use in language-related features. These Emacs variables control
1864 how Emacs interacts with these features.
1866 @defvar locale-coding-system
1867 @cindex keyboard input decoding on X
1868 This variable specifies the coding system to use for decoding system
1869 error messages and---on X Window system only---keyboard input, for
1870 encoding the format argument to @code{format-time-string}, and for
1871 decoding the return value of @code{format-time-string}.
1874 @defvar system-messages-locale
1875 This variable specifies the locale to use for generating system error
1876 messages. Changing the locale can cause messages to come out in a
1877 different language or in a different orthography. If the variable is
1878 @code{nil}, the locale is specified by environment variables in the
1879 usual POSIX fashion.
1882 @defvar system-time-locale
1883 This variable specifies the locale to use for formatting time values.
1884 Changing the locale can cause messages to appear according to the
1885 conventions of a different language. If the variable is @code{nil}, the
1886 locale is specified by environment variables in the usual POSIX fashion.
1889 @defun locale-info item
1890 This function returns locale data @var{item} for the current POSIX
1891 locale, if available. @var{item} should be one of these symbols:
1895 Return the character set as a string (locale item @code{CODESET}).
1898 Return a 7-element vector of day names (locale items
1899 @code{DAY_1} through @code{DAY_7});
1902 Return a 12-element vector of month names (locale items @code{MON_1}
1903 through @code{MON_12}).
1906 Return a list @code{(@var{width} @var{height})} for the default paper
1907 size measured in millimeters (locale items @code{PAPER_WIDTH} and
1908 @code{PAPER_HEIGHT}).
1911 If the system can't provide the requested information, or if
1912 @var{item} is not one of those symbols, the value is @code{nil}. All
1913 strings in the return value are decoded using
1914 @code{locale-coding-system}. @xref{Locales,,, libc, The GNU Libc Manual},
1915 for more information about locales and locale items.
1919 arch-tag: be705bf8-941b-4c35-84fc-ad7d20ddb7cb