2 @c This is part of the GNU Emacs Lisp Reference Manual.
3 @c Copyright (C) 1998, 1999 Free Software Foundation, Inc.
4 @c See the file elisp.texi for copying conditions.
5 @setfilename ../info/characters
6 @node Non-ASCII Characters, Searching and Matching, Text, Top
7 @chapter Non-@sc{ascii} Characters
8 @cindex multibyte characters
9 @cindex non-@sc{ascii} characters
11 This chapter covers the special issues relating to non-@sc{ascii}
12 characters and how they are stored in strings and buffers.
15 * Text Representations:: Unibyte and multibyte representations
16 * Converting Representations:: Converting unibyte to multibyte and vice versa.
17 * Selecting a Representation:: Treating a byte sequence as unibyte or multi.
18 * Character Codes:: How unibyte and multibyte relate to
19 codes of individual characters.
20 * Character Sets:: The space of possible characters codes
21 is divided into various character sets.
22 * Chars and Bytes:: More information about multibyte encodings.
23 * Splitting Characters:: Converting a character to its byte sequence.
24 * Scanning Charsets:: Which character sets are used in a buffer?
25 * Translation of Characters:: Translation tables are used for conversion.
26 * Coding Systems:: Coding systems are conversions for saving files.
27 * Input Methods:: Input methods allow users to enter various
28 non-ASCII characters without speciak keyboards.
29 * Locales:: Interacting with the POSIX locale.
32 @node Text Representations
33 @section Text Representations
34 @cindex text representations
36 Emacs has two @dfn{text representations}---two ways to represent text
37 in a string or buffer. These are called @dfn{unibyte} and
38 @dfn{multibyte}. Each string, and each buffer, uses one of these two
39 representations. For most purposes, you can ignore the issue of
40 representations, because Emacs converts text between them as
41 appropriate. Occasionally in Lisp programming you will need to pay
42 attention to the difference.
45 In unibyte representation, each character occupies one byte and
46 therefore the possible character codes range from 0 to 255. Codes 0
47 through 127 are @sc{ascii} characters; the codes from 128 through 255
48 are used for one non-@sc{ascii} character set (you can choose which
49 character set by setting the variable @code{nonascii-insert-offset}).
52 @cindex multibyte text
53 @cindex trailing codes
54 In multibyte representation, a character may occupy more than one
55 byte, and as a result, the full range of Emacs character codes can be
56 stored. The first byte of a multibyte character is always in the range
57 128 through 159 (octal 0200 through 0237). These values are called
58 @dfn{leading codes}. The second and subsequent bytes of a multibyte
59 character are always in the range 160 through 255 (octal 0240 through
60 0377); these values are @dfn{trailing codes}.
62 Some sequences of bytes are not valid in multibyte text: for example,
63 a single isolated byte in the range 128 through 159 is not allowed. But
64 character codes 128 through 159 can appear in multibyte text,
65 represented as two-byte sequences. All the character codes 128 through
66 255 are possible (though slightly abnormal) in multibyte text; they
67 appear in multibyte buffers and strings when you do explicit encoding
68 and decoding (@pxref{Explicit Encoding}).
70 In a buffer, the buffer-local value of the variable
71 @code{enable-multibyte-characters} specifies the representation used.
72 The representation for a string is determined and recorded in the string
73 when the string is constructed.
75 @defvar enable-multibyte-characters
76 This variable specifies the current buffer's text representation.
77 If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
78 it contains unibyte text.
80 You cannot set this variable directly; instead, use the function
81 @code{set-buffer-multibyte} to change a buffer's representation.
84 @defvar default-enable-multibyte-characters
85 This variable's value is entirely equivalent to @code{(default-value
86 'enable-multibyte-characters)}, and setting this variable changes that
87 default value. Setting the local binding of
88 @code{enable-multibyte-characters} in a specific buffer is not allowed,
89 but changing the default value is supported, and it is a reasonable
90 thing to do, because it has no effect on existing buffers.
92 The @samp{--unibyte} command line option does its job by setting the
93 default value to @code{nil} early in startup.
96 @defun position-bytes position
97 @tindex position-bytes
98 Return the byte-position corresponding to buffer position @var{position}
99 in the current buffer.
102 @defun byte-to-position byte-position
103 @tindex byte-to-position
104 Return the buffer position corresponding to byte-position
105 @var{byte-position} in the current buffer.
108 @defun multibyte-string-p string
109 Return @code{t} if @var{string} is a multibyte string.
112 @node Converting Representations
113 @section Converting Text Representations
115 Emacs can convert unibyte text to multibyte; it can also convert
116 multibyte text to unibyte, though this conversion loses information. In
117 general these conversions happen when inserting text into a buffer, or
118 when putting text from several strings together in one string. You can
119 also explicitly convert a string's contents to either representation.
121 Emacs chooses the representation for a string based on the text that
122 it is constructed from. The general rule is to convert unibyte text to
123 multibyte text when combining it with other multibyte text, because the
124 multibyte representation is more general and can hold whatever
125 characters the unibyte text has.
127 When inserting text into a buffer, Emacs converts the text to the
128 buffer's representation, as specified by
129 @code{enable-multibyte-characters} in that buffer. In particular, when
130 you insert multibyte text into a unibyte buffer, Emacs converts the text
131 to unibyte, even though this conversion cannot in general preserve all
132 the characters that might be in the multibyte text. The other natural
133 alternative, to convert the buffer contents to multibyte, is not
134 acceptable because the buffer's representation is a choice made by the
135 user that cannot be overridden automatically.
137 Converting unibyte text to multibyte text leaves @sc{ascii} characters
138 unchanged, and likewise character codes 128 through 159. It converts
139 the non-@sc{ascii} codes 160 through 255 by adding the value
140 @code{nonascii-insert-offset} to each character code. By setting this
141 variable, you specify which character set the unibyte characters
142 correspond to (@pxref{Character Sets}). For example, if
143 @code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
144 'latin-iso8859-1) 128)}, then the unibyte non-@sc{ascii} characters
145 correspond to Latin 1. If it is 2688, which is @code{(- (make-char
146 'greek-iso8859-7) 128)}, then they correspond to Greek letters.
148 Converting multibyte text to unibyte is simpler: it discards all but
149 the low 8 bits of each character code. If @code{nonascii-insert-offset}
150 has a reasonable value, corresponding to the beginning of some character
151 set, this conversion is the inverse of the other: converting unibyte
152 text to multibyte and back to unibyte reproduces the original unibyte
155 @defvar nonascii-insert-offset
156 This variable specifies the amount to add to a non-@sc{ascii} character
157 when converting unibyte text to multibyte. It also applies when
158 @code{self-insert-command} inserts a character in the unibyte
159 non-@sc{ascii} range, 128 through 255. However, the functions
160 @code{insert} and @code{insert-char} do not perform this conversion.
162 The right value to use to select character set @var{cs} is @code{(-
163 (make-char @var{cs}) 128)}. If the value of
164 @code{nonascii-insert-offset} is zero, then conversion actually uses the
165 value for the Latin 1 character set, rather than zero.
168 @defvar nonascii-translation-table
169 This variable provides a more general alternative to
170 @code{nonascii-insert-offset}. You can use it to specify independently
171 how to translate each code in the range of 128 through 255 into a
172 multibyte character. The value should be a char-table, or @code{nil}.
173 If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
176 @defun string-make-unibyte string
177 This function converts the text of @var{string} to unibyte
178 representation, if it isn't already, and returns the result. If
179 @var{string} is a unibyte string, it is returned unchanged.
180 Multibyte character codes are converted to unibyte
181 by using just the low 8 bits.
184 @defun string-make-multibyte string
185 This function converts the text of @var{string} to multibyte
186 representation, if it isn't already, and returns the result. If
187 @var{string} is a multibyte string, it is returned unchanged.
188 The function @code{unibyte-char-to-multibyte} is used to convert
189 each unibyte character to a multibyte character.
192 @node Selecting a Representation
193 @section Selecting a Representation
195 Sometimes it is useful to examine an existing buffer or string as
196 multibyte when it was unibyte, or vice versa.
198 @defun set-buffer-multibyte multibyte
199 Set the representation type of the current buffer. If @var{multibyte}
200 is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
201 is @code{nil}, the buffer becomes unibyte.
203 This function leaves the buffer contents unchanged when viewed as a
204 sequence of bytes. As a consequence, it can change the contents viewed
205 as characters; a sequence of two bytes which is treated as one character
206 in multibyte representation will count as two characters in unibyte
207 representation. Character codes 128 through 159 are an exception. They
208 are represented by one byte in a unibyte buffer, but when the buffer is
209 set to multibyte, they are converted to two-byte sequences, and vice
212 This function sets @code{enable-multibyte-characters} to record which
213 representation is in use. It also adjusts various data in the buffer
214 (including overlays, text properties and markers) so that they cover the
215 same text as they did before.
217 You cannot use @code{set-buffer-multibyte} on an indirect buffer,
218 because indirect buffers always inherit the representation of the
222 @defun string-as-unibyte string
223 This function returns a string with the same bytes as @var{string} but
224 treating each byte as a character. This means that the value may have
225 more characters than @var{string} has.
227 If @var{string} is already a unibyte string, then the value is
228 @var{string} itself. Otherwise it is a newly created string, with no
229 text properties. If @var{string} is multibyte, any characters it
230 contains of charset @var{eight-bit-control} or @var{eight-bit-graphic}
231 are converted to the corresponding single byte.
234 @defun string-as-multibyte string
235 This function returns a string with the same bytes as @var{string} but
236 treating each multibyte sequence as one character. This means that the
237 value may have fewer characters than @var{string} has.
239 If @var{string} is already a multibyte string, then the value is
240 @var{string} itself. Otherwise it is a newly created string, with no
241 text properties. If @var{string} is unibyte and contains any individual
242 8-bit bytes (i.e.@: not part of a multibyte form), they are converted to
243 the corresponding multibyte character of charset @var{eight-bit-control}
244 or @var{eight-bit-graphic}.
247 @node Character Codes
248 @section Character Codes
249 @cindex character codes
251 The unibyte and multibyte text representations use different character
252 codes. The valid character codes for unibyte representation range from
253 0 to 255---the values that can fit in one byte. The valid character
254 codes for multibyte representation range from 0 to 524287, but not all
255 values in that range are valid. The values 128 through 255 are not
256 entirely proper in multibyte text, but they can occur if you do explicit
257 encoding and decoding (@pxref{Explicit Encoding}). Some other character
258 codes cannot occur at all in multibyte text. Only the @sc{ascii} codes
259 0 through 127 are completely legitimate in both representations.
261 @defun char-valid-p charcode &optional genericp
262 This returns @code{t} if @var{charcode} is valid for either one of the two
263 text representations.
274 If the optional argument @var{genericp} is non-nil, this function
275 returns @code{t} if @var{charcode} is a generic character
276 (@pxref{Splitting Characters}).
280 @section Character Sets
281 @cindex character sets
283 Emacs classifies characters into various @dfn{character sets}, each of
284 which has a name which is a symbol. Each character belongs to one and
285 only one character set.
287 In general, there is one character set for each distinct script. For
288 example, @code{latin-iso8859-1} is one character set,
289 @code{greek-iso8859-7} is another, and @code{ascii} is another. An
290 Emacs character set can hold at most 9025 characters; therefore, in some
291 cases, characters that would logically be grouped together are split
292 into several character sets. For example, one set of Chinese
293 characters, generally known as Big 5, is divided into two Emacs
294 character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
296 @sc{ascii} characters are in character set @code{ascii}. The
297 non-@sc{ascii} characters 128 through 159 are in character set
298 @code{eight-bit-control}, and codes 160 through 255 are in character set
299 @code{eight-bit-graphic}.
301 @defun charsetp object
302 Returns @code{t} if @var{object} is a symbol that names a character set,
303 @code{nil} otherwise.
307 This function returns a list of all defined character set names.
310 @defun char-charset character
311 This function returns the name of the character set that @var{character}
315 @defun charset-plist charset
316 @tindex charset-plist
317 This function returns the charset property list of the character set
318 @var{charset}. Although @var{charset} is a symbol, this is not the same
319 as the property list of that symbol. Charset properties are used for
320 special purposes within Emacs; for example,
321 @code{preferred-coding-system} helps determine which coding system to
322 use to encode characters in a charset.
325 @node Chars and Bytes
326 @section Characters and Bytes
327 @cindex bytes and characters
329 @cindex introduction sequence
330 @cindex dimension (of character set)
331 In multibyte representation, each character occupies one or more
332 bytes. Each character set has an @dfn{introduction sequence}, which is
333 normally one or two bytes long. (Exception: the @sc{ascii} character
334 set and the @sc{eight-bit-graphic} character set have a zero-length
335 introduction sequence.) The introduction sequence is the beginning of
336 the byte sequence for any character in the character set. The rest of
337 the character's bytes distinguish it from the other characters in the
338 same character set. Depending on the character set, there are either
339 one or two distinguishing bytes; the number of such bytes is called the
340 @dfn{dimension} of the character set.
342 @defun charset-dimension charset
343 This function returns the dimension of @var{charset}; at present, the
344 dimension is always 1 or 2.
347 @defun charset-bytes charset
348 @tindex charset-bytes
349 This function returns the number of bytes used to represent a character
350 in character set @var{charset}.
353 This is the simplest way to determine the byte length of a character
354 set's introduction sequence:
357 (- (charset-bytes @var{charset})
358 (charset-dimension @var{charset}))
361 @node Splitting Characters
362 @section Splitting Characters
364 The functions in this section convert between characters and the byte
365 values used to represent them. For most purposes, there is no need to
366 be concerned with the sequence of bytes used to represent a character,
367 because Emacs translates automatically when necessary.
369 @defun split-char character
370 Return a list containing the name of the character set of
371 @var{character}, followed by one or two byte values (integers) which
372 identify @var{character} within that character set. The number of byte
373 values is the character set's dimension.
377 @result{} (latin-iso8859-1 72)
381 @result{} (eight-bit-control 128)
385 @defun make-char charset &optional code1 code2
386 This function returns the character in character set @var{charset} whose
387 position codes are @var{code1} and @var{code2}. This is roughly the
388 inverse of @code{split-char}. Normally, you should specify either one
389 or both of @var{code1} and @var{code2} according to the dimension of
390 @var{charset}. For example,
393 (make-char 'latin-iso8859-1 72)
398 @cindex generic characters
399 If you call @code{make-char} with no @var{byte-values}, the result is
400 a @dfn{generic character} which stands for @var{charset}. A generic
401 character is an integer, but it is @emph{not} valid for insertion in the
402 buffer as a character. It can be used in @code{char-table-range} to
403 refer to the whole character set (@pxref{Char-Tables}).
404 @code{char-valid-p} returns @code{nil} for generic characters.
408 (make-char 'latin-iso8859-1)
412 (char-valid-p 2176 t)
415 @result{} (latin-iso8859-1 0)
418 The character sets @sc{ascii}, @sc{eight-bit-control}, and
419 @sc{eight-bit-graphic} don't have corresponding generic characters. If
420 @var{charset} is one of them and you don't supply @var{code1},
421 @code{make-char} returns the character code corresponding to the
422 smallest code in @var{charset}.
424 @node Scanning Charsets
425 @section Scanning for Character Sets
427 Sometimes it is useful to find out which character sets appear in a
428 part of a buffer or a string. One use for this is in determining which
429 coding systems (@pxref{Coding Systems}) are capable of representing all
430 of the text in question.
432 @defun find-charset-region beg end &optional translation
433 This function returns a list of the character sets that appear in the
434 current buffer between positions @var{beg} and @var{end}.
436 The optional argument @var{translation} specifies a translation table to
437 be used in scanning the text (@pxref{Translation of Characters}). If it
438 is non-@code{nil}, then each character in the region is translated
439 through this table, and the value returned describes the translated
440 characters instead of the characters actually in the buffer.
443 @defun find-charset-string string &optional translation
444 This function returns a list of the character sets that appear in the
445 string @var{string}. It is just like @code{find-charset-region}, except
446 that it applies to the contents of @var{string} instead of part of the
450 @node Translation of Characters
451 @section Translation of Characters
452 @cindex character translation tables
453 @cindex translation tables
455 A @dfn{translation table} specifies a mapping of characters
456 into characters. These tables are used in encoding and decoding, and
457 for other purposes. Some coding systems specify their own particular
458 translation tables; there are also default translation tables which
459 apply to all other coding systems.
461 @defun make-translation-table &rest translations
462 This function returns a translation table based on the argument
463 @var{translations}. Each element of @var{translations} should be a
464 list of elements of the form @code{(@var{from} . @var{to})}; this says
465 to translate the character @var{from} into @var{to}.
467 The arguments and the forms in each argument are processed in order,
468 and if a previous form already translates @var{to} to some other
469 character, say @var{to-alt}, @var{from} is also translated to
472 You can also map one whole character set into another character set with
473 the same dimension. To do this, you specify a generic character (which
474 designates a character set) for @var{from} (@pxref{Splitting Characters}).
475 In this case, @var{to} should also be a generic character, for another
476 character set of the same dimension. Then the translation table
477 translates each character of @var{from}'s character set into the
478 corresponding character of @var{to}'s character set.
481 In decoding, the translation table's translations are applied to the
482 characters that result from ordinary decoding. If a coding system has
483 property @code{character-translation-table-for-decode}, that specifies
484 the translation table to use. Otherwise, if
485 @code{standard-translation-table-for-decode} is non-@code{nil}, decoding
488 In encoding, the translation table's translations are applied to the
489 characters in the buffer, and the result of translation is actually
490 encoded. If a coding system has property
491 @code{character-translation-table-for-encode}, that specifies the
492 translation table to use. Otherwise the variable
493 @code{standard-translation-table-for-encode} specifies the translation
496 @defvar standard-translation-table-for-decode
497 This is the default translation table for decoding, for
498 coding systems that don't specify any other translation table.
501 @defvar standard-translation-table-for-encode
502 This is the default translation table for encoding, for
503 coding systems that don't specify any other translation table.
507 @section Coding Systems
509 @cindex coding system
510 When Emacs reads or writes a file, and when Emacs sends text to a
511 subprocess or receives text from a subprocess, it normally performs
512 character code conversion and end-of-line conversion as specified
513 by a particular @dfn{coding system}.
515 How to define a coding system is an arcane matter, and is not
519 * Coding System Basics:: Basic concepts.
520 * Encoding and I/O:: How file I/O functions handle coding systems.
521 * Lisp and Coding Systems:: Functions to operate on coding system names.
522 * User-Chosen Coding Systems:: Asking the user to choose a coding system.
523 * Default Coding Systems:: Controlling the default choices.
524 * Specifying Coding Systems:: Requesting a particular coding system
525 for a single file operation.
526 * Explicit Encoding:: Encoding or decoding text without doing I/O.
527 * Terminal I/O Encoding:: Use of encoding for terminal I/O.
528 * MS-DOS File Types:: How DOS "text" and "binary" files
529 relate to coding systems.
532 @node Coding System Basics
533 @subsection Basic Concepts of Coding Systems
535 @cindex character code conversion
536 @dfn{Character code conversion} involves conversion between the encoding
537 used inside Emacs and some other encoding. Emacs supports many
538 different encodings, in that it can convert to and from them. For
539 example, it can convert text to or from encodings such as Latin 1, Latin
540 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some
541 cases, Emacs supports several alternative encodings for the same
542 characters; for example, there are three coding systems for the Cyrillic
543 (Russian) alphabet: ISO, Alternativnyj, and KOI8.
545 Most coding systems specify a particular character code for
546 conversion, but some of them leave the choice unspecified---to be chosen
547 heuristically for each file, based on the data.
549 @cindex end of line conversion
550 @dfn{End of line conversion} handles three different conventions used
551 on various systems for representing end of line in files. The Unix
552 convention is to use the linefeed character (also called newline). The
553 DOS convention is to use a carriage-return and a linefeed at the end of
554 a line. The Mac convention is to use just carriage-return.
556 @cindex base coding system
557 @cindex variant coding system
558 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
559 conversion unspecified, to be chosen based on the data. @dfn{Variant
560 coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
561 @code{latin-1-mac} specify the end-of-line conversion explicitly as
562 well. Most base coding systems have three corresponding variants whose
563 names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
565 The coding system @code{raw-text} is special in that it prevents
566 character code conversion, and causes the buffer visited with that
567 coding system to be a unibyte buffer. It does not specify the
568 end-of-line conversion, allowing that to be determined as usual by the
569 data, and has the usual three variants which specify the end-of-line
570 conversion. @code{no-conversion} is equivalent to @code{raw-text-unix}:
571 it specifies no conversion of either character codes or end-of-line.
573 The coding system @code{emacs-mule} specifies that the data is
574 represented in the internal Emacs encoding. This is like
575 @code{raw-text} in that no code conversion happens, but different in
576 that the result is multibyte data.
578 @defun coding-system-get coding-system property
579 This function returns the specified property of the coding system
580 @var{coding-system}. Most coding system properties exist for internal
581 purposes, but one that you might find useful is @code{mime-charset}.
582 That property's value is the name used in MIME for the character coding
583 which this coding system can read and write. Examples:
586 (coding-system-get 'iso-latin-1 'mime-charset)
588 (coding-system-get 'iso-2022-cn 'mime-charset)
589 @result{} iso-2022-cn
590 (coding-system-get 'cyrillic-koi8 'mime-charset)
594 The value of the @code{mime-charset} property is also defined
595 as an alias for the coding system.
598 @node Encoding and I/O
599 @subsection Encoding and I/O
601 The principal purpose of coding systems is for use in reading and
602 writing files. The function @code{insert-file-contents} uses
603 a coding system for decoding the file data, and @code{write-region}
604 uses one to encode the buffer contents.
606 You can specify the coding system to use either explicitly
607 (@pxref{Specifying Coding Systems}), or implicitly using the defaulting
608 mechanism (@pxref{Default Coding Systems}). But these methods may not
609 completely specify what to do. For example, they may choose a coding
610 system such as @code{undefined} which leaves the character code
611 conversion to be determined from the data. In these cases, the I/O
612 operation finishes the job of choosing a coding system. Very often
613 you will want to find out afterwards which coding system was chosen.
615 @defvar buffer-file-coding-system
616 This variable records the coding system that was used for visiting the
617 current buffer. It is used for saving the buffer, and for writing part
618 of the buffer with @code{write-region}. When those operations ask the
619 user to specify a different coding system,
620 @code{buffer-file-coding-system} is updated to the coding system
623 However, @code{buffer-file-coding-system} does not affect sending text
627 @defvar save-buffer-coding-system
628 This variable specifies the coding system for saving the buffer (by
629 overriding @code{buffer-file-coding-system}). Note that it is not used
630 for @code{write-region}.
632 When a command to save the buffer starts out to use
633 @code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
634 and that coding system cannot handle
635 the actual text in the buffer, the command asks the user to choose
636 another coding system. After that happens, the command also updates
637 @code{buffer-file-coding-system} to represent the coding system that the
641 @defvar last-coding-system-used
642 I/O operations for files and subprocesses set this variable to the
643 coding system name that was used. The explicit encoding and decoding
644 functions (@pxref{Explicit Encoding}) set it too.
646 @strong{Warning:} Since receiving subprocess output sets this variable,
647 it can change whenever Emacs waits; therefore, you should copy the
648 value shortly after the function call that stores the value you are
652 The variable @code{selection-coding-system} specifies how to encode
653 selections for the window system. @xref{Window System Selections}.
655 @node Lisp and Coding Systems
656 @subsection Coding Systems in Lisp
658 Here are the Lisp facilities for working with coding systems:
660 @defun coding-system-list &optional base-only
661 This function returns a list of all coding system names (symbols). If
662 @var{base-only} is non-@code{nil}, the value includes only the
663 base coding systems. Otherwise, it includes alias and variant coding
667 @defun coding-system-p object
668 This function returns @code{t} if @var{object} is a coding system
672 @defun check-coding-system coding-system
673 This function checks the validity of @var{coding-system}.
674 If that is valid, it returns @var{coding-system}.
675 Otherwise it signals an error with condition @code{coding-system-error}.
678 @defun coding-system-change-eol-conversion coding-system eol-type
679 This function returns a coding system which is like @var{coding-system}
680 except for its eol conversion, which is specified by @code{eol-type}.
681 @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
682 @code{nil}. If it is @code{nil}, the returned coding system determines
683 the end-of-line conversion from the data.
686 @defun coding-system-change-text-conversion eol-coding text-coding
687 This function returns a coding system which uses the end-of-line
688 conversion of @var{eol-coding}, and the text conversion of
689 @var{text-coding}. If @var{text-coding} is @code{nil}, it returns
690 @code{undecided}, or one of its variants according to @var{eol-coding}.
693 @defun find-coding-systems-region from to
694 This function returns a list of coding systems that could be used to
695 encode a text between @var{from} and @var{to}. All coding systems in
696 the list can safely encode any multibyte characters in that portion of
699 If the text contains no multibyte characters, the function returns the
700 list @code{(undecided)}.
703 @defun find-coding-systems-string string
704 This function returns a list of coding systems that could be used to
705 encode the text of @var{string}. All coding systems in the list can
706 safely encode any multibyte characters in @var{string}. If the text
707 contains no multibyte characters, this returns the list
711 @defun find-coding-systems-for-charsets charsets
712 This function returns a list of coding systems that could be used to
713 encode all the character sets in the list @var{charsets}.
716 @defun detect-coding-region start end &optional highest
717 This function chooses a plausible coding system for decoding the text
718 from @var{start} to @var{end}. This text should be a byte sequence
719 (@pxref{Explicit Encoding}).
721 Normally this function returns a list of coding systems that could
722 handle decoding the text that was scanned. They are listed in order of
723 decreasing priority. But if @var{highest} is non-@code{nil}, then the
724 return value is just one coding system, the one that is highest in
727 If the region contains only @sc{ascii} characters, the value
728 is @code{undecided} or @code{(undecided)}.
731 @defun detect-coding-string string highest
732 This function is like @code{detect-coding-region} except that it
733 operates on the contents of @var{string} instead of bytes in the buffer.
736 @xref{Process Information}, for how to examine or set the coding
737 systems used for I/O to a subprocess.
739 @node User-Chosen Coding Systems
740 @subsection User-Chosen Coding Systems
742 @defun select-safe-coding-system from to &optional default-coding-system accept-default-p
743 This function selects a coding system for encoding specified text,
744 asking the user to choose if necessary. Normally the specified text
745 is the text in the current buffer between @var{from} and @var{to},
746 defaulting to the whole buffer if they are @code{nil}. If @var{from}
747 is a string, the string is the target text, and @var{to} is ignored.
749 If @var{default-coding-system} is non-@code{nil}, that is the first
750 coding system to try; if that can handle the text,
751 @code{select-safe-coding-system} returns that coding system. It can
752 also be a list of coding systems; then the function tries each of them
753 one by one. After trying all of them, it next tries the user's most
754 preferred coding system (@pxref{Recognize Coding,
755 prefer-coding-system, the description of @code{prefer-coding-system},
756 emacs, GNU Emacs Manual}), and after that the current buffer's value
757 of @code{buffer-file-coding-system} (if it is not @code{undecided}).
759 If one of those coding systems can safely encode all the specified
760 text, @code{select-safe-coding-system} chooses it and returns it.
761 Otherwise, it asks the user to choose from a list of coding systems
762 which can encode all the text, and returns the user's choice.
764 The optional argument @var{accept-default-p}, if non-@code{nil},
765 should be a function to determine whether the coding system selected
766 without user interaction is acceptable. If this function returns
767 @code{nil}, the silently selected coding system is rejected, and the
768 user is asked to select a coding system from a list of possible
771 @vindex select-safe-coding-system-accept-default-p
772 If the variable @code{select-safe-coding-system-accept-default-p} is
773 non-@code{nil}, its value overrides the value of
774 @var{accept-default-p}.
777 Here are two functions you can use to let the user specify a coding
778 system, with completion. @xref{Completion}.
780 @defun read-coding-system prompt &optional default
781 This function reads a coding system using the minibuffer, prompting with
782 string @var{prompt}, and returns the coding system name as a symbol. If
783 the user enters null input, @var{default} specifies which coding system
784 to return. It should be a symbol or a string.
787 @defun read-non-nil-coding-system prompt
788 This function reads a coding system using the minibuffer, prompting with
789 string @var{prompt}, and returns the coding system name as a symbol. If
790 the user tries to enter null input, it asks the user to try again.
791 @xref{Coding Systems}.
794 @node Default Coding Systems
795 @subsection Default Coding Systems
797 This section describes variables that specify the default coding
798 system for certain files or when running certain subprograms, and the
799 function that I/O operations use to access them.
801 The idea of these variables is that you set them once and for all to the
802 defaults you want, and then do not change them again. To specify a
803 particular coding system for a particular operation in a Lisp program,
804 don't change these variables; instead, override them using
805 @code{coding-system-for-read} and @code{coding-system-for-write}
806 (@pxref{Specifying Coding Systems}).
808 @defvar auto-coding-regexp-alist
809 This variable is an alist of text patterns and corresponding coding
810 systems. Each element has the form @code{(@var{regexp}
811 . @var{coding-system})}; a file whose first few kilobytes match
812 @var{regexp} is decoded with @var{coding-system} when its contents are
813 read into a buffer. The settings in this alist take priority over
814 @code{coding:} tags in the files and the contents of
815 @code{file-coding-system-alist} (see below). The default value is set
816 so that Emacs automatically recognizes mail files in Babyl format and
817 reads them with no code conversions.
820 @defvar file-coding-system-alist
821 This variable is an alist that specifies the coding systems to use for
822 reading and writing particular files. Each element has the form
823 @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
824 expression that matches certain file names. The element applies to file
825 names that match @var{pattern}.
827 The @sc{cdr} of the element, @var{coding}, should be either a coding
828 system, a cons cell containing two coding systems, or a function name (a
829 symbol with a function definition). If @var{coding} is a coding system,
830 that coding system is used for both reading the file and writing it. If
831 @var{coding} is a cons cell containing two coding systems, its @sc{car}
832 specifies the coding system for decoding, and its @sc{cdr} specifies the
833 coding system for encoding.
835 If @var{coding} is a function name, the function must return a coding
836 system or a cons cell containing two coding systems. This value is used
840 @defvar process-coding-system-alist
841 This variable is an alist specifying which coding systems to use for a
842 subprocess, depending on which program is running in the subprocess. It
843 works like @code{file-coding-system-alist}, except that @var{pattern} is
844 matched against the program name used to start the subprocess. The coding
845 system or systems specified in this alist are used to initialize the
846 coding systems used for I/O to the subprocess, but you can specify
847 other coding systems later using @code{set-process-coding-system}.
850 @strong{Warning:} Coding systems such as @code{undecided}, which
851 determine the coding system from the data, do not work entirely reliably
852 with asynchronous subprocess output. This is because Emacs handles
853 asynchronous subprocess output in batches, as it arrives. If the coding
854 system leaves the character code conversion unspecified, or leaves the
855 end-of-line conversion unspecified, Emacs must try to detect the proper
856 conversion from one batch at a time, and this does not always work.
858 Therefore, with an asynchronous subprocess, if at all possible, use a
859 coding system which determines both the character code conversion and
860 the end of line conversion---that is, one like @code{latin-1-unix},
861 rather than @code{undecided} or @code{latin-1}.
863 @defvar network-coding-system-alist
864 This variable is an alist that specifies the coding system to use for
865 network streams. It works much like @code{file-coding-system-alist},
866 with the difference that the @var{pattern} in an element may be either a
867 port number or a regular expression. If it is a regular expression, it
868 is matched against the network service name used to open the network
872 @defvar default-process-coding-system
873 This variable specifies the coding systems to use for subprocess (and
874 network stream) input and output, when nothing else specifies what to
877 The value should be a cons cell of the form @code{(@var{input-coding}
878 . @var{output-coding})}. Here @var{input-coding} applies to input from
879 the subprocess, and @var{output-coding} applies to output to it.
882 @defun find-operation-coding-system operation &rest arguments
883 This function returns the coding system to use (by default) for
884 performing @var{operation} with @var{arguments}. The value has this
888 (@var{decoding-system} @var{encoding-system})
891 The first element, @var{decoding-system}, is the coding system to use
892 for decoding (in case @var{operation} does decoding), and
893 @var{encoding-system} is the coding system for encoding (in case
894 @var{operation} does encoding).
896 The argument @var{operation} should be a symbol, one of
897 @code{insert-file-contents}, @code{write-region}, @code{call-process},
898 @code{call-process-region}, @code{start-process}, or
899 @code{open-network-stream}. These are the names of the Emacs I/O primitives
900 that can do coding system conversion.
902 The remaining arguments should be the same arguments that might be given
903 to that I/O primitive. Depending on the primitive, one of those
904 arguments is selected as the @dfn{target}. For example, if
905 @var{operation} does file I/O, whichever argument specifies the file
906 name is the target. For subprocess primitives, the process name is the
907 target. For @code{open-network-stream}, the target is the service name
910 This function looks up the target in @code{file-coding-system-alist},
911 @code{process-coding-system-alist}, or
912 @code{network-coding-system-alist}, depending on @var{operation}.
913 @xref{Default Coding Systems}.
916 @node Specifying Coding Systems
917 @subsection Specifying a Coding System for One Operation
919 You can specify the coding system for a specific operation by binding
920 the variables @code{coding-system-for-read} and/or
921 @code{coding-system-for-write}.
923 @defvar coding-system-for-read
924 If this variable is non-@code{nil}, it specifies the coding system to
925 use for reading a file, or for input from a synchronous subprocess.
927 It also applies to any asynchronous subprocess or network stream, but in
928 a different way: the value of @code{coding-system-for-read} when you
929 start the subprocess or open the network stream specifies the input
930 decoding method for that subprocess or network stream. It remains in
931 use for that subprocess or network stream unless and until overridden.
933 The right way to use this variable is to bind it with @code{let} for a
934 specific I/O operation. Its global value is normally @code{nil}, and
935 you should not globally set it to any other value. Here is an example
936 of the right way to use the variable:
939 ;; @r{Read the file with no character code conversion.}
940 ;; @r{Assume @sc{crlf} represents end-of-line.}
941 (let ((coding-system-for-write 'emacs-mule-dos))
942 (insert-file-contents filename))
945 When its value is non-@code{nil}, @code{coding-system-for-read} takes
946 precedence over all other methods of specifying a coding system to use for
947 input, including @code{file-coding-system-alist},
948 @code{process-coding-system-alist} and
949 @code{network-coding-system-alist}.
952 @defvar coding-system-for-write
953 This works much like @code{coding-system-for-read}, except that it
954 applies to output rather than input. It affects writing to files,
955 as well as sending output to subprocesses and net connections.
957 When a single operation does both input and output, as do
958 @code{call-process-region} and @code{start-process}, both
959 @code{coding-system-for-read} and @code{coding-system-for-write}
963 @defvar inhibit-eol-conversion
964 When this variable is non-@code{nil}, no end-of-line conversion is done,
965 no matter which coding system is specified. This applies to all the
966 Emacs I/O and subprocess primitives, and to the explicit encoding and
967 decoding functions (@pxref{Explicit Encoding}).
970 @node Explicit Encoding
971 @subsection Explicit Encoding and Decoding
972 @cindex encoding text
973 @cindex decoding text
975 All the operations that transfer text in and out of Emacs have the
976 ability to use a coding system to encode or decode the text.
977 You can also explicitly encode and decode text using the functions
980 The result of encoding, and the input to decoding, are not ordinary
981 text. They logically consist of a series of byte values; that is, a
982 series of characters whose codes are in the range 0 through 255. In a
983 multibyte buffer or string, character codes 128 through 159 are
984 represented by multibyte sequences, but this is invisible to Lisp
987 The usual way to read a file into a buffer as a sequence of bytes, so
988 you can decode the contents explicitly, is with
989 @code{insert-file-contents-literally} (@pxref{Reading from Files});
990 alternatively, specify a non-@code{nil} @var{rawfile} argument when
991 visiting a file with @code{find-file-noselect}. These methods result in
994 The usual way to use the byte sequence that results from explicitly
995 encoding text is to copy it to a file or process---for example, to write
996 it with @code{write-region} (@pxref{Writing to Files}), and suppress
997 encoding by binding @code{coding-system-for-write} to
998 @code{no-conversion}.
1000 Here are the functions to perform explicit encoding or decoding. The
1001 decoding functions produce sequences of bytes; the encoding functions
1002 are meant to operate on sequences of bytes. All of these functions
1003 discard text properties.
1005 @defun encode-coding-region start end coding-system
1006 This function encodes the text from @var{start} to @var{end} according
1007 to coding system @var{coding-system}. The encoded text replaces the
1008 original text in the buffer. The result of encoding is logically a
1009 sequence of bytes, but the buffer remains multibyte if it was multibyte
1013 @defun encode-coding-string string coding-system
1014 This function encodes the text in @var{string} according to coding
1015 system @var{coding-system}. It returns a new string containing the
1016 encoded text. The result of encoding is a unibyte string.
1019 @defun decode-coding-region start end coding-system
1020 This function decodes the text from @var{start} to @var{end} according
1021 to coding system @var{coding-system}. The decoded text replaces the
1022 original text in the buffer. To make explicit decoding useful, the text
1023 before decoding ought to be a sequence of byte values, but both
1024 multibyte and unibyte buffers are acceptable.
1027 @defun decode-coding-string string coding-system
1028 This function decodes the text in @var{string} according to coding
1029 system @var{coding-system}. It returns a new string containing the
1030 decoded text. To make explicit decoding useful, the contents of
1031 @var{string} ought to be a sequence of byte values, but a multibyte
1032 string is acceptable.
1035 @node Terminal I/O Encoding
1036 @subsection Terminal I/O Encoding
1038 Emacs can decode keyboard input using a coding system, and encode
1039 terminal output. This is useful for terminals that transmit or display
1040 text using a particular encoding such as Latin-1. Emacs does not set
1041 @code{last-coding-system-used} for encoding or decoding for the
1044 @defun keyboard-coding-system
1045 This function returns the coding system that is in use for decoding
1046 keyboard input---or @code{nil} if no coding system is to be used.
1049 @defun set-keyboard-coding-system coding-system
1050 This function specifies @var{coding-system} as the coding system to
1051 use for decoding keyboard input. If @var{coding-system} is @code{nil},
1052 that means do not decode keyboard input.
1055 @defun terminal-coding-system
1056 This function returns the coding system that is in use for encoding
1057 terminal output---or @code{nil} for no encoding.
1060 @defun set-terminal-coding-system coding-system
1061 This function specifies @var{coding-system} as the coding system to use
1062 for encoding terminal output. If @var{coding-system} is @code{nil},
1063 that means do not encode terminal output.
1066 @node MS-DOS File Types
1067 @subsection MS-DOS File Types
1068 @cindex DOS file types
1069 @cindex MS-DOS file types
1070 @cindex Windows file types
1071 @cindex file types on MS-DOS and Windows
1072 @cindex text files and binary files
1073 @cindex binary files and text files
1075 On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
1076 end-of-line conversion for a file by looking at the file's name. This
1077 feature classifies files as @dfn{text files} and @dfn{binary files}. By
1078 ``binary file'' we mean a file of literal byte values that are not
1079 necessarily meant to be characters; Emacs does no end-of-line conversion
1080 and no character code conversion for them. On the other hand, the bytes
1081 in a text file are intended to represent characters; when you create a
1082 new file whose name implies that it is a text file, Emacs uses DOS
1083 end-of-line conversion.
1085 @defvar buffer-file-type
1086 This variable, automatically buffer-local in each buffer, records the
1087 file type of the buffer's visited file. When a buffer does not specify
1088 a coding system with @code{buffer-file-coding-system}, this variable is
1089 used to determine which coding system to use when writing the contents
1090 of the buffer. It should be @code{nil} for text, @code{t} for binary.
1091 If it is @code{t}, the coding system is @code{no-conversion}.
1092 Otherwise, @code{undecided-dos} is used.
1094 Normally this variable is set by visiting a file; it is set to
1095 @code{nil} if the file was visited without any actual conversion.
1098 @defopt file-name-buffer-file-type-alist
1099 This variable holds an alist for recognizing text and binary files.
1100 Each element has the form (@var{regexp} . @var{type}), where
1101 @var{regexp} is matched against the file name, and @var{type} may be
1102 @code{nil} for text, @code{t} for binary, or a function to call to
1103 compute which. If it is a function, then it is called with a single
1104 argument (the file name) and should return @code{t} or @code{nil}.
1106 When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
1107 which coding system to use when reading a file. For a text file,
1108 @code{undecided-dos} is used. For a binary file, @code{no-conversion}
1111 If no element in this alist matches a given file name, then
1112 @code{default-buffer-file-type} says how to treat the file.
1115 @defopt default-buffer-file-type
1116 This variable says how to handle files for which
1117 @code{file-name-buffer-file-type-alist} says nothing about the type.
1119 If this variable is non-@code{nil}, then these files are treated as
1120 binary: the coding system @code{no-conversion} is used. Otherwise,
1121 nothing special is done for them---the coding system is deduced solely
1122 from the file contents, in the usual Emacs fashion.
1126 @section Input Methods
1127 @cindex input methods
1129 @dfn{Input methods} provide convenient ways of entering non-@sc{ascii}
1130 characters from the keyboard. Unlike coding systems, which translate
1131 non-@sc{ascii} characters to and from encodings meant to be read by
1132 programs, input methods provide human-friendly commands. (@xref{Input
1133 Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1134 use input methods to enter text.) How to define input methods is not
1135 yet documented in this manual, but here we describe how to use them.
1137 Each input method has a name, which is currently a string;
1138 in the future, symbols may also be usable as input method names.
1140 @defvar current-input-method
1141 This variable holds the name of the input method now active in the
1142 current buffer. (It automatically becomes local in each buffer when set
1143 in any fashion.) It is @code{nil} if no input method is active in the
1147 @defvar default-input-method
1148 This variable holds the default input method for commands that choose an
1149 input method. Unlike @code{current-input-method}, this variable is
1153 @defun set-input-method input-method
1154 This function activates input method @var{input-method} for the current
1155 buffer. It also sets @code{default-input-method} to @var{input-method}.
1156 If @var{input-method} is @code{nil}, this function deactivates any input
1157 method for the current buffer.
1160 @defun read-input-method-name prompt &optional default inhibit-null
1161 This function reads an input method name with the minibuffer, prompting
1162 with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1163 by default, if the user enters empty input. However, if
1164 @var{inhibit-null} is non-@code{nil}, empty input signals an error.
1166 The returned value is a string.
1169 @defvar input-method-alist
1170 This variable defines all the supported input methods.
1171 Each element defines one input method, and should have the form:
1174 (@var{input-method} @var{language-env} @var{activate-func}
1175 @var{title} @var{description} @var{args}...)
1178 Here @var{input-method} is the input method name, a string;
1179 @var{language-env} is another string, the name of the language
1180 environment this input method is recommended for. (That serves only for
1181 documentation purposes.)
1183 @var{activate-func} is a function to call to activate this method. The
1184 @var{args}, if any, are passed as arguments to @var{activate-func}. All
1185 told, the arguments to @var{activate-func} are @var{input-method} and
1188 @var{title} is a string to display in the mode line while this method is
1189 active. @var{description} is a string describing this method and what
1193 The fundamental interface to input methods is through the
1194 variable @code{input-method-function}. @xref{Reading One Event}.
1200 POSIX defines a concept of ``locales'' which control which language
1201 to use in language-related features. These Emacs variables control
1202 how Emacs interacts with these features.
1204 @defvar locale-coding-system
1205 @tindex locale-coding-system
1206 This variable specifies the coding system to use for decoding system
1207 error messages, for encoding the format argument to
1208 @code{format-time-string}, and for decoding the return value of
1209 @code{format-time-string}.
1212 @defvar system-messages-locale
1213 @tindex system-messages-locale
1214 This variable specifies the locale to use for generating system error
1215 messages. Changing the locale can cause messages to come out in a
1216 different language or in a different orthography. If the variable is
1217 @code{nil}, the locale is specified by environment variables in the
1218 usual POSIX fashion.
1221 @defvar system-time-locale
1222 @tindex system-time-locale
1223 This variable specifies the locale to use for formatting time values.
1224 Changing the locale can cause messages to appear according to the
1225 conventions of a different language. If the variable is @code{nil}, the
1226 locale is specified by environment variables in the usual POSIX fashion.