2 @c This is part of the GNU Emacs Lisp Reference Manual.
3 @c Copyright (C) 1998 Free Software Foundation, Inc.
4 @c See the file elisp.texi for copying conditions.
5 @setfilename ../info/characters
6 @node Non-ASCII Characters, Searching and Matching, Text, Top
7 @chapter Non-ASCII Characters
8 @cindex multibyte characters
9 @cindex non-ASCII characters
11 This chapter covers the special issues relating to non-@sc{ASCII}
12 characters and how they are stored in strings and buffers.
15 * Text Representations::
16 * Converting Representations::
17 * Selecting a Representation::
21 * Splitting Characters::
23 * Translation of Characters::
28 @node Text Representations
29 @section Text Representations
30 @cindex text representations
32 Emacs has two @dfn{text representations}---two ways to represent text
33 in a string or buffer. These are called @dfn{unibyte} and
34 @dfn{multibyte}. Each string, and each buffer, uses one of these two
35 representations. For most purposes, you can ignore the issue of
36 representations, because Emacs converts text between them as
37 appropriate. Occasionally in Lisp programming you will need to pay
38 attention to the difference.
41 In unibyte representation, each character occupies one byte and
42 therefore the possible character codes range from 0 to 255. Codes 0
43 through 127 are @sc{ASCII} characters; the codes from 128 through 255
44 are used for one non-@sc{ASCII} character set (you can choose which
45 character set by setting the variable @code{nonascii-insert-offset}).
48 @cindex multibyte text
49 @cindex trailing codes
50 In multibyte representation, a character may occupy more than one
51 byte, and as a result, the full range of Emacs character codes can be
52 stored. The first byte of a multibyte character is always in the range
53 128 through 159 (octal 0200 through 0237). These values are called
54 @dfn{leading codes}. The second and subsequent bytes of a multibyte
55 character are always in the range 160 through 255 (octal 0240 through
56 0377); these values are @dfn{trailing codes}.
58 In a buffer, the buffer-local value of the variable
59 @code{enable-multibyte-characters} specifies the representation used.
60 The representation for a string is determined based on the string
61 contents when the string is constructed.
63 @defvar enable-multibyte-characters
64 @tindex enable-multibyte-characters
65 This variable specifies the current buffer's text representation.
66 If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
67 it contains unibyte text.
69 You cannot set this variable directly; instead, use the function
70 @code{set-buffer-multibyte} to change a buffer's representation.
73 @defvar default-enable-multibyte-characters
74 @tindex default-enable-multibyte-characters
75 This variable's value is entirely equivalent to @code{(default-value
76 'enable-multibyte-characters)}, and setting this variable changes that
77 default value. Setting the local binding of
78 @code{enable-multibyte-characters} in a specific buffer is not allowed,
79 but changing the default value is supported, and it is a reasonable
80 thing to do, because it has no effect on existing buffers.
82 The @samp{--unibyte} command line option does its job by setting the
83 default value to @code{nil} early in startup.
86 @defun multibyte-string-p string
87 @tindex multibyte-string-p
88 Return @code{t} if @var{string} contains multibyte characters.
91 @node Converting Representations
92 @section Converting Text Representations
94 Emacs can convert unibyte text to multibyte; it can also convert
95 multibyte text to unibyte, though this conversion loses information. In
96 general these conversions happen when inserting text into a buffer, or
97 when putting text from several strings together in one string. You can
98 also explicitly convert a string's contents to either representation.
100 Emacs chooses the representation for a string based on the text that
101 it is constructed from. The general rule is to convert unibyte text to
102 multibyte text when combining it with other multibyte text, because the
103 multibyte representation is more general and can hold whatever
104 characters the unibyte text has.
106 When inserting text into a buffer, Emacs converts the text to the
107 buffer's representation, as specified by
108 @code{enable-multibyte-characters} in that buffer. In particular, when
109 you insert multibyte text into a unibyte buffer, Emacs converts the text
110 to unibyte, even though this conversion cannot in general preserve all
111 the characters that might be in the multibyte text. The other natural
112 alternative, to convert the buffer contents to multibyte, is not
113 acceptable because the buffer's representation is a choice made by the
114 user that cannot be overridden automatically.
116 Converting unibyte text to multibyte text leaves @sc{ASCII} characters
117 unchanged, and likewise 128 through 159. It converts the non-@sc{ASCII}
118 codes 160 through 255 by adding the value @code{nonascii-insert-offset}
119 to each character code. By setting this variable, you specify which
120 character set the unibyte characters correspond to (@pxref{Character
121 Sets}). For example, if @code{nonascii-insert-offset} is 2048, which is
122 @code{(- (make-char 'latin-iso8859-1) 128)}, then the unibyte
123 non-@sc{ASCII} characters correspond to Latin 1. If it is 2688, which
124 is @code{(- (make-char 'greek-iso8859-7) 128)}, then they correspond to
127 Converting multibyte text to unibyte is simpler: it performs
128 logical-and of each character code with 255. If
129 @code{nonascii-insert-offset} has a reasonable value, corresponding to
130 the beginning of some character set, this conversion is the inverse of
131 the other: converting unibyte text to multibyte and back to unibyte
132 reproduces the original unibyte text.
134 @defvar nonascii-insert-offset
135 @tindex nonascii-insert-offset
136 This variable specifies the amount to add to a non-@sc{ASCII} character
137 when converting unibyte text to multibyte. It also applies when
138 @code{self-insert-command} inserts a character in the unibyte
139 non-@sc{ASCII} range, 128 through 255. However, the function
140 @code{insert-char} does not perform this conversion.
142 The right value to use to select character set @var{cs} is @code{(-
143 (make-char @var{cs}) 128)}. If the value of
144 @code{nonascii-insert-offset} is zero, then conversion actually uses the
145 value for the Latin 1 character set, rather than zero.
148 @defvar nonascii-translation-table
149 @tindex nonascii-translation-table
150 This variable provides a more general alternative to
151 @code{nonascii-insert-offset}. You can use it to specify independently
152 how to translate each code in the range of 128 through 255 into a
153 multibyte character. The value should be a vector, or @code{nil}.
154 If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
157 @defun string-make-unibyte string
158 @tindex string-make-unibyte
159 This function converts the text of @var{string} to unibyte
160 representation, if it isn't already, and returns the result. If
161 @var{string} is a unibyte string, it is returned unchanged.
164 @defun string-make-multibyte string
165 @tindex string-make-multibyte
166 This function converts the text of @var{string} to multibyte
167 representation, if it isn't already, and returns the result. If
168 @var{string} is a multibyte string, it is returned unchanged.
171 @node Selecting a Representation
172 @section Selecting a Representation
174 Sometimes it is useful to examine an existing buffer or string as
175 multibyte when it was unibyte, or vice versa.
177 @defun set-buffer-multibyte multibyte
178 @tindex set-buffer-multibyte
179 Set the representation type of the current buffer. If @var{multibyte}
180 is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
181 is @code{nil}, the buffer becomes unibyte.
183 This function leaves the buffer contents unchanged when viewed as a
184 sequence of bytes. As a consequence, it can change the contents viewed
185 as characters; a sequence of two bytes which is treated as one character
186 in multibyte representation will count as two characters in unibyte
189 This function sets @code{enable-multibyte-characters} to record which
190 representation is in use. It also adjusts various data in the buffer
191 (including overlays, text properties and markers) so that they cover the
192 same text as they did before.
195 @defun string-as-unibyte string
196 @tindex string-as-unibyte
197 This function returns a string with the same bytes as @var{string} but
198 treating each byte as a character. This means that the value may have
199 more characters than @var{string} has.
201 If @var{string} is unibyte already, then the value is @var{string}
205 @defun string-as-multibyte string
206 @tindex string-as-multibyte
207 This function returns a string with the same bytes as @var{string} but
208 treating each multibyte sequence as one character. This means that the
209 value may have fewer characters than @var{string} has.
211 If @var{string} is multibyte already, then the value is @var{string}
215 @node Character Codes
216 @section Character Codes
217 @cindex character codes
219 The unibyte and multibyte text representations use different character
220 codes. The valid character codes for unibyte representation range from
221 0 to 255---the values that can fit in one byte. The valid character
222 codes for multibyte representation range from 0 to 524287, but not all
223 values in that range are valid. In particular, the values 128 through
224 255 are not legitimate in multibyte text (though they can occur in ``raw
225 bytes''; @pxref{Explicit Encoding}). Only the @sc{ASCII} codes 0
226 through 127 are fully legitimate in both representations.
228 @defun char-valid-p charcode
229 This returns @code{t} if @var{charcode} is valid for either one of the two
230 text representations.
243 @section Character Sets
244 @cindex character sets
246 Emacs classifies characters into various @dfn{character sets}, each of
247 which has a name which is a symbol. Each character belongs to one and
248 only one character set.
250 In general, there is one character set for each distinct script. For
251 example, @code{latin-iso8859-1} is one character set,
252 @code{greek-iso8859-7} is another, and @code{ascii} is another. An
253 Emacs character set can hold at most 9025 characters; therefore, in some
254 cases, characters that would logically be grouped together are split
255 into several character sets. For example, one set of Chinese
256 characters, generally known as Big 5, is divided into two Emacs
257 character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
259 @defun charsetp object
261 Return @code{t} if @var{object} is a character set name symbol,
262 @code{nil} otherwise.
267 This function returns a list of all defined character set names.
270 @defun char-charset character
272 This function returns the name of the character
273 set that @var{character} belongs to.
276 @node Chars and Bytes
277 @section Characters and Bytes
278 @cindex bytes and characters
280 @cindex introduction sequence
281 @cindex dimension (of character set)
282 In multibyte representation, each character occupies one or more
283 bytes. Each character set has an @dfn{introduction sequence}, which is
284 normally one or two bytes long. (Exception: the @sc{ASCII} character
285 set has a zero-length introduction sequence.) The introduction sequence
286 is the beginning of the byte sequence for any character in the character
287 set. The rest of the character's bytes distinguish it from the other
288 characters in the same character set. Depending on the character set,
289 there are either one or two distinguishing bytes; the number of such
290 bytes is called the @dfn{dimension} of the character set.
292 @defun charset-dimension charset
293 @tindex charset-dimension
294 This function returns the dimension of @var{charset};
295 at present, the dimension is always 1 or 2.
298 This is the simplest way to determine the byte length of a character
299 set's introduction sequence:
302 (- (char-bytes (make-char @var{charset}))
303 (charset-dimension @var{charset}))
306 @node Splitting Characters
307 @section Splitting Characters
309 The functions in this section convert between characters and the byte
310 values used to represent them. For most purposes, there is no need to
311 be concerned with the sequence of bytes used to represent a character,
312 because Emacs translates automatically when necessary.
314 @defun char-bytes character
316 This function returns the number of bytes used to represent the
317 character @var{character}. This depends only on the character set that
318 @var{character} belongs to; it equals the dimension of that character
319 set (@pxref{Character Sets}), plus the length of its introduction
331 The reason this function can give correct results for both multibyte and
332 unibyte representations is that the non-@sc{ASCII} character codes used
333 in those two representations do not overlap.
336 @defun split-char character
338 Return a list containing the name of the character set of
339 @var{character}, followed by one or two byte values (integers) which
340 identify @var{character} within that character set. The number of byte
341 values is the character set's dimension.
345 @result{} (latin-iso8859-1 72)
350 Unibyte non-@sc{ASCII} characters are considered as part of
351 the @code{ascii} character set:
355 @result{} (ascii 192)
359 @defun make-char charset &rest byte-values
361 This function returns the character in character set @var{charset}
362 identified by @var{byte-values}. This is roughly the inverse of
363 @code{split-char}. Normally, you should specify either one or two
364 @var{byte-values}, according to the dimension of @var{charset}. For
368 (make-char 'latin-iso8859-1 72)
373 @cindex generic characters
374 If you call @code{make-char} with no @var{byte-values}, the result is
375 a @dfn{generic character} which stands for @var{charset}. A generic
376 character is an integer, but it is @emph{not} valid for insertion in the
377 buffer as a character. It can be used in @code{char-table-range} to
378 refer to the whole character set (@pxref{Char-Tables}).
379 @code{char-valid-p} returns @code{nil} for generic characters.
383 (make-char 'latin-iso8859-1)
388 @result{} (latin-iso8859-1 0)
391 @node Scanning Charsets
392 @section Scanning for Character Sets
394 Sometimes it is useful to find out which character sets appear in a
395 part of a buffer or a string. One use for this is in determining which
396 coding systems (@pxref{Coding Systems}) are capable of representing all
397 of the text in question.
399 @defun find-charset-region beg end &optional translation
400 @tindex find-charset-region
401 This function returns a list of the character sets that appear in the
402 current buffer between positions @var{beg} and @var{end}.
404 The optional argument @var{translation} specifies a translation table to
405 be used in scanning the text (@pxref{Translation of Characters}). If it
406 is non-@code{nil}, then each character in the region is translated
407 through this table, and the value returned describes the translated
408 characters instead of the characters actually in the buffer.
411 @defun find-charset-string string &optional translation
412 @tindex find-charset-string
413 This function returns a list of the character sets
414 that appear in the string @var{string}.
416 The optional argument @var{translation} specifies a
417 translation table; see @code{find-charset-region}, above.
420 @node Translation of Characters
421 @section Translation of Characters
422 @cindex character translation tables
423 @cindex translation tables
425 A @dfn{translation table} specifies a mapping of characters
426 into characters. These tables are used in encoding and decoding, and
427 for other purposes. Some coding systems specify their own particular
428 translation tables; there are also default translation tables which
429 apply to all other coding systems.
431 @defun make-translation-table translations
432 This function returns a translation table based on the arguments
433 @var{translations}. Each argument---each element of
434 @var{translations}---should be a list of the form @code{(@var{from}
435 . @var{to})}; this says to translate the character @var{from} into
438 You can also map one whole character set into another character set with
439 the same dimension. To do this, you specify a generic character (which
440 designates a character set) for @var{from} (@pxref{Splitting Characters}).
441 In this case, @var{to} should also be a generic character, for another
442 character set of the same dimension. Then the translation table
443 translates each character of @var{from}'s character set into the
444 corresponding character of @var{to}'s character set.
447 In decoding, the translation table's translations are applied to the
448 characters that result from ordinary decoding. If a coding system has
449 property @code{character-translation-table-for-decode}, that specifies
450 the translation table to use. Otherwise, if
451 @code{standard-character-translation-table-for-decode} is
452 non-@code{nil}, decoding uses that table.
454 In encoding, the translation table's translations are applied to the
455 characters in the buffer, and the result of translation is actually
456 encoded. If a coding system has property
457 @code{character-translation-table-for-encode}, that specifies the
458 translation table to use. Otherwise the variable
459 @code{standard-character-translation-table-for-encode} specifies the
462 @defvar standard-character-translation-table-for-decode
463 This is the default translation table for decoding, for
464 coding systems that don't specify any other translation table.
467 @defvar standard-character-translation-table-for-encode
468 This is the default translation table for encoding, for
469 coding systems that don't specify any other translation table.
473 @section Coding Systems
475 @cindex coding system
476 When Emacs reads or writes a file, and when Emacs sends text to a
477 subprocess or receives text from a subprocess, it normally performs
478 character code conversion and end-of-line conversion as specified
479 by a particular @dfn{coding system}.
482 * Coding System Basics::
484 * Lisp and Coding Systems::
485 * User-Chosen Coding Systems::
486 * Default Coding Systems::
487 * Specifying Coding Systems::
488 * Explicit Encoding::
489 * Terminal I/O Encoding::
490 * MS-DOS File Types::
493 @node Coding System Basics
494 @subsection Basic Concepts of Coding Systems
496 @cindex character code conversion
497 @dfn{Character code conversion} involves conversion between the encoding
498 used inside Emacs and some other encoding. Emacs supports many
499 different encodings, in that it can convert to and from them. For
500 example, it can convert text to or from encodings such as Latin 1, Latin
501 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some
502 cases, Emacs supports several alternative encodings for the same
503 characters; for example, there are three coding systems for the Cyrillic
504 (Russian) alphabet: ISO, Alternativnyj, and KOI8.
506 Most coding systems specify a particular character code for
507 conversion, but some of them leave this unspecified---to be chosen
508 heuristically based on the data.
510 @cindex end of line conversion
511 @dfn{End of line conversion} handles three different conventions used
512 on various systems for representing end of line in files. The Unix
513 convention is to use the linefeed character (also called newline). The
514 DOS convention is to use the two character sequence, carriage-return
515 linefeed, at the end of a line. The Mac convention is to use just
518 @cindex base coding system
519 @cindex variant coding system
520 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
521 conversion unspecified, to be chosen based on the data. @dfn{Variant
522 coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
523 @code{latin-1-mac} specify the end-of-line conversion explicitly as
524 well. Most base coding systems have three corresponding variants whose
525 names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
527 The coding system @code{raw-text} is special in that it prevents
528 character code conversion, and causes the buffer visited with that
529 coding system to be a unibyte buffer. It does not specify the
530 end-of-line conversion, allowing that to be determined as usual by the
531 data, and has the usual three variants which specify the end-of-line
532 conversion. @code{no-conversion} is equivalent to @code{raw-text-unix}:
533 it specifies no conversion of either character codes or end-of-line.
535 The coding system @code{emacs-mule} specifies that the data is
536 represented in the internal Emacs encoding. This is like
537 @code{raw-text} in that no code conversion happens, but different in
538 that the result is multibyte data.
540 @defun coding-system-get coding-system property
541 @tindex coding-system-get
542 This function returns the specified property of the coding system
543 @var{coding-system}. Most coding system properties exist for internal
544 purposes, but one that you might find useful is @code{mime-charset}.
545 That property's value is the name used in MIME for the character coding
546 which this coding system can read and write. Examples:
549 (coding-system-get 'iso-latin-1 'mime-charset)
551 (coding-system-get 'iso-2022-cn 'mime-charset)
552 @result{} iso-2022-cn
553 (coding-system-get 'cyrillic-koi8 'mime-charset)
557 The value of the @code{mime-charset} property is also defined
558 as an alias for the coding system.
561 @node Encoding and I/O
562 @subsection Encoding and I/O
564 The principal purpose of coding systems is for use in reading and
565 writing files. The function @code{insert-file-contents} uses
566 a coding system for decoding the file data, and @code{write-region}
567 uses one to encode the buffer contents.
569 You can specify the coding system to use either explicitly
570 (@pxref{Specifying Coding Systems}), or implicitly using the defaulting
571 mechanism (@pxref{Default Coding Systems}). But these methods may not
572 completely specify what to do. For example, they may choose a coding
573 system such as @code{undefined} which leaves the character code
574 conversion to be determined from the data. In these cases, the I/O
575 operation finishes the job of choosing a coding system. Very often
576 you will want to find out afterwards which coding system was chosen.
578 @defvar buffer-file-coding-system
579 @tindex buffer-file-coding-system
580 This variable records the coding system that was used for visiting the
581 current buffer. It is used for saving the buffer, and for writing part
582 of the buffer with @code{write-region}. When those operations ask the
583 user to specify a different coding system,
584 @code{buffer-file-coding-system} is updated to the coding system
588 @defvar save-buffer-coding-system
589 @tindex save-buffer-coding-system
590 This variable specifies the coding system for saving the buffer---but it
591 is not used for @code{write-region}. When saving the buffer asks the
592 user to specify a different coding system, and
593 @code{save-buffer-coding-system} was used, then it is updated to the
594 coding system that was specified.
597 @defvar last-coding-system-used
598 @tindex last-coding-system-used
599 I/O operations for files and subprocesses set this variable to the
600 coding system name that was used. The explicit encoding and decoding
601 functions (@pxref{Explicit Encoding}) set it too.
603 @strong{Warning:} Since receiving subprocess output sets this variable,
604 it can change whenever Emacs waits; therefore, you should use copy the
605 value shortly after the function call which stores the value you are
609 The variable @code{selection-coding-system} specifies how to encode
610 selections for the window system. @xref{Window System Selections}.
612 @node Lisp and Coding Systems
613 @subsection Coding Systems in Lisp
615 Here are Lisp facilities for working with coding systems;
617 @defun coding-system-list &optional base-only
618 @tindex coding-system-list
619 This function returns a list of all coding system names (symbols). If
620 @var{base-only} is non-@code{nil}, the value includes only the
621 base coding systems. Otherwise, it includes variant coding systems as well.
624 @defun coding-system-p object
625 @tindex coding-system-p
626 This function returns @code{t} if @var{object} is a coding system
630 @defun check-coding-system coding-system
631 @tindex check-coding-system
632 This function checks the validity of @var{coding-system}.
633 If that is valid, it returns @var{coding-system}.
634 Otherwise it signals an error with condition @code{coding-system-error}.
637 @defun coding-system-change-eol-conversion coding-system eol-type
638 @tindex coding-system-change-eol-conversion
639 This function returns a coding system which is like @var{coding-system}
640 except for its eol conversion, which is specified by @code{eol-type}.
641 @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
642 @code{nil}. If it is @code{nil}, the returned coding system determines
643 the end-of-line conversion from the data.
646 @defun coding-system-change-text-conversion eol-coding text-coding
647 @tindex coding-system-change-text-conversion
648 This function returns a coding system which uses the end-of-line
649 conversion of @var{eol-coding}, and the text conversion of
650 @var{text-coding}. If @var{text-coding} is @code{nil}, it returns
651 @code{undecided}, or one of its variants according to @var{eol-coding}.
654 @defun find-coding-systems-region from to
655 @tindex find-coding-systems-region
656 This function returns a list of coding systems that could be used to
657 encode a text between @var{from} and @var{to}. All coding systems in
658 the list can safely encode any multibyte characters in that portion of
661 If the text contains no multibyte characters, the function returns the
662 list @code{(undecided)}.
665 @defun find-coding-systems-string string
666 @tindex find-coding-systems-string
667 This function returns a list of coding systems that could be used to
668 encode the text of @var{string}. All coding systems in the list can
669 safely encode any multibyte characters in @var{string}. If the text
670 contains no multibyte characters, this returns the list
674 @defun find-coding-systems-for-charsets charsets
675 @tindex find-coding-systems-for-charsets
676 This function returns a list of coding systems that could be used to
677 encode all the character sets in the list @var{charsets}.
680 @defun detect-coding-region start end &optional highest
681 @tindex detect-coding-region
682 This function chooses a plausible coding system for decoding the text
683 from @var{start} to @var{end}. This text should be ``raw bytes''
684 (@pxref{Explicit Encoding}).
686 Normally this function returns a list of coding systems that could
687 handle decoding the text that was scanned. They are listed in order of
688 decreasing priority. But if @var{highest} is non-@code{nil}, then the
689 return value is just one coding system, the one that is highest in
692 If the region contains only @sc{ASCII} characters, the value
693 is @code{undecided} or @code{(undecided)}.
696 @defun detect-coding-string string highest
697 @tindex detect-coding-string
698 This function is like @code{detect-coding-region} except that it
699 operates on the contents of @var{string} instead of bytes in the buffer.
702 @xref{Process Information}, for how to examine or set the coding
703 systems used for I/O to a subprocess.
705 @node User-Chosen Coding Systems
706 @subsection User-Chosen Coding Systems
708 @tindex select-safe-coding-system
709 @defun select-safe-coding-system from to &optional preferred-coding-system
710 This function selects a coding system for encoding the text between
711 @var{from} and @var{to}, asking the user to choose if necessary.
713 The optional argument @var{preferred-coding-system} specifies a coding
714 system to try first. If that one can handle the text in the specified
715 region, then it is used. If this argument is omitted, the current
716 buffer's value of @code{buffer-file-coding-system} is tried first.
718 If the region contains some multibyte characters that the preferred
719 coding system cannot encode, this function asks the user to choose from
720 a list of coding systems which can encode the text, and returns the
723 One other kludgy feature: if @var{from} is a string, the string is the
724 target text, and @var{to} is ignored.
727 Here are two functions you can use to let the user specify a coding
728 system, with completion. @xref{Completion}.
730 @defun read-coding-system prompt &optional default
731 @tindex read-coding-system
732 This function reads a coding system using the minibuffer, prompting with
733 string @var{prompt}, and returns the coding system name as a symbol. If
734 the user enters null input, @var{default} specifies which coding system
735 to return. It should be a symbol or a string.
738 @defun read-non-nil-coding-system prompt
739 @tindex read-non-nil-coding-system
740 This function reads a coding system using the minibuffer, prompting with
741 string @var{prompt}, and returns the coding system name as a symbol. If
742 the user tries to enter null input, it asks the user to try again.
743 @xref{Coding Systems}.
746 @node Default Coding Systems
747 @subsection Default Coding Systems
749 This section describes variables that specify the default coding
750 system for certain files or when running certain subprograms, and the
751 function that I/O operations use to access them.
753 The idea of these variables is that you set them once and for all to the
754 defaults you want, and then do not change them again. To specify a
755 particular coding system for a particular operation in a Lisp program,
756 don't change these variables; instead, override them using
757 @code{coding-system-for-read} and @code{coding-system-for-write}
758 (@pxref{Specifying Coding Systems}).
760 @defvar file-coding-system-alist
761 @tindex file-coding-system-alist
762 This variable is an alist that specifies the coding systems to use for
763 reading and writing particular files. Each element has the form
764 @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
765 expression that matches certain file names. The element applies to file
766 names that match @var{pattern}.
768 The @sc{cdr} of the element, @var{coding}, should be either a coding
769 system, a cons cell containing two coding systems, or a function symbol.
770 If @var{val} is a coding system, that coding system is used for both
771 reading the file and writing it. If @var{val} is a cons cell containing
772 two coding systems, its @sc{car} specifies the coding system for
773 decoding, and its @sc{cdr} specifies the coding system for encoding.
775 If @var{val} is a function symbol, the function must return a coding
776 system or a cons cell containing two coding systems. This value is used
780 @defvar process-coding-system-alist
781 @tindex process-coding-system-alist
782 This variable is an alist specifying which coding systems to use for a
783 subprocess, depending on which program is running in the subprocess. It
784 works like @code{file-coding-system-alist}, except that @var{pattern} is
785 matched against the program name used to start the subprocess. The coding
786 system or systems specified in this alist are used to initialize the
787 coding systems used for I/O to the subprocess, but you can specify
788 other coding systems later using @code{set-process-coding-system}.
791 @strong{Warning:} Coding systems such as @code{undecided} which
792 determine the coding system from the data do not work entirely reliably
793 with asynchronous subprocess output. This is because Emacs handles
794 asynchronous subprocess output in batches, as it arrives. If the coding
795 system leaves the character code conversion unspecified, or leaves the
796 end-of-line conversion unspecified, Emacs must try to detect the proper
797 conversion from one batch at a time, and this does not always work.
799 Therefore, with an asynchronous subprocess, if at all possible, use a
800 coding system which determines both the character code conversion and
801 the end of line conversion---that is, one like @code{latin-1-unix},
802 rather than @code{undecided} or @code{latin-1}.
804 @defvar network-coding-system-alist
805 @tindex network-coding-system-alist
806 This variable is an alist that specifies the coding system to use for
807 network streams. It works much like @code{file-coding-system-alist},
808 with the difference that the @var{pattern} in an element may be either a
809 port number or a regular expression. If it is a regular expression, it
810 is matched against the network service name used to open the network
814 @defvar default-process-coding-system
815 @tindex default-process-coding-system
816 This variable specifies the coding systems to use for subprocess (and
817 network stream) input and output, when nothing else specifies what to
820 The value should be a cons cell of the form @code{(@var{input-coding}
821 . @var{output-coding})}. Here @var{input-coding} applies to input from
822 the subprocess, and @var{output-coding} applies to output to it.
825 @defun find-operation-coding-system operation &rest arguments
826 @tindex find-operation-coding-system
827 This function returns the coding system to use (by default) for
828 performing @var{operation} with @var{arguments}. The value has this
832 (@var{decoding-system} @var{encoding-system})
835 The first element, @var{decoding-system}, is the coding system to use
836 for decoding (in case @var{operation} does decoding), and
837 @var{encoding-system} is the coding system for encoding (in case
838 @var{operation} does encoding).
840 The argument @var{operation} should be an Emacs I/O primitive:
841 @code{insert-file-contents}, @code{write-region}, @code{call-process},
842 @code{call-process-region}, @code{start-process}, or
843 @code{open-network-stream}.
845 The remaining arguments should be the same arguments that might be given
846 to that I/O primitive. Depending on which primitive, one of those
847 arguments is selected as the @dfn{target}. For example, if
848 @var{operation} does file I/O, whichever argument specifies the file
849 name is the target. For subprocess primitives, the process name is the
850 target. For @code{open-network-stream}, the target is the service name
853 This function looks up the target in @code{file-coding-system-alist},
854 @code{process-coding-system-alist}, or
855 @code{network-coding-system-alist}, depending on @var{operation}.
856 @xref{Default Coding Systems}.
859 @node Specifying Coding Systems
860 @subsection Specifying a Coding System for One Operation
862 You can specify the coding system for a specific operation by binding
863 the variables @code{coding-system-for-read} and/or
864 @code{coding-system-for-write}.
866 @defvar coding-system-for-read
867 @tindex coding-system-for-read
868 If this variable is non-@code{nil}, it specifies the coding system to
869 use for reading a file, or for input from a synchronous subprocess.
871 It also applies to any asynchronous subprocess or network stream, but in
872 a different way: the value of @code{coding-system-for-read} when you
873 start the subprocess or open the network stream specifies the input
874 decoding method for that subprocess or network stream. It remains in
875 use for that subprocess or network stream unless and until overridden.
877 The right way to use this variable is to bind it with @code{let} for a
878 specific I/O operation. Its global value is normally @code{nil}, and
879 you should not globally set it to any other value. Here is an example
880 of the right way to use the variable:
883 ;; @r{Read the file with no character code conversion.}
884 ;; @r{Assume @sc{crlf} represents end-of-line.}
885 (let ((coding-system-for-write 'emacs-mule-dos))
886 (insert-file-contents filename))
889 When its value is non-@code{nil}, @code{coding-system-for-read} takes
890 precedence over all other methods of specifying a coding system to use for
891 input, including @code{file-coding-system-alist},
892 @code{process-coding-system-alist} and
893 @code{network-coding-system-alist}.
896 @defvar coding-system-for-write
897 @tindex coding-system-for-write
898 This works much like @code{coding-system-for-read}, except that it
899 applies to output rather than input. It affects writing to files,
900 subprocesses, and net connections.
902 When a single operation does both input and output, as do
903 @code{call-process-region} and @code{start-process}, both
904 @code{coding-system-for-read} and @code{coding-system-for-write}
908 @defvar inhibit-eol-conversion
909 @tindex inhibit-eol-conversion
910 When this variable is non-@code{nil}, no end-of-line conversion is done,
911 no matter which coding system is specified. This applies to all the
912 Emacs I/O and subprocess primitives, and to the explicit encoding and
913 decoding functions (@pxref{Explicit Encoding}).
916 @node Explicit Encoding
917 @subsection Explicit Encoding and Decoding
918 @cindex encoding text
919 @cindex decoding text
921 All the operations that transfer text in and out of Emacs have the
922 ability to use a coding system to encode or decode the text.
923 You can also explicitly encode and decode text using the functions
927 The result of encoding, and the input to decoding, are not ordinary
928 text. They are ``raw bytes''---bytes that represent text in the same
929 way that an external file would. When a buffer contains raw bytes, it
930 is most natural to mark that buffer as using unibyte representation,
931 using @code{set-buffer-multibyte} (@pxref{Selecting a Representation}),
932 but this is not required. If the buffer's contents are only temporarily
933 raw, leave the buffer multibyte, which will be correct after you decode
936 The usual way to get raw bytes in a buffer, for explicit decoding, is
937 to read them from a file with @code{insert-file-contents-literally}
938 (@pxref{Reading from Files}) or specify a non-@code{nil} @var{rawfile}
939 argument when visiting a file with @code{find-file-noselect}.
941 The usual way to use the raw bytes that result from explicitly
942 encoding text is to copy them to a file or process---for example, to
943 write them with @code{write-region} (@pxref{Writing to Files}), and
944 suppress encoding for that @code{write-region} call by binding
945 @code{coding-system-for-write} to @code{no-conversion}.
947 Raw bytes sometimes contain overlong byte-sequences that look like a
948 proper multibyte character plus extra bytes containing trailing codes.
949 For most purposes, Emacs treats such a sequence in a buffer or string as
950 a single character, and if you look at its character code, you get the
951 value that corresponds to the multibyte character sequence---the extra
952 bytes are disregarded. This behavior is not quite clean, but raw bytes
953 are used only in limited places in Emacs, so as a practical matter
954 problems can be avoided.
956 @defun encode-coding-region start end coding-system
957 @tindex encode-coding-region
958 This function encodes the text from @var{start} to @var{end} according
959 to coding system @var{coding-system}. The encoded text replaces the
960 original text in the buffer. The result of encoding is ``raw bytes,''
961 but the buffer remains multibyte if it was multibyte before.
964 @defun encode-coding-string string coding-system
965 @tindex encode-coding-string
966 This function encodes the text in @var{string} according to coding
967 system @var{coding-system}. It returns a new string containing the
968 encoded text. The result of encoding is a unibyte string of ``raw bytes.''
971 @defun decode-coding-region start end coding-system
972 @tindex decode-coding-region
973 This function decodes the text from @var{start} to @var{end} according
974 to coding system @var{coding-system}. The decoded text replaces the
975 original text in the buffer. To make explicit decoding useful, the text
976 before decoding ought to be ``raw bytes.''
979 @defun decode-coding-string string coding-system
980 @tindex decode-coding-string
981 This function decodes the text in @var{string} according to coding
982 system @var{coding-system}. It returns a new string containing the
983 decoded text. To make explicit decoding useful, the contents of
984 @var{string} ought to be ``raw bytes.''
987 @node Terminal I/O Encoding
988 @subsection Terminal I/O Encoding
990 Emacs can decode keyboard input using a coding system, and encode
991 terminal output. This is useful for terminals that transmit or display
992 text using a particular encoding such as Latin-1. Emacs does not set
993 @code{last-coding-system-used} for encoding or decoding for the
996 @defun keyboard-coding-system
997 @tindex keyboard-coding-system
998 This function returns the coding system that is in use for decoding
999 keyboard input---or @code{nil} if no coding system is to be used.
1002 @defun set-keyboard-coding-system coding-system
1003 @tindex set-keyboard-coding-system
1004 This function specifies @var{coding-system} as the coding system to
1005 use for decoding keyboard input. If @var{coding-system} is @code{nil},
1006 that means do not decode keyboard input.
1009 @defun terminal-coding-system
1010 @tindex terminal-coding-system
1011 This function returns the coding system that is in use for encoding
1012 terminal output---or @code{nil} for no encoding.
1015 @defun set-terminal-coding-system coding-system
1016 @tindex set-terminal-coding-system
1017 This function specifies @var{coding-system} as the coding system to use
1018 for encoding terminal output. If @var{coding-system} is @code{nil},
1019 that means do not encode terminal output.
1022 @node MS-DOS File Types
1023 @subsection MS-DOS File Types
1024 @cindex DOS file types
1025 @cindex MS-DOS file types
1026 @cindex Windows file types
1027 @cindex file types on MS-DOS and Windows
1028 @cindex text files and binary files
1029 @cindex binary files and text files
1031 Emacs on MS-DOS and on MS-Windows recognizes certain file names as
1032 text files or binary files. By ``binary file'' we mean a file of
1033 literal byte values that are not necessary meant to be characters.
1034 Emacs does no end-of-line conversion and no character code conversion
1035 for a binary file. Meanwhile, when you create a new file which is
1036 marked by its name as a ``text file'', Emacs uses DOS end-of-line
1039 @defvar buffer-file-type
1040 This variable, automatically buffer-local in each buffer, records the
1041 file type of the buffer's visited file. When a buffer does not specify
1042 a coding system with @code{buffer-file-coding-system}, this variable is
1043 used to determine which coding system to use when writing the contents
1044 of the buffer. It should be @code{nil} for text, @code{t} for binary.
1045 If it is @code{t}, the coding system is @code{no-conversion}.
1046 Otherwise, @code{undecided-dos} is used.
1048 Normally this variable is set by visiting a file; it is set to
1049 @code{nil} if the file was visited without any actual conversion.
1052 @defopt file-name-buffer-file-type-alist
1053 This variable holds an alist for recognizing text and binary files.
1054 Each element has the form (@var{regexp} . @var{type}), where
1055 @var{regexp} is matched against the file name, and @var{type} may be
1056 @code{nil} for text, @code{t} for binary, or a function to call to
1057 compute which. If it is a function, then it is called with a single
1058 argument (the file name) and should return @code{t} or @code{nil}.
1060 Emacs when running on MS-DOS or MS-Windows checks this alist to decide
1061 which coding system to use when reading a file. For a text file,
1062 @code{undecided-dos} is used. For a binary file, @code{no-conversion}
1065 If no element in this alist matches a given file name, then
1066 @code{default-buffer-file-type} says how to treat the file.
1069 @defopt default-buffer-file-type
1070 This variable says how to handle files for which
1071 @code{file-name-buffer-file-type-alist} says nothing about the type.
1073 If this variable is non-@code{nil}, then these files are treated as
1074 binary: the coding system @code{no-conversion} is used. Otherwise,
1075 nothing special is done for them---the coding system is deduced solely
1076 from the file contents, in the usual Emacs fashion.
1080 @section Input Methods
1081 @cindex input methods
1083 @dfn{Input methods} provide convenient ways of entering non-@sc{ASCII}
1084 characters from the keyboard. Unlike coding systems, which translate
1085 non-@sc{ASCII} characters to and from encodings meant to be read by
1086 programs, input methods provide human-friendly commands. (@xref{Input
1087 Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1088 use input methods to enter text.) How to define input methods is not
1089 yet documented in this manual, but here we describe how to use them.
1091 Each input method has a name, which is currently a string;
1092 in the future, symbols may also be usable as input method names.
1094 @tindex current-input-method
1095 @defvar current-input-method
1096 This variable holds the name of the input method now active in the
1097 current buffer. (It automatically becomes local in each buffer when set
1098 in any fashion.) It is @code{nil} if no input method is active in the
1102 @tindex default-input-method
1103 @defvar default-input-method
1104 This variable holds the default input method for commands that choose an
1105 input method. Unlike @code{current-input-method}, this variable is
1109 @tindex set-input-method
1110 @defun set-input-method input-method
1111 This function activates input method @var{input-method} for the current
1112 buffer. It also sets @code{default-input-method} to @var{input-method}.
1113 If @var{input-method} is @code{nil}, this function deactivates any input
1114 method for the current buffer.
1117 @tindex read-input-method-name
1118 @defun read-input-method-name prompt &optional default inhibit-null
1119 This function reads an input method name with the minibuffer, prompting
1120 with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1121 by default, if the user enters empty input. However, if
1122 @var{inhibit-null} is non-@code{nil}, empty input signals an error.
1124 The returned value is a string.
1127 @tindex input-method-alist
1128 @defvar input-method-alist
1129 This variable defines all the supported input methods.
1130 Each element defines one input method, and should have the form:
1133 (@var{input-method} @var{language-env} @var{activate-func}
1134 @var{title} @var{description} @var{args}...)
1137 Here @var{input-method} is the input method name, a string;
1138 @var{language-env} is another string, the name of the language
1139 environment this input method is recommended for. (That serves only for
1140 documentation purposes.)
1142 @var{title} is a string to display in the mode line while this method is
1143 active. @var{description} is a string describing this method and what
1146 @var{activate-func} is a function to call to activate this method. The
1147 @var{args}, if any, are passed as arguments to @var{activate-func}. All
1148 told, the arguments to @var{activate-func} are @var{input-method} and
1152 The fundamental interface to input methods is through the
1153 variable @code{input-method-function}. @xref{Reading One Event}.