Require button.el.
[emacs.git] / src / coding.c
blobf5b4f80868162611bafead86e6c06e1053eae50c
1 /* Coding system handler (conversion, detection, and etc).
2 Copyright (C) 1995, 1997, 1998, 2002 Electrotechnical Laboratory, JAPAN.
3 Licensed to the Free Software Foundation.
4 Copyright (C) 2001,2002 Free Software Foundation, Inc.
6 This file is part of GNU Emacs.
8 GNU Emacs is free software; you can redistribute it and/or modify
9 it under the terms of the GNU General Public License as published by
10 the Free Software Foundation; either version 2, or (at your option)
11 any later version.
13 GNU Emacs is distributed in the hope that it will be useful,
14 but WITHOUT ANY WARRANTY; without even the implied warranty of
15 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 GNU General Public License for more details.
18 You should have received a copy of the GNU General Public License
19 along with GNU Emacs; see the file COPYING. If not, write to
20 the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
21 Boston, MA 02111-1307, USA. */
23 /*** TABLE OF CONTENTS ***
25 0. General comments
26 1. Preamble
27 2. Emacs' internal format (emacs-mule) handlers
28 3. ISO2022 handlers
29 4. Shift-JIS and BIG5 handlers
30 5. CCL handlers
31 6. End-of-line handlers
32 7. C library functions
33 8. Emacs Lisp library functions
34 9. Post-amble
38 /*** 0. General comments ***/
41 /*** GENERAL NOTE on CODING SYSTEMS ***
43 A coding system is an encoding mechanism for one or more character
44 sets. Here's a list of coding systems which Emacs can handle. When
45 we say "decode", it means converting some other coding system to
46 Emacs' internal format (emacs-mule), and when we say "encode",
47 it means converting the coding system emacs-mule to some other
48 coding system.
50 0. Emacs' internal format (emacs-mule)
52 Emacs itself holds a multi-lingual character in buffers and strings
53 in a special format. Details are described in section 2.
55 1. ISO2022
57 The most famous coding system for multiple character sets. X's
58 Compound Text, various EUCs (Extended Unix Code), and coding
59 systems used in Internet communication such as ISO-2022-JP are
60 all variants of ISO2022. Details are described in section 3.
62 2. SJIS (or Shift-JIS or MS-Kanji-Code)
64 A coding system to encode character sets: ASCII, JISX0201, and
65 JISX0208. Widely used for PC's in Japan. Details are described in
66 section 4.
68 3. BIG5
70 A coding system to encode the character sets ASCII and Big5. Widely
71 used for Chinese (mainly in Taiwan and Hong Kong). Details are
72 described in section 4. In this file, when we write "BIG5"
73 (all uppercase), we mean the coding system, and when we write
74 "Big5" (capitalized), we mean the character set.
76 4. Raw text
78 A coding system for text containing random 8-bit code. Emacs does
79 no code conversion on such text except for end-of-line format.
81 5. Other
83 If a user wants to read/write text encoded in a coding system not
84 listed above, he can supply a decoder and an encoder for it as CCL
85 (Code Conversion Language) programs. Emacs executes the CCL program
86 while reading/writing.
88 Emacs represents a coding system by a Lisp symbol that has a property
89 `coding-system'. But, before actually using the coding system, the
90 information about it is set in a structure of type `struct
91 coding_system' for rapid processing. See section 6 for more details.
95 /*** GENERAL NOTES on END-OF-LINE FORMAT ***
97 How end-of-line of text is encoded depends on the operating system.
98 For instance, Unix's format is just one byte of `line-feed' code,
99 whereas DOS's format is two-byte sequence of `carriage-return' and
100 `line-feed' codes. MacOS's format is usually one byte of
101 `carriage-return'.
103 Since text character encoding and end-of-line encoding are
104 independent, any coding system described above can have any
105 end-of-line format. So Emacs has information about end-of-line
106 format in each coding-system. See section 6 for more details.
110 /*** GENERAL NOTES on `detect_coding_XXX ()' functions ***
112 These functions check if a text between SRC and SRC_END is encoded
113 in the coding system category XXX. Each returns an integer value in
114 which appropriate flag bits for the category XXX are set. The flag
115 bits are defined in macros CODING_CATEGORY_MASK_XXX. Below is the
116 template for these functions. If MULTIBYTEP is nonzero, 8-bit codes
117 of the range 0x80..0x9F are in multibyte form. */
118 #if 0
120 detect_coding_emacs_mule (src, src_end, multibytep)
121 unsigned char *src, *src_end;
122 int multibytep;
126 #endif
128 /*** GENERAL NOTES on `decode_coding_XXX ()' functions ***
130 These functions decode SRC_BYTES length of unibyte text at SOURCE
131 encoded in CODING to Emacs' internal format. The resulting
132 multibyte text goes to a place pointed to by DESTINATION, the length
133 of which should not exceed DST_BYTES.
135 These functions set the information about original and decoded texts
136 in the members `produced', `produced_char', `consumed', and
137 `consumed_char' of the structure *CODING. They also set the member
138 `result' to one of CODING_FINISH_XXX indicating how the decoding
139 finished.
141 DST_BYTES zero means that the source area and destination area are
142 overlapped, which means that we can produce a decoded text until it
143 reaches the head of the not-yet-decoded source text.
145 Below is a template for these functions. */
146 #if 0
147 static void
148 decode_coding_XXX (coding, source, destination, src_bytes, dst_bytes)
149 struct coding_system *coding;
150 unsigned char *source, *destination;
151 int src_bytes, dst_bytes;
155 #endif
157 /*** GENERAL NOTES on `encode_coding_XXX ()' functions ***
159 These functions encode SRC_BYTES length text at SOURCE from Emacs'
160 internal multibyte format to CODING. The resulting unibyte text
161 goes to a place pointed to by DESTINATION, the length of which
162 should not exceed DST_BYTES.
164 These functions set the information about original and encoded texts
165 in the members `produced', `produced_char', `consumed', and
166 `consumed_char' of the structure *CODING. They also set the member
167 `result' to one of CODING_FINISH_XXX indicating how the encoding
168 finished.
170 DST_BYTES zero means that the source area and destination area are
171 overlapped, which means that we can produce encoded text until it
172 reaches at the head of the not-yet-encoded source text.
174 Below is a template for these functions. */
175 #if 0
176 static void
177 encode_coding_XXX (coding, source, destination, src_bytes, dst_bytes)
178 struct coding_system *coding;
179 unsigned char *source, *destination;
180 int src_bytes, dst_bytes;
184 #endif
186 /*** COMMONLY USED MACROS ***/
188 /* The following two macros ONE_MORE_BYTE and TWO_MORE_BYTES safely
189 get one, two, and three bytes from the source text respectively.
190 If there are not enough bytes in the source, they jump to
191 `label_end_of_loop'. The caller should set variables `coding',
192 `src' and `src_end' to appropriate pointer in advance. These
193 macros are called from decoding routines `decode_coding_XXX', thus
194 it is assumed that the source text is unibyte. */
196 #define ONE_MORE_BYTE(c1) \
197 do { \
198 if (src >= src_end) \
200 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
201 goto label_end_of_loop; \
203 c1 = *src++; \
204 } while (0)
206 #define TWO_MORE_BYTES(c1, c2) \
207 do { \
208 if (src + 1 >= src_end) \
210 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
211 goto label_end_of_loop; \
213 c1 = *src++; \
214 c2 = *src++; \
215 } while (0)
218 /* Like ONE_MORE_BYTE, but 8-bit bytes of data at SRC are in multibyte
219 form if MULTIBYTEP is nonzero. */
221 #define ONE_MORE_BYTE_CHECK_MULTIBYTE(c1, multibytep) \
222 do { \
223 if (src >= src_end) \
225 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
226 goto label_end_of_loop; \
228 c1 = *src++; \
229 if (multibytep && c1 == LEADING_CODE_8_BIT_CONTROL) \
230 c1 = *src++ - 0x20; \
231 } while (0)
233 /* Set C to the next character at the source text pointed by `src'.
234 If there are not enough characters in the source, jump to
235 `label_end_of_loop'. The caller should set variables `coding'
236 `src', `src_end', and `translation_table' to appropriate pointers
237 in advance. This macro is used in encoding routines
238 `encode_coding_XXX', thus it assumes that the source text is in
239 multibyte form except for 8-bit characters. 8-bit characters are
240 in multibyte form if coding->src_multibyte is nonzero, else they
241 are represented by a single byte. */
243 #define ONE_MORE_CHAR(c) \
244 do { \
245 int len = src_end - src; \
246 int bytes; \
247 if (len <= 0) \
249 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
250 goto label_end_of_loop; \
252 if (coding->src_multibyte \
253 || UNIBYTE_STR_AS_MULTIBYTE_P (src, len, bytes)) \
254 c = STRING_CHAR_AND_LENGTH (src, len, bytes); \
255 else \
256 c = *src, bytes = 1; \
257 if (!NILP (translation_table)) \
258 c = translate_char (translation_table, c, -1, 0, 0); \
259 src += bytes; \
260 } while (0)
263 /* Produce a multibyte form of character C to `dst'. Jump to
264 `label_end_of_loop' if there's not enough space at `dst'.
266 If we are now in the middle of a composition sequence, the decoded
267 character may be ALTCHAR (for the current composition). In that
268 case, the character goes to coding->cmp_data->data instead of
269 `dst'.
271 This macro is used in decoding routines. */
273 #define EMIT_CHAR(c) \
274 do { \
275 if (! COMPOSING_P (coding) \
276 || coding->composing == COMPOSITION_RELATIVE \
277 || coding->composing == COMPOSITION_WITH_RULE) \
279 int bytes = CHAR_BYTES (c); \
280 if ((dst + bytes) > (dst_bytes ? dst_end : src)) \
282 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
283 goto label_end_of_loop; \
285 dst += CHAR_STRING (c, dst); \
286 coding->produced_char++; \
289 if (COMPOSING_P (coding) \
290 && coding->composing != COMPOSITION_RELATIVE) \
292 CODING_ADD_COMPOSITION_COMPONENT (coding, c); \
293 coding->composition_rule_follows \
294 = coding->composing != COMPOSITION_WITH_ALTCHARS; \
296 } while (0)
299 #define EMIT_ONE_BYTE(c) \
300 do { \
301 if (dst >= (dst_bytes ? dst_end : src)) \
303 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
304 goto label_end_of_loop; \
306 *dst++ = c; \
307 } while (0)
309 #define EMIT_TWO_BYTES(c1, c2) \
310 do { \
311 if (dst + 2 > (dst_bytes ? dst_end : src)) \
313 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
314 goto label_end_of_loop; \
316 *dst++ = c1, *dst++ = c2; \
317 } while (0)
319 #define EMIT_BYTES(from, to) \
320 do { \
321 if (dst + (to - from) > (dst_bytes ? dst_end : src)) \
323 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
324 goto label_end_of_loop; \
326 while (from < to) \
327 *dst++ = *from++; \
328 } while (0)
331 /*** 1. Preamble ***/
333 #ifdef emacs
334 #include <config.h>
335 #endif
337 #include <stdio.h>
339 #ifdef emacs
341 #include "lisp.h"
342 #include "buffer.h"
343 #include "charset.h"
344 #include "composite.h"
345 #include "ccl.h"
346 #include "coding.h"
347 #include "window.h"
349 #else /* not emacs */
351 #include "mulelib.h"
353 #endif /* not emacs */
355 Lisp_Object Qcoding_system, Qeol_type;
356 Lisp_Object Qbuffer_file_coding_system;
357 Lisp_Object Qpost_read_conversion, Qpre_write_conversion;
358 Lisp_Object Qno_conversion, Qundecided;
359 Lisp_Object Qcoding_system_history;
360 Lisp_Object Qsafe_chars;
361 Lisp_Object Qvalid_codes;
363 extern Lisp_Object Qinsert_file_contents, Qwrite_region;
364 Lisp_Object Qcall_process, Qcall_process_region, Qprocess_argument;
365 Lisp_Object Qstart_process, Qopen_network_stream;
366 Lisp_Object Qtarget_idx;
368 Lisp_Object Vselect_safe_coding_system_function;
370 int coding_system_require_warning;
372 /* Mnemonic string for each format of end-of-line. */
373 Lisp_Object eol_mnemonic_unix, eol_mnemonic_dos, eol_mnemonic_mac;
374 /* Mnemonic string to indicate format of end-of-line is not yet
375 decided. */
376 Lisp_Object eol_mnemonic_undecided;
378 /* Format of end-of-line decided by system. This is CODING_EOL_LF on
379 Unix, CODING_EOL_CRLF on DOS/Windows, and CODING_EOL_CR on Mac. */
380 int system_eol_type;
382 #ifdef emacs
384 /* Information about which coding system is safe for which chars.
385 The value has the form (GENERIC-LIST . NON-GENERIC-ALIST).
387 GENERIC-LIST is a list of generic coding systems which can encode
388 any characters.
390 NON-GENERIC-ALIST is an alist of non generic coding systems vs the
391 corresponding char table that contains safe chars. */
392 Lisp_Object Vcoding_system_safe_chars;
394 Lisp_Object Vcoding_system_list, Vcoding_system_alist;
396 Lisp_Object Qcoding_system_p, Qcoding_system_error;
398 /* Coding system emacs-mule and raw-text are for converting only
399 end-of-line format. */
400 Lisp_Object Qemacs_mule, Qraw_text;
402 /* Coding-systems are handed between Emacs Lisp programs and C internal
403 routines by the following three variables. */
404 /* Coding-system for reading files and receiving data from process. */
405 Lisp_Object Vcoding_system_for_read;
406 /* Coding-system for writing files and sending data to process. */
407 Lisp_Object Vcoding_system_for_write;
408 /* Coding-system actually used in the latest I/O. */
409 Lisp_Object Vlast_coding_system_used;
411 /* A vector of length 256 which contains information about special
412 Latin codes (especially for dealing with Microsoft codes). */
413 Lisp_Object Vlatin_extra_code_table;
415 /* Flag to inhibit code conversion of end-of-line format. */
416 int inhibit_eol_conversion;
418 /* Flag to inhibit ISO2022 escape sequence detection. */
419 int inhibit_iso_escape_detection;
421 /* Flag to make buffer-file-coding-system inherit from process-coding. */
422 int inherit_process_coding_system;
424 /* Coding system to be used to encode text for terminal display. */
425 struct coding_system terminal_coding;
427 /* Coding system to be used to encode text for terminal display when
428 terminal coding system is nil. */
429 struct coding_system safe_terminal_coding;
431 /* Coding system of what is sent from terminal keyboard. */
432 struct coding_system keyboard_coding;
434 /* Default coding system to be used to write a file. */
435 struct coding_system default_buffer_file_coding;
437 Lisp_Object Vfile_coding_system_alist;
438 Lisp_Object Vprocess_coding_system_alist;
439 Lisp_Object Vnetwork_coding_system_alist;
441 Lisp_Object Vlocale_coding_system;
443 #endif /* emacs */
445 Lisp_Object Qcoding_category, Qcoding_category_index;
447 /* List of symbols `coding-category-xxx' ordered by priority. */
448 Lisp_Object Vcoding_category_list;
450 /* Table of coding categories (Lisp symbols). */
451 Lisp_Object Vcoding_category_table;
453 /* Table of names of symbol for each coding-category. */
454 char *coding_category_name[CODING_CATEGORY_IDX_MAX] = {
455 "coding-category-emacs-mule",
456 "coding-category-sjis",
457 "coding-category-iso-7",
458 "coding-category-iso-7-tight",
459 "coding-category-iso-8-1",
460 "coding-category-iso-8-2",
461 "coding-category-iso-7-else",
462 "coding-category-iso-8-else",
463 "coding-category-ccl",
464 "coding-category-big5",
465 "coding-category-utf-8",
466 "coding-category-utf-16-be",
467 "coding-category-utf-16-le",
468 "coding-category-raw-text",
469 "coding-category-binary"
472 /* Table of pointers to coding systems corresponding to each coding
473 categories. */
474 struct coding_system *coding_system_table[CODING_CATEGORY_IDX_MAX];
476 /* Table of coding category masks. Nth element is a mask for a coding
477 category of which priority is Nth. */
478 static
479 int coding_priorities[CODING_CATEGORY_IDX_MAX];
481 /* Flag to tell if we look up translation table on character code
482 conversion. */
483 Lisp_Object Venable_character_translation;
484 /* Standard translation table to look up on decoding (reading). */
485 Lisp_Object Vstandard_translation_table_for_decode;
486 /* Standard translation table to look up on encoding (writing). */
487 Lisp_Object Vstandard_translation_table_for_encode;
489 Lisp_Object Qtranslation_table;
490 Lisp_Object Qtranslation_table_id;
491 Lisp_Object Qtranslation_table_for_decode;
492 Lisp_Object Qtranslation_table_for_encode;
494 /* Alist of charsets vs revision number. */
495 Lisp_Object Vcharset_revision_alist;
497 /* Default coding systems used for process I/O. */
498 Lisp_Object Vdefault_process_coding_system;
500 /* Char table for translating Quail and self-inserting input. */
501 Lisp_Object Vtranslation_table_for_input;
503 /* Global flag to tell that we can't call post-read-conversion and
504 pre-write-conversion functions. Usually the value is zero, but it
505 is set to 1 temporarily while such functions are running. This is
506 to avoid infinite recursive call. */
507 static int inhibit_pre_post_conversion;
509 Lisp_Object Qchar_coding_system;
511 /* Return `safe-chars' property of CODING_SYSTEM (symbol). Don't check
512 its validity. */
514 Lisp_Object
515 coding_safe_chars (coding_system)
516 Lisp_Object coding_system;
518 Lisp_Object coding_spec, plist, safe_chars;
520 coding_spec = Fget (coding_system, Qcoding_system);
521 plist = XVECTOR (coding_spec)->contents[3];
522 safe_chars = Fplist_get (XVECTOR (coding_spec)->contents[3], Qsafe_chars);
523 return (CHAR_TABLE_P (safe_chars) ? safe_chars : Qt);
526 #define CODING_SAFE_CHAR_P(safe_chars, c) \
527 (EQ (safe_chars, Qt) || !NILP (CHAR_TABLE_REF (safe_chars, c)))
530 /*** 2. Emacs internal format (emacs-mule) handlers ***/
532 /* Emacs' internal format for representation of multiple character
533 sets is a kind of multi-byte encoding, i.e. characters are
534 represented by variable-length sequences of one-byte codes.
536 ASCII characters and control characters (e.g. `tab', `newline') are
537 represented by one-byte sequences which are their ASCII codes, in
538 the range 0x00 through 0x7F.
540 8-bit characters of the range 0x80..0x9F are represented by
541 two-byte sequences of LEADING_CODE_8_BIT_CONTROL and (their 8-bit
542 code + 0x20).
544 8-bit characters of the range 0xA0..0xFF are represented by
545 one-byte sequences which are their 8-bit code.
547 The other characters are represented by a sequence of `base
548 leading-code', optional `extended leading-code', and one or two
549 `position-code's. The length of the sequence is determined by the
550 base leading-code. Leading-code takes the range 0x81 through 0x9D,
551 whereas extended leading-code and position-code take the range 0xA0
552 through 0xFF. See `charset.h' for more details about leading-code
553 and position-code.
555 --- CODE RANGE of Emacs' internal format ---
556 character set range
557 ------------- -----
558 ascii 0x00..0x7F
559 eight-bit-control LEADING_CODE_8_BIT_CONTROL + 0xA0..0xBF
560 eight-bit-graphic 0xA0..0xBF
561 ELSE 0x81..0x9D + [0xA0..0xFF]+
562 ---------------------------------------------
564 As this is the internal character representation, the format is
565 usually not used externally (i.e. in a file or in a data sent to a
566 process). But, it is possible to have a text externally in this
567 format (i.e. by encoding by the coding system `emacs-mule').
569 In that case, a sequence of one-byte codes has a slightly different
570 form.
572 Firstly, all characters in eight-bit-control are represented by
573 one-byte sequences which are their 8-bit code.
575 Next, character composition data are represented by the byte
576 sequence of the form: 0x80 METHOD BYTES CHARS COMPONENT ...,
577 where,
578 METHOD is 0xF0 plus one of composition method (enum
579 composition_method),
581 BYTES is 0xA0 plus the byte length of these composition data,
583 CHARS is 0xA0 plus the number of characters composed by these
584 data,
586 COMPONENTs are characters of multibyte form or composition
587 rules encoded by two-byte of ASCII codes.
589 In addition, for backward compatibility, the following formats are
590 also recognized as composition data on decoding.
592 0x80 MSEQ ...
593 0x80 0xFF MSEQ RULE MSEQ RULE ... MSEQ
595 Here,
596 MSEQ is a multibyte form but in these special format:
597 ASCII: 0xA0 ASCII_CODE+0x80,
598 other: LEADING_CODE+0x20 FOLLOWING-BYTE ...,
599 RULE is a one byte code of the range 0xA0..0xF0 that
600 represents a composition rule.
603 enum emacs_code_class_type emacs_code_class[256];
605 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
606 Check if a text is encoded in Emacs' internal format. If it is,
607 return CODING_CATEGORY_MASK_EMACS_MULE, else return 0. */
609 static int
610 detect_coding_emacs_mule (src, src_end, multibytep)
611 unsigned char *src, *src_end;
612 int multibytep;
614 unsigned char c;
615 int composing = 0;
616 /* Dummy for ONE_MORE_BYTE. */
617 struct coding_system dummy_coding;
618 struct coding_system *coding = &dummy_coding;
620 while (1)
622 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
624 if (composing)
626 if (c < 0xA0)
627 composing = 0;
628 else if (c == 0xA0)
630 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
631 c &= 0x7F;
633 else
634 c -= 0x20;
637 if (c < 0x20)
639 if (c == ISO_CODE_ESC || c == ISO_CODE_SI || c == ISO_CODE_SO)
640 return 0;
642 else if (c >= 0x80 && c < 0xA0)
644 if (c == 0x80)
645 /* Old leading code for a composite character. */
646 composing = 1;
647 else
649 unsigned char *src_base = src - 1;
650 int bytes;
652 if (!UNIBYTE_STR_AS_MULTIBYTE_P (src_base, src_end - src_base,
653 bytes))
654 return 0;
655 src = src_base + bytes;
659 label_end_of_loop:
660 return CODING_CATEGORY_MASK_EMACS_MULE;
664 /* Record the starting position START and METHOD of one composition. */
666 #define CODING_ADD_COMPOSITION_START(coding, start, method) \
667 do { \
668 struct composition_data *cmp_data = coding->cmp_data; \
669 int *data = cmp_data->data + cmp_data->used; \
670 coding->cmp_data_start = cmp_data->used; \
671 data[0] = -1; \
672 data[1] = cmp_data->char_offset + start; \
673 data[3] = (int) method; \
674 cmp_data->used += 4; \
675 } while (0)
677 /* Record the ending position END of the current composition. */
679 #define CODING_ADD_COMPOSITION_END(coding, end) \
680 do { \
681 struct composition_data *cmp_data = coding->cmp_data; \
682 int *data = cmp_data->data + coding->cmp_data_start; \
683 data[0] = cmp_data->used - coding->cmp_data_start; \
684 data[2] = cmp_data->char_offset + end; \
685 } while (0)
687 /* Record one COMPONENT (alternate character or composition rule). */
689 #define CODING_ADD_COMPOSITION_COMPONENT(coding, component) \
690 do { \
691 coding->cmp_data->data[coding->cmp_data->used++] = component; \
692 if (coding->cmp_data->used - coding->cmp_data_start \
693 == COMPOSITION_DATA_MAX_BUNCH_LENGTH) \
695 CODING_ADD_COMPOSITION_END (coding, coding->produced_char); \
696 coding->composing = COMPOSITION_NO; \
698 } while (0)
701 /* Get one byte from a data pointed by SRC and increment SRC. If SRC
702 is not less than SRC_END, return -1 without incrementing Src. */
704 #define SAFE_ONE_MORE_BYTE() (src >= src_end ? -1 : *src++)
707 /* Decode a character represented as a component of composition
708 sequence of Emacs 20 style at SRC. Set C to that character, store
709 its multibyte form sequence at P, and set P to the end of that
710 sequence. If no valid character is found, set C to -1. */
712 #define DECODE_EMACS_MULE_COMPOSITION_CHAR(c, p) \
713 do { \
714 int bytes; \
716 c = SAFE_ONE_MORE_BYTE (); \
717 if (c < 0) \
718 break; \
719 if (CHAR_HEAD_P (c)) \
720 c = -1; \
721 else if (c == 0xA0) \
723 c = SAFE_ONE_MORE_BYTE (); \
724 if (c < 0xA0) \
725 c = -1; \
726 else \
728 c -= 0xA0; \
729 *p++ = c; \
732 else if (BASE_LEADING_CODE_P (c - 0x20)) \
734 unsigned char *p0 = p; \
736 c -= 0x20; \
737 *p++ = c; \
738 bytes = BYTES_BY_CHAR_HEAD (c); \
739 while (--bytes) \
741 c = SAFE_ONE_MORE_BYTE (); \
742 if (c < 0) \
743 break; \
744 *p++ = c; \
746 if (UNIBYTE_STR_AS_MULTIBYTE_P (p0, p - p0, bytes)) \
747 c = STRING_CHAR (p0, bytes); \
748 else \
749 c = -1; \
751 else \
752 c = -1; \
753 } while (0)
756 /* Decode a composition rule represented as a component of composition
757 sequence of Emacs 20 style at SRC. Set C to the rule. If not
758 valid rule is found, set C to -1. */
760 #define DECODE_EMACS_MULE_COMPOSITION_RULE(c) \
761 do { \
762 c = SAFE_ONE_MORE_BYTE (); \
763 c -= 0xA0; \
764 if (c < 0 || c >= 81) \
765 c = -1; \
766 else \
768 gref = c / 9, nref = c % 9; \
769 c = COMPOSITION_ENCODE_RULE (gref, nref); \
771 } while (0)
774 /* Decode composition sequence encoded by `emacs-mule' at the source
775 pointed by SRC. SRC_END is the end of source. Store information
776 of the composition in CODING->cmp_data.
778 For backward compatibility, decode also a composition sequence of
779 Emacs 20 style. In that case, the composition sequence contains
780 characters that should be extracted into a buffer or string. Store
781 those characters at *DESTINATION in multibyte form.
783 If we encounter an invalid byte sequence, return 0.
784 If we encounter an insufficient source or destination, or
785 insufficient space in CODING->cmp_data, return 1.
786 Otherwise, return consumed bytes in the source.
789 static INLINE int
790 decode_composition_emacs_mule (coding, src, src_end,
791 destination, dst_end, dst_bytes)
792 struct coding_system *coding;
793 unsigned char *src, *src_end, **destination, *dst_end;
794 int dst_bytes;
796 unsigned char *dst = *destination;
797 int method, data_len, nchars;
798 unsigned char *src_base = src++;
799 /* Store components of composition. */
800 int component[COMPOSITION_DATA_MAX_BUNCH_LENGTH];
801 int ncomponent;
802 /* Store multibyte form of characters to be composed. This is for
803 Emacs 20 style composition sequence. */
804 unsigned char buf[MAX_COMPOSITION_COMPONENTS * MAX_MULTIBYTE_LENGTH];
805 unsigned char *bufp = buf;
806 int c, i, gref, nref;
808 if (coding->cmp_data->used + COMPOSITION_DATA_MAX_BUNCH_LENGTH
809 >= COMPOSITION_DATA_SIZE)
811 coding->result = CODING_FINISH_INSUFFICIENT_CMP;
812 return -1;
815 ONE_MORE_BYTE (c);
816 if (c - 0xF0 >= COMPOSITION_RELATIVE
817 && c - 0xF0 <= COMPOSITION_WITH_RULE_ALTCHARS)
819 int with_rule;
821 method = c - 0xF0;
822 with_rule = (method == COMPOSITION_WITH_RULE
823 || method == COMPOSITION_WITH_RULE_ALTCHARS);
824 ONE_MORE_BYTE (c);
825 data_len = c - 0xA0;
826 if (data_len < 4
827 || src_base + data_len > src_end)
828 return 0;
829 ONE_MORE_BYTE (c);
830 nchars = c - 0xA0;
831 if (c < 1)
832 return 0;
833 for (ncomponent = 0; src < src_base + data_len; ncomponent++)
835 /* If it is longer than this, it can't be valid. */
836 if (ncomponent >= COMPOSITION_DATA_MAX_BUNCH_LENGTH)
837 return 0;
839 if (ncomponent % 2 && with_rule)
841 ONE_MORE_BYTE (gref);
842 gref -= 32;
843 ONE_MORE_BYTE (nref);
844 nref -= 32;
845 c = COMPOSITION_ENCODE_RULE (gref, nref);
847 else
849 int bytes;
850 if (UNIBYTE_STR_AS_MULTIBYTE_P (src, src_end - src, bytes))
851 c = STRING_CHAR (src, bytes);
852 else
853 c = *src, bytes = 1;
854 src += bytes;
856 component[ncomponent] = c;
859 else
861 /* This may be an old Emacs 20 style format. See the comment at
862 the section 2 of this file. */
863 while (src < src_end && !CHAR_HEAD_P (*src)) src++;
864 if (src == src_end
865 && !(coding->mode & CODING_MODE_LAST_BLOCK))
866 goto label_end_of_loop;
868 src_end = src;
869 src = src_base + 1;
870 if (c < 0xC0)
872 method = COMPOSITION_RELATIVE;
873 for (ncomponent = 0; ncomponent < MAX_COMPOSITION_COMPONENTS;)
875 DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
876 if (c < 0)
877 break;
878 component[ncomponent++] = c;
880 if (ncomponent < 2)
881 return 0;
882 nchars = ncomponent;
884 else if (c == 0xFF)
886 method = COMPOSITION_WITH_RULE;
887 src++;
888 DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
889 if (c < 0)
890 return 0;
891 component[0] = c;
892 for (ncomponent = 1;
893 ncomponent < MAX_COMPOSITION_COMPONENTS * 2 - 1;)
895 DECODE_EMACS_MULE_COMPOSITION_RULE (c);
896 if (c < 0)
897 break;
898 component[ncomponent++] = c;
899 DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
900 if (c < 0)
901 break;
902 component[ncomponent++] = c;
904 if (ncomponent < 3)
905 return 0;
906 nchars = (ncomponent + 1) / 2;
908 else
909 return 0;
912 if (buf == bufp || dst + (bufp - buf) <= (dst_bytes ? dst_end : src))
914 CODING_ADD_COMPOSITION_START (coding, coding->produced_char, method);
915 for (i = 0; i < ncomponent; i++)
916 CODING_ADD_COMPOSITION_COMPONENT (coding, component[i]);
917 CODING_ADD_COMPOSITION_END (coding, coding->produced_char + nchars);
918 if (buf < bufp)
920 unsigned char *p = buf;
921 EMIT_BYTES (p, bufp);
922 *destination += bufp - buf;
923 coding->produced_char += nchars;
925 return (src - src_base);
927 label_end_of_loop:
928 return -1;
931 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
933 static void
934 decode_coding_emacs_mule (coding, source, destination, src_bytes, dst_bytes)
935 struct coding_system *coding;
936 unsigned char *source, *destination;
937 int src_bytes, dst_bytes;
939 unsigned char *src = source;
940 unsigned char *src_end = source + src_bytes;
941 unsigned char *dst = destination;
942 unsigned char *dst_end = destination + dst_bytes;
943 /* SRC_BASE remembers the start position in source in each loop.
944 The loop will be exited when there's not enough source code, or
945 when there's not enough destination area to produce a
946 character. */
947 unsigned char *src_base;
949 coding->produced_char = 0;
950 while ((src_base = src) < src_end)
952 unsigned char tmp[MAX_MULTIBYTE_LENGTH], *p;
953 int bytes;
955 if (*src == '\r')
957 int c = *src++;
959 if (coding->eol_type == CODING_EOL_CR)
960 c = '\n';
961 else if (coding->eol_type == CODING_EOL_CRLF)
963 ONE_MORE_BYTE (c);
964 if (c != '\n')
966 src--;
967 c = '\r';
970 *dst++ = c;
971 coding->produced_char++;
972 continue;
974 else if (*src == '\n')
976 if ((coding->eol_type == CODING_EOL_CR
977 || coding->eol_type == CODING_EOL_CRLF)
978 && coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
980 coding->result = CODING_FINISH_INCONSISTENT_EOL;
981 goto label_end_of_loop;
983 *dst++ = *src++;
984 coding->produced_char++;
985 continue;
987 else if (*src == 0x80 && coding->cmp_data)
989 /* Start of composition data. */
990 int consumed = decode_composition_emacs_mule (coding, src, src_end,
991 &dst, dst_end,
992 dst_bytes);
993 if (consumed < 0)
994 goto label_end_of_loop;
995 else if (consumed > 0)
997 src += consumed;
998 continue;
1000 bytes = CHAR_STRING (*src, tmp);
1001 p = tmp;
1002 src++;
1004 else if (UNIBYTE_STR_AS_MULTIBYTE_P (src, src_end - src, bytes))
1006 p = src;
1007 src += bytes;
1009 else
1011 bytes = CHAR_STRING (*src, tmp);
1012 p = tmp;
1013 src++;
1015 if (dst + bytes >= (dst_bytes ? dst_end : src))
1017 coding->result = CODING_FINISH_INSUFFICIENT_DST;
1018 break;
1020 while (bytes--) *dst++ = *p++;
1021 coding->produced_char++;
1023 label_end_of_loop:
1024 coding->consumed = coding->consumed_char = src_base - source;
1025 coding->produced = dst - destination;
1029 /* Encode composition data stored at DATA into a special byte sequence
1030 starting by 0x80. Update CODING->cmp_data_start and maybe
1031 CODING->cmp_data for the next call. */
1033 #define ENCODE_COMPOSITION_EMACS_MULE(coding, data) \
1034 do { \
1035 unsigned char buf[1024], *p0 = buf, *p; \
1036 int len = data[0]; \
1037 int i; \
1039 buf[0] = 0x80; \
1040 buf[1] = 0xF0 + data[3]; /* METHOD */ \
1041 buf[3] = 0xA0 + (data[2] - data[1]); /* COMPOSED-CHARS */ \
1042 p = buf + 4; \
1043 if (data[3] == COMPOSITION_WITH_RULE \
1044 || data[3] == COMPOSITION_WITH_RULE_ALTCHARS) \
1046 p += CHAR_STRING (data[4], p); \
1047 for (i = 5; i < len; i += 2) \
1049 int gref, nref; \
1050 COMPOSITION_DECODE_RULE (data[i], gref, nref); \
1051 *p++ = 0x20 + gref; \
1052 *p++ = 0x20 + nref; \
1053 p += CHAR_STRING (data[i + 1], p); \
1056 else \
1058 for (i = 4; i < len; i++) \
1059 p += CHAR_STRING (data[i], p); \
1061 buf[2] = 0xA0 + (p - buf); /* COMPONENTS-BYTES */ \
1063 if (dst + (p - buf) + 4 > (dst_bytes ? dst_end : src)) \
1065 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
1066 goto label_end_of_loop; \
1068 while (p0 < p) \
1069 *dst++ = *p0++; \
1070 coding->cmp_data_start += data[0]; \
1071 if (coding->cmp_data_start == coding->cmp_data->used \
1072 && coding->cmp_data->next) \
1074 coding->cmp_data = coding->cmp_data->next; \
1075 coding->cmp_data_start = 0; \
1077 } while (0)
1080 static void encode_eol P_ ((struct coding_system *, const unsigned char *,
1081 unsigned char *, int, int));
1083 static void
1084 encode_coding_emacs_mule (coding, source, destination, src_bytes, dst_bytes)
1085 struct coding_system *coding;
1086 unsigned char *source, *destination;
1087 int src_bytes, dst_bytes;
1089 unsigned char *src = source;
1090 unsigned char *src_end = source + src_bytes;
1091 unsigned char *dst = destination;
1092 unsigned char *dst_end = destination + dst_bytes;
1093 unsigned char *src_base;
1094 int c;
1095 int char_offset;
1096 int *data;
1098 Lisp_Object translation_table;
1100 translation_table = Qnil;
1102 /* Optimization for the case that there's no composition. */
1103 if (!coding->cmp_data || coding->cmp_data->used == 0)
1105 encode_eol (coding, source, destination, src_bytes, dst_bytes);
1106 return;
1109 char_offset = coding->cmp_data->char_offset;
1110 data = coding->cmp_data->data + coding->cmp_data_start;
1111 while (1)
1113 src_base = src;
1115 /* If SRC starts a composition, encode the information about the
1116 composition in advance. */
1117 if (coding->cmp_data_start < coding->cmp_data->used
1118 && char_offset + coding->consumed_char == data[1])
1120 ENCODE_COMPOSITION_EMACS_MULE (coding, data);
1121 char_offset = coding->cmp_data->char_offset;
1122 data = coding->cmp_data->data + coding->cmp_data_start;
1125 ONE_MORE_CHAR (c);
1126 if (c == '\n' && (coding->eol_type == CODING_EOL_CRLF
1127 || coding->eol_type == CODING_EOL_CR))
1129 if (coding->eol_type == CODING_EOL_CRLF)
1130 EMIT_TWO_BYTES ('\r', c);
1131 else
1132 EMIT_ONE_BYTE ('\r');
1134 else if (SINGLE_BYTE_CHAR_P (c))
1135 EMIT_ONE_BYTE (c);
1136 else
1137 EMIT_BYTES (src_base, src);
1138 coding->consumed_char++;
1140 label_end_of_loop:
1141 coding->consumed = src_base - source;
1142 coding->produced = coding->produced_char = dst - destination;
1143 return;
1147 /*** 3. ISO2022 handlers ***/
1149 /* The following note describes the coding system ISO2022 briefly.
1150 Since the intention of this note is to help understand the
1151 functions in this file, some parts are NOT ACCURATE or are OVERLY
1152 SIMPLIFIED. For thorough understanding, please refer to the
1153 original document of ISO2022. This is equivalent to the standard
1154 ECMA-35, obtainable from <URL:http://www.ecma.ch/> (*).
1156 ISO2022 provides many mechanisms to encode several character sets
1157 in 7-bit and 8-bit environments. For 7-bit environments, all text
1158 is encoded using bytes less than 128. This may make the encoded
1159 text a little bit longer, but the text passes more easily through
1160 several types of gateway, some of which strip off the MSB (Most
1161 Significant Bit).
1163 There are two kinds of character sets: control character sets and
1164 graphic character sets. The former contain control characters such
1165 as `newline' and `escape' to provide control functions (control
1166 functions are also provided by escape sequences). The latter
1167 contain graphic characters such as 'A' and '-'. Emacs recognizes
1168 two control character sets and many graphic character sets.
1170 Graphic character sets are classified into one of the following
1171 four classes, according to the number of bytes (DIMENSION) and
1172 number of characters in one dimension (CHARS) of the set:
1173 - DIMENSION1_CHARS94
1174 - DIMENSION1_CHARS96
1175 - DIMENSION2_CHARS94
1176 - DIMENSION2_CHARS96
1178 In addition, each character set is assigned an identification tag,
1179 unique for each set, called the "final character" (denoted as <F>
1180 hereafter). The <F> of each character set is decided by ECMA(*)
1181 when it is registered in ISO. The code range of <F> is 0x30..0x7F
1182 (0x30..0x3F are for private use only).
1184 Note (*): ECMA = European Computer Manufacturers Association
1186 Here are examples of graphic character sets [NAME(<F>)]:
1187 o DIMENSION1_CHARS94 -- ASCII('B'), right-half-of-JISX0201('I'), ...
1188 o DIMENSION1_CHARS96 -- right-half-of-ISO8859-1('A'), ...
1189 o DIMENSION2_CHARS94 -- GB2312('A'), JISX0208('B'), ...
1190 o DIMENSION2_CHARS96 -- none for the moment
1192 A code area (1 byte=8 bits) is divided into 4 areas, C0, GL, C1, and GR.
1193 C0 [0x00..0x1F] -- control character plane 0
1194 GL [0x20..0x7F] -- graphic character plane 0
1195 C1 [0x80..0x9F] -- control character plane 1
1196 GR [0xA0..0xFF] -- graphic character plane 1
1198 A control character set is directly designated and invoked to C0 or
1199 C1 by an escape sequence. The most common case is that:
1200 - ISO646's control character set is designated/invoked to C0, and
1201 - ISO6429's control character set is designated/invoked to C1,
1202 and usually these designations/invocations are omitted in encoded
1203 text. In a 7-bit environment, only C0 can be used, and a control
1204 character for C1 is encoded by an appropriate escape sequence to
1205 fit into the environment. All control characters for C1 are
1206 defined to have corresponding escape sequences.
1208 A graphic character set is at first designated to one of four
1209 graphic registers (G0 through G3), then these graphic registers are
1210 invoked to GL or GR. These designations and invocations can be
1211 done independently. The most common case is that G0 is invoked to
1212 GL, G1 is invoked to GR, and ASCII is designated to G0. Usually
1213 these invocations and designations are omitted in encoded text.
1214 In a 7-bit environment, only GL can be used.
1216 When a graphic character set of CHARS94 is invoked to GL, codes
1217 0x20 and 0x7F of the GL area work as control characters SPACE and
1218 DEL respectively, and codes 0xA0 and 0xFF of the GR area should not
1219 be used.
1221 There are two ways of invocation: locking-shift and single-shift.
1222 With locking-shift, the invocation lasts until the next different
1223 invocation, whereas with single-shift, the invocation affects the
1224 following character only and doesn't affect the locking-shift
1225 state. Invocations are done by the following control characters or
1226 escape sequences:
1228 ----------------------------------------------------------------------
1229 abbrev function cntrl escape seq description
1230 ----------------------------------------------------------------------
1231 SI/LS0 (shift-in) 0x0F none invoke G0 into GL
1232 SO/LS1 (shift-out) 0x0E none invoke G1 into GL
1233 LS2 (locking-shift-2) none ESC 'n' invoke G2 into GL
1234 LS3 (locking-shift-3) none ESC 'o' invoke G3 into GL
1235 LS1R (locking-shift-1 right) none ESC '~' invoke G1 into GR (*)
1236 LS2R (locking-shift-2 right) none ESC '}' invoke G2 into GR (*)
1237 LS3R (locking-shift 3 right) none ESC '|' invoke G3 into GR (*)
1238 SS2 (single-shift-2) 0x8E ESC 'N' invoke G2 for one char
1239 SS3 (single-shift-3) 0x8F ESC 'O' invoke G3 for one char
1240 ----------------------------------------------------------------------
1241 (*) These are not used by any known coding system.
1243 Control characters for these functions are defined by macros
1244 ISO_CODE_XXX in `coding.h'.
1246 Designations are done by the following escape sequences:
1247 ----------------------------------------------------------------------
1248 escape sequence description
1249 ----------------------------------------------------------------------
1250 ESC '(' <F> designate DIMENSION1_CHARS94<F> to G0
1251 ESC ')' <F> designate DIMENSION1_CHARS94<F> to G1
1252 ESC '*' <F> designate DIMENSION1_CHARS94<F> to G2
1253 ESC '+' <F> designate DIMENSION1_CHARS94<F> to G3
1254 ESC ',' <F> designate DIMENSION1_CHARS96<F> to G0 (*)
1255 ESC '-' <F> designate DIMENSION1_CHARS96<F> to G1
1256 ESC '.' <F> designate DIMENSION1_CHARS96<F> to G2
1257 ESC '/' <F> designate DIMENSION1_CHARS96<F> to G3
1258 ESC '$' '(' <F> designate DIMENSION2_CHARS94<F> to G0 (**)
1259 ESC '$' ')' <F> designate DIMENSION2_CHARS94<F> to G1
1260 ESC '$' '*' <F> designate DIMENSION2_CHARS94<F> to G2
1261 ESC '$' '+' <F> designate DIMENSION2_CHARS94<F> to G3
1262 ESC '$' ',' <F> designate DIMENSION2_CHARS96<F> to G0 (*)
1263 ESC '$' '-' <F> designate DIMENSION2_CHARS96<F> to G1
1264 ESC '$' '.' <F> designate DIMENSION2_CHARS96<F> to G2
1265 ESC '$' '/' <F> designate DIMENSION2_CHARS96<F> to G3
1266 ----------------------------------------------------------------------
1268 In this list, "DIMENSION1_CHARS94<F>" means a graphic character set
1269 of dimension 1, chars 94, and final character <F>, etc...
1271 Note (*): Although these designations are not allowed in ISO2022,
1272 Emacs accepts them on decoding, and produces them on encoding
1273 CHARS96 character sets in a coding system which is characterized as
1274 7-bit environment, non-locking-shift, and non-single-shift.
1276 Note (**): If <F> is '@', 'A', or 'B', the intermediate character
1277 '(' can be omitted. We refer to this as "short-form" hereafter.
1279 Now you may notice that there are a lot of ways of encoding the
1280 same multilingual text in ISO2022. Actually, there exist many
1281 coding systems such as Compound Text (used in X11's inter client
1282 communication, ISO-2022-JP (used in Japanese Internet), ISO-2022-KR
1283 (used in Korean Internet), EUC (Extended UNIX Code, used in Asian
1284 localized platforms), and all of these are variants of ISO2022.
1286 In addition to the above, Emacs handles two more kinds of escape
1287 sequences: ISO6429's direction specification and Emacs' private
1288 sequence for specifying character composition.
1290 ISO6429's direction specification takes the following form:
1291 o CSI ']' -- end of the current direction
1292 o CSI '0' ']' -- end of the current direction
1293 o CSI '1' ']' -- start of left-to-right text
1294 o CSI '2' ']' -- start of right-to-left text
1295 The control character CSI (0x9B: control sequence introducer) is
1296 abbreviated to the escape sequence ESC '[' in a 7-bit environment.
1298 Character composition specification takes the following form:
1299 o ESC '0' -- start relative composition
1300 o ESC '1' -- end composition
1301 o ESC '2' -- start rule-base composition (*)
1302 o ESC '3' -- start relative composition with alternate chars (**)
1303 o ESC '4' -- start rule-base composition with alternate chars (**)
1304 Since these are not standard escape sequences of any ISO standard,
1305 the use of them with these meanings is restricted to Emacs only.
1307 (*) This form is used only in Emacs 20.5 and older versions,
1308 but the newer versions can safely decode it.
1309 (**) This form is used only in Emacs 21.1 and newer versions,
1310 and the older versions can't decode it.
1312 Here's a list of example usages of these composition escape
1313 sequences (categorized by `enum composition_method').
1315 COMPOSITION_RELATIVE:
1316 ESC 0 CHAR [ CHAR ] ESC 1
1317 COMPOSITION_WITH_RULE:
1318 ESC 2 CHAR [ RULE CHAR ] ESC 1
1319 COMPOSITION_WITH_ALTCHARS:
1320 ESC 3 ALTCHAR [ ALTCHAR ] ESC 0 CHAR [ CHAR ] ESC 1
1321 COMPOSITION_WITH_RULE_ALTCHARS:
1322 ESC 4 ALTCHAR [ RULE ALTCHAR ] ESC 0 CHAR [ CHAR ] ESC 1 */
1324 enum iso_code_class_type iso_code_class[256];
1326 #define CHARSET_OK(idx, charset, c) \
1327 (coding_system_table[idx] \
1328 && (charset == CHARSET_ASCII \
1329 || (safe_chars = coding_safe_chars (coding_system_table[idx]->symbol), \
1330 CODING_SAFE_CHAR_P (safe_chars, c))) \
1331 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding_system_table[idx], \
1332 charset) \
1333 != CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION))
1335 #define SHIFT_OUT_OK(idx) \
1336 (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding_system_table[idx], 1) >= 0)
1338 #define COMPOSITION_OK(idx) \
1339 (coding_system_table[idx]->composing != COMPOSITION_DISABLED)
1341 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
1342 Check if a text is encoded in ISO2022. If it is, return an
1343 integer in which appropriate flag bits any of:
1344 CODING_CATEGORY_MASK_ISO_7
1345 CODING_CATEGORY_MASK_ISO_7_TIGHT
1346 CODING_CATEGORY_MASK_ISO_8_1
1347 CODING_CATEGORY_MASK_ISO_8_2
1348 CODING_CATEGORY_MASK_ISO_7_ELSE
1349 CODING_CATEGORY_MASK_ISO_8_ELSE
1350 are set. If a code which should never appear in ISO2022 is found,
1351 returns 0. */
1353 static int
1354 detect_coding_iso2022 (src, src_end, multibytep)
1355 unsigned char *src, *src_end;
1356 int multibytep;
1358 int mask = CODING_CATEGORY_MASK_ISO;
1359 int mask_found = 0;
1360 int reg[4], shift_out = 0, single_shifting = 0;
1361 int c, c1, charset;
1362 /* Dummy for ONE_MORE_BYTE. */
1363 struct coding_system dummy_coding;
1364 struct coding_system *coding = &dummy_coding;
1365 Lisp_Object safe_chars;
1367 reg[0] = CHARSET_ASCII, reg[1] = reg[2] = reg[3] = -1;
1368 while (mask && src < src_end)
1370 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
1371 retry:
1372 switch (c)
1374 case ISO_CODE_ESC:
1375 if (inhibit_iso_escape_detection)
1376 break;
1377 single_shifting = 0;
1378 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
1379 if (c >= '(' && c <= '/')
1381 /* Designation sequence for a charset of dimension 1. */
1382 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
1383 if (c1 < ' ' || c1 >= 0x80
1384 || (charset = iso_charset_table[0][c >= ','][c1]) < 0)
1385 /* Invalid designation sequence. Just ignore. */
1386 break;
1387 reg[(c - '(') % 4] = charset;
1389 else if (c == '$')
1391 /* Designation sequence for a charset of dimension 2. */
1392 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
1393 if (c >= '@' && c <= 'B')
1394 /* Designation for JISX0208.1978, GB2312, or JISX0208. */
1395 reg[0] = charset = iso_charset_table[1][0][c];
1396 else if (c >= '(' && c <= '/')
1398 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
1399 if (c1 < ' ' || c1 >= 0x80
1400 || (charset = iso_charset_table[1][c >= ','][c1]) < 0)
1401 /* Invalid designation sequence. Just ignore. */
1402 break;
1403 reg[(c - '(') % 4] = charset;
1405 else
1406 /* Invalid designation sequence. Just ignore. */
1407 break;
1409 else if (c == 'N' || c == 'O')
1411 /* ESC <Fe> for SS2 or SS3. */
1412 mask &= CODING_CATEGORY_MASK_ISO_7_ELSE;
1413 break;
1415 else if (c >= '0' && c <= '4')
1417 /* ESC <Fp> for start/end composition. */
1418 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_7))
1419 mask_found |= CODING_CATEGORY_MASK_ISO_7;
1420 else
1421 mask &= ~CODING_CATEGORY_MASK_ISO_7;
1422 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_7_TIGHT))
1423 mask_found |= CODING_CATEGORY_MASK_ISO_7_TIGHT;
1424 else
1425 mask &= ~CODING_CATEGORY_MASK_ISO_7_TIGHT;
1426 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_8_1))
1427 mask_found |= CODING_CATEGORY_MASK_ISO_8_1;
1428 else
1429 mask &= ~CODING_CATEGORY_MASK_ISO_8_1;
1430 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_8_2))
1431 mask_found |= CODING_CATEGORY_MASK_ISO_8_2;
1432 else
1433 mask &= ~CODING_CATEGORY_MASK_ISO_8_2;
1434 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_7_ELSE))
1435 mask_found |= CODING_CATEGORY_MASK_ISO_7_ELSE;
1436 else
1437 mask &= ~CODING_CATEGORY_MASK_ISO_7_ELSE;
1438 if (COMPOSITION_OK (CODING_CATEGORY_IDX_ISO_8_ELSE))
1439 mask_found |= CODING_CATEGORY_MASK_ISO_8_ELSE;
1440 else
1441 mask &= ~CODING_CATEGORY_MASK_ISO_8_ELSE;
1442 break;
1444 else
1445 /* Invalid escape sequence. Just ignore. */
1446 break;
1448 /* We found a valid designation sequence for CHARSET. */
1449 mask &= ~CODING_CATEGORY_MASK_ISO_8BIT;
1450 c = MAKE_CHAR (charset, 0, 0);
1451 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7, charset, c))
1452 mask_found |= CODING_CATEGORY_MASK_ISO_7;
1453 else
1454 mask &= ~CODING_CATEGORY_MASK_ISO_7;
1455 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7_TIGHT, charset, c))
1456 mask_found |= CODING_CATEGORY_MASK_ISO_7_TIGHT;
1457 else
1458 mask &= ~CODING_CATEGORY_MASK_ISO_7_TIGHT;
1459 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7_ELSE, charset, c))
1460 mask_found |= CODING_CATEGORY_MASK_ISO_7_ELSE;
1461 else
1462 mask &= ~CODING_CATEGORY_MASK_ISO_7_ELSE;
1463 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_8_ELSE, charset, c))
1464 mask_found |= CODING_CATEGORY_MASK_ISO_8_ELSE;
1465 else
1466 mask &= ~CODING_CATEGORY_MASK_ISO_8_ELSE;
1467 break;
1469 case ISO_CODE_SO:
1470 if (inhibit_iso_escape_detection)
1471 break;
1472 single_shifting = 0;
1473 if (shift_out == 0
1474 && (reg[1] >= 0
1475 || SHIFT_OUT_OK (CODING_CATEGORY_IDX_ISO_7_ELSE)
1476 || SHIFT_OUT_OK (CODING_CATEGORY_IDX_ISO_8_ELSE)))
1478 /* Locking shift out. */
1479 mask &= ~CODING_CATEGORY_MASK_ISO_7BIT;
1480 mask_found |= CODING_CATEGORY_MASK_ISO_SHIFT;
1482 break;
1484 case ISO_CODE_SI:
1485 if (inhibit_iso_escape_detection)
1486 break;
1487 single_shifting = 0;
1488 if (shift_out == 1)
1490 /* Locking shift in. */
1491 mask &= ~CODING_CATEGORY_MASK_ISO_7BIT;
1492 mask_found |= CODING_CATEGORY_MASK_ISO_SHIFT;
1494 break;
1496 case ISO_CODE_CSI:
1497 single_shifting = 0;
1498 case ISO_CODE_SS2:
1499 case ISO_CODE_SS3:
1501 int newmask = CODING_CATEGORY_MASK_ISO_8_ELSE;
1503 if (inhibit_iso_escape_detection)
1504 break;
1505 if (c != ISO_CODE_CSI)
1507 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_1]->flags
1508 & CODING_FLAG_ISO_SINGLE_SHIFT)
1509 newmask |= CODING_CATEGORY_MASK_ISO_8_1;
1510 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_2]->flags
1511 & CODING_FLAG_ISO_SINGLE_SHIFT)
1512 newmask |= CODING_CATEGORY_MASK_ISO_8_2;
1513 single_shifting = 1;
1515 if (VECTORP (Vlatin_extra_code_table)
1516 && !NILP (XVECTOR (Vlatin_extra_code_table)->contents[c]))
1518 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_1]->flags
1519 & CODING_FLAG_ISO_LATIN_EXTRA)
1520 newmask |= CODING_CATEGORY_MASK_ISO_8_1;
1521 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_2]->flags
1522 & CODING_FLAG_ISO_LATIN_EXTRA)
1523 newmask |= CODING_CATEGORY_MASK_ISO_8_2;
1525 mask &= newmask;
1526 mask_found |= newmask;
1528 break;
1530 default:
1531 if (c < 0x80)
1533 single_shifting = 0;
1534 break;
1536 else if (c < 0xA0)
1538 single_shifting = 0;
1539 if (VECTORP (Vlatin_extra_code_table)
1540 && !NILP (XVECTOR (Vlatin_extra_code_table)->contents[c]))
1542 int newmask = 0;
1544 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_1]->flags
1545 & CODING_FLAG_ISO_LATIN_EXTRA)
1546 newmask |= CODING_CATEGORY_MASK_ISO_8_1;
1547 if (coding_system_table[CODING_CATEGORY_IDX_ISO_8_2]->flags
1548 & CODING_FLAG_ISO_LATIN_EXTRA)
1549 newmask |= CODING_CATEGORY_MASK_ISO_8_2;
1550 mask &= newmask;
1551 mask_found |= newmask;
1553 else
1554 return 0;
1556 else
1558 mask &= ~(CODING_CATEGORY_MASK_ISO_7BIT
1559 | CODING_CATEGORY_MASK_ISO_7_ELSE);
1560 mask_found |= CODING_CATEGORY_MASK_ISO_8_1;
1561 /* Check the length of succeeding codes of the range
1562 0xA0..0FF. If the byte length is odd, we exclude
1563 CODING_CATEGORY_MASK_ISO_8_2. We can check this only
1564 when we are not single shifting. */
1565 if (!single_shifting
1566 && mask & CODING_CATEGORY_MASK_ISO_8_2)
1568 int i = 1;
1570 c = -1;
1571 while (src < src_end)
1573 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
1574 if (c < 0xA0)
1575 break;
1576 i++;
1579 if (i & 1 && src < src_end)
1580 mask &= ~CODING_CATEGORY_MASK_ISO_8_2;
1581 else
1582 mask_found |= CODING_CATEGORY_MASK_ISO_8_2;
1583 if (c >= 0)
1584 /* This means that we have read one extra byte. */
1585 goto retry;
1588 break;
1591 label_end_of_loop:
1592 return (mask & mask_found);
1595 /* Decode a character of which charset is CHARSET, the 1st position
1596 code is C1, the 2nd position code is C2, and return the decoded
1597 character code. If the variable `translation_table' is non-nil,
1598 returned the translated code. */
1600 #define DECODE_ISO_CHARACTER(charset, c1, c2) \
1601 (NILP (translation_table) \
1602 ? MAKE_CHAR (charset, c1, c2) \
1603 : translate_char (translation_table, -1, charset, c1, c2))
1605 /* Set designation state into CODING. */
1606 #define DECODE_DESIGNATION(reg, dimension, chars, final_char) \
1607 do { \
1608 int charset, c; \
1610 if (final_char < '0' || final_char >= 128) \
1611 goto label_invalid_code; \
1612 charset = ISO_CHARSET_TABLE (make_number (dimension), \
1613 make_number (chars), \
1614 make_number (final_char)); \
1615 c = MAKE_CHAR (charset, 0, 0); \
1616 if (charset >= 0 \
1617 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset) == reg \
1618 || CODING_SAFE_CHAR_P (safe_chars, c))) \
1620 if (coding->spec.iso2022.last_invalid_designation_register == 0 \
1621 && reg == 0 \
1622 && charset == CHARSET_ASCII) \
1624 /* We should insert this designation sequence as is so \
1625 that it is surely written back to a file. */ \
1626 coding->spec.iso2022.last_invalid_designation_register = -1; \
1627 goto label_invalid_code; \
1629 coding->spec.iso2022.last_invalid_designation_register = -1; \
1630 if ((coding->mode & CODING_MODE_DIRECTION) \
1631 && CHARSET_REVERSE_CHARSET (charset) >= 0) \
1632 charset = CHARSET_REVERSE_CHARSET (charset); \
1633 CODING_SPEC_ISO_DESIGNATION (coding, reg) = charset; \
1635 else \
1637 coding->spec.iso2022.last_invalid_designation_register = reg; \
1638 goto label_invalid_code; \
1640 } while (0)
1642 /* Allocate a memory block for storing information about compositions.
1643 The block is chained to the already allocated blocks. */
1645 void
1646 coding_allocate_composition_data (coding, char_offset)
1647 struct coding_system *coding;
1648 int char_offset;
1650 struct composition_data *cmp_data
1651 = (struct composition_data *) xmalloc (sizeof *cmp_data);
1653 cmp_data->char_offset = char_offset;
1654 cmp_data->used = 0;
1655 cmp_data->prev = coding->cmp_data;
1656 cmp_data->next = NULL;
1657 if (coding->cmp_data)
1658 coding->cmp_data->next = cmp_data;
1659 coding->cmp_data = cmp_data;
1660 coding->cmp_data_start = 0;
1663 /* Handle composition start sequence ESC 0, ESC 2, ESC 3, or ESC 4.
1664 ESC 0 : relative composition : ESC 0 CHAR ... ESC 1
1665 ESC 2 : rulebase composition : ESC 2 CHAR RULE CHAR RULE ... CHAR ESC 1
1666 ESC 3 : altchar composition : ESC 3 ALT ... ESC 0 CHAR ... ESC 1
1667 ESC 4 : alt&rule composition : ESC 4 ALT RULE .. ALT ESC 0 CHAR ... ESC 1
1670 #define DECODE_COMPOSITION_START(c1) \
1671 do { \
1672 if (coding->composing == COMPOSITION_DISABLED) \
1674 *dst++ = ISO_CODE_ESC; \
1675 *dst++ = c1 & 0x7f; \
1676 coding->produced_char += 2; \
1678 else if (!COMPOSING_P (coding)) \
1680 /* This is surely the start of a composition. We must be sure \
1681 that coding->cmp_data has enough space to store the \
1682 information about the composition. If not, terminate the \
1683 current decoding loop, allocate one more memory block for \
1684 coding->cmp_data in the caller, then start the decoding \
1685 loop again. We can't allocate memory here directly because \
1686 it may cause buffer/string relocation. */ \
1687 if (!coding->cmp_data \
1688 || (coding->cmp_data->used + COMPOSITION_DATA_MAX_BUNCH_LENGTH \
1689 >= COMPOSITION_DATA_SIZE)) \
1691 coding->result = CODING_FINISH_INSUFFICIENT_CMP; \
1692 goto label_end_of_loop; \
1694 coding->composing = (c1 == '0' ? COMPOSITION_RELATIVE \
1695 : c1 == '2' ? COMPOSITION_WITH_RULE \
1696 : c1 == '3' ? COMPOSITION_WITH_ALTCHARS \
1697 : COMPOSITION_WITH_RULE_ALTCHARS); \
1698 CODING_ADD_COMPOSITION_START (coding, coding->produced_char, \
1699 coding->composing); \
1700 coding->composition_rule_follows = 0; \
1702 else \
1704 /* We are already handling a composition. If the method is \
1705 the following two, the codes following the current escape \
1706 sequence are actual characters stored in a buffer. */ \
1707 if (coding->composing == COMPOSITION_WITH_ALTCHARS \
1708 || coding->composing == COMPOSITION_WITH_RULE_ALTCHARS) \
1710 coding->composing = COMPOSITION_RELATIVE; \
1711 coding->composition_rule_follows = 0; \
1714 } while (0)
1716 /* Handle composition end sequence ESC 1. */
1718 #define DECODE_COMPOSITION_END(c1) \
1719 do { \
1720 if (! COMPOSING_P (coding)) \
1722 *dst++ = ISO_CODE_ESC; \
1723 *dst++ = c1; \
1724 coding->produced_char += 2; \
1726 else \
1728 CODING_ADD_COMPOSITION_END (coding, coding->produced_char); \
1729 coding->composing = COMPOSITION_NO; \
1731 } while (0)
1733 /* Decode a composition rule from the byte C1 (and maybe one more byte
1734 from SRC) and store one encoded composition rule in
1735 coding->cmp_data. */
1737 #define DECODE_COMPOSITION_RULE(c1) \
1738 do { \
1739 int rule = 0; \
1740 (c1) -= 32; \
1741 if (c1 < 81) /* old format (before ver.21) */ \
1743 int gref = (c1) / 9; \
1744 int nref = (c1) % 9; \
1745 if (gref == 4) gref = 10; \
1746 if (nref == 4) nref = 10; \
1747 rule = COMPOSITION_ENCODE_RULE (gref, nref); \
1749 else if (c1 < 93) /* new format (after ver.21) */ \
1751 ONE_MORE_BYTE (c2); \
1752 rule = COMPOSITION_ENCODE_RULE (c1 - 81, c2 - 32); \
1754 CODING_ADD_COMPOSITION_COMPONENT (coding, rule); \
1755 coding->composition_rule_follows = 0; \
1756 } while (0)
1759 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
1761 static void
1762 decode_coding_iso2022 (coding, source, destination, src_bytes, dst_bytes)
1763 struct coding_system *coding;
1764 unsigned char *source, *destination;
1765 int src_bytes, dst_bytes;
1767 unsigned char *src = source;
1768 unsigned char *src_end = source + src_bytes;
1769 unsigned char *dst = destination;
1770 unsigned char *dst_end = destination + dst_bytes;
1771 /* Charsets invoked to graphic plane 0 and 1 respectively. */
1772 int charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1773 int charset1 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 1);
1774 /* SRC_BASE remembers the start position in source in each loop.
1775 The loop will be exited when there's not enough source code
1776 (within macro ONE_MORE_BYTE), or when there's not enough
1777 destination area to produce a character (within macro
1778 EMIT_CHAR). */
1779 unsigned char *src_base;
1780 int c, charset;
1781 Lisp_Object translation_table;
1782 Lisp_Object safe_chars;
1784 safe_chars = coding_safe_chars (coding->symbol);
1786 if (NILP (Venable_character_translation))
1787 translation_table = Qnil;
1788 else
1790 translation_table = coding->translation_table_for_decode;
1791 if (NILP (translation_table))
1792 translation_table = Vstandard_translation_table_for_decode;
1795 coding->result = CODING_FINISH_NORMAL;
1797 while (1)
1799 int c1, c2;
1801 src_base = src;
1802 ONE_MORE_BYTE (c1);
1804 /* We produce no character or one character. */
1805 switch (iso_code_class [c1])
1807 case ISO_0x20_or_0x7F:
1808 if (COMPOSING_P (coding) && coding->composition_rule_follows)
1810 DECODE_COMPOSITION_RULE (c1);
1811 continue;
1813 if (charset0 < 0 || CHARSET_CHARS (charset0) == 94)
1815 /* This is SPACE or DEL. */
1816 charset = CHARSET_ASCII;
1817 break;
1819 /* This is a graphic character, we fall down ... */
1821 case ISO_graphic_plane_0:
1822 if (COMPOSING_P (coding) && coding->composition_rule_follows)
1824 DECODE_COMPOSITION_RULE (c1);
1825 continue;
1827 charset = charset0;
1828 break;
1830 case ISO_0xA0_or_0xFF:
1831 if (charset1 < 0 || CHARSET_CHARS (charset1) == 94
1832 || coding->flags & CODING_FLAG_ISO_SEVEN_BITS)
1833 goto label_invalid_code;
1834 /* This is a graphic character, we fall down ... */
1836 case ISO_graphic_plane_1:
1837 if (charset1 < 0)
1838 goto label_invalid_code;
1839 charset = charset1;
1840 break;
1842 case ISO_control_0:
1843 if (COMPOSING_P (coding))
1844 DECODE_COMPOSITION_END ('1');
1846 /* All ISO2022 control characters in this class have the
1847 same representation in Emacs internal format. */
1848 if (c1 == '\n'
1849 && (coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
1850 && (coding->eol_type == CODING_EOL_CR
1851 || coding->eol_type == CODING_EOL_CRLF))
1853 coding->result = CODING_FINISH_INCONSISTENT_EOL;
1854 goto label_end_of_loop;
1856 charset = CHARSET_ASCII;
1857 break;
1859 case ISO_control_1:
1860 if (COMPOSING_P (coding))
1861 DECODE_COMPOSITION_END ('1');
1862 goto label_invalid_code;
1864 case ISO_carriage_return:
1865 if (COMPOSING_P (coding))
1866 DECODE_COMPOSITION_END ('1');
1868 if (coding->eol_type == CODING_EOL_CR)
1869 c1 = '\n';
1870 else if (coding->eol_type == CODING_EOL_CRLF)
1872 ONE_MORE_BYTE (c1);
1873 if (c1 != ISO_CODE_LF)
1875 src--;
1876 c1 = '\r';
1879 charset = CHARSET_ASCII;
1880 break;
1882 case ISO_shift_out:
1883 if (! (coding->flags & CODING_FLAG_ISO_LOCKING_SHIFT)
1884 || CODING_SPEC_ISO_DESIGNATION (coding, 1) < 0)
1885 goto label_invalid_code;
1886 CODING_SPEC_ISO_INVOCATION (coding, 0) = 1;
1887 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1888 continue;
1890 case ISO_shift_in:
1891 if (! (coding->flags & CODING_FLAG_ISO_LOCKING_SHIFT))
1892 goto label_invalid_code;
1893 CODING_SPEC_ISO_INVOCATION (coding, 0) = 0;
1894 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1895 continue;
1897 case ISO_single_shift_2_7:
1898 case ISO_single_shift_2:
1899 if (! (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT))
1900 goto label_invalid_code;
1901 /* SS2 is handled as an escape sequence of ESC 'N' */
1902 c1 = 'N';
1903 goto label_escape_sequence;
1905 case ISO_single_shift_3:
1906 if (! (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT))
1907 goto label_invalid_code;
1908 /* SS2 is handled as an escape sequence of ESC 'O' */
1909 c1 = 'O';
1910 goto label_escape_sequence;
1912 case ISO_control_sequence_introducer:
1913 /* CSI is handled as an escape sequence of ESC '[' ... */
1914 c1 = '[';
1915 goto label_escape_sequence;
1917 case ISO_escape:
1918 ONE_MORE_BYTE (c1);
1919 label_escape_sequence:
1920 /* Escape sequences handled by Emacs are invocation,
1921 designation, direction specification, and character
1922 composition specification. */
1923 switch (c1)
1925 case '&': /* revision of following character set */
1926 ONE_MORE_BYTE (c1);
1927 if (!(c1 >= '@' && c1 <= '~'))
1928 goto label_invalid_code;
1929 ONE_MORE_BYTE (c1);
1930 if (c1 != ISO_CODE_ESC)
1931 goto label_invalid_code;
1932 ONE_MORE_BYTE (c1);
1933 goto label_escape_sequence;
1935 case '$': /* designation of 2-byte character set */
1936 if (! (coding->flags & CODING_FLAG_ISO_DESIGNATION))
1937 goto label_invalid_code;
1938 ONE_MORE_BYTE (c1);
1939 if (c1 >= '@' && c1 <= 'B')
1940 { /* designation of JISX0208.1978, GB2312.1980,
1941 or JISX0208.1980 */
1942 DECODE_DESIGNATION (0, 2, 94, c1);
1944 else if (c1 >= 0x28 && c1 <= 0x2B)
1945 { /* designation of DIMENSION2_CHARS94 character set */
1946 ONE_MORE_BYTE (c2);
1947 DECODE_DESIGNATION (c1 - 0x28, 2, 94, c2);
1949 else if (c1 >= 0x2C && c1 <= 0x2F)
1950 { /* designation of DIMENSION2_CHARS96 character set */
1951 ONE_MORE_BYTE (c2);
1952 DECODE_DESIGNATION (c1 - 0x2C, 2, 96, c2);
1954 else
1955 goto label_invalid_code;
1956 /* We must update these variables now. */
1957 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1958 charset1 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 1);
1959 continue;
1961 case 'n': /* invocation of locking-shift-2 */
1962 if (! (coding->flags & CODING_FLAG_ISO_LOCKING_SHIFT)
1963 || CODING_SPEC_ISO_DESIGNATION (coding, 2) < 0)
1964 goto label_invalid_code;
1965 CODING_SPEC_ISO_INVOCATION (coding, 0) = 2;
1966 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1967 continue;
1969 case 'o': /* invocation of locking-shift-3 */
1970 if (! (coding->flags & CODING_FLAG_ISO_LOCKING_SHIFT)
1971 || CODING_SPEC_ISO_DESIGNATION (coding, 3) < 0)
1972 goto label_invalid_code;
1973 CODING_SPEC_ISO_INVOCATION (coding, 0) = 3;
1974 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
1975 continue;
1977 case 'N': /* invocation of single-shift-2 */
1978 if (! (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT)
1979 || CODING_SPEC_ISO_DESIGNATION (coding, 2) < 0)
1980 goto label_invalid_code;
1981 charset = CODING_SPEC_ISO_DESIGNATION (coding, 2);
1982 ONE_MORE_BYTE (c1);
1983 if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
1984 goto label_invalid_code;
1985 break;
1987 case 'O': /* invocation of single-shift-3 */
1988 if (! (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT)
1989 || CODING_SPEC_ISO_DESIGNATION (coding, 3) < 0)
1990 goto label_invalid_code;
1991 charset = CODING_SPEC_ISO_DESIGNATION (coding, 3);
1992 ONE_MORE_BYTE (c1);
1993 if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
1994 goto label_invalid_code;
1995 break;
1997 case '0': case '2': case '3': case '4': /* start composition */
1998 DECODE_COMPOSITION_START (c1);
1999 continue;
2001 case '1': /* end composition */
2002 DECODE_COMPOSITION_END (c1);
2003 continue;
2005 case '[': /* specification of direction */
2006 if (coding->flags & CODING_FLAG_ISO_NO_DIRECTION)
2007 goto label_invalid_code;
2008 /* For the moment, nested direction is not supported.
2009 So, `coding->mode & CODING_MODE_DIRECTION' zero means
2010 left-to-right, and nonzero means right-to-left. */
2011 ONE_MORE_BYTE (c1);
2012 switch (c1)
2014 case ']': /* end of the current direction */
2015 coding->mode &= ~CODING_MODE_DIRECTION;
2017 case '0': /* end of the current direction */
2018 case '1': /* start of left-to-right direction */
2019 ONE_MORE_BYTE (c1);
2020 if (c1 == ']')
2021 coding->mode &= ~CODING_MODE_DIRECTION;
2022 else
2023 goto label_invalid_code;
2024 break;
2026 case '2': /* start of right-to-left direction */
2027 ONE_MORE_BYTE (c1);
2028 if (c1 == ']')
2029 coding->mode |= CODING_MODE_DIRECTION;
2030 else
2031 goto label_invalid_code;
2032 break;
2034 default:
2035 goto label_invalid_code;
2037 continue;
2039 default:
2040 if (! (coding->flags & CODING_FLAG_ISO_DESIGNATION))
2041 goto label_invalid_code;
2042 if (c1 >= 0x28 && c1 <= 0x2B)
2043 { /* designation of DIMENSION1_CHARS94 character set */
2044 ONE_MORE_BYTE (c2);
2045 DECODE_DESIGNATION (c1 - 0x28, 1, 94, c2);
2047 else if (c1 >= 0x2C && c1 <= 0x2F)
2048 { /* designation of DIMENSION1_CHARS96 character set */
2049 ONE_MORE_BYTE (c2);
2050 DECODE_DESIGNATION (c1 - 0x2C, 1, 96, c2);
2052 else
2053 goto label_invalid_code;
2054 /* We must update these variables now. */
2055 charset0 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 0);
2056 charset1 = CODING_SPEC_ISO_PLANE_CHARSET (coding, 1);
2057 continue;
2061 /* Now we know CHARSET and 1st position code C1 of a character.
2062 Produce a multibyte sequence for that character while getting
2063 2nd position code C2 if necessary. */
2064 if (CHARSET_DIMENSION (charset) == 2)
2066 ONE_MORE_BYTE (c2);
2067 if (c1 < 0x80 ? c2 < 0x20 || c2 >= 0x80 : c2 < 0xA0)
2068 /* C2 is not in a valid range. */
2069 goto label_invalid_code;
2071 c = DECODE_ISO_CHARACTER (charset, c1, c2);
2072 EMIT_CHAR (c);
2073 continue;
2075 label_invalid_code:
2076 coding->errors++;
2077 if (COMPOSING_P (coding))
2078 DECODE_COMPOSITION_END ('1');
2079 src = src_base;
2080 c = *src++;
2081 EMIT_CHAR (c);
2084 label_end_of_loop:
2085 coding->consumed = coding->consumed_char = src_base - source;
2086 coding->produced = dst - destination;
2087 return;
2091 /* ISO2022 encoding stuff. */
2094 It is not enough to say just "ISO2022" on encoding, we have to
2095 specify more details. In Emacs, each ISO2022 coding system
2096 variant has the following specifications:
2097 1. Initial designation to G0 through G3.
2098 2. Allows short-form designation?
2099 3. ASCII should be designated to G0 before control characters?
2100 4. ASCII should be designated to G0 at end of line?
2101 5. 7-bit environment or 8-bit environment?
2102 6. Use locking-shift?
2103 7. Use Single-shift?
2104 And the following two are only for Japanese:
2105 8. Use ASCII in place of JIS0201-1976-Roman?
2106 9. Use JISX0208-1983 in place of JISX0208-1978?
2107 These specifications are encoded in `coding->flags' as flag bits
2108 defined by macros CODING_FLAG_ISO_XXX. See `coding.h' for more
2109 details.
2112 /* Produce codes (escape sequence) for designating CHARSET to graphic
2113 register REG at DST, and increment DST. If <final-char> of CHARSET is
2114 '@', 'A', or 'B' and the coding system CODING allows, produce
2115 designation sequence of short-form. */
2117 #define ENCODE_DESIGNATION(charset, reg, coding) \
2118 do { \
2119 unsigned char final_char = CHARSET_ISO_FINAL_CHAR (charset); \
2120 char *intermediate_char_94 = "()*+"; \
2121 char *intermediate_char_96 = ",-./"; \
2122 int revision = CODING_SPEC_ISO_REVISION_NUMBER(coding, charset); \
2124 if (revision < 255) \
2126 *dst++ = ISO_CODE_ESC; \
2127 *dst++ = '&'; \
2128 *dst++ = '@' + revision; \
2130 *dst++ = ISO_CODE_ESC; \
2131 if (CHARSET_DIMENSION (charset) == 1) \
2133 if (CHARSET_CHARS (charset) == 94) \
2134 *dst++ = (unsigned char) (intermediate_char_94[reg]); \
2135 else \
2136 *dst++ = (unsigned char) (intermediate_char_96[reg]); \
2138 else \
2140 *dst++ = '$'; \
2141 if (CHARSET_CHARS (charset) == 94) \
2143 if (! (coding->flags & CODING_FLAG_ISO_SHORT_FORM) \
2144 || reg != 0 \
2145 || final_char < '@' || final_char > 'B') \
2146 *dst++ = (unsigned char) (intermediate_char_94[reg]); \
2148 else \
2149 *dst++ = (unsigned char) (intermediate_char_96[reg]); \
2151 *dst++ = final_char; \
2152 CODING_SPEC_ISO_DESIGNATION (coding, reg) = charset; \
2153 } while (0)
2155 /* The following two macros produce codes (control character or escape
2156 sequence) for ISO2022 single-shift functions (single-shift-2 and
2157 single-shift-3). */
2159 #define ENCODE_SINGLE_SHIFT_2 \
2160 do { \
2161 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2162 *dst++ = ISO_CODE_ESC, *dst++ = 'N'; \
2163 else \
2164 *dst++ = ISO_CODE_SS2; \
2165 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 1; \
2166 } while (0)
2168 #define ENCODE_SINGLE_SHIFT_3 \
2169 do { \
2170 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2171 *dst++ = ISO_CODE_ESC, *dst++ = 'O'; \
2172 else \
2173 *dst++ = ISO_CODE_SS3; \
2174 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 1; \
2175 } while (0)
2177 /* The following four macros produce codes (control character or
2178 escape sequence) for ISO2022 locking-shift functions (shift-in,
2179 shift-out, locking-shift-2, and locking-shift-3). */
2181 #define ENCODE_SHIFT_IN \
2182 do { \
2183 *dst++ = ISO_CODE_SI; \
2184 CODING_SPEC_ISO_INVOCATION (coding, 0) = 0; \
2185 } while (0)
2187 #define ENCODE_SHIFT_OUT \
2188 do { \
2189 *dst++ = ISO_CODE_SO; \
2190 CODING_SPEC_ISO_INVOCATION (coding, 0) = 1; \
2191 } while (0)
2193 #define ENCODE_LOCKING_SHIFT_2 \
2194 do { \
2195 *dst++ = ISO_CODE_ESC, *dst++ = 'n'; \
2196 CODING_SPEC_ISO_INVOCATION (coding, 0) = 2; \
2197 } while (0)
2199 #define ENCODE_LOCKING_SHIFT_3 \
2200 do { \
2201 *dst++ = ISO_CODE_ESC, *dst++ = 'o'; \
2202 CODING_SPEC_ISO_INVOCATION (coding, 0) = 3; \
2203 } while (0)
2205 /* Produce codes for a DIMENSION1 character whose character set is
2206 CHARSET and whose position-code is C1. Designation and invocation
2207 sequences are also produced in advance if necessary. */
2209 #define ENCODE_ISO_CHARACTER_DIMENSION1(charset, c1) \
2210 do { \
2211 if (CODING_SPEC_ISO_SINGLE_SHIFTING (coding)) \
2213 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2214 *dst++ = c1 & 0x7F; \
2215 else \
2216 *dst++ = c1 | 0x80; \
2217 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 0; \
2218 break; \
2220 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 0)) \
2222 *dst++ = c1 & 0x7F; \
2223 break; \
2225 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 1)) \
2227 *dst++ = c1 | 0x80; \
2228 break; \
2230 else \
2231 /* Since CHARSET is not yet invoked to any graphic planes, we \
2232 must invoke it, or, at first, designate it to some graphic \
2233 register. Then repeat the loop to actually produce the \
2234 character. */ \
2235 dst = encode_invocation_designation (charset, coding, dst); \
2236 } while (1)
2238 /* Produce codes for a DIMENSION2 character whose character set is
2239 CHARSET and whose position-codes are C1 and C2. Designation and
2240 invocation codes are also produced in advance if necessary. */
2242 #define ENCODE_ISO_CHARACTER_DIMENSION2(charset, c1, c2) \
2243 do { \
2244 if (CODING_SPEC_ISO_SINGLE_SHIFTING (coding)) \
2246 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2247 *dst++ = c1 & 0x7F, *dst++ = c2 & 0x7F; \
2248 else \
2249 *dst++ = c1 | 0x80, *dst++ = c2 | 0x80; \
2250 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 0; \
2251 break; \
2253 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 0)) \
2255 *dst++ = c1 & 0x7F, *dst++= c2 & 0x7F; \
2256 break; \
2258 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 1)) \
2260 *dst++ = c1 | 0x80, *dst++= c2 | 0x80; \
2261 break; \
2263 else \
2264 /* Since CHARSET is not yet invoked to any graphic planes, we \
2265 must invoke it, or, at first, designate it to some graphic \
2266 register. Then repeat the loop to actually produce the \
2267 character. */ \
2268 dst = encode_invocation_designation (charset, coding, dst); \
2269 } while (1)
2271 #define ENCODE_ISO_CHARACTER(c) \
2272 do { \
2273 int charset, c1, c2; \
2275 SPLIT_CHAR (c, charset, c1, c2); \
2276 if (CHARSET_DEFINED_P (charset)) \
2278 if (CHARSET_DIMENSION (charset) == 1) \
2280 if (charset == CHARSET_ASCII \
2281 && coding->flags & CODING_FLAG_ISO_USE_ROMAN) \
2282 charset = charset_latin_jisx0201; \
2283 ENCODE_ISO_CHARACTER_DIMENSION1 (charset, c1); \
2285 else \
2287 if (charset == charset_jisx0208 \
2288 && coding->flags & CODING_FLAG_ISO_USE_OLDJIS) \
2289 charset = charset_jisx0208_1978; \
2290 ENCODE_ISO_CHARACTER_DIMENSION2 (charset, c1, c2); \
2293 else \
2295 *dst++ = c1; \
2296 if (c2 >= 0) \
2297 *dst++ = c2; \
2299 } while (0)
2302 /* Instead of encoding character C, produce one or two `?'s. */
2304 #define ENCODE_UNSAFE_CHARACTER(c) \
2305 do { \
2306 ENCODE_ISO_CHARACTER (CODING_INHIBIT_CHARACTER_SUBSTITUTION); \
2307 if (CHARSET_WIDTH (CHAR_CHARSET (c)) > 1) \
2308 ENCODE_ISO_CHARACTER (CODING_INHIBIT_CHARACTER_SUBSTITUTION); \
2309 } while (0)
2312 /* Produce designation and invocation codes at a place pointed by DST
2313 to use CHARSET. The element `spec.iso2022' of *CODING is updated.
2314 Return new DST. */
2316 unsigned char *
2317 encode_invocation_designation (charset, coding, dst)
2318 int charset;
2319 struct coding_system *coding;
2320 unsigned char *dst;
2322 int reg; /* graphic register number */
2324 /* At first, check designations. */
2325 for (reg = 0; reg < 4; reg++)
2326 if (charset == CODING_SPEC_ISO_DESIGNATION (coding, reg))
2327 break;
2329 if (reg >= 4)
2331 /* CHARSET is not yet designated to any graphic registers. */
2332 /* At first check the requested designation. */
2333 reg = CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset);
2334 if (reg == CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION)
2335 /* Since CHARSET requests no special designation, designate it
2336 to graphic register 0. */
2337 reg = 0;
2339 ENCODE_DESIGNATION (charset, reg, coding);
2342 if (CODING_SPEC_ISO_INVOCATION (coding, 0) != reg
2343 && CODING_SPEC_ISO_INVOCATION (coding, 1) != reg)
2345 /* Since the graphic register REG is not invoked to any graphic
2346 planes, invoke it to graphic plane 0. */
2347 switch (reg)
2349 case 0: /* graphic register 0 */
2350 ENCODE_SHIFT_IN;
2351 break;
2353 case 1: /* graphic register 1 */
2354 ENCODE_SHIFT_OUT;
2355 break;
2357 case 2: /* graphic register 2 */
2358 if (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT)
2359 ENCODE_SINGLE_SHIFT_2;
2360 else
2361 ENCODE_LOCKING_SHIFT_2;
2362 break;
2364 case 3: /* graphic register 3 */
2365 if (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT)
2366 ENCODE_SINGLE_SHIFT_3;
2367 else
2368 ENCODE_LOCKING_SHIFT_3;
2369 break;
2373 return dst;
2376 /* Produce 2-byte codes for encoded composition rule RULE. */
2378 #define ENCODE_COMPOSITION_RULE(rule) \
2379 do { \
2380 int gref, nref; \
2381 COMPOSITION_DECODE_RULE (rule, gref, nref); \
2382 *dst++ = 32 + 81 + gref; \
2383 *dst++ = 32 + nref; \
2384 } while (0)
2386 /* Produce codes for indicating the start of a composition sequence
2387 (ESC 0, ESC 3, or ESC 4). DATA points to an array of integers
2388 which specify information about the composition. See the comment
2389 in coding.h for the format of DATA. */
2391 #define ENCODE_COMPOSITION_START(coding, data) \
2392 do { \
2393 coding->composing = data[3]; \
2394 *dst++ = ISO_CODE_ESC; \
2395 if (coding->composing == COMPOSITION_RELATIVE) \
2396 *dst++ = '0'; \
2397 else \
2399 *dst++ = (coding->composing == COMPOSITION_WITH_ALTCHARS \
2400 ? '3' : '4'); \
2401 coding->cmp_data_index = coding->cmp_data_start + 4; \
2402 coding->composition_rule_follows = 0; \
2404 } while (0)
2406 /* Produce codes for indicating the end of the current composition. */
2408 #define ENCODE_COMPOSITION_END(coding, data) \
2409 do { \
2410 *dst++ = ISO_CODE_ESC; \
2411 *dst++ = '1'; \
2412 coding->cmp_data_start += data[0]; \
2413 coding->composing = COMPOSITION_NO; \
2414 if (coding->cmp_data_start == coding->cmp_data->used \
2415 && coding->cmp_data->next) \
2417 coding->cmp_data = coding->cmp_data->next; \
2418 coding->cmp_data_start = 0; \
2420 } while (0)
2422 /* Produce composition start sequence ESC 0. Here, this sequence
2423 doesn't mean the start of a new composition but means that we have
2424 just produced components (alternate chars and composition rules) of
2425 the composition and the actual text follows in SRC. */
2427 #define ENCODE_COMPOSITION_FAKE_START(coding) \
2428 do { \
2429 *dst++ = ISO_CODE_ESC; \
2430 *dst++ = '0'; \
2431 coding->composing = COMPOSITION_RELATIVE; \
2432 } while (0)
2434 /* The following three macros produce codes for indicating direction
2435 of text. */
2436 #define ENCODE_CONTROL_SEQUENCE_INTRODUCER \
2437 do { \
2438 if (coding->flags == CODING_FLAG_ISO_SEVEN_BITS) \
2439 *dst++ = ISO_CODE_ESC, *dst++ = '['; \
2440 else \
2441 *dst++ = ISO_CODE_CSI; \
2442 } while (0)
2444 #define ENCODE_DIRECTION_R2L \
2445 ENCODE_CONTROL_SEQUENCE_INTRODUCER (dst), *dst++ = '2', *dst++ = ']'
2447 #define ENCODE_DIRECTION_L2R \
2448 ENCODE_CONTROL_SEQUENCE_INTRODUCER (dst), *dst++ = '0', *dst++ = ']'
2450 /* Produce codes for designation and invocation to reset the graphic
2451 planes and registers to initial state. */
2452 #define ENCODE_RESET_PLANE_AND_REGISTER \
2453 do { \
2454 int reg; \
2455 if (CODING_SPEC_ISO_INVOCATION (coding, 0) != 0) \
2456 ENCODE_SHIFT_IN; \
2457 for (reg = 0; reg < 4; reg++) \
2458 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg) >= 0 \
2459 && (CODING_SPEC_ISO_DESIGNATION (coding, reg) \
2460 != CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg))) \
2461 ENCODE_DESIGNATION \
2462 (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg), reg, coding); \
2463 } while (0)
2465 /* Produce designation sequences of charsets in the line started from
2466 SRC to a place pointed by DST, and return updated DST.
2468 If the current block ends before any end-of-line, we may fail to
2469 find all the necessary designations. */
2471 static unsigned char *
2472 encode_designation_at_bol (coding, translation_table, src, src_end, dst)
2473 struct coding_system *coding;
2474 Lisp_Object translation_table;
2475 unsigned char *src, *src_end, *dst;
2477 int charset, c, found = 0, reg;
2478 /* Table of charsets to be designated to each graphic register. */
2479 int r[4];
2481 for (reg = 0; reg < 4; reg++)
2482 r[reg] = -1;
2484 while (found < 4)
2486 ONE_MORE_CHAR (c);
2487 if (c == '\n')
2488 break;
2490 charset = CHAR_CHARSET (c);
2491 reg = CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset);
2492 if (reg != CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION && r[reg] < 0)
2494 found++;
2495 r[reg] = charset;
2499 label_end_of_loop:
2500 if (found)
2502 for (reg = 0; reg < 4; reg++)
2503 if (r[reg] >= 0
2504 && CODING_SPEC_ISO_DESIGNATION (coding, reg) != r[reg])
2505 ENCODE_DESIGNATION (r[reg], reg, coding);
2508 return dst;
2511 /* See the above "GENERAL NOTES on `encode_coding_XXX ()' functions". */
2513 static void
2514 encode_coding_iso2022 (coding, source, destination, src_bytes, dst_bytes)
2515 struct coding_system *coding;
2516 unsigned char *source, *destination;
2517 int src_bytes, dst_bytes;
2519 unsigned char *src = source;
2520 unsigned char *src_end = source + src_bytes;
2521 unsigned char *dst = destination;
2522 unsigned char *dst_end = destination + dst_bytes;
2523 /* Since the maximum bytes produced by each loop is 20, we subtract 19
2524 from DST_END to assure overflow checking is necessary only at the
2525 head of loop. */
2526 unsigned char *adjusted_dst_end = dst_end - 19;
2527 /* SRC_BASE remembers the start position in source in each loop.
2528 The loop will be exited when there's not enough source text to
2529 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
2530 there's not enough destination area to produce encoded codes
2531 (within macro EMIT_BYTES). */
2532 unsigned char *src_base;
2533 int c;
2534 Lisp_Object translation_table;
2535 Lisp_Object safe_chars;
2537 safe_chars = coding_safe_chars (coding->symbol);
2539 if (NILP (Venable_character_translation))
2540 translation_table = Qnil;
2541 else
2543 translation_table = coding->translation_table_for_encode;
2544 if (NILP (translation_table))
2545 translation_table = Vstandard_translation_table_for_encode;
2548 coding->consumed_char = 0;
2549 coding->errors = 0;
2550 while (1)
2552 src_base = src;
2554 if (dst >= (dst_bytes ? adjusted_dst_end : (src - 19)))
2556 coding->result = CODING_FINISH_INSUFFICIENT_DST;
2557 break;
2560 if (coding->flags & CODING_FLAG_ISO_DESIGNATE_AT_BOL
2561 && CODING_SPEC_ISO_BOL (coding))
2563 /* We have to produce designation sequences if any now. */
2564 dst = encode_designation_at_bol (coding, translation_table,
2565 src, src_end, dst);
2566 CODING_SPEC_ISO_BOL (coding) = 0;
2569 /* Check composition start and end. */
2570 if (coding->composing != COMPOSITION_DISABLED
2571 && coding->cmp_data_start < coding->cmp_data->used)
2573 struct composition_data *cmp_data = coding->cmp_data;
2574 int *data = cmp_data->data + coding->cmp_data_start;
2575 int this_pos = cmp_data->char_offset + coding->consumed_char;
2577 if (coding->composing == COMPOSITION_RELATIVE)
2579 if (this_pos == data[2])
2581 ENCODE_COMPOSITION_END (coding, data);
2582 cmp_data = coding->cmp_data;
2583 data = cmp_data->data + coding->cmp_data_start;
2586 else if (COMPOSING_P (coding))
2588 /* COMPOSITION_WITH_ALTCHARS or COMPOSITION_WITH_RULE_ALTCHAR */
2589 if (coding->cmp_data_index == coding->cmp_data_start + data[0])
2590 /* We have consumed components of the composition.
2591 What follows in SRC is the composition's base
2592 text. */
2593 ENCODE_COMPOSITION_FAKE_START (coding);
2594 else
2596 int c = cmp_data->data[coding->cmp_data_index++];
2597 if (coding->composition_rule_follows)
2599 ENCODE_COMPOSITION_RULE (c);
2600 coding->composition_rule_follows = 0;
2602 else
2604 if (coding->flags & CODING_FLAG_ISO_SAFE
2605 && ! CODING_SAFE_CHAR_P (safe_chars, c))
2606 ENCODE_UNSAFE_CHARACTER (c);
2607 else
2608 ENCODE_ISO_CHARACTER (c);
2609 if (coding->composing == COMPOSITION_WITH_RULE_ALTCHARS)
2610 coding->composition_rule_follows = 1;
2612 continue;
2615 if (!COMPOSING_P (coding))
2617 if (this_pos == data[1])
2619 ENCODE_COMPOSITION_START (coding, data);
2620 continue;
2625 ONE_MORE_CHAR (c);
2627 /* Now encode the character C. */
2628 if (c < 0x20 || c == 0x7F)
2630 if (c == '\r')
2632 if (! (coding->mode & CODING_MODE_SELECTIVE_DISPLAY))
2634 if (coding->flags & CODING_FLAG_ISO_RESET_AT_CNTL)
2635 ENCODE_RESET_PLANE_AND_REGISTER;
2636 *dst++ = c;
2637 continue;
2639 /* fall down to treat '\r' as '\n' ... */
2640 c = '\n';
2642 if (c == '\n')
2644 if (coding->flags & CODING_FLAG_ISO_RESET_AT_EOL)
2645 ENCODE_RESET_PLANE_AND_REGISTER;
2646 if (coding->flags & CODING_FLAG_ISO_INIT_AT_BOL)
2647 bcopy (coding->spec.iso2022.initial_designation,
2648 coding->spec.iso2022.current_designation,
2649 sizeof coding->spec.iso2022.initial_designation);
2650 if (coding->eol_type == CODING_EOL_LF
2651 || coding->eol_type == CODING_EOL_UNDECIDED)
2652 *dst++ = ISO_CODE_LF;
2653 else if (coding->eol_type == CODING_EOL_CRLF)
2654 *dst++ = ISO_CODE_CR, *dst++ = ISO_CODE_LF;
2655 else
2656 *dst++ = ISO_CODE_CR;
2657 CODING_SPEC_ISO_BOL (coding) = 1;
2659 else
2661 if (coding->flags & CODING_FLAG_ISO_RESET_AT_CNTL)
2662 ENCODE_RESET_PLANE_AND_REGISTER;
2663 *dst++ = c;
2666 else if (ASCII_BYTE_P (c))
2667 ENCODE_ISO_CHARACTER (c);
2668 else if (SINGLE_BYTE_CHAR_P (c))
2670 *dst++ = c;
2671 coding->errors++;
2673 else if (coding->flags & CODING_FLAG_ISO_SAFE
2674 && ! CODING_SAFE_CHAR_P (safe_chars, c))
2675 ENCODE_UNSAFE_CHARACTER (c);
2676 else
2677 ENCODE_ISO_CHARACTER (c);
2679 coding->consumed_char++;
2682 label_end_of_loop:
2683 coding->consumed = src_base - source;
2684 coding->produced = coding->produced_char = dst - destination;
2688 /*** 4. SJIS and BIG5 handlers ***/
2690 /* Although SJIS and BIG5 are not ISO coding systems, they are used
2691 quite widely. So, for the moment, Emacs supports them in the bare
2692 C code. But, in the future, they may be supported only by CCL. */
2694 /* SJIS is a coding system encoding three character sets: ASCII, right
2695 half of JISX0201-Kana, and JISX0208. An ASCII character is encoded
2696 as is. A character of charset katakana-jisx0201 is encoded by
2697 "position-code + 0x80". A character of charset japanese-jisx0208
2698 is encoded in 2-byte but two position-codes are divided and shifted
2699 so that it fits in the range below.
2701 --- CODE RANGE of SJIS ---
2702 (character set) (range)
2703 ASCII 0x00 .. 0x7F
2704 KATAKANA-JISX0201 0xA1 .. 0xDF
2705 JISX0208 (1st byte) 0x81 .. 0x9F and 0xE0 .. 0xEF
2706 (2nd byte) 0x40 .. 0x7E and 0x80 .. 0xFC
2707 -------------------------------
2711 /* BIG5 is a coding system encoding two character sets: ASCII and
2712 Big5. An ASCII character is encoded as is. Big5 is a two-byte
2713 character set and is encoded in two bytes.
2715 --- CODE RANGE of BIG5 ---
2716 (character set) (range)
2717 ASCII 0x00 .. 0x7F
2718 Big5 (1st byte) 0xA1 .. 0xFE
2719 (2nd byte) 0x40 .. 0x7E and 0xA1 .. 0xFE
2720 --------------------------
2722 Since the number of characters in Big5 is larger than maximum
2723 characters in Emacs' charset (96x96), it can't be handled as one
2724 charset. So, in Emacs, Big5 is divided into two: `charset-big5-1'
2725 and `charset-big5-2'. Both are DIMENSION2 and CHARS94. The former
2726 contains frequently used characters and the latter contains less
2727 frequently used characters. */
2729 /* Macros to decode or encode a character of Big5 in BIG5. B1 and B2
2730 are the 1st and 2nd position-codes of Big5 in BIG5 coding system.
2731 C1 and C2 are the 1st and 2nd position-codes of Emacs' internal
2732 format. CHARSET is `charset_big5_1' or `charset_big5_2'. */
2734 /* Number of Big5 characters which have the same code in 1st byte. */
2735 #define BIG5_SAME_ROW (0xFF - 0xA1 + 0x7F - 0x40)
2737 #define DECODE_BIG5(b1, b2, charset, c1, c2) \
2738 do { \
2739 unsigned int temp \
2740 = (b1 - 0xA1) * BIG5_SAME_ROW + b2 - (b2 < 0x7F ? 0x40 : 0x62); \
2741 if (b1 < 0xC9) \
2742 charset = charset_big5_1; \
2743 else \
2745 charset = charset_big5_2; \
2746 temp -= (0xC9 - 0xA1) * BIG5_SAME_ROW; \
2748 c1 = temp / (0xFF - 0xA1) + 0x21; \
2749 c2 = temp % (0xFF - 0xA1) + 0x21; \
2750 } while (0)
2752 #define ENCODE_BIG5(charset, c1, c2, b1, b2) \
2753 do { \
2754 unsigned int temp = (c1 - 0x21) * (0xFF - 0xA1) + (c2 - 0x21); \
2755 if (charset == charset_big5_2) \
2756 temp += BIG5_SAME_ROW * (0xC9 - 0xA1); \
2757 b1 = temp / BIG5_SAME_ROW + 0xA1; \
2758 b2 = temp % BIG5_SAME_ROW; \
2759 b2 += b2 < 0x3F ? 0x40 : 0x62; \
2760 } while (0)
2762 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2763 Check if a text is encoded in SJIS. If it is, return
2764 CODING_CATEGORY_MASK_SJIS, else return 0. */
2766 static int
2767 detect_coding_sjis (src, src_end, multibytep)
2768 unsigned char *src, *src_end;
2769 int multibytep;
2771 int c;
2772 /* Dummy for ONE_MORE_BYTE. */
2773 struct coding_system dummy_coding;
2774 struct coding_system *coding = &dummy_coding;
2776 while (1)
2778 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2779 if (c < 0x80)
2780 continue;
2781 if (c == 0x80 || c == 0xA0 || c > 0xEF)
2782 return 0;
2783 if (c <= 0x9F || c >= 0xE0)
2785 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2786 if (c < 0x40 || c == 0x7F || c > 0xFC)
2787 return 0;
2790 label_end_of_loop:
2791 return CODING_CATEGORY_MASK_SJIS;
2794 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2795 Check if a text is encoded in BIG5. If it is, return
2796 CODING_CATEGORY_MASK_BIG5, else return 0. */
2798 static int
2799 detect_coding_big5 (src, src_end, multibytep)
2800 unsigned char *src, *src_end;
2801 int multibytep;
2803 int c;
2804 /* Dummy for ONE_MORE_BYTE. */
2805 struct coding_system dummy_coding;
2806 struct coding_system *coding = &dummy_coding;
2808 while (1)
2810 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2811 if (c < 0x80)
2812 continue;
2813 if (c < 0xA1 || c > 0xFE)
2814 return 0;
2815 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2816 if (c < 0x40 || (c > 0x7F && c < 0xA1) || c > 0xFE)
2817 return 0;
2819 label_end_of_loop:
2820 return CODING_CATEGORY_MASK_BIG5;
2823 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2824 Check if a text is encoded in UTF-8. If it is, return
2825 CODING_CATEGORY_MASK_UTF_8, else return 0. */
2827 #define UTF_8_1_OCTET_P(c) ((c) < 0x80)
2828 #define UTF_8_EXTRA_OCTET_P(c) (((c) & 0xC0) == 0x80)
2829 #define UTF_8_2_OCTET_LEADING_P(c) (((c) & 0xE0) == 0xC0)
2830 #define UTF_8_3_OCTET_LEADING_P(c) (((c) & 0xF0) == 0xE0)
2831 #define UTF_8_4_OCTET_LEADING_P(c) (((c) & 0xF8) == 0xF0)
2832 #define UTF_8_5_OCTET_LEADING_P(c) (((c) & 0xFC) == 0xF8)
2833 #define UTF_8_6_OCTET_LEADING_P(c) (((c) & 0xFE) == 0xFC)
2835 static int
2836 detect_coding_utf_8 (src, src_end, multibytep)
2837 unsigned char *src, *src_end;
2838 int multibytep;
2840 unsigned char c;
2841 int seq_maybe_bytes;
2842 /* Dummy for ONE_MORE_BYTE. */
2843 struct coding_system dummy_coding;
2844 struct coding_system *coding = &dummy_coding;
2846 while (1)
2848 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2849 if (UTF_8_1_OCTET_P (c))
2850 continue;
2851 else if (UTF_8_2_OCTET_LEADING_P (c))
2852 seq_maybe_bytes = 1;
2853 else if (UTF_8_3_OCTET_LEADING_P (c))
2854 seq_maybe_bytes = 2;
2855 else if (UTF_8_4_OCTET_LEADING_P (c))
2856 seq_maybe_bytes = 3;
2857 else if (UTF_8_5_OCTET_LEADING_P (c))
2858 seq_maybe_bytes = 4;
2859 else if (UTF_8_6_OCTET_LEADING_P (c))
2860 seq_maybe_bytes = 5;
2861 else
2862 return 0;
2866 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
2867 if (!UTF_8_EXTRA_OCTET_P (c))
2868 return 0;
2869 seq_maybe_bytes--;
2871 while (seq_maybe_bytes > 0);
2874 label_end_of_loop:
2875 return CODING_CATEGORY_MASK_UTF_8;
2878 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2879 Check if a text is encoded in UTF-16 Big Endian (endian == 1) or
2880 Little Endian (otherwise). If it is, return
2881 CODING_CATEGORY_MASK_UTF_16_BE or CODING_CATEGORY_MASK_UTF_16_LE,
2882 else return 0. */
2884 #define UTF_16_INVALID_P(val) \
2885 (((val) == 0xFFFE) \
2886 || ((val) == 0xFFFF))
2888 #define UTF_16_HIGH_SURROGATE_P(val) \
2889 (((val) & 0xD800) == 0xD800)
2891 #define UTF_16_LOW_SURROGATE_P(val) \
2892 (((val) & 0xDC00) == 0xDC00)
2894 static int
2895 detect_coding_utf_16 (src, src_end, multibytep)
2896 unsigned char *src, *src_end;
2897 int multibytep;
2899 unsigned char c1, c2;
2900 /* Dummy for TWO_MORE_BYTES. */
2901 struct coding_system dummy_coding;
2902 struct coding_system *coding = &dummy_coding;
2904 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
2905 ONE_MORE_BYTE_CHECK_MULTIBYTE (c2, multibytep);
2907 if ((c1 == 0xFF) && (c2 == 0xFE))
2908 return CODING_CATEGORY_MASK_UTF_16_LE;
2909 else if ((c1 == 0xFE) && (c2 == 0xFF))
2910 return CODING_CATEGORY_MASK_UTF_16_BE;
2912 label_end_of_loop:
2913 return 0;
2916 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions".
2917 If SJIS_P is 1, decode SJIS text, else decode BIG5 test. */
2919 static void
2920 decode_coding_sjis_big5 (coding, source, destination,
2921 src_bytes, dst_bytes, sjis_p)
2922 struct coding_system *coding;
2923 unsigned char *source, *destination;
2924 int src_bytes, dst_bytes;
2925 int sjis_p;
2927 unsigned char *src = source;
2928 unsigned char *src_end = source + src_bytes;
2929 unsigned char *dst = destination;
2930 unsigned char *dst_end = destination + dst_bytes;
2931 /* SRC_BASE remembers the start position in source in each loop.
2932 The loop will be exited when there's not enough source code
2933 (within macro ONE_MORE_BYTE), or when there's not enough
2934 destination area to produce a character (within macro
2935 EMIT_CHAR). */
2936 unsigned char *src_base;
2937 Lisp_Object translation_table;
2939 if (NILP (Venable_character_translation))
2940 translation_table = Qnil;
2941 else
2943 translation_table = coding->translation_table_for_decode;
2944 if (NILP (translation_table))
2945 translation_table = Vstandard_translation_table_for_decode;
2948 coding->produced_char = 0;
2949 while (1)
2951 int c, charset, c1, c2;
2953 src_base = src;
2954 ONE_MORE_BYTE (c1);
2956 if (c1 < 0x80)
2958 charset = CHARSET_ASCII;
2959 if (c1 < 0x20)
2961 if (c1 == '\r')
2963 if (coding->eol_type == CODING_EOL_CRLF)
2965 ONE_MORE_BYTE (c2);
2966 if (c2 == '\n')
2967 c1 = c2;
2968 else
2969 /* To process C2 again, SRC is subtracted by 1. */
2970 src--;
2972 else if (coding->eol_type == CODING_EOL_CR)
2973 c1 = '\n';
2975 else if (c1 == '\n'
2976 && (coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
2977 && (coding->eol_type == CODING_EOL_CR
2978 || coding->eol_type == CODING_EOL_CRLF))
2980 coding->result = CODING_FINISH_INCONSISTENT_EOL;
2981 goto label_end_of_loop;
2985 else
2987 if (sjis_p)
2989 if (c1 == 0x80 || c1 == 0xA0 || c1 > 0xEF)
2990 goto label_invalid_code;
2991 if (c1 <= 0x9F || c1 >= 0xE0)
2993 /* SJIS -> JISX0208 */
2994 ONE_MORE_BYTE (c2);
2995 if (c2 < 0x40 || c2 == 0x7F || c2 > 0xFC)
2996 goto label_invalid_code;
2997 DECODE_SJIS (c1, c2, c1, c2);
2998 charset = charset_jisx0208;
3000 else
3001 /* SJIS -> JISX0201-Kana */
3002 charset = charset_katakana_jisx0201;
3004 else
3006 /* BIG5 -> Big5 */
3007 if (c1 < 0xA0 || c1 > 0xFE)
3008 goto label_invalid_code;
3009 ONE_MORE_BYTE (c2);
3010 if (c2 < 0x40 || (c2 > 0x7E && c2 < 0xA1) || c2 > 0xFE)
3011 goto label_invalid_code;
3012 DECODE_BIG5 (c1, c2, charset, c1, c2);
3016 c = DECODE_ISO_CHARACTER (charset, c1, c2);
3017 EMIT_CHAR (c);
3018 continue;
3020 label_invalid_code:
3021 coding->errors++;
3022 src = src_base;
3023 c = *src++;
3024 EMIT_CHAR (c);
3027 label_end_of_loop:
3028 coding->consumed = coding->consumed_char = src_base - source;
3029 coding->produced = dst - destination;
3030 return;
3033 /* See the above "GENERAL NOTES on `encode_coding_XXX ()' functions".
3034 This function can encode charsets `ascii', `katakana-jisx0201',
3035 `japanese-jisx0208', `chinese-big5-1', and `chinese-big5-2'. We
3036 are sure that all these charsets are registered as official charset
3037 (i.e. do not have extended leading-codes). Characters of other
3038 charsets are produced without any encoding. If SJIS_P is 1, encode
3039 SJIS text, else encode BIG5 text. */
3041 static void
3042 encode_coding_sjis_big5 (coding, source, destination,
3043 src_bytes, dst_bytes, sjis_p)
3044 struct coding_system *coding;
3045 unsigned char *source, *destination;
3046 int src_bytes, dst_bytes;
3047 int sjis_p;
3049 unsigned char *src = source;
3050 unsigned char *src_end = source + src_bytes;
3051 unsigned char *dst = destination;
3052 unsigned char *dst_end = destination + dst_bytes;
3053 /* SRC_BASE remembers the start position in source in each loop.
3054 The loop will be exited when there's not enough source text to
3055 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
3056 there's not enough destination area to produce encoded codes
3057 (within macro EMIT_BYTES). */
3058 unsigned char *src_base;
3059 Lisp_Object translation_table;
3061 if (NILP (Venable_character_translation))
3062 translation_table = Qnil;
3063 else
3065 translation_table = coding->translation_table_for_encode;
3066 if (NILP (translation_table))
3067 translation_table = Vstandard_translation_table_for_encode;
3070 while (1)
3072 int c, charset, c1, c2;
3074 src_base = src;
3075 ONE_MORE_CHAR (c);
3077 /* Now encode the character C. */
3078 if (SINGLE_BYTE_CHAR_P (c))
3080 switch (c)
3082 case '\r':
3083 if (!(coding->mode & CODING_MODE_SELECTIVE_DISPLAY))
3085 EMIT_ONE_BYTE (c);
3086 break;
3088 c = '\n';
3089 case '\n':
3090 if (coding->eol_type == CODING_EOL_CRLF)
3092 EMIT_TWO_BYTES ('\r', c);
3093 break;
3095 else if (coding->eol_type == CODING_EOL_CR)
3096 c = '\r';
3097 default:
3098 EMIT_ONE_BYTE (c);
3101 else
3103 SPLIT_CHAR (c, charset, c1, c2);
3104 if (sjis_p)
3106 if (charset == charset_jisx0208
3107 || charset == charset_jisx0208_1978)
3109 ENCODE_SJIS (c1, c2, c1, c2);
3110 EMIT_TWO_BYTES (c1, c2);
3112 else if (charset == charset_katakana_jisx0201)
3113 EMIT_ONE_BYTE (c1 | 0x80);
3114 else if (charset == charset_latin_jisx0201)
3115 EMIT_ONE_BYTE (c1);
3116 else
3117 /* There's no way other than producing the internal
3118 codes as is. */
3119 EMIT_BYTES (src_base, src);
3121 else
3123 if (charset == charset_big5_1 || charset == charset_big5_2)
3125 ENCODE_BIG5 (charset, c1, c2, c1, c2);
3126 EMIT_TWO_BYTES (c1, c2);
3128 else
3129 /* There's no way other than producing the internal
3130 codes as is. */
3131 EMIT_BYTES (src_base, src);
3134 coding->consumed_char++;
3137 label_end_of_loop:
3138 coding->consumed = src_base - source;
3139 coding->produced = coding->produced_char = dst - destination;
3143 /*** 5. CCL handlers ***/
3145 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
3146 Check if a text is encoded in a coding system of which
3147 encoder/decoder are written in CCL program. If it is, return
3148 CODING_CATEGORY_MASK_CCL, else return 0. */
3150 static int
3151 detect_coding_ccl (src, src_end, multibytep)
3152 unsigned char *src, *src_end;
3153 int multibytep;
3155 unsigned char *valid;
3156 int c;
3157 /* Dummy for ONE_MORE_BYTE. */
3158 struct coding_system dummy_coding;
3159 struct coding_system *coding = &dummy_coding;
3161 /* No coding system is assigned to coding-category-ccl. */
3162 if (!coding_system_table[CODING_CATEGORY_IDX_CCL])
3163 return 0;
3165 valid = coding_system_table[CODING_CATEGORY_IDX_CCL]->spec.ccl.valid_codes;
3166 while (1)
3168 ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
3169 if (! valid[c])
3170 return 0;
3172 label_end_of_loop:
3173 return CODING_CATEGORY_MASK_CCL;
3177 /*** 6. End-of-line handlers ***/
3179 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
3181 static void
3182 decode_eol (coding, source, destination, src_bytes, dst_bytes)
3183 struct coding_system *coding;
3184 unsigned char *source, *destination;
3185 int src_bytes, dst_bytes;
3187 unsigned char *src = source;
3188 unsigned char *dst = destination;
3189 unsigned char *src_end = src + src_bytes;
3190 unsigned char *dst_end = dst + dst_bytes;
3191 Lisp_Object translation_table;
3192 /* SRC_BASE remembers the start position in source in each loop.
3193 The loop will be exited when there's not enough source code
3194 (within macro ONE_MORE_BYTE), or when there's not enough
3195 destination area to produce a character (within macro
3196 EMIT_CHAR). */
3197 unsigned char *src_base;
3198 int c;
3200 translation_table = Qnil;
3201 switch (coding->eol_type)
3203 case CODING_EOL_CRLF:
3204 while (1)
3206 src_base = src;
3207 ONE_MORE_BYTE (c);
3208 if (c == '\r')
3210 ONE_MORE_BYTE (c);
3211 if (c != '\n')
3213 src--;
3214 c = '\r';
3217 else if (c == '\n'
3218 && (coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL))
3220 coding->result = CODING_FINISH_INCONSISTENT_EOL;
3221 goto label_end_of_loop;
3223 EMIT_CHAR (c);
3225 break;
3227 case CODING_EOL_CR:
3228 while (1)
3230 src_base = src;
3231 ONE_MORE_BYTE (c);
3232 if (c == '\n')
3234 if (coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
3236 coding->result = CODING_FINISH_INCONSISTENT_EOL;
3237 goto label_end_of_loop;
3240 else if (c == '\r')
3241 c = '\n';
3242 EMIT_CHAR (c);
3244 break;
3246 default: /* no need for EOL handling */
3247 while (1)
3249 src_base = src;
3250 ONE_MORE_BYTE (c);
3251 EMIT_CHAR (c);
3255 label_end_of_loop:
3256 coding->consumed = coding->consumed_char = src_base - source;
3257 coding->produced = dst - destination;
3258 return;
3261 /* See "GENERAL NOTES about `encode_coding_XXX ()' functions". Encode
3262 format of end-of-line according to `coding->eol_type'. It also
3263 convert multibyte form 8-bit characters to unibyte if
3264 CODING->src_multibyte is nonzero. If `coding->mode &
3265 CODING_MODE_SELECTIVE_DISPLAY' is nonzero, code '\r' in source text
3266 also means end-of-line. */
3268 static void
3269 encode_eol (coding, source, destination, src_bytes, dst_bytes)
3270 struct coding_system *coding;
3271 const unsigned char *source;
3272 unsigned char *destination;
3273 int src_bytes, dst_bytes;
3275 const unsigned char *src = source;
3276 unsigned char *dst = destination;
3277 const unsigned char *src_end = src + src_bytes;
3278 unsigned char *dst_end = dst + dst_bytes;
3279 Lisp_Object translation_table;
3280 /* SRC_BASE remembers the start position in source in each loop.
3281 The loop will be exited when there's not enough source text to
3282 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
3283 there's not enough destination area to produce encoded codes
3284 (within macro EMIT_BYTES). */
3285 const unsigned char *src_base;
3286 unsigned char *tmp;
3287 int c;
3288 int selective_display = coding->mode & CODING_MODE_SELECTIVE_DISPLAY;
3290 translation_table = Qnil;
3291 if (coding->src_multibyte
3292 && *(src_end - 1) == LEADING_CODE_8_BIT_CONTROL)
3294 src_end--;
3295 src_bytes--;
3296 coding->result = CODING_FINISH_INSUFFICIENT_SRC;
3299 if (coding->eol_type == CODING_EOL_CRLF)
3301 while (src < src_end)
3303 src_base = src;
3304 c = *src++;
3305 if (c >= 0x20)
3306 EMIT_ONE_BYTE (c);
3307 else if (c == '\n' || (c == '\r' && selective_display))
3308 EMIT_TWO_BYTES ('\r', '\n');
3309 else
3310 EMIT_ONE_BYTE (c);
3312 src_base = src;
3313 label_end_of_loop:
3316 else
3318 if (!dst_bytes || src_bytes <= dst_bytes)
3320 safe_bcopy (src, dst, src_bytes);
3321 src_base = src_end;
3322 dst += src_bytes;
3324 else
3326 if (coding->src_multibyte
3327 && *(src + dst_bytes - 1) == LEADING_CODE_8_BIT_CONTROL)
3328 dst_bytes--;
3329 safe_bcopy (src, dst, dst_bytes);
3330 src_base = src + dst_bytes;
3331 dst = destination + dst_bytes;
3332 coding->result = CODING_FINISH_INSUFFICIENT_DST;
3334 if (coding->eol_type == CODING_EOL_CR)
3336 for (tmp = destination; tmp < dst; tmp++)
3337 if (*tmp == '\n') *tmp = '\r';
3339 else if (selective_display)
3341 for (tmp = destination; tmp < dst; tmp++)
3342 if (*tmp == '\r') *tmp = '\n';
3345 if (coding->src_multibyte)
3346 dst = destination + str_as_unibyte (destination, dst - destination);
3348 coding->consumed = src_base - source;
3349 coding->produced = dst - destination;
3350 coding->produced_char = coding->produced;
3354 /*** 7. C library functions ***/
3356 /* In Emacs Lisp, a coding system is represented by a Lisp symbol which
3357 has a property `coding-system'. The value of this property is a
3358 vector of length 5 (called the coding-vector). Among elements of
3359 this vector, the first (element[0]) and the fifth (element[4])
3360 carry important information for decoding/encoding. Before
3361 decoding/encoding, this information should be set in fields of a
3362 structure of type `coding_system'.
3364 The value of the property `coding-system' can be a symbol of another
3365 subsidiary coding-system. In that case, Emacs gets coding-vector
3366 from that symbol.
3368 `element[0]' contains information to be set in `coding->type'. The
3369 value and its meaning is as follows:
3371 0 -- coding_type_emacs_mule
3372 1 -- coding_type_sjis
3373 2 -- coding_type_iso2022
3374 3 -- coding_type_big5
3375 4 -- coding_type_ccl encoder/decoder written in CCL
3376 nil -- coding_type_no_conversion
3377 t -- coding_type_undecided (automatic conversion on decoding,
3378 no-conversion on encoding)
3380 `element[4]' contains information to be set in `coding->flags' and
3381 `coding->spec'. The meaning varies by `coding->type'.
3383 If `coding->type' is `coding_type_iso2022', element[4] is a vector
3384 of length 32 (of which the first 13 sub-elements are used now).
3385 Meanings of these sub-elements are:
3387 sub-element[N] where N is 0 through 3: to be set in `coding->spec.iso2022'
3388 If the value is an integer of valid charset, the charset is
3389 assumed to be designated to graphic register N initially.
3391 If the value is minus, it is a minus value of charset which
3392 reserves graphic register N, which means that the charset is
3393 not designated initially but should be designated to graphic
3394 register N just before encoding a character in that charset.
3396 If the value is nil, graphic register N is never used on
3397 encoding.
3399 sub-element[N] where N is 4 through 11: to be set in `coding->flags'
3400 Each value takes t or nil. See the section ISO2022 of
3401 `coding.h' for more information.
3403 If `coding->type' is `coding_type_big5', element[4] is t to denote
3404 BIG5-ETen or nil to denote BIG5-HKU.
3406 If `coding->type' takes the other value, element[4] is ignored.
3408 Emacs Lisp's coding systems also carry information about format of
3409 end-of-line in a value of property `eol-type'. If the value is
3410 integer, 0 means CODING_EOL_LF, 1 means CODING_EOL_CRLF, and 2
3411 means CODING_EOL_CR. If it is not integer, it should be a vector
3412 of subsidiary coding systems of which property `eol-type' has one
3413 of the above values.
3417 /* Extract information for decoding/encoding from CODING_SYSTEM_SYMBOL
3418 and set it in CODING. If CODING_SYSTEM_SYMBOL is invalid, CODING
3419 is setup so that no conversion is necessary and return -1, else
3420 return 0. */
3423 setup_coding_system (coding_system, coding)
3424 Lisp_Object coding_system;
3425 struct coding_system *coding;
3427 Lisp_Object coding_spec, coding_type, eol_type, plist;
3428 Lisp_Object val;
3430 /* At first, zero clear all members. */
3431 bzero (coding, sizeof (struct coding_system));
3433 /* Initialize some fields required for all kinds of coding systems. */
3434 coding->symbol = coding_system;
3435 coding->heading_ascii = -1;
3436 coding->post_read_conversion = coding->pre_write_conversion = Qnil;
3437 coding->composing = COMPOSITION_DISABLED;
3438 coding->cmp_data = NULL;
3440 if (NILP (coding_system))
3441 goto label_invalid_coding_system;
3443 coding_spec = Fget (coding_system, Qcoding_system);
3445 if (!VECTORP (coding_spec)
3446 || XVECTOR (coding_spec)->size != 5
3447 || !CONSP (XVECTOR (coding_spec)->contents[3]))
3448 goto label_invalid_coding_system;
3450 eol_type = inhibit_eol_conversion ? Qnil : Fget (coding_system, Qeol_type);
3451 if (VECTORP (eol_type))
3453 coding->eol_type = CODING_EOL_UNDECIDED;
3454 coding->common_flags = CODING_REQUIRE_DETECTION_MASK;
3456 else if (XFASTINT (eol_type) == 1)
3458 coding->eol_type = CODING_EOL_CRLF;
3459 coding->common_flags
3460 = CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3462 else if (XFASTINT (eol_type) == 2)
3464 coding->eol_type = CODING_EOL_CR;
3465 coding->common_flags
3466 = CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3468 else
3469 coding->eol_type = CODING_EOL_LF;
3471 coding_type = XVECTOR (coding_spec)->contents[0];
3472 /* Try short cut. */
3473 if (SYMBOLP (coding_type))
3475 if (EQ (coding_type, Qt))
3477 coding->type = coding_type_undecided;
3478 coding->common_flags |= CODING_REQUIRE_DETECTION_MASK;
3480 else
3481 coding->type = coding_type_no_conversion;
3482 /* Initialize this member. Any thing other than
3483 CODING_CATEGORY_IDX_UTF_16_BE and
3484 CODING_CATEGORY_IDX_UTF_16_LE are ok because they have
3485 special treatment in detect_eol. */
3486 coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
3488 return 0;
3491 /* Get values of coding system properties:
3492 `post-read-conversion', `pre-write-conversion',
3493 `translation-table-for-decode', `translation-table-for-encode'. */
3494 plist = XVECTOR (coding_spec)->contents[3];
3495 /* Pre & post conversion functions should be disabled if
3496 inhibit_eol_conversion is nonzero. This is the case that a code
3497 conversion function is called while those functions are running. */
3498 if (! inhibit_pre_post_conversion)
3500 coding->post_read_conversion = Fplist_get (plist, Qpost_read_conversion);
3501 coding->pre_write_conversion = Fplist_get (plist, Qpre_write_conversion);
3503 val = Fplist_get (plist, Qtranslation_table_for_decode);
3504 if (SYMBOLP (val))
3505 val = Fget (val, Qtranslation_table_for_decode);
3506 coding->translation_table_for_decode = CHAR_TABLE_P (val) ? val : Qnil;
3507 val = Fplist_get (plist, Qtranslation_table_for_encode);
3508 if (SYMBOLP (val))
3509 val = Fget (val, Qtranslation_table_for_encode);
3510 coding->translation_table_for_encode = CHAR_TABLE_P (val) ? val : Qnil;
3511 val = Fplist_get (plist, Qcoding_category);
3512 if (!NILP (val))
3514 val = Fget (val, Qcoding_category_index);
3515 if (INTEGERP (val))
3516 coding->category_idx = XINT (val);
3517 else
3518 goto label_invalid_coding_system;
3520 else
3521 goto label_invalid_coding_system;
3523 /* If the coding system has non-nil `composition' property, enable
3524 composition handling. */
3525 val = Fplist_get (plist, Qcomposition);
3526 if (!NILP (val))
3527 coding->composing = COMPOSITION_NO;
3529 switch (XFASTINT (coding_type))
3531 case 0:
3532 coding->type = coding_type_emacs_mule;
3533 coding->common_flags
3534 |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3535 if (!NILP (coding->post_read_conversion))
3536 coding->common_flags |= CODING_REQUIRE_DECODING_MASK;
3537 if (!NILP (coding->pre_write_conversion))
3538 coding->common_flags |= CODING_REQUIRE_ENCODING_MASK;
3539 break;
3541 case 1:
3542 coding->type = coding_type_sjis;
3543 coding->common_flags
3544 |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3545 break;
3547 case 2:
3548 coding->type = coding_type_iso2022;
3549 coding->common_flags
3550 |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3552 Lisp_Object val, temp;
3553 Lisp_Object *flags;
3554 int i, charset, reg_bits = 0;
3556 val = XVECTOR (coding_spec)->contents[4];
3558 if (!VECTORP (val) || XVECTOR (val)->size != 32)
3559 goto label_invalid_coding_system;
3561 flags = XVECTOR (val)->contents;
3562 coding->flags
3563 = ((NILP (flags[4]) ? 0 : CODING_FLAG_ISO_SHORT_FORM)
3564 | (NILP (flags[5]) ? 0 : CODING_FLAG_ISO_RESET_AT_EOL)
3565 | (NILP (flags[6]) ? 0 : CODING_FLAG_ISO_RESET_AT_CNTL)
3566 | (NILP (flags[7]) ? 0 : CODING_FLAG_ISO_SEVEN_BITS)
3567 | (NILP (flags[8]) ? 0 : CODING_FLAG_ISO_LOCKING_SHIFT)
3568 | (NILP (flags[9]) ? 0 : CODING_FLAG_ISO_SINGLE_SHIFT)
3569 | (NILP (flags[10]) ? 0 : CODING_FLAG_ISO_USE_ROMAN)
3570 | (NILP (flags[11]) ? 0 : CODING_FLAG_ISO_USE_OLDJIS)
3571 | (NILP (flags[12]) ? 0 : CODING_FLAG_ISO_NO_DIRECTION)
3572 | (NILP (flags[13]) ? 0 : CODING_FLAG_ISO_INIT_AT_BOL)
3573 | (NILP (flags[14]) ? 0 : CODING_FLAG_ISO_DESIGNATE_AT_BOL)
3574 | (NILP (flags[15]) ? 0 : CODING_FLAG_ISO_SAFE)
3575 | (NILP (flags[16]) ? 0 : CODING_FLAG_ISO_LATIN_EXTRA)
3578 /* Invoke graphic register 0 to plane 0. */
3579 CODING_SPEC_ISO_INVOCATION (coding, 0) = 0;
3580 /* Invoke graphic register 1 to plane 1 if we can use full 8-bit. */
3581 CODING_SPEC_ISO_INVOCATION (coding, 1)
3582 = (coding->flags & CODING_FLAG_ISO_SEVEN_BITS ? -1 : 1);
3583 /* Not single shifting at first. */
3584 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 0;
3585 /* Beginning of buffer should also be regarded as bol. */
3586 CODING_SPEC_ISO_BOL (coding) = 1;
3588 for (charset = 0; charset <= MAX_CHARSET; charset++)
3589 CODING_SPEC_ISO_REVISION_NUMBER (coding, charset) = 255;
3590 val = Vcharset_revision_alist;
3591 while (CONSP (val))
3593 charset = get_charset_id (Fcar_safe (XCAR (val)));
3594 if (charset >= 0
3595 && (temp = Fcdr_safe (XCAR (val)), INTEGERP (temp))
3596 && (i = XINT (temp), (i >= 0 && (i + '@') < 128)))
3597 CODING_SPEC_ISO_REVISION_NUMBER (coding, charset) = i;
3598 val = XCDR (val);
3601 /* Checks FLAGS[REG] (REG = 0, 1, 2 3) and decide designations.
3602 FLAGS[REG] can be one of below:
3603 integer CHARSET: CHARSET occupies register I,
3604 t: designate nothing to REG initially, but can be used
3605 by any charsets,
3606 list of integer, nil, or t: designate the first
3607 element (if integer) to REG initially, the remaining
3608 elements (if integer) is designated to REG on request,
3609 if an element is t, REG can be used by any charsets,
3610 nil: REG is never used. */
3611 for (charset = 0; charset <= MAX_CHARSET; charset++)
3612 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset)
3613 = CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION;
3614 for (i = 0; i < 4; i++)
3616 if ((INTEGERP (flags[i])
3617 && (charset = XINT (flags[i]), CHARSET_VALID_P (charset)))
3618 || (charset = get_charset_id (flags[i])) >= 0)
3620 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i) = charset;
3621 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset) = i;
3623 else if (EQ (flags[i], Qt))
3625 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i) = -1;
3626 reg_bits |= 1 << i;
3627 coding->flags |= CODING_FLAG_ISO_DESIGNATION;
3629 else if (CONSP (flags[i]))
3631 Lisp_Object tail;
3632 tail = flags[i];
3634 coding->flags |= CODING_FLAG_ISO_DESIGNATION;
3635 if ((INTEGERP (XCAR (tail))
3636 && (charset = XINT (XCAR (tail)),
3637 CHARSET_VALID_P (charset)))
3638 || (charset = get_charset_id (XCAR (tail))) >= 0)
3640 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i) = charset;
3641 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset) =i;
3643 else
3644 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i) = -1;
3645 tail = XCDR (tail);
3646 while (CONSP (tail))
3648 if ((INTEGERP (XCAR (tail))
3649 && (charset = XINT (XCAR (tail)),
3650 CHARSET_VALID_P (charset)))
3651 || (charset = get_charset_id (XCAR (tail))) >= 0)
3652 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset)
3653 = i;
3654 else if (EQ (XCAR (tail), Qt))
3655 reg_bits |= 1 << i;
3656 tail = XCDR (tail);
3659 else
3660 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i) = -1;
3662 CODING_SPEC_ISO_DESIGNATION (coding, i)
3663 = CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, i);
3666 if (reg_bits && ! (coding->flags & CODING_FLAG_ISO_LOCKING_SHIFT))
3668 /* REG 1 can be used only by locking shift in 7-bit env. */
3669 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS)
3670 reg_bits &= ~2;
3671 if (! (coding->flags & CODING_FLAG_ISO_SINGLE_SHIFT))
3672 /* Without any shifting, only REG 0 and 1 can be used. */
3673 reg_bits &= 3;
3676 if (reg_bits)
3677 for (charset = 0; charset <= MAX_CHARSET; charset++)
3679 if (CHARSET_DEFINED_P (charset)
3680 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset)
3681 == CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION))
3683 /* There exist some default graphic registers to be
3684 used by CHARSET. */
3686 /* We had better avoid designating a charset of
3687 CHARS96 to REG 0 as far as possible. */
3688 if (CHARSET_CHARS (charset) == 96)
3689 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset)
3690 = (reg_bits & 2
3691 ? 1 : (reg_bits & 4 ? 2 : (reg_bits & 8 ? 3 : 0)));
3692 else
3693 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset)
3694 = (reg_bits & 1
3695 ? 0 : (reg_bits & 2 ? 1 : (reg_bits & 4 ? 2 : 3)));
3699 coding->common_flags |= CODING_REQUIRE_FLUSHING_MASK;
3700 coding->spec.iso2022.last_invalid_designation_register = -1;
3701 break;
3703 case 3:
3704 coding->type = coding_type_big5;
3705 coding->common_flags
3706 |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3707 coding->flags
3708 = (NILP (XVECTOR (coding_spec)->contents[4])
3709 ? CODING_FLAG_BIG5_HKU
3710 : CODING_FLAG_BIG5_ETEN);
3711 break;
3713 case 4:
3714 coding->type = coding_type_ccl;
3715 coding->common_flags
3716 |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
3718 val = XVECTOR (coding_spec)->contents[4];
3719 if (! CONSP (val)
3720 || setup_ccl_program (&(coding->spec.ccl.decoder),
3721 XCAR (val)) < 0
3722 || setup_ccl_program (&(coding->spec.ccl.encoder),
3723 XCDR (val)) < 0)
3724 goto label_invalid_coding_system;
3726 bzero (coding->spec.ccl.valid_codes, 256);
3727 val = Fplist_get (plist, Qvalid_codes);
3728 if (CONSP (val))
3730 Lisp_Object this;
3732 for (; CONSP (val); val = XCDR (val))
3734 this = XCAR (val);
3735 if (INTEGERP (this)
3736 && XINT (this) >= 0 && XINT (this) < 256)
3737 coding->spec.ccl.valid_codes[XINT (this)] = 1;
3738 else if (CONSP (this)
3739 && INTEGERP (XCAR (this))
3740 && INTEGERP (XCDR (this)))
3742 int start = XINT (XCAR (this));
3743 int end = XINT (XCDR (this));
3745 if (start >= 0 && start <= end && end < 256)
3746 while (start <= end)
3747 coding->spec.ccl.valid_codes[start++] = 1;
3752 coding->common_flags |= CODING_REQUIRE_FLUSHING_MASK;
3753 coding->spec.ccl.cr_carryover = 0;
3754 coding->spec.ccl.eight_bit_carryover[0] = 0;
3755 break;
3757 case 5:
3758 coding->type = coding_type_raw_text;
3759 break;
3761 default:
3762 goto label_invalid_coding_system;
3764 return 0;
3766 label_invalid_coding_system:
3767 coding->type = coding_type_no_conversion;
3768 coding->category_idx = CODING_CATEGORY_IDX_BINARY;
3769 coding->common_flags = 0;
3770 coding->eol_type = CODING_EOL_LF;
3771 coding->pre_write_conversion = coding->post_read_conversion = Qnil;
3772 return -1;
3775 /* Free memory blocks allocated for storing composition information. */
3777 void
3778 coding_free_composition_data (coding)
3779 struct coding_system *coding;
3781 struct composition_data *cmp_data = coding->cmp_data, *next;
3783 if (!cmp_data)
3784 return;
3785 /* Memory blocks are chained. At first, rewind to the first, then,
3786 free blocks one by one. */
3787 while (cmp_data->prev)
3788 cmp_data = cmp_data->prev;
3789 while (cmp_data)
3791 next = cmp_data->next;
3792 xfree (cmp_data);
3793 cmp_data = next;
3795 coding->cmp_data = NULL;
3798 /* Set `char_offset' member of all memory blocks pointed by
3799 coding->cmp_data to POS. */
3801 void
3802 coding_adjust_composition_offset (coding, pos)
3803 struct coding_system *coding;
3804 int pos;
3806 struct composition_data *cmp_data;
3808 for (cmp_data = coding->cmp_data; cmp_data; cmp_data = cmp_data->next)
3809 cmp_data->char_offset = pos;
3812 /* Setup raw-text or one of its subsidiaries in the structure
3813 coding_system CODING according to the already setup value eol_type
3814 in CODING. CODING should be setup for some coding system in
3815 advance. */
3817 void
3818 setup_raw_text_coding_system (coding)
3819 struct coding_system *coding;
3821 if (coding->type != coding_type_raw_text)
3823 coding->symbol = Qraw_text;
3824 coding->type = coding_type_raw_text;
3825 if (coding->eol_type != CODING_EOL_UNDECIDED)
3827 Lisp_Object subsidiaries;
3828 subsidiaries = Fget (Qraw_text, Qeol_type);
3830 if (VECTORP (subsidiaries)
3831 && XVECTOR (subsidiaries)->size == 3)
3832 coding->symbol
3833 = XVECTOR (subsidiaries)->contents[coding->eol_type];
3835 setup_coding_system (coding->symbol, coding);
3837 return;
3840 /* Emacs has a mechanism to automatically detect a coding system if it
3841 is one of Emacs' internal format, ISO2022, SJIS, and BIG5. But,
3842 it's impossible to distinguish some coding systems accurately
3843 because they use the same range of codes. So, at first, coding
3844 systems are categorized into 7, those are:
3846 o coding-category-emacs-mule
3848 The category for a coding system which has the same code range
3849 as Emacs' internal format. Assigned the coding-system (Lisp
3850 symbol) `emacs-mule' by default.
3852 o coding-category-sjis
3854 The category for a coding system which has the same code range
3855 as SJIS. Assigned the coding-system (Lisp
3856 symbol) `japanese-shift-jis' by default.
3858 o coding-category-iso-7
3860 The category for a coding system which has the same code range
3861 as ISO2022 of 7-bit environment. This doesn't use any locking
3862 shift and single shift functions. This can encode/decode all
3863 charsets. Assigned the coding-system (Lisp symbol)
3864 `iso-2022-7bit' by default.
3866 o coding-category-iso-7-tight
3868 Same as coding-category-iso-7 except that this can
3869 encode/decode only the specified charsets.
3871 o coding-category-iso-8-1
3873 The category for a coding system which has the same code range
3874 as ISO2022 of 8-bit environment and graphic plane 1 used only
3875 for DIMENSION1 charset. This doesn't use any locking shift
3876 and single shift functions. Assigned the coding-system (Lisp
3877 symbol) `iso-latin-1' by default.
3879 o coding-category-iso-8-2
3881 The category for a coding system which has the same code range
3882 as ISO2022 of 8-bit environment and graphic plane 1 used only
3883 for DIMENSION2 charset. This doesn't use any locking shift
3884 and single shift functions. Assigned the coding-system (Lisp
3885 symbol) `japanese-iso-8bit' by default.
3887 o coding-category-iso-7-else
3889 The category for a coding system which has the same code range
3890 as ISO2022 of 7-bit environment but uses locking shift or
3891 single shift functions. Assigned the coding-system (Lisp
3892 symbol) `iso-2022-7bit-lock' by default.
3894 o coding-category-iso-8-else
3896 The category for a coding system which has the same code range
3897 as ISO2022 of 8-bit environment but uses locking shift or
3898 single shift functions. Assigned the coding-system (Lisp
3899 symbol) `iso-2022-8bit-ss2' by default.
3901 o coding-category-big5
3903 The category for a coding system which has the same code range
3904 as BIG5. Assigned the coding-system (Lisp symbol)
3905 `cn-big5' by default.
3907 o coding-category-utf-8
3909 The category for a coding system which has the same code range
3910 as UTF-8 (cf. RFC2279). Assigned the coding-system (Lisp
3911 symbol) `utf-8' by default.
3913 o coding-category-utf-16-be
3915 The category for a coding system in which a text has an
3916 Unicode signature (cf. Unicode Standard) in the order of BIG
3917 endian at the head. Assigned the coding-system (Lisp symbol)
3918 `utf-16-be' by default.
3920 o coding-category-utf-16-le
3922 The category for a coding system in which a text has an
3923 Unicode signature (cf. Unicode Standard) in the order of
3924 LITTLE endian at the head. Assigned the coding-system (Lisp
3925 symbol) `utf-16-le' by default.
3927 o coding-category-ccl
3929 The category for a coding system of which encoder/decoder is
3930 written in CCL programs. The default value is nil, i.e., no
3931 coding system is assigned.
3933 o coding-category-binary
3935 The category for a coding system not categorized in any of the
3936 above. Assigned the coding-system (Lisp symbol)
3937 `no-conversion' by default.
3939 Each of them is a Lisp symbol and the value is an actual
3940 `coding-system' (this is also a Lisp symbol) assigned by a user.
3941 What Emacs does actually is to detect a category of coding system.
3942 Then, it uses a `coding-system' assigned to it. If Emacs can't
3943 decide a single possible category, it selects a category of the
3944 highest priority. Priorities of categories are also specified by a
3945 user in a Lisp variable `coding-category-list'.
3949 static
3950 int ascii_skip_code[256];
3952 /* Detect how a text of length SRC_BYTES pointed by SOURCE is encoded.
3953 If it detects possible coding systems, return an integer in which
3954 appropriate flag bits are set. Flag bits are defined by macros
3955 CODING_CATEGORY_MASK_XXX in `coding.h'. If PRIORITIES is non-NULL,
3956 it should point the table `coding_priorities'. In that case, only
3957 the flag bit for a coding system of the highest priority is set in
3958 the returned value. If MULTIBYTEP is nonzero, 8-bit codes of the
3959 range 0x80..0x9F are in multibyte form.
3961 How many ASCII characters are at the head is returned as *SKIP. */
3963 static int
3964 detect_coding_mask (source, src_bytes, priorities, skip, multibytep)
3965 unsigned char *source;
3966 int src_bytes, *priorities, *skip;
3967 int multibytep;
3969 register unsigned char c;
3970 unsigned char *src = source, *src_end = source + src_bytes;
3971 unsigned int mask, utf16_examined_p, iso2022_examined_p;
3972 int i;
3974 /* At first, skip all ASCII characters and control characters except
3975 for three ISO2022 specific control characters. */
3976 ascii_skip_code[ISO_CODE_SO] = 0;
3977 ascii_skip_code[ISO_CODE_SI] = 0;
3978 ascii_skip_code[ISO_CODE_ESC] = 0;
3980 label_loop_detect_coding:
3981 while (src < src_end && ascii_skip_code[*src]) src++;
3982 *skip = src - source;
3984 if (src >= src_end)
3985 /* We found nothing other than ASCII. There's nothing to do. */
3986 return 0;
3988 c = *src;
3989 /* The text seems to be encoded in some multilingual coding system.
3990 Now, try to find in which coding system the text is encoded. */
3991 if (c < 0x80)
3993 /* i.e. (c == ISO_CODE_ESC || c == ISO_CODE_SI || c == ISO_CODE_SO) */
3994 /* C is an ISO2022 specific control code of C0. */
3995 mask = detect_coding_iso2022 (src, src_end, multibytep);
3996 if (mask == 0)
3998 /* No valid ISO2022 code follows C. Try again. */
3999 src++;
4000 if (c == ISO_CODE_ESC)
4001 ascii_skip_code[ISO_CODE_ESC] = 1;
4002 else
4003 ascii_skip_code[ISO_CODE_SO] = ascii_skip_code[ISO_CODE_SI] = 1;
4004 goto label_loop_detect_coding;
4006 if (priorities)
4008 for (i = 0; i < CODING_CATEGORY_IDX_MAX; i++)
4010 if (mask & priorities[i])
4011 return priorities[i];
4013 return CODING_CATEGORY_MASK_RAW_TEXT;
4016 else
4018 int try;
4020 if (multibytep && c == LEADING_CODE_8_BIT_CONTROL)
4021 c = src[1] - 0x20;
4023 if (c < 0xA0)
4025 /* C is the first byte of SJIS character code,
4026 or a leading-code of Emacs' internal format (emacs-mule),
4027 or the first byte of UTF-16. */
4028 try = (CODING_CATEGORY_MASK_SJIS
4029 | CODING_CATEGORY_MASK_EMACS_MULE
4030 | CODING_CATEGORY_MASK_UTF_16_BE
4031 | CODING_CATEGORY_MASK_UTF_16_LE);
4033 /* Or, if C is a special latin extra code,
4034 or is an ISO2022 specific control code of C1 (SS2 or SS3),
4035 or is an ISO2022 control-sequence-introducer (CSI),
4036 we should also consider the possibility of ISO2022 codings. */
4037 if ((VECTORP (Vlatin_extra_code_table)
4038 && !NILP (XVECTOR (Vlatin_extra_code_table)->contents[c]))
4039 || (c == ISO_CODE_SS2 || c == ISO_CODE_SS3)
4040 || (c == ISO_CODE_CSI
4041 && (src < src_end
4042 && (*src == ']'
4043 || ((*src == '0' || *src == '1' || *src == '2')
4044 && src + 1 < src_end
4045 && src[1] == ']')))))
4046 try |= (CODING_CATEGORY_MASK_ISO_8_ELSE
4047 | CODING_CATEGORY_MASK_ISO_8BIT);
4049 else
4050 /* C is a character of ISO2022 in graphic plane right,
4051 or a SJIS's 1-byte character code (i.e. JISX0201),
4052 or the first byte of BIG5's 2-byte code,
4053 or the first byte of UTF-8/16. */
4054 try = (CODING_CATEGORY_MASK_ISO_8_ELSE
4055 | CODING_CATEGORY_MASK_ISO_8BIT
4056 | CODING_CATEGORY_MASK_SJIS
4057 | CODING_CATEGORY_MASK_BIG5
4058 | CODING_CATEGORY_MASK_UTF_8
4059 | CODING_CATEGORY_MASK_UTF_16_BE
4060 | CODING_CATEGORY_MASK_UTF_16_LE);
4062 /* Or, we may have to consider the possibility of CCL. */
4063 if (coding_system_table[CODING_CATEGORY_IDX_CCL]
4064 && (coding_system_table[CODING_CATEGORY_IDX_CCL]
4065 ->spec.ccl.valid_codes)[c])
4066 try |= CODING_CATEGORY_MASK_CCL;
4068 mask = 0;
4069 utf16_examined_p = iso2022_examined_p = 0;
4070 if (priorities)
4072 for (i = 0; i < CODING_CATEGORY_IDX_MAX; i++)
4074 if (!iso2022_examined_p
4075 && (priorities[i] & try & CODING_CATEGORY_MASK_ISO))
4077 mask |= detect_coding_iso2022 (src, src_end, multibytep);
4078 iso2022_examined_p = 1;
4080 else if (priorities[i] & try & CODING_CATEGORY_MASK_SJIS)
4081 mask |= detect_coding_sjis (src, src_end, multibytep);
4082 else if (priorities[i] & try & CODING_CATEGORY_MASK_UTF_8)
4083 mask |= detect_coding_utf_8 (src, src_end, multibytep);
4084 else if (!utf16_examined_p
4085 && (priorities[i] & try &
4086 CODING_CATEGORY_MASK_UTF_16_BE_LE))
4088 mask |= detect_coding_utf_16 (src, src_end, multibytep);
4089 utf16_examined_p = 1;
4091 else if (priorities[i] & try & CODING_CATEGORY_MASK_BIG5)
4092 mask |= detect_coding_big5 (src, src_end, multibytep);
4093 else if (priorities[i] & try & CODING_CATEGORY_MASK_EMACS_MULE)
4094 mask |= detect_coding_emacs_mule (src, src_end, multibytep);
4095 else if (priorities[i] & try & CODING_CATEGORY_MASK_CCL)
4096 mask |= detect_coding_ccl (src, src_end, multibytep);
4097 else if (priorities[i] & CODING_CATEGORY_MASK_RAW_TEXT)
4098 mask |= CODING_CATEGORY_MASK_RAW_TEXT;
4099 else if (priorities[i] & CODING_CATEGORY_MASK_BINARY)
4100 mask |= CODING_CATEGORY_MASK_BINARY;
4101 if (mask & priorities[i])
4102 return priorities[i];
4104 return CODING_CATEGORY_MASK_RAW_TEXT;
4106 if (try & CODING_CATEGORY_MASK_ISO)
4107 mask |= detect_coding_iso2022 (src, src_end, multibytep);
4108 if (try & CODING_CATEGORY_MASK_SJIS)
4109 mask |= detect_coding_sjis (src, src_end, multibytep);
4110 if (try & CODING_CATEGORY_MASK_BIG5)
4111 mask |= detect_coding_big5 (src, src_end, multibytep);
4112 if (try & CODING_CATEGORY_MASK_UTF_8)
4113 mask |= detect_coding_utf_8 (src, src_end, multibytep);
4114 if (try & CODING_CATEGORY_MASK_UTF_16_BE_LE)
4115 mask |= detect_coding_utf_16 (src, src_end, multibytep);
4116 if (try & CODING_CATEGORY_MASK_EMACS_MULE)
4117 mask |= detect_coding_emacs_mule (src, src_end, multibytep);
4118 if (try & CODING_CATEGORY_MASK_CCL)
4119 mask |= detect_coding_ccl (src, src_end, multibytep);
4121 return (mask | CODING_CATEGORY_MASK_RAW_TEXT | CODING_CATEGORY_MASK_BINARY);
4124 /* Detect how a text of length SRC_BYTES pointed by SRC is encoded.
4125 The information of the detected coding system is set in CODING. */
4127 void
4128 detect_coding (coding, src, src_bytes)
4129 struct coding_system *coding;
4130 const unsigned char *src;
4131 int src_bytes;
4133 unsigned int idx;
4134 int skip, mask;
4135 Lisp_Object val;
4137 val = Vcoding_category_list;
4138 mask = detect_coding_mask (src, src_bytes, coding_priorities, &skip,
4139 coding->src_multibyte);
4140 coding->heading_ascii = skip;
4142 if (!mask) return;
4144 /* We found a single coding system of the highest priority in MASK. */
4145 idx = 0;
4146 while (mask && ! (mask & 1)) mask >>= 1, idx++;
4147 if (! mask)
4148 idx = CODING_CATEGORY_IDX_RAW_TEXT;
4150 val = SYMBOL_VALUE (XVECTOR (Vcoding_category_table)->contents[idx]);
4152 if (coding->eol_type != CODING_EOL_UNDECIDED)
4154 Lisp_Object tmp;
4156 tmp = Fget (val, Qeol_type);
4157 if (VECTORP (tmp))
4158 val = XVECTOR (tmp)->contents[coding->eol_type];
4161 /* Setup this new coding system while preserving some slots. */
4163 int src_multibyte = coding->src_multibyte;
4164 int dst_multibyte = coding->dst_multibyte;
4166 setup_coding_system (val, coding);
4167 coding->src_multibyte = src_multibyte;
4168 coding->dst_multibyte = dst_multibyte;
4169 coding->heading_ascii = skip;
4173 /* Detect how end-of-line of a text of length SRC_BYTES pointed by
4174 SOURCE is encoded. Return one of CODING_EOL_LF, CODING_EOL_CRLF,
4175 CODING_EOL_CR, and CODING_EOL_UNDECIDED.
4177 How many non-eol characters are at the head is returned as *SKIP. */
4179 #define MAX_EOL_CHECK_COUNT 3
4181 static int
4182 detect_eol_type (source, src_bytes, skip)
4183 unsigned char *source;
4184 int src_bytes, *skip;
4186 unsigned char *src = source, *src_end = src + src_bytes;
4187 unsigned char c;
4188 int total = 0; /* How many end-of-lines are found so far. */
4189 int eol_type = CODING_EOL_UNDECIDED;
4190 int this_eol_type;
4192 *skip = 0;
4194 while (src < src_end && total < MAX_EOL_CHECK_COUNT)
4196 c = *src++;
4197 if (c == '\n' || c == '\r')
4199 if (*skip == 0)
4200 *skip = src - 1 - source;
4201 total++;
4202 if (c == '\n')
4203 this_eol_type = CODING_EOL_LF;
4204 else if (src >= src_end || *src != '\n')
4205 this_eol_type = CODING_EOL_CR;
4206 else
4207 this_eol_type = CODING_EOL_CRLF, src++;
4209 if (eol_type == CODING_EOL_UNDECIDED)
4210 /* This is the first end-of-line. */
4211 eol_type = this_eol_type;
4212 else if (eol_type != this_eol_type)
4214 /* The found type is different from what found before. */
4215 eol_type = CODING_EOL_INCONSISTENT;
4216 break;
4221 if (*skip == 0)
4222 *skip = src_end - source;
4223 return eol_type;
4226 /* Like detect_eol_type, but detect EOL type in 2-octet
4227 big-endian/little-endian format for coding systems utf-16-be and
4228 utf-16-le. */
4230 static int
4231 detect_eol_type_in_2_octet_form (source, src_bytes, skip, big_endian_p)
4232 unsigned char *source;
4233 int src_bytes, *skip, big_endian_p;
4235 unsigned char *src = source, *src_end = src + src_bytes;
4236 unsigned int c1, c2;
4237 int total = 0; /* How many end-of-lines are found so far. */
4238 int eol_type = CODING_EOL_UNDECIDED;
4239 int this_eol_type;
4240 int msb, lsb;
4242 if (big_endian_p)
4243 msb = 0, lsb = 1;
4244 else
4245 msb = 1, lsb = 0;
4247 *skip = 0;
4249 while ((src + 1) < src_end && total < MAX_EOL_CHECK_COUNT)
4251 c1 = (src[msb] << 8) | (src[lsb]);
4252 src += 2;
4254 if (c1 == '\n' || c1 == '\r')
4256 if (*skip == 0)
4257 *skip = src - 2 - source;
4258 total++;
4259 if (c1 == '\n')
4261 this_eol_type = CODING_EOL_LF;
4263 else
4265 if ((src + 1) >= src_end)
4267 this_eol_type = CODING_EOL_CR;
4269 else
4271 c2 = (src[msb] << 8) | (src[lsb]);
4272 if (c2 == '\n')
4273 this_eol_type = CODING_EOL_CRLF, src += 2;
4274 else
4275 this_eol_type = CODING_EOL_CR;
4279 if (eol_type == CODING_EOL_UNDECIDED)
4280 /* This is the first end-of-line. */
4281 eol_type = this_eol_type;
4282 else if (eol_type != this_eol_type)
4284 /* The found type is different from what found before. */
4285 eol_type = CODING_EOL_INCONSISTENT;
4286 break;
4291 if (*skip == 0)
4292 *skip = src_end - source;
4293 return eol_type;
4296 /* Detect how end-of-line of a text of length SRC_BYTES pointed by SRC
4297 is encoded. If it detects an appropriate format of end-of-line, it
4298 sets the information in *CODING. */
4300 void
4301 detect_eol (coding, src, src_bytes)
4302 struct coding_system *coding;
4303 const unsigned char *src;
4304 int src_bytes;
4306 Lisp_Object val;
4307 int skip;
4308 int eol_type;
4310 switch (coding->category_idx)
4312 case CODING_CATEGORY_IDX_UTF_16_BE:
4313 eol_type = detect_eol_type_in_2_octet_form (src, src_bytes, &skip, 1);
4314 break;
4315 case CODING_CATEGORY_IDX_UTF_16_LE:
4316 eol_type = detect_eol_type_in_2_octet_form (src, src_bytes, &skip, 0);
4317 break;
4318 default:
4319 eol_type = detect_eol_type (src, src_bytes, &skip);
4320 break;
4323 if (coding->heading_ascii > skip)
4324 coding->heading_ascii = skip;
4325 else
4326 skip = coding->heading_ascii;
4328 if (eol_type == CODING_EOL_UNDECIDED)
4329 return;
4330 if (eol_type == CODING_EOL_INCONSISTENT)
4332 #if 0
4333 /* This code is suppressed until we find a better way to
4334 distinguish raw text file and binary file. */
4336 /* If we have already detected that the coding is raw-text, the
4337 coding should actually be no-conversion. */
4338 if (coding->type == coding_type_raw_text)
4340 setup_coding_system (Qno_conversion, coding);
4341 return;
4343 /* Else, let's decode only text code anyway. */
4344 #endif /* 0 */
4345 eol_type = CODING_EOL_LF;
4348 val = Fget (coding->symbol, Qeol_type);
4349 if (VECTORP (val) && XVECTOR (val)->size == 3)
4351 int src_multibyte = coding->src_multibyte;
4352 int dst_multibyte = coding->dst_multibyte;
4353 struct composition_data *cmp_data = coding->cmp_data;
4355 setup_coding_system (XVECTOR (val)->contents[eol_type], coding);
4356 coding->src_multibyte = src_multibyte;
4357 coding->dst_multibyte = dst_multibyte;
4358 coding->heading_ascii = skip;
4359 coding->cmp_data = cmp_data;
4363 #define CONVERSION_BUFFER_EXTRA_ROOM 256
4365 #define DECODING_BUFFER_MAG(coding) \
4366 (coding->type == coding_type_iso2022 \
4367 ? 3 \
4368 : (coding->type == coding_type_ccl \
4369 ? coding->spec.ccl.decoder.buf_magnification \
4370 : 2))
4372 /* Return maximum size (bytes) of a buffer enough for decoding
4373 SRC_BYTES of text encoded in CODING. */
4376 decoding_buffer_size (coding, src_bytes)
4377 struct coding_system *coding;
4378 int src_bytes;
4380 return (src_bytes * DECODING_BUFFER_MAG (coding)
4381 + CONVERSION_BUFFER_EXTRA_ROOM);
4384 /* Return maximum size (bytes) of a buffer enough for encoding
4385 SRC_BYTES of text to CODING. */
4388 encoding_buffer_size (coding, src_bytes)
4389 struct coding_system *coding;
4390 int src_bytes;
4392 int magnification;
4394 if (coding->type == coding_type_ccl)
4395 magnification = coding->spec.ccl.encoder.buf_magnification;
4396 else if (CODING_REQUIRE_ENCODING (coding))
4397 magnification = 3;
4398 else
4399 magnification = 1;
4401 return (src_bytes * magnification + CONVERSION_BUFFER_EXTRA_ROOM);
4404 /* Working buffer for code conversion. */
4405 struct conversion_buffer
4407 int size; /* size of data. */
4408 int on_stack; /* 1 if allocated by alloca. */
4409 unsigned char *data;
4412 /* Don't use alloca for allocating memory space larger than this, lest
4413 we overflow their stack. */
4414 #define MAX_ALLOCA 16*1024
4416 /* Allocate LEN bytes of memory for BUF (struct conversion_buffer). */
4417 #define allocate_conversion_buffer(buf, len) \
4418 do { \
4419 if (len < MAX_ALLOCA) \
4421 buf.data = (unsigned char *) alloca (len); \
4422 buf.on_stack = 1; \
4424 else \
4426 buf.data = (unsigned char *) xmalloc (len); \
4427 buf.on_stack = 0; \
4429 buf.size = len; \
4430 } while (0)
4432 /* Double the allocated memory for *BUF. */
4433 static void
4434 extend_conversion_buffer (buf)
4435 struct conversion_buffer *buf;
4437 if (buf->on_stack)
4439 unsigned char *save = buf->data;
4440 buf->data = (unsigned char *) xmalloc (buf->size * 2);
4441 bcopy (save, buf->data, buf->size);
4442 buf->on_stack = 0;
4444 else
4446 buf->data = (unsigned char *) xrealloc (buf->data, buf->size * 2);
4448 buf->size *= 2;
4451 /* Free the allocated memory for BUF if it is not on stack. */
4452 static void
4453 free_conversion_buffer (buf)
4454 struct conversion_buffer *buf;
4456 if (!buf->on_stack)
4457 xfree (buf->data);
4461 ccl_coding_driver (coding, source, destination, src_bytes, dst_bytes, encodep)
4462 struct coding_system *coding;
4463 unsigned char *source, *destination;
4464 int src_bytes, dst_bytes, encodep;
4466 struct ccl_program *ccl
4467 = encodep ? &coding->spec.ccl.encoder : &coding->spec.ccl.decoder;
4468 unsigned char *dst = destination;
4470 ccl->suppress_error = coding->suppress_error;
4471 ccl->last_block = coding->mode & CODING_MODE_LAST_BLOCK;
4472 if (encodep)
4474 /* On encoding, EOL format is converted within ccl_driver. For
4475 that, setup proper information in the structure CCL. */
4476 ccl->eol_type = coding->eol_type;
4477 if (ccl->eol_type ==CODING_EOL_UNDECIDED)
4478 ccl->eol_type = CODING_EOL_LF;
4479 ccl->cr_consumed = coding->spec.ccl.cr_carryover;
4481 ccl->multibyte = coding->src_multibyte;
4482 if (coding->spec.ccl.eight_bit_carryover[0] != 0)
4484 /* Move carryover bytes to DESTINATION. */
4485 unsigned char *p = coding->spec.ccl.eight_bit_carryover;
4486 while (*p)
4487 *dst++ = *p++;
4488 coding->spec.ccl.eight_bit_carryover[0] = 0;
4489 if (dst_bytes)
4490 dst_bytes -= dst - destination;
4493 coding->produced = (ccl_driver (ccl, source, dst, src_bytes, dst_bytes,
4494 &(coding->consumed))
4495 + dst - destination);
4497 if (encodep)
4499 coding->produced_char = coding->produced;
4500 coding->spec.ccl.cr_carryover = ccl->cr_consumed;
4502 else if (!ccl->eight_bit_control)
4504 /* The produced bytes forms a valid multibyte sequence. */
4505 coding->produced_char
4506 = multibyte_chars_in_text (destination, coding->produced);
4507 coding->spec.ccl.eight_bit_carryover[0] = 0;
4509 else
4511 /* On decoding, the destination should always multibyte. But,
4512 CCL program might have been generated an invalid multibyte
4513 sequence. Here we make such a sequence valid as
4514 multibyte. */
4515 int bytes
4516 = dst_bytes ? dst_bytes : source + coding->consumed - destination;
4518 if ((coding->consumed < src_bytes
4519 || !ccl->last_block)
4520 && coding->produced >= 1
4521 && destination[coding->produced - 1] >= 0x80)
4523 /* We should not convert the tailing 8-bit codes to
4524 multibyte form even if they doesn't form a valid
4525 multibyte sequence. They may form a valid sequence in
4526 the next call. */
4527 int carryover = 0;
4529 if (destination[coding->produced - 1] < 0xA0)
4530 carryover = 1;
4531 else if (coding->produced >= 2)
4533 if (destination[coding->produced - 2] >= 0x80)
4535 if (destination[coding->produced - 2] < 0xA0)
4536 carryover = 2;
4537 else if (coding->produced >= 3
4538 && destination[coding->produced - 3] >= 0x80
4539 && destination[coding->produced - 3] < 0xA0)
4540 carryover = 3;
4543 if (carryover > 0)
4545 BCOPY_SHORT (destination + coding->produced - carryover,
4546 coding->spec.ccl.eight_bit_carryover,
4547 carryover);
4548 coding->spec.ccl.eight_bit_carryover[carryover] = 0;
4549 coding->produced -= carryover;
4552 coding->produced = str_as_multibyte (destination, bytes,
4553 coding->produced,
4554 &(coding->produced_char));
4557 switch (ccl->status)
4559 case CCL_STAT_SUSPEND_BY_SRC:
4560 coding->result = CODING_FINISH_INSUFFICIENT_SRC;
4561 break;
4562 case CCL_STAT_SUSPEND_BY_DST:
4563 coding->result = CODING_FINISH_INSUFFICIENT_DST;
4564 break;
4565 case CCL_STAT_QUIT:
4566 case CCL_STAT_INVALID_CMD:
4567 coding->result = CODING_FINISH_INTERRUPT;
4568 break;
4569 default:
4570 coding->result = CODING_FINISH_NORMAL;
4571 break;
4573 return coding->result;
4576 /* Decode EOL format of the text at PTR of BYTES length destructively
4577 according to CODING->eol_type. This is called after the CCL
4578 program produced a decoded text at PTR. If we do CRLF->LF
4579 conversion, update CODING->produced and CODING->produced_char. */
4581 static void
4582 decode_eol_post_ccl (coding, ptr, bytes)
4583 struct coding_system *coding;
4584 unsigned char *ptr;
4585 int bytes;
4587 Lisp_Object val, saved_coding_symbol;
4588 unsigned char *pend = ptr + bytes;
4589 int dummy;
4591 /* Remember the current coding system symbol. We set it back when
4592 an inconsistent EOL is found so that `last-coding-system-used' is
4593 set to the coding system that doesn't specify EOL conversion. */
4594 saved_coding_symbol = coding->symbol;
4596 coding->spec.ccl.cr_carryover = 0;
4597 if (coding->eol_type == CODING_EOL_UNDECIDED)
4599 /* Here, to avoid the call of setup_coding_system, we directly
4600 call detect_eol_type. */
4601 coding->eol_type = detect_eol_type (ptr, bytes, &dummy);
4602 if (coding->eol_type == CODING_EOL_INCONSISTENT)
4603 coding->eol_type = CODING_EOL_LF;
4604 if (coding->eol_type != CODING_EOL_UNDECIDED)
4606 val = Fget (coding->symbol, Qeol_type);
4607 if (VECTORP (val) && XVECTOR (val)->size == 3)
4608 coding->symbol = XVECTOR (val)->contents[coding->eol_type];
4610 coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
4613 if (coding->eol_type == CODING_EOL_LF
4614 || coding->eol_type == CODING_EOL_UNDECIDED)
4616 /* We have nothing to do. */
4617 ptr = pend;
4619 else if (coding->eol_type == CODING_EOL_CRLF)
4621 unsigned char *pstart = ptr, *p = ptr;
4623 if (! (coding->mode & CODING_MODE_LAST_BLOCK)
4624 && *(pend - 1) == '\r')
4626 /* If the last character is CR, we can't handle it here
4627 because LF will be in the not-yet-decoded source text.
4628 Record that the CR is not yet processed. */
4629 coding->spec.ccl.cr_carryover = 1;
4630 coding->produced--;
4631 coding->produced_char--;
4632 pend--;
4634 while (ptr < pend)
4636 if (*ptr == '\r')
4638 if (ptr + 1 < pend && *(ptr + 1) == '\n')
4640 *p++ = '\n';
4641 ptr += 2;
4643 else
4645 if (coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
4646 goto undo_eol_conversion;
4647 *p++ = *ptr++;
4650 else if (*ptr == '\n'
4651 && coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
4652 goto undo_eol_conversion;
4653 else
4654 *p++ = *ptr++;
4655 continue;
4657 undo_eol_conversion:
4658 /* We have faced with inconsistent EOL format at PTR.
4659 Convert all LFs before PTR back to CRLFs. */
4660 for (p--, ptr--; p >= pstart; p--)
4662 if (*p == '\n')
4663 *ptr-- = '\n', *ptr-- = '\r';
4664 else
4665 *ptr-- = *p;
4667 /* If carryover is recorded, cancel it because we don't
4668 convert CRLF anymore. */
4669 if (coding->spec.ccl.cr_carryover)
4671 coding->spec.ccl.cr_carryover = 0;
4672 coding->produced++;
4673 coding->produced_char++;
4674 pend++;
4676 p = ptr = pend;
4677 coding->eol_type = CODING_EOL_LF;
4678 coding->symbol = saved_coding_symbol;
4680 if (p < pend)
4682 /* As each two-byte sequence CRLF was converted to LF, (PEND
4683 - P) is the number of deleted characters. */
4684 coding->produced -= pend - p;
4685 coding->produced_char -= pend - p;
4688 else /* i.e. coding->eol_type == CODING_EOL_CR */
4690 unsigned char *p = ptr;
4692 for (; ptr < pend; ptr++)
4694 if (*ptr == '\r')
4695 *ptr = '\n';
4696 else if (*ptr == '\n'
4697 && coding->mode & CODING_MODE_INHIBIT_INCONSISTENT_EOL)
4699 for (; p < ptr; p++)
4701 if (*p == '\n')
4702 *p = '\r';
4704 ptr = pend;
4705 coding->eol_type = CODING_EOL_LF;
4706 coding->symbol = saved_coding_symbol;
4712 /* See "GENERAL NOTES about `decode_coding_XXX ()' functions". Before
4713 decoding, it may detect coding system and format of end-of-line if
4714 those are not yet decided. The source should be unibyte, the
4715 result is multibyte if CODING->dst_multibyte is nonzero, else
4716 unibyte. */
4719 decode_coding (coding, source, destination, src_bytes, dst_bytes)
4720 struct coding_system *coding;
4721 const unsigned char *source;
4722 unsigned char *destination;
4723 int src_bytes, dst_bytes;
4725 int extra = 0;
4727 if (coding->type == coding_type_undecided)
4728 detect_coding (coding, source, src_bytes);
4730 if (coding->eol_type == CODING_EOL_UNDECIDED
4731 && coding->type != coding_type_ccl)
4733 detect_eol (coding, source, src_bytes);
4734 /* We had better recover the original eol format if we
4735 encounter an inconsistent eol format while decoding. */
4736 coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
4739 coding->produced = coding->produced_char = 0;
4740 coding->consumed = coding->consumed_char = 0;
4741 coding->errors = 0;
4742 coding->result = CODING_FINISH_NORMAL;
4744 switch (coding->type)
4746 case coding_type_sjis:
4747 decode_coding_sjis_big5 (coding, source, destination,
4748 src_bytes, dst_bytes, 1);
4749 break;
4751 case coding_type_iso2022:
4752 decode_coding_iso2022 (coding, source, destination,
4753 src_bytes, dst_bytes);
4754 break;
4756 case coding_type_big5:
4757 decode_coding_sjis_big5 (coding, source, destination,
4758 src_bytes, dst_bytes, 0);
4759 break;
4761 case coding_type_emacs_mule:
4762 decode_coding_emacs_mule (coding, source, destination,
4763 src_bytes, dst_bytes);
4764 break;
4766 case coding_type_ccl:
4767 if (coding->spec.ccl.cr_carryover)
4769 /* Put the CR which was not processed by the previous call
4770 of decode_eol_post_ccl in DESTINATION. It will be
4771 decoded together with the following LF by the call to
4772 decode_eol_post_ccl below. */
4773 *destination = '\r';
4774 coding->produced++;
4775 coding->produced_char++;
4776 dst_bytes--;
4777 extra = coding->spec.ccl.cr_carryover;
4779 ccl_coding_driver (coding, source, destination + extra,
4780 src_bytes, dst_bytes, 0);
4781 if (coding->eol_type != CODING_EOL_LF)
4783 coding->produced += extra;
4784 coding->produced_char += extra;
4785 decode_eol_post_ccl (coding, destination, coding->produced);
4787 break;
4789 default:
4790 decode_eol (coding, source, destination, src_bytes, dst_bytes);
4793 if (coding->result == CODING_FINISH_INSUFFICIENT_SRC
4794 && coding->mode & CODING_MODE_LAST_BLOCK
4795 && coding->consumed == src_bytes)
4796 coding->result = CODING_FINISH_NORMAL;
4798 if (coding->mode & CODING_MODE_LAST_BLOCK
4799 && coding->result == CODING_FINISH_INSUFFICIENT_SRC)
4801 const unsigned char *src = source + coding->consumed;
4802 unsigned char *dst = destination + coding->produced;
4804 src_bytes -= coding->consumed;
4805 coding->errors++;
4806 if (COMPOSING_P (coding))
4807 DECODE_COMPOSITION_END ('1');
4808 while (src_bytes--)
4810 int c = *src++;
4811 dst += CHAR_STRING (c, dst);
4812 coding->produced_char++;
4814 coding->consumed = coding->consumed_char = src - source;
4815 coding->produced = dst - destination;
4816 coding->result = CODING_FINISH_NORMAL;
4819 if (!coding->dst_multibyte)
4821 coding->produced = str_as_unibyte (destination, coding->produced);
4822 coding->produced_char = coding->produced;
4825 return coding->result;
4828 /* See "GENERAL NOTES about `encode_coding_XXX ()' functions". The
4829 multibyteness of the source is CODING->src_multibyte, the
4830 multibyteness of the result is always unibyte. */
4833 encode_coding (coding, source, destination, src_bytes, dst_bytes)
4834 struct coding_system *coding;
4835 const unsigned char *source;
4836 unsigned char *destination;
4837 int src_bytes, dst_bytes;
4839 coding->produced = coding->produced_char = 0;
4840 coding->consumed = coding->consumed_char = 0;
4841 coding->errors = 0;
4842 coding->result = CODING_FINISH_NORMAL;
4844 switch (coding->type)
4846 case coding_type_sjis:
4847 encode_coding_sjis_big5 (coding, source, destination,
4848 src_bytes, dst_bytes, 1);
4849 break;
4851 case coding_type_iso2022:
4852 encode_coding_iso2022 (coding, source, destination,
4853 src_bytes, dst_bytes);
4854 break;
4856 case coding_type_big5:
4857 encode_coding_sjis_big5 (coding, source, destination,
4858 src_bytes, dst_bytes, 0);
4859 break;
4861 case coding_type_emacs_mule:
4862 encode_coding_emacs_mule (coding, source, destination,
4863 src_bytes, dst_bytes);
4864 break;
4866 case coding_type_ccl:
4867 ccl_coding_driver (coding, source, destination,
4868 src_bytes, dst_bytes, 1);
4869 break;
4871 default:
4872 encode_eol (coding, source, destination, src_bytes, dst_bytes);
4875 if (coding->mode & CODING_MODE_LAST_BLOCK
4876 && coding->result == CODING_FINISH_INSUFFICIENT_SRC)
4878 const unsigned char *src = source + coding->consumed;
4879 unsigned char *dst = destination + coding->produced;
4881 if (coding->type == coding_type_iso2022)
4882 ENCODE_RESET_PLANE_AND_REGISTER;
4883 if (COMPOSING_P (coding))
4884 *dst++ = ISO_CODE_ESC, *dst++ = '1';
4885 if (coding->consumed < src_bytes)
4887 int len = src_bytes - coding->consumed;
4889 BCOPY_SHORT (src, dst, len);
4890 if (coding->src_multibyte)
4891 len = str_as_unibyte (dst, len);
4892 dst += len;
4893 coding->consumed = src_bytes;
4895 coding->produced = coding->produced_char = dst - destination;
4896 coding->result = CODING_FINISH_NORMAL;
4899 if (coding->result == CODING_FINISH_INSUFFICIENT_SRC
4900 && coding->consumed == src_bytes)
4901 coding->result = CODING_FINISH_NORMAL;
4903 return coding->result;
4906 /* Scan text in the region between *BEG and *END (byte positions),
4907 skip characters which we don't have to decode by coding system
4908 CODING at the head and tail, then set *BEG and *END to the region
4909 of the text we actually have to convert. The caller should move
4910 the gap out of the region in advance if the region is from a
4911 buffer.
4913 If STR is not NULL, *BEG and *END are indices into STR. */
4915 static void
4916 shrink_decoding_region (beg, end, coding, str)
4917 int *beg, *end;
4918 struct coding_system *coding;
4919 unsigned char *str;
4921 unsigned char *begp_orig, *begp, *endp_orig, *endp, c;
4922 int eol_conversion;
4923 Lisp_Object translation_table;
4925 if (coding->type == coding_type_ccl
4926 || coding->type == coding_type_undecided
4927 || coding->eol_type != CODING_EOL_LF
4928 || !NILP (coding->post_read_conversion)
4929 || coding->composing != COMPOSITION_DISABLED)
4931 /* We can't skip any data. */
4932 return;
4934 if (coding->type == coding_type_no_conversion
4935 || coding->type == coding_type_raw_text
4936 || coding->type == coding_type_emacs_mule)
4938 /* We need no conversion, but don't have to skip any data here.
4939 Decoding routine handles them effectively anyway. */
4940 return;
4943 translation_table = coding->translation_table_for_decode;
4944 if (NILP (translation_table) && !NILP (Venable_character_translation))
4945 translation_table = Vstandard_translation_table_for_decode;
4946 if (CHAR_TABLE_P (translation_table))
4948 int i;
4949 for (i = 0; i < 128; i++)
4950 if (!NILP (CHAR_TABLE_REF (translation_table, i)))
4951 break;
4952 if (i < 128)
4953 /* Some ASCII character should be translated. We give up
4954 shrinking. */
4955 return;
4958 if (coding->heading_ascii >= 0)
4959 /* Detection routine has already found how much we can skip at the
4960 head. */
4961 *beg += coding->heading_ascii;
4963 if (str)
4965 begp_orig = begp = str + *beg;
4966 endp_orig = endp = str + *end;
4968 else
4970 begp_orig = begp = BYTE_POS_ADDR (*beg);
4971 endp_orig = endp = begp + *end - *beg;
4974 eol_conversion = (coding->eol_type == CODING_EOL_CR
4975 || coding->eol_type == CODING_EOL_CRLF);
4977 switch (coding->type)
4979 case coding_type_sjis:
4980 case coding_type_big5:
4981 /* We can skip all ASCII characters at the head. */
4982 if (coding->heading_ascii < 0)
4984 if (eol_conversion)
4985 while (begp < endp && *begp < 0x80 && *begp != '\r') begp++;
4986 else
4987 while (begp < endp && *begp < 0x80) begp++;
4989 /* We can skip all ASCII characters at the tail except for the
4990 second byte of SJIS or BIG5 code. */
4991 if (eol_conversion)
4992 while (begp < endp && endp[-1] < 0x80 && endp[-1] != '\r') endp--;
4993 else
4994 while (begp < endp && endp[-1] < 0x80) endp--;
4995 /* Do not consider LF as ascii if preceded by CR, since that
4996 confuses eol decoding. */
4997 if (begp < endp && endp < endp_orig && endp[-1] == '\r' && endp[0] == '\n')
4998 endp++;
4999 if (begp < endp && endp < endp_orig && endp[-1] >= 0x80)
5000 endp++;
5001 break;
5003 case coding_type_iso2022:
5004 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, 0) != CHARSET_ASCII)
5005 /* We can't skip any data. */
5006 break;
5007 if (coding->heading_ascii < 0)
5009 /* We can skip all ASCII characters at the head except for a
5010 few control codes. */
5011 while (begp < endp && (c = *begp) < 0x80
5012 && c != ISO_CODE_CR && c != ISO_CODE_SO
5013 && c != ISO_CODE_SI && c != ISO_CODE_ESC
5014 && (!eol_conversion || c != ISO_CODE_LF))
5015 begp++;
5017 switch (coding->category_idx)
5019 case CODING_CATEGORY_IDX_ISO_8_1:
5020 case CODING_CATEGORY_IDX_ISO_8_2:
5021 /* We can skip all ASCII characters at the tail. */
5022 if (eol_conversion)
5023 while (begp < endp && (c = endp[-1]) < 0x80 && c != '\r') endp--;
5024 else
5025 while (begp < endp && endp[-1] < 0x80) endp--;
5026 /* Do not consider LF as ascii if preceded by CR, since that
5027 confuses eol decoding. */
5028 if (begp < endp && endp < endp_orig && endp[-1] == '\r' && endp[0] == '\n')
5029 endp++;
5030 break;
5032 case CODING_CATEGORY_IDX_ISO_7:
5033 case CODING_CATEGORY_IDX_ISO_7_TIGHT:
5035 /* We can skip all characters at the tail except for 8-bit
5036 codes and ESC and the following 2-byte at the tail. */
5037 unsigned char *eight_bit = NULL;
5039 if (eol_conversion)
5040 while (begp < endp
5041 && (c = endp[-1]) != ISO_CODE_ESC && c != '\r')
5043 if (!eight_bit && c & 0x80) eight_bit = endp;
5044 endp--;
5046 else
5047 while (begp < endp
5048 && (c = endp[-1]) != ISO_CODE_ESC)
5050 if (!eight_bit && c & 0x80) eight_bit = endp;
5051 endp--;
5053 /* Do not consider LF as ascii if preceded by CR, since that
5054 confuses eol decoding. */
5055 if (begp < endp && endp < endp_orig
5056 && endp[-1] == '\r' && endp[0] == '\n')
5057 endp++;
5058 if (begp < endp && endp[-1] == ISO_CODE_ESC)
5060 if (endp + 1 < endp_orig && end[0] == '(' && end[1] == 'B')
5061 /* This is an ASCII designation sequence. We can
5062 surely skip the tail. But, if we have
5063 encountered an 8-bit code, skip only the codes
5064 after that. */
5065 endp = eight_bit ? eight_bit : endp + 2;
5066 else
5067 /* Hmmm, we can't skip the tail. */
5068 endp = endp_orig;
5070 else if (eight_bit)
5071 endp = eight_bit;
5074 break;
5076 default:
5077 abort ();
5079 *beg += begp - begp_orig;
5080 *end += endp - endp_orig;
5081 return;
5084 /* Like shrink_decoding_region but for encoding. */
5086 static void
5087 shrink_encoding_region (beg, end, coding, str)
5088 int *beg, *end;
5089 struct coding_system *coding;
5090 unsigned char *str;
5092 unsigned char *begp_orig, *begp, *endp_orig, *endp;
5093 int eol_conversion;
5094 Lisp_Object translation_table;
5096 if (coding->type == coding_type_ccl
5097 || coding->eol_type == CODING_EOL_CRLF
5098 || coding->eol_type == CODING_EOL_CR
5099 || (coding->cmp_data && coding->cmp_data->used > 0))
5101 /* We can't skip any data. */
5102 return;
5104 if (coding->type == coding_type_no_conversion
5105 || coding->type == coding_type_raw_text
5106 || coding->type == coding_type_emacs_mule
5107 || coding->type == coding_type_undecided)
5109 /* We need no conversion, but don't have to skip any data here.
5110 Encoding routine handles them effectively anyway. */
5111 return;
5114 translation_table = coding->translation_table_for_encode;
5115 if (NILP (translation_table) && !NILP (Venable_character_translation))
5116 translation_table = Vstandard_translation_table_for_encode;
5117 if (CHAR_TABLE_P (translation_table))
5119 int i;
5120 for (i = 0; i < 128; i++)
5121 if (!NILP (CHAR_TABLE_REF (translation_table, i)))
5122 break;
5123 if (i < 128)
5124 /* Some ASCII character should be translated. We give up
5125 shrinking. */
5126 return;
5129 if (str)
5131 begp_orig = begp = str + *beg;
5132 endp_orig = endp = str + *end;
5134 else
5136 begp_orig = begp = BYTE_POS_ADDR (*beg);
5137 endp_orig = endp = begp + *end - *beg;
5140 eol_conversion = (coding->eol_type == CODING_EOL_CR
5141 || coding->eol_type == CODING_EOL_CRLF);
5143 /* Here, we don't have to check coding->pre_write_conversion because
5144 the caller is expected to have handled it already. */
5145 switch (coding->type)
5147 case coding_type_iso2022:
5148 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, 0) != CHARSET_ASCII)
5149 /* We can't skip any data. */
5150 break;
5151 if (coding->flags & CODING_FLAG_ISO_DESIGNATE_AT_BOL)
5153 unsigned char *bol = begp;
5154 while (begp < endp && *begp < 0x80)
5156 begp++;
5157 if (begp[-1] == '\n')
5158 bol = begp;
5160 begp = bol;
5161 goto label_skip_tail;
5163 /* fall down ... */
5165 case coding_type_sjis:
5166 case coding_type_big5:
5167 /* We can skip all ASCII characters at the head and tail. */
5168 if (eol_conversion)
5169 while (begp < endp && *begp < 0x80 && *begp != '\n') begp++;
5170 else
5171 while (begp < endp && *begp < 0x80) begp++;
5172 label_skip_tail:
5173 if (eol_conversion)
5174 while (begp < endp && endp[-1] < 0x80 && endp[-1] != '\n') endp--;
5175 else
5176 while (begp < endp && *(endp - 1) < 0x80) endp--;
5177 break;
5179 default:
5180 abort ();
5183 *beg += begp - begp_orig;
5184 *end += endp - endp_orig;
5185 return;
5188 /* As shrinking conversion region requires some overhead, we don't try
5189 shrinking if the length of conversion region is less than this
5190 value. */
5191 static int shrink_conversion_region_threshhold = 1024;
5193 #define SHRINK_CONVERSION_REGION(beg, end, coding, str, encodep) \
5194 do { \
5195 if (*(end) - *(beg) > shrink_conversion_region_threshhold) \
5197 if (encodep) shrink_encoding_region (beg, end, coding, str); \
5198 else shrink_decoding_region (beg, end, coding, str); \
5200 } while (0)
5202 static Lisp_Object
5203 code_convert_region_unwind (dummy)
5204 Lisp_Object dummy;
5206 inhibit_pre_post_conversion = 0;
5207 return Qnil;
5210 /* Store information about all compositions in the range FROM and TO
5211 of OBJ in memory blocks pointed by CODING->cmp_data. OBJ is a
5212 buffer or a string, defaults to the current buffer. */
5214 void
5215 coding_save_composition (coding, from, to, obj)
5216 struct coding_system *coding;
5217 int from, to;
5218 Lisp_Object obj;
5220 Lisp_Object prop;
5221 int start, end;
5223 if (coding->composing == COMPOSITION_DISABLED)
5224 return;
5225 if (!coding->cmp_data)
5226 coding_allocate_composition_data (coding, from);
5227 if (!find_composition (from, to, &start, &end, &prop, obj)
5228 || end > to)
5229 return;
5230 if (start < from
5231 && (!find_composition (end, to, &start, &end, &prop, obj)
5232 || end > to))
5233 return;
5234 coding->composing = COMPOSITION_NO;
5237 if (COMPOSITION_VALID_P (start, end, prop))
5239 enum composition_method method = COMPOSITION_METHOD (prop);
5240 if (coding->cmp_data->used + COMPOSITION_DATA_MAX_BUNCH_LENGTH
5241 >= COMPOSITION_DATA_SIZE)
5242 coding_allocate_composition_data (coding, from);
5243 /* For relative composition, we remember start and end
5244 positions, for the other compositions, we also remember
5245 components. */
5246 CODING_ADD_COMPOSITION_START (coding, start - from, method);
5247 if (method != COMPOSITION_RELATIVE)
5249 /* We must store a*/
5250 Lisp_Object val, ch;
5252 val = COMPOSITION_COMPONENTS (prop);
5253 if (CONSP (val))
5254 while (CONSP (val))
5256 ch = XCAR (val), val = XCDR (val);
5257 CODING_ADD_COMPOSITION_COMPONENT (coding, XINT (ch));
5259 else if (VECTORP (val) || STRINGP (val))
5261 int len = (VECTORP (val)
5262 ? XVECTOR (val)->size : SCHARS (val));
5263 int i;
5264 for (i = 0; i < len; i++)
5266 ch = (STRINGP (val)
5267 ? Faref (val, make_number (i))
5268 : XVECTOR (val)->contents[i]);
5269 CODING_ADD_COMPOSITION_COMPONENT (coding, XINT (ch));
5272 else /* INTEGERP (val) */
5273 CODING_ADD_COMPOSITION_COMPONENT (coding, XINT (val));
5275 CODING_ADD_COMPOSITION_END (coding, end - from);
5277 start = end;
5279 while (start < to
5280 && find_composition (start, to, &start, &end, &prop, obj)
5281 && end <= to);
5283 /* Make coding->cmp_data point to the first memory block. */
5284 while (coding->cmp_data->prev)
5285 coding->cmp_data = coding->cmp_data->prev;
5286 coding->cmp_data_start = 0;
5289 /* Reflect the saved information about compositions to OBJ.
5290 CODING->cmp_data points to a memory block for the information. OBJ
5291 is a buffer or a string, defaults to the current buffer. */
5293 void
5294 coding_restore_composition (coding, obj)
5295 struct coding_system *coding;
5296 Lisp_Object obj;
5298 struct composition_data *cmp_data = coding->cmp_data;
5300 if (!cmp_data)
5301 return;
5303 while (cmp_data->prev)
5304 cmp_data = cmp_data->prev;
5306 while (cmp_data)
5308 int i;
5310 for (i = 0; i < cmp_data->used && cmp_data->data[i] > 0;
5311 i += cmp_data->data[i])
5313 int *data = cmp_data->data + i;
5314 enum composition_method method = (enum composition_method) data[3];
5315 Lisp_Object components;
5317 if (method == COMPOSITION_RELATIVE)
5318 components = Qnil;
5319 else
5321 int len = data[0] - 4, j;
5322 Lisp_Object args[MAX_COMPOSITION_COMPONENTS * 2 - 1];
5324 if (method == COMPOSITION_WITH_RULE_ALTCHARS
5325 && len % 2 == 0)
5326 len --;
5327 for (j = 0; j < len; j++)
5328 args[j] = make_number (data[4 + j]);
5329 components = (method == COMPOSITION_WITH_ALTCHARS
5330 ? Fstring (len, args) : Fvector (len, args));
5332 compose_text (data[1], data[2], components, Qnil, obj);
5334 cmp_data = cmp_data->next;
5338 /* Decode (if ENCODEP is zero) or encode (if ENCODEP is nonzero) the
5339 text from FROM to TO (byte positions are FROM_BYTE and TO_BYTE) by
5340 coding system CODING, and return the status code of code conversion
5341 (currently, this value has no meaning).
5343 How many characters (and bytes) are converted to how many
5344 characters (and bytes) are recorded in members of the structure
5345 CODING.
5347 If REPLACE is nonzero, we do various things as if the original text
5348 is deleted and a new text is inserted. See the comments in
5349 replace_range (insdel.c) to know what we are doing.
5351 If REPLACE is zero, it is assumed that the source text is unibyte.
5352 Otherwise, it is assumed that the source text is multibyte. */
5355 code_convert_region (from, from_byte, to, to_byte, coding, encodep, replace)
5356 int from, from_byte, to, to_byte, encodep, replace;
5357 struct coding_system *coding;
5359 int len = to - from, len_byte = to_byte - from_byte;
5360 int nchars_del = 0, nbytes_del = 0;
5361 int require, inserted, inserted_byte;
5362 int head_skip, tail_skip, total_skip = 0;
5363 Lisp_Object saved_coding_symbol;
5364 int first = 1;
5365 unsigned char *src, *dst;
5366 Lisp_Object deletion;
5367 int orig_point = PT, orig_len = len;
5368 int prev_Z;
5369 int multibyte_p = !NILP (current_buffer->enable_multibyte_characters);
5371 deletion = Qnil;
5372 saved_coding_symbol = coding->symbol;
5374 if (from < PT && PT < to)
5376 TEMP_SET_PT_BOTH (from, from_byte);
5377 orig_point = from;
5380 if (replace)
5382 int saved_from = from;
5383 int saved_inhibit_modification_hooks;
5385 prepare_to_modify_buffer (from, to, &from);
5386 if (saved_from != from)
5388 to = from + len;
5389 from_byte = CHAR_TO_BYTE (from), to_byte = CHAR_TO_BYTE (to);
5390 len_byte = to_byte - from_byte;
5393 /* The code conversion routine can not preserve text properties
5394 for now. So, we must remove all text properties in the
5395 region. Here, we must suppress all modification hooks. */
5396 saved_inhibit_modification_hooks = inhibit_modification_hooks;
5397 inhibit_modification_hooks = 1;
5398 Fset_text_properties (make_number (from), make_number (to), Qnil, Qnil);
5399 inhibit_modification_hooks = saved_inhibit_modification_hooks;
5402 if (! encodep && CODING_REQUIRE_DETECTION (coding))
5404 /* We must detect encoding of text and eol format. */
5406 if (from < GPT && to > GPT)
5407 move_gap_both (from, from_byte);
5408 if (coding->type == coding_type_undecided)
5410 detect_coding (coding, BYTE_POS_ADDR (from_byte), len_byte);
5411 if (coding->type == coding_type_undecided)
5413 /* It seems that the text contains only ASCII, but we
5414 should not leave it undecided because the deeper
5415 decoding routine (decode_coding) tries to detect the
5416 encodings again in vain. */
5417 coding->type = coding_type_emacs_mule;
5418 coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
5419 /* As emacs-mule decoder will handle composition, we
5420 need this setting to allocate coding->cmp_data
5421 later. */
5422 coding->composing = COMPOSITION_NO;
5425 if (coding->eol_type == CODING_EOL_UNDECIDED
5426 && coding->type != coding_type_ccl)
5428 detect_eol (coding, BYTE_POS_ADDR (from_byte), len_byte);
5429 if (coding->eol_type == CODING_EOL_UNDECIDED)
5430 coding->eol_type = CODING_EOL_LF;
5431 /* We had better recover the original eol format if we
5432 encounter an inconsistent eol format while decoding. */
5433 coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
5437 /* Now we convert the text. */
5439 /* For encoding, we must process pre-write-conversion in advance. */
5440 if (! inhibit_pre_post_conversion
5441 && encodep
5442 && SYMBOLP (coding->pre_write_conversion)
5443 && ! NILP (Ffboundp (coding->pre_write_conversion)))
5445 /* The function in pre-write-conversion may put a new text in a
5446 new buffer. */
5447 struct buffer *prev = current_buffer;
5448 Lisp_Object new;
5450 record_unwind_protect (code_convert_region_unwind, Qnil);
5451 /* We should not call any more pre-write/post-read-conversion
5452 functions while this pre-write-conversion is running. */
5453 inhibit_pre_post_conversion = 1;
5454 call2 (coding->pre_write_conversion,
5455 make_number (from), make_number (to));
5456 inhibit_pre_post_conversion = 0;
5457 /* Discard the unwind protect. */
5458 specpdl_ptr--;
5460 if (current_buffer != prev)
5462 len = ZV - BEGV;
5463 new = Fcurrent_buffer ();
5464 set_buffer_internal_1 (prev);
5465 del_range_2 (from, from_byte, to, to_byte, 0);
5466 TEMP_SET_PT_BOTH (from, from_byte);
5467 insert_from_buffer (XBUFFER (new), 1, len, 0);
5468 Fkill_buffer (new);
5469 if (orig_point >= to)
5470 orig_point += len - orig_len;
5471 else if (orig_point > from)
5472 orig_point = from;
5473 orig_len = len;
5474 to = from + len;
5475 from_byte = CHAR_TO_BYTE (from);
5476 to_byte = CHAR_TO_BYTE (to);
5477 len_byte = to_byte - from_byte;
5478 TEMP_SET_PT_BOTH (from, from_byte);
5482 if (replace)
5484 if (! EQ (current_buffer->undo_list, Qt))
5485 deletion = make_buffer_string_both (from, from_byte, to, to_byte, 1);
5486 else
5488 nchars_del = to - from;
5489 nbytes_del = to_byte - from_byte;
5493 if (coding->composing != COMPOSITION_DISABLED)
5495 if (encodep)
5496 coding_save_composition (coding, from, to, Fcurrent_buffer ());
5497 else
5498 coding_allocate_composition_data (coding, from);
5501 /* Try to skip the heading and tailing ASCIIs. */
5502 if (coding->type != coding_type_ccl)
5504 int from_byte_orig = from_byte, to_byte_orig = to_byte;
5506 if (from < GPT && GPT < to)
5507 move_gap_both (from, from_byte);
5508 SHRINK_CONVERSION_REGION (&from_byte, &to_byte, coding, NULL, encodep);
5509 if (from_byte == to_byte
5510 && (encodep || NILP (coding->post_read_conversion))
5511 && ! CODING_REQUIRE_FLUSHING (coding))
5513 coding->produced = len_byte;
5514 coding->produced_char = len;
5515 if (!replace)
5516 /* We must record and adjust for this new text now. */
5517 adjust_after_insert (from, from_byte_orig, to, to_byte_orig, len);
5518 return 0;
5521 head_skip = from_byte - from_byte_orig;
5522 tail_skip = to_byte_orig - to_byte;
5523 total_skip = head_skip + tail_skip;
5524 from += head_skip;
5525 to -= tail_skip;
5526 len -= total_skip; len_byte -= total_skip;
5529 /* For conversion, we must put the gap before the text in addition to
5530 making the gap larger for efficient decoding. The required gap
5531 size starts from 2000 which is the magic number used in make_gap.
5532 But, after one batch of conversion, it will be incremented if we
5533 find that it is not enough . */
5534 require = 2000;
5536 if (GAP_SIZE < require)
5537 make_gap (require - GAP_SIZE);
5538 move_gap_both (from, from_byte);
5540 inserted = inserted_byte = 0;
5542 GAP_SIZE += len_byte;
5543 ZV -= len;
5544 Z -= len;
5545 ZV_BYTE -= len_byte;
5546 Z_BYTE -= len_byte;
5548 if (GPT - BEG < BEG_UNCHANGED)
5549 BEG_UNCHANGED = GPT - BEG;
5550 if (Z - GPT < END_UNCHANGED)
5551 END_UNCHANGED = Z - GPT;
5553 if (!encodep && coding->src_multibyte)
5555 /* Decoding routines expects that the source text is unibyte.
5556 We must convert 8-bit characters of multibyte form to
5557 unibyte. */
5558 int len_byte_orig = len_byte;
5559 len_byte = str_as_unibyte (GAP_END_ADDR - len_byte, len_byte);
5560 if (len_byte < len_byte_orig)
5561 safe_bcopy (GAP_END_ADDR - len_byte_orig, GAP_END_ADDR - len_byte,
5562 len_byte);
5563 coding->src_multibyte = 0;
5566 for (;;)
5568 int result;
5570 /* The buffer memory is now:
5571 +--------+converted-text+---------+-------original-text-------+---+
5572 |<-from->|<--inserted-->|---------|<--------len_byte--------->|---|
5573 |<---------------------- GAP ----------------------->| */
5574 src = GAP_END_ADDR - len_byte;
5575 dst = GPT_ADDR + inserted_byte;
5577 if (encodep)
5578 result = encode_coding (coding, src, dst, len_byte, 0);
5579 else
5581 if (coding->composing != COMPOSITION_DISABLED)
5582 coding->cmp_data->char_offset = from + inserted;
5583 result = decode_coding (coding, src, dst, len_byte, 0);
5586 /* The buffer memory is now:
5587 +--------+-------converted-text----+--+------original-text----+---+
5588 |<-from->|<-inserted->|<-produced->|--|<-(len_byte-consumed)->|---|
5589 |<---------------------- GAP ----------------------->| */
5591 inserted += coding->produced_char;
5592 inserted_byte += coding->produced;
5593 len_byte -= coding->consumed;
5595 if (result == CODING_FINISH_INSUFFICIENT_CMP)
5597 coding_allocate_composition_data (coding, from + inserted);
5598 continue;
5601 src += coding->consumed;
5602 dst += coding->produced;
5604 if (result == CODING_FINISH_NORMAL)
5606 src += len_byte;
5607 break;
5609 if (! encodep && result == CODING_FINISH_INCONSISTENT_EOL)
5611 unsigned char *pend = dst, *p = pend - inserted_byte;
5612 Lisp_Object eol_type;
5614 /* Encode LFs back to the original eol format (CR or CRLF). */
5615 if (coding->eol_type == CODING_EOL_CR)
5617 while (p < pend) if (*p++ == '\n') p[-1] = '\r';
5619 else
5621 int count = 0;
5623 while (p < pend) if (*p++ == '\n') count++;
5624 if (src - dst < count)
5626 /* We don't have sufficient room for encoding LFs
5627 back to CRLF. We must record converted and
5628 not-yet-converted text back to the buffer
5629 content, enlarge the gap, then record them out of
5630 the buffer contents again. */
5631 int add = len_byte + inserted_byte;
5633 GAP_SIZE -= add;
5634 ZV += add; Z += add; ZV_BYTE += add; Z_BYTE += add;
5635 GPT += inserted_byte; GPT_BYTE += inserted_byte;
5636 make_gap (count - GAP_SIZE);
5637 GAP_SIZE += add;
5638 ZV -= add; Z -= add; ZV_BYTE -= add; Z_BYTE -= add;
5639 GPT -= inserted_byte; GPT_BYTE -= inserted_byte;
5640 /* Don't forget to update SRC, DST, and PEND. */
5641 src = GAP_END_ADDR - len_byte;
5642 dst = GPT_ADDR + inserted_byte;
5643 pend = dst;
5645 inserted += count;
5646 inserted_byte += count;
5647 coding->produced += count;
5648 p = dst = pend + count;
5649 while (count)
5651 *--p = *--pend;
5652 if (*p == '\n') count--, *--p = '\r';
5656 /* Suppress eol-format conversion in the further conversion. */
5657 coding->eol_type = CODING_EOL_LF;
5659 /* Set the coding system symbol to that for Unix-like EOL. */
5660 eol_type = Fget (saved_coding_symbol, Qeol_type);
5661 if (VECTORP (eol_type)
5662 && XVECTOR (eol_type)->size == 3
5663 && SYMBOLP (XVECTOR (eol_type)->contents[CODING_EOL_LF]))
5664 coding->symbol = XVECTOR (eol_type)->contents[CODING_EOL_LF];
5665 else
5666 coding->symbol = saved_coding_symbol;
5668 continue;
5670 if (len_byte <= 0)
5672 if (coding->type != coding_type_ccl
5673 || coding->mode & CODING_MODE_LAST_BLOCK)
5674 break;
5675 coding->mode |= CODING_MODE_LAST_BLOCK;
5676 continue;
5678 if (result == CODING_FINISH_INSUFFICIENT_SRC)
5680 /* The source text ends in invalid codes. Let's just
5681 make them valid buffer contents, and finish conversion. */
5682 if (multibyte_p)
5684 unsigned char *start = dst;
5686 inserted += len_byte;
5687 while (len_byte--)
5689 int c = *src++;
5690 dst += CHAR_STRING (c, dst);
5693 inserted_byte += dst - start;
5695 else
5697 inserted += len_byte;
5698 inserted_byte += len_byte;
5699 while (len_byte--)
5700 *dst++ = *src++;
5702 break;
5704 if (result == CODING_FINISH_INTERRUPT)
5706 /* The conversion procedure was interrupted by a user. */
5707 break;
5709 /* Now RESULT == CODING_FINISH_INSUFFICIENT_DST */
5710 if (coding->consumed < 1)
5712 /* It's quite strange to require more memory without
5713 consuming any bytes. Perhaps CCL program bug. */
5714 break;
5716 if (first)
5718 /* We have just done the first batch of conversion which was
5719 stopped because of insufficient gap. Let's reconsider the
5720 required gap size (i.e. SRT - DST) now.
5722 We have converted ORIG bytes (== coding->consumed) into
5723 NEW bytes (coding->produced). To convert the remaining
5724 LEN bytes, we may need REQUIRE bytes of gap, where:
5725 REQUIRE + LEN_BYTE = LEN_BYTE * (NEW / ORIG)
5726 REQUIRE = LEN_BYTE * (NEW - ORIG) / ORIG
5727 Here, we are sure that NEW >= ORIG. */
5728 float ratio;
5730 if (coding->produced <= coding->consumed)
5732 /* This happens because of CCL-based coding system with
5733 eol-type CRLF. */
5734 require = 0;
5736 else
5738 ratio = (coding->produced - coding->consumed) / coding->consumed;
5739 require = len_byte * ratio;
5741 first = 0;
5743 if ((src - dst) < (require + 2000))
5745 /* See the comment above the previous call of make_gap. */
5746 int add = len_byte + inserted_byte;
5748 GAP_SIZE -= add;
5749 ZV += add; Z += add; ZV_BYTE += add; Z_BYTE += add;
5750 GPT += inserted_byte; GPT_BYTE += inserted_byte;
5751 make_gap (require + 2000);
5752 GAP_SIZE += add;
5753 ZV -= add; Z -= add; ZV_BYTE -= add; Z_BYTE -= add;
5754 GPT -= inserted_byte; GPT_BYTE -= inserted_byte;
5757 if (src - dst > 0) *dst = 0; /* Put an anchor. */
5759 if (encodep && coding->dst_multibyte)
5761 /* The output is unibyte. We must convert 8-bit characters to
5762 multibyte form. */
5763 if (inserted_byte * 2 > GAP_SIZE)
5765 GAP_SIZE -= inserted_byte;
5766 ZV += inserted_byte; Z += inserted_byte;
5767 ZV_BYTE += inserted_byte; Z_BYTE += inserted_byte;
5768 GPT += inserted_byte; GPT_BYTE += inserted_byte;
5769 make_gap (inserted_byte - GAP_SIZE);
5770 GAP_SIZE += inserted_byte;
5771 ZV -= inserted_byte; Z -= inserted_byte;
5772 ZV_BYTE -= inserted_byte; Z_BYTE -= inserted_byte;
5773 GPT -= inserted_byte; GPT_BYTE -= inserted_byte;
5775 inserted_byte = str_to_multibyte (GPT_ADDR, GAP_SIZE, inserted_byte);
5778 /* If we shrank the conversion area, adjust it now. */
5779 if (total_skip > 0)
5781 if (tail_skip > 0)
5782 safe_bcopy (GAP_END_ADDR, GPT_ADDR + inserted_byte, tail_skip);
5783 inserted += total_skip; inserted_byte += total_skip;
5784 GAP_SIZE += total_skip;
5785 GPT -= head_skip; GPT_BYTE -= head_skip;
5786 ZV -= total_skip; ZV_BYTE -= total_skip;
5787 Z -= total_skip; Z_BYTE -= total_skip;
5788 from -= head_skip; from_byte -= head_skip;
5789 to += tail_skip; to_byte += tail_skip;
5792 prev_Z = Z;
5793 if (! EQ (current_buffer->undo_list, Qt))
5794 adjust_after_replace (from, from_byte, deletion, inserted, inserted_byte);
5795 else
5796 adjust_after_replace_noundo (from, from_byte, nchars_del, nbytes_del,
5797 inserted, inserted_byte);
5798 inserted = Z - prev_Z;
5800 if (!encodep && coding->cmp_data && coding->cmp_data->used)
5801 coding_restore_composition (coding, Fcurrent_buffer ());
5802 coding_free_composition_data (coding);
5804 if (! inhibit_pre_post_conversion
5805 && ! encodep && ! NILP (coding->post_read_conversion))
5807 Lisp_Object val;
5809 if (from != PT)
5810 TEMP_SET_PT_BOTH (from, from_byte);
5811 prev_Z = Z;
5812 record_unwind_protect (code_convert_region_unwind, Qnil);
5813 /* We should not call any more pre-write/post-read-conversion
5814 functions while this post-read-conversion is running. */
5815 inhibit_pre_post_conversion = 1;
5816 val = call1 (coding->post_read_conversion, make_number (inserted));
5817 inhibit_pre_post_conversion = 0;
5818 /* Discard the unwind protect. */
5819 specpdl_ptr--;
5820 CHECK_NUMBER (val);
5821 inserted += Z - prev_Z;
5824 if (orig_point >= from)
5826 if (orig_point >= from + orig_len)
5827 orig_point += inserted - orig_len;
5828 else
5829 orig_point = from;
5830 TEMP_SET_PT (orig_point);
5833 if (replace)
5835 signal_after_change (from, to - from, inserted);
5836 update_compositions (from, from + inserted, CHECK_BORDER);
5840 coding->consumed = to_byte - from_byte;
5841 coding->consumed_char = to - from;
5842 coding->produced = inserted_byte;
5843 coding->produced_char = inserted;
5846 return 0;
5849 Lisp_Object
5850 run_pre_post_conversion_on_str (str, coding, encodep)
5851 Lisp_Object str;
5852 struct coding_system *coding;
5853 int encodep;
5855 int count = SPECPDL_INDEX ();
5856 struct gcpro gcpro1, gcpro2;
5857 int multibyte = STRING_MULTIBYTE (str);
5858 Lisp_Object buffer;
5859 struct buffer *buf;
5860 Lisp_Object old_deactivate_mark;
5862 record_unwind_protect (Fset_buffer, Fcurrent_buffer ());
5863 record_unwind_protect (code_convert_region_unwind, Qnil);
5864 /* It is not crucial to specbind this. */
5865 old_deactivate_mark = Vdeactivate_mark;
5866 GCPRO2 (str, old_deactivate_mark);
5868 buffer = Fget_buffer_create (build_string (" *code-converting-work*"));
5869 buf = XBUFFER (buffer);
5871 buf->directory = current_buffer->directory;
5872 buf->read_only = Qnil;
5873 buf->filename = Qnil;
5874 buf->undo_list = Qt;
5875 buf->overlays_before = Qnil;
5876 buf->overlays_after = Qnil;
5878 set_buffer_internal (buf);
5879 /* We must insert the contents of STR as is without
5880 unibyte<->multibyte conversion. For that, we adjust the
5881 multibyteness of the working buffer to that of STR. */
5882 Ferase_buffer ();
5883 buf->enable_multibyte_characters = multibyte ? Qt : Qnil;
5885 insert_from_string (str, 0, 0,
5886 SCHARS (str), SBYTES (str), 0);
5887 UNGCPRO;
5888 inhibit_pre_post_conversion = 1;
5889 if (encodep)
5890 call2 (coding->pre_write_conversion, make_number (BEG), make_number (Z));
5891 else
5893 TEMP_SET_PT_BOTH (BEG, BEG_BYTE);
5894 call1 (coding->post_read_conversion, make_number (Z - BEG));
5896 inhibit_pre_post_conversion = 0;
5897 Vdeactivate_mark = old_deactivate_mark;
5898 str = make_buffer_string (BEG, Z, 1);
5899 return unbind_to (count, str);
5902 Lisp_Object
5903 decode_coding_string (str, coding, nocopy)
5904 Lisp_Object str;
5905 struct coding_system *coding;
5906 int nocopy;
5908 int len;
5909 struct conversion_buffer buf;
5910 int from, to_byte;
5911 Lisp_Object saved_coding_symbol;
5912 int result;
5913 int require_decoding;
5914 int shrinked_bytes = 0;
5915 Lisp_Object newstr;
5916 int consumed, consumed_char, produced, produced_char;
5918 from = 0;
5919 to_byte = SBYTES (str);
5921 saved_coding_symbol = coding->symbol;
5922 coding->src_multibyte = STRING_MULTIBYTE (str);
5923 coding->dst_multibyte = 1;
5924 if (CODING_REQUIRE_DETECTION (coding))
5926 /* See the comments in code_convert_region. */
5927 if (coding->type == coding_type_undecided)
5929 detect_coding (coding, SDATA (str), to_byte);
5930 if (coding->type == coding_type_undecided)
5932 coding->type = coding_type_emacs_mule;
5933 coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
5934 /* As emacs-mule decoder will handle composition, we
5935 need this setting to allocate coding->cmp_data
5936 later. */
5937 coding->composing = COMPOSITION_NO;
5940 if (coding->eol_type == CODING_EOL_UNDECIDED
5941 && coding->type != coding_type_ccl)
5943 saved_coding_symbol = coding->symbol;
5944 detect_eol (coding, SDATA (str), to_byte);
5945 if (coding->eol_type == CODING_EOL_UNDECIDED)
5946 coding->eol_type = CODING_EOL_LF;
5947 /* We had better recover the original eol format if we
5948 encounter an inconsistent eol format while decoding. */
5949 coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
5953 if (coding->type == coding_type_no_conversion
5954 || coding->type == coding_type_raw_text)
5955 coding->dst_multibyte = 0;
5957 require_decoding = CODING_REQUIRE_DECODING (coding);
5959 if (STRING_MULTIBYTE (str))
5961 /* Decoding routines expect the source text to be unibyte. */
5962 str = Fstring_as_unibyte (str);
5963 to_byte = SBYTES (str);
5964 nocopy = 1;
5965 coding->src_multibyte = 0;
5968 /* Try to skip the heading and tailing ASCIIs. */
5969 if (require_decoding && coding->type != coding_type_ccl)
5971 SHRINK_CONVERSION_REGION (&from, &to_byte, coding, SDATA (str),
5973 if (from == to_byte)
5974 require_decoding = 0;
5975 shrinked_bytes = from + (SBYTES (str) - to_byte);
5978 if (!require_decoding)
5980 coding->consumed = SBYTES (str);
5981 coding->consumed_char = SCHARS (str);
5982 if (coding->dst_multibyte)
5984 str = Fstring_as_multibyte (str);
5985 nocopy = 1;
5987 coding->produced = SBYTES (str);
5988 coding->produced_char = SCHARS (str);
5989 return (nocopy ? str : Fcopy_sequence (str));
5992 if (coding->composing != COMPOSITION_DISABLED)
5993 coding_allocate_composition_data (coding, from);
5994 len = decoding_buffer_size (coding, to_byte - from);
5995 allocate_conversion_buffer (buf, len);
5997 consumed = consumed_char = produced = produced_char = 0;
5998 while (1)
6000 result = decode_coding (coding, SDATA (str) + from + consumed,
6001 buf.data + produced, to_byte - from - consumed,
6002 buf.size - produced);
6003 consumed += coding->consumed;
6004 consumed_char += coding->consumed_char;
6005 produced += coding->produced;
6006 produced_char += coding->produced_char;
6007 if (result == CODING_FINISH_NORMAL
6008 || (result == CODING_FINISH_INSUFFICIENT_SRC
6009 && coding->consumed == 0))
6010 break;
6011 if (result == CODING_FINISH_INSUFFICIENT_CMP)
6012 coding_allocate_composition_data (coding, from + produced_char);
6013 else if (result == CODING_FINISH_INSUFFICIENT_DST)
6014 extend_conversion_buffer (&buf);
6015 else if (result == CODING_FINISH_INCONSISTENT_EOL)
6017 Lisp_Object eol_type;
6019 /* Recover the original EOL format. */
6020 if (coding->eol_type == CODING_EOL_CR)
6022 unsigned char *p;
6023 for (p = buf.data; p < buf.data + produced; p++)
6024 if (*p == '\n') *p = '\r';
6026 else if (coding->eol_type == CODING_EOL_CRLF)
6028 int num_eol = 0;
6029 unsigned char *p0, *p1;
6030 for (p0 = buf.data, p1 = p0 + produced; p0 < p1; p0++)
6031 if (*p0 == '\n') num_eol++;
6032 if (produced + num_eol >= buf.size)
6033 extend_conversion_buffer (&buf);
6034 for (p0 = buf.data + produced, p1 = p0 + num_eol; p0 > buf.data;)
6036 *--p1 = *--p0;
6037 if (*p0 == '\n') *--p1 = '\r';
6039 produced += num_eol;
6040 produced_char += num_eol;
6042 /* Suppress eol-format conversion in the further conversion. */
6043 coding->eol_type = CODING_EOL_LF;
6045 /* Set the coding system symbol to that for Unix-like EOL. */
6046 eol_type = Fget (saved_coding_symbol, Qeol_type);
6047 if (VECTORP (eol_type)
6048 && XVECTOR (eol_type)->size == 3
6049 && SYMBOLP (XVECTOR (eol_type)->contents[CODING_EOL_LF]))
6050 coding->symbol = XVECTOR (eol_type)->contents[CODING_EOL_LF];
6051 else
6052 coding->symbol = saved_coding_symbol;
6058 coding->consumed = consumed;
6059 coding->consumed_char = consumed_char;
6060 coding->produced = produced;
6061 coding->produced_char = produced_char;
6063 if (coding->dst_multibyte)
6064 newstr = make_uninit_multibyte_string (produced_char + shrinked_bytes,
6065 produced + shrinked_bytes);
6066 else
6067 newstr = make_uninit_string (produced + shrinked_bytes);
6068 if (from > 0)
6069 STRING_COPYIN (newstr, 0, SDATA (str), from);
6070 STRING_COPYIN (newstr, from, buf.data, produced);
6071 if (shrinked_bytes > from)
6072 STRING_COPYIN (newstr, from + produced,
6073 SDATA (str) + to_byte,
6074 shrinked_bytes - from);
6075 free_conversion_buffer (&buf);
6077 if (coding->cmp_data && coding->cmp_data->used)
6078 coding_restore_composition (coding, newstr);
6079 coding_free_composition_data (coding);
6081 if (SYMBOLP (coding->post_read_conversion)
6082 && !NILP (Ffboundp (coding->post_read_conversion)))
6083 newstr = run_pre_post_conversion_on_str (newstr, coding, 0);
6085 return newstr;
6088 Lisp_Object
6089 encode_coding_string (str, coding, nocopy)
6090 Lisp_Object str;
6091 struct coding_system *coding;
6092 int nocopy;
6094 int len;
6095 struct conversion_buffer buf;
6096 int from, to, to_byte;
6097 int result;
6098 int shrinked_bytes = 0;
6099 Lisp_Object newstr;
6100 int consumed, consumed_char, produced, produced_char;
6102 if (SYMBOLP (coding->pre_write_conversion)
6103 && !NILP (Ffboundp (coding->pre_write_conversion)))
6104 str = run_pre_post_conversion_on_str (str, coding, 1);
6106 from = 0;
6107 to = SCHARS (str);
6108 to_byte = SBYTES (str);
6110 /* Encoding routines determine the multibyteness of the source text
6111 by coding->src_multibyte. */
6112 coding->src_multibyte = STRING_MULTIBYTE (str);
6113 coding->dst_multibyte = 0;
6114 if (! CODING_REQUIRE_ENCODING (coding))
6116 coding->consumed = SBYTES (str);
6117 coding->consumed_char = SCHARS (str);
6118 if (STRING_MULTIBYTE (str))
6120 str = Fstring_as_unibyte (str);
6121 nocopy = 1;
6123 coding->produced = SBYTES (str);
6124 coding->produced_char = SCHARS (str);
6125 return (nocopy ? str : Fcopy_sequence (str));
6128 if (coding->composing != COMPOSITION_DISABLED)
6129 coding_save_composition (coding, from, to, str);
6131 /* Try to skip the heading and tailing ASCIIs. */
6132 if (coding->type != coding_type_ccl)
6134 SHRINK_CONVERSION_REGION (&from, &to_byte, coding, SDATA (str),
6136 if (from == to_byte)
6137 return (nocopy ? str : Fcopy_sequence (str));
6138 shrinked_bytes = from + (SBYTES (str) - to_byte);
6141 len = encoding_buffer_size (coding, to_byte - from);
6142 allocate_conversion_buffer (buf, len);
6144 consumed = consumed_char = produced = produced_char = 0;
6145 while (1)
6147 result = encode_coding (coding, SDATA (str) + from + consumed,
6148 buf.data + produced, to_byte - from - consumed,
6149 buf.size - produced);
6150 consumed += coding->consumed;
6151 consumed_char += coding->consumed_char;
6152 produced += coding->produced;
6153 produced_char += coding->produced_char;
6154 if (result == CODING_FINISH_NORMAL
6155 || (result == CODING_FINISH_INSUFFICIENT_SRC
6156 && coding->consumed == 0))
6157 break;
6158 /* Now result should be CODING_FINISH_INSUFFICIENT_DST. */
6159 extend_conversion_buffer (&buf);
6162 coding->consumed = consumed;
6163 coding->consumed_char = consumed_char;
6164 coding->produced = produced;
6165 coding->produced_char = produced_char;
6167 newstr = make_uninit_string (produced + shrinked_bytes);
6168 if (from > 0)
6169 STRING_COPYIN (newstr, 0, SDATA (str), from);
6170 STRING_COPYIN (newstr, from, buf.data, produced);
6171 if (shrinked_bytes > from)
6172 STRING_COPYIN (newstr, from + produced,
6173 SDATA (str) + to_byte,
6174 shrinked_bytes - from);
6176 free_conversion_buffer (&buf);
6177 coding_free_composition_data (coding);
6179 return newstr;
6183 #ifdef emacs
6184 /*** 8. Emacs Lisp library functions ***/
6186 DEFUN ("coding-system-p", Fcoding_system_p, Scoding_system_p, 1, 1, 0,
6187 doc: /* Return t if OBJECT is nil or a coding-system.
6188 See the documentation of `make-coding-system' for information
6189 about coding-system objects. */)
6190 (obj)
6191 Lisp_Object obj;
6193 if (NILP (obj))
6194 return Qt;
6195 if (!SYMBOLP (obj))
6196 return Qnil;
6197 /* Get coding-spec vector for OBJ. */
6198 obj = Fget (obj, Qcoding_system);
6199 return ((VECTORP (obj) && XVECTOR (obj)->size == 5)
6200 ? Qt : Qnil);
6203 DEFUN ("read-non-nil-coding-system", Fread_non_nil_coding_system,
6204 Sread_non_nil_coding_system, 1, 1, 0,
6205 doc: /* Read a coding system from the minibuffer, prompting with string PROMPT. */)
6206 (prompt)
6207 Lisp_Object prompt;
6209 Lisp_Object val;
6212 val = Fcompleting_read (prompt, Vcoding_system_alist, Qnil,
6213 Qt, Qnil, Qcoding_system_history, Qnil, Qnil);
6215 while (SCHARS (val) == 0);
6216 return (Fintern (val, Qnil));
6219 DEFUN ("read-coding-system", Fread_coding_system, Sread_coding_system, 1, 2, 0,
6220 doc: /* Read a coding system from the minibuffer, prompting with string PROMPT.
6221 If the user enters null input, return second argument DEFAULT-CODING-SYSTEM. */)
6222 (prompt, default_coding_system)
6223 Lisp_Object prompt, default_coding_system;
6225 Lisp_Object val;
6226 if (SYMBOLP (default_coding_system))
6227 default_coding_system = SYMBOL_NAME (default_coding_system);
6228 val = Fcompleting_read (prompt, Vcoding_system_alist, Qnil,
6229 Qt, Qnil, Qcoding_system_history,
6230 default_coding_system, Qnil);
6231 return (SCHARS (val) == 0 ? Qnil : Fintern (val, Qnil));
6234 DEFUN ("check-coding-system", Fcheck_coding_system, Scheck_coding_system,
6235 1, 1, 0,
6236 doc: /* Check validity of CODING-SYSTEM.
6237 If valid, return CODING-SYSTEM, else signal a `coding-system-error' error.
6238 It is valid if it is a symbol with a non-nil `coding-system' property.
6239 The value of property should be a vector of length 5. */)
6240 (coding_system)
6241 Lisp_Object coding_system;
6243 CHECK_SYMBOL (coding_system);
6244 if (!NILP (Fcoding_system_p (coding_system)))
6245 return coding_system;
6246 while (1)
6247 Fsignal (Qcoding_system_error, Fcons (coding_system, Qnil));
6250 Lisp_Object
6251 detect_coding_system (src, src_bytes, highest, multibytep)
6252 const unsigned char *src;
6253 int src_bytes, highest;
6254 int multibytep;
6256 int coding_mask, eol_type;
6257 Lisp_Object val, tmp;
6258 int dummy;
6260 coding_mask = detect_coding_mask (src, src_bytes, NULL, &dummy, multibytep);
6261 eol_type = detect_eol_type (src, src_bytes, &dummy);
6262 if (eol_type == CODING_EOL_INCONSISTENT)
6263 eol_type = CODING_EOL_UNDECIDED;
6265 if (!coding_mask)
6267 val = Qundecided;
6268 if (eol_type != CODING_EOL_UNDECIDED)
6270 Lisp_Object val2;
6271 val2 = Fget (Qundecided, Qeol_type);
6272 if (VECTORP (val2))
6273 val = XVECTOR (val2)->contents[eol_type];
6275 return (highest ? val : Fcons (val, Qnil));
6278 /* At first, gather possible coding systems in VAL. */
6279 val = Qnil;
6280 for (tmp = Vcoding_category_list; CONSP (tmp); tmp = XCDR (tmp))
6282 Lisp_Object category_val, category_index;
6284 category_index = Fget (XCAR (tmp), Qcoding_category_index);
6285 category_val = Fsymbol_value (XCAR (tmp));
6286 if (!NILP (category_val)
6287 && NATNUMP (category_index)
6288 && (coding_mask & (1 << XFASTINT (category_index))))
6290 val = Fcons (category_val, val);
6291 if (highest)
6292 break;
6295 if (!highest)
6296 val = Fnreverse (val);
6298 /* Then, replace the elements with subsidiary coding systems. */
6299 for (tmp = val; CONSP (tmp); tmp = XCDR (tmp))
6301 if (eol_type != CODING_EOL_UNDECIDED
6302 && eol_type != CODING_EOL_INCONSISTENT)
6304 Lisp_Object eol;
6305 eol = Fget (XCAR (tmp), Qeol_type);
6306 if (VECTORP (eol))
6307 XSETCAR (tmp, XVECTOR (eol)->contents[eol_type]);
6310 return (highest ? XCAR (val) : val);
6313 DEFUN ("detect-coding-region", Fdetect_coding_region, Sdetect_coding_region,
6314 2, 3, 0,
6315 doc: /* Detect how the byte sequence in the region is encoded.
6316 Return a list of possible coding systems used on decoding a byte
6317 sequence containing the bytes in the region between START and END when
6318 the coding system `undecided' is specified. The list is ordered by
6319 priority decided in the current language environment.
6321 If only ASCII characters are found, it returns a list of single element
6322 `undecided' or its subsidiary coding system according to a detected
6323 end-of-line format.
6325 If optional argument HIGHEST is non-nil, return the coding system of
6326 highest priority. */)
6327 (start, end, highest)
6328 Lisp_Object start, end, highest;
6330 int from, to;
6331 int from_byte, to_byte;
6332 int include_anchor_byte = 0;
6334 CHECK_NUMBER_COERCE_MARKER (start);
6335 CHECK_NUMBER_COERCE_MARKER (end);
6337 validate_region (&start, &end);
6338 from = XINT (start), to = XINT (end);
6339 from_byte = CHAR_TO_BYTE (from);
6340 to_byte = CHAR_TO_BYTE (to);
6342 if (from < GPT && to >= GPT)
6343 move_gap_both (to, to_byte);
6344 /* If we an anchor byte `\0' follows the region, we include it in
6345 the detecting source. Then code detectors can handle the tailing
6346 byte sequence more accurately.
6348 Fix me: This is not a perfect solution. It is better that we
6349 add one more argument, say LAST_BLOCK, to all detect_coding_XXX.
6351 if (to == Z || (to == GPT && GAP_SIZE > 0))
6352 include_anchor_byte = 1;
6353 return detect_coding_system (BYTE_POS_ADDR (from_byte),
6354 to_byte - from_byte + include_anchor_byte,
6355 !NILP (highest),
6356 !NILP (current_buffer
6357 ->enable_multibyte_characters));
6360 DEFUN ("detect-coding-string", Fdetect_coding_string, Sdetect_coding_string,
6361 1, 2, 0,
6362 doc: /* Detect how the byte sequence in STRING is encoded.
6363 Return a list of possible coding systems used on decoding a byte
6364 sequence containing the bytes in STRING when the coding system
6365 `undecided' is specified. The list is ordered by priority decided in
6366 the current language environment.
6368 If only ASCII characters are found, it returns a list of single element
6369 `undecided' or its subsidiary coding system according to a detected
6370 end-of-line format.
6372 If optional argument HIGHEST is non-nil, return the coding system of
6373 highest priority. */)
6374 (string, highest)
6375 Lisp_Object string, highest;
6377 CHECK_STRING (string);
6379 return detect_coding_system (SDATA (string),
6380 /* "+ 1" is to include the anchor byte
6381 `\0'. With this, code detectors can
6382 handle the tailing bytes more
6383 accurately. */
6384 SBYTES (string) + 1,
6385 !NILP (highest),
6386 STRING_MULTIBYTE (string));
6389 /* Subroutine for Fsafe_coding_systems_region_internal.
6391 Return a list of coding systems that safely encode the multibyte
6392 text between P and PEND. SAFE_CODINGS, if non-nil, is a list of
6393 possible coding systems. If it is nil, it means that we have not
6394 yet found any coding systems.
6396 WORK_TABLE is a copy of the char-table Vchar_coding_system_table. An
6397 element of WORK_TABLE is set to t once the element is looked up.
6399 If a non-ASCII single byte char is found, set
6400 *single_byte_char_found to 1. */
6402 static Lisp_Object
6403 find_safe_codings (p, pend, safe_codings, work_table, single_byte_char_found)
6404 unsigned char *p, *pend;
6405 Lisp_Object safe_codings, work_table;
6406 int *single_byte_char_found;
6408 int c, len, i;
6409 Lisp_Object val, ch;
6410 Lisp_Object prev, tail;
6412 while (p < pend)
6414 c = STRING_CHAR_AND_LENGTH (p, pend - p, len);
6415 p += len;
6416 if (ASCII_BYTE_P (c))
6417 /* We can ignore ASCII characters here. */
6418 continue;
6419 if (SINGLE_BYTE_CHAR_P (c))
6420 *single_byte_char_found = 1;
6421 if (NILP (safe_codings))
6422 /* Already all coding systems are excluded. */
6423 continue;
6424 /* Check the safe coding systems for C. */
6425 ch = make_number (c);
6426 val = Faref (work_table, ch);
6427 if (EQ (val, Qt))
6428 /* This element was already checked. Ignore it. */
6429 continue;
6430 /* Remember that we checked this element. */
6431 Faset (work_table, ch, Qt);
6433 for (prev = tail = safe_codings; CONSP (tail); tail = XCDR (tail))
6435 val = XCAR (tail);
6436 if (NILP (Faref (XCDR (val), ch)))
6438 /* Exclued this coding system from SAFE_CODINGS. */
6439 if (EQ (tail, safe_codings))
6440 safe_codings = XCDR (safe_codings);
6441 else
6442 XSETCDR (prev, XCDR (tail));
6444 else
6445 prev = tail;
6448 return safe_codings;
6451 DEFUN ("find-coding-systems-region-internal",
6452 Ffind_coding_systems_region_internal,
6453 Sfind_coding_systems_region_internal, 2, 2, 0,
6454 doc: /* Internal use only. */)
6455 (start, end)
6456 Lisp_Object start, end;
6458 Lisp_Object work_table, safe_codings;
6459 int non_ascii_p = 0;
6460 int single_byte_char_found = 0;
6461 const unsigned char *p1, *p1end, *p2, *p2end, *p;
6463 if (STRINGP (start))
6465 if (!STRING_MULTIBYTE (start))
6466 return Qt;
6467 p1 = SDATA (start), p1end = p1 + SBYTES (start);
6468 p2 = p2end = p1end;
6469 if (SCHARS (start) != SBYTES (start))
6470 non_ascii_p = 1;
6472 else
6474 int from, to, stop;
6476 CHECK_NUMBER_COERCE_MARKER (start);
6477 CHECK_NUMBER_COERCE_MARKER (end);
6478 if (XINT (start) < BEG || XINT (end) > Z || XINT (start) > XINT (end))
6479 args_out_of_range (start, end);
6480 if (NILP (current_buffer->enable_multibyte_characters))
6481 return Qt;
6482 from = CHAR_TO_BYTE (XINT (start));
6483 to = CHAR_TO_BYTE (XINT (end));
6484 stop = from < GPT_BYTE && GPT_BYTE < to ? GPT_BYTE : to;
6485 p1 = BYTE_POS_ADDR (from), p1end = p1 + (stop - from);
6486 if (stop == to)
6487 p2 = p2end = p1end;
6488 else
6489 p2 = BYTE_POS_ADDR (stop), p2end = p2 + (to - stop);
6490 if (XINT (end) - XINT (start) != to - from)
6491 non_ascii_p = 1;
6494 if (!non_ascii_p)
6496 /* We are sure that the text contains no multibyte character.
6497 Check if it contains eight-bit-graphic. */
6498 p = p1;
6499 for (p = p1; p < p1end && ASCII_BYTE_P (*p); p++);
6500 if (p == p1end)
6502 for (p = p2; p < p2end && ASCII_BYTE_P (*p); p++);
6503 if (p == p2end)
6504 return Qt;
6508 /* The text contains non-ASCII characters. */
6510 work_table = Fmake_char_table (Qchar_coding_system, Qnil);
6511 safe_codings = Fcopy_sequence (XCDR (Vcoding_system_safe_chars));
6513 safe_codings = find_safe_codings (p1, p1end, safe_codings, work_table,
6514 &single_byte_char_found);
6515 if (p2 < p2end)
6516 safe_codings = find_safe_codings (p2, p2end, safe_codings, work_table,
6517 &single_byte_char_found);
6518 if (EQ (safe_codings, XCDR (Vcoding_system_safe_chars)))
6519 safe_codings = Qt;
6520 else
6522 /* Turn safe_codings to a list of coding systems... */
6523 Lisp_Object val;
6525 if (single_byte_char_found)
6526 /* ... and append these for eight-bit chars. */
6527 val = Fcons (Qraw_text,
6528 Fcons (Qemacs_mule, Fcons (Qno_conversion, Qnil)));
6529 else
6530 /* ... and append generic coding systems. */
6531 val = Fcopy_sequence (XCAR (Vcoding_system_safe_chars));
6533 for (; CONSP (safe_codings); safe_codings = XCDR (safe_codings))
6534 val = Fcons (XCAR (XCAR (safe_codings)), val);
6535 safe_codings = val;
6538 return safe_codings;
6542 /* Search from position POS for such characters that are unencodable
6543 accoding to SAFE_CHARS, and return a list of their positions. P
6544 points where in the memory the character at POS exists. Limit the
6545 search at PEND or when Nth unencodable characters are found.
6547 If SAFE_CHARS is a char table, an element for an unencodable
6548 character is nil.
6550 If SAFE_CHARS is nil, all non-ASCII characters are unencodable.
6552 Otherwise, SAFE_CHARS is t, and only eight-bit-contrl and
6553 eight-bit-graphic characters are unencodable. */
6555 static Lisp_Object
6556 unencodable_char_position (safe_chars, pos, p, pend, n)
6557 Lisp_Object safe_chars;
6558 int pos;
6559 unsigned char *p, *pend;
6560 int n;
6562 Lisp_Object pos_list;
6564 pos_list = Qnil;
6565 while (p < pend)
6567 int len;
6568 int c = STRING_CHAR_AND_LENGTH (p, MAX_MULTIBYTE_LENGTH, len);
6570 if (c >= 128
6571 && (CHAR_TABLE_P (safe_chars)
6572 ? NILP (CHAR_TABLE_REF (safe_chars, c))
6573 : (NILP (safe_chars) || c < 256)))
6575 pos_list = Fcons (make_number (pos), pos_list);
6576 if (--n <= 0)
6577 break;
6579 pos++;
6580 p += len;
6582 return Fnreverse (pos_list);
6586 DEFUN ("unencodable-char-position", Funencodable_char_position,
6587 Sunencodable_char_position, 3, 5, 0,
6588 doc: /*
6589 Return position of first un-encodable character in a region.
6590 START and END specfiy the region and CODING-SYSTEM specifies the
6591 encoding to check. Return nil if CODING-SYSTEM does encode the region.
6593 If optional 4th argument COUNT is non-nil, it specifies at most how
6594 many un-encodable characters to search. In this case, the value is a
6595 list of positions.
6597 If optional 5th argument STRING is non-nil, it is a string to search
6598 for un-encodable characters. In that case, START and END are indexes
6599 to the string. */)
6600 (start, end, coding_system, count, string)
6601 Lisp_Object start, end, coding_system, count, string;
6603 int n;
6604 Lisp_Object safe_chars;
6605 struct coding_system coding;
6606 Lisp_Object positions;
6607 int from, to;
6608 unsigned char *p, *pend;
6610 if (NILP (string))
6612 validate_region (&start, &end);
6613 from = XINT (start);
6614 to = XINT (end);
6615 if (NILP (current_buffer->enable_multibyte_characters))
6616 return Qnil;
6617 p = CHAR_POS_ADDR (from);
6618 if (to == GPT)
6619 pend = GPT_ADDR;
6620 else
6621 pend = CHAR_POS_ADDR (to);
6623 else
6625 CHECK_STRING (string);
6626 CHECK_NATNUM (start);
6627 CHECK_NATNUM (end);
6628 from = XINT (start);
6629 to = XINT (end);
6630 if (from > to
6631 || to > SCHARS (string))
6632 args_out_of_range_3 (string, start, end);
6633 if (! STRING_MULTIBYTE (string))
6634 return Qnil;
6635 p = SDATA (string) + string_char_to_byte (string, from);
6636 pend = SDATA (string) + string_char_to_byte (string, to);
6639 setup_coding_system (Fcheck_coding_system (coding_system), &coding);
6641 if (NILP (count))
6642 n = 1;
6643 else
6645 CHECK_NATNUM (count);
6646 n = XINT (count);
6649 if (coding.type == coding_type_no_conversion
6650 || coding.type == coding_type_raw_text)
6651 return Qnil;
6653 if (coding.type == coding_type_undecided)
6654 safe_chars = Qnil;
6655 else
6656 safe_chars = coding_safe_chars (coding_system);
6658 if (STRINGP (string)
6659 || from >= GPT || to <= GPT)
6660 positions = unencodable_char_position (safe_chars, from, p, pend, n);
6661 else
6663 Lisp_Object args[2];
6665 args[0] = unencodable_char_position (safe_chars, from, p, GPT_ADDR, n);
6666 n -= XINT (Flength (args[0]));
6667 if (n <= 0)
6668 positions = args[0];
6669 else
6671 args[1] = unencodable_char_position (safe_chars, GPT, GAP_END_ADDR,
6672 pend, n);
6673 positions = Fappend (2, args);
6677 return (NILP (count) ? Fcar (positions) : positions);
6681 Lisp_Object
6682 code_convert_region1 (start, end, coding_system, encodep)
6683 Lisp_Object start, end, coding_system;
6684 int encodep;
6686 struct coding_system coding;
6687 int from, to;
6689 CHECK_NUMBER_COERCE_MARKER (start);
6690 CHECK_NUMBER_COERCE_MARKER (end);
6691 CHECK_SYMBOL (coding_system);
6693 validate_region (&start, &end);
6694 from = XFASTINT (start);
6695 to = XFASTINT (end);
6697 if (NILP (coding_system))
6698 return make_number (to - from);
6700 if (setup_coding_system (Fcheck_coding_system (coding_system), &coding) < 0)
6701 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system)));
6703 coding.mode |= CODING_MODE_LAST_BLOCK;
6704 coding.src_multibyte = coding.dst_multibyte
6705 = !NILP (current_buffer->enable_multibyte_characters);
6706 code_convert_region (from, CHAR_TO_BYTE (from), to, CHAR_TO_BYTE (to),
6707 &coding, encodep, 1);
6708 Vlast_coding_system_used = coding.symbol;
6709 return make_number (coding.produced_char);
6712 DEFUN ("decode-coding-region", Fdecode_coding_region, Sdecode_coding_region,
6713 3, 3, "r\nzCoding system: ",
6714 doc: /* Decode the current region from the specified coding system.
6715 When called from a program, takes three arguments:
6716 START, END, and CODING-SYSTEM. START and END are buffer positions.
6717 This function sets `last-coding-system-used' to the precise coding system
6718 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6719 not fully specified.)
6720 It returns the length of the decoded text. */)
6721 (start, end, coding_system)
6722 Lisp_Object start, end, coding_system;
6724 return code_convert_region1 (start, end, coding_system, 0);
6727 DEFUN ("encode-coding-region", Fencode_coding_region, Sencode_coding_region,
6728 3, 3, "r\nzCoding system: ",
6729 doc: /* Encode the current region into the specified coding system.
6730 When called from a program, takes three arguments:
6731 START, END, and CODING-SYSTEM. START and END are buffer positions.
6732 This function sets `last-coding-system-used' to the precise coding system
6733 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6734 not fully specified.)
6735 It returns the length of the encoded text. */)
6736 (start, end, coding_system)
6737 Lisp_Object start, end, coding_system;
6739 return code_convert_region1 (start, end, coding_system, 1);
6742 Lisp_Object
6743 code_convert_string1 (string, coding_system, nocopy, encodep)
6744 Lisp_Object string, coding_system, nocopy;
6745 int encodep;
6747 struct coding_system coding;
6749 CHECK_STRING (string);
6750 CHECK_SYMBOL (coding_system);
6752 if (NILP (coding_system))
6753 return (NILP (nocopy) ? Fcopy_sequence (string) : string);
6755 if (setup_coding_system (Fcheck_coding_system (coding_system), &coding) < 0)
6756 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system)));
6758 coding.mode |= CODING_MODE_LAST_BLOCK;
6759 string = (encodep
6760 ? encode_coding_string (string, &coding, !NILP (nocopy))
6761 : decode_coding_string (string, &coding, !NILP (nocopy)));
6762 Vlast_coding_system_used = coding.symbol;
6764 return string;
6767 DEFUN ("decode-coding-string", Fdecode_coding_string, Sdecode_coding_string,
6768 2, 3, 0,
6769 doc: /* Decode STRING which is encoded in CODING-SYSTEM, and return the result.
6770 Optional arg NOCOPY non-nil means it is OK to return STRING itself
6771 if the decoding operation is trivial.
6772 This function sets `last-coding-system-used' to the precise coding system
6773 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6774 not fully specified.) */)
6775 (string, coding_system, nocopy)
6776 Lisp_Object string, coding_system, nocopy;
6778 return code_convert_string1 (string, coding_system, nocopy, 0);
6781 DEFUN ("encode-coding-string", Fencode_coding_string, Sencode_coding_string,
6782 2, 3, 0,
6783 doc: /* Encode STRING to CODING-SYSTEM, and return the result.
6784 Optional arg NOCOPY non-nil means it is OK to return STRING itself
6785 if the encoding operation is trivial.
6786 This function sets `last-coding-system-used' to the precise coding system
6787 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6788 not fully specified.) */)
6789 (string, coding_system, nocopy)
6790 Lisp_Object string, coding_system, nocopy;
6792 return code_convert_string1 (string, coding_system, nocopy, 1);
6795 /* Encode or decode STRING according to CODING_SYSTEM.
6796 Do not set Vlast_coding_system_used.
6798 This function is called only from macros DECODE_FILE and
6799 ENCODE_FILE, thus we ignore character composition. */
6801 Lisp_Object
6802 code_convert_string_norecord (string, coding_system, encodep)
6803 Lisp_Object string, coding_system;
6804 int encodep;
6806 struct coding_system coding;
6808 CHECK_STRING (string);
6809 CHECK_SYMBOL (coding_system);
6811 if (NILP (coding_system))
6812 return string;
6814 if (setup_coding_system (Fcheck_coding_system (coding_system), &coding) < 0)
6815 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system)));
6817 coding.composing = COMPOSITION_DISABLED;
6818 coding.mode |= CODING_MODE_LAST_BLOCK;
6819 return (encodep
6820 ? encode_coding_string (string, &coding, 1)
6821 : decode_coding_string (string, &coding, 1));
6824 DEFUN ("decode-sjis-char", Fdecode_sjis_char, Sdecode_sjis_char, 1, 1, 0,
6825 doc: /* Decode a Japanese character which has CODE in shift_jis encoding.
6826 Return the corresponding character. */)
6827 (code)
6828 Lisp_Object code;
6830 unsigned char c1, c2, s1, s2;
6831 Lisp_Object val;
6833 CHECK_NUMBER (code);
6834 s1 = (XFASTINT (code)) >> 8, s2 = (XFASTINT (code)) & 0xFF;
6835 if (s1 == 0)
6837 if (s2 < 0x80)
6838 XSETFASTINT (val, s2);
6839 else if (s2 >= 0xA0 || s2 <= 0xDF)
6840 XSETFASTINT (val, MAKE_CHAR (charset_katakana_jisx0201, s2, 0));
6841 else
6842 error ("Invalid Shift JIS code: %x", XFASTINT (code));
6844 else
6846 if ((s1 < 0x80 || (s1 > 0x9F && s1 < 0xE0) || s1 > 0xEF)
6847 || (s2 < 0x40 || s2 == 0x7F || s2 > 0xFC))
6848 error ("Invalid Shift JIS code: %x", XFASTINT (code));
6849 DECODE_SJIS (s1, s2, c1, c2);
6850 XSETFASTINT (val, MAKE_CHAR (charset_jisx0208, c1, c2));
6852 return val;
6855 DEFUN ("encode-sjis-char", Fencode_sjis_char, Sencode_sjis_char, 1, 1, 0,
6856 doc: /* Encode a Japanese character CHAR to shift_jis encoding.
6857 Return the corresponding code in SJIS. */)
6858 (ch)
6859 Lisp_Object ch;
6861 int charset, c1, c2, s1, s2;
6862 Lisp_Object val;
6864 CHECK_NUMBER (ch);
6865 SPLIT_CHAR (XFASTINT (ch), charset, c1, c2);
6866 if (charset == CHARSET_ASCII)
6868 val = ch;
6870 else if (charset == charset_jisx0208
6871 && c1 > 0x20 && c1 < 0x7F && c2 > 0x20 && c2 < 0x7F)
6873 ENCODE_SJIS (c1, c2, s1, s2);
6874 XSETFASTINT (val, (s1 << 8) | s2);
6876 else if (charset == charset_katakana_jisx0201
6877 && c1 > 0x20 && c2 < 0xE0)
6879 XSETFASTINT (val, c1 | 0x80);
6881 else
6882 error ("Can't encode to shift_jis: %d", XFASTINT (ch));
6883 return val;
6886 DEFUN ("decode-big5-char", Fdecode_big5_char, Sdecode_big5_char, 1, 1, 0,
6887 doc: /* Decode a Big5 character which has CODE in BIG5 coding system.
6888 Return the corresponding character. */)
6889 (code)
6890 Lisp_Object code;
6892 int charset;
6893 unsigned char b1, b2, c1, c2;
6894 Lisp_Object val;
6896 CHECK_NUMBER (code);
6897 b1 = (XFASTINT (code)) >> 8, b2 = (XFASTINT (code)) & 0xFF;
6898 if (b1 == 0)
6900 if (b2 >= 0x80)
6901 error ("Invalid BIG5 code: %x", XFASTINT (code));
6902 val = code;
6904 else
6906 if ((b1 < 0xA1 || b1 > 0xFE)
6907 || (b2 < 0x40 || (b2 > 0x7E && b2 < 0xA1) || b2 > 0xFE))
6908 error ("Invalid BIG5 code: %x", XFASTINT (code));
6909 DECODE_BIG5 (b1, b2, charset, c1, c2);
6910 XSETFASTINT (val, MAKE_CHAR (charset, c1, c2));
6912 return val;
6915 DEFUN ("encode-big5-char", Fencode_big5_char, Sencode_big5_char, 1, 1, 0,
6916 doc: /* Encode the Big5 character CHAR to BIG5 coding system.
6917 Return the corresponding character code in Big5. */)
6918 (ch)
6919 Lisp_Object ch;
6921 int charset, c1, c2, b1, b2;
6922 Lisp_Object val;
6924 CHECK_NUMBER (ch);
6925 SPLIT_CHAR (XFASTINT (ch), charset, c1, c2);
6926 if (charset == CHARSET_ASCII)
6928 val = ch;
6930 else if ((charset == charset_big5_1
6931 && (XFASTINT (ch) >= 0x250a1 && XFASTINT (ch) <= 0x271ec))
6932 || (charset == charset_big5_2
6933 && XFASTINT (ch) >= 0x290a1 && XFASTINT (ch) <= 0x2bdb2))
6935 ENCODE_BIG5 (charset, c1, c2, b1, b2);
6936 XSETFASTINT (val, (b1 << 8) | b2);
6938 else
6939 error ("Can't encode to Big5: %d", XFASTINT (ch));
6940 return val;
6943 DEFUN ("set-terminal-coding-system-internal", Fset_terminal_coding_system_internal,
6944 Sset_terminal_coding_system_internal, 1, 1, 0,
6945 doc: /* Internal use only. */)
6946 (coding_system)
6947 Lisp_Object coding_system;
6949 CHECK_SYMBOL (coding_system);
6950 setup_coding_system (Fcheck_coding_system (coding_system), &terminal_coding);
6951 /* We had better not send unsafe characters to terminal. */
6952 terminal_coding.flags |= CODING_FLAG_ISO_SAFE;
6953 /* Character composition should be disabled. */
6954 terminal_coding.composing = COMPOSITION_DISABLED;
6955 /* Error notification should be suppressed. */
6956 terminal_coding.suppress_error = 1;
6957 terminal_coding.src_multibyte = 1;
6958 terminal_coding.dst_multibyte = 0;
6959 return Qnil;
6962 DEFUN ("set-safe-terminal-coding-system-internal", Fset_safe_terminal_coding_system_internal,
6963 Sset_safe_terminal_coding_system_internal, 1, 1, 0,
6964 doc: /* Internal use only. */)
6965 (coding_system)
6966 Lisp_Object coding_system;
6968 CHECK_SYMBOL (coding_system);
6969 setup_coding_system (Fcheck_coding_system (coding_system),
6970 &safe_terminal_coding);
6971 /* Character composition should be disabled. */
6972 safe_terminal_coding.composing = COMPOSITION_DISABLED;
6973 /* Error notification should be suppressed. */
6974 terminal_coding.suppress_error = 1;
6975 safe_terminal_coding.src_multibyte = 1;
6976 safe_terminal_coding.dst_multibyte = 0;
6977 return Qnil;
6980 DEFUN ("terminal-coding-system", Fterminal_coding_system,
6981 Sterminal_coding_system, 0, 0, 0,
6982 doc: /* Return coding system specified for terminal output. */)
6985 return terminal_coding.symbol;
6988 DEFUN ("set-keyboard-coding-system-internal", Fset_keyboard_coding_system_internal,
6989 Sset_keyboard_coding_system_internal, 1, 1, 0,
6990 doc: /* Internal use only. */)
6991 (coding_system)
6992 Lisp_Object coding_system;
6994 CHECK_SYMBOL (coding_system);
6995 setup_coding_system (Fcheck_coding_system (coding_system), &keyboard_coding);
6996 /* Character composition should be disabled. */
6997 keyboard_coding.composing = COMPOSITION_DISABLED;
6998 return Qnil;
7001 DEFUN ("keyboard-coding-system", Fkeyboard_coding_system,
7002 Skeyboard_coding_system, 0, 0, 0,
7003 doc: /* Return coding system specified for decoding keyboard input. */)
7006 return keyboard_coding.symbol;
7010 DEFUN ("find-operation-coding-system", Ffind_operation_coding_system,
7011 Sfind_operation_coding_system, 1, MANY, 0,
7012 doc: /* Choose a coding system for an operation based on the target name.
7013 The value names a pair of coding systems: (DECODING-SYSTEM . ENCODING-SYSTEM).
7014 DECODING-SYSTEM is the coding system to use for decoding
7015 \(in case OPERATION does decoding), and ENCODING-SYSTEM is the coding system
7016 for encoding (in case OPERATION does encoding).
7018 The first argument OPERATION specifies an I/O primitive:
7019 For file I/O, `insert-file-contents' or `write-region'.
7020 For process I/O, `call-process', `call-process-region', or `start-process'.
7021 For network I/O, `open-network-stream'.
7023 The remaining arguments should be the same arguments that were passed
7024 to the primitive. Depending on which primitive, one of those arguments
7025 is selected as the TARGET. For example, if OPERATION does file I/O,
7026 whichever argument specifies the file name is TARGET.
7028 TARGET has a meaning which depends on OPERATION:
7029 For file I/O, TARGET is a file name.
7030 For process I/O, TARGET is a process name.
7031 For network I/O, TARGET is a service name or a port number
7033 This function looks up what specified for TARGET in,
7034 `file-coding-system-alist', `process-coding-system-alist',
7035 or `network-coding-system-alist' depending on OPERATION.
7036 They may specify a coding system, a cons of coding systems,
7037 or a function symbol to call.
7038 In the last case, we call the function with one argument,
7039 which is a list of all the arguments given to this function.
7041 usage: (find-operation-coding-system OPERATION ARGUMENTS ...) */)
7042 (nargs, args)
7043 int nargs;
7044 Lisp_Object *args;
7046 Lisp_Object operation, target_idx, target, val;
7047 register Lisp_Object chain;
7049 if (nargs < 2)
7050 error ("Too few arguments");
7051 operation = args[0];
7052 if (!SYMBOLP (operation)
7053 || !INTEGERP (target_idx = Fget (operation, Qtarget_idx)))
7054 error ("Invalid first argument");
7055 if (nargs < 1 + XINT (target_idx))
7056 error ("Too few arguments for operation: %s",
7057 SDATA (SYMBOL_NAME (operation)));
7058 /* For write-region, if the 6th argument (i.e. VISIT, the 5th
7059 argument to write-region) is string, it must be treated as a
7060 target file name. */
7061 if (EQ (operation, Qwrite_region)
7062 && nargs > 5
7063 && STRINGP (args[5]))
7064 target_idx = make_number (4);
7065 target = args[XINT (target_idx) + 1];
7066 if (!(STRINGP (target)
7067 || (EQ (operation, Qopen_network_stream) && INTEGERP (target))))
7068 error ("Invalid argument %d", XINT (target_idx) + 1);
7070 chain = ((EQ (operation, Qinsert_file_contents)
7071 || EQ (operation, Qwrite_region))
7072 ? Vfile_coding_system_alist
7073 : (EQ (operation, Qopen_network_stream)
7074 ? Vnetwork_coding_system_alist
7075 : Vprocess_coding_system_alist));
7076 if (NILP (chain))
7077 return Qnil;
7079 for (; CONSP (chain); chain = XCDR (chain))
7081 Lisp_Object elt;
7082 elt = XCAR (chain);
7084 if (CONSP (elt)
7085 && ((STRINGP (target)
7086 && STRINGP (XCAR (elt))
7087 && fast_string_match (XCAR (elt), target) >= 0)
7088 || (INTEGERP (target) && EQ (target, XCAR (elt)))))
7090 val = XCDR (elt);
7091 /* Here, if VAL is both a valid coding system and a valid
7092 function symbol, we return VAL as a coding system. */
7093 if (CONSP (val))
7094 return val;
7095 if (! SYMBOLP (val))
7096 return Qnil;
7097 if (! NILP (Fcoding_system_p (val)))
7098 return Fcons (val, val);
7099 if (! NILP (Ffboundp (val)))
7101 val = call1 (val, Flist (nargs, args));
7102 if (CONSP (val))
7103 return val;
7104 if (SYMBOLP (val) && ! NILP (Fcoding_system_p (val)))
7105 return Fcons (val, val);
7107 return Qnil;
7110 return Qnil;
7113 DEFUN ("update-coding-systems-internal", Fupdate_coding_systems_internal,
7114 Supdate_coding_systems_internal, 0, 0, 0,
7115 doc: /* Update internal database for ISO2022 and CCL based coding systems.
7116 When values of any coding categories are changed, you must
7117 call this function. */)
7120 int i;
7122 for (i = CODING_CATEGORY_IDX_EMACS_MULE; i < CODING_CATEGORY_IDX_MAX; i++)
7124 Lisp_Object val;
7126 val = SYMBOL_VALUE (XVECTOR (Vcoding_category_table)->contents[i]);
7127 if (!NILP (val))
7129 if (! coding_system_table[i])
7130 coding_system_table[i] = ((struct coding_system *)
7131 xmalloc (sizeof (struct coding_system)));
7132 setup_coding_system (val, coding_system_table[i]);
7134 else if (coding_system_table[i])
7136 xfree (coding_system_table[i]);
7137 coding_system_table[i] = NULL;
7141 return Qnil;
7144 DEFUN ("set-coding-priority-internal", Fset_coding_priority_internal,
7145 Sset_coding_priority_internal, 0, 0, 0,
7146 doc: /* Update internal database for the current value of `coding-category-list'.
7147 This function is internal use only. */)
7150 int i = 0, idx;
7151 Lisp_Object val;
7153 val = Vcoding_category_list;
7155 while (CONSP (val) && i < CODING_CATEGORY_IDX_MAX)
7157 if (! SYMBOLP (XCAR (val)))
7158 break;
7159 idx = XFASTINT (Fget (XCAR (val), Qcoding_category_index));
7160 if (idx >= CODING_CATEGORY_IDX_MAX)
7161 break;
7162 coding_priorities[i++] = (1 << idx);
7163 val = XCDR (val);
7165 /* If coding-category-list is valid and contains all coding
7166 categories, `i' should be CODING_CATEGORY_IDX_MAX now. If not,
7167 the following code saves Emacs from crashing. */
7168 while (i < CODING_CATEGORY_IDX_MAX)
7169 coding_priorities[i++] = CODING_CATEGORY_MASK_RAW_TEXT;
7171 return Qnil;
7174 DEFUN ("define-coding-system-internal", Fdefine_coding_system_internal,
7175 Sdefine_coding_system_internal, 1, 1, 0,
7176 doc: /* Register CODING-SYSTEM as a base coding system.
7177 This function is internal use only. */)
7178 (coding_system)
7179 Lisp_Object coding_system;
7181 Lisp_Object safe_chars, slot;
7183 if (NILP (Fcheck_coding_system (coding_system)))
7184 Fsignal (Qcoding_system_error, Fcons (coding_system, Qnil));
7185 safe_chars = coding_safe_chars (coding_system);
7186 if (! EQ (safe_chars, Qt) && ! CHAR_TABLE_P (safe_chars))
7187 error ("No valid safe-chars property for %s",
7188 SDATA (SYMBOL_NAME (coding_system)));
7189 if (EQ (safe_chars, Qt))
7191 if (NILP (Fmemq (coding_system, XCAR (Vcoding_system_safe_chars))))
7192 XSETCAR (Vcoding_system_safe_chars,
7193 Fcons (coding_system, XCAR (Vcoding_system_safe_chars)));
7195 else
7197 slot = Fassq (coding_system, XCDR (Vcoding_system_safe_chars));
7198 if (NILP (slot))
7199 XSETCDR (Vcoding_system_safe_chars,
7200 nconc2 (XCDR (Vcoding_system_safe_chars),
7201 Fcons (Fcons (coding_system, safe_chars), Qnil)));
7202 else
7203 XSETCDR (slot, safe_chars);
7205 return Qnil;
7208 #endif /* emacs */
7211 /*** 9. Post-amble ***/
7213 void
7214 init_coding_once ()
7216 int i;
7218 /* Emacs' internal format specific initialize routine. */
7219 for (i = 0; i <= 0x20; i++)
7220 emacs_code_class[i] = EMACS_control_code;
7221 emacs_code_class[0x0A] = EMACS_linefeed_code;
7222 emacs_code_class[0x0D] = EMACS_carriage_return_code;
7223 for (i = 0x21 ; i < 0x7F; i++)
7224 emacs_code_class[i] = EMACS_ascii_code;
7225 emacs_code_class[0x7F] = EMACS_control_code;
7226 for (i = 0x80; i < 0xFF; i++)
7227 emacs_code_class[i] = EMACS_invalid_code;
7228 emacs_code_class[LEADING_CODE_PRIVATE_11] = EMACS_leading_code_3;
7229 emacs_code_class[LEADING_CODE_PRIVATE_12] = EMACS_leading_code_3;
7230 emacs_code_class[LEADING_CODE_PRIVATE_21] = EMACS_leading_code_4;
7231 emacs_code_class[LEADING_CODE_PRIVATE_22] = EMACS_leading_code_4;
7233 /* ISO2022 specific initialize routine. */
7234 for (i = 0; i < 0x20; i++)
7235 iso_code_class[i] = ISO_control_0;
7236 for (i = 0x21; i < 0x7F; i++)
7237 iso_code_class[i] = ISO_graphic_plane_0;
7238 for (i = 0x80; i < 0xA0; i++)
7239 iso_code_class[i] = ISO_control_1;
7240 for (i = 0xA1; i < 0xFF; i++)
7241 iso_code_class[i] = ISO_graphic_plane_1;
7242 iso_code_class[0x20] = iso_code_class[0x7F] = ISO_0x20_or_0x7F;
7243 iso_code_class[0xA0] = iso_code_class[0xFF] = ISO_0xA0_or_0xFF;
7244 iso_code_class[ISO_CODE_CR] = ISO_carriage_return;
7245 iso_code_class[ISO_CODE_SO] = ISO_shift_out;
7246 iso_code_class[ISO_CODE_SI] = ISO_shift_in;
7247 iso_code_class[ISO_CODE_SS2_7] = ISO_single_shift_2_7;
7248 iso_code_class[ISO_CODE_ESC] = ISO_escape;
7249 iso_code_class[ISO_CODE_SS2] = ISO_single_shift_2;
7250 iso_code_class[ISO_CODE_SS3] = ISO_single_shift_3;
7251 iso_code_class[ISO_CODE_CSI] = ISO_control_sequence_introducer;
7253 setup_coding_system (Qnil, &keyboard_coding);
7254 setup_coding_system (Qnil, &terminal_coding);
7255 setup_coding_system (Qnil, &safe_terminal_coding);
7256 setup_coding_system (Qnil, &default_buffer_file_coding);
7258 bzero (coding_system_table, sizeof coding_system_table);
7260 bzero (ascii_skip_code, sizeof ascii_skip_code);
7261 for (i = 0; i < 128; i++)
7262 ascii_skip_code[i] = 1;
7264 #if defined (MSDOS) || defined (WINDOWSNT)
7265 system_eol_type = CODING_EOL_CRLF;
7266 #else
7267 system_eol_type = CODING_EOL_LF;
7268 #endif
7270 inhibit_pre_post_conversion = 0;
7273 #ifdef emacs
7275 void
7276 syms_of_coding ()
7278 Qtarget_idx = intern ("target-idx");
7279 staticpro (&Qtarget_idx);
7281 Qcoding_system_history = intern ("coding-system-history");
7282 staticpro (&Qcoding_system_history);
7283 Fset (Qcoding_system_history, Qnil);
7285 /* Target FILENAME is the first argument. */
7286 Fput (Qinsert_file_contents, Qtarget_idx, make_number (0));
7287 /* Target FILENAME is the third argument. */
7288 Fput (Qwrite_region, Qtarget_idx, make_number (2));
7290 Qcall_process = intern ("call-process");
7291 staticpro (&Qcall_process);
7292 /* Target PROGRAM is the first argument. */
7293 Fput (Qcall_process, Qtarget_idx, make_number (0));
7295 Qcall_process_region = intern ("call-process-region");
7296 staticpro (&Qcall_process_region);
7297 /* Target PROGRAM is the third argument. */
7298 Fput (Qcall_process_region, Qtarget_idx, make_number (2));
7300 Qstart_process = intern ("start-process");
7301 staticpro (&Qstart_process);
7302 /* Target PROGRAM is the third argument. */
7303 Fput (Qstart_process, Qtarget_idx, make_number (2));
7305 Qopen_network_stream = intern ("open-network-stream");
7306 staticpro (&Qopen_network_stream);
7307 /* Target SERVICE is the fourth argument. */
7308 Fput (Qopen_network_stream, Qtarget_idx, make_number (3));
7310 Qcoding_system = intern ("coding-system");
7311 staticpro (&Qcoding_system);
7313 Qeol_type = intern ("eol-type");
7314 staticpro (&Qeol_type);
7316 Qbuffer_file_coding_system = intern ("buffer-file-coding-system");
7317 staticpro (&Qbuffer_file_coding_system);
7319 Qpost_read_conversion = intern ("post-read-conversion");
7320 staticpro (&Qpost_read_conversion);
7322 Qpre_write_conversion = intern ("pre-write-conversion");
7323 staticpro (&Qpre_write_conversion);
7325 Qno_conversion = intern ("no-conversion");
7326 staticpro (&Qno_conversion);
7328 Qundecided = intern ("undecided");
7329 staticpro (&Qundecided);
7331 Qcoding_system_p = intern ("coding-system-p");
7332 staticpro (&Qcoding_system_p);
7334 Qcoding_system_error = intern ("coding-system-error");
7335 staticpro (&Qcoding_system_error);
7337 Fput (Qcoding_system_error, Qerror_conditions,
7338 Fcons (Qcoding_system_error, Fcons (Qerror, Qnil)));
7339 Fput (Qcoding_system_error, Qerror_message,
7340 build_string ("Invalid coding system"));
7342 Qcoding_category = intern ("coding-category");
7343 staticpro (&Qcoding_category);
7344 Qcoding_category_index = intern ("coding-category-index");
7345 staticpro (&Qcoding_category_index);
7347 Vcoding_category_table
7348 = Fmake_vector (make_number (CODING_CATEGORY_IDX_MAX), Qnil);
7349 staticpro (&Vcoding_category_table);
7351 int i;
7352 for (i = 0; i < CODING_CATEGORY_IDX_MAX; i++)
7354 XVECTOR (Vcoding_category_table)->contents[i]
7355 = intern (coding_category_name[i]);
7356 Fput (XVECTOR (Vcoding_category_table)->contents[i],
7357 Qcoding_category_index, make_number (i));
7361 Vcoding_system_safe_chars = Fcons (Qnil, Qnil);
7362 staticpro (&Vcoding_system_safe_chars);
7364 Qtranslation_table = intern ("translation-table");
7365 staticpro (&Qtranslation_table);
7366 Fput (Qtranslation_table, Qchar_table_extra_slots, make_number (1));
7368 Qtranslation_table_id = intern ("translation-table-id");
7369 staticpro (&Qtranslation_table_id);
7371 Qtranslation_table_for_decode = intern ("translation-table-for-decode");
7372 staticpro (&Qtranslation_table_for_decode);
7374 Qtranslation_table_for_encode = intern ("translation-table-for-encode");
7375 staticpro (&Qtranslation_table_for_encode);
7377 Qsafe_chars = intern ("safe-chars");
7378 staticpro (&Qsafe_chars);
7380 Qchar_coding_system = intern ("char-coding-system");
7381 staticpro (&Qchar_coding_system);
7383 /* Intern this now in case it isn't already done.
7384 Setting this variable twice is harmless.
7385 But don't staticpro it here--that is done in alloc.c. */
7386 Qchar_table_extra_slots = intern ("char-table-extra-slots");
7387 Fput (Qsafe_chars, Qchar_table_extra_slots, make_number (0));
7388 Fput (Qchar_coding_system, Qchar_table_extra_slots, make_number (0));
7390 Qvalid_codes = intern ("valid-codes");
7391 staticpro (&Qvalid_codes);
7393 Qemacs_mule = intern ("emacs-mule");
7394 staticpro (&Qemacs_mule);
7396 Qraw_text = intern ("raw-text");
7397 staticpro (&Qraw_text);
7399 defsubr (&Scoding_system_p);
7400 defsubr (&Sread_coding_system);
7401 defsubr (&Sread_non_nil_coding_system);
7402 defsubr (&Scheck_coding_system);
7403 defsubr (&Sdetect_coding_region);
7404 defsubr (&Sdetect_coding_string);
7405 defsubr (&Sfind_coding_systems_region_internal);
7406 defsubr (&Sunencodable_char_position);
7407 defsubr (&Sdecode_coding_region);
7408 defsubr (&Sencode_coding_region);
7409 defsubr (&Sdecode_coding_string);
7410 defsubr (&Sencode_coding_string);
7411 defsubr (&Sdecode_sjis_char);
7412 defsubr (&Sencode_sjis_char);
7413 defsubr (&Sdecode_big5_char);
7414 defsubr (&Sencode_big5_char);
7415 defsubr (&Sset_terminal_coding_system_internal);
7416 defsubr (&Sset_safe_terminal_coding_system_internal);
7417 defsubr (&Sterminal_coding_system);
7418 defsubr (&Sset_keyboard_coding_system_internal);
7419 defsubr (&Skeyboard_coding_system);
7420 defsubr (&Sfind_operation_coding_system);
7421 defsubr (&Supdate_coding_systems_internal);
7422 defsubr (&Sset_coding_priority_internal);
7423 defsubr (&Sdefine_coding_system_internal);
7425 DEFVAR_LISP ("coding-system-list", &Vcoding_system_list,
7426 doc: /* List of coding systems.
7428 Do not alter the value of this variable manually. This variable should be
7429 updated by the functions `make-coding-system' and
7430 `define-coding-system-alias'. */);
7431 Vcoding_system_list = Qnil;
7433 DEFVAR_LISP ("coding-system-alist", &Vcoding_system_alist,
7434 doc: /* Alist of coding system names.
7435 Each element is one element list of coding system name.
7436 This variable is given to `completing-read' as TABLE argument.
7438 Do not alter the value of this variable manually. This variable should be
7439 updated by the functions `make-coding-system' and
7440 `define-coding-system-alias'. */);
7441 Vcoding_system_alist = Qnil;
7443 DEFVAR_LISP ("coding-category-list", &Vcoding_category_list,
7444 doc: /* List of coding-categories (symbols) ordered by priority.
7446 On detecting a coding system, Emacs tries code detection algorithms
7447 associated with each coding-category one by one in this order. When
7448 one algorithm agrees with a byte sequence of source text, the coding
7449 system bound to the corresponding coding-category is selected. */);
7451 int i;
7453 Vcoding_category_list = Qnil;
7454 for (i = CODING_CATEGORY_IDX_MAX - 1; i >= 0; i--)
7455 Vcoding_category_list
7456 = Fcons (XVECTOR (Vcoding_category_table)->contents[i],
7457 Vcoding_category_list);
7460 DEFVAR_LISP ("coding-system-for-read", &Vcoding_system_for_read,
7461 doc: /* Specify the coding system for read operations.
7462 It is useful to bind this variable with `let', but do not set it globally.
7463 If the value is a coding system, it is used for decoding on read operation.
7464 If not, an appropriate element is used from one of the coding system alists:
7465 There are three such tables, `file-coding-system-alist',
7466 `process-coding-system-alist', and `network-coding-system-alist'. */);
7467 Vcoding_system_for_read = Qnil;
7469 DEFVAR_LISP ("coding-system-for-write", &Vcoding_system_for_write,
7470 doc: /* Specify the coding system for write operations.
7471 Programs bind this variable with `let', but you should not set it globally.
7472 If the value is a coding system, it is used for encoding of output,
7473 when writing it to a file and when sending it to a file or subprocess.
7475 If this does not specify a coding system, an appropriate element
7476 is used from one of the coding system alists:
7477 There are three such tables, `file-coding-system-alist',
7478 `process-coding-system-alist', and `network-coding-system-alist'.
7479 For output to files, if the above procedure does not specify a coding system,
7480 the value of `buffer-file-coding-system' is used. */);
7481 Vcoding_system_for_write = Qnil;
7483 DEFVAR_LISP ("last-coding-system-used", &Vlast_coding_system_used,
7484 doc: /* Coding system used in the latest file or process I/O. */);
7485 Vlast_coding_system_used = Qnil;
7487 DEFVAR_BOOL ("inhibit-eol-conversion", &inhibit_eol_conversion,
7488 doc: /* *Non-nil means always inhibit code conversion of end-of-line format.
7489 See info node `Coding Systems' and info node `Text and Binary' concerning
7490 such conversion. */);
7491 inhibit_eol_conversion = 0;
7493 DEFVAR_BOOL ("inherit-process-coding-system", &inherit_process_coding_system,
7494 doc: /* Non-nil means process buffer inherits coding system of process output.
7495 Bind it to t if the process output is to be treated as if it were a file
7496 read from some filesystem. */);
7497 inherit_process_coding_system = 0;
7499 DEFVAR_LISP ("file-coding-system-alist", &Vfile_coding_system_alist,
7500 doc: /* Alist to decide a coding system to use for a file I/O operation.
7501 The format is ((PATTERN . VAL) ...),
7502 where PATTERN is a regular expression matching a file name,
7503 VAL is a coding system, a cons of coding systems, or a function symbol.
7504 If VAL is a coding system, it is used for both decoding and encoding
7505 the file contents.
7506 If VAL is a cons of coding systems, the car part is used for decoding,
7507 and the cdr part is used for encoding.
7508 If VAL is a function symbol, the function must return a coding system
7509 or a cons of coding systems which are used as above. The function gets
7510 the arguments with which `find-operation-coding-system' was called.
7512 See also the function `find-operation-coding-system'
7513 and the variable `auto-coding-alist'. */);
7514 Vfile_coding_system_alist = Qnil;
7516 DEFVAR_LISP ("process-coding-system-alist", &Vprocess_coding_system_alist,
7517 doc: /* Alist to decide a coding system to use for a process I/O operation.
7518 The format is ((PATTERN . VAL) ...),
7519 where PATTERN is a regular expression matching a program name,
7520 VAL is a coding system, a cons of coding systems, or a function symbol.
7521 If VAL is a coding system, it is used for both decoding what received
7522 from the program and encoding what sent to the program.
7523 If VAL is a cons of coding systems, the car part is used for decoding,
7524 and the cdr part is used for encoding.
7525 If VAL is a function symbol, the function must return a coding system
7526 or a cons of coding systems which are used as above.
7528 See also the function `find-operation-coding-system'. */);
7529 Vprocess_coding_system_alist = Qnil;
7531 DEFVAR_LISP ("network-coding-system-alist", &Vnetwork_coding_system_alist,
7532 doc: /* Alist to decide a coding system to use for a network I/O operation.
7533 The format is ((PATTERN . VAL) ...),
7534 where PATTERN is a regular expression matching a network service name
7535 or is a port number to connect to,
7536 VAL is a coding system, a cons of coding systems, or a function symbol.
7537 If VAL is a coding system, it is used for both decoding what received
7538 from the network stream and encoding what sent to the network stream.
7539 If VAL is a cons of coding systems, the car part is used for decoding,
7540 and the cdr part is used for encoding.
7541 If VAL is a function symbol, the function must return a coding system
7542 or a cons of coding systems which are used as above.
7544 See also the function `find-operation-coding-system'. */);
7545 Vnetwork_coding_system_alist = Qnil;
7547 DEFVAR_LISP ("locale-coding-system", &Vlocale_coding_system,
7548 doc: /* Coding system to use with system messages.
7549 Also used for decoding keyboard input on X Window system. */);
7550 Vlocale_coding_system = Qnil;
7552 /* The eol mnemonics are reset in startup.el system-dependently. */
7553 DEFVAR_LISP ("eol-mnemonic-unix", &eol_mnemonic_unix,
7554 doc: /* *String displayed in mode line for UNIX-like (LF) end-of-line format. */);
7555 eol_mnemonic_unix = build_string (":");
7557 DEFVAR_LISP ("eol-mnemonic-dos", &eol_mnemonic_dos,
7558 doc: /* *String displayed in mode line for DOS-like (CRLF) end-of-line format. */);
7559 eol_mnemonic_dos = build_string ("\\");
7561 DEFVAR_LISP ("eol-mnemonic-mac", &eol_mnemonic_mac,
7562 doc: /* *String displayed in mode line for MAC-like (CR) end-of-line format. */);
7563 eol_mnemonic_mac = build_string ("/");
7565 DEFVAR_LISP ("eol-mnemonic-undecided", &eol_mnemonic_undecided,
7566 doc: /* *String displayed in mode line when end-of-line format is not yet determined. */);
7567 eol_mnemonic_undecided = build_string (":");
7569 DEFVAR_LISP ("enable-character-translation", &Venable_character_translation,
7570 doc: /* *Non-nil enables character translation while encoding and decoding. */);
7571 Venable_character_translation = Qt;
7573 DEFVAR_LISP ("standard-translation-table-for-decode",
7574 &Vstandard_translation_table_for_decode,
7575 doc: /* Table for translating characters while decoding. */);
7576 Vstandard_translation_table_for_decode = Qnil;
7578 DEFVAR_LISP ("standard-translation-table-for-encode",
7579 &Vstandard_translation_table_for_encode,
7580 doc: /* Table for translating characters while encoding. */);
7581 Vstandard_translation_table_for_encode = Qnil;
7583 DEFVAR_LISP ("charset-revision-table", &Vcharset_revision_alist,
7584 doc: /* Alist of charsets vs revision numbers.
7585 While encoding, if a charset (car part of an element) is found,
7586 designate it with the escape sequence identifying revision (cdr part of the element). */);
7587 Vcharset_revision_alist = Qnil;
7589 DEFVAR_LISP ("default-process-coding-system",
7590 &Vdefault_process_coding_system,
7591 doc: /* Cons of coding systems used for process I/O by default.
7592 The car part is used for decoding a process output,
7593 the cdr part is used for encoding a text to be sent to a process. */);
7594 Vdefault_process_coding_system = Qnil;
7596 DEFVAR_LISP ("latin-extra-code-table", &Vlatin_extra_code_table,
7597 doc: /* Table of extra Latin codes in the range 128..159 (inclusive).
7598 This is a vector of length 256.
7599 If Nth element is non-nil, the existence of code N in a file
7600 \(or output of subprocess) doesn't prevent it to be detected as
7601 a coding system of ISO 2022 variant which has a flag
7602 `accept-latin-extra-code' t (e.g. iso-latin-1) on reading a file
7603 or reading output of a subprocess.
7604 Only 128th through 159th elements has a meaning. */);
7605 Vlatin_extra_code_table = Fmake_vector (make_number (256), Qnil);
7607 DEFVAR_LISP ("select-safe-coding-system-function",
7608 &Vselect_safe_coding_system_function,
7609 doc: /* Function to call to select safe coding system for encoding a text.
7611 If set, this function is called to force a user to select a proper
7612 coding system which can encode the text in the case that a default
7613 coding system used in each operation can't encode the text.
7615 The default value is `select-safe-coding-system' (which see). */);
7616 Vselect_safe_coding_system_function = Qnil;
7618 DEFVAR_BOOL ("coding-system-require-warning",
7619 &coding_system_require_warning,
7620 doc: /* Internal use only.
7621 If non-nil, on writing a file, `select-safe-coding-system-function' is
7622 called even if `coding-system-for-write' is non-nil. The command
7623 `universal-coding-system-argument' binds this variable to t temporarily. */);
7624 coding_system_require_warning = 0;
7627 DEFVAR_BOOL ("inhibit-iso-escape-detection",
7628 &inhibit_iso_escape_detection,
7629 doc: /* If non-nil, Emacs ignores ISO2022's escape sequence on code detection.
7631 By default, on reading a file, Emacs tries to detect how the text is
7632 encoded. This code detection is sensitive to escape sequences. If
7633 the sequence is valid as ISO2022, the code is determined as one of
7634 the ISO2022 encodings, and the file is decoded by the corresponding
7635 coding system (e.g. `iso-2022-7bit').
7637 However, there may be a case that you want to read escape sequences in
7638 a file as is. In such a case, you can set this variable to non-nil.
7639 Then, as the code detection ignores any escape sequences, no file is
7640 detected as encoded in some ISO2022 encoding. The result is that all
7641 escape sequences become visible in a buffer.
7643 The default value is nil, and it is strongly recommended not to change
7644 it. That is because many Emacs Lisp source files that contain
7645 non-ASCII characters are encoded by the coding system `iso-2022-7bit'
7646 in Emacs's distribution, and they won't be decoded correctly on
7647 reading if you suppress escape sequence detection.
7649 The other way to read escape sequences in a file without decoding is
7650 to explicitly specify some coding system that doesn't use ISO2022's
7651 escape sequence (e.g `latin-1') on reading by \\[universal-coding-system-argument]. */);
7652 inhibit_iso_escape_detection = 0;
7654 DEFVAR_LISP ("translation-table-for-input", &Vtranslation_table_for_input,
7655 doc: /* Char table for translating self-inserting characters.
7656 This is applied to the result of input methods, not their input. See also
7657 `keyboard-translate-table'. */);
7658 Vtranslation_table_for_input = Qnil;
7661 char *
7662 emacs_strerror (error_number)
7663 int error_number;
7665 char *str;
7667 synchronize_system_messages_locale ();
7668 str = strerror (error_number);
7670 if (! NILP (Vlocale_coding_system))
7672 Lisp_Object dec = code_convert_string_norecord (build_string (str),
7673 Vlocale_coding_system,
7675 str = (char *) SDATA (dec);
7678 return str;
7681 #endif /* emacs */