1 -*-mode: text; coding: utf-8;-*-
3 Copyright (C) 2002-2017 Free Software Foundation, Inc.
4 See the end of the file for license conditions.
6 Importing a new Unicode Standard version into Emacs
7 -------------------------------------------------------------
9 Emacs uses the following files from the Unicode Character Database
17 . NormalizationTest.txt
19 . BidiCharacterTest.txt
21 First, the first 7 files need to be copied into admin/unidata/, and
22 then Emacs should be rebuilt for them to take effect. Rebuilding
23 Emacs updates several derived files elsewhere in the Emacs source
24 tree, mainly in lisp/international/.
26 When Emacs is rebuilt for the first time after importing the new
27 files, pay attention to any warning or error messages. In particular,
28 admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
29 new bidirectional attributes of characters, because unidata-gen.el,
30 bidi.c and dispextern.h need to be updated in that case; failure to do
31 so will cause aborts in redisplay.
33 Next, review the changes in UnicodeData.txt vs the previous version
34 used by Emacs. Any changes, be it introduction of new scripts or
35 addition of codepoints to existing scripts, might need corresponding
36 changes in the data used for filling the category-table, case-table,
37 and char-width-table. The additional scripts should cause automatic
38 updates in charscript.el, but it is a good idea to look at the results
39 and see if any changes in admin/unidata/blocks.awk are required.
41 The setting of char-width-table around line 1200 of characters.el
42 should be checked against the latest version of the Unicode file
43 EastAsianWidth.txt, and any discrepancies fixed.
45 Any new scripts added by UnicodeData.txt will also need updates to
46 script-representative-chars defined in fontset.el, and also the list
47 of OTF script tags in otf-script-alist, whose source is on this page:
49 https://www.microsoft.com/typography/otspec/scripttags.htm
51 Other databases in fontset.el might also need to be updated as needed.
53 The function 'ucs-names', defined in lisp/international/mule-cmds.el,
54 might need to be updated because it knows about used and unused ranges
55 of Unicode codepoints, which a new release of the Unicode Standard
58 Next, test normalization functions against NormalizationTests.txt,
59 in the test/ directory run:
61 make lisp/international/ucs-normalize-tests
63 See commentary in test/lisp/international/ucs-normalize-tests.el
64 regarding failing lines.
66 The file BidiCharacterTest.txt should be copied to the test suite, and
67 if its format has changed, the file biditest.el there should be
68 modified to follow suit.
70 Problems, fixmes and other unicode-related issues
71 -------------------------------------------------------------
73 Notes by fx to record various things of variable importance. Handa
74 needs to check them -- don't take too seriously, especially with
75 regard to completeness.
77 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
78 undesirable effects. E.g.:
79 (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
80 (multibyte-string-p (concat [?£])) => nil
81 (text-char-description ?£) => "M-#"
83 These examples are all fixed by the change of 2002-10-14, but
84 there still exist questionable SINGLE_BYTE_CHAR_P in the
85 code (keymap.c and print.c).
87 * Rationalize character syntax and its relationship to the Unicode
88 database. (Applies mainly to symbol an punctuation syntax.)
90 * Fontset handling and customization needs work. We want to relate
91 fonts to scripts, probably based on the Unicode blocks. The
92 presence of small-repertoire 10646-encoded fonts in XFree 4 is a
93 pain, not currently worked round.
95 With the change on 2002-07-26, multiple fonts can be
96 specified in a fontset for a specific range of characters.
97 Each range can also be specified by script. Before using
98 ISO10646 fonts, Emacs checks their repertories to avoid such
99 fonts that don't have a glyph for a specific character.
101 fx has worked on fontset customization, but was stymied by
102 basic problems with the way the default face is dealt with
103 (and something else, I think). This needs revisiting.
105 * Work is also needed on charset and coding system priorities.
107 * The relevant bits of latin1-disp.el need porting (and probably
108 re-naming/updating). See also cyril-util.el.
110 * Quail files need more work now the encoding is largely irrelevant.
112 * What to do with the old coding categories stuff?
114 * The preferred-coding-system property of charsets should probably be
115 junked unless it can be made more useful now.
117 * find-multibyte-characters needs looking at.
119 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
122 * Lazy-load tables for unify-charset somehow?
124 Actually, Emacs clears out all charset maps and unify-map just
125 before dumping, and they are loaded again on demand by the
126 dumped emacs. But, those maps (char tables) generated while
127 temacs is running can't be removed from the dumped emacs.
129 * iso-2022 charsets get unified on i/o.
131 With the change on 2003-01-06, decoding routines put the 'charset'
132 property onto decoded text, and iso-2022 encoder pay attention
133 to it. Thus, for instance, reading and writing by
134 iso-2022-7bit preserve the original designation sequences.
135 The property name 'preferred-charset' may be better?
137 We may have to utilize this property to decide a font.
139 * Revisit locale processing: look at treating the language and
140 charset parts separately. (Language should affect things like
141 spelling and calendar, but that's not a Unicode issue.)
143 * Handle Unicode combining characters usefully, e.g. diacritics, and
144 handle more scripts specifically (à la Devanagari). There are
145 issues with canonicalization.
147 * We need tabular input methods, e.g. for maths symbols. (Not
148 specific to Unicode.)
150 * Need multibyte text in menus, e.g. for the above. (Not specific to
151 Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
153 * Populate char-width-table correctly for Unicode characters and
154 worry about what happens when double-width charsets covering
155 non-CJK characters are unified.
157 * There are type errors lurking, e.g. in
158 Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
160 * Old auto-save files, and similar files, such as Gnus drafts,
161 containing non-ASCII characters probably won't be re-read correctly.
167 Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
168 subset), but there are a few exceptions, listed below. Perhaps
169 someday many of these files will be converted to UTF-8, for
170 convenience when using tools like 'grep -r', but this might need
171 nontrivial changes to the build process.
175 These are verbatim copies of files taken from external sources.
176 They haven't been converted to UTF-8.
178 leim/CXTERM-DIC/4Corner.tit
179 leim/CXTERM-DIC/ARRAY30.tit
180 leim/CXTERM-DIC/ECDICT.tit
181 leim/CXTERM-DIC/ETZY.tit
182 leim/CXTERM-DIC/PY-b5.tit
183 leim/CXTERM-DIC/Punct-b5.tit
184 leim/CXTERM-DIC/QJ-b5.tit
185 leim/CXTERM-DIC/ZOZY.tit
186 leim/MISC-DIC/CTLau-b5.html
187 leim/MISC-DIC/cangjie-table.b5
191 These are verbatim copies of files taken from external sources.
192 They haven't been converted to UTF-8.
194 leim/CXTERM-DIC/CCDOSPY.tit
195 leim/CXTERM-DIC/Punct.tit
196 leim/CXTERM-DIC/QJ.tit
197 leim/CXTERM-DIC/SW.tit
198 leim/CXTERM-DIC/TONEPY.tit
199 leim/MISC-DIC/CTLau.html
200 leim/MISC-DIC/pinyin.map
201 leim/MISC-DIC/ziranma.cin
205 This file contains non-ASCII characters in unibyte strings. When
206 editing a keyboard layout it's more convenient to see 'é' than
207 '\202', and the MS-DOS compiler requires the single byte if a
208 backslash escape is not being used.
214 This file is externally generated from leim/MISC-DIC/cangjie-table.b5
215 by Big5->CNS converter. It hasn't been converted to UTF-8.
217 leim/MISC-DIC/cangjie-table.cns
221 SKK-JISYO.L is a verbatim copy of a file taken from an external source.
222 It hasn't been converted to UTF-8.
224 leim/SKK-DIC/SKK-JISYO.L
228 This is a verbatim copy of a file taken from an external source.
229 It hasn't been converted to UTF-8.
231 admin/charsets/mapfiles/cns2ucsdkw.txt
235 This file switches between CJK charsets, which is not encoded in UTF-8.
239 Each of these files contains just one CJK charset, but Emacs
240 currently has no easy way to specify set-charset-priority on a
241 per-file basis, so converting any of these files to UTF-8 might
242 change the file's appearance when viewed by an Emacs that is
243 operating in some other language environment.
245 etc/tutorials/TUTORIAL.ja
246 lisp/international/ja-dic-cnv.el
247 lisp/international/ja-dic-utl.el
248 lisp/international/kinsoku.el
249 lisp/international/kkc.el
250 lisp/international/titdic-cnv.el
251 lisp/language/japan-util.el
252 lisp/language/japanese.el
253 lisp/leim/quail/cyril-jis.el
254 lisp/leim/quail/hanja-jis.el
255 lisp/leim/quail/japanese.el
256 lisp/leim/quail/py-punct.el
257 lisp/leim/quail/pypunct-b5.el
259 This file contains just Chinese characters, and has same problem.
260 Also, it contains characters that cannot be encoded in UTF-8.
262 lisp/international/titdic-cnv.el
266 These files contain characters that cannot be encoded in UTF-8.
268 lisp/language/ethio-util.el
269 lisp/language/ethiopic.el
270 lisp/language/ind-util.el
271 lisp/language/tibet-util.el
272 lisp/language/tibetan.el
273 lisp/leim/quail/ethiopic.el
274 lisp/leim/quail/tibetan.el
278 These files contain binary data, and are not text files.
279 Some of the entries in this list are patterns, and stand for any
280 files with the listed extension.
290 etc/package-keyring.gpg
292 nextstep/GNUstep/Emacs.base/Resources/emacs.tiff
296 This file is part of GNU Emacs.
298 GNU Emacs is free software: you can redistribute it and/or modify
299 it under the terms of the GNU General Public License as published by
300 the Free Software Foundation, either version 3 of the License, or
301 (at your option) any later version.
303 GNU Emacs is distributed in the hope that it will be useful,
304 but WITHOUT ANY WARRANTY; without even the implied warranty of
305 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
306 GNU General Public License for more details.
308 You should have received a copy of the GNU General Public License
309 along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>.