1 -*-mode: text; coding: utf-8;-*-
3 Copyright (C) 2002-2017 Free Software Foundation, Inc.
4 See the end of the file for license conditions.
6 Importing a new Unicode Standard version into Emacs
7 -------------------------------------------------------------
9 Emacs uses the following files from the Unicode Character Database
17 . NormalizationTest.txt
18 . BidiCharacterTest.txt
20 First, the first 6 files need to be copied into admin/unidata/, and
21 then Emacs should be rebuilt for them to take effect. Rebuilding
22 Emacs updates several derived files elsewhere in the Emacs source
23 tree, mainly in lisp/international/.
25 When Emacs is rebuilt for the first time after importing the new
26 files, pay attention to any warning or error messages. In particular,
27 admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
28 new bidirectional attributes of characters, because unidata-gen.el,
29 bidi.c and dispextern.h need to be updated in that case; failure to do
30 so will cause aborts in redisplay.
32 Next, review the changes in UnicodeData.txt vs the previous version
33 used by Emacs. Any changes, be it introduction of new scripts or
34 addition of codepoints to existing scripts, might need corresponding
35 changes in the data used for filling the category-table, case-table,
36 and char-width-table. The additional scripts should cause automatic
37 updates in charscript.el, but it is a good idea to look at the results
38 and see if any changes in admin/unidata/blocks.awk are required.
40 The setting of char-width-table around line 1200 of characters.el
41 should be checked against the latest version of the Unicode file
42 EastAsianWidth.txt, and any discrepancies fixed.
44 Any new scripts added by UnicodeData.txt will also need updates to
45 script-representative-chars defined in fontset.el, and also the list
46 of OTF script tags in otf-script-alist, whose source is on this page:
48 https://www.microsoft.com/typography/otspec/scripttags.htm
50 Other databases in fontset.el might also need to be updated as needed.
52 The function 'ucs-names', defined in lisp/international/mule-cmds.el,
53 might need to be updated because it knows about used and unused ranges
54 of Unicode codepoints, which a new release of the Unicode Standard
57 Next, test normalization functions against NormalizationTests.txt,
58 in the test/ directory run:
60 make lisp/international/ucs-normalize-tests
62 See commentary in test/lisp/international/ucs-normalize-tests.el
63 regarding failing lines.
65 The file BidiCharacterTest.txt should be copied to the test suite, and
66 if its format has changed, the file biditest.el there should be
67 modified to follow suit.
69 Problems, fixmes and other unicode-related issues
70 -------------------------------------------------------------
72 Notes by fx to record various things of variable importance. Handa
73 needs to check them -- don't take too seriously, especially with
74 regard to completeness.
76 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
77 undesirable effects. E.g.:
78 (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
79 (multibyte-string-p (concat [?£])) => nil
80 (text-char-description ?£) => "M-#"
82 These examples are all fixed by the change of 2002-10-14, but
83 there still exist questionable SINGLE_BYTE_CHAR_P in the
84 code (keymap.c and print.c).
86 * Rationalize character syntax and its relationship to the Unicode
87 database. (Applies mainly to symbol an punctuation syntax.)
89 * Fontset handling and customization needs work. We want to relate
90 fonts to scripts, probably based on the Unicode blocks. The
91 presence of small-repertoire 10646-encoded fonts in XFree 4 is a
92 pain, not currently worked round.
94 With the change on 2002-07-26, multiple fonts can be
95 specified in a fontset for a specific range of characters.
96 Each range can also be specified by script. Before using
97 ISO10646 fonts, Emacs checks their repertories to avoid such
98 fonts that don't have a glyph for a specific character.
100 fx has worked on fontset customization, but was stymied by
101 basic problems with the way the default face is dealt with
102 (and something else, I think). This needs revisiting.
104 * Work is also needed on charset and coding system priorities.
106 * The relevant bits of latin1-disp.el need porting (and probably
107 re-naming/updating). See also cyril-util.el.
109 * Quail files need more work now the encoding is largely irrelevant.
111 * What to do with the old coding categories stuff?
113 * The preferred-coding-system property of charsets should probably be
114 junked unless it can be made more useful now.
116 * find-multibyte-characters needs looking at.
118 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
121 * Lazy-load tables for unify-charset somehow?
123 Actually, Emacs clears out all charset maps and unify-map just
124 before dumping, and they are loaded again on demand by the
125 dumped emacs. But, those maps (char tables) generated while
126 temacs is running can't be removed from the dumped emacs.
128 * iso-2022 charsets get unified on i/o.
130 With the change on 2003-01-06, decoding routines put the 'charset'
131 property onto decoded text, and iso-2022 encoder pay attention
132 to it. Thus, for instance, reading and writing by
133 iso-2022-7bit preserve the original designation sequences.
134 The property name 'preferred-charset' may be better?
136 We may have to utilize this property to decide a font.
138 * Revisit locale processing: look at treating the language and
139 charset parts separately. (Language should affect things like
140 spelling and calendar, but that's not a Unicode issue.)
142 * Handle Unicode combining characters usefully, e.g. diacritics, and
143 handle more scripts specifically (à la Devanagari). There are
144 issues with canonicalization.
146 * We need tabular input methods, e.g. for maths symbols. (Not
147 specific to Unicode.)
149 * Need multibyte text in menus, e.g. for the above. (Not specific to
150 Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
152 * Populate char-width-table correctly for Unicode characters and
153 worry about what happens when double-width charsets covering
154 non-CJK characters are unified.
156 * There are type errors lurking, e.g. in
157 Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
159 * Old auto-save files, and similar files, such as Gnus drafts,
160 containing non-ASCII characters probably won't be re-read correctly.
166 Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
167 subset), but there are a few exceptions, listed below. Perhaps
168 someday many of these files will be converted to UTF-8, for
169 convenience when using tools like 'grep -r', but this might need
170 nontrivial changes to the build process.
174 These are verbatim copies of files taken from external sources.
175 They haven't been converted to UTF-8.
177 leim/CXTERM-DIC/4Corner.tit
178 leim/CXTERM-DIC/ARRAY30.tit
179 leim/CXTERM-DIC/ECDICT.tit
180 leim/CXTERM-DIC/ETZY.tit
181 leim/CXTERM-DIC/PY-b5.tit
182 leim/CXTERM-DIC/Punct-b5.tit
183 leim/CXTERM-DIC/QJ-b5.tit
184 leim/CXTERM-DIC/ZOZY.tit
185 leim/MISC-DIC/CTLau-b5.html
186 leim/MISC-DIC/cangjie-table.b5
190 These are verbatim copies of files taken from external sources.
191 They haven't been converted to UTF-8.
193 leim/CXTERM-DIC/CCDOSPY.tit
194 leim/CXTERM-DIC/Punct.tit
195 leim/CXTERM-DIC/QJ.tit
196 leim/CXTERM-DIC/SW.tit
197 leim/CXTERM-DIC/TONEPY.tit
198 leim/MISC-DIC/CTLau.html
199 leim/MISC-DIC/pinyin.map
200 leim/MISC-DIC/ziranma.cin
204 This file contains non-ASCII characters in unibyte strings. When
205 editing a keyboard layout it's more convenient to see 'é' than
206 '\202', and the MS-DOS compiler requires the single byte if a
207 backslash escape is not being used.
213 This file is externally generated from leim/MISC-DIC/cangjie-table.b5
214 by Big5->CNS converter. It hasn't been converted to UTF-8.
216 leim/MISC-DIC/cangjie-table.cns
220 SKK-JISYO.L is a verbatim copy of a file taken from an external source.
221 It hasn't been converted to UTF-8.
223 leim/SKK-DIC/SKK-JISYO.L
227 This is a verbatim copy of a file taken from an external source.
228 It hasn't been converted to UTF-8.
230 admin/charsets/mapfiles/cns2ucsdkw.txt
234 This file switches between CJK charsets, which is not encoded in UTF-8.
238 Each of these files contains just one CJK charset, but Emacs
239 currently has no easy way to specify set-charset-priority on a
240 per-file basis, so converting any of these files to UTF-8 might
241 change the file's appearance when viewed by an Emacs that is
242 operating in some other language environment.
244 etc/tutorials/TUTORIAL.ja
245 lisp/international/ja-dic-cnv.el
246 lisp/international/ja-dic-utl.el
247 lisp/international/kinsoku.el
248 lisp/international/kkc.el
249 lisp/international/titdic-cnv.el
250 lisp/language/japan-util.el
251 lisp/language/japanese.el
252 lisp/leim/quail/cyril-jis.el
253 lisp/leim/quail/hanja-jis.el
254 lisp/leim/quail/japanese.el
255 lisp/leim/quail/py-punct.el
256 lisp/leim/quail/pypunct-b5.el
258 This file contains just Chinese characters, and has same problem.
259 Also, it contains characters that cannot be encoded in UTF-8.
261 lisp/international/titdic-cnv.el
265 These files contain characters that cannot be encoded in UTF-8.
267 lisp/language/ethio-util.el
268 lisp/language/ethiopic.el
269 lisp/language/ind-util.el
270 lisp/language/tibet-util.el
271 lisp/language/tibetan.el
272 lisp/leim/quail/ethiopic.el
273 lisp/leim/quail/tibetan.el
277 These files contain binary data, and are not text files.
278 Some of the entries in this list are patterns, and stand for any
279 files with the listed extension.
289 etc/package-keyring.gpg
291 nextstep/GNUstep/Emacs.base/Resources/emacs.tiff
295 This file is part of GNU Emacs.
297 GNU Emacs is free software: you can redistribute it and/or modify
298 it under the terms of the GNU General Public License as published by
299 the Free Software Foundation, either version 3 of the License, or
300 (at your option) any later version.
302 GNU Emacs is distributed in the hope that it will be useful,
303 but WITHOUT ANY WARRANTY; without even the implied warranty of
304 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
305 GNU General Public License for more details.
307 You should have received a copy of the GNU General Public License
308 along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.