x86-64: Integrate Paul Khuong's interleaved raw slot feature.
[sbcl.git] / src / code / early-raw-slots.lisp
blob9e1ae187b6d4839036a926ac6f88ec14ab38b922
1 ;;;; This software is part of the SBCL system. See the README file for
2 ;;;; more information.
3 ;;;;
4 ;;;; This software is derived from the CMU CL system, which was
5 ;;;; written at Carnegie Mellon University and released into the
6 ;;;; public domain. The software is in the public domain and is
7 ;;;; provided with absolutely no warranty. See the COPYING and CREDITS
8 ;;;; files for more information.
10 (in-package "SB!KERNEL")
12 ;;; This file has to be loaded during cold-init as early as you'd like to
13 ;;; have any defstructs that use raw slots. %COMPILER-DEFSTRUCT needs the
14 ;;; raw-slot-data-list both at compile-time and load-time.
16 ;;; STRUCTURE-OBJECT supports two different strategies to place raw slots
17 ;;; (containing "just bits", not Lisp descriptors) within it in a way
18 ;;; that GC has knowledge of. No backend supports both strategies though.
20 ;;; The older strategy is "non-interleaved".
21 ;;; Consider a structure of 3 tagged slots (A,B,C) and 2 raw slots,
22 ;;; where (for simplicity) each raw slot takes space equal to one Lisp word.
23 ;;; (In general raw slots can take >1 word)
24 ;;; Lisp code arranges so that raw slots are last.
25 ;;; Word offsets are listed on the left
26 ;;; 0 : header = (instance-length << 8) | instance-header-widetag
27 ;;; 1 : dsd-index 0 = ptr to LAYOUT
28 ;;; 2 : dsd-index 1 = tagged slot A
29 ;;; 3 : dsd-index 2 = ... B
30 ;;; 4 : dsd-index 3 = ... C
31 ;;; 5 : filler
32 ;;; 6 : dsd-index 1 = second raw slot
33 ;;; 7 : dsd-index 0 = first raw slot
34 ;;;
35 ;;; Note that numbering of raw slots with respect to their DSD-INDEX
36 ;;; restarts at 0, so there are two "spaces" of dsd-indices, the non-raw
37 ;;; and the raw space. Also note that filler was added in the middle, so
38 ;;; that adding INSTANCE-LENGTH to the object's address always gets you
39 ;;; to exactly the 0th raw slot. The filler can't be squeezed out, because
40 ;;; all Lisp objects must consume an even number of words, and the length
41 ;;; of an instance reflects the number of physical - not logical - words
42 ;;; that follow the instance header.
43 ;;;
44 ;;; This strategy for placement of raw slots is easy for GC because GC's
45 ;;; view of an instance is simply some number of boxed words followed by
46 ;;; some number of ignored words.
47 ;;; However, this strategy presents a difficulty for Lisp in that a raw
48 ;;; slot at a given index is not at a fixed offset relative to the base of
49 ;;; the object - it is fixed relative to the _last_ word of the object.
50 ;;; This has to do with the requirement that structure accessors defined by
51 ;;; a parent type work correctly on a descendant type, while preserving the
52 ;;; simple-for-GC aspect. If another DEFSTRUCT says to :INCLUDE the above,
53 ;;; adding two more tagged slots D and E, the slot named D occupies word 5
54 ;;; ('filler' above), E occupies word 6, and the two raw slots shift down.
55 ;;; To read raw slot at index N requires adding to the object pointer
56 ;;; the number of words represented by instance-length and subtracting the
57 ;;; raw slot index.
58 ;;; Aside from instance-length, the only additional piece of information
59 ;;; that GC needs to know to scavenge a structure is the number of raw slots,
60 ;;; which is obtained from the object's layout in the N-UNTAGGED-SLOTS slot.
62 ;;; Assuming that it is more important to simplify runtime access than
63 ;;; to simplify GC, we can use the newer strategy, "interleaved" raw slots.
64 ;;; Interleaving freely intermingles tagged data with untagged data
65 ;;; following the layout. This permits descendant structures to add
66 ;;; slots of any kind to the end without changing any physical placement
67 ;;; that was already determined, and eliminates the runtime computation
68 ;;; of the offset to raw slots. It is also generally easier to understand.
69 ;;; The trade-off is that GC (and a few other things - structure dumping,
70 ;;; EQUALP checking, to name a few) have to be able to determine for each
71 ;;; slot whether it is a Lisp descriptor or just bits. This is done
72 ;;; with the LAYOUT-UNTAGGED-BITMAP of an object's layout.
73 ;;; The bitmap stores a '1' for each bit representing a raw word,
74 ;;; and could be a BIGNUM given a spectacularly huge structure.
76 ;;; Also note that in both strategies there are possibly some alignment
77 ;;; concerns which must be accounted for when DEFSTRUCT lays out slots,
78 ;;; by injecting padding words appropriately.
79 ;;; For example COMPLEX-DOUBLE-FLOAT *should* be aligned to twice the
80 ;;; alignment of a DOUBLE-FLOAT. It is not, as things stand,
81 ;;; but this is considered a minor bug.
82 ;;; Interleaving is supported only on x86-64, but porting should be
83 ;;; straightforward, because if anything the VOPs become simpler.
85 ;; To utilize a word-sized slot in a defstruct without having to resort to
86 ;; writing (myslot :type (unsigned-byte #.sb!vm:n-word-bits)), or even
87 ;; worse (:type #+sb-xc-host <sometype> #-sb-xc-host <othertype>),
88 ;; these abstractions are provided as soon as the raw slots defs are.
89 ;; 'signed-word' is here for companionship - slots of that type are not raw.
90 (def!type sb!vm:word () `(unsigned-byte ,sb!vm:n-word-bits))
91 (def!type sb!vm:signed-word () `(signed-byte ,sb!vm:n-word-bits))
93 ;; These definitions pertain to how a LAYOUT stores the raw-slot metadata,
94 ;; and we need them before 'class.lisp' is compiled (why, I'm can't remember).
95 ;; LAYOUT-RAW-SLOT-METADATA is an abstraction over whichever kind of
96 ;; metadata we have - it will be one or the other.
97 #!-interleaved-raw-slots
98 (progn (deftype layout-raw-slot-metadata-type () 'index)
99 (defmacro layout-raw-slot-metadata (x) `(layout-n-untagged-slots ,x)))
100 ;; It would be possible to represent an unlimited number of trailing untagged
101 ;; slots (maybe) without consing a bignum if we wished to allow signed integers
102 ;; for the raw slot bitmap, but that's probably confusing and pointless, so...
103 #!+interleaved-raw-slots
104 (progn (deftype layout-raw-slot-metadata-type () 'unsigned-byte)
105 (defmacro layout-raw-slot-metadata (x) `(layout-untagged-bitmap ,x)))
107 ;; information about how a slot of a given DSD-RAW-TYPE is to be accessed
108 (eval-when (:compile-toplevel :load-toplevel :execute)
109 (defstruct (raw-slot-data
110 (:copier nil)
111 (:predicate nil))
112 ;; the raw slot type, or T for a non-raw slot
114 ;; (Non-raw slots are in the ordinary place you'd expect, directly
115 ;; indexed off the instance pointer. Raw slots are indexed from the end
116 ;; of the instance and skipped by GC.)
117 (raw-type (missing-arg) :type (or symbol cons) :read-only t)
118 ;; What operator is used to access a slot of this type?
119 (accessor-name (missing-arg) :type symbol :read-only t)
120 (init-vop (missing-arg) :type symbol :read-only t)
121 ;; How many words are each value of this type?
122 (n-words (missing-arg) :type (and index (integer 1)) :read-only t)
123 ;; Necessary alignment in units of words. Note that instances
124 ;; themselves are aligned by exactly two words, so specifying more
125 ;; than two words here would not work.
126 (alignment 1 :type (integer 1 2) :read-only t)
127 (comparer (missing-arg) :type function :read-only t)))
129 #!-sb-fluid (declaim (freeze-type raw-slot-data))
131 (defglobal *raw-slot-data-list*
132 (macrolet ((make-comparer (accessor-name)
133 ;; Not a symbol, because there aren't any so-named functions.
134 `(named-lambda ,(string (symbolicate accessor-name "="))
135 (index x y)
136 (declare (optimize speed (safety 0)))
137 (= (,accessor-name x index)
138 (,accessor-name y index)))))
139 (let ((double-float-alignment
140 ;; white list of architectures that can load unaligned doubles:
141 #!+(or x86 x86-64 ppc) 1
142 ;; at least sparc, mips and alpha can't:
143 #!-(or x86 x86-64 ppc) 2))
144 (list
145 (make-raw-slot-data :raw-type 'sb!vm:word
146 :accessor-name '%raw-instance-ref/word
147 :init-vop 'sb!vm::raw-instance-init/word
148 :n-words 1
149 :comparer (make-comparer %raw-instance-ref/word))
150 (make-raw-slot-data :raw-type 'single-float
151 :accessor-name '%raw-instance-ref/single
152 :init-vop 'sb!vm::raw-instance-init/single
153 ;; KLUDGE: On 64 bit architectures, we
154 ;; could pack two SINGLE-FLOATs into the
155 ;; same word if raw slots were indexed
156 ;; using bytes instead of words. However,
157 ;; I don't personally find optimizing
158 ;; SINGLE-FLOAT memory usage worthwile
159 ;; enough. And the other datatype that
160 ;; would really benefit is (UNSIGNED-BYTE
161 ;; 32), but that is a subtype of FIXNUM, so
162 ;; we store it unraw anyway. :-( -- DFL
163 :n-words 1
164 :comparer (make-comparer %raw-instance-ref/single))
165 (make-raw-slot-data :raw-type 'double-float
166 :accessor-name '%raw-instance-ref/double
167 :init-vop 'sb!vm::raw-instance-init/double
168 :alignment double-float-alignment
169 :n-words (/ 8 sb!vm:n-word-bytes)
170 :comparer (make-comparer %raw-instance-ref/double))
171 (make-raw-slot-data :raw-type 'complex-single-float
172 :accessor-name '%raw-instance-ref/complex-single
173 :init-vop 'sb!vm::raw-instance-init/complex-single
174 :n-words (/ 8 sb!vm:n-word-bytes)
175 :comparer (make-comparer %raw-instance-ref/complex-single))
176 (make-raw-slot-data :raw-type 'complex-double-float
177 :accessor-name '%raw-instance-ref/complex-double
178 :init-vop 'sb!vm::raw-instance-init/complex-double
179 :alignment double-float-alignment
180 :n-words (/ 16 sb!vm:n-word-bytes)
181 :comparer (make-comparer %raw-instance-ref/complex-double))
182 #!+long-float
183 (make-raw-slot-data :raw-type long-float
184 :accessor-name '%raw-instance-ref/long
185 :init-vop 'sb!vm::raw-instance-init/long
186 :n-words #!+x86 3 #!+sparc 4
187 :comparer (make-comparer %raw-instance-ref/long))
188 #!+long-float
189 (make-raw-slot-data :raw-type complex-long-float
190 :accessor-name '%raw-instance-ref/complex-long
191 :init-vop 'sb!vm::raw-instance-init/complex-long
192 :n-words #!+x86 6 #!+sparc 8
193 :comparer (make-comparer %raw-instance-ref/complex-long))))))
195 (declaim (ftype (sfunction (symbol) raw-slot-data) raw-slot-data-or-lose))
196 (defun raw-slot-data-or-lose (type)
197 (or (car (member type *raw-slot-data-list* :key #'raw-slot-data-raw-type))
198 (error "Invalid raw slot type: ~S" type)))
200 (defun raw-slot-words (type)
201 (raw-slot-data-n-words (raw-slot-data-or-lose type)))
203 ;; DO-INSTANCE-TAGGED-SLOT will iterate over the slots of THING that
204 ;; contain tagged objects. INDEX-VAR is bound to successive slot-indices,
205 ;; and is usually used as the second argument to %INSTANCE-REF.
206 ;; START, if supplied, should be either 0 or 1 to include,
207 ;; or respectively exclude, the object's layout slot.
208 ;; END, if supplied, represents the upper bound of the scan and should be
209 ;; the LAYOUT-LENGTH of the object; it defaults to %INSTANCE-LENGTH if
210 ;; unsupplied, will therefore possibly include one word that can, in theory,
211 ;; covertly hold one tagged object more than indicated by LAYOUT-LENGTH
212 ;; depending on ODDP of LAYOUT-LENGTH.
213 ;; END works correctly whether or not the backend supports slot interleaving,
214 ;; but it is probably a bug if anyone uses the padding slot for storage.
215 ;; LAYOUT is optional and somewhat unnecessary, but since some uses of
216 ;; this macro already have a layout in hand, it can be supplied.
217 ;; [If the compiler were smarter about doing fewer memory accesses,
218 ;; there would be no need at all for the LAYOUT - if it had already been
219 ;; accessed, it shouldn't be another memory read]
220 ;; Note also that THING is usually a STRUCTURE-OBJECT, not a condition or
221 ;; standard-object. Iterating over a CONDITION means iterating over the
222 ;; slots comprising the primitive representation, not the manifest slots.
223 ;; Similarly for STANDARD-OBJECT. Additionally, in the latter case
224 ;; it would be a bug to specify LAYOUT-LENGTH as the :END parameter.
225 (defmacro do-instance-tagged-slot ((index-var thing
226 &key (start 0) end layout) &body body)
227 (with-unique-names (instance limit bitmap)
228 (declare (ignorable bitmap))
229 (unless layout
230 (setq layout `(%instance-layout ,instance)))
231 (unless end
232 (setq end `(%instance-length ,instance)))
233 `(let ((,instance ,thing))
234 ;; If the macro is given both :LAYOUT and :END, it never uses
235 ;; the local rebinding of INSTANCE, which is ok.
236 (declare (ignorable ,instance))
237 #!+interleaved-raw-slots
238 (let ((,bitmap (layout-untagged-bitmap ,layout)))
239 (do ((,index-var ,start (1+ ,index-var))
240 (,limit ,end))
241 ((>= ,index-var ,limit))
242 (declare (index ,index-var))
243 (unless (logbitp ,index-var ,bitmap)
244 ,@body)))
245 #!-interleaved-raw-slots
246 (do ((,index-var ,start (1+ ,index-var))
247 (,limit (- ,end (layout-n-untagged-slots ,layout))))
248 ((>= ,index-var ,limit))
249 (declare (index ,index-var))
250 ,@body))))