Add better error reporting for MemoryErrors caused by str->float conversions.
[python.git] / Demo / metaclasses / index.html
blobeee473a814f91bd179195390dba543e2163185c4
1 <HTML>
3 <HEAD>
4 <TITLE>Metaclasses in Python 1.5</TITLE>
5 </HEAD>
7 <BODY BGCOLOR="FFFFFF">
9 <H1>Metaclasses in Python 1.5</H1>
10 <H2>(A.k.a. The Killer Joke :-)</H2>
12 <HR>
14 (<i>Postscript:</i> reading this essay is probably not the best way to
15 understand the metaclass hook described here. See a <A
16 HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
17 which may give a gentler introduction to the matter. You may also
18 want to search Deja News for messages with "metaclass" in the subject
19 posted to comp.lang.python in July and August 1998.)
21 <HR>
23 <P>In previous Python releases (and still in 1.5), there is something
24 called the ``Don Beaudry hook'', after its inventor and champion.
25 This allows C extensions to provide alternate class behavior, thereby
26 allowing the Python class syntax to be used to define other class-like
27 entities. Don Beaudry has used this in his infamous <A
28 HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
29 Fulton has used it in his <A
30 HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
31 Classes</A> package. (It has also been referred to as the ``Don
32 Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
33 about it -- in fact, it is rather elegant and deep, even though
34 there's something dark to it.)
36 <P>(On first reading, you may want to skip directly to the examples in
37 the section "Writing Metaclasses in Python" below, unless you want
38 your head to explode.)
40 <P>
42 <HR>
44 <P>Documentation of the Don Beaudry hook has purposefully been kept
45 minimal, since it is a feature of incredible power, and is easily
46 abused. Basically, it checks whether the <b>type of the base
47 class</b> is callable, and if so, it is called to create the new
48 class.
50 <P>Note the two indirection levels. Take a simple example:
52 <PRE>
53 class B:
54 pass
56 class C(B):
57 pass
58 </PRE>
60 Take a look at the second class definition, and try to fathom ``the
61 type of the base class is callable.''
63 <P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
64 particular 6.22 in the <A
65 HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
66 for more on this topic.)
68 <P>
70 <UL>
72 <LI>The <b>base class</b> is B; this one's easy.<P>
74 <LI>Since B is a class, its type is ``class''; so the <b>type of the
75 base class</b> is the type ``class''. This is also known as
76 types.ClassType, assuming the standard module <code>types</code> has
77 been imported.<P>
79 <LI>Now is the type ``class'' <b>callable</b>? No, because types (in
80 core Python) are never callable. Classes are callable (calling a
81 class creates a new instance) but types aren't.<P>
83 </UL>
85 <P>So our conclusion is that in our example, the type of the base
86 class (of C) is not callable. So the Don Beaudry hook does not apply,
87 and the default class creation mechanism is used (which is also used
88 when there is no base class). In fact, the Don Beaudry hook never
89 applies when using only core Python, since the type of a core object
90 is never callable.
92 <P>So what do Don and Jim do in order to use Don's hook? Write an
93 extension that defines at least two new Python object types. The
94 first would be the type for ``class-like'' objects usable as a base
95 class, to trigger Don's hook. This type must be made callable.
96 That's why we need a second type. Whether an object is callable
97 depends on its type. So whether a type object is callable depends on
98 <i>its</i> type, which is a <i>meta-type</i>. (In core Python there
99 is only one meta-type, the type ``type'' (types.TypeType), which is
100 the type of all type objects, even itself.) A new meta-type must
101 be defined that makes the type of the class-like objects callable.
102 (Normally, a third type would also be needed, the new ``instance''
103 type, but this is not an absolute requirement -- the new class type
104 could return an object of some existing type when invoked to create an
105 instance.)
107 <P>Still confused? Here's a simple device due to Don himself to
108 explain metaclasses. Take a simple class definition; assume B is a
109 special class that triggers Don's hook:
111 <PRE>
112 class C(B):
113 a = 1
114 b = 2
115 </PRE>
117 This can be though of as equivalent to:
119 <PRE>
120 C = type(B)('C', (B,), {'a': 1, 'b': 2})
121 </PRE>
123 If that's too dense for you, here's the same thing written out using
124 temporary variables:
126 <PRE>
127 creator = type(B) # The type of the base class
128 name = 'C' # The name of the new class
129 bases = (B,) # A tuple containing the base class(es)
130 namespace = {'a': 1, 'b': 2} # The namespace of the class statement
131 C = creator(name, bases, namespace)
132 </PRE>
134 This is analogous to what happens without the Don Beaudry hook, except
135 that in that case the creator function is set to the default class
136 creator.
138 <P>In either case, the creator is called with three arguments. The
139 first one, <i>name</i>, is the name of the new class (as given at the
140 top of the class statement). The <i>bases</i> argument is a tuple of
141 base classes (a singleton tuple if there's only one base class, like
142 the example). Finally, <i>namespace</i> is a dictionary containing
143 the local variables collected during execution of the class statement.
145 <P>Note that the contents of the namespace dictionary is simply
146 whatever names were defined in the class statement. A little-known
147 fact is that when Python executes a class statement, it enters a new
148 local namespace, and all assignments and function definitions take
149 place in this namespace. Thus, after executing the following class
150 statement:
152 <PRE>
153 class C:
154 a = 1
155 def f(s): pass
156 </PRE>
158 the class namespace's contents would be {'a': 1, 'f': &lt;function f
159 ...&gt;}.
161 <P>But enough already about writing Python metaclasses in C; read the
162 documentation of <A
163 HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
164 HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
165 Classes</A> for more information.
169 <HR>
171 <H2>Writing Metaclasses in Python</H2>
173 <P>In Python 1.5, the requirement to write a C extension in order to
174 write metaclasses has been dropped (though you can still do
175 it, of course). In addition to the check ``is the type of the base
176 class callable,'' there's a check ``does the base class have a
177 __class__ attribute.'' If so, it is assumed that the __class__
178 attribute refers to a class.
180 <P>Let's repeat our simple example from above:
182 <PRE>
183 class C(B):
184 a = 1
185 b = 2
186 </PRE>
188 Assuming B has a __class__ attribute, this translates into:
190 <PRE>
191 C = B.__class__('C', (B,), {'a': 1, 'b': 2})
192 </PRE>
194 This is exactly the same as before except that instead of type(B),
195 B.__class__ is invoked. If you have read <A HREF=
196 "http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
197 >FAQ question 6.22</A> you will understand that while there is a big
198 technical difference between type(B) and B.__class__, they play the
199 same role at different abstraction levels. And perhaps at some point
200 in the future they will really be the same thing (at which point you
201 would be able to derive subclasses from built-in types).
203 <P>At this point it may be worth mentioning that C.__class__ is the
204 same object as B.__class__, i.e., C's metaclass is the same as B's
205 metaclass. In other words, subclassing an existing class creates a
206 new (meta)inststance of the base class's metaclass.
208 <P>Going back to the example, the class B.__class__ is instantiated,
209 passing its constructor the same three arguments that are passed to
210 the default class constructor or to an extension's metaclass:
211 <i>name</i>, <i>bases</i>, and <i>namespace</i>.
213 <P>It is easy to be confused by what exactly happens when using a
214 metaclass, because we lose the absolute distinction between classes
215 and instances: a class is an instance of a metaclass (a
216 ``metainstance''), but technically (i.e. in the eyes of the python
217 runtime system), the metaclass is just a class, and the metainstance
218 is just an instance. At the end of the class statement, the metaclass
219 whose metainstance is used as a base class is instantiated, yielding a
220 second metainstance (of the same metaclass). This metainstance is
221 then used as a (normal, non-meta) class; instantiation of the class
222 means calling the metainstance, and this will return a real instance.
223 And what class is that an instance of? Conceptually, it is of course
224 an instance of our metainstance; but in most cases the Python runtime
225 system will see it as an instance of a a helper class used by the
226 metaclass to implement its (non-meta) instances...
228 <P>Hopefully an example will make things clearer. Let's presume we
229 have a metaclass MetaClass1. It's helper class (for non-meta
230 instances) is callled HelperClass1. We now (manually) instantiate
231 MetaClass1 once to get an empty special base class:
233 <PRE>
234 BaseClass1 = MetaClass1("BaseClass1", (), {})
235 </PRE>
237 We can now use BaseClass1 as a base class in a class statement:
239 <PRE>
240 class MySpecialClass(BaseClass1):
241 i = 1
242 def f(s): pass
243 </PRE>
245 At this point, MySpecialClass is defined; it is a metainstance of
246 MetaClass1 just like BaseClass1, and in fact the expression
247 ``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
248 yields true.
250 <P>We are now ready to create instances of MySpecialClass. Let's
251 assume that no constructor arguments are required:
253 <PRE>
254 x = MySpecialClass()
255 y = MySpecialClass()
256 print x.__class__, y.__class__
257 </PRE>
259 The print statement shows that x and y are instances of HelperClass1.
260 How did this happen? MySpecialClass is an instance of MetaClass1
261 (``meta'' is irrelevant here); when an instance is called, its
262 __call__ method is invoked, and presumably the __call__ method defined
263 by MetaClass1 returns an instance of HelperClass1.
265 <P>Now let's see how we could use metaclasses -- what can we do
266 with metaclasses that we can't easily do without them? Here's one
267 idea: a metaclass could automatically insert trace calls for all
268 method calls. Let's first develop a simplified example, without
269 support for inheritance or other ``advanced'' Python features (we'll
270 add those later).
272 <PRE>
273 import types
275 class Tracing:
276 def __init__(self, name, bases, namespace):
277 """Create a new class."""
278 self.__name__ = name
279 self.__bases__ = bases
280 self.__namespace__ = namespace
281 def __call__(self):
282 """Create a new instance."""
283 return Instance(self)
285 class Instance:
286 def __init__(self, klass):
287 self.__klass__ = klass
288 def __getattr__(self, name):
289 try:
290 value = self.__klass__.__namespace__[name]
291 except KeyError:
292 raise AttributeError, name
293 if type(value) is not types.FunctionType:
294 return value
295 return BoundMethod(value, self)
297 class BoundMethod:
298 def __init__(self, function, instance):
299 self.function = function
300 self.instance = instance
301 def __call__(self, *args):
302 print "calling", self.function, "for", self.instance, "with", args
303 return apply(self.function, (self.instance,) + args)
305 Trace = Tracing('Trace', (), {})
307 class MyTracedClass(Trace):
308 def method1(self, a):
309 self.a = a
310 def method2(self):
311 return self.a
313 aninstance = MyTracedClass()
315 aninstance.method1(10)
317 print "the answer is %d" % aninstance.method2()
318 </PRE>
320 Confused already? The intention is to read this from top down. The
321 Tracing class is the metaclass we're defining. Its structure is
322 really simple.
326 <UL>
328 <LI>The __init__ method is invoked when a new Tracing instance is
329 created, e.g. the definition of class MyTracedClass later in the
330 example. It simply saves the class name, base classes and namespace
331 as instance variables.<P>
333 <LI>The __call__ method is invoked when a Tracing instance is called,
334 e.g. the creation of aninstance later in the example. It returns an
335 instance of the class Instance, which is defined next.<P>
337 </UL>
339 <P>The class Instance is the class used for all instances of classes
340 built using the Tracing metaclass, e.g. aninstance. It has two
341 methods:
345 <UL>
347 <LI>The __init__ method is invoked from the Tracing.__call__ method
348 above to initialize a new instance. It saves the class reference as
349 an instance variable. It uses a funny name because the user's
350 instance variables (e.g. self.a later in the example) live in the same
351 namespace.<P>
353 <LI>The __getattr__ method is invoked whenever the user code
354 references an attribute of the instance that is not an instance
355 variable (nor a class variable; but except for __init__ and
356 __getattr__ there are no class variables). It will be called, for
357 example, when aninstance.method1 is referenced in the example, with
358 self set to aninstance and name set to the string "method1".<P>
360 </UL>
362 <P>The __getattr__ method looks the name up in the __namespace__
363 dictionary. If it isn't found, it raises an AttributeError exception.
364 (In a more realistic example, it would first have to look through the
365 base classes as well.) If it is found, there are two possibilities:
366 it's either a function or it isn't. If it's not a function, it is
367 assumed to be a class variable, and its value is returned. If it's a
368 function, we have to ``wrap'' it in instance of yet another helper
369 class, BoundMethod.
371 <P>The BoundMethod class is needed to implement a familiar feature:
372 when a method is defined, it has an initial argument, self, which is
373 automatically bound to the relevant instance when it is called. For
374 example, aninstance.method1(10) is equivalent to method1(aninstance,
375 10). In the example if this call, first a temporary BoundMethod
376 instance is created with the following constructor call: temp =
377 BoundMethod(method1, aninstance); then this instance is called as
378 temp(10). After the call, the temporary instance is discarded.
382 <UL>
384 <LI>The __init__ method is invoked for the constructor call
385 BoundMethod(method1, aninstance). It simply saves away its
386 arguments.<P>
388 <LI>The __call__ method is invoked when the bound method instance is
389 called, as in temp(10). It needs to call method1(aninstance, 10).
390 However, even though self.function is now method1 and self.instance is
391 aninstance, it can't call self.function(self.instance, args) directly,
392 because it should work regardless of the number of arguments passed.
393 (For simplicity, support for keyword arguments has been omitted.)<P>
395 </UL>
397 <P>In order to be able to support arbitrary argument lists, the
398 __call__ method first constructs a new argument tuple. Conveniently,
399 because of the notation *args in __call__'s own argument list, the
400 arguments to __call__ (except for self) are placed in the tuple args.
401 To construct the desired argument list, we concatenate a singleton
402 tuple containing the instance with the args tuple: (self.instance,) +
403 args. (Note the trailing comma used to construct the singleton
404 tuple.) In our example, the resulting argument tuple is (aninstance,
405 10).
407 <P>The intrinsic function apply() takes a function and an argument
408 tuple and calls the function for it. In our example, we are calling
409 apply(method1, (aninstance, 10)) which is equivalent to calling
410 method(aninstance, 10).
412 <P>From here on, things should come together quite easily. The output
413 of the example code is something like this:
415 <PRE>
416 calling &lt;function method1 at ae8d8&gt; for &lt;Instance instance at 95ab0&gt; with (10,)
417 calling &lt;function method2 at ae900&gt; for &lt;Instance instance at 95ab0&gt; with ()
418 the answer is 10
419 </PRE>
421 <P>That was about the shortest meaningful example that I could come up
422 with. A real tracing metaclass (for example, <A
423 HREF="#Trace">Trace.py</A> discussed below) needs to be more
424 complicated in two dimensions.
426 <P>First, it needs to support more advanced Python features such as
427 class variables, inheritance, __init__ methods, and keyword arguments.
429 <P>Second, it needs to provide a more flexible way to handle the
430 actual tracing information; perhaps it should be possible to write
431 your own tracing function that gets called, perhaps it should be
432 possible to enable and disable tracing on a per-class or per-instance
433 basis, and perhaps a filter so that only interesting calls are traced;
434 it should also be able to trace the return value of the call (or the
435 exception it raised if an error occurs). Even the Trace.py example
436 doesn't support all these features yet.
440 <HR>
442 <H1>Real-life Examples</H1>
444 <P>Have a look at some very preliminary examples that I coded up to
445 teach myself how to write metaclasses:
447 <DL>
449 <DT><A HREF="Enum.py">Enum.py</A>
451 <DD>This (ab)uses the class syntax as an elegant way to define
452 enumerated types. The resulting classes are never instantiated --
453 rather, their class attributes are the enumerated values. For
454 example:
456 <PRE>
457 class Color(Enum):
458 red = 1
459 green = 2
460 blue = 3
461 print Color.red
462 </PRE>
464 will print the string ``Color.red'', while ``Color.red==1'' is true,
465 and ``Color.red + 1'' raise a TypeError exception.
469 <DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
471 <DD>The resulting classes work much like standard
472 classes, but by setting a special class or instance attribute
473 __trace_output__ to point to a file, all calls to the class's methods
474 are traced. It was a bit of a struggle to get this right. This
475 should probably redone using the generic metaclass below.
479 <DT><A HREF="Meta.py">Meta.py</A>
481 <DD>A generic metaclass. This is an attempt at finding out how much
482 standard class behavior can be mimicked by a metaclass. The
483 preliminary answer appears to be that everything's fine as long as the
484 class (or its clients) don't look at the instance's __class__
485 attribute, nor at the class's __dict__ attribute. The use of
486 __getattr__ internally makes the classic implementation of __getattr__
487 hooks tough; we provide a similar hook _getattr_ instead.
488 (__setattr__ and __delattr__ are not affected.)
489 (XXX Hm. Could detect presence of __getattr__ and rename it.)
493 <DT><A HREF="Eiffel.py">Eiffel.py</A>
495 <DD>Uses the above generic metaclass to implement Eiffel style
496 pre-conditions and post-conditions.
500 <DT><A HREF="Synch.py">Synch.py</A>
502 <DD>Uses the above generic metaclass to implement synchronized
503 methods.
507 <DT><A HREF="Simple.py">Simple.py</A>
509 <DD>The example module used above.
513 </DL>
515 <P>A pattern seems to be emerging: almost all these uses of
516 metaclasses (except for Enum, which is probably more cute than useful)
517 mostly work by placing wrappers around method calls. An obvious
518 problem with that is that it's not easy to combine the features of
519 different metaclasses, while this would actually be quite useful: for
520 example, I wouldn't mind getting a trace from the test run of the
521 Synch module, and it would be interesting to add preconditions to it
522 as well. This needs more research. Perhaps a metaclass could be
523 provided that allows stackable wrappers...
527 <HR>
529 <H2>Things You Could Do With Metaclasses</H2>
531 <P>There are lots of things you could do with metaclasses. Most of
532 these can also be done with creative use of __getattr__, but
533 metaclasses make it easier to modify the attribute lookup behavior of
534 classes. Here's a partial list.
538 <UL>
540 <LI>Enforce different inheritance semantics, e.g. automatically call
541 base class methods when a derived class overrides<P>
543 <LI>Implement class methods (e.g. if the first argument is not named
544 'self')<P>
546 <LI>Implement that each instance is initialized with <b>copies</b> of
547 all class variables<P>
549 <LI>Implement a different way to store instance variables (e.g. in a
550 list kept outside the instance but indexed by the instance's id())<P>
552 <LI>Automatically wrap or trap all or certain methods
554 <UL>
556 <LI>for tracing
558 <LI>for precondition and postcondition checking
560 <LI>for synchronized methods
562 <LI>for automatic value caching
564 </UL>
567 <LI>When an attribute is a parameterless function, call it on
568 reference (to mimic it being an instance variable); same on assignment<P>
570 <LI>Instrumentation: see how many times various attributes are used<P>
572 <LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
573 them when they are being used recursively)<P>
575 <LI>Abuse class syntax for other things<P>
577 <LI>Experiment with automatic type checking<P>
579 <LI>Delegation (or acquisition)<P>
581 <LI>Dynamic inheritance patterns<P>
583 <LI>Automatic caching of methods<P>
585 </UL>
589 <HR>
591 <H4>Credits</H4>
593 <P>Many thanks to David Ascher and Donald Beaudry for their comments
594 on earlier draft of this paper. Also thanks to Matt Conway and Tommy
595 Burnette for putting a seed for the idea of metaclasses in my
596 mind, nearly three years ago, even though at the time my response was
597 ``you can do that with __getattr__ hooks...'' :-)
601 <HR>
603 </BODY>
605 </HTML>