6 % Count up the patches and bugs
8 \title{What's New in Python
2.5}
10 \author{A.M. Kuchling
}
11 \authoraddress{\email{amk@amk.ca
}}
17 This article explains the new features in Python
2.5. The final
18 release of Python
2.5 is scheduled for August
2006;
19 \pep{356} describes the planned release schedule.
21 The changes in Python
2.5 are an interesting mix of language and
22 library improvements. The library enhancements will be more important
23 to Python's user community, I think, because several widely-useful
24 packages were added. New modules include ElementTree for XML
25 processing (section~
\ref{module-etree
}), the SQLite database module
26 (section~
\ref{module-sqlite
}), and the
\module{ctypes
} module for
27 calling C functions (section~
\ref{module-ctypes
}).
29 The language changes are of middling significance. Some pleasant new
30 features were added, but most of them aren't features that you'll use
31 every day. Conditional expressions were finally added to the language
32 using a novel syntax; see section~
\ref{pep-
308}. The new
33 '
\keyword{with
}' statement will make writing cleanup code easier
34 (section~
\ref{pep-
343}). Values can now be passed into generators
35 (section~
\ref{pep-
342}). Imports are now visible as either absolute
36 or relative (section~
\ref{pep-
328}). Some corner cases of exception
37 handling are handled better (section~
\ref{pep-
341}). All these
38 improvements are worthwhile, but they're improvements to one specific
39 language feature or another; none of them are broad modifications to
42 This article doesn't try to be a complete specification of the new
43 features; instead changes are briefly introduced using helpful
44 examples. For full details, you should always refer to the
45 documentation for Python
2.5.
46 % XXX add hyperlink when the documentation becomes available online.
47 If you want to understand the complete implementation and design
48 rationale, refer to the PEP for a particular new feature.
50 Comments, suggestions, and error reports for this
document are
51 welcome; please e-mail them to the author or open a bug in the Python
54 %======================================================================
55 \section{PEP
308: Conditional Expressions
\label{pep-
308}}
57 For a long time, people have been requesting a way to write
58 conditional expressions, which are expressions that return value A or
59 value B depending on whether a Boolean value is true or false. A
60 conditional expression lets you write a single assignment statement
61 that has the same effect as the following:
70 There have been endless tedious discussions of syntax on both
71 python-dev and comp.lang.python. A vote was even held that found the
72 majority of voters wanted conditional expressions in some form,
73 but there was no syntax that was preferred by a clear majority.
74 Candidates included C's
\code{cond ? true_v : false_v
},
75 \code{if cond then true_v else false_v
}, and
16 other variations.
77 GvR eventually chose a surprising syntax:
80 x = true_value if condition else false_value
83 Evaluation is still lazy as in existing Boolean expressions, so the
84 order of evaluation jumps around a bit. The
\var{condition
}
85 expression in the middle is evaluated first, and the
\var{true_value
}
86 expression is evaluated only if the condition was true. Similarly,
87 the
\var{false_value
} expression is only evaluated when the condition
90 This syntax may seem strange and backwards; why does the condition go
91 in the
\emph{middle
} of the expression, and not in the front as in C's
92 \code{c ? x : y
}? The decision was checked by applying the new syntax
93 to the modules in the standard library and seeing how the resulting
94 code read. In many cases where a conditional expression is used, one
95 value seems to be the 'common case' and one value is an 'exceptional
96 case', used only on rarer occasions when the condition isn't met. The
97 conditional syntax makes this pattern a bit more obvious:
100 contents = ((doc + '
\n') if doc else '')
103 I read the above statement as meaning ``here
\var{contents
} is
104 usually assigned a value of
\code{doc+'
\e n'
}; sometimes
105 \var{doc
} is empty, in which special case an empty string is returned.''
106 I doubt I will use conditional expressions very often where there
107 isn't a clear common and uncommon case.
109 There was some discussion of whether the language should require
110 surrounding conditional expressions with parentheses. The decision
111 was made to
\emph{not
} require parentheses in the Python language's
112 grammar, but as a matter of style I think you should always use them.
113 Consider these two statements:
116 # First version -- no parens
117 level =
1 if logging else
0
119 # Second version -- with parens
120 level = (
1 if logging else
0)
123 In the first version, I think a reader's eye might group the statement
124 into 'level =
1', 'if logging', 'else
0', and think that the condition
125 decides whether the assignment to
\var{level
} is performed. The
126 second version reads better, in my opinion, because it makes it clear
127 that the assignment is always performed and the choice is being made
130 Another reason for including the brackets: a few odd combinations of
131 list comprehensions and lambdas could look like incorrect conditional
132 expressions. See
\pep{308} for some examples. If you put parentheses
133 around your conditional expressions, you won't run into this case.
138 \seepep{308}{Conditional Expressions
}{PEP written by
139 Guido van~Rossum and Raymond D. Hettinger; implemented by Thomas
145 %======================================================================
146 \section{PEP
309: Partial Function Application
\label{pep-
309}}
148 The
\module{functools
} module is intended to contain tools for
149 functional-style programming.
151 One useful tool in this module is the
\function{partial()
} function.
152 For programs written in a functional style, you'll sometimes want to
153 construct variants of existing functions that have some of the
154 parameters filled in. Consider a Python function
\code{f(a, b, c)
};
155 you could create a new function
\code{g(b, c)
} that was equivalent to
156 \code{f(
1, b, c)
}. This is called ``partial function application''.
158 \function{partial
} takes the arguments
159 \code{(
\var{function
},
\var{arg1
},
\var{arg2
}, ...
160 \var{kwarg1
}=
\var{value1
},
\var{kwarg2
}=
\var{value2
})
}. The resulting
161 object is callable, so you can just call it to invoke
\var{function
}
162 with the filled-in arguments.
164 Here's a small but realistic example:
169 def log (message, subsystem):
170 "Write the contents of 'message' to the specified subsystem."
171 print '
%s: %s' % (subsystem, message)
174 server_log = functools.partial(log, subsystem='server')
175 server_log('Unable to open socket')
178 Here's another example, from a program that uses PyGTK. Here a
179 context-sensitive pop-up menu is being constructed dynamically. The
180 callback provided for the menu option is a partially applied version
181 of the
\method{open_item()
} method, where the first argument has been
187 def open_item(self, path):
190 open_func = functools.partial(self.open_item, item_path)
191 popup_menu.append( ("Open", open_func,
1) )
195 Another function in the
\module{functools
} module is the
196 \function{update_wrapper(
\var{wrapper
},
\var{wrapped
})
} function that
197 helps you write well-behaved decorators.
\function{update_wrapper()
}
198 copies the name, module, and docstring attribute to a wrapper function
199 so that tracebacks inside the wrapped function are easier to
200 understand. For example, you might write:
204 def wrapper
(*args, **kwds):
205 print 'Calling decorated function'
206 return f(*args, **kwds)
207 functools.update_wrapper(wrapper, f)
211 \function{wraps()} is a decorator that can be used inside your own
212 decorators to copy the wrapped function's information. An alternate
213 version of the previous example would be:
218 def wrapper(*args, **kwds):
219 print 'Calling decorated function'
220 return f(*args, **kwds)
226 \seepep{309}{Partial Function Application}{PEP proposed and written by
227 Peter Harris; implemented by Hye-Shik Chang and Nick Coghlan, with
228 adaptations by Raymond Hettinger.}
233 %======================================================================
234 \section{PEP 314: Metadata for Python Software Packages v1.1\label{pep-314}}
236 Some simple dependency support was added to Distutils. The
237 \function{setup()} function now has \code{requires}, \code{provides},
238 and \code{obsoletes} keyword parameters. When you build a source
239 distribution using the \code{sdist} command, the dependency
240 information will be recorded in the \file{PKG-INFO} file.
242 Another new keyword parameter is \code{download_url}, which should be
243 set to a URL for the package's source code. This means it's now
244 possible to look up an entry in the package index, determine the
245 dependencies for a package, and download the required packages.
249 setup(name='PyPackage',
251 requires=['numarray', 'zlib (>=1.1.4)'],
252 obsoletes=['OldPackage']
253 download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
258 Another new enhancement to the Python package index at
259 \url{http://cheeseshop.python.org} is storing source and binary
260 archives for a package. The new \command{upload} Distutils command
261 will upload a package to the repository.
263 Before a package can be uploaded, you must be able to build a
264 distribution using the \command{sdist} Distutils command. Once that
265 works, you can run \code{python setup.py upload} to add your package
266 to the PyPI archive. Optionally you can GPG-sign the package by
267 supplying the \longprogramopt{sign} and
268 \longprogramopt{identity} options.
270 Package uploading was implemented by Martin von~L\"owis and Richard Jones.
274 \seepep{314}{Metadata for Python Software Packages v1.1}{PEP proposed
275 and written by A.M. Kuchling, Richard Jones, and Fred Drake;
276 implemented by Richard Jones and Fred Drake.}
281 %======================================================================
282 \section{PEP 328: Absolute and Relative Imports\label{pep-328}}
284 The simpler part of PEP 328 was implemented in Python 2.4: parentheses
285 could now be used to enclose the names imported from a module using
286 the \code{from ... import ...} statement, making it easier to import
287 many different names.
289 The more complicated part has been implemented in Python 2.5:
290 importing a module can be specified to use absolute or
291 package-relative imports. The plan is to move toward making absolute
292 imports the default in future versions of Python.
294 Let's say you have a package directory like this:
302 This defines a package named \module{pkg} containing the
303 \module{pkg.main} and \module{pkg.string} submodules.
305 Consider the code in the \file{main.py} module. What happens if it
306 executes the statement \code{import string}? In Python 2.4 and
307 earlier, it will first look in the package's directory to perform a
308 relative import, finds \file{pkg/string.py}, imports the contents of
309 that file as the \module{pkg.string} module, and that module is bound
310 to the name \samp{string} in the \module{pkg.main} module's namespace.
312 That's fine if \module{pkg.string} was what you wanted. But what if
313 you wanted Python's standard \module{string} module? There's no clean
314 way to ignore \module{pkg.string} and look for the standard module;
315 generally you had to look at the contents of \code{sys.modules}, which
317 Holger Krekel's \module{py.std} package provides a tidier way to perform
318 imports from the standard library, \code{import py ; py.std.string.join()},
319 but that package isn't available on all Python installations.
321 Reading code which relies on relative imports is also less clear,
322 because a reader may be confused about which module, \module{string}
323 or \module{pkg.string}, is intended to be used. Python users soon
324 learned not to duplicate the names of standard library modules in the
325 names of their packages' submodules, but you can't protect against
326 having your submodule's name being used for a new module added in a
327 future version of Python.
329 In Python 2.5, you can switch \keyword{import}'s behaviour to
330 absolute imports using a \code{from __future__ import absolute_import}
331 directive. This absolute-import behaviour will become the default in
332 a future version (probably Python 2.7). Once absolute imports
333 are the default, \code{import string} will
334 always find the standard library's version.
335 It's suggested that users should begin using absolute imports as much
336 as possible, so it's preferable to begin writing \code{from pkg import
337 string} in your code.
339 Relative imports are still possible by adding a leading period
340 to the module name when using the \code{from ... import} form:
343 # Import names from pkg.string
344 from .string import name1, name2
349 This imports the \module{string} module relative to the current
350 package, so in \module{pkg.main} this will import \var{name1} and
351 \var{name2} from \module{pkg.string}. Additional leading periods
352 perform the relative import starting from the parent of the current
353 package. For example, code in the \module{A.B.C} module can do:
356 from . import D # Imports A.B.D
357 from .. import E # Imports A.E
358 from ..F import G # Imports A.F.G
361 Leading periods cannot be used with the \code{import \var{modname}}
362 form of the import statement, only the \code{from ... import} form.
366 \seepep{328}{Imports: Multi-Line and Absolute/Relative}
367 {PEP written by Aahz; implemented by Thomas Wouters.}
369 \seeurl{http://codespeak.net/py/current/doc/index.html}
370 {The py library by Holger Krekel, which contains the \module{py.std} package.}
375 %======================================================================
376 \section{PEP 338: Executing Modules as Scripts\label{pep-338}}
378 The \programopt{-m} switch added in Python 2.4 to execute a module as
379 a script gained a few more abilities. Instead of being implemented in
380 C code inside the Python interpreter, the switch now uses an
381 implementation in a new module, \module{runpy}.
383 The \module{runpy} module implements a more sophisticated import
384 mechanism so that it's now possible to run modules in a package such
385 as \module{pychecker.checker}. The module also supports alternative
386 import mechanisms such as the \module{zipimport} module. This means
387 you can add a .zip archive's path to \code{sys.path} and then use the
388 \programopt{-m} switch to execute code from the archive.
393 \seepep{338}{Executing modules as scripts}{PEP written and
394 implemented by Nick Coghlan.}
399 %======================================================================
400 \section{PEP 341: Unified try/except/finally\label{pep-341}}
402 Until Python 2.5, the \keyword{try} statement came in two
403 flavours. You could use a \keyword{finally} block to ensure that code
404 is always executed, or one or more \keyword{except} blocks to catch
405 specific exceptions. You couldn't combine both \keyword{except} blocks and a
406 \keyword{finally} block, because generating the right bytecode for the
407 combined version was complicated and it wasn't clear what the
408 semantics of the combined should be.
410 GvR spent some time working with Java, which does support the
411 equivalent of combining \keyword{except} blocks and a
412 \keyword{finally} block, and this clarified what the statement should
413 mean. In Python 2.5, you can now write:
428 The code in \var{block-1} is executed. If the code raises an
429 exception, the various \keyword{except} blocks are tested: if the
430 exception is of class \class{Exception1}, \var{handler-1} is executed;
431 otherwise if it's of class \class{Exception2}, \var{handler-2} is
432 executed, and so forth. If no exception is raised, the
433 \var{else-block} is executed.
435 No matter what happened previously, the \var{final-block} is executed
436 once the code block is complete and any raised exceptions handled.
437 Even if there's an error in an exception handler or the
438 \var{else-block} and a new exception is raised, the
439 code in the \var{final-block} is still run.
443 \seepep{341}{Unifying try-except and try-finally}{PEP written by Georg Brandl;
444 implementation by Thomas Lee.}
449 %======================================================================
450 \section{PEP 342: New Generator Features\label{pep-342}}
452 Python 2.5 adds a simple way to pass values \emph{into} a generator.
453 As introduced in Python 2.3, generators only produce output; once a
454 generator's code was invoked to create an iterator, there was no way to
455 pass any new information into the function when its execution is
456 resumed. Sometimes the ability to pass in some information would be
457 useful. Hackish solutions to this include making the generator's code
458 look at a global variable and then changing the global variable's
459 value, or passing in some mutable object that callers then modify.
461 To refresh your memory of basic generators, here's a simple example:
464 def counter (maximum):
471 When you call \code{counter(10)}, the result is an iterator that
472 returns the values from 0 up to 9. On encountering the
473 \keyword{yield} statement, the iterator returns the provided value and
474 suspends the function's execution, preserving the local variables.
475 Execution resumes on the following call to the iterator's
476 \method{next()} method, picking up after the \keyword{yield} statement.
478 In Python 2.3, \keyword{yield} was a statement; it didn't return any
479 value. In 2.5, \keyword{yield} is now an expression, returning a
480 value that can be assigned to a variable or otherwise operated on:
486 I recommend that you always put parentheses around a \keyword{yield}
487 expression when you're doing something with the returned value, as in
488 the above example. The parentheses aren't always necessary, but it's
489 easier to always add them instead of having to remember when they're
492 (\pep{342} explains the exact rules, which are that a
493 \keyword{yield}-expression must always be parenthesized except when it
494 occurs at the top-level expression on the right-hand side of an
495 assignment. This means you can write \code{val = yield i} but have to
496 use parentheses when there's an operation, as in \code{val = (yield i)
499 Values are sent into a generator by calling its
500 \method{send(\var{value})} method. The generator's code is then
501 resumed and the \keyword{yield} expression returns the specified
502 \var{value}. If the regular \method{next()} method is called, the
503 \keyword{yield} returns \constant{None}.
505 Here's the previous example, modified to allow changing the value of
506 the internal counter.
509 def counter (maximum):
513 # If value provided, change counter
520 And here's an example of changing the counter:
533 Traceback (most recent call last):
534 File ``t.py'', line 15, in ?
539 Because \keyword{yield} will often be returning \constant{None}, you
540 should always check for this case. Don't just use its value in
541 expressions unless you're sure that the \method{send()} method
542 will be the only method used resume your generator function.
544 In addition to \method{send()}, there are two other new methods on
549 \item \method{throw(\var{type}, \var{value}=None,
550 \var{traceback}=None)} is used to raise an exception inside the
551 generator; the exception is raised by the \keyword{yield} expression
552 where the generator's execution is paused.
554 \item \method{close()} raises a new \exception{GeneratorExit}
555 exception inside the generator to terminate the iteration.
557 exception, the generator's code must either raise
558 \exception{GeneratorExit} or \exception{StopIteration}; catching the
559 exception and doing anything else is illegal and will trigger
560 a \exception{RuntimeError}. \method{close()} will also be called by
561 Python's garbage collector when the generator is garbage-collected.
563 If you need to run cleanup code when a \exception{GeneratorExit} occurs,
564 I suggest using a \code{try: ... finally:} suite instead of
565 catching \exception{GeneratorExit}.
569 The cumulative effect of these changes is to turn generators from
570 one-way producers of information into both producers and consumers.
572 Generators also become \emph{coroutines}, a more generalized form of
573 subroutines. Subroutines are entered at one point and exited at
574 another point (the top of the function, and a \keyword{return}
575 statement), but coroutines can be entered, exited, and resumed at
576 many different points (the \keyword{yield} statements). We'll have to
577 figure out patterns for using coroutines effectively in Python.
579 The addition of the \method{close()} method has one side effect that
580 isn't obvious. \method{close()} is called when a generator is
581 garbage-collected, so this means the generator's code gets one last
582 chance to run before the generator is destroyed. This last chance
583 means that \code{try...finally} statements in generators can now be
584 guaranteed to work; the \keyword{finally} clause will now always get a
585 chance to run. The syntactic restriction that you couldn't mix
586 \keyword{yield} statements with a \code{try...finally} suite has
587 therefore been removed. This seems like a minor bit of language
588 trivia, but using generators and \code{try...finally} is actually
589 necessary in order to implement the \keyword{with} statement
590 described by PEP 343. I'll look at this new statement in the following
593 Another even more esoteric effect of this change: previously, the
594 \member{gi_frame} attribute of a generator was always a frame object.
595 It's now possible for \member{gi_frame} to be \code{None}
596 once the generator has been exhausted.
600 \seepep{342}{Coroutines via Enhanced Generators}{PEP written by
601 Guido van~Rossum and Phillip J. Eby;
602 implemented by Phillip J. Eby. Includes examples of
603 some fancier uses of generators as coroutines.}
605 \seeurl{http://en.wikipedia.org/wiki/Coroutine}{The Wikipedia entry for
608 \seeurl{http://www.sidhe.org/\~{}dan/blog/archives/000178.html}{An
609 explanation of coroutines from a Perl point of view, written by Dan
615 %======================================================================
616 \section{PEP 343: The 'with' statement\label{pep-343}}
618 The '\keyword{with}' statement clarifies code that previously would
619 use \code{try...finally} blocks to ensure that clean-up code is
620 executed. In this section, I'll discuss the statement as it will
621 commonly be used. In the next section, I'll examine the
622 implementation details and show how to write objects for use with this
625 The '\keyword{with}' statement is a new control-flow structure whose
629 with expression [as variable]:
633 The expression is evaluated, and it should result in an object that
634 supports the context management protocol. This object may return a
635 value that can optionally be bound to the name \var{variable}. (Note
636 carefully that \var{variable} is \emph{not} assigned the result of
637 \var{expression}.) The object can then run set-up code
638 before \var{with-block} is executed and some clean-up code
639 is executed after the block is done, even if the block raised an exception.
641 To enable the statement in Python 2.5, you need
642 to add the following directive to your module:
645 from __future__ import with_statement
648 The statement will always be enabled in Python 2.6.
650 Some standard Python objects now support the context management
651 protocol and can be used with the '\keyword{with}' statement. File
652 objects are one example:
655 with open('/etc/passwd', 'r') as f:
658 ... more processing code ...
661 After this statement has executed, the file object in \var{f} will
662 have been automatically closed, even if the 'for' loop
663 raised an exception part-way through the block.
665 The \module{threading} module's locks and condition variables
666 also support the '\keyword{with}' statement:
669 lock = threading.Lock()
671 # Critical section of code
675 The lock is acquired before the block is executed and always released once
676 the block is complete.
678 The \module{decimal} module's contexts, which encapsulate the desired
679 precision and rounding characteristics for computations, provide a
680 \method{context_manager()} method for getting a context manager:
685 # Displays with default precision of 28 digits
686 v1 = decimal.Decimal('578')
689 ctx = decimal.Context(prec=16)
690 with ctx.context_manager():
691 # All code in this block uses a precision of 16 digits.
692 # The original context is restored on exiting the block.
696 \subsection{Writing Context Managers\label{context-managers}}
698 Under the hood, the '\keyword{with}' statement is fairly complicated.
699 Most people will only use '\keyword{with}' in company with existing
700 objects and don't need to know these details, so you can skip the rest
701 of this section if you like. Authors of new objects will need to
702 understand the details of the underlying implementation and should
705 A high-level explanation of the context management protocol is:
709 \item The expression is evaluated and should result in an object
710 called a ``context manager''. The context manager must have
711 \method{__enter__()} and \method{__exit__()} methods.
713 \item The context manager's \method{__enter__()} method is called. The value
714 returned is assigned to \var{VAR}. If no \code{'as \var{VAR}'} clause
715 is present, the value is simply discarded.
717 \item The code in \var{BLOCK} is executed.
719 \item If \var{BLOCK} raises an exception, the
720 \method{__exit__(\var{type}, \var{value}, \var{traceback})} is called
721 with the exception details, the same values returned by
722 \function{sys.exc_info()}. The method's return value controls whether
723 the exception is re-raised: any false value re-raises the exception,
724 and \code{True} will result in suppressing it. You'll only rarely
725 want to suppress the exception, because if you do
726 the author of the code containing the
727 '\keyword{with}' statement will never realize anything went wrong.
729 \item If \var{BLOCK} didn't raise an exception,
730 the \method{__exit__()} method is still called,
731 but \var{type}, \var{value}, and \var{traceback} are all \code{None}.
735 Let's think through an example. I won't present detailed code but
736 will only sketch the methods necessary for a database that supports
739 (For people unfamiliar with database terminology: a set of changes to
740 the database are grouped into a transaction. Transactions can be
741 either committed, meaning that all the changes are written into the
742 database, or rolled back, meaning that the changes are all discarded
743 and the database is unchanged. See any database textbook for more
745 % XXX find a shorter reference?
747 Let's assume there's an object representing a database connection.
748 Our goal will be to let the user write code like this:
751 db_connection = DatabaseConnection()
752 with db_connection as cursor:
753 cursor.execute('insert into ...')
754 cursor.execute('delete from ...')
755 # ... more operations ...
758 The transaction should be committed if the code in the block
759 runs flawlessly or rolled back if there's an exception.
760 Here's the basic interface
761 for \class{DatabaseConnection} that I'll assume:
764 class DatabaseConnection:
767 "Returns a cursor object and starts a new transaction"
769 "Commits current transaction"
771 "Rolls back current transaction"
774 The \method {__enter__()} method is pretty easy, having only to start
775 a new transaction. For this application the resulting cursor object
776 would be a useful result, so the method will return it. The user can
777 then add \code{as cursor} to their '\keyword{with}' statement to bind
778 the cursor to a variable name.
781 class DatabaseConnection:
783 def __enter__ (self):
784 # Code to start a new transaction
785 cursor = self.cursor()
789 The \method{__exit__()} method is the most complicated because it's
790 where most of the work has to be done. The method has to check if an
791 exception occurred. If there was no exception, the transaction is
792 committed. The transaction is rolled back if there was an exception.
794 In the code below, execution will just fall off the end of the
795 function, returning the default value of \code{None}. \code{None} is
796 false, so the exception will be re-raised automatically. If you
797 wished, you could be more explicit and add a \keyword{return}
798 statement at the marked location.
801 class DatabaseConnection:
803 def __exit__ (self, type, value, tb):
805 # No exception, so commit
808 # Exception occurred, so rollback.
814 \subsection{The contextlib module\label{module-contextlib}}
816 The new \module{contextlib} module provides some functions and a
817 decorator that are useful for writing objects for use with the
818 '\keyword{with}' statement.
820 The decorator is called \function{contextmanager}, and lets you write
821 a single generator function instead of defining a new class. The generator
822 should yield exactly one value. The code up to the \keyword{yield}
823 will be executed as the \method{__enter__()} method, and the value
824 yielded will be the method's return value that will get bound to the
825 variable in the '\keyword{with}' statement's \keyword{as} clause, if
826 any. The code after the \keyword{yield} will be executed in the
827 \method{__exit__()} method. Any exception raised in the block will be
828 raised by the \keyword{yield} statement.
830 Our database example from the previous section could be written
831 using this decorator as:
834 from contextlib import contextmanager
837 def db_transaction (connection):
838 cursor = connection.cursor()
842 connection.rollback()
847 db = DatabaseConnection()
848 with db_transaction(db) as cursor:
852 The \module{contextlib} module also has a \function{nested(\var{mgr1},
853 \var{mgr2}, ...)} function that combines a number of context managers so you
854 don't need to write nested '\keyword{with}' statements. In this
855 example, the single '\keyword{with}' statement both starts a database
856 transaction and acquires a thread lock:
859 lock = threading.Lock()
860 with nested (db_transaction(db), lock) as (cursor, locked):
864 Finally, the \function{closing(\var{object})} function
865 returns \var{object} so that it can be bound to a variable,
866 and calls \code{\var{object}.close()} at the end of the block.
870 from contextlib import closing
872 with closing(urllib.urlopen('http://www.yahoo.com')) as f:
874 sys.stdout.write(line)
879 \seepep{343}{The ``with'' statement}{PEP written by Guido van~Rossum
880 and Nick Coghlan; implemented by Mike Bland, Guido van~Rossum, and
881 Neal Norwitz. The PEP shows the code generated for a '\keyword{with}'
882 statement, which can be helpful in learning how the statement works.}
884 \seeurl{../lib/module-contextlib.html}{The documentation
885 for the \module{contextlib} module.}
890 %======================================================================
891 \section{PEP 352: Exceptions as New-Style Classes\label{pep-352}}
893 Exception classes can now be new-style classes, not just classic
894 classes, and the built-in \exception{Exception} class and all the
895 standard built-in exceptions (\exception{NameError},
896 \exception{ValueError}, etc.) are now new-style classes.
898 The inheritance hierarchy for exceptions has been rearranged a bit.
899 In 2.5, the inheritance relationships are:
902 BaseException # New in Python 2.5
906 |- (all other current built-in exceptions)
909 This rearrangement was done because people often want to catch all
910 exceptions that indicate program errors. \exception{KeyboardInterrupt} and
911 \exception{SystemExit} aren't errors, though, and usually represent an explicit
912 action such as the user hitting Control-C or code calling
913 \function{sys.exit()}. A bare \code{except:} will catch all exceptions,
914 so you commonly need to list \exception{KeyboardInterrupt} and
915 \exception{SystemExit} in order to re-raise them. The usual pattern is:
920 except (KeyboardInterrupt, SystemExit):
924 # Continue running program...
927 In Python 2.5, you can now write \code{except Exception} to achieve
928 the same result, catching all the exceptions that usually indicate errors
929 but leaving \exception{KeyboardInterrupt} and
930 \exception{SystemExit} alone. As in previous versions,
931 a bare \code{except:} still catches all exceptions.
933 The goal for Python 3.0 is to require any class raised as an exception
934 to derive from \exception{BaseException} or some descendant of
935 \exception{BaseException}, and future releases in the
936 Python 2.x series may begin to enforce this constraint. Therefore, I
937 suggest you begin making all your exception classes derive from
938 \exception{Exception} now. It's been suggested that the bare
939 \code{except:} form should be removed in Python 3.0, but Guido van~Rossum
940 hasn't decided whether to do this or not.
942 Raising of strings as exceptions, as in the statement \code{raise
943 "Error occurred"}, is deprecated in Python 2.5 and will trigger a
944 warning. The aim is to be able to remove the string-exception feature
950 \seepep{352}{Required Superclass for Exceptions}{PEP written by
951 Brett Cannon and Guido van~Rossum; implemented by Brett Cannon.}
956 %======================================================================
957 \section{PEP 353: Using ssize_t as the index type\label{pep-353}}
959 A wide-ranging change to Python's C API, using a new
960 \ctype{Py_ssize_t} type definition instead of \ctype{int},
961 will permit the interpreter to handle more data on 64-bit platforms.
962 This change doesn't affect Python's capacity on 32-bit platforms.
964 Various pieces of the Python interpreter used C's \ctype{int} type to
965 store sizes or counts; for example, the number of items in a list or
966 tuple were stored in an \ctype{int}. The C compilers for most 64-bit
967 platforms still define \ctype{int} as a 32-bit type, so that meant
968 that lists could only hold up to \code{2**31 - 1} = 2147483647 items.
969 (There are actually a few different programming models that 64-bit C
970 compilers can use -- see
971 \url{http://www.unix.org/version2/whatsnew/lp64_wp.html} for a
972 discussion -- but the most commonly available model leaves \ctype{int}
975 A limit of 2147483647 items doesn't really matter on a 32-bit platform
976 because you'll run out of memory before hitting the length limit.
977 Each list item requires space for a pointer, which is 4 bytes, plus
978 space for a \ctype{PyObject} representing the item. 2147483647*4 is
979 already more bytes than a 32-bit address space can contain.
981 It's possible to address that much memory on a 64-bit platform,
982 however. The pointers for a list that size would only require 16~GiB
983 of space, so it's not unreasonable that Python programmers might
984 construct lists that large. Therefore, the Python interpreter had to
985 be changed to use some type other than \ctype{int}, and this will be a
986 64-bit type on 64-bit platforms. The change will cause
987 incompatibilities on 64-bit machines, so it was deemed worth making
988 the transition now, while the number of 64-bit users is still
989 relatively small. (In 5 or 10 years, we may \emph{all} be on 64-bit
990 machines, and the transition would be more painful then.)
992 This change most strongly affects authors of C extension modules.
993 Python strings and container types such as lists and tuples
994 now use \ctype{Py_ssize_t} to store their size.
995 Functions such as \cfunction{PyList_Size()}
996 now return \ctype{Py_ssize_t}. Code in extension modules
997 may therefore need to have some variables changed to
1000 The \cfunction{PyArg_ParseTuple()} and \cfunction{Py_BuildValue()} functions
1001 have a new conversion code, \samp{n}, for \ctype{Py_ssize_t}.
1002 \cfunction{PyArg_ParseTuple()}'s \samp{s\#} and \samp{t\#} still output
1003 \ctype{int} by default, but you can define the macro
1004 \csimplemacro{PY_SSIZE_T_CLEAN} before including \file{Python.h}
1005 to make them return \ctype{Py_ssize_t}.
1007 \pep{353} has a section on conversion guidelines that
1008 extension authors should read to learn about supporting 64-bit
1013 \seepep{353}{Using ssize_t as the index type}{PEP written and implemented by Martin von~L\"owis.}
1018 %======================================================================
1019 \section{PEP 357: The '__index__' method\label{pep-357}}
1021 The NumPy developers had a problem that could only be solved by adding
1022 a new special method, \method{__index__}. When using slice notation,
1023 as in \code{[\var{start}:\var{stop}:\var{step}]}, the values of the
1024 \var{start}, \var{stop}, and \var{step} indexes must all be either
1025 integers or long integers. NumPy defines a variety of specialized
1026 integer types corresponding to unsigned and signed integers of 8, 16,
1027 32, and 64 bits, but there was no way to signal that these types could
1028 be used as slice indexes.
1030 Slicing can't just use the existing \method{__int__} method because
1031 that method is also used to implement coercion to integers. If
1032 slicing used \method{__int__}, floating-point numbers would also
1033 become legal slice indexes and that's clearly an undesirable
1036 Instead, a new special method called \method{__index__} was added. It
1037 takes no arguments and returns an integer giving the slice index to
1042 def __index__ (self):
1046 The return value must be either a Python integer or long integer.
1047 The interpreter will check that the type returned is correct, and
1048 raises a \exception{TypeError} if this requirement isn't met.
1050 A corresponding \member{nb_index} slot was added to the C-level
1051 \ctype{PyNumberMethods} structure to let C extensions implement this
1052 protocol. \cfunction{PyNumber_Index(\var{obj})} can be used in
1053 extension code to call the \method{__index__} function and retrieve
1058 \seepep{357}{Allowing Any Object to be Used for Slicing}{PEP written
1059 and implemented by Travis Oliphant.}
1064 %======================================================================
1065 \section{Other Language Changes\label{other-lang}}
1067 Here are all of the changes that Python 2.5 makes to the core Python
1072 \item The \class{dict} type has a new hook for letting subclasses
1073 provide a default value when a key isn't contained in the dictionary.
1074 When a key isn't found, the dictionary's
1075 \method{__missing__(\var{key})}
1076 method will be called. This hook is used to implement
1077 the new \class{defaultdict} class in the \module{collections}
1078 module. The following example defines a dictionary
1079 that returns zero for any missing key:
1082 class zerodict (dict):
1083 def __missing__ (self, key):
1086 d = zerodict({1:1, 2:2})
1087 print d[1], d[2] # Prints 1, 2
1088 print d[3], d[4] # Prints 0, 0
1091 \item Both 8-bit and Unicode strings have new \method{partition(sep)}
1092 and \method{rpartition(sep)} methods that simplify a common use case.
1093 The \method{find(S)} method is often used to get an index which is
1094 then used to slice the string and obtain the pieces that are before
1095 and after the separator.
1097 \method{partition(sep)} condenses this
1098 pattern into a single method call that returns a 3-tuple containing
1099 the substring before the separator, the separator itself, and the
1100 substring after the separator. If the separator isn't found, the
1101 first element of the tuple is the entire string and the other two
1102 elements are empty. \method{rpartition(sep)} also returns a 3-tuple
1103 but starts searching from the end of the string; the \samp{r} stands
1109 >>> ('http://www.python.org').partition('://')
1110 ('http', '://', 'www.python.org')
1111 >>> (u'Subject: a quick question').partition(':')
1112 (u'Subject', u':', u' a quick question')
1113 >>> ('file:/usr/share/doc/index.html').partition('://')
1114 ('file:/usr/share/doc/index.html', '', '')
1115 >>> 'www.python.org'.rpartition('.')
1116 ('www.python', '.', 'org')
1119 (Implemented by Fredrik Lundh following a suggestion by Raymond Hettinger.)
1121 \item The \method{startswith()} and \method{endswith()} methods
1122 of string types now accept tuples of strings to check for.
1125 def is_image_file (filename):
1126 return filename.endswith(('.gif', '.jpg', '.tiff'))
1129 (Implemented by Georg Brandl following a suggestion by Tom Lynn.)
1132 \item The \function{min()} and \function{max()} built-in functions
1133 gained a \code{key} keyword parameter analogous to the \code{key}
1134 argument for \method{sort()}. This parameter supplies a function that
1135 takes a single argument and is called for every value in the list;
1136 \function{min()}/\function{max()} will return the element with the
1137 smallest/largest return value from this function.
1138 For example, to find the longest string in a list, you can do:
1141 L = ['medium', 'longest', 'short']
1143 print max(L, key=len)
1144 # Prints 'short', because lexicographically 'short' has the largest value
1148 (Contributed by Steven Bethard and Raymond Hettinger.)
1150 \item Two new built-in functions, \function{any()} and
1151 \function{all()}, evaluate whether an iterator contains any true or
1152 false values. \function{any()} returns \constant{True} if any value
1153 returned by the iterator is true; otherwise it will return
1154 \constant{False}. \function{all()} returns \constant{True} only if
1155 all of the values returned by the iterator evaluate as being true.
1156 (Suggested by GvR, and implemented by Raymond Hettinger.)
1158 \item ASCII is now the default encoding for modules. It's now
1159 a syntax error if a module contains string literals with 8-bit
1160 characters but doesn't have an encoding declaration. In Python 2.4
1161 this triggered a warning, not a syntax error. See \pep{263}
1162 for how to declare a module's encoding; for example, you might add
1163 a line like this near the top of the source file:
1166 # -*- coding: latin1 -*-
1169 \item One error that Python programmers sometimes make is forgetting
1170 to include an \file{__init__.py} module in a package directory.
1171 Debugging this mistake can be confusing, and usually requires running
1172 Python with the \programopt{-v} switch to log all the paths searched.
1173 In Python 2.5, a new \exception{ImportWarning} warning is raised when
1174 an import would have picked up a directory as a package but no
1175 \file{__init__.py} was found. (Implemented by Thomas Wouters.)
1177 To suppress these warnings, you can either supply
1178 \code{\programopt{-W}'ignore:Not importing directory'} when running the Python
1179 interpreter, or use the \module{warnings} module to suppress the
1183 warnings.filterwarnings('ignore', 'Not importing directory',
1187 \item The list of base classes in a class definition can now be empty.
1188 As an example, this is now legal:
1194 (Implemented by Brett Cannon.)
1199 %======================================================================
1200 \subsection{Interactive Interpreter Changes\label{interactive}}
1202 In the interactive interpreter, \code{quit} and \code{exit}
1203 have long been strings so that new users get a somewhat helpful message
1204 when they try to quit:
1208 'Use Ctrl-D (i.e. EOF) to exit.'
1211 In Python 2.5, \code{quit} and \code{exit} are now objects that still
1212 produce string representations of themselves, but are also callable.
1213 Newbies who try \code{quit()} or \code{exit()} will now exit the
1214 interpreter as they expect. (Implemented by Georg Brandl.)
1217 %======================================================================
1218 \subsection{Optimizations\label{opts}}
1220 Several of the optimizations were developed at the NeedForSpeed
1221 sprint, an event held in Reykjavik, Iceland, from May 21--28 2006.
1222 The sprint focused on speed enhancements to the CPython implementation
1223 and was funded by EWT LLC with local support from CCP Games. Those
1224 optimizations added at this sprint are specially marked in the
1229 \item When they were introduced
1230 in Python 2.4, the built-in \class{set} and \class{frozenset} types
1231 were built on top of Python's dictionary type.
1232 In 2.5 the internal data structure has been customized for implementing sets,
1233 and as a result sets will use a third less memory and are somewhat faster.
1234 (Implemented by Raymond Hettinger.)
1236 \item The speed of some Unicode operations, such as finding
1237 substrings, string splitting, and character map encoding and decoding,
1238 has been improved. (Substring search and splitting improvements were
1239 added by Fredrik Lundh and Andrew Dalke at the NeedForSpeed
1240 sprint. Character maps were improved by Walter D\"orwald and
1241 Martin von~L\"owis.)
1242 % Patch 1313939, 1359618
1244 \item The \function{long(\var{str}, \var{base})} function is now
1245 faster on long digit strings because fewer intermediate results are
1246 calculated. The peak is for strings of around 800--1000 digits where
1247 the function is 6 times faster.
1248 (Contributed by Alan McIntyre and committed at the NeedForSpeed sprint.)
1251 \item The \module{struct} module now compiles structure format
1252 strings into an internal representation and caches this
1253 representation, yielding a 20\% speedup. (Contributed by Bob Ippolito
1254 at the NeedForSpeed sprint.)
1256 \item The \module{re} module got a 1 or 2\% speedup by switching to
1257 Python's allocator functions instead of the system's
1258 \cfunction{malloc()} and \cfunction{free()}.
1259 (Contributed by Jack Diederich at the NeedForSpeed sprint.)
1261 \item The code generator's peephole optimizer now performs
1262 simple constant folding in expressions. If you write something like
1263 \code{a = 2+3}, the code generator will do the arithmetic and produce
1264 code corresponding to \code{a = 5}.
1266 \item Function calls are now faster because code objects now keep
1267 the most recently finished frame (a ``zombie frame'') in an internal
1268 field of the code object, reusing it the next time the code object is
1269 invoked. (Original patch by Michael Hudson, modified by Armin Rigo
1270 and Richard Jones; committed at the NeedForSpeed sprint.)
1273 Frame objects are also slightly smaller, which may improve cache locality
1274 and reduce memory usage a bit. (Contributed by Neal Norwitz.)
1277 \item Python's built-in exceptions are now new-style classes, a change
1278 that speeds up instantiation considerably. Exception handling in
1279 Python 2.5 is therefore about 30\% faster than in 2.4.
1280 (Contributed by Richard Jones, Georg Brandl and Sean Reifschneider at
1281 the NeedForSpeed sprint.)
1283 \item Importing now caches the paths tried, recording whether
1284 they exist or not so that the interpreter makes fewer
1285 \cfunction{open()} and \cfunction{stat()} calls on startup.
1286 (Contributed by Martin von~L\"owis and Georg Brandl.)
1291 The net result of the 2.5 optimizations is that Python 2.5 runs the
1292 pystone benchmark around XXX\% faster than Python 2.4.
1295 %======================================================================
1296 \section{New, Improved, and Removed Modules\label{modules}}
1298 The standard library received many enhancements and bug fixes in
1299 Python 2.5. Here's a partial list of the most notable changes, sorted
1300 alphabetically by module name. Consult the \file{Misc/NEWS} file in
1301 the source tree for a more complete list of changes, or look through
1302 the SVN logs for all the details.
1306 \item The \module{audioop} module now supports the a-LAW encoding,
1307 and the code for u-LAW encoding has been improved. (Contributed by
1310 \item The \module{codecs} module gained support for incremental
1311 codecs. The \function{codec.lookup()} function now
1312 returns a \class{CodecInfo} instance instead of a tuple.
1313 \class{CodecInfo} instances behave like a 4-tuple to preserve backward
1314 compatibility but also have the attributes \member{encode},
1315 \member{decode}, \member{incrementalencoder}, \member{incrementaldecoder},
1316 \member{streamwriter}, and \member{streamreader}. Incremental codecs
1317 can receive input and produce output in multiple chunks; the output is
1318 the same as if the entire input was fed to the non-incremental codec.
1319 See the \module{codecs} module documentation for details.
1320 (Designed and implemented by Walter D\"orwald.)
1323 \item The \module{collections} module gained a new type,
1324 \class{defaultdict}, that subclasses the standard \class{dict}
1325 type. The new type mostly behaves like a dictionary but constructs a
1326 default value when a key isn't present, automatically adding it to the
1327 dictionary for the requested key value.
1329 The first argument to \class{defaultdict}'s constructor is a factory
1330 function that gets called whenever a key is requested but not found.
1331 This factory function receives no arguments, so you can use built-in
1332 type constructors such as \function{list()} or \function{int()}. For
1334 you can make an index of words based on their initial letter like this:
1337 words = """Nel mezzo del cammin di nostra vita
1338 mi ritrovai per una selva oscura
1339 che la diritta via era smarrita""".lower().split()
1341 index = defaultdict(list)
1345 index[init_letter].append(w)
1348 Printing \code{index} results in the following output:
1351 defaultdict(<type 'list'>, {'c': ['cammin', 'che'], 'e': ['era'],
1352 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'],
1353 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'],
1354 'p': ['per'], 's': ['selva', 'smarrita'],
1355 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']}
1358 The \class{deque} double-ended queue type supplied by the
1359 \module{collections} module now has a \method{remove(\var{value})}
1360 method that removes the first occurrence of \var{value} in the queue,
1361 raising \exception{ValueError} if the value isn't found.
1363 \item New module: The \module{contextlib} module contains helper functions for use
1364 with the new '\keyword{with}' statement. See
1365 section~\ref{module-contextlib} for more about this module.
1367 \item New module: The \module{cProfile} module is a C implementation of
1368 the existing \module{profile} module that has much lower overhead.
1369 The module's interface is the same as \module{profile}: you run
1370 \code{cProfile.run('main()')} to profile a function, can save profile
1371 data to a file, etc. It's not yet known if the Hotshot profiler,
1372 which is also written in C but doesn't match the \module{profile}
1373 module's interface, will continue to be maintained in future versions
1374 of Python. (Contributed by Armin Rigo.)
1376 Also, the \module{pstats} module for analyzing the data measured by
1377 the profiler now supports directing the output to any file object
1378 by supplying a \var{stream} argument to the \class{Stats} constructor.
1379 (Contributed by Skip Montanaro.)
1381 \item The \module{csv} module, which parses files in
1382 comma-separated value format, received several enhancements and a
1383 number of bugfixes. You can now set the maximum size in bytes of a
1384 field by calling the \method{csv.field_size_limit(\var{new_limit})}
1385 function; omitting the \var{new_limit} argument will return the
1386 currently-set limit. The \class{reader} class now has a
1387 \member{line_num} attribute that counts the number of physical lines
1388 read from the source; records can span multiple physical lines, so
1389 \member{line_num} is not the same as the number of records read.
1390 (Contributed by Skip Montanaro and Andrew McNamara.)
1392 \item The \class{datetime} class in the \module{datetime}
1393 module now has a \method{strptime(\var{string}, \var{format})}
1394 method for parsing date strings, contributed by Josh Spoerri.
1395 It uses the same format characters as \function{time.strptime()} and
1396 \function{time.strftime()}:
1399 from datetime import datetime
1401 ts = datetime.strptime('10:13:15 2006-03-07',
1402 '%H:%M:%S %Y-%m-%d')
1405 \item The \method{SequenceMatcher.get_matching_blocks()} method
1406 in the \module{difflib} module now guarantees to return a minimal list
1407 of blocks describing matching subsequences. Previously, the algorithm would
1408 occasionally break a block of matching elements into two list entries.
1409 (Enhancement by Tim Peters.)
1411 \item The \module{doctest} module gained a \code{SKIP} option that
1412 keeps an example from being executed at all. This is intended for
1413 code snippets that are usage examples intended for the reader and
1414 aren't actually test cases.
1416 An \var{encoding} parameter was added to the \function{testfile()}
1417 function and the \class{DocFileSuite} class to specify the file's
1418 encoding. This makes it easier to use non-ASCII characters in
1419 tests contained within a docstring. (Contributed by Bjorn Tillenius.)
1422 \item The \module{email} package has been updated to version 4.0.
1423 % XXX need to provide some more detail here
1424 (Contributed by Barry Warsaw.)
1426 \item The \module{fileinput} module was made more flexible.
1427 Unicode filenames are now supported, and a \var{mode} parameter that
1428 defaults to \code{"r"} was added to the
1429 \function{input()} function to allow opening files in binary or
1430 universal-newline mode. Another new parameter, \var{openhook},
1431 lets you use a function other than \function{open()}
1432 to open the input files. Once you're iterating over
1433 the set of files, the \class{FileInput} object's new
1434 \method{fileno()} returns the file descriptor for the currently opened file.
1435 (Contributed by Georg Brandl.)
1437 \item In the \module{gc} module, the new \function{get_count()} function
1438 returns a 3-tuple containing the current collection counts for the
1439 three GC generations. This is accounting information for the garbage
1440 collector; when these counts reach a specified threshold, a garbage
1441 collection sweep will be made. The existing \function{gc.collect()}
1442 function now takes an optional \var{generation} argument of 0, 1, or 2
1443 to specify which generation to collect.
1444 (Contributed by Barry Warsaw.)
1446 \item The \function{nsmallest()} and
1447 \function{nlargest()} functions in the \module{heapq} module
1448 now support a \code{key} keyword parameter similar to the one
1449 provided by the \function{min()}/\function{max()} functions
1450 and the \method{sort()} methods. For example:
1454 >>> L = ["short", 'medium', 'longest', 'longer still']
1455 >>> heapq.nsmallest(2, L) # Return two lowest elements, lexicographically
1456 ['longer still', 'longest']
1457 >>> heapq.nsmallest(2, L, key=len) # Return two shortest elements
1461 (Contributed by Raymond Hettinger.)
1463 \item The \function{itertools.islice()} function now accepts
1464 \code{None} for the start and step arguments. This makes it more
1465 compatible with the attributes of slice objects, so that you can now write
1469 s = slice(5) # Create slice object
1470 itertools.islice(iterable, s.start, s.stop, s.step)
1473 (Contributed by Raymond Hettinger.)
1475 \item The \module{mailbox} module underwent a massive rewrite to add
1476 the capability to modify mailboxes in addition to reading them. A new
1477 set of classes that include \class{mbox}, \class{MH}, and
1478 \class{Maildir} are used to read mailboxes, and have an
1479 \method{add(\var{message})} method to add messages,
1480 \method{remove(\var{key})} to remove messages, and
1481 \method{lock()}/\method{unlock()} to lock/unlock the mailbox. The
1482 following example converts a maildir-format mailbox into an mbox-format one:
1487 # 'factory=None' uses email.Message.Message as the class representing
1488 # individual messages.
1489 src = mailbox.Maildir('maildir', factory=None)
1490 dest = mailbox.mbox('/tmp/mbox')
1496 (Contributed by Gregory K. Johnson. Funding was provided by Google's
1497 2005 Summer of Code.)
1499 \item New module: the \module{msilib} module allows creating
1500 Microsoft Installer \file{.msi} files and CAB files. Some support
1501 for reading the \file{.msi} database is also included.
1502 (Contributed by Martin von~L\"owis.)
1504 \item The \module{nis} module now supports accessing domains other
1505 than the system default domain by supplying a \var{domain} argument to
1506 the \function{nis.match()} and \function{nis.maps()} functions.
1507 (Contributed by Ben Bell.)
1509 \item The \module{operator} module's \function{itemgetter()}
1510 and \function{attrgetter()} functions now support multiple fields.
1511 A call such as \code{operator.attrgetter('a', 'b')}
1512 will return a function
1513 that retrieves the \member{a} and \member{b} attributes. Combining
1514 this new feature with the \method{sort()} method's \code{key} parameter
1515 lets you easily sort lists using multiple fields.
1516 (Contributed by Raymond Hettinger.)
1518 \item The \module{optparse} module was updated to version 1.5.1 of the
1519 Optik library. The \class{OptionParser} class gained an
1520 \member{epilog} attribute, a string that will be printed after the
1521 help message, and a \method{destroy()} method to break reference
1522 cycles created by the object. (Contributed by Greg Ward.)
1524 \item The \module{os} module underwent several changes. The
1525 \member{stat_float_times} variable now defaults to true, meaning that
1526 \function{os.stat()} will now return time values as floats. (This
1527 doesn't necessarily mean that \function{os.stat()} will return times
1528 that are precise to fractions of a second; not all systems support
1531 Constants named \member{os.SEEK_SET}, \member{os.SEEK_CUR}, and
1532 \member{os.SEEK_END} have been added; these are the parameters to the
1533 \function{os.lseek()} function. Two new constants for locking are
1534 \member{os.O_SHLOCK} and \member{os.O_EXLOCK}.
1536 Two new functions, \function{wait3()} and \function{wait4()}, were
1537 added. They're similar the \function{waitpid()} function which waits
1538 for a child process to exit and returns a tuple of the process ID and
1539 its exit status, but \function{wait3()} and \function{wait4()} return
1540 additional information. \function{wait3()} doesn't take a process ID
1541 as input, so it waits for any child process to exit and returns a
1542 3-tuple of \var{process-id}, \var{exit-status}, \var{resource-usage}
1543 as returned from the \function{resource.getrusage()} function.
1544 \function{wait4(\var{pid})} does take a process ID.
1545 (Contributed by Chad J. Schroeder.)
1547 On FreeBSD, the \function{os.stat()} function now returns
1548 times with nanosecond resolution, and the returned object
1549 now has \member{st_gen} and \member{st_birthtime}.
1550 The \member{st_flags} member is also available, if the platform supports it.
1551 (Contributed by Antti Louko and Diego Petten\`o.)
1552 % (Patch 1180695, 1212117)
1554 \item The Python debugger provided by the \module{pdb} module
1555 can now store lists of commands to execute when a breakpoint is
1556 reached and execution stops. Once breakpoint \#1 has been created,
1557 enter \samp{commands 1} and enter a series of commands to be executed,
1558 finishing the list with \samp{end}. The command list can include
1559 commands that resume execution, such as \samp{continue} or
1560 \samp{next}. (Contributed by Gr\'egoire Dooms.)
1563 \item The \module{pickle} and \module{cPickle} modules no
1564 longer accept a return value of \code{None} from the
1565 \method{__reduce__()} method; the method must return a tuple of
1566 arguments instead. The ability to return \code{None} was deprecated
1567 in Python 2.4, so this completes the removal of the feature.
1569 \item The \module{pkgutil} module, containing various utility
1570 functions for finding packages, was enhanced to support PEP 302's
1571 import hooks and now also works for packages stored in ZIP-format archives.
1572 (Contributed by Phillip J. Eby.)
1574 \item The pybench benchmark suite by Marc-Andr\'e~Lemburg is now
1575 included in the \file{Tools/pybench} directory. The pybench suite is
1576 an improvement on the commonly used \file{pystone.py} program because
1577 pybench provides a more detailed measurement of the interpreter's
1578 speed. It times particular operations such as function calls,
1579 tuple slicing, method lookups, and numeric operations, instead of
1580 performing many different operations and reducing the result to a
1581 single number as \file{pystone.py} does.
1583 \item The \module{pyexpat} module now uses version 2.0 of the Expat parser.
1584 (Contributed by Trent Mick.)
1586 \item The old \module{regex} and \module{regsub} modules, which have been
1587 deprecated ever since Python 2.0, have finally been deleted.
1588 Other deleted modules: \module{statcache}, \module{tzparse},
1591 \item Also deleted: the \file{lib-old} directory,
1592 which includes ancient modules such as \module{dircmp} and
1593 \module{ni}, was removed. \file{lib-old} wasn't on the default
1594 \code{sys.path}, so unless your programs explicitly added the directory to
1595 \code{sys.path}, this removal shouldn't affect your code.
1597 \item The \module{rlcompleter} module is no longer
1598 dependent on importing the \module{readline} module and
1599 therefore now works on non-{\UNIX} platforms.
1600 (Patch from Robert Kiendl.)
1603 \item The \module{SimpleXMLRPCServer} and \module{DocXMLRPCServer}
1604 classes now have a \member{rpc_paths} attribute that constrains
1605 XML-RPC operations to a limited set of URL paths; the default is
1606 to allow only \code{'/'} and \code{'/RPC2'}. Setting
1607 \member{rpc_paths} to \code{None} or an empty tuple disables
1611 \item The \module{socket} module now supports \constant{AF_NETLINK}
1612 sockets on Linux, thanks to a patch from Philippe Biondi.
1613 Netlink sockets are a Linux-specific mechanism for communications
1614 between a user-space process and kernel code; an introductory
1615 article about them is at \url{http://www.linuxjournal.com/article/7356}.
1616 In Python code, netlink addresses are represented as a tuple of 2 integers,
1617 \code{(\var{pid}, \var{group_mask})}.
1619 Two new methods on socket objects, \method{recv_buf(\var{buffer})} and
1620 \method{recvfrom_buf(\var{buffer})}, store the received data in an object
1621 that supports the buffer protocol instead of returning the data as a
1622 string. This means you can put the data directly into an array or a
1625 Socket objects also gained \method{getfamily()}, \method{gettype()},
1626 and \method{getproto()} accessor methods to retrieve the family, type,
1627 and protocol values for the socket.
1629 \item New module: the \module{spwd} module provides functions for
1630 accessing the shadow password database on systems that support
1633 \item The \module{struct} is now faster because it
1634 compiles format strings into \class{Struct} objects
1635 with \method{pack()} and \method{unpack()} methods. This is similar
1636 to how the \module{re} module lets you create compiled regular
1637 expression objects. You can still use the module-level
1638 \function{pack()} and \function{unpack()} functions; they'll create
1639 \class{Struct} objects and cache them. Or you can use
1640 \class{Struct} instances directly:
1643 s = struct.Struct('ih3s')
1645 data = s.pack(1972, 187, 'abc')
1646 year, number, name = s.unpack(data)
1649 You can also pack and unpack data to and from buffer objects directly
1650 using the \method{pack_to(\var{buffer}, \var{offset}, \var{v1},
1651 \var{v2}, ...)} and \method{unpack_from(\var{buffer}, \var{offset})}
1652 methods. This lets you store data directly into an array or a
1655 (\class{Struct} objects were implemented by Bob Ippolito at the
1656 NeedForSpeed sprint. Support for buffer objects was added by Martin
1657 Blais, also at the NeedForSpeed sprint.)
1659 \item The Python developers switched from CVS to Subversion during the 2.5
1660 development process. Information about the exact build version is
1661 available as the \code{sys.subversion} variable, a 3-tuple of
1662 \code{(\var{interpreter-name}, \var{branch-name},
1663 \var{revision-range})}. For example, at the time of writing my copy
1664 of 2.5 was reporting \code{('CPython', 'trunk', '45313:45315')}.
1666 This information is also available to C extensions via the
1667 \cfunction{Py_GetBuildInfo()} function that returns a
1668 string of build information like this:
1669 \code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}.
1670 (Contributed by Barry Warsaw.)
1672 \item The \class{TarFile} class in the \module{tarfile} module now has
1673 an \method{extractall()} method that extracts all members from the
1674 archive into the current working directory. It's also possible to set
1675 a different directory as the extraction target, and to unpack only a
1676 subset of the archive's members.
1678 A tarfile's compression can be autodetected by
1679 using the mode \code{'r|*'}.
1681 (Contributed by Lars Gust\"abel.)
1683 \item The \module{threading} module now lets you set the stack size
1684 used when new threads are created. The
1685 \function{stack_size(\optional{\var{size}})} function returns the
1686 currently configured stack size, and supplying the optional \var{size}
1687 parameter sets a new value. Not all platforms support changing the
1688 stack size, but Windows, POSIX threading, and OS/2 all do.
1689 (Contributed by Andrew MacIntyre.)
1692 \item The \module{unicodedata} module has been updated to use version 4.1.0
1693 of the Unicode character database. Version 3.2.0 is required
1694 by some specifications, so it's still available as
1695 \member{unicodedata.ucd_3_2_0}.
1697 \item New module: the \module{uuid} module generates
1698 universally unique identifiers (UUIDs) according to \rfc{4122}. The
1699 RFC defines several different UUID versions that are generated from a
1700 starting string, from system properties, or purely randomly. This
1701 module contains a \class{UUID} class and
1702 functions named \function{uuid1()},
1703 \function{uuid3()}, \function{uuid4()}, and
1704 \function{uuid5()} to generate different versions of UUID. (Version 2 UUIDs
1705 are not specified in \rfc{4122} and are not supported by this module.)
1709 >>> # make a UUID based on the host ID and current time
1711 UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')
1713 >>> # make a UUID using an MD5 hash of a namespace UUID and a name
1714 >>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
1715 UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')
1717 >>> # make a random UUID
1719 UUID('16fd2706-8baf-433b-82eb-8c7fada847da')
1721 >>> # make a UUID using a SHA-1 hash of a namespace UUID and a name
1722 >>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
1723 UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')
1726 (Contributed by Ka-Ping Yee.)
1728 \item The \module{weakref} module's \class{WeakKeyDictionary} and
1729 \class{WeakValueDictionary} types gained new methods for iterating
1730 over the weak references contained in the dictionary.
1731 \method{iterkeyrefs()} and \method{keyrefs()} methods were
1732 added to \class{WeakKeyDictionary}, and
1733 \method{itervaluerefs()} and \method{valuerefs()} were added to
1734 \class{WeakValueDictionary}. (Contributed by Fred L.~Drake, Jr.)
1736 \item The \module{webbrowser} module received a number of
1738 It's now usable as a script with \code{python -m webbrowser}, taking a
1739 URL as the argument; there are a number of switches
1740 to control the behaviour (\programopt{-n} for a new browser window,
1741 \programopt{-t} for a new tab). New module-level functions,
1742 \function{open_new()} and \function{open_new_tab()}, were added
1743 to support this. The module's \function{open()} function supports an
1744 additional feature, an \var{autoraise} parameter that signals whether
1745 to raise the open window when possible. A number of additional
1746 browsers were added to the supported list such as Firefox, Opera,
1747 Konqueror, and elinks. (Contributed by Oleg Broytmann and Georg
1751 \item The standard library's XML-related package
1752 has been renamed to \module{xmlcore}. The \module{xml} module will
1753 now import either the \module{xmlcore} or PyXML version of subpackages
1754 such as \module{xml.dom}. The renaming means it will always be
1755 possible to import the standard library's XML support whether or not
1756 the PyXML package is installed.
1758 \item The \module{xmlrpclib} module now supports returning
1759 \class{datetime} objects for the XML-RPC date type. Supply
1760 \code{use_datetime=True} to the \function{loads()} function
1761 or the \class{Unmarshaller} class to enable this feature.
1762 (Contributed by Skip Montanaro.)
1765 \item The \module{zipfile} module now supports the ZIP64 version of the
1766 format, meaning that a .zip archive can now be larger than 4~GiB and
1767 can contain individual files larger than 4~GiB. (Contributed by
1771 \item The \module{zlib} module's \class{Compress} and \class{Decompress}
1772 objects now support a \method{copy()} method that makes a copy of the
1773 object's internal state and returns a new
1774 \class{Compress} or \class{Decompress} object.
1775 (Contributed by Chris AtLee.)
1782 %======================================================================
1783 \subsection{The ctypes package\label{module-ctypes}}
1785 The \module{ctypes} package, written by Thomas Heller, has been added
1786 to the standard library. \module{ctypes} lets you call arbitrary functions
1787 in shared libraries or DLLs. Long-time users may remember the \module{dl} module, which
1788 provides functions for loading shared libraries and calling functions in them. The \module{ctypes} package is much fancier.
1790 To load a shared library or DLL, you must create an instance of the
1791 \class{CDLL} class and provide the name or path of the shared library
1792 or DLL. Once that's done, you can call arbitrary functions
1793 by accessing them as attributes of the \class{CDLL} object.
1798 libc = ctypes.CDLL('libc.so.6')
1799 result = libc.printf("Line of output\n")
1802 Type constructors for the various C types are provided: \function{c_int},
1803 \function{c_float}, \function{c_double}, \function{c_char_p} (equivalent to \ctype{char *}), and so forth. Unlike Python's types, the C versions are all mutable; you can assign to their \member{value} attribute
1804 to change the wrapped value. Python integers and strings will be automatically
1805 converted to the corresponding C types, but for other types you
1806 must call the correct type constructor. (And I mean \emph{must};
1807 getting it wrong will often result in the interpreter crashing
1808 with a segmentation fault.)
1810 You shouldn't use \function{c_char_p} with a Python string when the C function will be modifying the memory area, because Python strings are
1811 supposed to be immutable; breaking this rule will cause puzzling bugs. When you need a modifiable memory area,
1812 use \function{create_string_buffer()}:
1815 s = "this is a string"
1816 buf = ctypes.create_string_buffer(s)
1820 C functions are assumed to return integers, but you can set
1821 the \member{restype} attribute of the function object to
1825 >>> libc.atof('2.71828')
1827 >>> libc.atof.restype = ctypes.c_double
1828 >>> libc.atof('2.71828')
1832 \module{ctypes} also provides a wrapper for Python's C API
1833 as the \code{ctypes.pythonapi} object. This object does \emph{not}
1834 release the global interpreter lock before calling a function, because the lock must be held when calling into the interpreter's code.
1835 There's a \class{py_object()} type constructor that will create a
1836 \ctype{PyObject *} pointer. A simple usage:
1842 ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d),
1843 ctypes.py_object("abc"), ctypes.py_object(1))
1844 # d is now {'abc', 1}.
1847 Don't forget to use \class{py_object()}; if it's omitted you end
1848 up with a segmentation fault.
1850 \module{ctypes} has been around for a while, but people still write
1851 and distribution hand-coded extension modules because you can't rely on \module{ctypes} being present.
1852 Perhaps developers will begin to write
1853 Python wrappers atop a library accessed through \module{ctypes} instead
1854 of extension modules, now that \module{ctypes} is included with core Python.
1858 \seeurl{http://starship.python.net/crew/theller/ctypes/}
1859 {The ctypes web page, with a tutorial, reference, and FAQ.}
1861 \seeurl{../lib/module-ctypes.html}{The documentation
1862 for the \module{ctypes} module.}
1867 %======================================================================
1868 \subsection{The ElementTree package\label{module-etree}}
1870 A subset of Fredrik Lundh's ElementTree library for processing XML has
1871 been added to the standard library as \module{xml.etree}. The
1872 available modules are
1873 \module{ElementTree}, \module{ElementPath}, and
1874 \module{ElementInclude} from ElementTree 1.2.6.
1875 The \module{cElementTree} accelerator module is also included.
1877 The rest of this section will provide a brief overview of using
1878 ElementTree. Full documentation for ElementTree is available at
1879 \url{http://effbot.org/zone/element-index.htm}.
1881 ElementTree represents an XML document as a tree of element nodes.
1882 The text content of the document is stored as the \member{.text}
1883 and \member{.tail} attributes of
1884 (This is one of the major differences between ElementTree and
1885 the Document Object Model; in the DOM there are many different
1886 types of node, including \class{TextNode}.)
1888 The most commonly used parsing function is \function{parse()}, that
1889 takes either a string (assumed to contain a filename) or a file-like
1890 object and returns an \class{ElementTree} instance:
1893 from xml.etree import ElementTree as ET
1895 tree = ET.parse('ex-1.xml')
1897 feed = urllib.urlopen(
1898 'http://planet.python.org/rss10.xml')
1899 tree = ET.parse(feed)
1902 Once you have an \class{ElementTree} instance, you
1903 can call its \method{getroot()} method to get the root \class{Element} node.
1905 There's also an \function{XML()} function that takes a string literal
1906 and returns an \class{Element} node (not an \class{ElementTree}).
1907 This function provides a tidy way to incorporate XML fragments,
1908 approaching the convenience of an XML literal:
1911 svg = ET.XML("""<svg width="10px" version="1.0">
1913 svg.set('height', '320px')
1917 Each XML element supports some dictionary-like and some list-like
1918 access methods. Dictionary-like operations are used to access attribute
1919 values, and list-like operations are used to access child nodes.
1921 \begin{tableii}{c|l}{code}{Operation}{Result}
1922 \lineii{elem[n]}{Returns n'th child element.}
1923 \lineii{elem[m:n]}{Returns list of m'th through n'th child elements.}
1924 \lineii{len(elem)}{Returns number of child elements.}
1925 \lineii{list(elem)}{Returns list of child elements.}
1926 \lineii{elem.append(elem2)}{Adds \var{elem2} as a child.}
1927 \lineii{elem.insert(index, elem2)}{Inserts \var{elem2} at the specified location.}
1928 \lineii{del elem[n]}{Deletes n'th child element.}
1929 \lineii{elem.keys()}{Returns list of attribute names.}
1930 \lineii{elem.get(name)}{Returns value of attribute \var{name}.}
1931 \lineii{elem.set(name, value)}{Sets new value for attribute \var{name}.}
1932 \lineii{elem.attrib}{Retrieves the dictionary containing attributes.}
1933 \lineii{del elem.attrib[name]}{Deletes attribute \var{name}.}
1936 Comments and processing instructions are also represented as
1937 \class{Element} nodes. To check if a node is a comment or processing
1941 if elem.tag is ET.Comment:
1943 elif elem.tag is ET.ProcessingInstruction:
1947 To generate XML output, you should call the
1948 \method{ElementTree.write()} method. Like \function{parse()},
1949 it can take either a string or a file-like object:
1952 # Encoding is US-ASCII
1953 tree.write('output.xml')
1956 f = open('output.xml', 'w')
1957 tree.write(f, encoding='utf-8')
1960 (Caution: the default encoding used for output is ASCII. For general
1961 XML work, where an element's name may contain arbitrary Unicode
1962 characters, ASCII isn't a very useful encoding because it will raise
1963 an exception if an element's name contains any characters with values
1964 greater than 127. Therefore, it's best to specify a different
1965 encoding such as UTF-8 that can handle any Unicode character.)
1967 This section is only a partial description of the ElementTree interfaces.
1968 Please read the package's official documentation for more details.
1972 \seeurl{http://effbot.org/zone/element-index.htm}
1973 {Official documentation for ElementTree.}
1978 %======================================================================
1979 \subsection{The hashlib package\label{module-hashlib}}
1981 A new \module{hashlib} module, written by Gregory P. Smith,
1982 has been added to replace the
1983 \module{md5} and \module{sha} modules. \module{hashlib} adds support
1984 for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512).
1985 When available, the module uses OpenSSL for fast platform optimized
1986 implementations of algorithms.
1988 The old \module{md5} and \module{sha} modules still exist as wrappers
1989 around hashlib to preserve backwards compatibility. The new module's
1990 interface is very close to that of the old modules, but not identical.
1991 The most significant difference is that the constructor functions
1992 for creating new hashing objects are named differently.
2009 # Hash that weren't previously available
2010 h = hashlib.sha224()
2011 h = hashlib.sha256()
2012 h = hashlib.sha384()
2013 h = hashlib.sha512()
2016 h = hashlib.new('md5') # Provide algorithm as a string
2019 Once a hash object has been created, its methods are the same as before:
2020 \method{update(\var{string})} hashes the specified string into the
2021 current digest state, \method{digest()} and \method{hexdigest()}
2022 return the digest value as a binary string or a string of hex digits,
2023 and \method{copy()} returns a new hashing object with the same digest state.
2027 \seeurl{../lib/module-hashlib.html}{The documentation
2028 for the \module{hashlib} module.}
2033 %======================================================================
2034 \subsection{The sqlite3 package\label{module-sqlite}}
2036 The pysqlite module (\url{http://www.pysqlite.org}), a wrapper for the
2037 SQLite embedded database, has been added to the standard library under
2038 the package name \module{sqlite3}.
2040 SQLite is a C library that provides a SQL-language database that
2041 stores data in disk files without requiring a separate server process.
2042 pysqlite was written by Gerhard H\"aring and provides a SQL interface
2043 compliant with the DB-API 2.0 specification described by
2044 \pep{249}. This means that it should be possible to write the first
2045 version of your applications using SQLite for data storage. If
2046 switching to a larger database such as PostgreSQL or Oracle is
2047 later necessary, the switch should be relatively easy.
2049 If you're compiling the Python source yourself, note that the source
2050 tree doesn't include the SQLite code, only the wrapper module.
2051 You'll need to have the SQLite libraries and headers installed before
2052 compiling Python, and the build process will compile the module when
2053 the necessary headers are available.
2055 To use the module, you must first create a \class{Connection} object
2056 that represents the database. Here the data will be stored in the
2057 \file{/tmp/example} file:
2060 conn = sqlite3.connect('/tmp/example')
2063 You can also supply the special name \samp{:memory:} to create
2066 Once you have a \class{Connection}, you can create a \class{Cursor}
2067 object and call its \method{execute()} method to perform SQL commands:
2073 c.execute('''create table stocks
2074 (date timestamp, trans varchar, symbol varchar,
2075 qty decimal, price decimal)''')
2077 # Insert a row of data
2078 c.execute("""insert into stocks
2079 values ('2006-01-05','BUY','RHAT',100,35.14)""")
2082 Usually your SQL operations will need to use values from Python
2083 variables. You shouldn't assemble your query using Python's string
2084 operations because doing so is insecure; it makes your program
2085 vulnerable to an SQL injection attack.
2087 Instead, use the DB-API's parameter substitution. Put \samp{?} as a
2088 placeholder wherever you want to use a value, and then provide a tuple
2089 of values as the second argument to the cursor's \method{execute()}
2090 method. (Other database modules may use a different placeholder,
2091 such as \samp{\%s} or \samp{:1}.) For example:
2094 # Never do this -- insecure!
2096 c.execute("... where symbol = '%s'" % symbol)
2100 c.execute('select * from stocks where symbol=?', t)
2103 for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
2104 ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
2105 ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
2107 c.execute('insert into stocks values (?,?,?,?,?)', t)
2110 To retrieve data after executing a SELECT statement, you can either
2111 treat the cursor as an iterator, call the cursor's \method{fetchone()}
2112 method to retrieve a single matching row,
2113 or call \method{fetchall()} to get a list of the matching rows.
2115 This example uses the iterator form:
2118 >>> c = conn.cursor()
2119 >>> c.execute('select * from stocks order by price')
2123 (u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
2124 (u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
2125 (u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
2126 (u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
2130 For more information about the SQL dialect supported by SQLite, see
2131 \url{http://www.sqlite.org}.
2135 \seeurl{http://www.pysqlite.org}
2136 {The pysqlite web page.}
2138 \seeurl{http://www.sqlite.org}
2139 {The SQLite web page; the documentation describes the syntax and the
2140 available data types for the supported SQL dialect.}
2142 \seeurl{../lib/module-sqlite3.html}{The documentation
2143 for the \module{sqlite3} module.}
2145 \seepep{249}{Database API Specification 2.0}{PEP written by
2146 Marc-Andr\'e Lemburg.}
2151 %======================================================================
2152 \subsection{The wsgiref package\label{module-wsgiref}}
2154 % XXX should this be in a PEP 333 section instead?
2156 The Web Server Gateway Interface (WSGI) v1.0 defines a standard
2157 interface between web servers and Python web applications and is
2158 described in \pep{333}. The \module{wsgiref} package is a reference
2159 implementation of the WSGI specification.
2161 The package includes a basic HTTP server that will run a WSGI
2162 application; this server is useful for debugging but isn't intended for
2163 production use. Setting up a server takes only a few lines of code:
2166 from wsgiref import simple_server
2172 httpd = simple_server.make_server(host, port, wsgi_app)
2173 httpd.serve_forever()
2176 % XXX discuss structure of WSGI applications?
2177 % XXX provide an example using Django or some other framework?
2181 \seeurl{http://www.wsgi.org}{A central web site for WSGI-related resources.}
2183 \seepep{333}{Python Web Server Gateway Interface v1.0}{PEP written by
2189 % ======================================================================
2190 \section{Build and C API Changes\label{build-api}}
2192 Changes to Python's build process and to the C API include:
2196 \item The largest change to the C API came from \pep{353},
2197 which modifies the interpreter to use a \ctype{Py_ssize_t} type
2198 definition instead of \ctype{int}. See the earlier
2199 section~\ref{pep-353} for a discussion of this change.
2201 \item The design of the bytecode compiler has changed a great deal, to
2202 no longer generate bytecode by traversing the parse tree. Instead
2203 the parse tree is converted to an abstract syntax tree (or AST), and it is
2204 the abstract syntax tree that's traversed to produce the bytecode.
2206 It's possible for Python code to obtain AST objects by using the
2207 \function{compile()} built-in and specifying \code{_ast.PyCF_ONLY_AST}
2209 \var{flags} parameter:
2212 from _ast import PyCF_ONLY_AST
2213 ast = compile("""a=0
2216 """, "<string>", 'exec', PyCF_ONLY_AST)
2218 assignment = ast.body[0]
2219 for_loop = ast.body[1]
2222 No official documentation has been written for the AST code yet, but
2223 \pep{339} discusses the design. To start learning about the code, read the
2224 definition of the various AST nodes in \file{Parser/Python.asdl}. A
2225 Python script reads this file and generates a set of C structure
2226 definitions in \file{Include/Python-ast.h}. The
2227 \cfunction{PyParser_ASTFromString()} and
2228 \cfunction{PyParser_ASTFromFile()}, defined in
2229 \file{Include/pythonrun.h}, take Python source as input and return the
2230 root of an AST representing the contents. This AST can then be turned
2231 into a code object by \cfunction{PyAST_Compile()}. For more
2232 information, read the source code, and then ask questions on
2235 % List of names taken from Jeremy's python-dev post at
2236 % http://mail.python.org/pipermail/python-dev/2005-October/057500.html
2237 The AST code was developed under Jeremy Hylton's management, and
2238 implemented by (in alphabetical order) Brett Cannon, Nick Coghlan,
2239 Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters,
2240 Armin Rigo, and Neil Schemenauer, plus the participants in a number of
2241 AST sprints at conferences such as PyCon.
2243 \item The built-in set types now have an official C API. Call
2244 \cfunction{PySet_New()} and \cfunction{PyFrozenSet_New()} to create a
2245 new set, \cfunction{PySet_Add()} and \cfunction{PySet_Discard()} to
2246 add and remove elements, and \cfunction{PySet_Contains} and
2247 \cfunction{PySet_Size} to examine the set's state.
2248 (Contributed by Raymond Hettinger.)
2250 \item C code can now obtain information about the exact revision
2251 of the Python interpreter by calling the
2252 \cfunction{Py_GetBuildInfo()} function that returns a
2253 string of build information like this:
2254 \code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}.
2255 (Contributed by Barry Warsaw.)
2257 \item Two new macros can be used to indicate C functions that are
2258 local to the current file so that a faster calling convention can be
2259 used. \cfunction{Py_LOCAL(\var{type})} declares the function as
2260 returning a value of the specified \var{type} and uses a fast-calling
2261 qualifier. \cfunction{Py_LOCAL_INLINE(\var{type})} does the same thing
2262 and also requests the function be inlined. If
2263 \cfunction{PY_LOCAL_AGGRESSIVE} is defined before \file{python.h} is
2264 included, a set of more aggressive optimizations are enabled for the
2265 module; you should benchmark the results to find out if these
2266 optimizations actually make the code faster. (Contributed by Fredrik
2267 Lundh at the NeedForSpeed sprint.)
2269 \item \cfunction{PyErr_NewException(\var{name}, \var{base},
2270 \var{dict})} can now accept a tuple of base classes as its \var{base}
2271 argument. (Contributed by Georg Brandl.)
2273 \item The CPython interpreter is still written in C, but
2274 the code can now be compiled with a {\Cpp} compiler without errors.
2275 (Implemented by Anthony Baxter, Martin von~L\"owis, Skip Montanaro.)
2277 \item The \cfunction{PyRange_New()} function was removed. It was
2278 never documented, never used in the core code, and had dangerously lax
2279 error checking. In the unlikely case that your extensions were using
2280 it, you can replace it by something like the following:
2282 range = PyObject_CallFunction((PyObject*) &PyRange_Type, "lll",
2289 %======================================================================
2290 \subsection{Port-Specific Changes
\label{ports
}}
2294 \item MacOS X (
10.3 and higher): dynamic loading of modules
2295 now uses the
\cfunction{dlopen()
} function instead of MacOS-specific
2298 \item MacOS X: a
\longprogramopt{enable-universalsdk
} switch was added
2299 to the
\program{configure
} script that compiles the interpreter as a
2300 universal binary able to run on both PowerPC and Intel processors.
2301 (Contributed by Ronald Oussoren.)
2303 \item Windows:
\file{.dll
} is no longer supported as a filename extension for
2304 extension modules.
\file{.pyd
} is now the only filename extension that will
2310 %======================================================================
2311 \section{Other Changes and Fixes
\label{section-other
}}
2313 As usual, there were a bunch of other improvements and bugfixes
2314 scattered throughout the source tree. A search through the SVN change
2315 logs finds there were XXX patches applied and YYY bugs fixed between
2316 Python
2.4 and
2.5. Both figures are likely to be underestimates.
2318 Some of the more notable changes are:
2322 \item Evan Jones's patch to obmalloc, first described in a talk
2323 at PyCon DC
2005, was applied. Python
2.4 allocated small objects in
2324 256K-sized arenas, but never freed arenas. With this patch, Python
2325 will free arenas when they're empty. The net effect is that on some
2326 platforms, when you allocate many objects, Python's memory usage may
2327 actually drop when you delete them, and the memory may be returned to
2328 the operating system. (Implemented by Evan Jones, and reworked by Tim
2331 Note that this change means extension modules need to be more careful
2332 with how they allocate memory. Python's API has many different
2333 functions for allocating memory that are grouped into families. For
2334 example,
\cfunction{PyMem_Malloc()
},
\cfunction{PyMem_Realloc()
}, and
2335 \cfunction{PyMem_Free()
} are one family that allocates raw memory,
2336 while
\cfunction{PyObject_Malloc()
},
\cfunction{PyObject_Realloc()
},
2337 and
\cfunction{PyObject_Free()
} are another family that's supposed to
2338 be used for creating Python objects.
2340 Previously these different families all reduced to the platform's
2341 \cfunction{malloc()
} and
\cfunction{free()
} functions. This meant
2342 it didn't matter if you got things wrong and allocated memory with the
2343 \cfunction{PyMem
} function but freed it with the
\cfunction{PyObject
}
2344 function. With the obmalloc change, these families now do different
2345 things, and mismatches will probably result in a segfault. You should
2346 carefully test your C extension modules with Python
2.5.
2348 \item Coverity, a company that markets a source code analysis tool
2349 called Prevent, provided the results of their examination of the Python
2350 source code. The analysis found about
60 bugs that
2351 were quickly fixed. Many of the bugs were refcounting problems, often
2352 occurring in error-handling code. See
2353 \url{http://scan.coverity.com
} for the statistics.
2358 %======================================================================
2359 \section{Porting to Python
2.5\label{porting
}}
2361 This section lists previously described changes that may require
2362 changes to your code:
2366 \item ASCII is now the default encoding for modules. It's now
2367 a syntax error if a module contains string literals with
8-bit
2368 characters but doesn't have an encoding declaration. In Python
2.4
2369 this triggered a warning, not a syntax error.
2371 \item Previously, the
\member{gi_frame
} attribute of a generator
2372 was always a frame object. Because of the
\pep{342} changes
2373 described in section~
\ref{pep-
342}, it's now possible
2374 for
\member{gi_frame
} to be
\code{None
}.
2377 \item Library: The
\module{pickle
} and
\module{cPickle
} modules no
2378 longer accept a return value of
\code{None
} from the
2379 \method{__reduce__()
} method; the method must return a tuple of
2380 arguments instead. The modules also no longer accept the deprecated
2381 \var{bin
} keyword parameter.
2383 \item Library: The
\module{SimpleXMLRPCServer
} and
\module{DocXMLRPCServer
}
2384 classes now have a
\member{rpc_paths
} attribute that constrains
2385 XML-RPC operations to a limited set of URL paths; the default is
2386 to allow only
\code{'/'
} and
\code{'/RPC2'
}. Setting
2387 \member{rpc_paths
} to
\code{None
} or an empty tuple disables
2390 \item Library: the
\module{xml
} package has been renamed to
\module{xmlcore
}.
2391 The PyXML package will therefore be
\module{xml
}, and the Python
2392 distribution's code will always be accessible as
\module{xmlcore
}.
2394 \item C API: Many functions now use
\ctype{Py_ssize_t
}
2395 instead of
\ctype{int
} to allow processing more data on
64-bit
2396 machines. Extension code may need to make the same change to avoid
2397 warnings and to support
64-bit machines. See the earlier
2398 section~
\ref{pep-
353} for a discussion of this change.
2401 The obmalloc changes mean that
2402 you must be careful to not mix usage
2403 of the
\cfunction{PyMem_*()
} and
\cfunction{PyObject_*()
}
2404 families of functions. Memory allocated with
2405 one family's
\cfunction{*_Malloc()
} must be
2406 freed with the corresponding family's
\cfunction{*_Free()
} function.
2411 %======================================================================
2412 \section{Acknowledgements
\label{acks
}}
2414 The author would like to thank the following people for offering
2415 suggestions, corrections and assistance with various drafts of this
2416 article: Nick Coghlan, Phillip J. Eby, Ralf W. Grosse-Kunstleve, Kent
2417 Johnson, Martin von~L\"owis, Fredrik Lundh, Gustavo Niemeyer, James
2418 Pryor, Mike Rovner, Scott Weikart, Barry Warsaw, Thomas Wouters.