Added a test for the ability to specify a class attribute in Formatter configuration...

[python.git] / Doc / whatsnew / whatsnew25.tex

\documentclass{howto}
\usepackage{distutils}
% $Id$


\title{What's New in Python 2.5}
\release{0.0}
\author{A.M. Kuchling}
\authoraddress{\email{amk@amk.ca}}

\begin{document}
\maketitle
\tableofcontents

This article explains the new features in Python 2.5.  No release date
for Python 2.5 has been set; it will probably be released in early 2006.

% Compare with previous release in 2 - 3 sentences here.

This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 2.5.
% add hyperlink when the documentation becomes available online.
If you want to understand the complete implementation and design
rationale, refer to the PEP for a particular new feature.


%======================================================================
\section{PEP 309: Partial Function Application}

The \module{functional} module is intended to contain tools for
functional-style programming.  Currently it only contains
\class{partial}, but new functions will probably be added in future
versions of Python.

For programs written in a functional style, it can be useful to
construct variants of existing functions that have some of the
parameters filled in.  Consider a Python function \code{f(a, b, c)};
you could create a new function \code{g(b, c)} that was equivalent to
\code{f(1, b, c)}.  This is called ``partial function application'',
and is provided by the \class{partial} class in the new
\module{functional} module.

The constructor for \class{partial} takes the arguments
\code{(\var{function}, \var{arg1}, \var{arg2}, ...
\var{kwarg1}=\var{value1}, \var{kwarg2}=\var{value2})}.  The resulting
object is callable, so you can just call it to invoke \var{function}
with the filled-in arguments.

Here's a small but realistic example:

\begin{verbatim}
import functional

def log (message, subsystem):
    "Write the contents of 'message' to the specified subsystem."
    print '%s: %s' % (subsystem, message)
    ...

server_log = functional.partial(log, subsystem='server')
\end{verbatim}

Here's another example, from a program that uses PyGTk.  Here a
context-sensitive pop-up menu is being constructed dynamically.  The
callback provided for the menu option is a partially applied version
of the \method{open_item()} method, where the first argument has been
provided.

\begin{verbatim}
...
class Application:
    def open_item(self, path):
       ...
    def init (self):
        open_func = functional.partial(self.open_item, item_path)
        popup_menu.append( ("Open", open_func, 1) )
\end{verbatim}


\begin{seealso}

\seepep{309}{Partial Function Application}{PEP proposed and written by
Peter Harris; implemented by Hye-Shik Chang, with adaptations by
Raymond Hettinger.}

\end{seealso}


%======================================================================
\section{PEP 314: Metadata for Python Software Packages v1.1}

Some simple dependency support was added to Distutils.  The
\function{setup()} function now has \code{requires},\code{provides},
and \code{obsoletes}.  When you build a source distribution using the
\code{sdist} command, the dependency information will be recorded in
the \file{PKG-INFO} file.  

Another new keyword is \code{download_url}, which should be set to a
URL for the package's source code.  This means it's now possible to
look up an entry in the package index, determine the dependencies for
a package, and download the required packages.  

% XXX put example here
 
\begin{seealso}

\seepep{314}{Metadata for Python Software Packages v1.1}{PEP proposed
and written by A.M. Kuchling, Richard Jones, and Fred Drake; 
implemented by Richard Jones and Fred Drake.}

\end{seealso}


%======================================================================
\section{PEP 342: New Generator Features}

As introduced in Python 2.3, generators only produce output; once a
generator's code was invoked to create an iterator, there's no way to
pass new parameters into the function when its execution is resumed.
Hackish solutions to this include making the generator's code look at
a global variable and then changing the global variable's value, or
passing in some mutable object that callers then modify.  Python
2.5 adds the ability to pass values \emph{into} a generator.

To refresh your memory of basic generators, here's a simple example:

\begin{verbatim}
def counter (maximum):
    i = 0
    while i < maximum:
        yield i
        i += 1
\end{verbatim}

When you call \code{counter(10)}, the result is an iterator that
returns the values from 0 up to 9.  On encountering the
\keyword{yield} statement, the iterator returns the provided value and
suspends the function's execution, preserving the local variables.
Execution resumes on the following call to the iterator's 
\method{next()} method, picking up after the \keyword{yield}.

In Python 2.3, \keyword{yield} was a statement; it didn't return any
value.  In 2.5, \keyword{yield} is now an expression, returning a
value that can be assigned to a variable or otherwise operated on:

\begin{verbatim}
val = (yield i)
\end{verbatim}

I recommend that you always put parentheses around a \keyword{yield}
expression when you're doing something with the returned value, as in
the above example.  The parentheses aren't always necessary, but it's
easier to always add them instead of having to remember when they're
needed.  The exact rules are that a \keyword{yield}-expression must
always be parenthesized except when it occurs at the top-level
expression on the right-hand side of an assignment, meaning
you can to write \code{val = yield i} but \code{val = (yield i) + 12}.

Values are sent into a generator by calling its
\method{send(\var{value})} method.  The generator's code is then
resumed and the \keyword{yield} expression produces \var{value}.
If the regular \method{next()} method is called, the \keyword{yield} 
returns \constant{None}.

Here's the previous example, modified to allow changing the value of
the internal counter.

\begin{verbatim}
def counter (maximum):
    i = 0
    while i < maximum:
        val = (yield i)
        # If value provided, change counter
        if val is not None:
            i = val
        else:
            i += 1
\end{verbatim}

And here's an example of changing the counter:

\begin{verbatim}
>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
  File ``t.py'', line 15, in ?
    print it.next()
StopIteration
\end{verbatim}

Because \keyword{yield} will often be returning \constant{None}, 
you shouldn't just use its value in expressions unless you're sure 
that only the \method{send()} method will be used.

There are two other new methods on generators in addition to 
\method{send()}:

\begin{itemize}

  \item \method{throw(\var{type}, \var{value}=None,
  \var{traceback}=None)} is used to raise an exception inside the
  generator; the exception is raised by the \keyword{yield} expression
  where the generator's execution is paused.

  \item \method{close()} raises a new \exception{GeneratorExit}
  exception inside the generator to terminate the iteration.  
  On receiving this
  exception, the generator's code must either raise
  \exception{GeneratorExit} or \exception{StopIteration}; catching the 
  exception and doing anything else is illegal and will trigger
  a \exception{RuntimeError}.  \method{close()} will also be called by 
  Python's garbage collection when the generator is garbage-collected.

  If you need to run cleanup code in case of a \exception{GeneratorExit},
  I suggest using a \code{try: ... finally:} suite instead of 
  catching \exception{GeneratorExit}.

\end{itemize}

The cumulative effect of these changes is to turn generators from
one-way producers of information into both producers and consumers.
Generators also become \emph{coroutines}, a more generalized form of
subroutines; subroutines are entered at one point and exited at
another point (the top of the function, and a \keyword{return
statement}), but coroutines can be entered, exited, and resumed at
many different points (the \keyword{yield} statements).science term

  
\begin{seealso}

\seepep{342}{Coroutines via Enhanced Generators}{PEP written by 
Guido van Rossum and Phillip J. Eby;
implemented by Phillip J. Eby.  Includes examples of 
some fancier uses of generators as coroutines.}

\seeurl{http://en.wikipedia.org/wiki/Coroutine}{The Wikipedia entry for 
coroutines.}

\seeurl{http://www.sidhe.org/~dan/blog/archives/000178.html}{An
explanation of coroutines from a Perl point of view, written by Dan
Sugalski.}

\end{seealso}


%======================================================================
\section{Other Language Changes}

Here are all of the changes that Python 2.5 makes to the core Python
language.

\begin{itemize}

\item The \function{min()} and \function{max()} built-in functions
gained a \code{key} keyword argument analogous to the \code{key}
argument for \method{sort()}.  This argument supplies a function
that takes a single argument and is called for every value in the list; 
\function{min()}/\function{max()} will return the element with the 
smallest/largest return value from this function.
For example, to find the longest string in a list, you can do:

\begin{verbatim}
L = ['medium', 'longest', 'short']
# Prints 'longest'
print max(L, key=len)              
# Prints 'short', because lexicographically 'short' has the largest value
print max(L)         
\end{verbatim}

(Contributed by Steven Bethard and Raymond Hettinger.)

\item Two new built-in functions, \function{any()} and
\function{all()}, evaluate whether an iterator contains any true or
false values.  \function{any()} returns \constant{True} if any value
returned by the iterator is true; otherwise it will return
\constant{False}.  \function{all()} returns \constant{True} only if
all of the values returned by the iterator evaluate as being true.

% XXX who added?


\item The list of base classes in a class definition can now be empty.  
As an example, this is now legal:

\begin{verbatim}
class C():
    pass
\end{verbatim}
(Implemented by Brett Cannon.)

\end{itemize}


%======================================================================
\subsection{Optimizations}

\begin{itemize}

\item When they were introduced 
in Python 2.4, the built-in \class{set} and \class{frozenset} types
were built on top of Python's dictionary type.  
In 2.5 the internal data structure has been customized for implementing sets,
and as a result sets will use a third less memory and are somewhat faster.
(Implemented by Raymond Hettinger.)

\end{itemize}

The net result of the 2.5 optimizations is that Python 2.5 runs the
pystone benchmark around XX\% faster than Python 2.4.


%======================================================================
\section{New, Improved, and Deprecated Modules}

As usual, Python's standard library received a number of enhancements and
bug fixes.  Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the
\file{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the CVS logs for all the
details.

\begin{itemize}

% collections.deque now has .remove()

% the cPickle module no longer accepts the deprecated None option in the
% args tuple returned by __reduce__().

% csv module improvements

% datetime.datetime() now has a strptime class method which can be used to
% create datetime object using a string and format.

\item A new \module{hashlib} module has been added to replace the
\module{md5} and \module{sha} modules.  \module{hashlib} adds support
for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512).
When available, the module uses OpenSSL for fast platform optimized
implementations of algorithms.  The old \module{md5} and \module{sha}
modules still exist as wrappers around hashlib to preserve backwards
compatibility.  (Contributed by Gregory P. Smith.)

\item The \function{nsmallest()} and 
\function{nlargest()} functions in the \module{heapq} module 
now support a \code{key} keyword argument similar to the one
provided by the \function{min()}/\function{max()} functions
and the \method{sort()} methods.  For example:
Example:

\begin{verbatim}
>>> import heapq
>>> L = ["short", 'medium', 'longest', 'longer still']
>>> heapq.nsmallest(2, L)  # Return two lowest elements, lexicographically
['longer still', 'longest']
>>> heapq.nsmallest(2, L, key=len)   # Return two shortest elements
['short', 'medium']
\end{verbatim}

(Contributed by Raymond Hettinger.)

\item The \function{itertools.islice()} function now accepts
\code{None} for the start and step arguments.  This makes it more
compatible with the attributes of slice objects, so that you can now write
the following:

\begin{verbatim}
s = slice(5)     # Create slice object
itertools.islice(iterable, s.start, s.stop, s.step)
\end{verbatim}

(Contributed by Raymond Hettinger.)

\item The \module{operator} module's \function{itemgetter()} 
and \function{attrgetter()} functions now support multiple fields.  
A call such as \code{operator.attrgetter('a', 'b')}
will return a function 
that retrieves the \member{a} and \member{b} attributes.  Combining 
this new feature with the \method{sort()} method's \code{key} parameter 
lets you easily sort lists using multiple fields.

% XXX who added?


\item The \module{os} module underwent a number of changes.  The
\member{stat_float_times} variable now defaults to true, meaning that
\function{os.stat()} will now return time values as floats.  (This
doesn't necessarily mean that \function{os.stat()} will return times
that are precise to fractions of a second; not all systems support
such precision.)

Constants named \member{os.SEEK_SET}, \member{os.SEEK_CUR}, and
\member{os.SEEK_END} have been added; these are the parameters to the
\function{os.lseek()} function.  Two new constants for locking are
\member{os.O_SHLOCK} and \member{os.O_EXLOCK}.

On FreeBSD, the \function{os.stat()} function now returns 
times with nanosecond resolution, and the returned object
now has \member{st_gen} and \member{st_birthtime}.
The \member{st_flags} member is also available, if the platform supports it.
% XXX patch 1180695, 1212117

\item The \module{socket} module now supports \constant{AF_NETLINK}
sockets on Linux, thanks to a patch from Philippe Biondi.  
Netlink sockets are a Linux-specific mechanism for communications
between a user-space process and kernel code; an introductory 
article about them is at \url{http://www.linuxjournal.com/article/7356}.
In Python code, netlink addresses are represented as a tuple of 2 integers, 
\code{(\var{pid}, \var{group_mask})}.

\item New module: \module{spwd} provides functions for accessing the
shadow password database on systems that support it.  
% XXX give example

\item The \class{TarFile} class in the \module{tarfile} module now has
an \method{extractall()} method that extracts all members from the
archive into the current working directory.  It's also possible to set
a different directory as the extraction target, and to unpack only a
subset of the archive's members.  

A tarfile's compression can be autodetected by 
using the mode \code{'r|*'}.
% patch 918101
(Contributed by Lars Gust\"abel.)

\item A new package \module{xml.etree} has been added, which contains
a subset of the ElementTree XML library.  Available modules are
\module{ElementTree}, \module{ElementPath}, and
\module{ElementInclude}, from ElementTree 1.2.6. (Contributed by
Fredrik Lundh.)

\item The \module{xmlrpclib} module now supports returning 
      \class{datetime} objects for the XML-RPC date type.  Supply 
      \code{use_datetime=True} to the \function{loads()} function
      or the \class{Unmarshaller} class to enable this feature.
% XXX patch 1120353


\end{itemize}



%======================================================================
% whole new modules get described in \subsections here

% XXX new distutils features: upload

% XXX should hashlib perhaps be described here instead?
% XXX should xml.etree perhaps be described here instead?



% ======================================================================
\section{Build and C API Changes}

Changes to Python's build process and to the C API include:

\begin{itemize}

\item The design of the bytecode compiler has changed a great deal, no
longer generating bytecode by traversing the parse tree.  Instead
the parse tree is converted to an abstract syntax tree (or AST), and it is 
the abstract syntax tree that's traversed to produce the bytecode.

No documentation has been written for the AST code yet.  To start
learning about it, read the definition of the various AST nodes in
\file{Parser/Python.asdl}.  A Python script reads this file and
generates a set of C structure definitions in
\file{Include/Python-ast.h}.  The \cfunction{PyParser_ASTFromString()}
and \cfunction{PyParser_ASTFromFile()}, defined in
\file{Include/pythonrun.h}, take Python source as input and return the
root of an AST representing the contents.  This AST can then be turned
into a code object by \cfunction{PyAST_Compile()}.  For more
information, read the source code, and then ask questions on
python-dev.

% List of names taken from Jeremy's python-dev post at 
% http://mail.python.org/pipermail/python-dev/2005-October/057500.html
The AST code was developed under Jeremy Hylton's management, and
implemented by (in alphabetical order) Brett Cannon, Nick Coghlan,
Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters,
Armin Rigo, and Neil Schemenauer, plus the participants in a number of
AST sprints at conferences such as PyCon.
 
\item The built-in set types now have an official C API.  Call
\cfunction{PySet_New()} and \cfunction{PyFrozenSet_New()} to create a
new set, \cfunction{PySet_Add()} and \cfunction{PySet_Discard()} to
add and remove elements, and \cfunction{PySet_Contains} and
\cfunction{PySet_Size} to examine the set's state.

\item The \cfunction{PyRange_New()} function was removed.  It was
never documented, never used in the core code, and had dangerously lax
error checking.

\end{itemize}


%======================================================================
\subsection{Port-Specific Changes}

Platform-specific changes go here.


%======================================================================
\section{Other Changes and Fixes \label{section-other}}

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the CVS change
logs finds there were XXX patches applied and YYY bugs fixed between
Python 2.4 and 2.5.  Both figures are likely to be underestimates.

Some of the more notable changes are:

\begin{itemize}

\item Details go here.

\end{itemize}


%======================================================================
\section{Porting to Python 2.5}

This section lists previously described changes that may require
changes to your code:

\begin{itemize}

\item Some old deprecated modules (\module{statcache}, \module{tzparse},
      \module{whrandom})  have been moved to \file{Lib/lib-old}.
You can get access to these modules  again by adding the directory 
to your \code{sys.path}:

\begin{verbatim}
import os
from distutils import sysconfig

lib_dir = sysconfig.get_python_lib(standard_lib=True)
old_dir = os.path.join(lib_dir, 'lib-old')
sys.path.append(old_dir)
\end{verbatim}

Doing so is discouraged, however; it's better to update any code that
still uses these modules.

% the pickle module no longer uses the deprecated bin parameter.

\end{itemize}


%======================================================================
\section{Acknowledgements \label{acks}}

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: .

\end{document}