1 :mod:`urllib2` --- extensible library for opening URLs
2 ======================================================
5 :synopsis: Next generation URL opening library.
6 .. moduleauthor:: Jeremy Hylton <jhylton@users.sourceforge.net>
7 .. sectionauthor:: Moshe Zadka <moshez@users.sourceforge.net>
11 The :mod:`urllib2` module has been split across several modules in
12 Python 3.0 named :mod:`urllib.request` and :mod:`urllib.error`.
13 The :term:`2to3` tool will automatically adapt imports when converting
17 The :mod:`urllib2` module defines functions and classes which help in opening
18 URLs (mostly HTTP) in a complex world --- basic and digest authentication,
19 redirections, cookies and more.
21 The :mod:`urllib2` module defines the following functions:
24 .. function:: urlopen(url[, data][, timeout])
26 Open the URL *url*, which can be either a string or a :class:`Request` object.
28 *data* may be a string specifying additional data to send to the server, or
29 ``None`` if no such data is needed. Currently HTTP requests are the only ones
30 that use *data*; the HTTP request will be a POST instead of a GET when the
31 *data* parameter is provided. *data* should be a buffer in the standard
32 :mimetype:`application/x-www-form-urlencoded` format. The
33 :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and
34 returns a string in this format.
36 The optional *timeout* parameter specifies a timeout in seconds for blocking
37 operations like the connection attempt (if not specified, the global default
38 timeout setting will be used). This actually only works for HTTP, HTTPS,
39 FTP and FTPS connections.
41 This function returns a file-like object with two additional methods:
43 * :meth:`geturl` --- return the URL of the resource retrieved, commonly used to
44 determine if a redirect was followed
46 * :meth:`info` --- return the meta-information of the page, such as headers, in
47 the form of an ``httplib.HTTPMessage`` instance
48 (see `Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_)
50 Raises :exc:`URLError` on errors.
52 Note that ``None`` may be returned if no handler handles the request (though the
53 default installed global :class:`OpenerDirector` uses :class:`UnknownHandler` to
54 ensure this never happens).
56 .. versionchanged:: 2.6
60 .. function:: install_opener(opener)
62 Install an :class:`OpenerDirector` instance as the default global opener.
63 Installing an opener is only necessary if you want urlopen to use that opener;
64 otherwise, simply call :meth:`OpenerDirector.open` instead of :func:`urlopen`.
65 The code does not check for a real :class:`OpenerDirector`, and any class with
66 the appropriate interface will work.
69 .. function:: build_opener([handler, ...])
71 Return an :class:`OpenerDirector` instance, which chains the handlers in the
72 order given. *handler*\s can be either instances of :class:`BaseHandler`, or
73 subclasses of :class:`BaseHandler` (in which case it must be possible to call
74 the constructor without any parameters). Instances of the following classes
75 will be in front of the *handler*\s, unless the *handler*\s contain them,
76 instances of them or subclasses of them: :class:`ProxyHandler`,
77 :class:`UnknownHandler`, :class:`HTTPHandler`, :class:`HTTPDefaultErrorHandler`,
78 :class:`HTTPRedirectHandler`, :class:`FTPHandler`, :class:`FileHandler`,
79 :class:`HTTPErrorProcessor`.
81 If the Python installation has SSL support (i.e., if the :mod:`ssl` module can be imported),
82 :class:`HTTPSHandler` will also be added.
84 Beginning in Python 2.3, a :class:`BaseHandler` subclass may also change its
85 :attr:`handler_order` member variable to modify its position in the handlers
88 The following exceptions are raised as appropriate:
91 .. exception:: URLError
93 The handlers raise this exception (or derived exceptions) when they run into a
94 problem. It is a subclass of :exc:`IOError`.
98 The reason for this error. It can be a message string or another exception
99 instance (:exc:`socket.error` for remote URLs, :exc:`OSError` for local
103 .. exception:: HTTPError
105 Though being an exception (a subclass of :exc:`URLError`), an :exc:`HTTPError`
106 can also function as a non-exceptional file-like return value (the same thing
107 that :func:`urlopen` returns). This is useful when handling exotic HTTP
108 errors, such as requests for authentication.
112 An HTTP status code as defined in `RFC 2616 <http://www.faqs.org/rfcs/rfc2616.html>`_.
113 This numeric value corresponds to a value found in the dictionary of
114 codes as found in :attr:`BaseHTTPServer.BaseHTTPRequestHandler.responses`.
118 The following classes are provided:
121 .. class:: Request(url[, data][, headers][, origin_req_host][, unverifiable])
123 This class is an abstraction of a URL request.
125 *url* should be a string containing a valid URL.
127 *data* may be a string specifying additional data to send to the server, or
128 ``None`` if no such data is needed. Currently HTTP requests are the only ones
129 that use *data*; the HTTP request will be a POST instead of a GET when the
130 *data* parameter is provided. *data* should be a buffer in the standard
131 :mimetype:`application/x-www-form-urlencoded` format. The
132 :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and
133 returns a string in this format.
135 *headers* should be a dictionary, and will be treated as if :meth:`add_header`
136 was called with each key and value as arguments. This is often used to "spoof"
137 the ``User-Agent`` header, which is used by a browser to identify itself --
138 some HTTP servers only allow requests coming from common browsers as opposed
139 to scripts. For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0
140 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib2`'s
141 default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6).
143 The final two arguments are only of interest for correct handling of third-party
146 *origin_req_host* should be the request-host of the origin transaction, as
147 defined by :rfc:`2965`. It defaults to ``cookielib.request_host(self)``. This
148 is the host name or IP address of the original request that was initiated by the
149 user. For example, if the request is for an image in an HTML document, this
150 should be the request-host of the request for the page containing the image.
152 *unverifiable* should indicate whether the request is unverifiable, as defined
153 by RFC 2965. It defaults to False. An unverifiable request is one whose URL
154 the user did not have the option to approve. For example, if the request is for
155 an image in an HTML document, and the user had no option to approve the
156 automatic fetching of the image, this should be true.
159 .. class:: OpenerDirector()
161 The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained
162 together. It manages the chaining of handlers, and recovery from errors.
165 .. class:: BaseHandler()
167 This is the base class for all registered handlers --- and handles only the
168 simple mechanics of registration.
171 .. class:: HTTPDefaultErrorHandler()
173 A class which defines a default handler for HTTP error responses; all responses
174 are turned into :exc:`HTTPError` exceptions.
177 .. class:: HTTPRedirectHandler()
179 A class to handle redirections.
182 .. class:: HTTPCookieProcessor([cookiejar])
184 A class to handle HTTP Cookies.
187 .. class:: ProxyHandler([proxies])
189 Cause requests to go through a proxy. If *proxies* is given, it must be a
190 dictionary mapping protocol names to URLs of proxies. The default is to read the
191 list of proxies from the environment variables :envvar:`<protocol>_proxy`.
192 To disable autodetected proxy pass an empty dictionary.
195 .. class:: HTTPPasswordMgr()
197 Keep a database of ``(realm, uri) -> (user, password)`` mappings.
200 .. class:: HTTPPasswordMgrWithDefaultRealm()
202 Keep a database of ``(realm, uri) -> (user, password)`` mappings. A realm of
203 ``None`` is considered a catch-all realm, which is searched if no other realm
207 .. class:: AbstractBasicAuthHandler([password_mgr])
209 This is a mixin class that helps with HTTP authentication, both to the remote
210 host and to a proxy. *password_mgr*, if given, should be something that is
211 compatible with :class:`HTTPPasswordMgr`; refer to section
212 :ref:`http-password-mgr` for information on the interface that must be
216 .. class:: HTTPBasicAuthHandler([password_mgr])
218 Handle authentication with the remote host. *password_mgr*, if given, should be
219 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
220 :ref:`http-password-mgr` for information on the interface that must be
224 .. class:: ProxyBasicAuthHandler([password_mgr])
226 Handle authentication with the proxy. *password_mgr*, if given, should be
227 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
228 :ref:`http-password-mgr` for information on the interface that must be
232 .. class:: AbstractDigestAuthHandler([password_mgr])
234 This is a mixin class that helps with HTTP authentication, both to the remote
235 host and to a proxy. *password_mgr*, if given, should be something that is
236 compatible with :class:`HTTPPasswordMgr`; refer to section
237 :ref:`http-password-mgr` for information on the interface that must be
241 .. class:: HTTPDigestAuthHandler([password_mgr])
243 Handle authentication with the remote host. *password_mgr*, if given, should be
244 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
245 :ref:`http-password-mgr` for information on the interface that must be
249 .. class:: ProxyDigestAuthHandler([password_mgr])
251 Handle authentication with the proxy. *password_mgr*, if given, should be
252 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
253 :ref:`http-password-mgr` for information on the interface that must be
257 .. class:: HTTPHandler()
259 A class to handle opening of HTTP URLs.
262 .. class:: HTTPSHandler()
264 A class to handle opening of HTTPS URLs.
267 .. class:: FileHandler()
272 .. class:: FTPHandler()
277 .. class:: CacheFTPHandler()
279 Open FTP URLs, keeping a cache of open FTP connections to minimize delays.
282 .. class:: UnknownHandler()
284 A catch-all class to handle unknown URLs.
292 The following methods describe all of :class:`Request`'s public interface, and
293 so all must be overridden in subclasses.
296 .. method:: Request.add_data(data)
298 Set the :class:`Request` data to *data*. This is ignored by all handlers except
299 HTTP handlers --- and there it should be a byte string, and will change the
300 request to be ``POST`` rather than ``GET``.
303 .. method:: Request.get_method()
305 Return a string indicating the HTTP request method. This is only meaningful for
306 HTTP requests, and currently always returns ``'GET'`` or ``'POST'``.
309 .. method:: Request.has_data()
311 Return whether the instance has a non-\ ``None`` data.
314 .. method:: Request.get_data()
316 Return the instance's data.
319 .. method:: Request.add_header(key, val)
321 Add another header to the request. Headers are currently ignored by all
322 handlers except HTTP handlers, where they are added to the list of headers sent
323 to the server. Note that there cannot be more than one header with the same
324 name, and later calls will overwrite previous calls in case the *key* collides.
325 Currently, this is no loss of HTTP functionality, since all headers which have
326 meaning when used more than once have a (header-specific) way of gaining the
327 same functionality using only one header.
330 .. method:: Request.add_unredirected_header(key, header)
332 Add a header that will not be added to a redirected request.
334 .. versionadded:: 2.4
337 .. method:: Request.has_header(header)
339 Return whether the instance has the named header (checks both regular and
342 .. versionadded:: 2.4
345 .. method:: Request.get_full_url()
347 Return the URL given in the constructor.
350 .. method:: Request.get_type()
352 Return the type of the URL --- also known as the scheme.
355 .. method:: Request.get_host()
357 Return the host to which a connection will be made.
360 .. method:: Request.get_selector()
362 Return the selector --- the part of the URL that is sent to the server.
365 .. method:: Request.set_proxy(host, type)
367 Prepare the request by connecting to a proxy server. The *host* and *type* will
368 replace those of the instance, and the instance's selector will be the original
369 URL given in the constructor.
372 .. method:: Request.get_origin_req_host()
374 Return the request-host of the origin transaction, as defined by :rfc:`2965`.
375 See the documentation for the :class:`Request` constructor.
378 .. method:: Request.is_unverifiable()
380 Return whether the request is unverifiable, as defined by RFC 2965. See the
381 documentation for the :class:`Request` constructor.
384 .. _opener-director-objects:
386 OpenerDirector Objects
387 ----------------------
389 :class:`OpenerDirector` instances have the following methods:
392 .. method:: OpenerDirector.add_handler(handler)
394 *handler* should be an instance of :class:`BaseHandler`. The following
395 methods are searched, and added to the possible chains (note that HTTP errors
398 * :samp:`{protocol}_open` --- signal that the handler knows how to open
401 * :samp:`http_error_{type}` --- signal that the handler knows how to handle
402 HTTP errors with HTTP error code *type*.
404 * :samp:`{protocol}_error` --- signal that the handler knows how to handle
405 errors from (non-\ ``http``) *protocol*.
407 * :samp:`{protocol}_request` --- signal that the handler knows how to
408 pre-process *protocol* requests.
410 * :samp:`{protocol}_response` --- signal that the handler knows how to
411 post-process *protocol* responses.
414 .. method:: OpenerDirector.open(url[, data][, timeout])
416 Open the given *url* (which can be a request object or a string), optionally
417 passing the given *data*. Arguments, return values and exceptions raised are
418 the same as those of :func:`urlopen` (which simply calls the :meth:`open`
419 method on the currently installed global :class:`OpenerDirector`). The
420 optional *timeout* parameter specifies a timeout in seconds for blocking
421 operations like the connection attempt (if not specified, the global default
422 timeout setting will be usedi). The timeout feature actually works only for
423 HTTP, HTTPS, FTP and FTPS connections).
425 .. versionchanged:: 2.6
429 .. method:: OpenerDirector.error(proto[, arg[, ...]])
431 Handle an error of the given protocol. This will call the registered error
432 handlers for the given protocol with the given arguments (which are protocol
433 specific). The HTTP protocol is a special case which uses the HTTP response
434 code to determine the specific error handler; refer to the :meth:`http_error_\*`
435 methods of the handler classes.
437 Return values and exceptions raised are the same as those of :func:`urlopen`.
439 OpenerDirector objects open URLs in three stages:
441 The order in which these methods are called within each stage is determined by
442 sorting the handler instances.
444 #. Every handler with a method named like :samp:`{protocol}_request` has that
445 method called to pre-process the request.
447 #. Handlers with a method named like :samp:`{protocol}_open` are called to handle
448 the request. This stage ends when a handler either returns a non-\ :const:`None`
449 value (ie. a response), or raises an exception (usually :exc:`URLError`).
450 Exceptions are allowed to propagate.
452 In fact, the above algorithm is first tried for methods named
453 :meth:`default_open`. If all such methods return :const:`None`, the
454 algorithm is repeated for methods named like :samp:`{protocol}_open`. If all
455 such methods return :const:`None`, the algorithm is repeated for methods
456 named :meth:`unknown_open`.
458 Note that the implementation of these methods may involve calls of the parent
459 :class:`OpenerDirector` instance's :meth:`.open` and :meth:`.error` methods.
461 #. Every handler with a method named like :samp:`{protocol}_response` has that
462 method called to post-process the response.
465 .. _base-handler-objects:
470 :class:`BaseHandler` objects provide a couple of methods that are directly
471 useful, and others that are meant to be used by derived classes. These are
472 intended for direct use:
475 .. method:: BaseHandler.add_parent(director)
477 Add a director as parent.
480 .. method:: BaseHandler.close()
484 The following members and methods should only be used by classes derived from
485 :class:`BaseHandler`.
489 The convention has been adopted that subclasses defining
490 :meth:`protocol_request` or :meth:`protocol_response` methods are named
491 :class:`\*Processor`; all others are named :class:`\*Handler`.
494 .. attribute:: BaseHandler.parent
496 A valid :class:`OpenerDirector`, which can be used to open using a different
497 protocol, or handle errors.
500 .. method:: BaseHandler.default_open(req)
502 This method is *not* defined in :class:`BaseHandler`, but subclasses should
503 define it if they want to catch all URLs.
505 This method, if implemented, will be called by the parent
506 :class:`OpenerDirector`. It should return a file-like object as described in
507 the return value of the :meth:`open` of :class:`OpenerDirector`, or ``None``.
508 It should raise :exc:`URLError`, unless a truly exceptional thing happens (for
509 example, :exc:`MemoryError` should not be mapped to :exc:`URLError`).
511 This method will be called before any protocol-specific open method.
514 .. method:: BaseHandler.protocol_open(req)
517 ("protocol" is to be replaced by the protocol name.)
519 This method is *not* defined in :class:`BaseHandler`, but subclasses should
520 define it if they want to handle URLs with the given *protocol*.
522 This method, if defined, will be called by the parent :class:`OpenerDirector`.
523 Return values should be the same as for :meth:`default_open`.
526 .. method:: BaseHandler.unknown_open(req)
528 This method is *not* defined in :class:`BaseHandler`, but subclasses should
529 define it if they want to catch all URLs with no specific registered handler to
532 This method, if implemented, will be called by the :attr:`parent`
533 :class:`OpenerDirector`. Return values should be the same as for
534 :meth:`default_open`.
537 .. method:: BaseHandler.http_error_default(req, fp, code, msg, hdrs)
539 This method is *not* defined in :class:`BaseHandler`, but subclasses should
540 override it if they intend to provide a catch-all for otherwise unhandled HTTP
541 errors. It will be called automatically by the :class:`OpenerDirector` getting
542 the error, and should not normally be called in other circumstances.
544 *req* will be a :class:`Request` object, *fp* will be a file-like object with
545 the HTTP error body, *code* will be the three-digit code of the error, *msg*
546 will be the user-visible explanation of the code and *hdrs* will be a mapping
547 object with the headers of the error.
549 Return values and exceptions raised should be the same as those of
553 .. method:: BaseHandler.http_error_nnn(req, fp, code, msg, hdrs)
555 *nnn* should be a three-digit HTTP error code. This method is also not defined
556 in :class:`BaseHandler`, but will be called, if it exists, on an instance of a
557 subclass, when an HTTP error with code *nnn* occurs.
559 Subclasses should override this method to handle specific HTTP errors.
561 Arguments, return values and exceptions raised should be the same as for
562 :meth:`http_error_default`.
565 .. method:: BaseHandler.protocol_request(req)
568 ("protocol" is to be replaced by the protocol name.)
570 This method is *not* defined in :class:`BaseHandler`, but subclasses should
571 define it if they want to pre-process requests of the given *protocol*.
573 This method, if defined, will be called by the parent :class:`OpenerDirector`.
574 *req* will be a :class:`Request` object. The return value should be a
575 :class:`Request` object.
578 .. method:: BaseHandler.protocol_response(req, response)
581 ("protocol" is to be replaced by the protocol name.)
583 This method is *not* defined in :class:`BaseHandler`, but subclasses should
584 define it if they want to post-process responses of the given *protocol*.
586 This method, if defined, will be called by the parent :class:`OpenerDirector`.
587 *req* will be a :class:`Request` object. *response* will be an object
588 implementing the same interface as the return value of :func:`urlopen`. The
589 return value should implement the same interface as the return value of
593 .. _http-redirect-handler:
595 HTTPRedirectHandler Objects
596 ---------------------------
600 Some HTTP redirections require action from this module's client code. If this
601 is the case, :exc:`HTTPError` is raised. See :rfc:`2616` for details of the
602 precise meanings of the various redirection codes.
605 .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs)
607 Return a :class:`Request` or ``None`` in response to a redirect. This is called
608 by the default implementations of the :meth:`http_error_30\*` methods when a
609 redirection is received from the server. If a redirection should take place,
610 return a new :class:`Request` to allow :meth:`http_error_30\*` to perform the
611 redirect. Otherwise, raise :exc:`HTTPError` if no other handler should try to
612 handle this URL, or return ``None`` if you can't but another handler might.
616 The default implementation of this method does not strictly follow :rfc:`2616`,
617 which says that 301 and 302 responses to ``POST`` requests must not be
618 automatically redirected without confirmation by the user. In reality, browsers
619 do allow automatic redirection of these responses, changing the POST to a
620 ``GET``, and the default implementation reproduces this behavior.
623 .. method:: HTTPRedirectHandler.http_error_301(req, fp, code, msg, hdrs)
625 Redirect to the ``Location:`` URL. This method is called by the parent
626 :class:`OpenerDirector` when getting an HTTP 'moved permanently' response.
629 .. method:: HTTPRedirectHandler.http_error_302(req, fp, code, msg, hdrs)
631 The same as :meth:`http_error_301`, but called for the 'found' response.
634 .. method:: HTTPRedirectHandler.http_error_303(req, fp, code, msg, hdrs)
636 The same as :meth:`http_error_301`, but called for the 'see other' response.
639 .. method:: HTTPRedirectHandler.http_error_307(req, fp, code, msg, hdrs)
641 The same as :meth:`http_error_301`, but called for the 'temporary redirect'
645 .. _http-cookie-processor:
647 HTTPCookieProcessor Objects
648 ---------------------------
650 .. versionadded:: 2.4
652 :class:`HTTPCookieProcessor` instances have one attribute:
655 .. attribute:: HTTPCookieProcessor.cookiejar
657 The :class:`cookielib.CookieJar` in which cookies are stored.
666 .. method:: ProxyHandler.protocol_open(request)
669 ("protocol" is to be replaced by the protocol name.)
671 The :class:`ProxyHandler` will have a method :samp:`{protocol}_open` for every
672 *protocol* which has a proxy in the *proxies* dictionary given in the
673 constructor. The method will modify requests to go through the proxy, by
674 calling ``request.set_proxy()``, and call the next handler in the chain to
675 actually execute the protocol.
678 .. _http-password-mgr:
680 HTTPPasswordMgr Objects
681 -----------------------
683 These methods are available on :class:`HTTPPasswordMgr` and
684 :class:`HTTPPasswordMgrWithDefaultRealm` objects.
687 .. method:: HTTPPasswordMgr.add_password(realm, uri, user, passwd)
689 *uri* can be either a single URI, or a sequence of URIs. *realm*, *user* and
690 *passwd* must be strings. This causes ``(user, passwd)`` to be used as
691 authentication tokens when authentication for *realm* and a super-URI of any of
692 the given URIs is given.
695 .. method:: HTTPPasswordMgr.find_user_password(realm, authuri)
697 Get user/password for given realm and URI, if any. This method will return
698 ``(None, None)`` if there is no matching user/password.
700 For :class:`HTTPPasswordMgrWithDefaultRealm` objects, the realm ``None`` will be
701 searched if the given *realm* has no matching user/password.
704 .. _abstract-basic-auth-handler:
706 AbstractBasicAuthHandler Objects
707 --------------------------------
710 .. method:: AbstractBasicAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
712 Handle an authentication request by getting a user/password pair, and re-trying
713 the request. *authreq* should be the name of the header where the information
714 about the realm is included in the request, *host* specifies the URL and path to
715 authenticate for, *req* should be the (failed) :class:`Request` object, and
716 *headers* should be the error headers.
718 *host* is either an authority (e.g. ``"python.org"``) or a URL containing an
719 authority component (e.g. ``"http://python.org/"``). In either case, the
720 authority must not contain a userinfo component (so, ``"python.org"`` and
721 ``"python.org:80"`` are fine, ``"joe:password@python.org"`` is not).
724 .. _http-basic-auth-handler:
726 HTTPBasicAuthHandler Objects
727 ----------------------------
730 .. method:: HTTPBasicAuthHandler.http_error_401(req, fp, code, msg, hdrs)
732 Retry the request with authentication information, if available.
735 .. _proxy-basic-auth-handler:
737 ProxyBasicAuthHandler Objects
738 -----------------------------
741 .. method:: ProxyBasicAuthHandler.http_error_407(req, fp, code, msg, hdrs)
743 Retry the request with authentication information, if available.
746 .. _abstract-digest-auth-handler:
748 AbstractDigestAuthHandler Objects
749 ---------------------------------
752 .. method:: AbstractDigestAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
754 *authreq* should be the name of the header where the information about the realm
755 is included in the request, *host* should be the host to authenticate to, *req*
756 should be the (failed) :class:`Request` object, and *headers* should be the
760 .. _http-digest-auth-handler:
762 HTTPDigestAuthHandler Objects
763 -----------------------------
766 .. method:: HTTPDigestAuthHandler.http_error_401(req, fp, code, msg, hdrs)
768 Retry the request with authentication information, if available.
771 .. _proxy-digest-auth-handler:
773 ProxyDigestAuthHandler Objects
774 ------------------------------
777 .. method:: ProxyDigestAuthHandler.http_error_407(req, fp, code, msg, hdrs)
779 Retry the request with authentication information, if available.
782 .. _http-handler-objects:
788 .. method:: HTTPHandler.http_open(req)
790 Send an HTTP request, which can be either GET or POST, depending on
794 .. _https-handler-objects:
800 .. method:: HTTPSHandler.https_open(req)
802 Send an HTTPS request, which can be either GET or POST, depending on
806 .. _file-handler-objects:
812 .. method:: FileHandler.file_open(req)
814 Open the file locally, if there is no host name, or the host name is
815 ``'localhost'``. Change the protocol to ``ftp`` otherwise, and retry opening it
816 using :attr:`parent`.
819 .. _ftp-handler-objects:
825 .. method:: FTPHandler.ftp_open(req)
827 Open the FTP file indicated by *req*. The login is always done with empty
828 username and password.
831 .. _cacheftp-handler-objects:
833 CacheFTPHandler Objects
834 -----------------------
836 :class:`CacheFTPHandler` objects are :class:`FTPHandler` objects with the
837 following additional methods:
840 .. method:: CacheFTPHandler.setTimeout(t)
842 Set timeout of connections to *t* seconds.
845 .. method:: CacheFTPHandler.setMaxConns(m)
847 Set maximum number of cached connections to *m*.
850 .. _unknown-handler-objects:
852 UnknownHandler Objects
853 ----------------------
856 .. method:: UnknownHandler.unknown_open()
858 Raise a :exc:`URLError` exception.
861 .. _http-error-processor-objects:
863 HTTPErrorProcessor Objects
864 --------------------------
866 .. versionadded:: 2.4
869 .. method:: HTTPErrorProcessor.unknown_open()
871 Process HTTP error responses.
873 For 200 error codes, the response object is returned immediately.
875 For non-200 error codes, this simply passes the job on to the
876 :samp:`{protocol}_error_code` handler methods, via
877 :meth:`OpenerDirector.error`. Eventually,
878 :class:`urllib2.HTTPDefaultErrorHandler` will raise an :exc:`HTTPError` if no
879 other handler handles the error.
882 .. _urllib2-examples:
887 This example gets the python.org main page and displays the first 100 bytes of
891 >>> f = urllib2.urlopen('http://www.python.org/')
892 >>> print f.read(100)
893 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
894 <?xml-stylesheet href="./css/ht2html
896 Here we are sending a data-stream to the stdin of a CGI and reading the data it
897 returns to us. Note that this example will only work when the Python
898 installation supports SSL. ::
901 >>> req = urllib2.Request(url='https://localhost/cgi-bin/test.cgi',
902 ... data='This data is passed to stdin of the CGI')
903 >>> f = urllib2.urlopen(req)
905 Got Data: "This data is passed to stdin of the CGI"
907 The code for the sample CGI used in the above example is::
909 #!/usr/bin/env python
911 data = sys.stdin.read()
912 print 'Content-type: text-plain\n\nGot Data: "%s"' % data
914 Use of Basic HTTP Authentication::
917 # Create an OpenerDirector with support for Basic HTTP Authentication...
918 auth_handler = urllib2.HTTPBasicAuthHandler()
919 auth_handler.add_password(realm='PDQ Application',
920 uri='https://mahler:8092/site-updates.py',
922 passwd='kadidd!ehopper')
923 opener = urllib2.build_opener(auth_handler)
924 # ...and install it globally so it can be used with urlopen.
925 urllib2.install_opener(opener)
926 urllib2.urlopen('http://www.example.com/login.html')
928 :func:`build_opener` provides many handlers by default, including a
929 :class:`ProxyHandler`. By default, :class:`ProxyHandler` uses the environment
930 variables named ``<scheme>_proxy``, where ``<scheme>`` is the URL scheme
931 involved. For example, the :envvar:`http_proxy` environment variable is read to
932 obtain the HTTP proxy's URL.
934 This example replaces the default :class:`ProxyHandler` with one that uses
935 programmatically-supplied proxy URLs, and adds proxy authorization support with
936 :class:`ProxyBasicAuthHandler`. ::
938 proxy_handler = urllib2.ProxyHandler({'http': 'http://www.example.com:3128/'})
939 proxy_auth_handler = urllib2.HTTPBasicAuthHandler()
940 proxy_auth_handler.add_password('realm', 'host', 'username', 'password')
942 opener = build_opener(proxy_handler, proxy_auth_handler)
943 # This time, rather than install the OpenerDirector, we use it directly:
944 opener.open('http://www.example.com/login.html')
948 Use the *headers* argument to the :class:`Request` constructor, or::
951 req = urllib2.Request('http://www.example.com/')
952 req.add_header('Referer', 'http://www.python.org/')
953 r = urllib2.urlopen(req)
955 :class:`OpenerDirector` automatically adds a :mailheader:`User-Agent` header to
956 every :class:`Request`. To change this::
959 opener = urllib2.build_opener()
960 opener.addheaders = [('User-agent', 'Mozilla/5.0')]
961 opener.open('http://www.example.com/')
963 Also, remember that a few standard headers (:mailheader:`Content-Length`,
964 :mailheader:`Content-Type` and :mailheader:`Host`) are added when the
965 :class:`Request` is passed to :func:`urlopen` (or :meth:`OpenerDirector.open`).