Doc/lib/libregex.tex

   1 \section{\module{regex} ---
   2          Regular expression operations}
   3 \declaremodule{builtin}{regex}
   4
   5 \modulesynopsis{Regular expression search and match operations.
   6                 \strong{Obsolete!}}
   7
   8
   9 This module provides regular expression matching operations similar to
  10 those found in Emacs.
  11
  12 \strong{Obsolescence note:}
  13 This module is obsolete as of Python version 1.5; it is still being
  14 maintained because much existing code still uses it.  All new code in
  15 need of regular expressions should use the new
  16 \code{re}\refstmodindex{re} module, which supports the more powerful
  17 and regular Perl-style regular expressions.  Existing code should be
  18 converted.  The standard library module
  19 \code{reconvert}\refstmodindex{reconvert} helps in converting
  20 \code{regex} style regular expressions to \code{re}\refstmodindex{re}
  21 style regular expressions.  (For more conversion help, see Andrew
  22 Kuchling's\index{Kuchling, Andrew} ``\module{regex-to-re} HOWTO'' at
  23 \url{http://www.python.org/doc/howto/regex-to-re/}.)
  24
  25 By default the patterns are Emacs-style regular expressions
  26 (with one exception).  There is
  27 a way to change the syntax to match that of several well-known
  28 \UNIX{} utilities.  The exception is that Emacs' \samp{\e s}
  29 pattern is not supported, since the original implementation references
  30 the Emacs syntax tables.
  31
  32 This module is 8-bit clean: both patterns and strings may contain null
  33 bytes and characters whose high bit is set.
  34
  35 \strong{Please note:} There is a little-known fact about Python string
  36 literals which means that you don't usually have to worry about
  37 doubling backslashes, even though they are used to escape special
  38 characters in string literals as well as in regular expressions.  This
  39 is because Python doesn't remove backslashes from string literals if
  40 they are followed by an unrecognized escape character.
  41 \emph{However}, if you want to include a literal \dfn{backslash} in a
  42 regular expression represented as a string literal, you have to
  43 \emph{quadruple} it or enclose it in a singleton character class.
  44 E.g.\  to extract \LaTeX\ \samp{\e section\{\textrm{\ldots}\}} headers
  45 from a document, you can use this pattern:
  46 \code{'[\e ]section\{\e (.*\e )\}'}.  \emph{Another exception:}
  47 the escape sequence \samp{\e b} is significant in string literals
  48 (where it means the ASCII bell character) as well as in Emacs regular
  49 expressions (where it stands for a word boundary), so in order to
  50 search for a word boundary, you should use the pattern \code{'\e \e b'}.
  51 Similarly, a backslash followed by a digit 0-7 should be doubled to
  52 avoid interpretation as an octal escape.
  53
  54 \subsection{Regular Expressions}
  55
  56 A regular expression (or RE) specifies a set of strings that matches
  57 it; the functions in this module let you check if a particular string
  58 matches a given regular expression (or if a given regular expression
  59 matches a particular string, which comes down to the same thing).
  60
  61 Regular expressions can be concatenated to form new regular
  62 expressions; if \emph{A} and \emph{B} are both regular expressions,
  63 then \emph{AB} is also an regular expression.  If a string \emph{p}
  64 matches A and another string \emph{q} matches B, the string \emph{pq}
  65 will match AB.  Thus, complex expressions can easily be constructed
  66 from simpler ones like the primitives described here.  For details of
  67 the theory and implementation of regular expressions, consult almost
  68 any textbook about compiler construction.
  69
  70 % XXX The reference could be made more specific, say to
  71 % "Compilers: Principles, Techniques and Tools", by Alfred V. Aho,
  72 % Ravi Sethi, and Jeffrey D. Ullman, or some FA text.
  73
  74 A brief explanation of the format of regular expressions follows.
  75
  76 Regular expressions can contain both special and ordinary characters.
  77 Ordinary characters, like '\code{A}', '\code{a}', or '\code{0}', are
  78 the simplest regular expressions; they simply match themselves.  You
  79 can concatenate ordinary characters, so '\code{last}' matches the
  80 characters 'last'.  (In the rest of this section, we'll write RE's in
  81 \code{this special font}, usually without quotes, and strings to be
  82 matched 'in single quotes'.)
  83
  84 Special characters either stand for classes of ordinary characters, or
  85 affect how the regular expressions around them are interpreted.
  86
  87 The special characters are:
  88 \begin{itemize}
  89 \item[\code{.}] (Dot.)  Matches any character except a newline.
  90 \item[\code{\^}] (Caret.)  Matches the start of the string.
  91 \item[\code{\$}] Matches the end of the string.
  92 \code{foo} matches both 'foo' and 'foobar', while the regular
  93 expression '\code{foo\$}' matches only 'foo'.
  94 \item[\code{*}] Causes the resulting RE to
  95 match 0 or more repetitions of the preceding RE.  \code{ab*} will
  96 match 'a', 'ab', or 'a' followed by any number of 'b's.
  97 \item[\code{+}] Causes the
  98 resulting RE to match 1 or more repetitions of the preceding RE.
  99 \code{ab+} will match 'a' followed by any non-zero number of 'b's; it
 100 will not match just 'a'.
 101 \item[\code{?}] Causes the resulting RE to
 102 match 0 or 1 repetitions of the preceding RE.  \code{ab?} will
 103 match either 'a' or 'ab'.
 104
 105 \item[\code{\e}] Either escapes special characters (permitting you to match
 106 characters like '*?+\&\$'), or signals a special sequence; special
 107 sequences are discussed below.  Remember that Python also uses the
 108 backslash as an escape sequence in string literals; if the escape
 109 sequence isn't recognized by Python's parser, the backslash and
 110 subsequent character are included in the resulting string.  However,
 111 if Python would recognize the resulting sequence, the backslash should
 112 be repeated twice.
 113
 114 \item[\code{[]}] Used to indicate a set of characters.  Characters can
 115 be listed individually, or a range is indicated by giving two
 116 characters and separating them by a '-'.  Special characters are
 117 not active inside sets.  For example, \code{[akm\$]}
 118 will match any of the characters 'a', 'k', 'm', or '\$'; \code{[a-z]} will
 119 match any lowercase letter.
 120
 121 If you want to include a \code{]} inside a
 122 set, it must be the first character of the set; to include a \code{-},
 123 place it as the first or last character.
 124
 125 Characters \emph{not} within a range can be matched by including a
 126 \code{\^} as the first character of the set; \code{\^} elsewhere will
 127 simply match the '\code{\^}' character.
 128 \end{itemize}
 129
 130 The special sequences consist of '\code{\e}' and a character
 131 from the list below.  If the ordinary character is not on the list,
 132 then the resulting RE will match the second character.  For example,
 133 \code{\e\$} matches the character '\$'.  Ones where the backslash
 134 should be doubled in string literals are indicated.
 135
 136 \begin{itemize}
 137 \item[\code{\e|}]\code{A\e|B}, where A and B can be arbitrary REs,
 138 creates a regular expression that will match either A or B.  This can
 139 be used inside groups (see below) as well.
 140 %
 141 \item[\code{\e( \e)}] Indicates the start and end of a group; the
 142 contents of a group can be matched later in the string with the
 143 \code{\e [1-9]} special sequence, described next.
 144 \end{itemize}
 145
 146 \begin{fulllineitems}
 147 \item[\code{\e \e 1, ... \e \e 7, \e 8, \e 9}]
 148 Matches the contents of the group of the same
 149 number.  For example, \code{\e (.+\e ) \e \e 1} matches 'the the' or
 150 '55 55', but not 'the end' (note the space after the group).  This
 151 special sequence can only be used to match one of the first 9 groups;
 152 groups with higher numbers can be matched using the \code{\e v}
 153 sequence.  (\code{\e 8} and \code{\e 9} don't need a double backslash
 154 because they are not octal digits.)
 155 \end{fulllineitems}
 156
 157 \begin{itemize}
 158 \item[\code{\e \e b}] Matches the empty string, but only at the
 159 beginning or end of a word.  A word is defined as a sequence of
 160 alphanumeric characters, so the end of a word is indicated by
 161 whitespace or a non-alphanumeric character.
 162 %
 163 \item[\code{\e B}] Matches the empty string, but when it is \emph{not} at the
 164 beginning or end of a word.
 165 %
 166 \item[\code{\e v}] Must be followed by a two digit decimal number, and
 167 matches the contents of the group of the same number.  The group
 168 number must be between 1 and 99, inclusive.
 169 %
 170 \item[\code{\e w}]Matches any alphanumeric character; this is
 171 equivalent to the set \code{[a-zA-Z0-9]}.
 172 %
 173 \item[\code{\e W}] Matches any non-alphanumeric character; this is
 174 equivalent to the set \code{[\^a-zA-Z0-9]}.
 175 \item[\code{\e <}] Matches the empty string, but only at the beginning of a
 176 word.  A word is defined as a sequence of alphanumeric characters, so
 177 the end of a word is indicated by whitespace or a non-alphanumeric
 178 character.
 179 \item[\code{\e >}] Matches the empty string, but only at the end of a
 180 word.
 181
 182 \item[\code{\e \e \e \e}] Matches a literal backslash.
 183
 184 % In Emacs, the following two are start of buffer/end of buffer.  In
 185 % Python they seem to be synonyms for ^$.
 186 \item[\code{\e `}] Like \code{\^}, this only matches at the start of the
 187 string.
 188 \item[\code{\e \e '}] Like \code{\$}, this only matches at the end of
 189 the string.
 190 % end of buffer
 191 \end{itemize}
 192
 193 \subsection{Module Contents}
 194 \nodename{Contents of Module regex}
 195
 196 The module defines these functions, and an exception:
 197
 198
 199 \begin{funcdesc}{match}{pattern, string}
 200   Return how many characters at the beginning of \var{string} match
 201   the regular expression \var{pattern}.  Return \code{-1} if the
 202   string does not match the pattern (this is different from a
 203   zero-length match!).
 204 \end{funcdesc}
 205
 206 \begin{funcdesc}{search}{pattern, string}
 207   Return the first position in \var{string} that matches the regular
 208   expression \var{pattern}.  Return \code{-1} if no position in the string
 209   matches the pattern (this is different from a zero-length match
 210   anywhere!).
 211 \end{funcdesc}
 212
 213 \begin{funcdesc}{compile}{pattern\optional{, translate}}
 214   Compile a regular expression pattern into a regular expression
 215   object, which can be used for matching using its \code{match()} and
 216   \code{search()} methods, described below.  The optional argument
 217   \var{translate}, if present, must be a 256-character string
 218   indicating how characters (both of the pattern and of the strings to
 219   be matched) are translated before comparing them; the \var{i}-th
 220   element of the string gives the translation for the character with
 221   \ASCII{} code \var{i}.  This can be used to implement
 222   case-insensitive matching; see the \code{casefold} data item below.
 223
 224   The sequence
 225
 226 \begin{verbatim}
 227 prog = regex.compile(pat)
 228 result = prog.match(str)
 229 \end{verbatim}
 230 %
 231 is equivalent to
 232
 233 \begin{verbatim}
 234 result = regex.match(pat, str)
 235 \end{verbatim}
 236
 237 but the version using \code{compile()} is more efficient when multiple
 238 regular expressions are used concurrently in a single program.  (The
 239 compiled version of the last pattern passed to \code{regex.match()} or
 240 \code{regex.search()} is cached, so programs that use only a single
 241 regular expression at a time needn't worry about compiling regular
 242 expressions.)
 243 \end{funcdesc}
 244
 245 \begin{funcdesc}{set_syntax}{flags}
 246   Set the syntax to be used by future calls to \code{compile()},
 247   \code{match()} and \code{search()}.  (Already compiled expression
 248   objects are not affected.)  The argument is an integer which is the
 249   OR of several flag bits.  The return value is the previous value of
 250   the syntax flags.  Names for the flags are defined in the standard
 251   module \code{regex_syntax}\refstmodindex{regex_syntax}; read the
 252   file \file{regex_syntax.py} for more information.
 253 \end{funcdesc}
 254
 255 \begin{funcdesc}{get_syntax}{}
 256   Returns the current value of the syntax flags as an integer.
 257 \end{funcdesc}
 258
 259 \begin{funcdesc}{symcomp}{pattern\optional{, translate}}
 260 This is like \code{compile()}, but supports symbolic group names: if a
 261 parenthesis-enclosed group begins with a group name in angular
 262 brackets, e.g. \code{'\e(<id>[a-z][a-z0-9]*\e)'}, the group can
 263 be referenced by its name in arguments to the \code{group()} method of
 264 the resulting compiled regular expression object, like this:
 265 \code{p.group('id')}.  Group names may contain alphanumeric characters
 266 and \code{'_'} only.
 267 \end{funcdesc}
 268
 269 \begin{excdesc}{error}
 270   Exception raised when a string passed to one of the functions here
 271   is not a valid regular expression (e.g., unmatched parentheses) or
 272   when some other error occurs during compilation or matching.  (It is
 273   never an error if a string contains no match for a pattern.)
 274 \end{excdesc}
 275
 276 \begin{datadesc}{casefold}
 277 A string suitable to pass as the \var{translate} argument to
 278 \code{compile()} to map all upper case characters to their lowercase
 279 equivalents.
 280 \end{datadesc}
 281
 282 \noindent
 283 Compiled regular expression objects support these methods:
 284
 285 \setindexsubitem{(regex method)}
 286 \begin{funcdesc}{match}{string\optional{, pos}}
 287   Return how many characters at the beginning of \var{string} match
 288   the compiled regular expression.  Return \code{-1} if the string
 289   does not match the pattern (this is different from a zero-length
 290   match!).
 291
 292   The optional second parameter, \var{pos}, gives an index in the string
 293   where the search is to start; it defaults to \code{0}.  This is not
 294   completely equivalent to slicing the string; the \code{'\^'} pattern
 295   character matches at the real beginning of the string and at positions
 296   just after a newline, not necessarily at the index where the search
 297   is to start.
 298 \end{funcdesc}
 299
 300 \begin{funcdesc}{search}{string\optional{, pos}}
 301   Return the first position in \var{string} that matches the regular
 302   expression \code{pattern}.  Return \code{-1} if no position in the
 303   string matches the pattern (this is different from a zero-length
 304   match anywhere!).
 305
 306   The optional second parameter has the same meaning as for the
 307   \code{match()} method.
 308 \end{funcdesc}
 309
 310 \begin{funcdesc}{group}{index, index, ...}
 311 This method is only valid when the last call to the \code{match()}
 312 or \code{search()} method found a match.  It returns one or more
 313 groups of the match.  If there is a single \var{index} argument,
 314 the result is a single string; if there are multiple arguments, the
 315 result is a tuple with one item per argument.  If the \var{index} is
 316 zero, the corresponding return value is the entire matching string; if
 317 it is in the inclusive range [1..99], it is the string matching the
 318 corresponding parenthesized group (using the default syntax,
 319 groups are parenthesized using \code{{\e}(} and \code{{\e})}).  If no
 320 such group exists, the corresponding result is \code{None}.
 321
 322 If the regular expression was compiled by \code{symcomp()} instead of
 323 \code{compile()}, the \var{index} arguments may also be strings
 324 identifying groups by their group name.
 325 \end{funcdesc}
 326
 327 \noindent
 328 Compiled regular expressions support these data attributes:
 329
 330 \setindexsubitem{(regex attribute)}
 331
 332 \begin{datadesc}{regs}
 333 When the last call to the \code{match()} or \code{search()} method found a
 334 match, this is a tuple of pairs of indexes corresponding to the
 335 beginning and end of all parenthesized groups in the pattern.  Indices
 336 are relative to the string argument passed to \code{match()} or
 337 \code{search()}.  The 0-th tuple gives the beginning and end or the
 338 whole pattern.  When the last match or search failed, this is
 339 \code{None}.
 340 \end{datadesc}
 341
 342 \begin{datadesc}{last}
 343 When the last call to the \code{match()} or \code{search()} method found a
 344 match, this is the string argument passed to that method.  When the
 345 last match or search failed, this is \code{None}.
 346 \end{datadesc}
 347
 348 \begin{datadesc}{translate}
 349 This is the value of the \var{translate} argument to
 350 \code{regex.compile()} that created this regular expression object.  If
 351 the \var{translate} argument was omitted in the \code{regex.compile()}
 352 call, this is \code{None}.
 353 \end{datadesc}
 354
 355 \begin{datadesc}{givenpat}
 356 The regular expression pattern as passed to \code{compile()} or
 357 \code{symcomp()}.
 358 \end{datadesc}
 359
 360 \begin{datadesc}{realpat}
 361 The regular expression after stripping the group names for regular
 362 expressions compiled with \code{symcomp()}.  Same as \code{givenpat}
 363 otherwise.
 364 \end{datadesc}
 365
 366 \begin{datadesc}{groupindex}
 367 A dictionary giving the mapping from symbolic group names to numerical
 368 group indexes for regular expressions compiled with \code{symcomp()}.
 369 \code{None} otherwise.
 370 \end{datadesc}