postfix-master/BUILTIN_FILTER_README.html

   1 <!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
   2         "http://www.w3.org/TR/html4/loose.dtd">
   3
   4 <html>
   5
   6 <head>
   7
   8 <title>Postfix Built-in Content Inspection</title>
   9
  10 <meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
  11
  12 </head>
  13
  14 <body>
  15
  16 <h1><img src="postfix-logo.jpg" width="203" height="98" alt="">
  17 Postfix Built-in Content Inspection</h1>
  18
  19 <hr>
  20
  21 <h2>Built-in content inspection introduction </h2>
  22
  23 <p> Postfix supports a built-in filter mechanism that examines
  24 message header and message body content, one line at a time, before
  25 it is stored in the Postfix queue. The filter is usually implemented
  26 with POSIX or PCRE regular expressions, as described in the
  27 <a href="header_checks.5.html">header_checks(5)</a> manual page. </p>
  28
  29 <p>  The original purpose of the built-in filter is to stop an
  30 outbreak of specific email worms or viruses, and it does this job
  31 well. The filter has also helped to block bounced junk email,
  32 bounced email from worms or viruses, and notifications from virus
  33 detection systems.  Information about this secondary application
  34 is given in the <a href="BACKSCATTER_README.html">BACKSCATTER_README</a> document.  </p>
  35
  36 <p> Because the built-in filter is optimized for stopping specific
  37 worms and virus outbreaks, it has <a href="#limitations">limitations</a>
  38 that make it NOT suitable for general junk email and virus detection.
  39 For that, you should use one of the external content inspection
  40 methods that are described in the <a href="FILTER_README.html">FILTER_README</a>, <a href="SMTPD_PROXY_README.html">SMTPD_PROXY_README</a>
  41 and <a href="MILTER_README.html">MILTER_README</a> documents.  </p>
  42
  43 <p> The following diagram gives an over-all picture of how Postfix
  44 built-in content inspection works: </p>
  45
  46 <blockquote>
  47
  48 <table>
  49
  50 <tr>
  51
  52     <td colspan="4"> <td bgcolor="#f0f0ff" align="center"
  53     valign="middle"> Postmaster<br> notifications </td>
  54
  55 </tr>
  56
  57 <tr>
  58
  59     <td colspan="4"> <td align="center"> <tt> |<br>v </tt></td>
  60
  61 </tr>
  62
  63 <tr>
  64
  65         <td bgcolor="#f0f0ff" align="center" valign="middle">
  66         Network or<br> local users </td>
  67
  68     <td align="center" valign="middle"> <tt> -&gt; </tt> </td>
  69
  70         <td bgcolor="#f0f0ff" align="center" valign="middle">
  71
  72         <b> Built-in<br> filter</b> </td>
  73
  74     <td align="center" valign="middle"> <tt> -&gt; </tt> </td>
  75
  76         <td bgcolor="#f0f0ff" align="center" valign="middle">
  77         Postfix<br> queue </td>
  78
  79     <td align="center" valign="middle"> <tt> -&gt; </tt> </td>
  80
  81         <td bgcolor="#f0f0ff" align="center" valign="middle">
  82         Delivery<br> agents </td>
  83
  84     <td align="center" valign="middle"> <tt> -&gt; </tt> </td>
  85
  86         <td bgcolor="#f0f0ff" align="center" valign="middle">
  87         Network or<br> local mailbox </td>
  88
  89 </tr>
  90
  91 <tr>
  92
  93     <td colspan="4"> <td align="center"> ^<br> <tt> | </tt> </td>
  94     <td> </td> <td align="center"> <tt> |<br>v </tt> </td>
  95
  96 </tr>
  97
  98 <tr>
  99
 100     <td colspan="4"> <td colspan="3" bgcolor="#f0f0ff" align="center"
 101     valign="middle"> Undeliverable mail<br> Forwarded mail</td>
 102
 103 </tr>
 104
 105 </table>
 106
 107 </blockquote>
 108
 109 <p> The picture makes clear that the filter works while Postfix is
 110 receiving new mail.  This means that Postfix can reject mail from
 111 the network without having to return undeliverable mail to the
 112 originator address (which is often spoofed anyway). However, this
 113 ability comes at a price:  if mail inspection takes too much time,
 114 then the remote client will time out, and the client may send the
 115 same message repeatedly. </p>
 116
 117 <p>Topics covered by this document: </p>
 118
 119 <ul>
 120
 121 <li><a href="#what">What mail is subjected to header/body checks </a>
 122
 123 <li><a href="#limitations">Limitations of Postfix header/body checks </a>
 124
 125 <li><a href="#daily">Preventing daily mail status reports from being blocked </a>
 126
 127 <li><a href="#remote_only">Configuring header/body checks for mail from outside users only</a>
 128
 129 <li><a href="#domain_except">Configuring header/body checks for mail to some domains only</a>
 130
 131 </ul>
 132
 133 <h2><a name="what">What mail is subjected to header/body checks </a></h2>
 134
 135 <p> Postfix header/body checks are implemented by the <a href="cleanup.8.html">cleanup(8)</a>
 136 server before it injects mail into the <a href="QSHAPE_README.html#incoming_queue">incoming queue</a>.  The diagram
 137 below zooms in on the <a href="cleanup.8.html">cleanup(8)</a> server, and shows that this server
 138 handles mail from many different sources. In order to keep the
 139 diagram readable, the sources of postmaster notifications are not
 140 shown, because they can be produced by many Postfix daemon processes.
 141 </p>
 142
 143 <blockquote>
 144
 145 <table>
 146
 147 <tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center"
 148 valign="middle"> <a href="bounce.8.html">bounce(8)</a><br> (undeliverable) </td> </tr>
 149
 150 <tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b>
 151 <a href="smtpd.8.html">smtpd(8)</a><br> (network)</b> </td> <td align="left" valign="bottom">
 152 <tt> \ </tt> </td> <td align="center" valign="middle"> <tt> |<br>v
 153 </tt> </td> </tr>
 154
 155 <tr> <td> </td> <td> </td> </tr>
 156
 157 <tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b>
 158 <a href="qmqpd.8.html">qmqpd(8)</a><br> (network)</b> </td> <td align="center" valign="middle">
 159 <tt> -\<br>-/ </tt> </td> <td bgcolor="#f0f0ff" align="center"
 160 valign="middle"> <a href="cleanup.8.html">cleanup(8)</a> </td> <td align="center" valign="middle">
 161 <tt> -&gt; </tt> </td> <td bgcolor="#f0f0ff" align="center"
 162 valign="middle"> <a href="QSHAPE_README.html#incoming_queue">
 163 incoming<br> queue </a> </td> </tr>
 164
 165 <tr> <td bgcolor="#f0f0ff" align="center" valign="middle"> <b>
 166 <a href="pickup.8.html">pickup(8)</a><br> (local)</b> </td> <td align="left" valign="top"> <tt>
 167 / </tt> </td> <td align="center" valign="middle"> ^<br> <tt> |
 168 </tt> </td> </tr>
 169
 170 <tr> <td colspan="2"> </td> <td bgcolor="#f0f0ff" align="center"
 171 valign="middle"> <a href="local.8.html">local(8)</a><br> (forwarded) </td> </tr>
 172
 173 </table>
 174
 175 </blockquote>
 176
 177 <p> For efficiency reasons, only mail that enters from outside of
 178 Postfix is inspected with header/body checks. It would be inefficient
 179 to filter already filtered mail again, and it would be undesirable
 180 to block postmaster notifications.  The table below summarizes what
 181 mail is and is not subject to header/body checks. </p>
 182
 183 <blockquote>
 184
 185 <table border="1">
 186
 187 <tr> <th> Message type </th> <th> Source </th> <th> Header/body checks? </th> </tr>
 188
 189 <tr> <td> Undeliverable mail </td> <td> <a href="bounce.8.html">bounce(8)</a> </td> <td> No </td> </tr>
 190
 191 <tr> <td> Network mail </td> <td> <a href="smtpd.8.html">smtpd(8)</a> </td> <td> Configurable </td> </tr>
 192
 193 <tr> <td> Network mail </td> <td> <a href="qmqpd.8.html">qmqpd(8)</a> </td>  <td> Configurable </td> </tr>
 194
 195 <tr> <td> Local submission </td> <td> <a href="pickup.8.html">pickup(8)</a> </td>  <td> Configurable </td> </tr>
 196
 197 <tr> <td> Local forwarding </td> <td> <a href="local.8.html">local(8)</a> </td> <td> No </td> </tr>
 198
 199 <tr> <td> Postmaster notice </td> <td> many </td> <td> No </td> </tr>
 200
 201 </table>
 202
 203 </blockquote>
 204
 205 <p> How does Postfix decide what mail needs to be filtered? It
 206 would be clumsy to make the decision in the <a href="cleanup.8.html">cleanup(8)</a> server, as
 207 this program receives mail from so many different sources.  Instead,
 208 header/body checks are requested by the source.  Examples of how
 209 to turn off header/body checks for mail received with <a href="smtpd.8.html">smtpd(8)</a>,
 210 <a href="qmqpd.8.html">qmqpd(8)</a> or <a href="pickup.8.html">pickup(8)</a> are given below under "<a
 211 href="#remote_only">Configuring header/body checks for mail from
 212 outside users only</a>" and "<a href="#domain_except">Configuring
 213 header/body checks for mail to some domains only</a>".  </p>
 214
 215 <h2><a name="limitations">Limitations of Postfix header/body checks </a></h2>
 216
 217 <ul>
 218
 219 <li> <p> Header/body checks do not decode message headers or message
 220 body content. For example, if text in the message body is BASE64
 221 encoded (<a href="http://tools.ietf.org/html/rfc2045">RFC 2045</a>) then your regular expressions will have to match
 222 the BASE64 encoded form. Likewise, message headers with encoded
 223 non-ASCII characters (<a href="http://tools.ietf.org/html/rfc2047">RFC 2047</a>) need to be matched in their encoded
 224 form. </p>
 225
 226 <li> <p> Header/body checks cannot filter on a combination of
 227 message headers or body lines. Header/body checks examine content
 228 one message header at a time, or one message body line at a time,
 229 and cannot carry a decision over to the next message header or body
 230 line.  </p>
 231
 232 <li> <p> Header/body checks cannot depend on the recipient of a
 233 message.  </p>
 234
 235 <ul>
 236
 237 <li> <p> One message can have multiple recipients, and all recipients
 238 of a message receive the same treatment. Workarounds have been
 239 proposed that involve selectively deferring some recipients of
 240 multi-recipient mail, but that results in poor SMTP performance
 241 and does not work for non-SMTP mail.  </p>
 242
 243 <li> <p> Some sources of mail send the headers and content ahead
 244 of the recipient information. It would be inefficient to buffer up
 245 an entire message before deciding if it needs to be filtered, and
 246 it would be clumsy to filter mail and to buffer up all the actions
 247 until it is known whether those actions need to be executed.  </p>
 248
 249 </ul>
 250
 251 <li> <p> Despite warnings, some people try to use the built-in
 252 filter feature for general junk email and/or virus blocking, using
 253 hundreds or even thousands of regular expressions. This can result
 254 in catastrophic performance failure. The symptoms are as follows:
 255 </p>
 256
 257 <ul>
 258
 259 <li> <p> The <a href="cleanup.8.html">cleanup(8)</a> processes use up all available CPU time in
 260 order to process the regular expressions, and/or they use up all
 261 available memory so that the system begins to swap. This slows down
 262 all incoming mail deliveries.  </p>
 263
 264 <li> <p> As Postfix needs more and more time to receive an email
 265 message, the number of simultaneous SMTP sessions increases to the
 266 point that the SMTP server process limit is reached. </p>
 267
 268 <li> <p> While all SMTP server processes are waiting for the
 269 <a href="cleanup.8.html">cleanup(8)</a> servers to finish, new SMTP clients have to wait until
 270 an SMTP server process becomes available. This causes mail deliveries
 271 to time out before they have even begun.  </p>
 272
 273 </ul>
 274
 275 <p> The remedy for this type of performance problem is simple:
 276 don't use header/body checks for general junk email and/or virus
 277 blocking, and don't filter mail before it is queued.  When performance
 278 is a concern, use an external content filter that runs after mail
 279 is queued, as described in the <a href="FILTER_README.html">FILTER_README</a> document. </p>
 280
 281 </ul>
 282
 283 <h2><a name="daily">Preventing daily mail status reports from being blocked </a></h2>
 284
 285 <p>The following is quoted from Jim Seymour's Pflogsumm FAQ at
 286 <a href="http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt">http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt</a>. Pflogsumm
 287 is a program that analyzes Postfix logs, including the logging from
 288 rejected mail. If these logs contain text that was rejected by
 289 Postfix <a href="postconf.5.html#body_checks">body_checks</a> patterns, then the logging is also likely to
 290 be rejected by those same <a href="postconf.5.html#body_checks">body_checks</a> patterns. This problem does
 291 not exist with <a href="postconf.5.html#header_checks">header_checks</a> patterns, because those are not applied
 292 to the text that is part of the mail status report. </p>
 293
 294 <blockquote>
 295
 296 <p>You configure Postfix to do body checks, Postfix does its thing,
 297 Pflogsumm reports it and Postfix catches the same string in the
 298 Pflogsumm report.  There are several solutions to this. </p>
 299
 300 <p> Wolfgang Zeikat contributed this: </p>
 301
 302 <blockquote>
 303 <pre>
 304 #!/usr/bin/perl
 305 use MIME::Lite;
 306
 307 ### Create a new message:
 308 $msg = MIME::Lite-&gt;new(
 309     From     =&gt; 'your@send.er',
 310     To       =&gt; 'your@recipie.nt',
 311     # Cc     =&gt; 'some@other.com, some@more.com',
 312     Subject  =&gt; 'pflogsumm',
 313     Date     =&gt; `date`,
 314     Type     =&gt; 'text/plain',
 315     Encoding =&gt; 'base64',
 316     Path     =&gt; '/tmp/pflogg',
 317 );
 318
 319 $msg-&gt;send;
 320 </pre>
 321 </blockquote>
 322
 323 <p> Where "/tmp/pflogg" is the output of Pflogsumm.  This puts Pflogsumm's
 324 output in a base64 MIME attachment. </p>
 325
 326 </blockquote>
 327
 328 <p> Note by Wietse: if you run this on a machine that is accessible
 329 by untrusted users, it is safer to store the Pflogsumm report in
 330 a directory that is not world writable. </p>
 331
 332 <blockquote>
 333
 334 <p> In a follow-up to a thread in the postfix-users mailing list, Ralf
 335 Hildebrandt noted: </p>
 336
 337 <blockquote> <p> "mpack does the same thing." </p> </blockquote>
 338
 339 </blockquote>
 340
 341 <p> And it does. Which tool one should use is a matter of preference.
 342 </p>
 343
 344 <p> Other solutions involve additional <a href="postconf.5.html#body_checks">body_checks</a> rules that make
 345 exceptions for daily mail status reports, but this is not recommended.
 346 Such rules slow down all mail and complicate Postfix maintenance.
 347 </p>
 348
 349 <h2><a name="remote_only">Configuring header/body checks for mail from outside users only</a></h2>
 350
 351 <p> The following information applies to Postfix 2.1 and later.
 352 Earlier
 353 Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature.
 354 </p>
 355
 356 <p> The easiest approach is to configure ONE Postfix instance with
 357 multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>: </p>
 358
 359 <ul>
 360
 361 <li> <p> Two SMTP server IP addresses for mail from inside users
 362 only, with header/body filtering turned off, and a local mail pickup
 363 service with header/body filtering turned off.  </p>
 364
 365 <pre>
 366 /etc/postfix.<a href="master.5.html">master.cf</a>:
 367     # ==================================================================
 368     # service      type  private unpriv  chroot  wakeup  maxproc command
 369     #                    (yes)   (yes)   (yes)   (never) (100)
 370     # ==================================================================
 371     1.2.3.4:smtp   inet  n       -       n       -       -       smtpd
 372         -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a>
 373     127.0.0.1:smtp inet  n       -       n       -       -       smtpd
 374         -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a>
 375     pickup         fifo  n       -       n       60      1       pickup
 376         -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a>
 377 </pre>
 378
 379 <li> <p> Add some firewall rule to prevent access to 1.2.3.4:smtp
 380 from the outside world. </p>
 381
 382 <li> <p> One SMTP server address for mail from outside users with
 383 header/body filtering turned on via <a href="postconf.5.html">main.cf</a>. </p>
 384
 385 <pre>
 386 /etc/postfix.<a href="master.5.html">master.cf</a>:
 387     # =================================================================
 388     # service     type  private unpriv  chroot  wakeup  maxproc command
 389     #                   (yes)   (yes)   (yes)   (never) (100)
 390     # =================================================================
 391     1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
 392 </pre>
 393
 394 </ul>
 395
 396 <h2><a name="domain_except">Configuring header/body checks for mail to some domains only</a></h2>
 397
 398 <p> The following information applies to Postfix 2.1. Earlier
 399 Postfix versions do not support the <a href="postconf.5.html#receive_override_options">receive_override_options</a> feature.
 400 </p>
 401
 402 <p> If you are MX service provider and want to apply disable
 403 head/body checks for some domains, you can configure ONE Postfix
 404 instance with multiple SMTP server IP addresses in <a href="master.5.html">master.cf</a>. Each
 405 address provides a different service. </p>
 406
 407 <blockquote>
 408
 409 <pre>
 410 /etc/postfix.<a href="master.5.html">master.cf</a>:
 411     # =================================================================
 412     # service     type  private unpriv  chroot  wakeup  maxproc command
 413     #                   (yes)   (yes)   (yes)   (never) (100)
 414     # =================================================================
 415     # SMTP service for domains with header/body checks turned on.
 416     1.2.3.4:smtp  inet  n       -       n       -       -       smtpd
 417
 418     # SMTP service for domains with header/body checks turned off.
 419     1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
 420         -o <a href="postconf.5.html#receive_override_options">receive_override_options</a>=<a href="postconf.5.html#no_header_body_checks">no_header_body_checks</a>
 421 </pre>
 422 </blockquote>
 423
 424 <p> Once this is set up you can configure MX records in the DNS
 425 that route each domain to the proper SMTP server instance. </p>
 426
 427 </body>
 428
 429 </html>