1 <?xml version="1.0" encoding="UTF-8" ?>
2 <!DOCTYPE manualpage SYSTEM "../style/manualpage.dtd">
3 <?xml-stylesheet type="text/xsl" href="../style/manual.en.xsl"?>
4 <!-- $LastChangedRevision$ -->
7 Licensed to the Apache Software Foundation (ASF) under one or more
8 contributor license agreements. See the NOTICE file distributed with
9 this work for additional information regarding copyright ownership.
10 The ASF licenses this file to You under the Apache License, Version 2.0
11 (the "License"); you may not use this file except in compliance with
12 the License. You may obtain a copy of the License at
14 http://www.apache.org/licenses/LICENSE-2.0
16 Unless required by applicable law or agreed to in writing, software
17 distributed under the License is distributed on an "AS IS" BASIS,
18 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
19 See the License for the specific language governing permissions and
20 limitations under the License.
23 <manualpage metafile="perf-tuning.xml.meta">
24 <parentdocument href="./">Miscellaneous Documentation</parentdocument>
26 <title>Apache Performance Tuning</title>
30 <p>Apache 2.x is a general-purpose webserver, designed to
31 provide a balance of flexibility, portability, and performance.
32 Although it has not been designed specifically to set benchmark
33 records, Apache 2.x is capable of high performance in many
34 real-world situations.</p>
36 <p>Compared to Apache 1.3, release 2.x contains many additional
37 optimizations to increase throughput and scalability. Most of
38 these improvements are enabled by default. However, there are
39 compile-time and run-time configuration choices that can
40 significantly affect performance. This document describes the
41 options that a server administrator can configure to tune the
42 performance of an Apache 2.x installation. Some of these
43 configuration options enable the httpd to better take advantage
44 of the capabilities of the hardware and OS, while others allow
45 the administrator to trade functionality for speed.</p>
49 <section id="hardware">
51 <title>Hardware and Operating System Issues</title>
53 <p>The single biggest hardware issue affecting webserver
54 performance is RAM. A webserver should never ever have to swap,
55 as swapping increases the latency of each request beyond a point
56 that users consider "fast enough". This causes users to hit
57 stop and reload, further increasing the load. You can, and
58 should, control the <directive module="mpm_common"
59 >MaxClients</directive> setting so that your server
60 does not spawn so many children it starts swapping. This procedure
61 for doing this is simple: determine the size of your average Apache
62 process, by looking at your process list via a tool such as
63 <code>top</code>, and divide this into your total available memory,
64 leaving some room for other processes.</p>
66 <p>Beyond that the rest is mundane: get a fast enough CPU, a
67 fast enough network card, and fast enough disks, where "fast
68 enough" is something that needs to be determined by
71 <p>Operating system choice is largely a matter of local
72 concerns. But some guidelines that have proven generally
77 <p>Run the latest stable release and patchlevel of the
78 operating system that you choose. Many OS suppliers have
79 introduced significant performance improvements to their
80 TCP stacks and thread libraries in recent years.</p>
84 <p>If your OS supports a <code>sendfile(2)</code> system
85 call, make sure you install the release and/or patches
86 needed to enable it. (With Linux, for example, this means
87 using Linux 2.4 or later. For early releases of Solaris 8,
88 you may need to apply a patch.) On systems where it is
89 available, <code>sendfile</code> enables Apache 2 to deliver
90 static content faster and with lower CPU utilization.</p>
96 <section id="runtime">
98 <title>Run-Time Configuration Issues</title>
102 <module>mod_dir</module>
103 <module>mpm_common</module>
104 <module>mod_status</module>
107 <directive module="core">AllowOverride</directive>
108 <directive module="mod_dir">DirectoryIndex</directive>
109 <directive module="core">HostnameLookups</directive>
110 <directive module="core">EnableMMAP</directive>
111 <directive module="core">EnableSendfile</directive>
112 <directive module="core">KeepAliveTimeout</directive>
113 <directive module="prefork">MaxSpareServers</directive>
114 <directive module="prefork">MinSpareServers</directive>
115 <directive module="core">Options</directive>
116 <directive module="mpm_common">StartServers</directive>
122 <title>HostnameLookups and other DNS considerations</title>
124 <p>Prior to Apache 1.3, <directive module="core"
125 >HostnameLookups</directive> defaulted to <code>On</code>.
126 This adds latency to every request because it requires a
127 DNS lookup to complete before the request is finished. In
128 Apache 1.3 this setting defaults to <code>Off</code>. If you need
129 to have addresses in your log files resolved to hostnames, use the
130 <program>logresolve</program>
131 program that comes with Apache, or one of the numerous log
132 reporting packages which are available.</p>
134 <p>It is recommended that you do this sort of postprocessing of
135 your log files on some machine other than the production web
136 server machine, in order that this activity not adversely affect
137 server performance.</p>
139 <p>If you use any <code><directive module="mod_access_compat"
140 >Allow</directive> from domain</code> or <code><directive
141 module="mod_access_compat">Deny</directive> from domain</code>
142 directives (i.e., using a hostname, or a domain name, rather than
143 an IP address) then you will pay for
144 two DNS lookups (a reverse, followed by a forward lookup
145 to make sure that the reverse is not being spoofed). For best
146 performance, therefore, use IP addresses, rather than names, when
147 using these directives, if possible.</p>
149 <p>Note that it's possible to scope the directives, such as
150 within a <code><Location /server-status></code> section.
151 In this case the DNS lookups are only performed on requests
152 matching the criteria. Here's an example which disables lookups
153 except for <code>.html</code> and <code>.cgi</code> files:</p>
156 HostnameLookups off<br />
157 <Files ~ "\.(html|cgi)$"><br />
159 HostnameLookups on<br />
164 <p>But even still, if you just need DNS names in some CGIs you
165 could consider doing the <code>gethostbyname</code> call in the
166 specific CGIs that need it.</p>
170 <section id="symlinks">
172 <title>FollowSymLinks and SymLinksIfOwnerMatch</title>
174 <p>Wherever in your URL-space you do not have an <code>Options
175 FollowSymLinks</code>, or you do have an <code>Options
176 SymLinksIfOwnerMatch</code> Apache will have to issue extra
177 system calls to check up on symlinks. One extra call per
178 filename component. For example, if you had:</p>
181 DocumentRoot /www/htdocs<br />
182 <Directory /><br />
184 Options SymLinksIfOwnerMatch<br />
189 <p>and a request is made for the URI <code>/index.html</code>.
190 Then Apache will perform <code>lstat(2)</code> on
191 <code>/www</code>, <code>/www/htdocs</code>, and
192 <code>/www/htdocs/index.html</code>. The results of these
193 <code>lstats</code> are never cached, so they will occur on
194 every single request. If you really desire the symlinks
195 security checking you can do something like this:</p>
198 DocumentRoot /www/htdocs<br />
199 <Directory /><br />
201 Options FollowSymLinks<br />
203 </Directory><br />
205 <Directory /www/htdocs><br />
207 Options -FollowSymLinks +SymLinksIfOwnerMatch<br />
212 <p>This at least avoids the extra checks for the
213 <directive module="core">DocumentRoot</directive> path.
214 Note that you'll need to add similar sections if you
215 have any <directive module="mod_alias">Alias</directive> or
216 <directive module="mod_rewrite">RewriteRule</directive> paths
217 outside of your document root. For highest performance,
218 and no symlink protection, set <code>FollowSymLinks</code>
219 everywhere, and never set <code>SymLinksIfOwnerMatch</code>.</p>
223 <section id="htacess">
225 <title>AllowOverride</title>
227 <p>Wherever in your URL-space you allow overrides (typically
228 <code>.htaccess</code> files) Apache will attempt to open
229 <code>.htaccess</code> for each filename component. For
233 DocumentRoot /www/htdocs<br />
234 <Directory /><br />
236 AllowOverride all<br />
241 <p>and a request is made for the URI <code>/index.html</code>.
242 Then Apache will attempt to open <code>/.htaccess</code>,
243 <code>/www/.htaccess</code>, and
244 <code>/www/htdocs/.htaccess</code>. The solutions are similar
245 to the previous case of <code>Options FollowSymLinks</code>.
246 For highest performance use <code>AllowOverride None</code>
247 everywhere in your filesystem.</p>
251 <section id="negotiation">
253 <title>Negotiation</title>
255 <p>If at all possible, avoid content-negotiation if you're
256 really interested in every last ounce of performance. In
257 practice the benefits of negotiation outweigh the performance
258 penalties. There's one case where you can speed up the server.
259 Instead of using a wildcard such as:</p>
265 <p>Use a complete list of options:</p>
268 DirectoryIndex index.cgi index.pl index.shtml index.html
271 <p>where you list the most common choice first.</p>
273 <p>Also note that explicitly creating a <code>type-map</code>
274 file provides better performance than using
275 <code>MultiViews</code>, as the necessary information can be
276 determined by reading this single file, rather than having to
277 scan the directory for files.</p>
279 <p>If your site needs content negotiation consider using
280 <code>type-map</code> files, rather than the <code>Options
281 MultiViews</code> directive to accomplish the negotiation. See the
282 <a href="../content-negotiation.html">Content Negotiation</a>
283 documentation for a full discussion of the methods of negotiation,
284 and instructions for creating <code>type-map</code> files.</p>
290 <title>Memory-mapping</title>
292 <p>In situations where Apache 2.x needs to look at the contents
293 of a file being delivered--for example, when doing server-side-include
294 processing--it normally memory-maps the file if the OS supports
295 some form of <code>mmap(2)</code>.</p>
297 <p>On some platforms, this memory-mapping improves performance.
298 However, there are cases where memory-mapping can hurt the performance
299 or even the stability of the httpd:</p>
303 <p>On some operating systems, <code>mmap</code> does not scale
304 as well as <code>read(2)</code> when the number of CPUs increases.
305 On multiprocessor Solaris servers, for example, Apache 2.x sometimes
306 delivers server-parsed files faster when <code>mmap</code> is disabled.</p>
310 <p>If you memory-map a file located on an NFS-mounted filesystem
311 and a process on another NFS client machine deletes or truncates
312 the file, your process may get a bus error the next time it tries
313 to access the mapped file content.</p>
317 <p>For installations where either of these factors applies, you
318 should use <code>EnableMMAP off</code> to disable the memory-mapping
319 of delivered files. (Note: This directive can be overridden on
320 a per-directory basis.)</p>
326 <title>Sendfile</title>
328 <p>In situations where Apache 2.x can ignore the contents of the file
329 to be delivered -- for example, when serving static file content --
330 it normally uses the kernel sendfile support the file if the OS
331 supports the <code>sendfile(2)</code> operation.</p>
333 <p>On most platforms, using sendfile improves performance by eliminating
334 separate read and send mechanics. However, there are cases where using
335 sendfile can harm the stability of the httpd:</p>
339 <p>Some platforms may have broken sendfile support that the build
340 system did not detect, especially if the binaries were built on
341 another box and moved to such a machine with broken sendfile support.</p>
344 <p>With an NFS-mounted files, the kernel may be unable
345 to reliably serve the network file through it's own cache.</p>
349 <p>For installations where either of these factors applies, you
350 should use <code>EnableSendfile off</code> to disable sendfile
351 delivery of file contents. (Note: This directive can be overridden
352 on a per-directory basis.)</p>
356 <section id="process">
358 <title>Process Creation</title>
360 <p>Prior to Apache 1.3 the <directive module="prefork"
361 >MinSpareServers</directive>, <directive module="prefork"
362 >MaxSpareServers</directive>, and <directive module="mpm_common"
363 >StartServers</directive> settings all had drastic effects on
364 benchmark results. In particular, Apache required a "ramp-up"
365 period in order to reach a number of children sufficient to serve
366 the load being applied. After the initial spawning of
367 <directive module="mpm_common">StartServers</directive> children,
368 only one child per second would be created to satisfy the
369 <directive module="prefork">MinSpareServers</directive>
370 setting. So a server being accessed by 100 simultaneous
371 clients, using the default <directive module="mpm_common"
372 >StartServers</directive> of <code>5</code> would take on
373 the order 95 seconds to spawn enough children to handle
374 the load. This works fine in practice on real-life servers,
375 because they aren't restarted frequently. But does really
376 poorly on benchmarks which might only run for ten minutes.</p>
378 <p>The one-per-second rule was implemented in an effort to
379 avoid swamping the machine with the startup of new children. If
380 the machine is busy spawning children it can't service
381 requests. But it has such a drastic effect on the perceived
382 performance of Apache that it had to be replaced. As of Apache
383 1.3, the code will relax the one-per-second rule. It will spawn
384 one, wait a second, then spawn two, wait a second, then spawn
385 four, and it will continue exponentially until it is spawning
386 32 children per second. It will stop whenever it satisfies the
387 <directive module="prefork">MinSpareServers</directive>
390 <p>This appears to be responsive enough that it's almost
391 unnecessary to twiddle the <directive module="prefork"
392 >MinSpareServers</directive>, <directive module="prefork"
393 >MaxSpareServers</directive> and <directive module="mpm_common"
394 >StartServers</directive> knobs. When more than 4 children are
395 spawned per second, a message will be emitted to the
396 <directive module="core">ErrorLog</directive>. If you
397 see a lot of these errors then consider tuning these settings.
398 Use the <module>mod_status</module> output as a guide.</p>
400 <p>Related to process creation is process death induced by the
401 <directive module="mpm_common">MaxRequestsPerChild</directive>
402 setting. By default this is <code>0</code>,
403 which means that there is no limit to the number of requests
404 handled per child. If your configuration currently has this set
405 to some very low number, such as <code>30</code>, you may want to bump this
406 up significantly. If you are running SunOS or an old version of
407 Solaris, limit this to <code>10000</code> or so because of memory leaks.</p>
409 <p>When keep-alives are in use, children will be kept busy
410 doing nothing waiting for more requests on the already open
411 connection. The default <directive module="core"
412 >KeepAliveTimeout</directive> of <code>5</code>
413 seconds attempts to minimize this effect. The tradeoff here is
414 between network bandwidth and server resources. In no event
415 should you raise this above about <code>60</code> seconds, as <a
416 href="http://www.research.digital.com/wrl/techreports/abstracts/95.4.html">
417 most of the benefits are lost</a>.</p>
423 <section id="compiletime">
425 <title>Compile-Time Configuration Issues</title>
429 <title>Choosing an MPM</title>
431 <p>Apache 2.x supports pluggable concurrency models, called
432 <a href="../mpm.html">Multi-Processing Modules</a> (MPMs).
433 When building Apache, you must choose an MPM to use. There
434 are platform-specific MPMs for some platforms:
435 <module>beos</module>, <module>mpm_netware</module>,
436 <module>mpmt_os2</module>, and <module>mpm_winnt</module>. For
437 general Unix-type systems, there are several MPMs from which
438 to choose. The choice of MPM can affect the speed and scalability
443 <li>The <module>worker</module> MPM uses multiple child
444 processes with many threads each. Each thread handles
445 one connection at a time. Worker generally is a good
446 choice for high-traffic servers because it has a smaller
447 memory footprint than the prefork MPM.</li>
449 <li>The <module>prefork</module> MPM uses multiple child
450 processes with one thread each. Each process handles
451 one connection at a time. On many systems, prefork is
452 comparable in speed to worker, but it uses more memory.
453 Prefork's threadless design has advantages over worker
454 in some situations: it can be used with non-thread-safe
455 third-party modules, and it is easier to debug on platforms
456 with poor thread debugging support.</li>
460 <p>For more information on these and other MPMs, please
461 see the MPM <a href="../mpm.html">documentation</a>.</p>
465 <section id="modules">
467 <title>Modules</title>
469 <p>Since memory usage is such an important consideration in
470 performance, you should attempt to eliminate modules that you are
471 not actually using. If you have built the modules as <a
472 href="../dso.html">DSOs</a>, eliminating modules is a simple
473 matter of commenting out the associated <directive
474 module="mod_so">LoadModule</directive> directive for that module.
475 This allows you to experiment with removing modules, and seeing
476 if your site still functions in their absense.</p>
478 <p>If, on the other hand, you have modules statically linked
479 into your Apache binary, you will need to recompile Apache in
480 order to remove unwanted modules.</p>
482 <p>An associated question that arises here is, of course, what
483 modules you need, and which ones you don't. The answer here
484 will, of course, vary from one web site to another. However, the
485 <em>minimal</em> list of modules which you can get by with tends
486 to include <module>mod_mime</module>, <module>mod_dir</module>,
487 and <module>mod_log_config</module>. <code>mod_log_config</code> is,
488 of course, optional, as you can run a web site without log
489 files. This is, however, not recommended.</p>
495 <title>Atomic Operations</title>
497 <p>Some modules, such as <module>mod_cache</module> and
498 recent development builds of the worker MPM, use APR's
499 atomic API. This API provides atomic operations that can
500 be used for lightweight thread synchronization.</p>
502 <p>By default, APR implements these operations using the
503 most efficient mechanism available on each target
504 OS/CPU platform. Many modern CPUs, for example, have
505 an instruction that does an atomic compare-and-swap (CAS)
506 operation in hardware. On some platforms, however, APR
507 defaults to a slower, mutex-based implementation of the
508 atomic API in order to ensure compatibility with older
509 CPU models that lack such instructions. If you are
510 building Apache for one of these platforms, and you plan
511 to run only on newer CPUs, you can select a faster atomic
512 implementation at build time by configuring Apache with
513 the <code>--enable-nonportable-atomics</code> option:</p>
517 ./configure --with-mpm=worker --enable-nonportable-atomics=yes
520 <p>The <code>--enable-nonportable-atomics</code> option is
521 relevant for the following platforms:</p>
525 <li>Solaris on SPARC<br />
526 By default, APR uses mutex-based atomics on Solaris/SPARC.
527 If you configure with <code>--enable-nonportable-atomics</code>,
528 however, APR generates code that uses a SPARC v8plus opcode for
529 fast hardware compare-and-swap. If you configure Apache with
530 this option, the atomic operations will be more efficient
531 (allowing for lower CPU utilization and higher concurrency),
532 but the resulting executable will run only on UltraSPARC
536 <li>Linux on x86<br />
537 By default, APR uses mutex-based atomics on Linux. If you
538 configure with <code>--enable-nonportable-atomics</code>,
539 however, APR generates code that uses a 486 opcode for fast
540 hardware compare-and-swap. This will result in more efficient
541 atomic operations, but the resulting executable will run only
542 on 486 and later chips (and not on 386).
551 <title>mod_status and ExtendedStatus On</title>
553 <p>If you include <module>mod_status</module> and you also set
554 <code>ExtendedStatus On</code> when building and running
555 Apache, then on every request Apache will perform two calls to
556 <code>gettimeofday(2)</code> (or <code>times(2)</code>
557 depending on your operating system), and (pre-1.3) several
558 extra calls to <code>time(2)</code>. This is all done so that
559 the status report contains timing indications. For highest
560 performance, set <code>ExtendedStatus off</code> (which is the
567 <title>accept Serialization - multiple sockets</title>
569 <note type="warning"><title>Warning:</title>
570 <p>This section has not been fully updated
571 to take into account changes made in the 2.x version of the
572 Apache HTTP Server. Some of the information may still be
573 relevant, but please use it with care.</p>
576 <p>This discusses a shortcoming in the Unix socket API. Suppose
577 your web server uses multiple <directive module="mpm_common"
578 >Listen</directive> statements to listen on either multiple
579 ports or multiple addresses. In order to test each socket
580 to see if a connection is ready Apache uses
581 <code>select(2)</code>. <code>select(2)</code> indicates that a
582 socket has <em>zero</em> or <em>at least one</em> connection
583 waiting on it. Apache's model includes multiple children, and
584 all the idle ones test for new connections at the same time. A
585 naive implementation looks something like this (these examples
586 do not match the code, they're contrived for pedagogical
594 fd_set accept_fds;<br />
596 FD_ZERO (&accept_fds);<br />
597 for (i = first_socket; i <= last_socket; ++i) {<br />
599 FD_SET (i, &accept_fds);<br />
602 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
603 if (rc < 1) continue;<br />
604 new_connection = -1;<br />
605 for (i = first_socket; i <= last_socket; ++i) {<br />
607 if (FD_ISSET (i, &accept_fds)) {<br />
609 new_connection = accept (i, NULL, NULL);<br />
610 if (new_connection != -1) break;<br />
615 if (new_connection != -1) break;<br />
618 process the new_connection;<br />
623 <p>But this naive implementation has a serious starvation problem.
624 Recall that multiple children execute this loop at the same
625 time, and so multiple children will block at
626 <code>select</code> when they are in between requests. All
627 those blocked children will awaken and return from
628 <code>select</code> when a single request appears on any socket
629 (the number of children which awaken varies depending on the
630 operating system and timing issues). They will all then fall
631 down into the loop and try to <code>accept</code> the
632 connection. But only one will succeed (assuming there's still
633 only one connection ready), the rest will be <em>blocked</em>
634 in <code>accept</code>. This effectively locks those children
635 into serving requests from that one socket and no other
636 sockets, and they'll be stuck there until enough new requests
637 appear on that socket to wake them all up. This starvation
638 problem was first documented in <a
639 href="http://bugs.apache.org/index/full/467">PR#467</a>. There
640 are at least two solutions.</p>
642 <p>One solution is to make the sockets non-blocking. In this
643 case the <code>accept</code> won't block the children, and they
644 will be allowed to continue immediately. But this wastes CPU
645 time. Suppose you have ten idle children in
646 <code>select</code>, and one connection arrives. Then nine of
647 those children will wake up, try to <code>accept</code> the
648 connection, fail, and loop back into <code>select</code>,
649 accomplishing nothing. Meanwhile none of those children are
650 servicing requests that occurred on other sockets until they
651 get back up to the <code>select</code> again. Overall this
652 solution does not seem very fruitful unless you have as many
653 idle CPUs (in a multiprocessor box) as you have idle children,
654 not a very likely situation.</p>
656 <p>Another solution, the one used by Apache, is to serialize
657 entry into the inner loop. The loop looks like this
658 (differences highlighted):</p>
663 <strong>accept_mutex_on ();</strong><br />
666 fd_set accept_fds;<br />
668 FD_ZERO (&accept_fds);<br />
669 for (i = first_socket; i <= last_socket; ++i) {<br />
671 FD_SET (i, &accept_fds);<br />
674 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
675 if (rc < 1) continue;<br />
676 new_connection = -1;<br />
677 for (i = first_socket; i <= last_socket; ++i) {<br />
679 if (FD_ISSET (i, &accept_fds)) {<br />
681 new_connection = accept (i, NULL, NULL);<br />
682 if (new_connection != -1) break;<br />
687 if (new_connection != -1) break;<br />
690 <strong>accept_mutex_off ();</strong><br />
691 process the new_connection;<br />
696 <p><a id="serialize" name="serialize">The functions</a>
697 <code>accept_mutex_on</code> and <code>accept_mutex_off</code>
698 implement a mutual exclusion semaphore. Only one child can have
699 the mutex at any time. There are several choices for
700 implementing these mutexes. The choice is defined in
701 <code>src/conf.h</code> (pre-1.3) or
702 <code>src/include/ap_config.h</code> (1.3 or later). Some
703 architectures do not have any locking choice made, on these
704 architectures it is unsafe to use multiple
705 <directive module="mpm_common">Listen</directive>
708 <p>The directive <directive
709 module="mpm_common">AcceptMutex</directive> can be used to
710 change the selected mutex implementation at run-time.</p>
713 <dt><code>AcceptMutex flock</code></dt>
716 <p>This method uses the <code>flock(2)</code> system call to
717 lock a lock file (located by the <directive module="mpm_common"
718 >LockFile</directive> directive).</p>
721 <dt><code>AcceptMutex fcntl</code></dt>
724 <p>This method uses the <code>fcntl(2)</code> system call to
725 lock a lock file (located by the <directive module="mpm_common"
726 >LockFile</directive> directive).</p>
729 <dt><code>AcceptMutex sysvsem</code></dt>
732 <p>(1.3 or later) This method uses SysV-style semaphores to
733 implement the mutex. Unfortunately SysV-style semaphores have
734 some bad side-effects. One is that it's possible Apache will
735 die without cleaning up the semaphore (see the
736 <code>ipcs(8)</code> man page). The other is that the
737 semaphore API allows for a denial of service attack by any
738 CGIs running under the same uid as the webserver
739 (<em>i.e.</em>, all CGIs, unless you use something like
740 <program>suexec</program> or <code>cgiwrapper</code>). For these
741 reasons this method is not used on any architecture except
742 IRIX (where the previous two are prohibitively expensive
743 on most IRIX boxes).</p>
746 <dt><code>AcceptMutex pthread</code></dt>
749 <p>(1.3 or later) This method uses POSIX mutexes and should
750 work on any architecture implementing the full POSIX threads
751 specification, however appears to only work on Solaris (2.5
752 or later), and even then only in certain configurations. If
753 you experiment with this you should watch out for your server
754 hanging and not responding. Static content only servers may
758 <dt><code>AcceptMutex posixsem</code></dt>
761 <p>(2.0 or later) This method uses POSIX semaphores. The
762 semaphore ownership is not recovered if a thread in the process
763 holding the mutex segfaults, resulting in a hang of the web
769 <p>If your system has another method of serialization which
770 isn't in the above list then it may be worthwhile adding code
773 <p>Another solution that has been considered but never
774 implemented is to partially serialize the loop -- that is, let
775 in a certain number of processes. This would only be of
776 interest on multiprocessor boxes where it's possible multiple
777 children could run simultaneously, and the serialization
778 actually doesn't take advantage of the full bandwidth. This is
779 a possible area of future investigation, but priority remains
780 low because highly parallel web servers are not the norm.</p>
782 <p>Ideally you should run servers without multiple
783 <directive module="mpm_common">Listen</directive>
784 statements if you want the highest performance.
791 <title>accept Serialization - single socket</title>
793 <p>The above is fine and dandy for multiple socket servers, but
794 what about single socket servers? In theory they shouldn't
795 experience any of these same problems because all children can
796 just block in <code>accept(2)</code> until a connection
797 arrives, and no starvation results. In practice this hides
798 almost the same "spinning" behaviour discussed above in the
799 non-blocking solution. The way that most TCP stacks are
800 implemented, the kernel actually wakes up all processes blocked
801 in <code>accept</code> when a single connection arrives. One of
802 those processes gets the connection and returns to user-space,
803 the rest spin in the kernel and go back to sleep when they
804 discover there's no connection for them. This spinning is
805 hidden from the user-land code, but it's there nonetheless.
806 This can result in the same load-spiking wasteful behaviour
807 that a non-blocking solution to the multiple sockets case
810 <p>For this reason we have found that many architectures behave
811 more "nicely" if we serialize even the single socket case. So
812 this is actually the default in almost all cases. Crude
813 experiments under Linux (2.0.30 on a dual Pentium pro 166
814 w/128Mb RAM) have shown that the serialization of the single
815 socket case causes less than a 3% decrease in requests per
816 second over unserialized single-socket. But unserialized
817 single-socket showed an extra 100ms latency on each request.
818 This latency is probably a wash on long haul lines, and only an
819 issue on LANs. If you want to override the single socket
820 serialization you can define
821 <code>SINGLE_LISTEN_UNSERIALIZED_ACCEPT</code> and then
822 single-socket servers will not serialize at all.</p>
828 <title>Lingering Close</title>
830 <p>As discussed in <a
831 href="http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt">
832 draft-ietf-http-connection-00.txt</a> section 8, in order for
833 an HTTP server to <strong>reliably</strong> implement the
834 protocol it needs to shutdown each direction of the
835 communication independently (recall that a TCP connection is
836 bi-directional, each half is independent of the other). This
837 fact is often overlooked by other servers, but is correctly
838 implemented in Apache as of 1.2.</p>
840 <p>When this feature was added to Apache it caused a flurry of
841 problems on various versions of Unix because of a
842 shortsightedness. The TCP specification does not state that the
843 <code>FIN_WAIT_2</code> state has a timeout, but it doesn't prohibit it.
844 On systems without the timeout, Apache 1.2 induces many sockets
845 stuck forever in the <code>FIN_WAIT_2</code> state. In many cases this
846 can be avoided by simply upgrading to the latest TCP/IP patches
847 supplied by the vendor. In cases where the vendor has never
848 released patches (<em>i.e.</em>, SunOS4 -- although folks with
849 a source license can patch it themselves) we have decided to
850 disable this feature.</p>
852 <p>There are two ways of accomplishing this. One is the socket
853 option <code>SO_LINGER</code>. But as fate would have it, this
854 has never been implemented properly in most TCP/IP stacks. Even
855 on those stacks with a proper implementation (<em>i.e.</em>,
856 Linux 2.0.31) this method proves to be more expensive (cputime)
857 than the next solution.</p>
859 <p>For the most part, Apache implements this in a function
860 called <code>lingering_close</code> (in
861 <code>http_main.c</code>). The function looks roughly like
865 void lingering_close (int s)<br />
868 char junk_buffer[2048];<br />
870 /* shutdown the sending side */<br />
871 shutdown (s, 1);<br />
873 signal (SIGALRM, lingering_death);<br />
878 select (s for reading, 2 second timeout);<br />
879 if (error) break;<br />
880 if (s is ready for reading) {<br />
882 if (read (s, junk_buffer, sizeof (junk_buffer)) <= 0) {<br />
887 /* just toss away whatever is here */<br />
898 <p>This naturally adds some expense at the end of a connection,
899 but it is required for a reliable implementation. As HTTP/1.1
900 becomes more prevalent, and all connections are persistent,
901 this expense will be amortized over more requests. If you want
902 to play with fire and disable this feature you can define
903 <code>NO_LINGCLOSE</code>, but this is not recommended at all.
904 In particular, as HTTP/1.1 pipelined persistent connections
905 come into use <code>lingering_close</code> is an absolute
907 href="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
908 pipelined connections are faster</a>, so you want to support
915 <title>Scoreboard File</title>
917 <p>Apache's parent and children communicate with each other
918 through something called the scoreboard. Ideally this should be
919 implemented in shared memory. For those operating systems that
920 we either have access to, or have been given detailed ports
921 for, it typically is implemented using shared memory. The rest
922 default to using an on-disk file. The on-disk file is not only
923 slow, but it is unreliable (and less featured). Peruse the
924 <code>src/main/conf.h</code> file for your architecture and
925 look for either <code>USE_MMAP_SCOREBOARD</code> or
926 <code>USE_SHMGET_SCOREBOARD</code>. Defining one of those two
927 (as well as their companions <code>HAVE_MMAP</code> and
928 <code>HAVE_SHMGET</code> respectively) enables the supplied
929 shared memory code. If your system has another type of shared
930 memory, edit the file <code>src/main/http_main.c</code> and add
931 the hooks necessary to use it in Apache. (Send us back a patch
934 <note>Historical note: The Linux port of Apache didn't start to
935 use shared memory until version 1.2 of Apache. This oversight
936 resulted in really poor and unreliable behaviour of earlier
937 versions of Apache on Linux.</note>
943 <title>DYNAMIC_MODULE_LIMIT</title>
945 <p>If you have no intention of using dynamically loaded modules
946 (you probably don't if you're reading this and tuning your
947 server for every last ounce of performance) then you should add
948 <code>-DDYNAMIC_MODULE_LIMIT=0</code> when building your
949 server. This will save RAM that's allocated only for supporting
950 dynamically loaded modules.</p>
958 <title>Appendix: Detailed Analysis of a Trace</title>
960 <p>Here is a system call trace of Apache 2.0.38 with the worker MPM
961 on Solaris 8. This trace was collected using:</p>
964 truss -l -p <var>httpd_child_pid</var>.
967 <p>The <code>-l</code> option tells truss to log the ID of the
968 LWP (lightweight process--Solaris's form of kernel-level thread)
969 that invokes each system call.</p>
971 <p>Other systems may have different system call tracing utilities
972 such as <code>strace</code>, <code>ktrace</code>, or <code>par</code>.
973 They all produce similar output.</p>
975 <p>In this trace, a client has requested a 10KB static file
976 from the httpd. Traces of non-static requests or requests
977 with content negotiation look wildly different (and quite ugly
981 <pre>/67: accept(3, 0x00200BEC, 0x00200C0C, 1) (sleeping...)
982 /67: accept(3, 0x00200BEC, 0x00200C0C, 1) = 9</pre>
985 <p>In this trace, the listener thread is running within LWP #67.</p>
987 <note>Note the lack of <code>accept(2)</code> serialization. On this
988 particular platform, the worker MPM uses an unserialized accept by
989 default unless it is listening on multiple ports.</note>
992 <pre>/65: lwp_park(0x00000000, 0) = 0
993 /67: lwp_unpark(65, 1) = 0</pre>
996 <p>Upon accepting the connection, the listener thread wakes up
997 a worker thread to do the request processing. In this trace,
998 the worker thread that handles the request is mapped to LWP #65.</p>
1001 <pre>/65: getsockname(9, 0x00200BA4, 0x00200BC4, 1) = 0</pre>
1004 <p>In order to implement virtual hosts, Apache needs to know
1005 the local socket address used to accept the connection. It
1006 is possible to eliminate this call in many situations (such
1007 as when there are no virtual hosts, or when
1008 <directive module="mpm_common">Listen</directive> directives
1009 are used which do not have wildcard addresses). But
1010 no effort has yet been made to do these optimizations. </p>
1013 <pre>/65: brk(0x002170E8) = 0
1014 /65: brk(0x002190E8) = 0</pre>
1017 <p>The <code>brk(2)</code> calls allocate memory from the heap.
1018 It is rare to see these in a system call trace, because the httpd
1019 uses custom memory allocators (<code>apr_pool</code> and
1020 <code>apr_bucket_alloc</code>) for most request processing.
1021 In this trace, the httpd has just been started, so it must
1022 call <code>malloc(3)</code> to get the blocks of raw memory
1023 with which to create the custom memory allocators.</p>
1026 <pre>/65: fcntl(9, F_GETFL, 0x00000000) = 2
1027 /65: fstat64(9, 0xFAF7B818) = 0
1028 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B910, 2190656) = 0
1029 /65: fstat64(9, 0xFAF7B818) = 0
1030 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B914, 2190656) = 0
1031 /65: setsockopt(9, 65535, 8192, 0xFAF7B918, 4, 2190656) = 0
1032 /65: fcntl(9, F_SETFL, 0x00000082) = 0</pre>
1035 <p>Next, the worker thread puts the connection to the client (file
1036 descriptor 9) in non-blocking mode. The <code>setsockopt(2)</code>
1037 and <code>getsockopt(2)</code> calls are a side-effect of how
1038 Solaris's libc handles <code>fcntl(2)</code> on sockets.</p>
1041 <pre>/65: read(9, " G E T / 1 0 k . h t m".., 8000) = 97</pre>
1044 <p>The worker thread reads the request from the client.</p>
1047 <pre>/65: stat("/var/httpd/apache/httpd-8999/htdocs/10k.html", 0xFAF7B978) = 0
1048 /65: open("/var/httpd/apache/httpd-8999/htdocs/10k.html", O_RDONLY) = 10</pre>
1051 <p>This httpd has been configured with <code>Options FollowSymLinks</code>
1052 and <code>AllowOverride None</code>. Thus it doesn't need to
1053 <code>lstat(2)</code> each directory in the path leading up to the
1054 requested file, nor check for <code>.htaccess</code> files.
1055 It simply calls <code>stat(2)</code> to verify that the file:
1056 1) exists, and 2) is a regular file, not a directory.</p>
1059 <pre>/65: sendfilev(0, 9, 0x00200F90, 2, 0xFAF7B53C) = 10269</pre>
1062 <p>In this example, the httpd is able to send the HTTP response
1063 header and the requested file with a single <code>sendfilev(2)</code>
1064 system call. Sendfile semantics vary among operating systems. On some other
1065 systems, it is necessary to do a <code>write(2)</code> or
1066 <code>writev(2)</code> call to send the headers before calling
1067 <code>sendfile(2)</code>.</p>
1070 <pre>/65: write(4, " 1 2 7 . 0 . 0 . 1 - ".., 78) = 78</pre>
1073 <p>This <code>write(2)</code> call records the request in the
1074 access log. Note that one thing missing from this trace is a
1075 <code>time(2)</code> call. Unlike Apache 1.3, Apache 2.x uses
1076 <code>gettimeofday(3)</code> to look up the time. On some operating
1077 systems, like Linux or Solaris, <code>gettimeofday</code> has an
1078 optimized implementation that doesn't require as much overhead
1079 as a typical system call.</p>
1082 <pre>/65: shutdown(9, 1, 1) = 0
1083 /65: poll(0xFAF7B980, 1, 2000) = 1
1084 /65: read(9, 0xFAF7BC20, 512) = 0
1085 /65: close(9) = 0</pre>
1088 <p>The worker thread does a lingering close of the connection.</p>
1091 <pre>/65: close(10) = 0
1092 /65: lwp_park(0x00000000, 0) (sleeping...)</pre>
1095 <p>Finally the worker thread closes the file that it has just delivered
1096 and blocks until the listener assigns it another connection.</p>
1099 <pre>/67: accept(3, 0x001FEB74, 0x001FEB94, 1) (sleeping...)</pre>
1102 <p>Meanwhile, the listener thread is able to accept another connection
1103 as soon as it has dispatched this connection to a worker thread (subject
1104 to some flow-control logic in the worker MPM that throttles the listener
1105 if all the available workers are busy). Though it isn't apparent from
1106 this trace, the next <code>accept(2)</code> can (and usually does, under
1107 high load conditions) occur in parallel with the worker thread's handling
1108 of the just-accepted connection.</p>