1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V4.1//EN">
4 <?dbhtml banner-text="Made possible by PowerDNS">
5 <?dbhtml banner-href="http://www.powerdns.com">
8 <Title>Linux Advanced Routing & Traffic Control HOWTO</Title>
11 <FirstName>Bert</FirstName><Surname>Hubert</Surname>
13 <orgname>Netherlabs BV</orgname>
14 <address><email>bert.hubert@netherlabs.nl</email></address>
19 <collabname>Gregory Maxwell</collabname>
21 <address><email>greg@linuxpower.cx</email></address>
26 <collabname>Remco van Mook</collabname>
28 <address><email>remco@virtu.nl</email></address>
33 <collabname>Martijn van Oosterhout</collabname>
35 <address><email>kleptog@cupid.suninternet.com</email></address>
40 <collabname>Paul B Schroeder</collabname>
42 <address><email>paulsch@us.ibm.com</email></address>
47 <collabname>Jasper Spaans</collabname>
49 <address><email>jasper@spaans.ds9a.nl</email></address>
56 <revnumber role="rcs">$Revision$</revnumber>
57 <date role="rcs">$Date$</date>
58 <revremark>DocBook Edition</revremark>
63 <Para>A very hands-on approach to <application>iproute2</application>,
64 traffic shaping and a bit of <application>netfilter</application>.
70 <chapter id="lartc.dedication">
71 <Title>Dedication</Title>
74 This document is dedicated to lots of people, and is my attempt to do
75 something back. To list but a few:
95 The good folks from Google
101 The staff of Casema Internet
111 <chapter id="lartc.intro">
112 <Title>Introduction</Title>
115 Welcome, gentle reader.
119 This document hopes to enlighten you on how to do more with Linux 2.2/2.4
120 routing. Unbeknownst to most users, you already run tools which allow you to
121 do spectacular things. Commands like <command>route</command> and
122 <command>ifconfig</command> are actually
123 very thin wrappers for the very powerful iproute2 infrastructure.
127 I hope that this HOWTO will become as readable as the ones by Rusty Russell
128 of (amongst other things) netfilter fame.
132 You can always reach us by writing to the <ULink
133 URL="mailto:HOWTO@ds9a.nl"
135 >. However, please consider posting to the mailing
136 list (see the relevant section) if you have questions which are not directly
137 related to this HOWTO. We are no free helpdesk, but we often will answer questions
142 Before losing your way in this HOWTO, if all you want to do is simple
143 traffic shaping, skip everything and head to the <citetitle><xref linkend="lartc.other"></citetitle> chapter, and read about CBQ.init.
146 <Sect1 id="lartc.intro.disclaimer">
147 <Title>Disclaimer & License</Title>
150 This document is distributed in the hope that it will be useful,
151 but WITHOUT ANY WARRANTY; without even the implied warranty of
152 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
156 In short, if your STM-64 backbone breaks down and distributes pornography to
157 your most esteemed customers - it's never our fault. Sorry.
161 Copyright (c) 2002 by bert hubert, Gregory Maxwell, Martijn van
162 Oosterhout, Remco van Mook, Paul B. Schroeder and others. This material may
163 be distributed only subject to the terms and conditions set forth in the
164 Open Publication License, v1.0 or later (the latest version is presently
165 available at http://www.opencontent.org/openpub/).
169 Please freely copy and distribute (sell or give away) this document in any
170 format. It's requested that corrections and/or comments be forwarded to the
175 It is also requested that if you publish this HOWTO in hardcopy that you
176 send the authors some samples for <quote>review purposes</quote> :-)
181 <Sect1 id="lartc.intro.prior">
182 <Title>Prior knowledge</Title>
185 As the title implies, this is the <quote>Advanced</quote> HOWTO.
186 While by no means rocket science, some prior knowledge is assumed.
190 Here are some other references which might help teach you more:
194 <ULink URL="http://netfilter.samba.org/unreliable-guides/networking-concepts-HOWTO/index.html">
195 Rusty Russell's networking-concepts-HOWTO</ULink>
198 <Para>Very nice introduction, explaining what a network is, and how it is
199 connected to other networks.
204 <Term>Linux Networking-HOWTO (Previously the Net-3 HOWTO)</Term>
206 <Para>Great stuff, although very verbose. It teaches you a lot of stuff
207 that's already configured if you are able to connect to the Internet.
208 Should be located in <filename>/usr/doc/HOWTO/NET3-4-HOWTO.txt</filename>
209 but can be also be found
210 <ULink URL="http://www.linuxports.com/howto/networking">online</ULink>.
219 <Sect1 id="lartc.intro.linux">
220 <Title>What Linux can do for you</Title>
223 A small list of things that are possible:
228 <Para>Throttle bandwidth for certain computers
232 <Para>Throttle bandwidth TO certain computers
236 <Para>Help you to fairly share your bandwidth
240 <Para>Protect your network from DoS attacks
244 <Para>Protect the Internet from your customers
248 <Para>Multiplex several servers as one, for load balancing or
249 enhanced availability
253 <Para>Restrict access to your computers
257 <Para>Limit access of your users to other hosts
261 <Para>Do routing based on user id (yes!), MAC address, source IP
262 address, port, type of service, time of day or content
268 Currently, not many people are using these advanced features. This is for
269 several reasons. While the provided documentation is verbose, it is not very
270 hands-on. Traffic control is almost undocumented.
275 <Sect1 id="lartc.intro.houskeeping">
276 <Title>Housekeeping notes</Title>
279 There are several things which should be noted about this document. While I
280 wrote most of it, I really don't want it to stay that way. I am a strong
281 believer in Open Source, so I encourage you to send feedback, updates,
282 patches etcetera. Do not hesitate to inform me of typos or plain old errors.
283 If my English sounds somewhat wooden, please realize that I'm not a native
284 speaker. Feel free to send suggestions.
288 If you feel to you are better qualified to maintain a section, or think that
289 you can author and maintain new sections, you are welcome to do so. The SGML
290 of this HOWTO is available via CVS, I very much envision more people
295 In aid of this, you will find lots of FIXME notices. Patches are always
296 welcome! Wherever you find a FIXME, you should know that you are treading in
297 unknown territory. This is not to say that there are no errors elsewhere,
298 but be extra careful. If you have validated something, please let us know so
299 we can remove the FIXME notice.
303 About this HOWTO, I will take some liberties along the road. For example, I
304 postulate a 10Mbit Internet connection, while I know full well that those
310 <Sect1 id="lartc.intro.cvs">
311 <Title>Access, CVS & submitting updates</Title>
314 The canonical location for the HOWTO is
315 <ULink URL="http://www.ds9a.nl/lartc">here</ULink>.
319 We now have anonymous CVS access available to the world at large. This is
320 good in a number of ways. You can easily upgrade to newer versions of this
321 HOWTO and submitting patches is no work at all.
325 Furthermore, it allows the authors to work on the source independently,
330 $ export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
332 CVS password: [enter 'cvs' (without 's)]
334 cvs server: Updating 2.4routing
335 U 2.4routing/lartc.db
339 If you made changes and want to contribute them, run <userinput>
340 cvs -z3 diff -uBb</userinput>,
341 and mail the output to <email>howto@ds9a.nl</email>, we
342 can then integrate it easily. Thanks! Please make sure that you edit the
343 .db file, by the way, the other files are generated from that one.
347 A Makefile is supplied which should help you create postscript, dvi, pdf,
348 html and plain text. You may need to install
349 <application>docbook</application>, <application>docbook-utils</application>,
350 <application>ghostscript</application> and <application>tetex</application>
355 Be careful not to edit 2.4routing.sgml! It contains an older version of the
356 HOWTO. The right file is lartc.db.
360 <Sect1 id="lartc.intro.mlist">
361 <Title>Mailing list</Title>
364 The authors receive an increasing amount of mail about this HOWTO. Because
365 of the clear interest of the community, it has been decided to start a
366 mailinglist where people can talk to each other about Advanced Routing and
367 Traffic Control. You can subscribe to the list
368 <ULink URL="http://mailman.ds9a.nl/mailman/listinfo/lartc">here</ULink>.
372 It should be pointed out that the authors are very hesitant of answering
373 questions not asked on the list. We would like the archive of the list to
374 become some kind of knowledge base. If you have a question, please search
375 the archive, and then post to the mailinglist.
380 <Sect1 id="lartc.intro.layout">
381 <Title>Layout of this document</Title>
384 We will be doing interesting stuff almost immediately, which also means that
385 there will initially be parts that are explained incompletely or are not
386 perfect. Please gloss over these parts and assume that all will become clear.
390 Routing and filtering are two distinct things. Filtering is documented very
391 well by Rusty's HOWTOs, available here:
396 <Para><ULink URL="http://netfilter.samba.org/unreliable-guides/">
397 Rusty's Remarkably Unreliable Guides</ULink>
402 <Para>We will be focusing mostly on what is possible by combining netfilter
410 <chapter id="lartc.iproute2">
411 <Title>Introduction to iproute2</Title>
413 <Sect1 id="lartc.iproute2.why">
414 <Title>Why iproute2?</Title>
417 Most Linux distributions, and most UNIX's, currently use the
418 venerable <command>arp</command>, <command>ifconfig</command> and
419 <command>route</command> commands.
420 While these tools work, they show some unexpected behaviour under Linux 2.2
422 For example, GRE tunnels are an integral part of routing these days, but
423 require completely different tools.
427 With <application>iproute2</application>, tunnels are an integral part of
432 The 2.2 and above Linux kernels include a completely redesigned network
433 subsystem. This new networking code brings Linux performance and a feature
434 set with little competition in the general OS arena. In fact, the new
435 routing, filtering, and classifying code is more featureful than the one
436 provided by many dedicated routers and firewalls and traffic shaping
441 As new networking concepts have been invented, people have found ways to
442 plaster them on top of the existing framework in existing OSes. This
443 constant layering of cruft has lead to networking code that is filled with
444 strange behaviour, much like most human languages. In the past, Linux
445 emulated SunOS's handling of many of these things, which was not ideal.
449 This new framework makes it possible to clearly express features
450 previously beyond Linux's reach.
455 <Sect1 id="lartc.iproute2.tour">
456 <Title>iproute2 tour</Title>
459 Linux has a sophisticated system for bandwidth provisioning called Traffic
460 Control. This system supports various method for classifying, prioritizing,
461 sharing, and limiting both inbound and outbound traffic.
465 We'll start off with a tiny tour of iproute2 possibilities.
470 <Sect1 id="lartc.iproute2.package">
471 <Title>Prerequisites</Title>
474 You should make sure that you have the userland tools installed. This
475 package is called 'iproute' on both RedHat and Debian, and may otherwise be
476 found at <filename>ftp://ftp.inr.ac.ru/ip-routing/iproute2-2.2.4-now-ss??????.tar.gz"</filename>.
481 <ULink URL="ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz">here</ULink>
482 for the latest version.
486 Some parts of iproute require you to have certain kernel options enabled. It
487 should also be noted that all releases of RedHat up to and including 6.2
488 come without most of the traffic control features in the default kernel.
492 RedHat 7.2 has everything in by default.
496 Also make sure that you have netlink support, should you choose to roll your
497 own kernel. Iproute2 needs it.
502 <Sect1 id="lartc.iproute2.explore">
503 <Title>Exploring your current configuration</Title>
506 This may come as a surprise, but iproute2 is already configured! The current
507 commands <command>ifconfig</command> and <command>route</command> are already using the advanced
508 syscalls, but mostly with very default (ie. boring) settings.
512 The <command>ip</command> tool is central, and we'll ask it to display our interfaces
517 <Title><command>ip</command> shows us our links</Title>
520 [ahu@home ahu]$ ip link list
521 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
522 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
523 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
524 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
525 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
526 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
527 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
528 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
529 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
535 Your mileage may vary, but this is what it shows on my NAT router at
536 home. I'll only explain part of the output as not everything is directly
541 We first see the loopback interface. While your computer may function
542 somewhat without one, I'd advise against it. The MTU size (Maximum Transfer
543 Unit) is 3924 octets, and it is not supposed to queue. Which makes sense
544 because the loopback interface is a figment of your kernel's imagination.
548 I'll skip the dummy interface for now, and it may not be present on your
549 computer. Then there are my two physical network interfaces, one at the side
550 of my cable modem, the other one serves my home ethernet segment.
551 Furthermore, we see a ppp0 interface.
555 Note the absence of IP addresses. iproute disconnects the concept of 'links'
556 and 'IP addresses'. With IP aliasing, the concept of 'the' IP address had
557 become quite irrelevant anyhow.
561 It does show us the MAC addresses though, the hardware identifier of our
568 <Title><command>ip</command> shows us our IP addresses</Title>
571 [ahu@home ahu]$ ip address show
572 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
573 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
574 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
575 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
576 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
577 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
578 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
579 inet 10.0.0.1/8 brd 10.255.255.255 scope global eth0
580 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
581 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
582 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
584 inet 212.64.94.251 peer 212.64.94.1/32 scope global ppp0
588 This contains more information. It shows all our addresses, and to which
589 cards they belong. 'inet' stands for Internet (IPv4). There are lots of other
590 address families, but these don't concern us right now.
594 Let's examine eth0 somewhat closer. It says that it is related to the inet
595 address '10.0.0.1/8'. What does this mean? The /8 stands for the number of
596 bits that are in the Network Address. There are 32 bits, so we have 24 bits
597 left that are part of our network. The first 8 bits of 10.0.0.1 correspond
598 to 10.0.0.0, our Network Address, and our netmask is 255.0.0.0.
602 The other bits are connected to this interface, so 10.250.3.13 is directly
603 available on eth0, as is 10.0.0.1 for example.
607 With ppp0, the same concept goes, though the numbers are different. Its
608 address is 212.64.94.251, without a subnet mask. This means that we have a
609 point-to-point connection and that every address, with the exception of
610 212.64.94.251, is remote. There is more information, however. It tells us
611 that on the other side of the link there is, yet again, only one address,
612 212.64.94.1. The /32 tells us that there are no 'network bits'.
616 It is absolutely vital that you grasp these concepts. Refer to the
617 documentation mentioned at the beginning of this HOWTO if you have trouble.
621 You may also note 'qdisc', which stands for Queueing Discipline. This will
622 become vital later on.
628 <Title><command>ip</command> shows us our routes</Title>
631 Well, we now know how to find 10.x.y.z addresses, and we are able to reach
632 212.64.94.1. This is not enough however, so we need instructions on how to
633 reach the world. The Internet is available via our ppp connection, and it
634 appears that 212.64.94.1 is willing to spread our packets around the
635 world, and deliver results back to us.
639 [ahu@home ahu]$ ip route show
640 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
641 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
642 127.0.0.0/8 dev lo scope link
643 default via 212.64.94.1 dev ppp0
647 This is pretty much self explanatory. The first 4 lines of output explicitly
648 state what was already implied by <command>ip address show</command>, the last line
649 tells us that the rest of the world can be found via 212.64.94.1, our
650 default gateway. We can see that it is a gateway because of the word
651 via, which tells us that we need to send packets to 212.64.94.1, and that it
652 will take care of things.
656 For reference, this is what the old <command>route</command> utility shows us:
660 [ahu@home ahu]$ route -n
661 Kernel IP routing table
662 Destination Gateway Genmask Flags Metric Ref Use
664 212.64.94.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0
665 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
666 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
667 0.0.0.0 212.64.94.1 0.0.0.0 UG 0 0 0 ppp0
674 <Sect1 id="lartc.iproute2.arp">
678 ARP is the Address Resolution Protocol as described in
679 <ULink URL="http://www.faqs.org/rfcs/rfc826.html">RFC 826</ULink>.
680 ARP is used by a networked machine to resolve the hardware location/address of
681 another machine on the same
682 local network. Machines on the Internet are generally known by their names
684 addresses. This is how a machine on the foo.com network is able to communicate
685 with another machine which is on the bar.net network. An IP address, though,
686 cannot tell you the physical location of a machine. This is where ARP comes
691 Let's take a very simple example. Suppose I have a network composed of several
692 machines. Two of the machines which are currently on my network are foo
693 with an IP address of 10.0.0.1 and bar with an IP address of 10.0.0.2.
694 Now foo wants to ping bar to see that he is alive, but alas, foo has no idea
695 where bar is. So when foo decides to ping bar he will need to send
697 This ARP request is akin to foo shouting out on the network "Bar (10.0.0.2)!
698 Where are you?" As a result of this every machine on the network will hear
699 foo shouting, but only bar (10.0.0.2) will respond. Bar will then send an
700 ARP reply directly back to foo which is akin
702 "Foo (10.0.0.1) I am here at 00:60:94:E9:08:12." After this simple transaction
703 that's used to locate his friend on the network, foo is able to communicate
704 with bar until he (his arp cache) forgets where bar is (typically after
709 Now let's see how this works.
710 You can view your machines current arp/neighbor cache/table like so:
714 [root@espa041 /home/src/iputils]# ip neigh show
715 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
716 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
720 As you can see my machine espa041 (9.3.76.41) knows where to find espa042
722 espagate (9.3.76.1). Now let's add another machine to the arp cache.
726 [root@espa041 /home/paulsch/.gnome-desktop]# ping -c 1 espa043
727 PING espa043.austin.ibm.com (9.3.76.43) from 9.3.76.41 : 56(84) bytes of data.
728 64 bytes from 9.3.76.43: icmp_seq=0 ttl=255 time=0.9 ms
730 --- espa043.austin.ibm.com ping statistics ---
731 1 packets transmitted, 1 packets received, 0% packet loss
732 round-trip min/avg/max = 0.9/0.9/0.9 ms
734 [root@espa041 /home/src/iputils]# ip neigh show
735 9.3.76.43 dev eth0 lladdr 00:06:29:21:80:20 nud reachable
736 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
737 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
741 As a result of espa041 trying to contact espa043, espa043's hardware
742 address/location has now been added to the arp/neighbor cache.
743 So until the entry for
744 espa043 times out (as a result of no communication between the two) espa041
745 knows where to find espa043 and has no need to send an ARP request.
749 Now let's delete espa043 from our arp cache:
753 [root@espa041 /home/src/iputils]# ip neigh delete 9.3.76.43 dev eth0
754 [root@espa041 /home/src/iputils]# ip neigh show
755 9.3.76.43 dev eth0 nud failed
756 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
757 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud stale
761 Now espa041 has again forgotten where to find espa043 and will need to send
762 another ARP request the next time he needs to communicate with espa043.
763 You can also see from the above output that espagate (9.3.76.1) has been
764 changed to the "stale" state. This means that the location shown is still
765 valid, but it will have to be confirmed at the first transaction to that
773 <chapter id="lartc.rpdb">
774 <Title>Rules - routing policy database</Title>
777 If you have a large router, you may well cater for the needs of different
778 people, who should be served differently. The routing policy database allows
779 you to do this by having multiple sets of routing tables.
783 If you want to use this feature, make sure that your kernel is compiled with
784 the "IP: advanced router" and "IP: policy routing" features.
788 When the kernel needs to make a routing decision, it finds out which table
789 needs to be consulted. By default, there are three tables. The old 'route'
790 tool modifies the main and local tables, as does the ip tool (by default).
793 <Para>The default rules:
797 [ahu@home ahu]$ ip rule list
798 0: from all lookup local
799 32766: from all lookup main
800 32767: from all lookup default
804 This lists the priority of all rules. We see that all rules apply to all
805 packets ('from all'). We've seen the 'main' table before, it is output by
806 <userinput>ip route ls</userinput>, but the 'local' and 'default' table are new.
810 If we want to do fancy things, we generate rules which point to different
811 tables which allow us to override system wide routing rules.
815 For the exact semantics on what the kernel does when there are more matching
816 rules, see Alexey's ip-cref documentation.
819 <Sect1 id="lartc.rpdb.simple">
820 <Title>Simple source policy routing</Title>
823 Let's take a real example once again, I have 2 (actually 3, about time I
824 returned them) cable modems, connected to a Linux NAT ('masquerading')
825 router. People living here pay me to use the Internet. Suppose one of my
826 house mates only visits hotmail and wants to pay less. This is fine with me,
827 but they'll end up using the low-end cable modem.
831 The 'fast' cable modem is known as 212.64.94.251 and is a PPP link to
832 212.64.94.1. The 'slow' cable modem is known by various ip addresses,
833 212.64.78.148 in this example and is a link to 195.96.98.253.
836 <Para>The local table:
840 [ahu@home ahu]$ ip route list table local
841 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
842 local 10.0.0.1 dev eth0 proto kernel scope host src 10.0.0.1
843 broadcast 10.0.0.0 dev eth0 proto kernel scope link src 10.0.0.1
844 local 212.64.94.251 dev ppp0 proto kernel scope host src 212.64.94.251
845 broadcast 10.255.255.255 dev eth0 proto kernel scope link src 10.0.0.1
846 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
847 local 212.64.78.148 dev ppp2 proto kernel scope host src 212.64.78.148
848 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
849 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
853 Lots of obvious things, but things that need to be specified somewhere.
854 Well, here they are. The default table is empty.
857 <Para>Let's view the 'main' table:
861 [ahu@home ahu]$ ip route list table main
862 195.96.98.253 dev ppp2 proto kernel scope link src 212.64.78.148
863 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
864 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
865 127.0.0.0/8 dev lo scope link
866 default via 212.64.94.1 dev ppp0
870 We now generate a new rule which we call 'John', for our hypothetical
871 house mate. Although we can work with pure numbers, it's far easier if we add
872 our tables to /etc/iproute2/rt_tables.
876 # echo 200 John >> /etc/iproute2/rt_tables
877 # ip rule add from 10.0.0.10 table John
879 0: from all lookup local
880 32765: from 10.0.0.10 lookup John
881 32766: from all lookup main
882 32767: from all lookup default
886 Now all that is left is to generate John's table, and flush the route cache:
890 # ip route add default via 195.96.98.253 dev ppp2 table John
891 # ip route flush cache
895 And we are done. It is left as an exercise for the reader to implement this
901 <sect1 id="lartc.rpdb.multiple-links">
902 <title>Routing for multiple uplinks/providers</title>
904 A common configuration is the following, in which there are two providers
905 that connect a local network (or even a single machine) to the big Internet.
911 +-------------+ Provider 1 +-------
913 ___/ \_ +------+-------+ +------------+ |
916 | Local network -----+ Linux router | | Internet
919 \___/ +------+-------+ +------------+ |
921 +-------------+ Provider 2 +-------
923 +------------+ \________
927 There are usually two questions given this setup.
929 <sect2><title>Split access</title>
931 The first is how to route answers to packets coming in over a
932 particular provider, say Provider 1, back out again over that same provider.
935 Let us first set some symbolical names. Let <command>$IF1</command> be the name of the
936 first interface (if1 in the picture above) and <command>$IF2</command> the name of the
937 second interface. Then let <command>$IP1</command> be the IP address associated with
938 <command>$IF1</command> and <command>$IP2</command> the IP address associated with
939 <command>$IF2</command>. Next, let <command>$P1</command> be the IP address of the gateway at
940 Provider 1, and <command>$P2</command> the IP address of the gateway at provider 2.
941 Finally, let <command>$P1_NET</command> be the IP network <command>$P1</command> is in,
942 and <command>$P2_NET</command> the IP network <command>$P2</command> is in.
945 One creates two additional routing tables, say <command>T1</command> and <command>T2</command>.
946 These are added in /etc/iproute2/rt_tables. Then you set up routing in
947 these tables as follows:
951 ip route add $P1_NET dev $IF1 src $IP1 table T1
952 ip route add default via $P1 table T1
953 ip route add $P2_NET dev $IF2 src $IP2 table T2
954 ip route add default via $P2 table T2
957 Nothing spectacular, just build a route to the gateway and build a
958 default route via that gateway, as you would do in the case of a single
959 upstream provider, but put the routes in a separate table per provider.
960 Note that the network route suffices, as it tells you how to find any host
961 in that network, which includes the gateway, as specified above.
964 Next you set up the main routing table. It is a good idea to route
965 things to the direct neighbour through the interface connected to that
966 neighbour. Note the `src' arguments, they make sure the right outgoing IP
970 ip route add $P1_NET dev $IF1 src $IP1
971 ip route add $P2_NET dev $IF2 src $IP2
974 Then, your preference for default route:
977 ip route add default via $P1
980 Next, you set up the routing rules. These actually choose what routing table
981 to route with. You want to make sure that you route out a given
982 interface if you already have the corresponding source address:
985 ip rule add from $IP1 table T1
986 ip rule add from $IP2 table T2
989 This set of commands makes sure all answers to traffic coming in on a
990 particular interface get answered from that interface.
993 Now, this is just the very basic setup. It will work for all processes
994 running on the router itself, and for the local network, if it is
995 masqueraded. If it is not, then you either have IP space from both providers
996 or you are going to want to masquerade to one of the two providers. In both
997 cases you will want to add rules selecting which provider to route out from
998 based on the IP address of the machine in the local network.
1001 <sect2><title>Load balancing</title>
1003 The second question is how to balance traffic going out over the two providers.
1004 This is actually not hard if you already have set up split access as above.
1007 Instead of choosing one of the two providers as your default route,
1008 you now set up the default route to be a multipath route. In the default
1009 kernel this will balance routes over the two providers. It is done
1010 as follows (once more building on the example in the section on
1014 ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
1015 nexthop via $P2 dev $IF2 weight 1
1018 This will balance the routes over both providers. The <command>weight</command>
1019 parameters can be tweaked to favor one provider over the other.
1022 Note that balancing will not be perfect, as it is route based, and routes
1023 are cached. This means that routes to often-used sites will always
1024 be over the same provider.
1027 Furthermore, if you really want to do this, you probably also want to look
1028 at Julian Anastasov's patches at <ulink
1029 url="http://www.linuxvirtualserver.org/~julian/#routes">http://www.linuxvirtualserver.org/~julian/#routes
1030 </ulink>, Julian's route patch page. They will make things nicer to work with.
1036 <chapter id="lartc.tunnel">
1037 <Title>GRE and other tunnels</Title>
1040 There are 3 kinds of tunnels in Linux. There's IP in IP tunneling, GRE tunneling and tunnels that live outside the kernel (like, for example PPTP).
1043 <Sect1 id="lartc.tunnel.remarks">
1044 <Title>A few general remarks about tunnels:</Title>
1047 Tunnels can be used to do some very unusual and very cool stuff. They can
1048 also make things go horribly wrong when you don't configure them right.
1049 Don't point your default route to a tunnel device unless you know
1050 <Emphasis>EXACTLY</Emphasis> what you are doing :-). Furthermore, tunneling increases
1051 overhead, because it needs an extra set of IP headers. Typically this is 20
1052 bytes per packet, so if the normal packet size (MTU) on a network is 1500
1053 bytes, a packet that is sent through a tunnel can only be 1480 bytes big.
1054 This is not necessarily a problem, but be sure to read up on IP packet
1055 fragmentation/reassembly when you plan to connect large networks with
1056 tunnels. Oh, and of course, the fastest way to dig a tunnel is to dig at
1062 <Sect1 id="lartc.tunnel.ip-ip">
1063 <Title>IP in IP tunneling</Title>
1066 This kind of tunneling has been available in Linux for a long time. It requires 2 kernel modules,
1067 ipip.o and new_tunnel.o.
1071 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
1072 So we have network A:
1077 netmask 255.255.255.0
1081 <Para>The router has address 172.16.17.18 on network C.
1084 <Para>and network B:
1089 netmask 255.255.255.0
1093 <Para>The router has address 172.19.20.21 on network C.
1097 As far as network C is concerned, we assume that it will pass any packet sent
1098 from A to B and vice versa. You might even use the Internet for this.
1101 <Para>Here's what you do:
1104 <Para>First, make sure the modules are installed:
1112 <Para>Then, on the router of network A, you do the following:
1116 ifconfig tunl0 10.0.1.1 pointopoint 172.19.20.21
1117 route add -net 10.0.2.0 netmask 255.255.255.0 dev tunl0
1120 <Para>And on the router of network B:
1124 ifconfig tunl0 10.0.2.1 pointopoint 172.16.17.18
1125 route add -net 10.0.1.0 netmask 255.255.255.0 dev tunl0
1128 <Para>And if you're finished with your tunnel:
1135 <Para>Presto, you're done. You can't forward broadcast or IPv6 traffic through
1136 an IP-in-IP tunnel, though. You just connect 2 IPv4 networks that normally wouldn't be able to talk to each other, that's all. As far as compatibility goes, this code has been around a long time, so it's compatible all the way back to 1.3 kernels. Linux IP-in-IP tunneling doesn't work with other Operating Systems or routers, as far as I know. It's simple, it works. Use it if you have to, otherwise use GRE.
1141 <Sect1 id="lartc.tunnel.gre">
1142 <Title>GRE tunneling</Title>
1145 GRE is a tunneling protocol that was originally developed by Cisco, and it
1146 can do a few more things than IP-in-IP tunneling. For example, you can also
1147 transport multicast traffic and IPv6 through a GRE tunnel.
1151 In Linux, you'll need the ip_gre.o module.
1155 <Title>IPv4 Tunneling</Title>
1158 Let's do IPv4 tunneling first:
1162 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
1166 So we have network A:
1170 netmask 255.255.255.0
1174 The router has address 172.16.17.18 on network C.
1175 Let's call this network neta (ok, hardly original)
1183 netmask 255.255.255.0
1187 The router has address 172.19.20.21 on network C.
1188 Let's call this network netb (still not original)
1192 As far as network C is concerned, we assume that it will pass any packet sent
1193 from A to B and vice versa. How and why, we do not care.
1196 <Para>On the router of network A, you do the following:
1200 ip tunnel add netb mode gre remote 172.19.20.21 local 172.16.17.18 ttl 255
1202 ip addr add 10.0.1.1 dev netb
1203 ip route add 10.0.2.0/24 dev netb
1207 Let's discuss this for a bit. In line 1, we added a tunnel device, and
1208 called it netb (which is kind of obvious because that's where we want it to
1209 go). Furthermore we told it to use the GRE protocol (mode gre), that the
1210 remote address is 172.19.20.21 (the router at the other end), that our
1211 tunneling packets should originate from 172.16.17.18 (which allows your
1212 router to have several IP addresses on network C and let you decide which
1213 one to use for tunneling) and that the TTL field of the packet should be set
1218 The second line enables the device.
1222 In the third line we gave the newly born interface netb the address
1223 10.0.1.1. This is OK for smaller networks, but when you're starting up a
1224 mining expedition (LOTS of tunnels), you might want to consider using
1225 another IP range for tunneling interfaces (in this example, you could use
1230 In the fourth line we set the route for network B. Note the different notation for the netmask. If you're not familiar with this notation, here's how it works: you write out the netmask in binary form, and you count all the ones. If you don't know how to do that, just remember that 255.0.0.0 is /8, 255.255.0.0 is /16 and 255.255.255.0 is /24. Oh, and 255.255.254.0 is /23, in case you were wondering.
1234 But enough about this, let's go on with the router of network B.
1237 ip tunnel add neta mode gre remote 172.16.17.18 local 172.19.20.21 ttl 255
1239 ip addr add 10.0.2.1 dev neta
1240 ip route add 10.0.1.0/24 dev neta
1243 And when you want to remove the tunnel on router A:
1246 ip link set netb down
1250 Of course, you can replace netb with neta for router B.
1256 <Title>IPv6 Tunneling</Title>
1259 See Section 6 for a short bit about IPv6 Addresses.
1263 On with the tunnels.
1267 Let's assume that you have the following IPv6 network, and you want to connect it to 6bone, or a friend.
1273 Network 3ffe:406:5:1:5:a:2:1/96
1276 Your IPv4 address is 172.16.17.18, and the 6bone router has IPv4 address 172.22.23.24.
1282 ip tunnel add sixbone mode sit remote 172.22.23.24 local 172.16.17.18 ttl 255
1283 ip link set sixbone up
1284 ip addr add 3ffe:406:5:1:5:a:2:1/96 dev sixbone
1285 ip route add 3ffe::/15 dev sixbone
1291 Let's discuss this. In the first line, we created a tunnel device called sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it where to go to (remote) and where to come from (local). TTL is set to maximum, 255. Next, we made the device active (up). After that, we added our own network address, and set a route for 3ffe::/15 (which is currently all of 6bone) through the tunnel.
1295 GRE tunnels are currently the preferred type of tunneling. It's a standard that is also widely adopted outside the Linux community and therefore a Good Thing.
1302 <Sect1 id="lartc.tunnel.userland">
1303 <Title>Userland tunnels</Title>
1306 There are literally dozens of implementations of tunneling outside the kernel. Best known are of course PPP and PPTP, but there are lots more (some proprietary, some secure, some that don't even use IP) and that is really beyond the scope of this HOWTO.
1313 <chapter id="lartc.ipv6-tunnel">
1314 <Title>IPv6 tunneling with Cisco and/or 6bone</Title>
1317 By Marco Davids <marco@sara.nl>
1325 As far as I am concerned, this IPv6-IPv4 tunneling is not per definition
1326 GRE tunneling. You could tunnel IPv6 over IPv4 by means of GRE tunnel devices
1327 (GRE tunnels ANY to IPv4), but the device used here ("sit") only tunnels
1328 IPv6 over IPv4 and is therefore something different.
1331 <Sect1 id="lartc.tunnel-ipv6.addressing">
1332 <Title>IPv6 Tunneling</Title>
1335 This is another application of the tunneling capabilities of Linux. It is
1336 popular among the IPv6 early adopters, or pioneers if you like.
1337 The 'hands-on' example described below is certainly not the only way
1338 to do IPv6 tunneling. However, it is the method that is often used to tunnel
1339 between Linux and a Cisco IPv6 capable router and experience tells us that
1340 this is just the thing many people are after. Ten to one this applies to
1345 A short bit about IPv6 addresses:
1349 IPv6 addresses are, compared to IPv4 addresses, really big: 128 bits
1350 against 32 bits. And this provides us just with the thing we need: many, many
1351 IP-addresses: 340,282,266,920,938,463,463,374,607,431,768,211,465 to be
1352 precise. Apart from this, IPv6 (or IPng, for IP Next Generation) is supposed
1353 to provide for smaller routing tables on the Internet's backbone routers,
1354 simpler configuration of equipment, better security at the IP level and
1355 better support for QoS.
1359 An example: 2002:836b:9820:0000:0000:0000:836b:9886
1363 Writing down IPv6 addresses can be quite a burden. Therefore, to make
1364 life easier there are some rules:
1373 Don't use leading zeroes. Same as in IPv4.
1380 Use colons to separate every 16 bits or two bytes.
1387 When you have lots of consecutive zeroes,
1388 you can write this down as ::. You can only do this once in an
1389 address and only for quantities of 16 bits, though.
1398 The address 2002:836b:9820:0000:0000:0000:836b:9886 can be written down
1399 as 2002:836b:9820::836b:9886, which is somewhat friendlier.
1403 Another example, the address 3ffe:0000:0000:0000:0000:0020:34A1:F32C can be
1404 written down as 3ffe::20:34A1:F32C, which is a lot shorter.
1408 IPv6 is intended to be the successor of the current IPv4. Because it
1409 is relatively new technology, there is no worldwide native IPv6 network
1410 yet. To be able to move forward swiftly, the 6bone was introduced.
1414 Native IPv6 networks are connected to each other by encapsulating the IPv6
1415 protocol in IPv4 packets and sending them over the existing IPv4 infrastructure
1416 from one IPv6 site to another.
1420 That is precisely where the tunnel steps in.
1424 To be able to use IPv6, we should have a kernel that supports it. There
1425 are many good documents on how to achieve this. But it all comes down to
1432 Get yourself a recent Linux distribution, with suitable glibc.
1438 Then get yourself an up-to-date kernel source.
1444 If you are all set, then you can go ahead and compile an IPv6 capable
1451 Go to /usr/src/linux and type:
1463 Choose "Networking Options"
1469 Select "The IPv6 protocol", "IPv6: enable EUI-64 token format", "IPv6:
1470 disable provider based addresses"
1476 HINT: Don't go for the 'module' option. Often this won't work well.
1480 In other words, compile IPv6 as 'built-in' in your kernel.
1481 You can then save your config like usual and go ahead with compiling
1486 HINT: Before doing so, consider editing the Makefile:
1487 EXTRAVERSION = -x ; --> ; EXTRAVERSION = -x-IPv6
1491 There is a lot of good documentation about compiling and installing
1492 a kernel, however this document is about something else. If you run into
1493 problems at this stage, go and look for documentation about compiling a
1494 Linux kernel according to your own specifications.
1498 The file /usr/src/linux/README might be a good start.
1499 After you accomplished all this, and rebooted with your brand new kernel,
1500 you might want to issue an '/sbin/ifconfig -a' and notice the brand
1501 new 'sit0-device'. SIT stands for Simple Internet Transition. You may give
1502 yourself a compliment; you are now one major step closer to IP, the Next
1507 Now on to the next step. You want to connect your host, or maybe even
1508 your entire LAN to another IPv6 capable network. This might be the "6bone"
1509 that is setup especially for this particular purpose.
1513 Let's assume that you have the following IPv6 network: 3ffe:604:6:8::/64 and
1514 you want to connect it to 6bone, or a friend. Please note that the /64
1515 subnet notation works just like with regular IP addresses.
1519 Your IPv4 address is 145.100.24.181 and the 6bone router has IPv4 address
1524 # ip tunnel add sixbone mode sit remote 145.100.1.5 [local 145.100.24.181 ttl 255]
1525 # ip link set sixbone up
1526 # ip addr add 3FFE:604:6:7::2/126 dev sixbone
1527 # ip route add 3ffe::0/16 dev sixbone
1531 Let's discuss this. In the first line, we created a tunnel device called
1532 sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it
1533 where to go to (remote) and where to come from (local). TTL is set to
1538 Next, we made the device active (up). After that, we added our own network
1539 address, and set a route for 3ffe::/15 (which is currently all of 6bone)
1540 through the tunnel. If the particular machine you run this on is your IPv6
1541 gateway, then consider adding the following lines:
1545 # echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
1546 # /usr/local/sbin/radvd
1550 The latter, radvd is -like zebra- a router advertisement daemon, to
1551 support IPv6's autoconfiguration features. Search for it with your favourite
1552 search-engine if you like.
1553 You can check things like this:
1557 # /sbin/ip -f inet6 addr
1561 If you happen to have radvd running on your IPv6 gateway and boot your
1562 IPv6 capable Linux on a machine on your local LAN, you would be able to
1563 enjoy the benefits of IPv6 autoconfiguration:
1567 # /sbin/ip -f inet6 addr
1568 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue inet6 ::1/128 scope host
1570 3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
1571 inet6 3ffe:604:6:8:5054:4cff:fe01:e3d6/64 scope global dynamic
1572 valid_lft forever preferred_lft 604646sec inet6 fe80::5054:4cff:fe01:e3d6/10
1577 You could go ahead and configure your bind for IPv6 addresses. The A
1578 type has an equivalent for IPv6: AAAA. The in-addr.arpa's equivalent is:
1579 ip6.int. There's a lot of information available on this topic.
1583 There is an increasing number of IPv6-aware applications available,
1584 including secure shell, telnet, inetd, Mozilla the browser, Apache the
1585 webserver and a lot of others. But this is all outside the scope of this
1586 Routing document ;-)
1590 On the Cisco side the configuration would be something like this:
1595 description IPv6 tunnel
1597 no ip directed-broadcast
1599 ipv6 address 3FFE:604:6:7::1/126
1600 tunnel source Serial0
1601 tunnel destination 145.100.24.181
1604 ipv6 route 3FFE:604:6:8::/64 Tunnel1
1607 But if you don't have a Cisco at your disposal, try one of the many
1608 IPv6 tunnel brokers available on the Internet. They are willing to configure
1609 their Cisco with an extra tunnel for you. Mostly by means of a friendly
1610 web interface. Search for "ipv6 tunnel broker" on your favourite search engine.
1617 <chapter id="lartc.ipsec">
1618 <Title>IPSEC: secure IP over the Internet</Title>
1620 There are two kinds of IPSEC available for Linux these days. For 2.2 and 2.4, there is FreeS/WAN. They
1621 have <ULink URL="http://www.freeswan.org/">an official site</ulink> and <ulink url="http://www.freeswan.ca">
1622 an unofficial one</ulink> that is actually maintained. FreeS/WAN has traditionally not been merged with
1623 the mainline kernel for a number of reasons. Most often mentioned are 'political' issues with Americans
1624 working on crypto tainting its exportability. Furthermore, it does not integrate too well with the Linux kernel,
1625 leading it to be a bad candidate for actual merging.
1628 Additionally, many parties have voiced worries about the quality of the code.
1631 As of Linux 2.5.47, there is a native IPSEC implementation in the kernel. It was written by Alexey Kuznetsov and
1632 Dave Miller, inspired by the work of the USAGI IPv6 group. With its merge, James Morris' CrypoAPI also became
1633 part of the kernel - it does the actual crypting.
1636 This HOWTO will only document the 2.5+ version of IPSEC. FreeS/WAN is recommended for Linux 2.4 users for now, but be aware
1637 that its configuration wil differ from the native IPSEC.
1642 As this is written, 2.5.47 has not even been released! A lot of what follows is likely to change. To try this with 2.5.46,
1643 fetch patches from the linux kernel mailinglist. Crude userspace utilities are <ulink url="ftp://ftp.inr.ac.ru/ip-routing/ipsec">
1649 First, we'll show how to setup secure communication between two hosts. This sample can be made much
1650 more complicated and powerful but we'll only do that in a later section.
1652 <sect1 id="lartc.ipsec.intro"><title>Intro</title>
1654 IPSEC is a complicated subject. A lot of information is available online, this HOWTO will concentrate on getting you
1655 up and running and explaining the basic principles.
1660 Many iptables configurations drop IPSEC packets! To pass IPSEC, use: 'iptables -A xxx -p 50' and 'iptables -A xxx -p 51'
1665 IPSEC offers a secure version of the Internet Protocol. Security in this context means two different things: encryption and authentication.
1666 A naive vision of security offers only encryption but it can easily be shown that is insufficient - you may be communicating encyphered,
1667 but no guarantee is offered that the remote party is the one you expect it to be.
1670 This is why SSL certificates exist - just doing crypto without authentication does not cut it.
1673 IPSEC supports 'Encapsulated Security Payload' (ESP) for encryption and 'Authentication Header' (AH) for authenticating the remote partner.
1674 You should configure both of them.
1677 Both ESP and AH rely on security associations. A security association (SA) consists of a source, a destination and an instruction. A sample
1678 authentication SA may look like this:
1680 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1682 This says 'traffic going from 10.0.0.11 to 10.0.0.216 that needs an AH can be signed using HMAC-MD5 using secret 1234567890123456'. This instruction
1683 is labelled with SPI id '15700', more about that later.
1684 The interesting bit about SAs is that they are symmetrical. Both sides of a conversation share exactly the same SA. It is not mirrored on the
1685 other side. Do note however that there is no 'autoreverse' rule - this SA only describes a possible authentication from 10.0.0.11 to
1686 10.0.0.216. For two-way traffic, two SAs are needed.
1691 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1693 This says 'traffic going from 10.0.0.11 to 10.0.0.216 that needs encryption can be encyphered using 3des-cbc with key 123456789012123456789012'. The
1697 So far, we've seen that SAs describe possible instructions, but do not in fact describe policy as to when these need to be used. In fact,
1698 there could be an arbitrary number of nearly identical SAs with only differing SPI ids. Incidentally, SPI stands for Security Parameter Index.
1699 To do actual crypto, we need to describe a policy. This policy can include things as 'use ipsec if available' or 'drop traffic unless we have ispec'.
1702 A typical simple Security Policy (SP) looks like this:
1704 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1705 esp/transport//require
1706 ah/transport//require;
1708 If entered on host 10.0.0.216, this means that all traffic going out to 10.0.0.11 must be encrypted
1709 and be wrapped in an AH authenticating header. Note that this does not descibe which SA is to be used,
1710 that is left as an exercise for the kernel to determine.
1713 Outgoing packets are however labelled with the SAs SPI ids the kernel used for encryption and authentication
1714 so the remote can lookup the corresponding verification and decryption instruction.
1717 What follows is a very simple configuration for talking from host 10.0.0.216 to 10.0.0.11 using
1718 encryption and authentication. Note that the reverse path is plaintext in this first version and that
1719 this configuration should not be deployed.
1725 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1726 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1728 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1729 esp/transport//required
1730 ah/transport//required;
1734 On host 10.0.0.11, the same Security Associations, no Security Policy:
1737 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1738 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1742 With the above configuration in place (these files can be executed if 'setkey' is installed in /sbin),
1743 'ping 10.0.0.11' from 10.0.0.216 looks like this using tcpdump:
1745 22:37:52 10.0.0.216 > 10.0.0.11: AH(spi=0x00005fb4,seq=0xa): ESP(spi=0x00005fb5,seq=0xa) (DF)
1746 22:37:52 10.0.0.11 > 10.0.0.216: icmp: echo reply
1748 Note how the ping back from 10.0.0.11 is indeed plainly visible. The forward ping cannot be read by tcpdump
1749 of course, but it does show the Security Parameter Index of AH and ESP, which tells 10.0.0.11 how to
1750 verify the authenticity of our packet and how to decrypt it.
1753 A few things must be mentioned however. The configuration above is shown in a lot of IPSEC examples and it is very dangerous.
1754 The problem is that the above contains policy on how 10.0.0.216 should treat packets going to 10.0.0.11, and that it explains how 10.0.0.11
1755 should treat those packets but it does NOT instruct 10.0.0.11 to discard unauthenticated or unencrypted traffic!
1758 Anybody can now insert spoofed and completely unencrypted data and 10.0.0.11 will accept it. To remedy the above, we need an incoming
1759 Security Policy on 10.0.0.11, as follows:
1762 spdadd 10.0.0.216 10.0.0.11 any -P IN ipsec
1763 esp/transport//require
1764 ah/transport//require;
1766 This instructs 10.0.0.11 that any traffic coming to it from 10.0.0.216 is required to have valid ESP and AH.
1769 Now, to complete this configuration, we need return traffic to be encrypted and authenticated as well of course. The full configuration on
1777 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1778 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1781 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1782 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1784 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1785 esp/transport//require
1786 ah/transport//require;
1788 spdadd 10.0.0.11 10.0.0.216 any -P in ipsec
1789 esp/transport//require
1790 ah/transport//require;
1802 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1803 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1806 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1807 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1810 spdadd 10.0.0.11 10.0.0.216 any -P out ipsec
1811 esp/transport//require
1812 ah/transport//require;
1814 spdadd 10.0.0.216 10.0.0.11 any -P in ipsec
1815 esp/transport//require
1816 ah/transport//require;
1821 Note that in this example we used identical keys for both directions of traffic. This is not required however.
1824 <sect1 id="lartc.ipsec.automatic.keying"><title>Automatic keying</title>
1826 In the previous section, encryption was configured using simple shared secrets. In other words, to remain secure,
1827 we need to transfer our encryption configuration over a trusted channel. If we were to configure the remote host
1828 over telnet, any third party would know our shared secret and the setup would not be secure.
1831 Furthermore, because the secret is shared, it is not a secret. The remote can't do a lot with our secret, but we do
1832 need to make sure that we use a different secret for communicating with all our partners. This requires a large number of keys,
1833 if there are 10 parties, this needs at least 50 different secrets.
1836 To solve this, IPSEC provides Internet Key Exchange to automatically exchange randomly generated keys which are
1837 transmitted using asymmetric encryption technology.
1840 The Linux 2.5 IPSEC implementation works with the KAME 'racoon' IKE daemon.
1846 <sect1 id="lartc.ipsec.tunnel"><title>IPSEC tunnels</title>
1848 So far, we've only seen IPSEC in so called 'transport' mode where both endpoints understand IPSEC directly. As this is often not
1849 the case, it is often necessary to have only routers understand IPSEC, and have them do the work for the hosts behind them.
1850 This is called 'tunnel mode'.
1853 Setting this up is a breeze. To tunnel all traffic to 130.161.0.0/16 from 10.0.0.216 via 10.0.0.11, we issue the following on
1860 add 10.0.0.216 10.0.0.11 esp 34501
1862 -E 3des-cbc "123456789012123456789012";
1864 spdadd 10.0.0.0/24 130.161.0.0/16 any -P out ipsec
1865 esp/tunnel/10.0.0.216-10.0.0.11/require;
1867 Note the '-m tunnel', it is vitally important! This first configures an ESP encryption SA between our tunnel endpoints,
1868 10.0.0.216 and 10.0.0.11.
1871 Next the actual tunnel is configured. It instructs the kernel to encrypt all traffic it has to route from 10.0.0.0/24 to
1872 130.161.0.0. Furthermore, this traffic then has to be shipped to 10.0.0.11.
1875 10.0.0.11 also needs some configuration:
1881 add 10.0.0.216 10.0.0.11 esp 34501
1883 -E 3des-cbc "123456789012123456789012";
1885 spdadd 10.0.0.0/24 130.161.0.0/16 any -P in ipsec
1886 esp/tunnel/10.0.0.216-10.0.0.11/require;
1888 Note that this is exactly identical, except for the change from '-P out' to '-P in'. As with earlier examples,
1889 we've now only configured traffic going one way. Completing the other half of the tunnel is left as an
1890 exercise for the reader.
1893 Another name for this setup is 'proxy ESP', which is somewhat clearer.
1898 The IPSEC tunnel needs to have IP Forwarding enabled in the kernel!
1905 <chapter id="lartc.multicast">
1906 <Title>Multicast routing</Title>
1909 FIXME: Editor Vacancy!
1913 The Multicast-HOWTO is ancient (relatively-speaking) and may be inaccurate
1914 or misleading in places, for that reason.
1918 Before you can do any multicast routing, you need to configure the Linux
1919 kernel to support the type of multicast routing you want to do. This, in
1920 turn, requires you to decide what type of multicast routing you expect to
1921 be using. There are essentially four "common" types - DVMRP (the Multicast
1922 version of the RIP unicast protocol), MOSPF (the same, but for OSPF), PIM-SM
1923 ("Protocol Independent Multicasting - Sparse Mode", which assumes that users
1924 of any multicast group are spread out, rather than clumped) and PIM-DM (the
1925 same, but "Dense Mode", which assumes that there will be significant clumps
1926 of users of the same multicast group).
1930 In the Linux kernel, you will notice that these options don't appear. This is
1931 because the protocol itself is handled by a routing application, such as
1932 Zebra, mrouted, or pimd. However, you still have to have a good idea of which
1933 you're going to use, to select the right options in the kernel.
1937 For all multicast routing, you will definitely need to enable "multicasting"
1938 and "multicast routing". For DVMRP and MOSPF, this is sufficient. If you are
1939 going to use PIM, you must also enable PIMv1 or PIMv2, depending on whether
1940 the network you are connecting to uses version 1 or 2 of the PIM protocol.
1944 Once you have all that sorted out, and your new Linux kernel compiled, you
1945 will see that the IP protocols listed, at boot time, now include IGMP. This
1946 is a protocol for managing multicast groups. At the time of writing, Linux
1947 supports IGMP versions 1 and 2 only, although version 3 does exist and has
1948 been documented. This doesn't really affect us that much, as IGMPv3 is still
1949 new enough that the extra capabilities of IGMPv3 aren't going to be that
1950 much use. Because IGMP deals with groups, only the features present in the
1951 simplest version of IGMP over the entire group are going to be used. For the
1952 most part, that will be IGMPv2, although IGMPv1 is sill going to be
1957 So far, so good. We've enabled multicasting. Now, we have to tell the Linux
1958 kernel to actually do something with it, so we can start routing. This means
1959 adding the Multicast virtual network to the router table:
1963 ip route add 224.0.0.0/4 dev eth0
1967 (Assuming, of course, that you're multicasting over eth0! Substitute the
1968 device of your choice, for this.)
1972 Now, tell Linux to forward packets...
1976 echo 1 > /proc/sys/net/ipv4/ip_forward
1980 At this point, you may be wondering if this is ever going to do anything. So,
1981 to test our connection, we ping the default group, 224.0.0.1, to see if anyone
1982 is alive. All machines on your LAN with multicasting enabled <Emphasis>should</Emphasis>
1983 respond, but nothing else. You'll notice that none of the machines that
1984 respond have an IP address of 224.0.0.1. What a surprise! :) This is a group
1985 address (a "broadcast" to subscribers), and all members of the group will
1986 respond with their own address, not the group address.
1994 At this point, you're ready to do actual multicast routing. Well, assuming
1995 that you have two networks to route between.
2004 <chapter id="lartc.qdisc">
2005 <Title>Queueing Disciplines for Bandwidth Management</Title>
2008 Now, when I discovered this, it <Emphasis>really</Emphasis> blew me away. Linux 2.2/2.4
2009 comes with everything to manage bandwidth in ways comparable to high-end
2010 dedicated bandwidth management systems.
2014 Linux even goes far beyond what Frame and ATM provide.
2017 <Para>Just to prevent confusion, <command>tc</command> uses the following
2018 rules for bandwith specification:
2020 <literallayout class='monospaced'>
2021 mbps = 1024 kbps = 1024 * 1024 bps => byte/s
2022 mbit = 1024 kbit => kilo bit/s.
2023 mb = 1024 kb = 1024 * 1024 b => byte
2024 mbit = 1024 kbit => kilo bit.
2027 Internally, the number is stored in bps and b.
2030 <Para>But when <command>tc</command> prints the rate, it uses following :
2033 <literallayout class='monospaced'>
2034 1Mbit = 1024 Kbit = 1024 * 1024 bps => byte/s
2037 <Sect1 id="lartc.qdisc.explain">
2038 <Title>Queues and Queueing Disciplines explained</Title>
2041 With queueing we determine the way in which data is <Emphasis>SENT</Emphasis>.
2042 It is important to realise that we can only shape data that we transmit.
2046 With the way the Internet works, we have no direct control of what people
2047 send us. It's a bit like your (physical!) mailbox at home. There is no way
2048 you can influence the world to modify the amount of mail they send you,
2049 short of contacting everybody.
2053 However, the Internet is mostly based on TCP/IP which has a few features
2054 that help us. TCP/IP has no way of knowing the capacity of the network
2055 between two hosts, so it just starts sending data faster and faster ('slow
2056 start') and when packets start getting lost, because there is no room to
2057 send them, it will slow down. In fact it is a bit smarter than this, but
2058 more about that later.
2062 This is the equivalent of not reading half of your mail, and hoping that
2063 people will stop sending it to you. With the difference that it works for
2068 If you have a router and wish to prevent certain hosts within your network
2069 from downloading too fast, you need to do your shaping on the *inner* interface
2070 of your router, the one that sends data to your own computers.
2074 You also have to be sure you are controlling the bottleneck of the link.
2075 If you have a 100Mbit NIC and you have a router that has a 256kbit link,
2076 you have to make sure you are not sending more data than your router can
2077 handle. Otherwise, it will be the router who is controlling the link and
2078 shaping the available bandwith. We need to 'own the queue' so to speak, and
2079 be the slowest link in the chain. Luckily this is easily possible.
2084 <Sect1 id="lartc.qdisc.classless">
2085 <Title>Simple, classless Queueing Disciplines</Title>
2088 As said, with queueing disciplines, we change the way data is sent.
2089 Classless queueing disciplines are those that, by and large accept data and
2090 only reschedule, delay or drop it.
2094 These can be used to shape traffic for an entire interface, without any
2095 subdivisions. It is vital that you understand this part of queueing before
2096 we go on the the classful qdisc-containing-qdiscs!
2100 By far the most widely used discipline is the pfifo_fast qdisc - this is the
2101 default. This also explains why these advanced features are so robust. They
2102 are nothing more than 'just another queue'.
2106 Each of these queues has specific strengths and weaknesses. Not all of them
2107 may be as well tested.
2111 <Title>pfifo_fast</Title>
2114 This queue is, as the name says, First In, First Out, which means that no
2115 packet receives special treatment. At least, not quite. This queue has 3 so
2116 called 'bands'. Within each band, FIFO rules apply. However, as long as
2117 there are packets waiting in band 0, band 1 won't be processed. Same goes
2118 for band 1 and band 2.
2122 The kernel honors the so called Type of Service flag of packets, and takes
2123 care to insert 'minimum delay' packets in band 0.
2127 Do not confuse this classless simple qdisc with the classful PRIO one!
2128 Although they behave similarly, pfifo_fast is classless and you cannot add
2129 other qdiscs to it with the tc command.
2133 <Title>Parameters & usage</Title>
2136 You can't configure the pfifo_fast qdisc as it is the hardwired default.
2137 This is how it is configured by default:
2141 <Term>priomap</Term>
2144 Determines how packet priorities, as assigned by the kernel, map to bands.
2145 Mapping occurs based on the TOS octet of the packet, which looks like this:
2152 +-----+-----+-----+-----+-----+-----+-----+-----+
2154 | PRECEDENCE | TOS | MBZ |
2156 +-----+-----+-----+-----+-----+-----+-----+-----+
2162 The four TOS bits (the 'TOS field') are defined as:
2165 Binary Decimcal Meaning
2166 -----------------------------------------
2167 1000 8 Minimize delay (md)
2168 0100 4 Maximize throughput (mt)
2169 0010 2 Maximize reliability (mr)
2170 0001 1 Minimize monetary cost (mmc)
2171 0000 0 Normal Service
2177 As there is 1 bit to the right of these four bits, the actual value of the
2178 TOS field is double the value of the TOS bits. Tcpdump -v -v shows you the
2179 value of the entire TOS field, not just the four bits. It is the value you
2180 see in the first column of this table:
2186 TOS Bits Means Linux Priority Band
2187 ------------------------------------------------------------
2188 0x0 0 Normal Service 0 Best Effort 1
2189 0x2 1 Minimize Monetary Cost 1 Filler 2
2190 0x4 2 Maximize Reliability 0 Best Effort 1
2191 0x6 3 mmc+mr 0 Best Effort 1
2192 0x8 4 Maximize Throughput 2 Bulk 2
2193 0xa 5 mmc+mt 2 Bulk 2
2194 0xc 6 mr+mt 2 Bulk 2
2195 0xe 7 mmc+mr+mt 2 Bulk 2
2196 0x10 8 Minimize Delay 6 Interactive 0
2197 0x12 9 mmc+md 6 Interactive 0
2198 0x14 10 mr+md 6 Interactive 0
2199 0x16 11 mmc+mr+md 6 Interactive 0
2200 0x18 12 mt+md 4 Int. Bulk 1
2201 0x1a 13 mmc+mt+md 4 Int. Bulk 1
2202 0x1c 14 mr+mt+md 4 Int. Bulk 1
2203 0x1e 15 mmc+mr+mt+md 4 Int. Bulk 1
2209 Lots of numbers. The second column contains the value of the relevant four
2210 TOS bits, followed by their translated meaning. For example, 15 stands for a
2211 packet wanting Minimal Monetary Cost, Maximum Reliability, Maximum
2212 Throughput AND Minimum Delay. I would call this a 'Dutch Packet'.
2216 The fourth column lists the way the Linux kernel interprets the TOS bits, by
2217 showing to which Priority they are mapped.
2221 The last column shows the result of the default priomap. On the command line,
2222 the default priomap looks like this:
2225 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
2231 This means that priority 4, for example, gets mapped to band number 1. The
2232 priomap also allows you to list higher priorities (> 7) which do not
2233 correspond to TOS mappings, but which are set by other means.
2237 This table from RFC 1349 (read it for more details) tells you how
2238 applications might very well set their TOS bits:
2241 TELNET 1000 (minimize delay)
2243 Control 1000 (minimize delay)
2244 Data 0100 (maximize throughput)
2246 TFTP 1000 (minimize delay)
2249 Command phase 1000 (minimize delay)
2250 DATA phase 0100 (maximize throughput)
2253 UDP Query 1000 (minimize delay)
2255 Zone Transfer 0100 (maximize throughput)
2257 NNTP 0001 (minimize monetary cost)
2261 Requests 0000 (mostly)
2262 Responses <same as request> (mostly)
2268 <Term>txqueuelen</Term>
2271 The length of this queue is gleaned from the interface configuration, which
2272 you can see and set with ifconfig and ip. To set the queue length to 10,
2273 execute: ifconfig eth0 txqueuelen 10
2277 You can't set this parameter with tc!
2288 <Title>Token Bucket Filter</Title>
2291 The Token Bucket Filter (TBF) is a simple qdisc that only passes packets
2292 arriving at a rate which is not exceeding some administratively set rate, but
2293 with the possibility to allow short bursts in excess of this rate.
2297 TBF is very precise, network- and processor friendly. It should be your
2298 first choice if you simply want to slow an interface down!
2302 The TBF implementation consists of a buffer (bucket), constantly filled by
2303 some virtual pieces of information called tokens, at a specific rate (token
2304 rate). The most important parameter of the bucket is its size, that is the
2305 number of tokens it can store.
2309 Each arriving token collects one incoming data packet from the data queue
2310 and is then deleted from the bucket. Associating this algorithm
2311 with the two flows -- token and data, gives us three possible scenarios:
2320 The data arrives in TBF at a rate that's <Emphasis>equal</Emphasis> to the rate
2321 of incoming tokens. In this case each incoming packet has its matching token
2322 and passes the queue without delay.
2329 The data arrives in TBF at a rate that's <Emphasis>smaller</Emphasis> than the
2330 token rate. Only a part of the tokens are deleted at output of each data packet
2331 that's sent out the queue, so the tokens accumulate, up to the bucket size.
2332 The unused tokens can then be used to send data a a speed that's exceeding the
2333 standard token rate, in case short data bursts occur.
2340 The data arrives in TBF at a rate <Emphasis>bigger</Emphasis> than the token rate.
2341 This means that the bucket will soon be devoid of tokens, which causes the
2342 TBF to throttle itself for a while. This is called an 'overlimit situation'.
2343 If packets keep coming in, packets will start to get dropped.
2352 The last scenario is very important, because it allows to
2353 administratively shape the bandwidth available to data that's passing
2358 The accumulation of tokens allows a short burst of overlimit data to be
2359 still passed without loss, but any lasting overload will cause packets to be
2360 constantly delayed, and then dropped.
2364 Please note that in the actual implementation, tokens correspond to bytes,
2369 <Title>Parameters & usage</Title>
2372 Even though you will probably not need to change them, tbf has some knobs
2373 available. First the parameters that are always available:
2377 <Term>limit or latency</Term>
2380 Limit is the number of bytes that can be queued waiting for tokens to become
2381 available. You can also specify this the other way around by setting the
2382 latency parameter, which specifies the maximum amount of time a packet can
2383 sit in the TBF. The latter calculation takes into account the size of the
2384 bucket, the rate and possibly the peakrate (if set).
2388 <Term>burst/buffer/maxburst</Term>
2391 Size of the bucket, in bytes. This is the maximum amount of bytes that
2392 tokens can be available for instantaneously. In general, larger shaping
2393 rates require a larger buffer. For 10mbit/s on Intel, you need at least
2394 10kbyte buffer if you want to reach your configured rate!
2398 If your buffer is too small, packets may be dropped because more tokens
2399 arrive per timer tick than fit in your bucket.
2406 A zero-sized packet does not use zero bandwidth. For ethernet, no packet
2407 uses less than 64 bytes. The Minimum Packet Unit determines the minimal
2408 token usage for a packet.
2415 The speedknob. See remarks above about limits!
2422 If the bucket contains tokens and is allowed to empty, by default it does so
2423 at infinite speed. If this is unacceptable, use the following parameters:
2430 <Term>peakrate</Term>
2433 If tokens are available, and packets arrive, they are sent out immediately
2434 by default, at 'lightspeed' so to speak. That may not be what you want,
2435 especially if you have a large bucket.
2439 The peakrate can be used to specify how quickly the bucket is allowed to be
2440 depleted. If doing everything by the book, this is achieved by releasing a
2441 packet, and then wait just long enough, and release the next. We calculated
2442 our waits so we send just at peakrate.
2446 However, due to de default 10ms timer resolution of Unix, with 10.000 bits
2447 average packets, we are limited to 1mbit/s of peakrate!
2451 <Term>mtu/minburst</Term>
2454 The 1mbit/s peakrate is not very useful if your regular rate is more than
2455 that. A higher peakrate is possible by sending out more packets per
2456 timertick, which effectively means that we create a second bucket!
2460 This second bucket defaults to a single packet, which is not a bucket at
2465 To calculate the maximum possible peakrate, multiply the configured mtu by
2466 100 (or more correctly, HZ, which is 100 on Intel, 1024 on Alpha).
2475 <Title>Sample configuration</Title>
2478 A simple but *very* useful configuration is this:
2481 # tc qdisc add dev ppp0 root tbf rate 220kbit latency 50ms burst 1540
2487 Ok, why is this useful? If you have a networking device with a large queue,
2488 like a DSL modem or a cable modem, and you talk to it over a fast device,
2489 like over an ethernet interface, you will find that uploading absolutely
2490 destroys interactivity.
2494 This is because uploading will fill the queue in the modem, which is
2495 probably *huge* because this helps actually achieving good data throughput
2496 uploading. But this is not what you want, you want to have the queue not too
2497 big so interactivity remains and you can still do other stuff while sending
2502 The line above slows down sending to a rate that does not lead to a queue in
2503 the modem - the queue will be in Linux, where we can control it to a limited
2508 Change 220kbit to your uplink's *actual* speed, minus a few percent. If you
2509 have a really fast modem, raise 'burst' a bit.
2516 <Sect2 id="lartc.sfq">
2517 <Title>Stochastic Fairness Queueing</Title>
2520 Stochastic Fairness Queueing (SFQ) is a simple implementation of the fair
2521 queueing algorithms family. It's less accurate than others, but it also
2522 requires less calculations while being almost perfectly fair.
2526 The key word in SFQ is conversation (or flow), which mostly corresponds to a
2527 TCP session or a UDP stream. Traffic is divided into a pretty large number
2528 of FIFO queues, one for each conversation. Traffic is then sent in a round
2529 robin fashion, giving each session the chance to send data in turn.
2533 This leads to very fair behaviour and disallows any single conversation from
2534 drowning out the rest. SFQ is called 'Stochastic' because it doesn't really
2535 allocate a queue for each session, it has an algorithm which divides traffic
2536 over a limited number of queues using a hashing algorithm.
2540 Because of the hash, multiple sessions might end up in the same bucket, which
2541 would halve each session's chance of sending a packet, thus halving the
2542 effective speed available. To prevent this situation from becoming
2543 noticeable, SFQ changes its hashing algorithm quite often so that any two
2544 colliding sessions will only do so for a small number of seconds.
2548 It is important to note that SFQ is only useful in case your actual outgoing
2549 interface is really full! If it isn't then there will be no queue on your
2550 linux machine and hence no effect. Later on we will describe how to combine
2551 SFQ with other qdiscs to get a best-of-both worlds situation.
2555 Specifically, setting SFQ on the ethernet interface heading to your
2556 cable modem or DSL router is pointless without further shaping!
2560 <Title>Parameters & usage</Title>
2563 The SFQ is pretty much self tuning:
2567 <Term>perturb</Term>
2570 Reconfigure hashing once this many seconds. If unset, hash will never be
2571 reconfigured. Not recommended. 10 seconds is probably a good value.
2575 <Term>quantum</Term>
2578 Amount of bytes a stream is allowed to dequeue before the next queue gets a
2579 turn. Defaults to 1 maximum sized packet (MTU-sized). Do not set below the
2589 <Title>Sample configuration</Title>
2592 If you have a device which has identical link speed and actual available
2593 rate, like a phone modem, this configuration will help promote fairness:
2596 # tc qdisc add dev ppp0 root sfq perturb 10
2598 qdisc sfq 800c: dev ppp0 quantum 1514b limit 128p flows 128/1024 perturb 10sec
2599 Sent 4812 bytes 62 pkts (dropped 0, overlimits 0)
2605 The number 800c: is the automatically assigned handle number, limit means
2606 that 128 packets can wait in this queue. There are 1024 hashbuckets
2607 available for accounting, of which 128 can be active at a time (no more
2608 packets fit in the queue!) Once every 10 seconds, the hashes are
2618 <Sect1 id="lartc.qdisc.advice">
2619 <Title>Advice for when to use which queue</Title>
2622 Summarizing, these are the simple queues that actually manage traffic by
2623 reordering, slowing or dropping packets.
2627 The following tips may help in choosing which queue to use. It mentions some
2628 qdiscs described in the
2629 <citetitle><xref linkend="lartc.adv-qdisc"></citetitle> chapter.
2635 To purely slow down outgoing traffic, use the Token Bucket Filter. Works up
2636 to huge bandwidths, if you scale the bucket.
2642 If your link is truly full and you want to make sure that no single session
2643 can dominate your outgoing bandwidth, use Stochastical Fairness Queueing.
2649 If you have a big backbone and know what you are doing, consider Random
2650 Early Drop (see Advanced chapter).
2656 To 'shape' incoming traffic which you are not forwarding, use the Ingress
2657 Policer. Incoming shaping is called 'policing', by the way, not 'shaping'.
2663 If you *are* forwarding it, use a TBF on the interface you are forwarding
2664 the data to. Unless you want to shape traffic that may go out over several
2665 interfaces, in which case the only common factor is the incoming interface.
2666 In that case use the Ingress Policer.
2672 If you don't want to shape, but only want to see if your interface is so
2673 loaded that it has to queue, use the pfifo queue (not pfifo_fast). It lacks
2674 internal bands but does account the size of its backlog.
2679 Finally - you can also do <quote>social shaping</quote>.
2680 You may not always be able to use technology to achieve what you want.
2681 Users experience technical constraints as hostile.
2682 A kind word may also help with getting your bandwidth to be divided right!
2689 <Sect1 id="lartc.qdisc.terminology">
2690 <Title>Terminology</Title>
2693 To properly understand more complicated configurations it is necessary to
2694 explain a few concepts first. Because of the complexity and he relative
2695 youth of the subject, a lot of different words are used when people in fact
2696 mean the same thing.
2700 The following is loosely based on
2701 <filename>draft-ietf-diffserv-model-06.txt</filename>,
2702 <citetitle>An Informal Management Model for Diffserv Routers</citetitle>.
2703 It can currently be found at
2704 <ulink url="http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt">
2705 http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt
2710 Read it for the strict definitions of the terms used.
2714 <Term>Queueing Discipline</Term>
2717 An algorithm that manages the queue of a device, either incoming (ingress)
2718 or outgoing (egress).
2722 <Term>Classless qdisc</Term>
2725 A qdisc with no configurable internal subdivisions.
2729 <Term>Classful qdisc</Term>
2732 A classful qdisc contains multiple classes. Each of these classes contains a
2733 further qdisc, which may again be classful, but need not be. According to
2734 the strict definition, pfifo_fast *is* classful, because it contains three
2735 bands which are, in fact, classes. However, from the user's configuration
2736 perspective, it is classless as the classes can't be touched with the tc
2741 <Term>Classes</Term>
2744 A classful qdisc may have many classes, which each are internal to the
2745 qdisc. Each of these classes may contain a real qdisc.
2749 <Term>Classifier</Term>
2752 Each classful qdisc needs to determine to which class it needs to send a
2753 packet. This is done using the classifier.
2760 Classification can be performed using filters. A filter contains a number of
2761 conditions which if matched, make the filter match.
2765 <Term>Scheduling</Term>
2768 A qdisc may, with the help of a classifier, decide that some packets need to
2769 go out earlier than others. This process is called Scheduling, and is
2770 performed for example by the pfifo_fast qdisc mentioned earlier. Scheduling
2771 is also called 'reordering', but this is confusing.
2775 <Term>Shaping</Term>
2778 The process of delaying packets before they go out to make traffic confirm
2779 to a configured maximum rate. Shaping is performed on egress. Colloquially,
2780 dropping packets to slow traffic down is also often called Shaping.
2784 <Term>Policing</Term>
2787 Delaying or dropping packets in order to make traffic stay below a
2788 configured bandwidth. In Linux, policing can only drop a packet and not
2789 delay it - there is no 'ingress queue'.
2793 <Term>Work-Conserving</Term>
2796 A work-conserving qdisc always delivers a packet if one is available. In
2797 other words, it never delays a packet if the network adaptor is ready to
2798 send one (in the case of an egress qdisc).
2802 <Term>non-Work-Conserving</Term>
2805 Some queues, like for example the Token Bucket Filter, may need to hold on
2806 to a packet for a certain time in order to limit the bandwidth. This means
2807 that they sometimes refuse to give up a packet, even though they have one
2815 Now that we have our terminology straight, let's see where all these things
2825 +---------------+-----------------------------------------+
2827 | -------> IP Stack |
2832 | | / ----------> Forwarding -> |
2837 | | Egress /--qdisc2--\ |
2838 --->->Ingress Classifier ---qdisc3---- | ->
2839 | Qdisc \__qdisc4__/ |
2842 +----------------------------------------------------------+
2845 Thanks to Jamal Hadi Salim for this ASCII representation.
2849 The big block represents the kernel. The leftmost arrow represents traffic
2850 entering your machine from the network. It is then fed to the Ingress
2851 Qdisc which may apply Filters to a packet, and decide to drop it. This
2852 is called 'Policing'.
2856 This happens at a very early stage, before it has seen a lot of the kernel.
2857 It is therefore a very good place to drop traffic very early, without
2858 consuming a lot of CPU power.
2862 If the packet is allowed to continue, it may be destined for a local
2863 application, in which case it enters the IP stack in order to be processed,
2864 and handed over to a userspace program. The packet may also be forwarded
2865 without entering an application, in which case it is destined for egress.
2866 Userspace programs may also deliver data, which is then examined and
2867 forwarded to the Egress Classifier.
2871 There it is investigated and enqueued to any of a number of qdiscs. In the
2872 unconfigured default case, there is only one egress qdisc installed, the
2873 pfifo_fast, which always receives the packet. This is called 'enqueueing'.
2877 The packet now sits in the qdisc, waiting for the kernel to ask for
2878 it for transmission over the network interface. This is called 'dequeueing'.
2882 This picture also holds in case there is only one network adaptor - the
2883 arrows entering and leaving the kernel should not be taken too literally.
2884 Each network adaptor has both ingress and egress hooks.
2889 <Sect1 id="lartc.qdisc.classful">
2890 <Title>Classful Queueing Disciplines</Title>
2893 Classful qdiscs are very useful if you have different kinds of traffic which
2894 should have differing treatment. One of the classful qdiscs is called 'CBQ'
2895 , 'Class Based Queueing' and it is so widely mentioned that people identify
2896 queueing with classes solely with CBQ, but this is not the case.
2900 CBQ is merely the oldest kid on the block - and also the most complex one.
2901 It may not always do what you want. This may come as something of a shock
2902 to many who fell for the 'sendmail effect', which teaches us that any
2903 complex technology which doesn't come with documentation must be the best
2908 More about CBQ and its alternatives shortly.
2912 <Title>Flow within classful qdiscs & classes</Title>
2915 When traffic enters a classful qdisc, it needs to be sent to any of the
2916 classes within - it needs to be 'classified'. To determine what to do with a
2917 packet, the so called 'filters' are consulted. It is important to know that
2918 the filters are called from within a qdisc, and not the other way around!
2922 The filters attached to that qdisc then return with a decision, and the
2923 qdisc uses this to enqueue the packet into one of the classes. Each subclass
2924 may try other filters to see if further instructions apply. If not, the
2925 class enqueues the packet to the qdisc it contains.
2929 Besides containing other qdiscs, most classful qdiscs also perform shaping.
2930 This is useful to perform both packet scheduling (with SFQ, for example) and
2931 rate control. You need this in cases where you have a high speed
2932 interface (for example, ethernet) to a slower device (a cable modem).
2936 If you were only to run SFQ, nothing would happen, as packets enter &
2937 leave your router without delay: the output interface is far faster than
2938 your actual link speed. There is no queue to schedule then.
2944 <Title>The qdisc family: roots, handles, siblings and parents</Title>
2947 Each interface has one egress 'root qdisc', by default the earlier mentioned
2948 classless pfifo_fast queueing discipline. Each qdisc can be assigned a
2949 handle, which can be used by later configuration statements to refer to that
2950 qdisc. Besides an egress qdisc, an interface may also have an ingress, which
2951 polices traffic coming in.
2955 The handles of these qdiscs consist of two parts, a major number and a minor
2956 number. It is habitual to name the root qdisc '1:', which is equal to '1:0'.
2957 The minor number of a qdisc is always 0.
2961 Classes need to have the same major number as their parent.
2965 <Title>How filters are used to classify traffic </Title>
2968 Recapping, a typical hierarchy might look like this:
2985 But don't let this tree fool you! You should *not* imagine the kernel to be
2986 at the apex of the tree and the network below, that is just not the case.
2987 Packets get enqueued and dequeued at the root qdisc, which is the only thing
2988 the kernel talks to.
2992 A packet might get classified in a chain like this:
2996 1: -> 1:1 -> 12: -> 12:2
3000 The packet now resides in a queue in a qdisc attached to class 12:2. In this
3001 example, a filter was attached to each 'node' in the tree, each choosing a
3002 branch to take next. This can make sense. However, this is also possible:
3010 In this case, a filter attached to the root decided to send the packet
3017 <Title>How packets are dequeued to the hardware</Title>
3020 When the kernel decides that it needs to extract packets to send to the
3021 interface, the root qdisc 1: gets a dequeue request, which is passed to
3022 1:1, which is in turn passed to 10:, 11: and 12:, which each query their
3023 siblings, and try to dequeue() from them. In this case, the kernel needs to
3024 walk the entire tree, because only 12:2 contains a packet.
3028 In short, nested classes ONLY talk to their parent qdiscs, never to an
3029 interface. Only the root qdisc gets dequeued by the kernel!
3033 The upshot of this is that classes never get dequeued faster than their
3034 parents allow. And this is exactly what we want: this way we can have SFQ in
3035 an inner class, which doesn't do any shaping, only scheduling, and have a
3036 shaping outer qdisc, which does the shaping.
3044 <Title>The PRIO qdisc</Title>
3047 The PRIO qdisc doesn't actually shape, it only subdivides traffic based on
3048 how you configured your filters. You can consider the PRIO qdisc a kind
3049 of pfifo_fast on steroids, whereby each band is a separate class instead of
3054 When a packet is enqueued to the PRIO qdisc, a class is chosen based on the
3055 filter commands you gave. By default, three classes are created. These
3056 classes by default contain pure FIFO qdiscs with no internal
3057 structure, but you can replace these by any qdisc you have available.
3061 Whenever a packet needs to be dequeued, class :1 is tried first. Higher
3062 classes are only used if lower bands all did not give up a packet.
3066 This qdisc is very useful in case you want to prioritize certain kinds of
3067 traffic without using only TOS-flags but using all the power of the tc
3068 filters. It can also contain more all qdiscs, whereas pfifo_fast is limited
3069 to simple fifo qdiscs.
3073 Because it doesn't actually shape, the same warning as for SFQ holds: either
3074 use it only if your physical link is really full or wrap it inside a
3075 classful qdisc that does shape. The last holds for almost all cable modems
3080 In formal words, the PRIO qdisc is a Work-Conserving scheduler.
3084 <Title>PRIO parameters & usage</Title>
3087 The following parameters are recognized by tc:
3094 Number of bands to create. Each band is in fact a class. If you change this
3095 number, you must also change:
3099 <Term>priomap</Term>
3102 If you do not provide tc filters to classify traffic, the PRIO qdisc looks
3103 at the TC_PRIO priority to decide how to enqueue traffic.
3107 This works just like with the pfifo_fast qdisc mentioned earlier, see there
3112 The bands are classes, and are called major:1 to major:3 by default, so if
3113 your PRIO qdisc is called 12:, tc filter traffic to 12:1 to grant it more
3118 Reiterating, band 0 goes to minor number 1! Band 1 to minor number 2, etc.
3124 <Title>Sample configuration</Title>
3127 We will create this tree:
3142 Bulk traffic will go to 30:, interactive traffic to 20: or 10:.
3149 # tc qdisc add dev eth0 root handle 1: prio
3150 ## This *instantly* creates classes 1:1, 1:2, 1:3
3152 # tc qdisc add dev eth0 parent 1:1 handle 10: sfq
3153 # tc qdisc add dev eth0 parent 1:2 handle 20: tbf rate 20kbit buffer 1600 limit 3000
3154 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
3160 Now let's see what we created:
3163 # tc -s qdisc ls dev eth0
3164 qdisc sfq 30: quantum 1514b
3165 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
3167 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3168 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
3170 qdisc sfq 10: quantum 1514b
3171 Sent 132 bytes 2 pkts (dropped 0, overlimits 0)
3173 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3174 Sent 174 bytes 3 pkts (dropped 0, overlimits 0)
3177 As you can see, band 0 has already had some traffic, and one packet was sent
3178 while running this command!
3182 We now do some bulk data transfer with a tool that properly sets TOS flags,
3183 and take another look:
3186 # scp tc ahu@10.0.0.11:./
3187 ahu@10.0.0.11's password:
3188 tc 100% |*****************************| 353 KB 00:00
3189 # tc -s qdisc ls dev eth0
3190 qdisc sfq 30: quantum 1514b
3191 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
3193 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3194 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
3196 qdisc sfq 10: quantum 1514b
3197 Sent 2230 bytes 31 pkts (dropped 0, overlimits 0)
3199 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3200 Sent 389140 bytes 326 pkts (dropped 0, overlimits 0)
3203 As you can see, all traffic went to handle 30:, which is the lowest priority
3204 band, just as intended. Now to verify that interactive traffic goes to
3205 higher bands, we create some interactive traffic:
3211 # tc -s qdisc ls dev eth0
3212 qdisc sfq 30: quantum 1514b
3213 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
3215 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3216 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
3218 qdisc sfq 10: quantum 1514b
3219 Sent 14926 bytes 193 pkts (dropped 0, overlimits 0)
3221 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3222 Sent 401836 bytes 488 pkts (dropped 0, overlimits 0)
3228 It worked - all additional traffic has gone to 10:, which is our highest
3229 priority qdisc. No traffic was sent to the lowest priority, which previously
3230 received our entire scp.
3238 <Title>The famous CBQ qdisc</Title>
3241 As said before, CBQ is the most complex qdisc available, the most hyped, the
3242 least understood, and probably the trickiest one to get right. This is not
3243 because the authors are evil or incompetent, far from it, it's just that the
3244 CBQ algorithm isn't all that precise and doesn't really match the way Linux
3249 Besides being classful, CBQ is also a shaper and it is in that aspect that
3250 it really doesn't work very well. It should work like this. If you try to
3251 shape a 10mbit/s connection to 1mbit/s, the link should be idle 90% of the
3252 time. If it isn't, we need to throttle so that it IS idle 90% of the time.
3256 This is pretty hard to measure, so CBQ instead derives the idle time from
3257 the number of microseconds that elapse between requests from the hardware
3258 layer for more data. Combined, this can be used to approximate how full or
3263 This is rather circumspect and doesn't always arrive at proper results. For
3264 example, what if the actual link speed of an interface that is not really
3265 able to transmit the full 100mbit/s of data, perhaps because of a badly
3266 implemented driver? A PCMCIA network card will also never achieve 100mbit/s
3267 because of the way the bus is designed - again, how do we calculate the idle
3272 It gets even worse if we consider not-quite-real network devices like PPP
3273 over Ethernet or PPTP over TCP/IP. The effective bandwidth in that case is
3274 probably determined by the efficiency of pipes to userspace - which is huge.
3278 People who have done measurements discover that CBQ is not always very
3279 accurate and sometimes completely misses the mark.
3283 In many circumstances however it works well. With the documentation provided
3284 here, you should be able to configure it to work well in most cases.
3288 <Title>CBQ shaping in detail</Title>
3291 As said before, CBQ works by making sure that the link is idle just long
3292 enough to bring down the real bandwidth to the configured rate. To do so, it
3293 calculates the time that should pass between average packets.
3297 During operations, the effective idletime is measured using an exponential
3298 weighted moving average (EWMA), which considers recent packets to be
3299 exponentially more important than past ones. The UNIX loadaverage is
3300 calculated in the same way.
3304 The calculated idle time is subtracted from the EWMA measured one, the
3305 resulting number is called 'avgidle'. A perfectly loaded link has an avgidle
3306 of zero: packets arrive exactly once every calculated interval.
3310 An overloaded link has a negative avgidle and if it gets too negative, CBQ
3311 shuts down for a while and is then 'overlimit'.
3315 Conversely, an idle link might amass a huge avgidle, which would then allow
3316 infinite bandwidths after a few hours of silence. To prevent this, avgidle is
3321 If overlimit, in theory, the CBQ could throttle itself for exactly the
3322 amount of time that was calculated to pass between packets, and then pass
3323 one packet, and throttle again. But see the 'minburst' parameter below.
3327 These are parameters you can specify in order to configure shaping:
3334 Average size of a packet, measured in bytes. Needed for calculating maxidle,
3335 which is derived from maxburst, which is specified in packets.
3339 <Term>bandwidth</Term>
3342 The physical bandwidth of your device, needed for idle time
3350 The time a packet takes to be transmitted over a device may grow in steps,
3351 based on the packet size. An 800 and an 806 size packet may take just as long
3352 to send, for example - this sets the granularity. Most often set to '8'.
3353 Must be an integral power of two.
3357 <Term>maxburst</Term>
3360 This number of packets is used to calculate maxidle so that when avgidle is
3361 at maxidle, this number of average packets can be burst before avgidle drops
3362 to 0. Set it higher to be more tolerant of bursts. You can't set maxidle
3363 directly, only via this parameter.
3367 <Term>minburst</Term>
3370 As mentioned before, CBQ needs to throttle in case of overlimit. The ideal
3371 solution is to do so for exactly the calculated idle time, and pass 1
3372 packet. However, Unix kernels generally have a hard time scheduling events
3373 shorter than 10ms, so it is better to throttle for a longer period, and then
3374 pass minburst packets in one go, and then sleep minburst times longer.
3378 The time to wait is called the offtime. Higher values of minburst lead to
3379 more accurate shaping in the long term, but to bigger bursts at millisecond
3384 <Term>minidle</Term>
3387 If avgidle is below 0, we are overlimits and need to wait until avgidle will
3388 be big enough to send one packet. To prevent a sudden burst from shutting
3389 down the link for a prolonged period of time, avgidle is reset to minidle if
3394 Minidle is specified in negative microseconds, so 10 means that avgidle is
3402 Minimum packet size - needed because even a zero size packet is padded
3403 to 64 bytes on ethernet, and so takes a certain time to transmit. CBQ needs
3404 to know this to accurately calculate the idle time.
3411 Desired rate of traffic leaving this qdisc - this is the 'speed knob'!
3418 Internally, CBQ has a lot of fine tuning. For example, classes which are
3419 known not to have data enqueued to them aren't queried. Overlimit classes
3420 are penalized by lowering their effective priority. All very smart &
3427 <Title>CBQ classful behaviour</Title>
3430 Besides shaping, using the aforementioned idletime approximations, CBQ also
3431 acts like the PRIO queue in the sense that classes can have differing
3432 priorities and that lower priority numbers will be polled before the higher
3437 Each time a packet is requested by the hardware layer to be sent out to the
3438 network, a weighted round robin process ('WRR') starts, beginning with the
3439 lower priority classes.
3443 These are then grouped and queried if they have data available. If so, it is
3444 returned. After a class has been allowed to dequeue a number of bytes, the
3445 next class within that priority is tried.
3449 The following parameters control the WRR process:
3456 When the outer CBQ is asked for a packet to send out on the interface, it
3457 will try all inner qdiscs (in the classes) in turn, in order of
3458 the 'priority' parameter. Each time a class gets its turn, it can only send out
3459 a limited amount of data. 'Allot' is the base unit of this amount. See
3460 the 'weight' parameter for more information.
3467 The CBQ can also act like the PRIO device. Inner classes with lower priority
3468 are tried first and as long as they have traffic, other classes are not
3476 Weight helps in the Weighted Round Robin process. Each class gets a chance
3477 to send in turn. If you have classes with significantly more bandwidth than
3478 other classes, it makes sense to allow them to send more data in one round
3483 A CBQ adds up all weights under a class, and normalizes them, so you can use
3484 arbitrary numbers: only the ratios are important. People have been
3485 using 'rate/10' as a rule of thumb and it appears to work well. The renormalized
3486 weight is multiplied by the 'allot' parameter to determine how much data can
3487 be sent in one round.
3494 Please note that all classes within an CBQ hierarchy need to share the same
3501 <Title>CBQ parameters that determine link sharing & borrowing</Title>
3504 Besides purely limiting certain kinds of traffic, it is also possible to
3505 specify which classes can borrow capacity from other classes or, conversely,
3513 <Term>Isolated/sharing</Term>
3516 A class that is configured with 'isolated' will not lend out bandwidth to
3517 sibling classes. Use this if you have competing or mutually-unfriendly
3518 agencies on your link who do want to give each other freebies.
3522 The control program tc also knows about 'sharing', which is the reverse
3527 <Term>bounded/borrow</Term>
3530 A class can also be 'bounded', which means that it will not try to borrow
3531 bandwidth from sibling classes. tc also knows about 'borrow', which is the
3532 reverse of 'bounded'.
3536 A typical situation might be where you have two agencies on your link which
3537 are both 'isolated' and 'bounded', which means that they are really limited
3538 to their assigned rate, and also won't allow each other to borrow.
3542 Within such an agency class, there might be other classes which are allowed
3549 <Title>Sample configuration</Title>
3552 This configuration limits webserver traffic to 5mbit and SMTP traffic to 3
3553 mbit. Together, they may not get more than 6mbit. We have a 100mbit NIC and
3554 the classes may borrow bandwidth from each other.
3557 # tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 100Mbit \
3559 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 100Mbit \
3560 rate 6Mbit weight 0.6Mbit prio 8 allot 1514 cell 8 maxburst 20 \
3564 This part installs the root and the customary 1:0 class. The 1:1 class is
3565 bounded, so the total bandwidth can't exceed 6mbit.
3569 As said before, CBQ requires a *lot* of knobs. All parameters are explained
3570 above, however. The corresponding HTB configuration is lots simpler.
3576 # tc class add dev eth0 parent 1:1 classid 1:3 cbq bandwidth 100Mbit \
3577 rate 5Mbit weight 0.5Mbit prio 5 allot 1514 cell 8 maxburst 20 \
3579 # tc class add dev eth0 parent 1:1 classid 1:4 cbq bandwidth 100Mbit \
3580 rate 3Mbit weight 0.3Mbit prio 5 allot 1514 cell 8 maxburst 20 \
3587 These are our two classes. Note how we scale the weight with the configured
3588 rate. Both classes are not bounded, but they are connected to class 1:1
3589 which is bounded. So the sum of bandwith of the 2 classes will never be
3590 more than 6mbit. The classids need to be within the same major number as
3591 the parent CBQ, by the way!
3597 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
3598 # tc qdisc add dev eth0 parent 1:4 handle 40: sfq
3604 Both classes have a FIFO qdisc by default. But we replaced these with an SFQ
3605 queue so each flow of data is treated equally.
3608 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
3609 sport 80 0xffff flowid 1:3
3610 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
3611 sport 25 0xffff flowid 1:4
3617 These commands, attached directly to the root, send traffic to the right
3622 Note that we use 'tc class add' to CREATE classes within a qdisc, but that
3623 we use 'tc qdisc add' to actually add qdiscs to these classes.
3627 You may wonder what happens to traffic that is not classified by any of the
3628 two rules. It appears that in this case, data will then be processed within
3629 1:0, and be unlimited.
3633 If SMTP+web together try to exceed the set limit of 6mbit/s, bandwidth will
3634 be divided according to the weight parameter, giving 5/8 of traffic to the
3635 webserver and 3/8 to the mail server.
3639 With this configuration you can also say that webserver traffic will always
3640 get at minimum 5/8 * 6 mbit = 3.75 mbit.
3646 <Title>Other CBQ parameters: split & defmap</Title>
3649 As said before, a classful qdisc needs to call filters to determine
3650 which class a packet will be enqueued to.
3654 Besides calling the filter, CBQ offers other options, defmap & split.
3655 This is pretty complicated to understand, and it is not vital. But as this
3656 is the only known place where defmap & split are properly explained, I'm
3661 As you will often want to filter on the Type of Service field only, a special
3662 syntax is provided. Whenever the CBQ needs to figure out where a packet
3663 needs to be enqueued, it checks if this node is a 'split node'. If so, one
3664 of the sub-qdiscs has indicated that it wishes to receive all packets with
3665 a certain configured priority, as might be derived from the TOS field, or
3666 socket options set by applications.
3670 The packets' priority bits are or-ed with the defmap field to see if a match
3671 exists. In other words, this is a short-hand way of creating a very fast
3672 filter, which only matches certain priorities. A defmap of ff (hex) will
3673 match everything, a map of 0 nothing. A sample configuration may help make
3680 # tc qdisc add dev eth1 root handle 1: cbq bandwidth 10Mbit allot 1514 \
3681 cell 8 avpkt 1000 mpu 64
3683 # tc class add dev eth1 parent 1:0 classid 1:1 cbq bandwidth 10Mbit \
3684 rate 10Mbit allot 1514 cell 8 weight 1Mbit prio 8 maxburst 20 \
3688 Standard CBQ preamble. I never get used to the sheer amount of numbers
3693 Defmap refers to TC_PRIO bits, which are defined as follows:
3699 TC_PRIO.. Num Corresponds to TOS
3700 -------------------------------------------------
3701 BESTEFFORT 0 Maximize Reliablity
3702 FILLER 1 Minimize Cost
3703 BULK 2 Maximize Throughput (0x8)
3705 INTERACTIVE 6 Minimize Delay (0x10)
3712 The TC_PRIO.. number corresponds to bits, counted from the right. See the
3713 pfifo_fast section for more details how TOS bits are converted to
3718 Now the interactive and the bulk classes:
3724 # tc class add dev eth1 parent 1:1 classid 1:2 cbq bandwidth 10Mbit \
3725 rate 1Mbit allot 1514 cell 8 weight 100Kbit prio 3 maxburst 20 \
3726 avpkt 1000 split 1:0 defmap c0
3728 # tc class add dev eth1 parent 1:1 classid 1:3 cbq bandwidth 10Mbit \
3729 rate 8Mbit allot 1514 cell 8 weight 800Kbit prio 7 maxburst 20 \
3730 avpkt 1000 split 1:0 defmap 3f
3736 The 'split qdisc' is 1:0, which is where the choice will be made. C0 is
3737 binary for 11000000, 3F for 00111111, so these two together will match
3738 everything. The first class matches bits 7 & 6, and thus corresponds
3739 to 'interactive' and 'control' traffic. The second class matches the rest.
3743 Node 1:0 now has a table like this:
3760 For additional fun, you can also pass a 'change mask', which indicates
3761 exactly which priorities you wish to change. You only need to use this if you
3762 are running 'tc class change'. For example, to add best effort traffic to
3763 1:2, we could run this:
3769 # tc class change dev eth1 classid 1:2 cbq defmap 01/01
3775 The priority map over at 1:0 now looks like this:
3795 FIXME: did not test 'tc class change', only looked at the source.
3803 <Title>Hierarchical Token Bucket </Title>
3806 Martin Devera (<devik>) rightly realised that CBQ is complex and does
3807 not seem optimized for many typical situations. His Hierarchical approach is
3808 well suited for setups where you have a fixed amount of bandwidth which you
3809 want to divide for different purposes, giving each purpose a guaranteed
3810 bandwidth, with the possibility of specifying how much bandwidth can be
3815 HTB works just like CBQ but does not resort to idle time calculations to
3816 shape. Instead, it is a classful Token Bucket Filter - hence the name. It
3817 has only a few parameters, which are well documented on his
3819 URL="http://luxik.cdi.cz/~devik/qos/htb/"
3825 As your HTB configuration gets more complex, your configuration scales
3826 well. With CBQ it is already complex even in simple cases! HTB3 (check
3827 <ulink url="http://luxik.cdi.cz/~devik/qos/htb/">its homepage</ulink> for
3828 details on HTB versions) is now part of the official kernel sources
3829 (from 2.4.20-pre1 and 2.5.31 onwards). However, maybe you still need to
3830 get a HTB3 patched version of 'tc': HTB kernel and userspace parts must
3831 be the same major version, or 'tc' will not work with HTB.
3836 If you already have a modern kernel, or are in a position to patch your
3837 kernel, by all means consider HTB.
3842 <Title>Sample configuration</Title>
3845 Functionally almost identical to the CBQ sample configuration above:
3851 # tc qdisc add dev eth0 root handle 1: htb default 30
3853 # tc class add dev eth0 parent 1: classid 1:1 htb rate 6mbit burst 15k
3855 # tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit burst 15k
3856 # tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 6mbit burst 15k
3857 # tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 6mbit burst 15k
3863 The author then recommends SFQ for beneath these classes:
3866 # tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
3867 # tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
3868 # tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10
3874 Add the filters which direct traffic to the right classes:
3877 # U32="tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32"
3878 # $U32 match ip dport 80 0xffff flowid 1:10
3879 # $U32 match ip sport 25 0xffff flowid 1:20
3882 And that's it - no unsightly unexplained numbers, no undocumented
3887 HTB certainly looks wonderful - if 10: and 20: both have their guaranteed
3888 bandwidth, and more is left to divide, they borrow in a 5:3 ratio, just as
3893 Unclassified traffic gets routed to 30:, which has little bandwidth of its
3894 own but can borrow everything that is left over. Because we chose SFQ
3895 internally, we get fairness thrown in for free!
3904 <Sect1 id="lartc.qdisc.filters">
3905 <Title>Classifying packets with filters</Title>
3908 To determine which class shall process a packet, the so-called 'classifier
3909 chain' is called each time a choice needs to be made. This chain consists of
3910 all filters attached to the classful qdisc that needs to decide.
3913 <Para>To reiterate the tree, which is not a tree:
3929 When enqueueing a packet, at each branch the filter chain is consulted for a
3930 relevant instruction. A typical setup might be to have a filter in 1:1 that
3931 directs a packet to 12: and a filter on 12: that sends the packet to 12:2.
3935 You might also attach this latter rule to 1:1, but you can make efficiency
3936 gains by having more specific tests lower in the chain.
3940 You can't filter a packet 'upwards', by the way. Also, with HTB, you should
3941 attach all filters to the root!
3945 And again - packets are only enqueued downwards! When they are dequeued,
3946 they go up again, where the interface lives. They do NOT fall off the end of
3947 the tree to the network adaptor!
3951 <Title>Some simple filtering examples</Title>
3954 As explained in the Classifier chapter, you can match on literally anything,
3955 using a very complicated syntax. To start, we will show how to do the
3956 obvious things, which luckily are quite easy.
3960 Let's say we have a PRIO qdisc called '10:' which contains three classes, and
3961 we want to assign all traffic from and to port 22 to the highest priority
3962 band, the filters would be:
3968 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
3969 ip dport 22 0xffff flowid 10:1
3970 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
3971 ip sport 80 0xffff flowid 10:1
3972 # tc filter add dev eth0 protocol ip parent 10: prio 2 flowid 10:2
3978 What does this say? It says: attach to eth0, node 10: a priority 1 u32
3979 filter that matches on IP destination port 22 *exactly* and send it to band
3980 10:1. And it then repeats the same for source port 80. The last command says
3981 that anything unmatched so far should go to band 10:2, the next-highest
3986 You need to add 'eth0', or whatever your interface is called, because each
3987 interface has a unique namespace of handles.
3991 To select on an IP address, use this:
3994 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
3995 match ip dst 4.3.2.1/32 flowid 10:1
3996 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
3997 match ip src 1.2.3.4/32 flowid 10:1
3998 # tc filter add dev eth0 protocol ip parent 10: prio 2 \
4005 This assigns traffic to 4.3.2.1 and traffic from 1.2.3.4 to the highest
4006 priority queue, and the rest to the next-highest one.
4010 You can concatenate matches, to match on traffic from 1.2.3.4 and from port
4014 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 match ip src 4.3.2.1/32
4015 match ip sport 80 0xffff flowid 10:1
4022 <Sect2 id="lartc.filtering.simple">
4023 <Title>All the filtering commands you will normally need</Title>
4026 Most shaping commands presented here start with this preamble:
4029 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 ..
4032 These are the so called 'u32' matches, which can match on ANY part of a
4037 <Term>On source/destination address</Term>
4040 Source mask 'match ip src 1.2.3.0/24', destination mask 'match ip dst
4041 4.3.2.0/24'. To match a single host, use /32, or omit the mask.
4045 <Term>On source/destination port, all IP protocols</Term>
4048 Source: 'match ip sport 80 0xffff', 'match ip dport 0xffff'
4052 <Term>On ip protocol (tcp, udp, icmp, gre, ipsec)</Term>
4055 Use the numbers from /etc/protocols, for example, icmp is 1: 'match ip
4060 <Term>On fwmark</Term>
4063 You can mark packets with either ipchains and have that mark survive routing
4064 across interfaces. This is really useful to for example only shape traffic on
4065 eth1 that came in on eth0. Syntax:
4066 # tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 6 fw flowid 1:1
4067 Note that this is not a u32 match!
4071 You can place a mark like this:
4074 # iptables -A PREROUTING -t mangle -i eth0 -j MARK --set-mark 6
4077 The number 6 is arbitrary.
4081 If you don't want to understand the full tc filter syntax, just use
4082 iptables, and only learn to select on fwmark.
4086 <Term>On the TOS field</Term>
4089 To select interactive, minimum delay traffic:
4092 # tc filter add dev ppp0 parent 1:0 protocol ip prio 10 u32 \
4093 match ip tos 0x10 0xff \
4097 Use 0x08 0xff for bulk traffic.
4104 For more filtering commands, see the Advanced Filters chapter.
4110 <Sect1 id="lartc.imq">
4111 <Title>The Intermediate queueing device (IMQ)</Title>
4114 The Intermediate queueing device is not a qdisc but its usage is tightly bound
4115 to qdiscs. Within linux, qdiscs are attached to network devices and everything
4116 that is queued to the device is first queued to the qdisc. From this concept,
4117 two limitations arise:
4121 1. Only egress shaping is possible (an ingress qdisc exists, but its
4122 possibilities are very limited compared to classful qdiscs).
4126 2. A qdisc can only see traffic of one interface, global limitations can't be
4131 IMQ is there to help solve those two limitations. In short, you can put
4132 everything you choose in a qdisc. Specially marked packets get intercepted
4133 in netfilter NF_IP_PRE_ROUTING and NF_IP_POST_ROUTING hooks and pass through
4134 the qdisc attached to an imq device. An iptables target is used for marking
4139 This enables you to do ingress shaping as you can just mark packets coming in from somewhere and/or treat interfaces as classes to set global limits.
4140 You can also do lots of other stuff like just putting your http traffic in a
4141 qdisc, put new connection requests in a qdisc, ...
4145 <Title>Sample configuration</Title>
4148 The first thing that might come to mind is use ingress shaping to give yourself
4149 a high guaranteed bandwidth. ;)
4150 Configuration is just like with any other interface:
4153 tc qdisc add dev imq0 root handle 1: htb default 20
4155 tc class add dev imq0 parent 1: classid 1:1 htb rate 2mbit burst 15k
4157 tc class add dev imq0 parent 1:1 classid 1:10 htb rate 1mbit
4158 tc class add dev imq0 parent 1:1 classid 1:20 htb rate 1mbit
4160 tc qdisc add dev imq0 parent 1:10 handle 10: pfifo
4161 tc qdisc add dev imq0 parent 1:20 handle 20: sfq
4163 tc filter add dev imq0 parent 10:0 protocol ip prio 1 u32 match \
4164 ip dst 10.0.0.230/32 flowid 1:10
4167 In this example u32 is used for classification. Other classifiers should work as
4169 Next traffic has to be selected and marked to be enqueued to imq0.
4172 iptables -t mangle -A PREROUTING -i eth0 -j IMQ --todev 0
4180 The IMQ iptables targets is valid in the PREROUTING and POSTROUTING chains of
4181 the mangle table. It's syntax is
4184 IMQ [ --todev n ] n : number of imq device
4187 An ip6tables target is also provided.
4191 Please note traffic is not enqueued when the target is hit but afterwards.
4192 The exact location where traffic enters the imq device depends on the
4193 direction of the traffic (in/out).
4194 These are the predefined netfilter hooks used by iptables:
4197 enum nf_ip_hook_priorities {
4198 NF_IP_PRI_FIRST = INT_MIN,
4199 NF_IP_PRI_CONNTRACK = -200,
4200 NF_IP_PRI_MANGLE = -150,
4201 NF_IP_PRI_NAT_DST = -100,
4202 NF_IP_PRI_FILTER = 0,
4203 NF_IP_PRI_NAT_SRC = 100,
4204 NF_IP_PRI_LAST = INT_MAX,
4211 For ingress traffic, imq registers itself with NF_IP_PRI_MANGLE + 1 priority
4212 which means packets enter the imq device directly after the mangle PREROUTING
4213 chain has been passed.
4217 For egress imq uses NF_IP_PRI_LAST which honours the fact that packets dropped
4218 by the filter table won't occupy bandwidth.
4222 The patches and some more information can be found at the
4224 URL="http://luxik.cdi.cz/~patrick/imq/"
4234 <chapter id="lartc.loadshare">
4235 <Title>Load sharing over multiple interfaces</Title>
4238 There are several ways of doing this. One of the easiest and straightforward
4239 ways is 'TEQL' - "True" (or "trivial") link equalizer. Like most things
4240 having to do with queueing, load sharing goes both ways. Both ends of a link
4241 may need to participate for full effect.
4245 Imagine this situation:
4251 +-------+ eth1 +-------+
4253 'network 1' ----| A | | B |---- 'network 2'
4255 +-------+ eth2 +-------+
4261 A and B are routers, and for the moment we'll assume both run Linux. If
4262 traffic is going from network 1 to network 2, router A needs to distribute
4263 the packets over both links to B. Router B needs to be configured to accept
4264 this. Same goes the other way around, when packets go from network 2 to
4265 network 1, router B needs to send the packets over both eth1 and eth2.
4269 The distributing part is done by a 'TEQL' device, like this (it couldn't be
4276 # tc qdisc add dev eth1 root teql0
4277 # tc qdisc add dev eth2 root teql0
4278 # ip link set dev teql0 up
4284 Don't forget the 'ip link set up' command!
4288 This needs to be done on both hosts. The device teql0 is basically a
4289 roundrobbin distributor over eth1 and eth2, for sending packets. No data
4290 ever comes in over an teql device, that just appears on the 'raw' eth1 and
4295 But now we just have devices, we also need proper routing. One way to do
4296 this is to assign a /31 network to both links, and a /31 to the teql0 device
4301 FIXME: does this need something like 'nobroadcast'? A /31 is too small to
4302 house a network address and a broadcast address - if this doesn't work as
4303 planned, try a /30, and adjust the ip addresses accordingly. You might even
4304 try to make eth1 and eth2 do without an IP address!
4311 # ip addr add dev eth1 10.0.0.0/31
4312 # ip addr add dev eth2 10.0.0.2/31
4313 # ip addr add dev teql0 10.0.0.4/31
4322 # ip addr add dev eth1 10.0.0.1/31
4323 # ip addr add dev eth2 10.0.0.3/31
4324 # ip addr add dev teql0 10.0.0.5/31
4330 Router A should now be able to ping 10.0.0.1, 10.0.0.3 and 10.0.0.5 over the
4331 2 real links and the 1 equalized device. Router B should be able to ping
4332 10.0.0.0, 10.0.0.2 and 10.0.0.4 over the links.
4336 If this works, Router A should make 10.0.0.5 its route for reaching network
4337 2, and Router B should make 10.0.0.4 its route for reaching network 1. For
4338 the special case where network 1 is your network at home, and network 2 is
4339 the Internet, Router A should make 10.0.0.5 its default gateway.
4342 <Sect1 id="lartc.loadshare.caveats">
4343 <Title>Caveats</Title>
4346 Nothing is as easy as it seems. eth1 and eth2 on both router A and B need to
4347 have return path filtering turned off, because they will otherwise drop
4348 packets destined for ip addresses other than their own:
4354 # echo 0 > /proc/net/ipv4/conf/eth1/rp_filter
4355 # echo 0 > /proc/net/ipv4/conf/eth2/rp_filter
4361 Then there is the nasty problem of packet reordering. Let's say 6 packets
4362 need to be sent from A to B - eth1 might get 1, 3 and 5. eth2 would then do
4363 2, 4 and 6. In an ideal world, router B would receive this in order, 1, 2,
4364 3, 4, 5, 6. But the possibility is very real that the kernel gets it like
4365 this: 2, 1, 4, 3, 6, 5. The problem is that this confuses TCP/IP. While not
4366 a problem for links carrying many different TCP/IP sessions, you won't be
4367 able to to a bundle multiple links and get to ftp a single file lots faster,
4368 except when your receiving or sending OS is Linux, which is not easily
4369 shaken by some simple reordering.
4373 However, for lots of applications, link load balancing is a great idea.
4377 <Sect1 id="lartc.loadshare.other">
4378 <Title>Other possibilities</Title>
4380 William Stearns has used an advanced tunneling setup to achieve good use of
4381 multiple, unrelated, internet connections together. It can be found on
4383 URL="http://www.stearns.org/tunnel/">his tunneling page</ULink>.
4386 The HOWTO may feature more about this in the future.
4391 <chapter id="lartc.netfilter">
4392 <Title>Netfilter & iproute - marking packets</Title>
4395 So far we've seen how iproute works, and netfilter was mentioned a few
4396 times. This would be a good time to browse through <ULink
4397 URL="http://netfilter.samba.org/unreliable-guides/"
4398 >Rusty's Remarkably Unreliable Guides</ULink
4401 URL="http://netfilter.filewatcher.org/"
4407 Netfilter allows us to filter packets, or mangle their headers. One special
4408 feature is that we can mark a packet with a number. This is done with the
4409 --set-mark facility.
4413 As an example, this command marks all packets destined for port 25, outgoing
4420 # iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 25 \
4421 -j MARK --set-mark 1
4427 Let's say that we have multiple connections, one that is fast (and
4428 expensive, per megabyte) and one that is slower, but flat fee. We would most
4429 certainly like outgoing mail to go via the cheap route.
4433 We've already marked the packets with a '1', we now instruct the routing
4434 policy database to act on this:
4440 # echo 201 mail.out >> /etc/iproute2/rt_tables
4441 # ip rule add fwmark 1 table mail.out
4443 0: from all lookup local
4444 32764: from all fwmark 1 lookup mail.out
4445 32766: from all lookup main
4446 32767: from all lookup default
4452 Now we generate the mail.out table with a route to the slow but cheap link:
4455 # /sbin/ip route add default via 195.96.98.253 dev ppp0 table mail.out
4461 And we are done. Should we want to make exceptions, there are lots of ways
4462 to achieve this. We can modify the netfilter statement to exclude certain
4463 hosts, or we can insert a rule with a lower priority that points to the main
4464 table for our excepted hosts.
4468 We can also use this feature to honour TOS bits by marking packets with a
4469 different type of service with different numbers, and creating rules to act
4470 on that. This way you can even dedicate, say, an ISDN line to interactive
4475 Needless to say, this also works fine on a host that's doing NAT
4480 IMPORTANT: We received a report that MASQ and SNAT at least collide
4481 with marking packets. Rusty Russell explains it in
4483 URL="http://lists.samba.org/pipermail/netfilter/2000-November/006089.html"
4484 >this posting</ULink
4485 >. Turn off the reverse path filter to make it work
4490 Note: to mark packets, you need to have some options enabled in your
4497 IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
4498 IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
4499 IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK) [Y/n/?]
4505 See also the <xref linkend="lartc.cookbook.squid"> in the
4506 <citetitle><xref linkend="lartc.cookbook"></citetitle>.
4511 <chapter id="lartc.adv-filter"
4512 xreflabel="Advanced filters for (re-)classifying packets">
4513 <Title>Advanced filters for (re-)classifying packets</Title>
4516 As explained in the section on classful queueing disciplines, filters are
4517 needed to classify packets into any of the sub-queues. These filters are
4518 called from within the classful qdisc.
4522 Here is an incomplete list of classifiers available:
4529 Bases the decision on how the firewall has marked the packet. This can be
4530 the easy way out if you don't want to learn tc filter syntax. See the
4531 Queueing chapter for details.
4538 Bases the decision on fields within the packet (i.e. source IP address, etc)
4545 Bases the decision on which route the packet will be routed by
4549 <Term>rsvp, rsvp6</Term>
4552 Routes packets based on <ULink
4553 URL="http://www.isi.edu/div7/rsvp/overview.html"
4556 on networks you control - the Internet does not respect RSVP.
4560 <Term>tcindex</Term>
4563 Used in the DSMARK qdisc, see the relevant section.
4570 Note that in general there are many ways in which you can classify packet
4571 and that it generally comes down to preference as to which system you wish
4576 Classifiers in general accept a few arguments in common. They are listed
4577 here for convenience:
4584 <Term>protocol</Term>
4587 The protocol this classifier will accept. Generally you will only be
4588 accepting only IP traffic. Required.
4595 The handle this classifier is to be attached to. This handle must be
4596 an already existing class. Required.
4603 The priority of this classifier. Lower numbers get tested first.
4610 This handle means different things to different filters.
4617 All the following sections will assume you are trying to shape the traffic
4618 going to <Literal remap="tt">HostA</Literal>. They will assume that the root class has been
4619 configured on 1: and that the class you want to send the selected traffic to
4623 <Sect1 id="lartc.adv-filter.u32">
4624 <Title>The <option>u32</option> classifier</Title>
4627 The U32 filter is the most advanced filter available in the current
4628 implementation. It entirely based on hashing tables, which make it
4629 robust when there are many filter rules.
4633 In its simplest form the U32 filter is a list of records, each
4634 consisting of two fields: a selector and an action. The selectors,
4635 described below, are compared with the currently processed IP packet
4636 until the first match occurs, and then the associated action is performed.
4637 The simplest type of action would be directing the packet into defined
4642 The command line of <Literal remap="tt">tc filter</Literal> program, used to configure the filter,
4643 consists of three parts: filter specification, a selector and an action.
4644 The filter specification can be defined as:
4650 tc filter add dev IF [ protocol PROTO ]
4651 [ (preference|priority) PRIO ]
4658 The <Literal remap="tt">protocol</Literal> field describes protocol that the filter will be
4659 applied to. We will only discuss case of <Literal remap="tt">ip</Literal> protocol. The
4660 <Literal remap="tt">preference</Literal> field (<Literal remap="tt">priority</Literal> can be used alternatively)
4661 sets the priority of currently defined filter. This is important, since
4662 you can have several filters (lists of rules) with different priorities.
4663 Each list will be passed in the order the rules were added, then list with
4664 lower priority (higher preference number) will be processed. The <Literal remap="tt">parent</Literal>
4665 field defines the CBQ tree top (e.g. 1:0), the filter should be attached
4670 The options described above apply to all filters, not only U32.
4674 <Title>U32 selector </Title>
4677 The U32 selector contains definition of the pattern, that will be matched
4678 to the currently processed packet. Precisely, it defines which bits are
4679 to be matched in the packet header and nothing more, but this simple
4680 method is very powerful. Let's take a look at the following examples,
4681 taken directly from a pretty complex, real-world filter:
4687 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
4688 match u32 00100000 00ff0000 at 0 flowid 1:10
4694 For now, leave the first line alone - all these parameters describe
4695 the filter's hash tables. Focus on the selector line, containing
4696 <Literal remap="tt">match</Literal> keyword. This selector will match to IP headers, whose
4697 second byte will be 0x10 (0010). As you can guess, the 00ff number is
4698 the match mask, telling the filter exactly which bits to match. Here
4699 it's 0xff, so the byte will match if it's exactly 0x10. The <Literal remap="tt">at</Literal>
4700 keyword means that the match is to be started at specified offset (in
4701 bytes) -- in this case it's beginning of the packet. Translating all
4702 that to human language, the packet will match if its Type of Service
4703 field will have `low delay' bits set. Let's analyze another rule:
4709 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
4710 match u32 00000016 0000ffff at nexthdr+0 flowid 1:10
4716 The <Literal remap="tt">nexthdr</Literal> option means next header encapsulated in the IP packet,
4717 i.e. header of upper-layer protocol. The match will also start here
4718 at the beginning of the next header. The match should occur in the
4719 second, 32-bit word of the header. In TCP and UDP protocols this field
4720 contains packet's destination port. The number is given in big-endian
4721 format, i.e. older bits first, so we simply read 0x0016 as 22 decimal,
4722 which stands for SSH service if this was TCP. As you guess, this match
4723 is ambiguous without a context, and we will discuss this later.
4727 Having understood all the above, we will find the following selector
4728 quite easy to read: <Literal remap="tt">match c0a80100 ffffff00 at 16</Literal>. What we
4729 got here is a three byte match at 17-th byte, counting from the IP
4730 header start. This will match for packets with destination address
4731 anywhere in 192.168.1/24 network. After analyzing the examples, we
4732 can summarize what we have learned.
4738 <Title>General selectors</Title>
4741 General selectors define the pattern, mask and offset the pattern
4742 will be matched to the packet contents. Using the general selectors
4743 you can match virtually any single bit in the IP (or upper layer)
4744 header. They are more difficult to write and read, though, than
4745 specific selectors that described below. The general selector syntax
4752 match [ u32 | u16 | u8 ] PATTERN MASK [ at OFFSET | nexthdr+OFFSET]
4758 One of the keywords <Literal remap="tt">u32</Literal>, <Literal remap="tt">u16</Literal> or <Literal remap="tt">u8</Literal> specifies
4759 length of the pattern in bits. PATTERN and MASK should follow, of length
4760 defined by the previous keyword. The OFFSET parameter is the offset,
4761 in bytes, to start matching. If <Literal remap="tt">nexthdr+</Literal> keyword is given,
4762 the offset is relative to start of the upper layer header.
4772 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
4773 match u8 64 0xff at 8 \
4780 Packet will match to this rule, if its time to live (TTL) is 64.
4781 TTL is the field starting just after 8-th byte of the IP header.
4787 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
4788 match u8 0x10 0xff at nexthdr+13 \
4796 FIXME: it has been pointed out that this syntax does not work currently.
4800 Use this to match ACKs on packets smaller than 64 bytes:
4806 ## match acks the hard way,
4808 ## IP header length 0x5(32 bit words),
4809 ## IP Total length 0x34 (ACK + 12 bytes of TCP options)
4810 ## TCP ack set (bit 5, offset 33)
4811 # tc filter add dev ppp14 parent 1:0 protocol ip prio 10 u32 \
4812 match ip protocol 6 0xff \
4813 match u8 0x05 0x0f at 0 \
4814 match u16 0x0000 0xffc0 at 2 \
4815 match u8 0x10 0xff at 33 \
4822 This rule will only match TCP packets with ACK bit set, and no further
4823 payload. Here we can see an example of using two selectors, the final result
4824 will be logical AND of their results. If we take a look at TCP header
4825 diagram, we can see that the ACK bit is second older bit (0x10) in the 14-th
4826 byte of the TCP header (<Literal remap="tt">at nexthdr+13</Literal>). As for the second
4827 selector, if we'd like to make our life harder, we could write <Literal remap="tt">match u8
4828 0x06 0xff at 9</Literal> instead of using the specific selector <Literal remap="tt">protocol
4829 tcp</Literal>, because 6 is the number of TCP protocol, present in 10-th byte of
4830 the IP header. On the other hand, in this example we couldn't use any
4831 specific selector for the first match - simply because there's no specific
4832 selector to match TCP ACK bits.
4838 <Title>Specific selectors</Title>
4841 The following table contains a list of all specific selectors
4842 the author of this section has found in the <Literal remap="tt">tc</Literal> program
4843 source code. They simply make your life easier and increase readability
4844 of your filter's configuration.
4848 FIXME: table placeholder - the table is in separate file ,,selector.html''
4852 FIXME: it's also still in Polish :-(
4856 FIXME: must be sgml'ized
4866 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
4867 match ip tos 0x10 0xff \
4874 FIXME: tcp dst match does not work as described below:
4878 The above rule will match packets which have the TOS field set to 0x10.
4879 The TOS field starts at second byte of the packet and is one byte big,
4880 so we could write an equivalent general selector: <Literal remap="tt">match u8 0x10 0xff
4881 at 1</Literal>. This gives us hint to the internals of U32 filter -- the
4882 specific rules are always translated to general ones, and in this
4883 form they are stored in the kernel memory. This leads to another conclusion
4884 -- the <Literal remap="tt">tcp</Literal> and <Literal remap="tt">udp</Literal> selectors are exactly the same
4885 and this is why you can't use single <Literal remap="tt">match tcp dst 53 0xffff</Literal>
4886 selector to match TCP packets sent to given port -- they will also
4887 match UDP packets sent to this port. You must remember to also specify
4888 the protocol and end up with the following rule:
4894 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
4895 match tcp dst 53 0xffff \
4896 match ip protocol 0x6 0xff \
4906 <Sect1 id="lartc.adv-filter.route">
4907 <Title>The <option>route</option> classifier</Title>
4910 This classifier filters based on the results of the routing tables. When a
4911 packet that is traversing through the classes reaches one that is marked
4912 with the "route" filter, it splits the packets up based on information in
4919 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 route
4925 Here we add a route classifier onto the parent node 1:0 with priority 100.
4926 When a packet reaches this node (which, since it is the root, will happen
4927 immediately) it will consult the routing table and if one matches will
4928 send it to the given class and give it a priority of 100. Then, to finally
4929 kick it into action, you add the appropriate routing entry:
4933 The trick here is to define 'realm' based on either destination or source.
4934 The way to do it is like this:
4940 # ip route add Host/Network via Gateway dev Device realm RealmNumber
4946 For instance, we can define our destination network 192.168.10.0 with a realm
4953 # ip route add 192.168.10.0/24 via 192.168.10.1 dev eth1 realm 10
4959 When adding route filters, we can use realm numbers to represent the
4960 networks or hosts and specify how the routes match the filters.
4966 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
4967 route to 10 classid 1:10
4973 The above rule says packets going to the network 192.168.10.0 match class id
4978 Route filter can also be used to match source routes. For example, there is
4979 a subnetwork attached to the Linux router on eth2.
4985 # ip route add 192.168.2.0/24 dev eth2 realm 2
4986 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
4987 route from 2 classid 1:2
4993 Here the filter specifies that packets from the subnetwork 192.168.2.0
4994 (realm 2) will match class id 1:2.
4999 <Sect1 id="lartc.adv-filter.policing">
5000 <Title>Policing filters</Title>
5003 To make even more complicated setups possible, you can have filters that
5004 only match up to a certain bandwidth. You can declare a filter to entirely
5005 cease matching above a certain rate, or only to not match only the bandwidth
5006 exceeding a certain rate.
5010 So if you decided to police at 4mbit/s, but 5mbit/s of traffic is present,
5011 you can stop matching either the entire 5mbit/s, or only not match 1mbit/s,
5012 and do send 4mbit/s to the configured class.
5016 If bandwidth exceeds the configured rate, you can drop a packet, reclassify
5017 it, or see if another filter will match it.
5021 <Title>Ways to police</Title>
5024 There are basically two ways to police. If you compiled the kernel
5025 with 'Estimators', the kernel can measure for each filter how much traffic
5026 it is passing, more or less. These estimators are very easy on the CPU, as
5027 they simply count 25 times per second how many data has been passed, and
5028 calculate the bitrate from that.
5032 The other way works again via a Token Bucket Filter, this time living within
5033 your filter. The TBF only matches traffic UP TO your configured bandwidth,
5034 if more is offered, only the excess is subject to the configured overlimit
5039 <Title>With the kernel estimator</Title>
5042 This is very simple and has only one parameter: avrate. Either the flow
5043 remains below avrate, and the filter classifies the traffic to the classid
5044 configured, or your rate exceeds it in which case the specified action is
5045 taken, which is 'reclassify' by default.
5049 The kernel uses an Exponential Weighted Moving Average for your bandwidth
5050 which makes it less sensitive to short bursts.
5056 <Title>With Token Bucket Filter</Title>
5059 Uses the following parameters:
5092 Which behave mostly identical to those described in the Token Bucket Filter
5093 section. Please note however that if you set the mtu of a TBF policer too
5094 low, *no* packets will pass, whereas the egress TBF qdisc will just pass
5099 Another difference is that a policer can only let a packet pass, or drop it.
5100 It cannot delay hold on to it in order to delay it.
5108 <Title>Overlimit actions</Title>
5111 If your filter decides that it is overlimit, it can take 'actions'.
5112 Currently, three actions are available:
5116 <Term>continue</Term>
5119 Causes this filter not to match, but perhaps other filters will.
5126 This is a very fierce option which simply discards traffic exceeding a
5127 certain rate. It is often used in the ingress policer and has limited uses.
5128 For example, you may have a name server that falls over if offered more than
5129 5mbit/s of packets, in which case an ingress filter could be used to make
5130 sure no more is ever offered.
5134 <Term>Pass/OK</Term>
5137 Pass on traffic ok. Might be used to disable a complicated filter, but leave
5142 <Term>reclassify</Term>
5145 Most often comes down to reclassification to Best Effort. This is the
5155 <Title>Examples</Title>
5158 The only real example known is mentioned in the 'Protecting your host
5159 from SYN floods' section.
5163 FIXME: if you have used this, please share your experience with us
5170 <Sect1 id="lartc.adv-filter.hashing">
5171 <Title>Hashing filters for very fast massive filtering</Title>
5174 If you have a need for thousands of rules, for example if you have a lot of
5175 clients or computers, all with different QoS specifications, you may find
5176 that the kernel spends a lot of time matching all those rules.
5180 By default, all filters reside in one big chain which is matched in
5181 descending order of priority. If you have 1000 rules, 1000 checks may be
5182 needed to determine what to do with a packet.
5186 Matching would go much quicker if you would have 256 chains with each four
5187 rules - if you could divide packets over those 256 chains, so that the right
5192 Hashing makes this possible. Let's say you have 1024 cable modem customers in
5193 your network, with IP addresses ranging from 1.2.0.0 to 1.2.3.255, and each
5194 has to go in another bin, for example 'lite', 'regular' and 'premium'. You
5195 would then have 1024 rules like this:
5201 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5203 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5206 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5207 1.2.3.254 classid 1:3
5208 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5209 1.2.3.255 classid 1:2
5215 To speed this up, we can use the last part of the IP address as a 'hash
5216 key'. We then get 256 tables, the first of which looks like this:
5219 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5221 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5223 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5225 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5232 The next one starts like this:
5235 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5243 This way, only four checks are needed at most, two on average.
5247 Configuration is pretty complicated, but very worth it by the time you have
5248 this many rules. First we make a filter root, then we create a table with
5252 # tc filter add dev eth1 parent 1:0 prio 5 protocol ip u32
5253 # tc filter add dev eth1 parent 1:0 prio 5 handle 2: protocol ip u32 divisor 256
5259 Now we add some rules to entries in the created table:
5265 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5266 match ip src 1.2.0.123 flowid 1:1
5267 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5268 match ip src 1.2.1.123 flowid 1:2
5269 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5270 match ip src 1.2.3.123 flowid 1:3
5271 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5272 match ip src 1.2.4.123 flowid 1:2
5275 This is entry 123, which contains matches for 1.2.0.123, 1.2.1.123,
5276 1.2.2.123, 1.2.3.123, and sends them to 1:1, 1:2, 1:3 and 1:2 respectively.
5277 Note that we need to specify our hash bucket in hex, 0x7b is 123.
5281 Next create a 'hashing filter' that directs traffic to the right entry in
5285 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 800:: \
5286 match ip src 1.2.0.0/16 \
5287 hashkey mask 0x000000ff at 12 \
5291 Ok, some numbers need explaining. The default hash table is called 800:: and
5292 all filtering starts there. Then we select the source address, which lives
5293 as position 12, 13, 14 and 15 in the IP header, and indicate that we are
5294 only interested in the last part. This we send to hash table 2:, which we
5299 It is quite complicated, but it does work in practice and performance will
5300 be staggering. Note that this example could be improved to the ideal case
5301 where each chain contains 1 filter!
5308 <chapter id="lartc.kernel">
5309 <Title>Kernel network parameters </Title>
5313 The kernel has lots of parameters which
5314 can be tuned for different circumstances. While, as usual, the default
5315 parameters serve 99% of installations very well, we don't call this the
5316 Advanced HOWTO for the fun of it!
5320 The interesting bits are in /proc/sys/net, take a look there. Not everything
5321 will be documented here initially, but we're working on it.
5325 In the meantime you may want to have a look at the Linux-Kernel sources;
5326 read the file Documentation/filesystems/proc.txt. Most of the
5327 features are explained there.
5334 <Sect1 id="lartc.kernel.rpf"
5335 xreflabel="Reverse Path Filtering">
5336 <Title>Reverse Path Filtering</Title>
5339 By default, routers route everything, even packets which 'obviously' don't
5340 belong on your network. A common example is private IP space escaping onto
5341 the Internet. If you have an interface with a route of 195.96.96.0/24 to it,
5342 you do not expect packets from 212.64.94.1 to arrive there.
5346 Lots of people will want to turn this feature off, so the kernel hackers
5347 have made it easy. There are files in /proc where you can tell
5348 the kernel to do this for you. The method is called "Reverse Path
5349 Filtering". Basically, if the reply to this packet wouldn't go out the
5350 interface this packet came in, then this is a bogus packet and should be
5355 The following fragment will turn this on for all current and future
5362 # for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
5363 > echo 2 > $i
5370 Going by the example above, if a packet arrived on the Linux router on eth1
5371 claiming to come from the Office+ISP subnet, it would be dropped. Similarly,
5372 if a packet came from the Office subnet, claiming to be from somewhere
5373 outside your firewall, it would be dropped also.
5377 The above is full reverse path filtering. The default is to only filter
5378 based on IPs that are on directly connected networks. This is because the
5379 full filtering breaks in the case of asymmetric routing (where packets come
5380 in one way and go out another, like satellite traffic, or if you have
5381 dynamic (bgp, ospf, rip) routes in your network. The data comes down
5382 through the satellite dish and replies go back through normal land-lines).
5386 If this exception applies to you (and you'll probably know if it does) you
5387 can simply turn off the rp_filter on the interface where the
5388 satellite data comes in. If you want to see if any packets are being
5389 dropped, the log_martians file in the same directory will tell
5390 the kernel to log them to your syslog.
5396 # echo 1 >/proc/sys/net/ipv4/conf/<interfacename>/log_martians
5402 FIXME: is setting the conf/{default,all}/* files enough? - martijn
5407 <Sect1 id="lartc.kernel.obscure">
5408 <Title>Obscure settings</Title>
5411 Ok, there are a lot of parameters which can be modified. We try to list them
5412 all. Also documented (partly) in Documentation/ip-sysctl.txt.
5416 Some of these settings have different defaults based on whether you
5417 answered 'Yes' to 'Configure as router and not host' while compiling your
5422 Oskar Andreasson also has a page on all these flags and it appears to be
5423 better than ours, so also check
5424 <ulink url="http://ipsysctl-tutorial.frozentux.net/">
5425 http://ipsysctl-tutorial.frozentux.net/</ulink>.
5429 <Title>Generic ipv4</Title>
5432 As a generic note, most rate limiting features don't work on loopback, so
5433 don't test them locally. The limits are supplied in 'jiffies', and are
5434 enforced using the earlier mentioned token bucket filter.
5438 The kernel has an internal clock which runs at 'HZ' ticks (or 'jiffies') per
5439 second. On Intel, 'HZ' is mostly 100. So setting a *_rate file to, say 50,
5440 would allow for 2 packets per second. The token bucket filter is also
5441 configured to allow for a burst of at most 6 packets, if enough tokens have
5446 Several entries in the following list have been copied from
5447 /usr/src/linux/Documentation/networking/ip-sysctl.txt, written by Alexey
5448 Kuznetsov <kuznet@ms2.inr.ac.ru> and Andi Kleen <ak@muc.de>
5452 <Term>/proc/sys/net/ipv4/icmp_destunreach_rate</Term>
5455 If the kernel decides that it can't deliver a packet, it will drop it, and
5456 send the source of the packet an ICMP notice to this effect.
5460 <Term>/proc/sys/net/ipv4/icmp_echo_ignore_all</Term>
5463 Don't act on echo packets at all. Please don't set this by default, but if
5464 you are used as a relay in a DoS attack, it may be useful.
5468 <Term>/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts [Useful]</Term>
5471 If you ping the broadcast address of a network, all hosts are supposed to
5472 respond. This makes for a dandy denial-of-service tool. Set this to 1 to
5473 ignore these broadcast messages.
5477 <Term>/proc/sys/net/ipv4/icmp_echoreply_rate</Term>
5480 The rate at which echo replies are sent to any one destination.
5484 <Term>/proc/sys/net/ipv4/icmp_ignore_bogus_error_responses</Term>
5487 Set this to ignore ICMP errors caused by hosts in the network reacting badly
5488 to frames sent to what they perceive to be the broadcast address.
5492 <Term>/proc/sys/net/ipv4/icmp_paramprob_rate</Term>
5495 A relatively unknown ICMP message, which is sent in response to incorrect
5496 packets with broken IP or TCP headers. With this file you can control the
5497 rate at which it is sent.
5501 <Term>/proc/sys/net/ipv4/icmp_timeexceed_rate</Term>
5504 This the famous cause of the 'Solaris middle star' in traceroutes. Limits
5505 number of ICMP Time Exceeded messages sent.
5509 <Term>/proc/sys/net/ipv4/igmp_max_memberships</Term>
5512 Maximum number of listening igmp (multicast) sockets on the host.
5513 FIXME: Is this true?
5517 <Term>/proc/sys/net/ipv4/inet_peer_gc_maxtime</Term>
5520 FIXME: Add a little explanation about the inet peer storage?
5522 Minimum interval between garbage collection passes. This interval is in
5523 effect under low (or absent) memory pressure on the pool. Measured in
5528 <Term>/proc/sys/net/ipv4/inet_peer_gc_mintime</Term>
5531 Minimum interval between garbage collection passes. This interval is in
5532 effect under high memory pressure on the pool. Measured in jiffies.
5536 <Term>/proc/sys/net/ipv4/inet_peer_maxttl</Term>
5539 Maximum time-to-live of entries. Unused entries will expire after this
5540 period of time if there is no memory pressure on the pool (i.e. when the
5541 number of entries in the pool is very small). Measured in jiffies.
5545 <Term>/proc/sys/net/ipv4/inet_peer_minttl</Term>
5548 Minimum time-to-live of entries. Should be enough to cover fragment
5549 time-to-live on the reassembling side. This minimum time-to-live
5550 is guaranteed if the pool size is less than inet_peer_threshold.
5551 Measured in jiffies.
5555 <Term>/proc/sys/net/ipv4/inet_peer_threshold</Term>
5558 The approximate size of the INET peer storage. Starting from this threshold
5559 entries will be thrown aggressively. This threshold also determines
5560 entries' time-to-live and time intervals between garbage collection passes.
5561 More entries, less time-to-live, less GC interval.
5565 <Term>/proc/sys/net/ipv4/ip_autoconfig</Term>
5568 This file contains the number one if the host received its IP configuration by
5569 RARP, BOOTP, DHCP or a similar mechanism. Otherwise it is zero.
5573 <Term>/proc/sys/net/ipv4/ip_default_ttl</Term>
5576 Time To Live of packets. Set to a safe 64. Raise it if you have a huge
5577 network. Don't do so for fun - routing loops cause much more damage that
5578 way. You might even consider lowering it in some circumstances.
5582 <Term>/proc/sys/net/ipv4/ip_dynaddr</Term>
5585 You need to set this if you use dial-on-demand with a dynamic interface
5586 address. Once your demand interface comes up, any local TCP sockets which haven't seen replies will be rebound to have the right address. This solves the problem that the
5587 connection that brings up your interface itself does not work, but the
5592 <Term>/proc/sys/net/ipv4/ip_forward</Term>
5595 If the kernel should attempt to forward packets. Off by default.
5599 <Term>/proc/sys/net/ipv4/ip_local_port_range</Term>
5602 Range of local ports for outgoing connections. Actually quite small by
5603 default, 1024 to 4999.
5607 <Term>/proc/sys/net/ipv4/ip_no_pmtu_disc</Term>
5610 Set this if you want to disable Path MTU discovery - a technique to
5611 determine the largest Maximum Transfer Unit possible on your path. See also
5612 the section on Path MTU discovery in the
5613 <citetitle><xref linkend="lartc.cookbook"></citetitle> chapter.
5617 <Term>/proc/sys/net/ipv4/ipfrag_high_thresh</Term>
5620 Maximum memory used to reassemble IP fragments. When
5621 ipfrag_high_thresh bytes of memory is allocated for this purpose,
5622 the fragment handler will toss packets until ipfrag_low_thresh
5627 <Term>/proc/sys/net/ipv4/ip_nonlocal_bind</Term>
5630 Set this if you want your applications to be able to bind to an address
5631 which doesn't belong to a device on your system. This can be useful when
5632 your machine is on a non-permanent (or even dynamic) link, so your services
5633 are able to start up and bind to a specific address when your link is down.
5637 <Term>/proc/sys/net/ipv4/ipfrag_low_thresh</Term>
5640 Minimum memory used to reassemble IP fragments.
5644 <Term>/proc/sys/net/ipv4/ipfrag_time</Term>
5647 Time in seconds to keep an IP fragment in memory.
5651 <Term>/proc/sys/net/ipv4/tcp_abort_on_overflow</Term>
5654 A boolean flag controlling the behaviour under lots of incoming connections.
5655 When enabled, this causes the kernel to actively send RST packets when a
5656 service is overloaded.
5660 <Term>/proc/sys/net/ipv4/tcp_fin_timeout</Term>
5663 Time to hold socket in state FIN-WAIT-2, if it was closed by our side. Peer
5664 can be broken and never close its side, or even died unexpectedly. Default
5665 value is 60sec. Usual value used in 2.2 was 180 seconds, you may restore it,
5666 but remember that if your machine is even underloaded WEB server, you risk
5667 to overflow memory with kilotons of dead sockets, FIN-WAIT-2 sockets are
5668 less dangerous than FIN-WAIT-1, because they eat maximum 1.5K of memory, but
5669 they tend to live longer. Cf. tcp_max_orphans.
5673 <Term>/proc/sys/net/ipv4/tcp_keepalive_time</Term>
5676 How often TCP sends out keepalive messages when keepalive is enabled.
5682 <Term>/proc/sys/net/ipv4/tcp_keepalive_intvl</Term>
5685 How frequent probes are retransmitted, when a probe isn't acknowledged.
5687 Default: 75 seconds.
5691 <Term>/proc/sys/net/ipv4/tcp_keepalive_probes</Term>
5694 How many keepalive probes TCP will send, until it decides that the
5695 connection is broken.
5699 Multiplied with tcp_keepalive_intvl, this gives the time a link can be
5700 non-responsive after a keepalive has been sent.
5704 <Term>/proc/sys/net/ipv4/tcp_max_orphans</Term>
5707 Maximal number of TCP sockets not attached to any user file handle, held by
5708 system. If this number is exceeded orphaned connections are reset
5709 immediately and warning is printed. This limit exists only to prevent simple
5710 DoS attacks, you _must_ not rely on this or lower the limit artificially,
5711 but rather increase it (probably, after increasing installed memory), if
5712 network conditions require more than default value, and tune network
5713 services to linger and kill such states more aggressively. Let me remind you
5714 again: each orphan eats up to 64K of unswappable memory.
5718 <Term>/proc/sys/net/ipv4/tcp_orphan_retries</Term>
5721 How may times to retry before killing TCP connection, closed by our side.
5722 Default value 7 corresponds to 50sec-16min depending on RTO. If your machine
5723 is a loaded WEB server, you should think about lowering this value, such
5724 sockets may consume significant resources. Cf. tcp_max_orphans.
5728 <Term>/proc/sys/net/ipv4/tcp_max_syn_backlog</Term>
5731 Maximal number of remembered connection requests, which still did not
5732 receive an acknowledgment from connecting client. Default value is 1024 for
5733 systems with more than 128Mb of memory, and 128 for low memory machines. If
5734 server suffers of overload, try to increase this number. Warning! If you
5735 make it greater than 1024, it would be better to change TCP_SYNQ_HSIZE in
5736 include/net/tcp.h to keep TCP_SYNQ_HSIZE*16<=tcp_max_syn_backlog and to
5741 <Term>/proc/sys/net/ipv4/tcp_max_tw_buckets</Term>
5744 Maximal number of timewait sockets held by system simultaneously. If this
5745 number is exceeded time-wait socket is immediately destroyed and warning is
5746 printed. This limit exists only to prevent simple DoS attacks, you _must_
5747 not lower the limit artificially, but rather increase it (probably, after
5748 increasing installed memory), if network conditions require more than
5753 <Term>/proc/sys/net/ipv4/tcp_retrans_collapse</Term>
5756 Bug-to-bug compatibility with some broken printers.
5757 On retransmit try to send bigger packets to work around bugs in
5762 <Term>/proc/sys/net/ipv4/tcp_retries1</Term>
5765 How many times to retry before deciding that something is wrong
5766 and it is necessary to report this suspicion to network layer.
5767 Minimal RFC value is 3, it is default, which corresponds
5768 to 3sec-8min depending on RTO.
5772 <Term>/proc/sys/net/ipv4/tcp_retries2</Term>
5775 How may times to retry before killing alive TCP connection.
5777 URL="http://www.ietf.org/rfc/rfc1122.txt"
5780 says that the limit should be longer than 100 sec.
5781 It is too small number. Default value 15 corresponds to 13-30min
5786 <Term>/proc/sys/net/ipv4/tcp_rfc1337</Term>
5789 This boolean enables a fix for 'time-wait assassination hazards in tcp', described
5790 in RFC 1337. If enabled, this causes the kernel to drop RST packets for
5791 sockets in the time-wait state.
5797 <Term>/proc/sys/net/ipv4/tcp_sack</Term>
5800 Use Selective ACK which can be used to signify that specific packets are
5801 missing - therefore helping fast recovery.
5805 <Term>/proc/sys/net/ipv4/tcp_stdurg</Term>
5808 Use the Host requirements interpretation of the TCP urg pointer
5811 Most hosts use the older BSD interpretation, so if you turn this on
5812 Linux might not communicate correctly with them.
5818 <Term>/proc/sys/net/ipv4/tcp_syn_retries</Term>
5821 Number of SYN packets the kernel will send before giving up on the new
5826 <Term>/proc/sys/net/ipv4/tcp_synack_retries</Term>
5829 To open the other side of the connection, the kernel sends a SYN with a
5830 piggybacked ACK on it, to acknowledge the earlier received SYN. This is part
5831 2 of the threeway handshake. This setting determines the number of SYN+ACK
5832 packets sent before the kernel gives up on the connection.
5836 <Term>/proc/sys/net/ipv4/tcp_timestamps</Term>
5839 Timestamps are used, amongst other things, to protect against wrapping
5840 sequence numbers. A 1 gigabit link might conceivably re-encounter a previous
5841 sequence number with an out-of-line value, because it was of a previous
5842 generation. The timestamp will let it recognize this 'ancient packet'.
5846 <Term>/proc/sys/net/ipv4/tcp_tw_recycle</Term>
5849 Enable fast recycling TIME-WAIT sockets. Default value is 1.
5850 It should not be changed without advice/request of technical experts.
5854 <Term>/proc/sys/net/ipv4/tcp_window_scaling</Term>
5857 TCP/IP normally allows windows up to 65535 bytes big. For really fast
5858 networks, this may not be enough. The window scaling options allows for
5859 almost gigabyte windows, which is good for high bandwidth*delay products.
5868 <Title>Per device settings</Title>
5871 DEV can either stand for a real interface, or for 'all' or 'default'.
5872 Default also changes settings for interfaces yet to be created.
5876 <Term>/proc/sys/net/ipv4/conf/DEV/accept_redirects</Term>
5879 If a router decides that you are using it for a wrong purpose (ie, it needs
5880 to resend your packet on the same interface), it will send us a ICMP
5881 Redirect. This is a slight security risk however, so you may want to turn it
5882 off, or use secure redirects.
5886 <Term>/proc/sys/net/ipv4/conf/DEV/accept_source_route</Term>
5889 Not used very much anymore. You used to be able to give a packet a list of
5890 IP addresses it should visit on its way. Linux can be made to honor this IP
5895 <Term>/proc/sys/net/ipv4/conf/DEV/bootp_relay</Term>
5898 Accept packets with source address 0.b.c.d with destinations not to this host
5899 as local ones. It is supposed that a BOOTP relay daemon will catch and forward
5904 The default is 0, since this feature is not implemented yet (kernel version
5909 <Term>/proc/sys/net/ipv4/conf/DEV/forwarding</Term>
5912 Enable or disable IP forwarding on this interface.
5916 <Term>/proc/sys/net/ipv4/conf/DEV/log_martians</Term>
5920 <citetitle><xref linkend="lartc.kernel.rpf"></citetitle>.
5924 <Term>/proc/sys/net/ipv4/conf/DEV/mc_forwarding</Term>
5927 If we do multicast forwarding on this interface
5931 <Term>/proc/sys/net/ipv4/conf/DEV/proxy_arp</Term>
5934 If you set this to 1, this interface will respond to ARP requests for
5935 addresses the kernel has routes to. Can be very useful when building 'ip
5936 pseudo bridges'. Do take care that your netmasks are very correct before
5937 enabling this! Also be aware that the rp_filter, mentioned elsewhere, also
5938 operates on ARP queries!
5942 <Term>/proc/sys/net/ipv4/conf/DEV/rp_filter</Term>
5946 <citetitle><xref linkend="lartc.kernel.rpf"></citetitle>.
5950 <Term>/proc/sys/net/ipv4/conf/DEV/secure_redirects</Term>
5953 Accept ICMP redirect messages only for gateways, listed in default gateway
5954 list. Enabled by default.
5958 <Term>/proc/sys/net/ipv4/conf/DEV/send_redirects</Term>
5961 If we send the above mentioned redirects.
5965 <Term>/proc/sys/net/ipv4/conf/DEV/shared_media</Term>
5968 If it is not set the kernel does not assume that different subnets on this
5969 device can communicate directly. Default setting is 'yes'.
5973 <Term>/proc/sys/net/ipv4/conf/DEV/tag</Term>
5985 <Title>Neighbor policy</Title>
5988 Dev can either stand for a real interface, or for 'all' or 'default'.
5989 Default also changes settings for interfaces yet to be created.
5993 <Term>/proc/sys/net/ipv4/neigh/DEV/anycast_delay</Term>
5996 Maximum for random delay of answers to neighbor solicitation messages in
5997 jiffies (1/100 sec). Not yet implemented (Linux does not have anycast support
6002 <Term>/proc/sys/net/ipv4/neigh/DEV/app_solicit</Term>
6005 Determines the number of requests to send to the user level ARP daemon. Use 0
6010 <Term>/proc/sys/net/ipv4/neigh/DEV/base_reachable_time</Term>
6013 A base value used for computing the random reachable time value as specified
6018 <Term>/proc/sys/net/ipv4/neigh/DEV/delay_first_probe_time</Term>
6021 Delay for the first time probe if the neighbor is reachable. (see
6026 <Term>/proc/sys/net/ipv4/neigh/DEV/gc_stale_time</Term>
6029 Determines how often to check for stale ARP entries. After an ARP entry is
6030 stale it will be resolved again (which is useful when an IP address migrates
6031 to another machine). When ucast_solicit is greater than 0 it first tries to
6032 send an ARP packet directly to the known host When that fails and
6033 mcast_solicit is greater than 0, an ARP request is broadcast.
6037 <Term>/proc/sys/net/ipv4/neigh/DEV/locktime</Term>
6040 An ARP/neighbor entry is only replaced with a new one if the old is at least
6041 locktime old. This prevents ARP cache thrashing.
6045 <Term>/proc/sys/net/ipv4/neigh/DEV/mcast_solicit</Term>
6048 Maximum number of retries for multicast solicitation.
6052 <Term>/proc/sys/net/ipv4/neigh/DEV/proxy_delay</Term>
6055 Maximum time (real time is random [0..proxytime]) before answering to an ARP
6056 request for which we have an proxy ARP entry. In some cases, this is used to
6057 prevent network flooding.
6061 <Term>/proc/sys/net/ipv4/neigh/DEV/proxy_qlen</Term>
6064 Maximum queue length of the delayed proxy arp timer. (see proxy_delay).
6068 <Term>/proc/sys/net/ipv4/neigh/DEV/retrans_time</Term>
6071 The time, expressed in jiffies (1/100 sec), between retransmitted Neighbor
6072 Solicitation messages. Used for address resolution and to determine if a
6073 neighbor is unreachable.
6077 <Term>/proc/sys/net/ipv4/neigh/DEV/ucast_solicit</Term>
6080 Maximum number of retries for unicast solicitation.
6084 <Term>/proc/sys/net/ipv4/neigh/DEV/unres_qlen</Term>
6087 Maximum queue length for a pending arp request - the number of packets which
6088 are accepted from other layers while the ARP address is still resolved.
6092 <Term>Internet QoS: Architectures and Mechanisms for Quality of Service,
6093 Zheng Wang, ISBN 1-55860-608-4</Term>
6096 Hardcover textbook covering topics
6097 related to Quality of Service. Good for understanding basic concepts.
6106 <Title>Routing settings</Title>
6112 <Term>/proc/sys/net/ipv4/route/error_burst</Term>
6115 These parameters are used to limit the warning messages written to the kernel
6116 log from the routing code. The higher the error_cost factor is, the fewer
6117 messages will be written. Error_burst controls when messages will be dropped.
6118 The default settings limit warning messages to one every five seconds.
6122 <Term>/proc/sys/net/ipv4/route/error_cost</Term>
6125 These parameters are used to limit the warning messages written to the kernel
6126 log from the routing code. The higher the error_cost factor is, the fewer
6127 messages will be written. Error_burst controls when messages will be dropped.
6128 The default settings limit warning messages to one every five seconds.
6132 <Term>/proc/sys/net/ipv4/route/flush</Term>
6135 Writing to this file results in a flush of the routing cache.
6139 <Term>/proc/sys/net/ipv4/route/gc_elasticity</Term>
6142 Values to control the frequency and behavior of the garbage collection
6143 algorithm for the routing cache. This can be important for when doing
6144 fail over. At least gc_timeout seconds will elapse before Linux will skip
6145 to another route because the previous one has died. By default set to 300,
6146 you may want to lower it if you want to have a speedy fail over.
6151 URL="http://mailman.ds9a.nl/pipermail/lartc/2002q1/002667.html"
6153 > by Ard van Breemen.
6157 <Term>/proc/sys/net/ipv4/route/gc_interval</Term>
6160 See /proc/sys/net/ipv4/route/gc_elasticity.
6164 <Term>/proc/sys/net/ipv4/route/gc_min_interval</Term>
6167 See /proc/sys/net/ipv4/route/gc_elasticity.
6171 <Term>/proc/sys/net/ipv4/route/gc_thresh</Term>
6174 See /proc/sys/net/ipv4/route/gc_elasticity.
6178 <Term>/proc/sys/net/ipv4/route/gc_timeout</Term>
6181 See /proc/sys/net/ipv4/route/gc_elasticity.
6185 <Term>/proc/sys/net/ipv4/route/max_delay</Term>
6188 Delays for flushing the routing cache.
6192 <Term>/proc/sys/net/ipv4/route/max_size</Term>
6195 Maximum size of the routing cache. Old entries will be purged once the cache
6196 reached has this size.
6200 <Term>/proc/sys/net/ipv4/route/min_adv_mss</Term>
6207 <Term>/proc/sys/net/ipv4/route/min_delay</Term>
6210 Delays for flushing the routing cache.
6214 <Term>/proc/sys/net/ipv4/route/min_pmtu</Term>
6221 <Term>/proc/sys/net/ipv4/route/mtu_expires</Term>
6228 <Term>/proc/sys/net/ipv4/route/redirect_load</Term>
6231 Factors which determine if more ICMP redirects should be sent to a specific
6232 host. No redirects will be sent once the load limit or the maximum number of
6233 redirects has been reached.
6237 <Term>/proc/sys/net/ipv4/route/redirect_number</Term>
6240 See /proc/sys/net/ipv4/route/redirect_load.
6244 <Term>/proc/sys/net/ipv4/route/redirect_silence</Term>
6247 Timeout for redirects. After this period redirects will be sent again, even if
6248 this has been stopped, because the load or number limit has been reached.
6260 <chapter id="lartc.adv-qdisc">
6261 <Title>Advanced & less common queueing disciplines</Title>
6264 Should you find that you have needs not addressed by the queues mentioned
6265 earlier, the kernel contains some other more specialized queues mentioned here.
6268 <Sect1 id="lartc.adv-qdisc.bfifo-pfifo">
6269 <Title><literal>bfifo</literal>/<literal>pfifo</literal></Title>
6272 These classless queues are even simpler than pfifo_fast in that they lack
6273 the internal bands - all traffic is really equal. They have one important
6274 benefit though, they have some statistics. So even if you don't need shaping
6275 or prioritizing, you can use this qdisc to determine the backlog on your
6280 pfifo has a length measured in packets, bfifo in bytes.
6284 <Title>Parameters & usage</Title>
6293 Specifies the length of the queue. Measured in bytes for bfifo, in packets
6294 for pfifo. Defaults to the interface txqueuelen (see pfifo_fast chapter)
6295 packets long or txqueuelen*mtu bytes for bfifo.
6305 <Sect1 id="lartc.adv-qdisc.csz">
6306 <Title>Clark-Shenker-Zhang algorithm (CSZ)</Title>
6309 This is so theoretical that not even Alexey (the main CBQ author) claims to
6310 understand it. From his source:
6315 David D. Clark, Scott Shenker and Lixia Zhang
6316 <citetitle>Supporting Real-Time Applications in an Integrated Services Packet
6317 Network: Architecture and Mechanism</citetitle>.
6321 As I understand it, the main idea is to create WFQ flows for each guaranteed
6322 service and to allocate the rest of bandwith to dummy flow-0. Flow-0
6323 comprises the predictive services and the best effort traffic; it is handled
6324 by a priority scheduler with the highest priority band allocated for
6325 predictive services, and the rest --- to the best effort packets.
6329 Note that in CSZ flows are NOT limited to their bandwidth. It is supposed
6330 that the flow passed admission control at the edge of the QoS network and it
6331 doesn't need further shaping. Any attempt to improve the flow or to shape it
6332 to a token bucket at intermediate hops will introduce undesired delays and
6337 At the moment CSZ is the only scheduler that provides true guaranteed
6338 service. Another schemes (including CBQ) do not provide guaranteed delay and
6343 Does not currently seem like a good candidate to use, unless you've read and
6344 understand the article mentioned.
6350 <Sect1 id="lartc.adv-qdisc.dsmark"
6352 <Title>DSMARK</Title>
6356 <author><firstname>Esteve</firstname><surname>Camps</surname></author>
6357 <address><email>marvin@grn.es</email></address>
6358 This text is an extract from my thesis on
6359 <citetitle>QoS Support in Linux</citetitle>, September 2000.
6363 <Para>Source documents:
6369 <ULink URL="ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz">
6370 Draft-almesberger-wajhak-diffserv-linux-01.txt</ULink>.
6374 <Para>Examples in iproute2 distribution.
6379 <ULink URL="http://www.qosforum.com/white-papers/qosprot_v3.pdf">
6380 White Paper-QoS protocols and architectures</ULink> and
6381 <ULink URL="http://www.qosforum.com/docs/faq">
6382 IP QoS Frequently Asked Questions</ULink> both by
6383 <citetitle>Quality of Service Forum</citetitle>.
6389 This chapter was written by Esteve Camps <esteve@hades.udg.es>.
6393 <Title>Introduction</Title>
6396 First of all, first of all, it would be a great idea for you to read RFCs
6397 written about this (RFC2474, RFC2475, RFC2597 and RFC2598) at
6398 <ULink URL="http://www.ietf.org/html.charters/diffserv-charter.html">
6399 IETF DiffServ working Group web site</ULink> and
6400 <ULink URL="http://diffserv.sf.net/">
6401 Werner Almesberger web site</ULink>
6402 (he wrote the code to support Differentiated Services on Linux).
6408 <Title>What is Dsmark related to?</Title>
6411 Dsmark is a queueing discipline that offers the capabilities needed in
6412 Differentiated Services (also called DiffServ or, simply, DS). DiffServ is
6413 one of two actual QoS architectures (the other one is called Integrated
6414 Services) that is based on a value carried by packets in the DS field of the
6419 One of the first solutions in IP designed to offer some QoS level was
6420 the Type of Service field (TOS byte) in IP header. By changing that value,
6421 we could choose a high/low level of throughput, delay or reliability.
6422 But this didn't provide sufficient flexibility to the needs of new
6423 services (such as real-time applications, interactive applications and
6424 others). After this, new architectures appeared. One of these was DiffServ
6425 which kept TOS bits and renamed DS field.
6431 <Title>Differentiated Services guidelines</Title>
6434 Differentiated Services is group-oriented. I mean, we don't know anything
6435 about flows (this will be the Integrated Services purpose); we know about
6436 flow aggregations and we will apply different behaviours depending on which
6437 aggregation a packet belongs to.
6441 When a packet arrives to an edge node (entry node to a DiffServ domain)
6442 entering to a DiffServ Domain we'll have to policy, shape and/or mark those
6443 packets (marking refers to assigning a value to the DS field. It's just like the
6444 cows :-) ). This will be the mark/value that the internal/core nodes on our
6445 DiffServ Domain will look at to determine which behaviour or QoS level
6450 As you can deduce, Differentiated Services involves a domain on which
6451 all DS rules will have to be applied. In fact you can think I
6452 will classify all the packets entering my domain. Once they enter my
6453 domain they will be subjected to the rules that my classification dictates
6454 and every traversed node will apply that QoS level.
6458 In fact, you can apply your own policies into your local domains, but some
6459 <Emphasis>Service Level Agreements</Emphasis> should be considered when connecting to
6464 At this point, you maybe have a lot of questions. DiffServ is more than I've
6465 explained. In fact, you can understand that I can not resume more than 3
6466 RFCs in just 50 lines :-).
6472 <Title>Working with Dsmark</Title>
6475 As the DiffServ bibliography specifies, we differentiate boundary nodes and
6476 interior nodes. These are two important points in the traffic path. Both
6477 types perform a classification when the packets arrive. Its result may be
6478 used in different places along the DS process before the packet is released
6479 to the network. It's just because of this that the diffserv code supplies an
6480 structure called sk_buff, including a new field called skb->tc_index
6481 where we'll store the result of initial classification that may be used in
6482 several points in DS treatment.
6486 The skb->tc_index value will be initially set by the DSMARK qdisc,
6487 retrieving it from the DS field in IP header of every received packet.
6488 Besides, cls_tcindex classifier will read all or part of skb->tcindex
6489 value and use it to select classes.
6493 But, first of all, take a look at DSMARK qdisc command and its parameters:
6496 ... dsmark indices INDICES [ default_index DEFAULT_INDEX ] [ set_tc_index ]
6499 What do these parameters mean?
6505 <Emphasis remap="bf">indices</Emphasis>: size of table of (mask,value) pairs. Maximum value is 2ˆn, where n>=0.
6511 <Emphasis remap="bf">Default_index</Emphasis>: the default table entry index if classifier finds no match.
6517 <Emphasis remap="bf">Set_tc_index</Emphasis>: instructs dsmark discipline to retrieve the DS field and store it onto skb->tc_index.
6523 Let's see the DSMARK process.
6529 <Title>How SCH_DSMARK works.</Title>
6532 This qdisc will apply the next steps:
6538 If we have declared set_tc_index option in qdisc command, DS field is retrieved and stored onto
6539 skb->tc_index variable.
6545 Classifier is invoked. The classifier will be executed and it will return a class ID that will be stored in
6546 skb->tc_index variable.If no filter matches are found, we consider the default_index option to be the
6547 classId to store. If neither set_tc_index nor default_index has been declared results may be
6554 After been sent to internal qdiscs where you can reuse the result of the filter, the classid returned by
6555 the internal qdisc is stored into skb->tc_index. We will use this value in the future to index a mask-
6556 value table. The final result to assign to the packet will be that resulting from next operation:
6559 New_Ds_field = ( Old_DS_field & mask ) | value
6568 Thus, new value will result from "anding" ds_field and mask values and next, this result "ORed" with
6569 value parameter. See next diagram to understand all this process:
6578 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
6580 | -- If you declare set_tc_index, we set DS | | <-----May change
6581 | value into skb->tc_index variable | |O DS field
6583 +-|-+ +------+ +---+-+ Internal +-+ +---N|-----|----+
6584 | | | | tc |--->| | |--> . . . -->| | | D| | |
6585 | | |----->|index |--->| | | Qdisc | |---->| v | |
6586 | | | |filter|--->| | | +---------------+ | ---->(mask,value) |
6587 -->| O | +------+ +-|-+--------------^----+ / | (. , .) |
6588 | | | ^ | | | | (. , .) |
6589 | | +----------|---------|----------------|-------|--+ (. , .) |
6590 | | sch_dsmark | | | | |
6591 +-|------------|---------|----------------|-------|------------------+
6592 | | | <- tc_index -> | |
6593 | |(read) | may change | | <--------------Index to the
6594 | | | | | (mask,value)
6595 v | v v | pairs table
6596 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ->
6603 How to do marking? Just change the mask and value of the class you want to remark. See next line of code:
6606 tc class change dev eth0 classid 1:1 dsmark mask 0x3 value 0xb8
6609 This changes the (mask,value) pair in hash table, to remark packets belonging to class 1:1.You have to "change" this values
6610 because of default values that (mask,value) gets initially (see table below).
6614 Now, we'll explain how TC_INDEX filter works and how fits into this. Besides, TCINDEX filter can be
6615 used in other configurations rather than those including DS services.
6621 <Title>TC_INDEX Filter</Title>
6624 This is the basic command to declare a TC_INDEX filter:
6627 ... tcindex [ hash SIZE ] [ mask MASK ] [ shift SHIFT ]
6628 [ pass_on | fall_through ]
6629 [ classid CLASSID ] [ police POLICE_SPEC ]
6632 Next, we show the example used to explain TC_INDEX operation mode. Pay attention to bolded words:
6635 tc qdisc add dev eth0 handle 1:0 root dsmark indices 64 <Emphasis remap="bf">set_tc_index</Emphasis>
6637 tc filter add dev eth0 parent 1:0 protocol ip prio 1 tcindex <Emphasis remap="bf">mask 0xfc shift 2</Emphasis>
6639 tc qdisc add dev eth0 parent 1:0 handle 2:0 cbq bandwidth 10Mbit cell 8 avpkt 1000 mpu 64
6641 # EF traffic class
6643 tc class add dev eth0 parent 2:0 classid 2:1 cbq bandwidth 10Mbit rate 1500Kbit avpkt 1000 prio 1 bounded isolated allot 1514 weight 1 maxburst 10
6645 # Packet fifo qdisc for EF traffic
6647 tc qdisc add dev eth0 parent 2:1 pfifo limit 5
6649 tc filter add dev eth0 parent 2:0 protocol ip prio 1 <Emphasis remap="bf">handle 0x2e</Emphasis> tcindex <Emphasis remap="bf">classid 2:1 pass_on</Emphasis>
6653 (This code is not complete. It's just an extract from EFCBQ example included in iproute2 distribution).
6657 First of all, suppose we receive a packet marked as EF . If you read RFC2598, you'll see that DSCP
6658 recommended value for EF traffic is 101110. This means that DS field will be 10111000 (remember that
6659 less significant bits in TOS byte are not used in DS) or 0xb8 in hexadecimal codification.
6667 +---+ +-------+ +---+-+ +------+ +-+ +-------+
6668 | | | | | | | |FILTER| +-+ +-+ | | | |
6669 | |----->| MASK | -> | | | -> |HANDLE|->| | | | -> | | -> | |
6670 | | . | =0xfc | | | | |0x2E | | +----+ | | | | |
6671 | | . | | | | | +------+ +--------+ | | | |
6672 | | . | | | | | | | | |
6673 -->| | . | SHIFT | | | | | | | |-->
6674 | | . | =2 | | | +----------------------------+ | | |
6675 | | | | | | CBQ 2:0 | | |
6676 | | +-------+ +---+--------------------------------+ | |
6678 | +-------------------------------------------------------------+ |
6680 +-------------------------------------------------------------------------+
6687 The packet arrives, then, set with 0xb8 value at DS field. As we explained before, dsmark qdisc identified
6688 by 1:0 id in the example, retrieves DS field and store it in skb->tc_index variable.
6689 Next step in the example will correspond to the filter associated to this qdisc (second line in the example).
6690 This will perform next operations:
6693 Value1 = skb->tc_index & MASK
6694 Key = Value1 >> SHIFT
6700 In the example, MASK=0xFC i SHIFT=2.
6703 Value1 = 10111000 & 11111100 = 10111000
6704 Key = 10111000 >> 2 = 00101110 -> 0x2E in hexadecimal
6710 The returned value will correspond to a qdisc internal filter handle (in the example, identifier 2:0). If a
6711 filter with this id exists, policing and metering conditions will be verified (in case that filter includes this)
6712 and the classid will be returned (in our example, classid 2:1) and stored in skb->tc_index variable.
6716 But if any filter with that identifier is found, the result will depend on fall_through flag declaration. If so,
6717 value key is returned as classid. If not, an error is returned and process continues with the rest filters. Be
6718 careful if you use fall_through flag; this can be done if a simple relation exists between values
6720 of skb->tc_index variable and class id's.
6724 The latest parameters to comment on are hash and pass_on. The first one
6725 relates to hash table size. Pass_on will be used to indicate that if no classid
6726 equal to the result of this filter is found, try next filter.
6727 The default action is fall_through (look at next table).
6731 Finally, let's see which possible values can be set to all this TCINDEX parameters:
6734 TC Name Value Default
6735 -----------------------------------------------------------------
6736 Hash 1...0x10000 Implementation dependent
6737 Mask 0...0xffff 0xffff
6739 Fall through / Pass_on Flag Fall_through
6740 Classid Major:minor None
6747 This kind of filter is very powerful. It's necessary to explore all possibilities. Besides, this filter is not only used in DiffServ configurations.
6748 You can use it as any other kind of filter.
6752 I recommend you to look at all DiffServ examples included in iproute2 distribution. I promise I will try to
6753 complement this text as soon as I can. Besides, all I have explained is the result of a lot of tests.
6754 I would thank you tell me if I'm wrong in any point.
6761 <Sect1 id="lartc.adv-qdisc.ingress">
6762 <Title>Ingress qdisc</Title>
6765 All qdiscs discussed so far are egress qdiscs. Each interface however can
6766 also have an ingress qdisc which is not used to send packets
6767 out to the network adaptor. Instead, it allows you to apply tc filters to
6768 packets coming in over the interface, regardless of whether they have a local
6769 destination or are to be forwarded.
6773 As the tc filters contain a full Token Bucket Filter implementation, and are
6774 also able to match on the kernel flow estimator, there is a lot of
6775 functionality available. This effectively allows you to police incoming
6776 traffic, before it even enters the IP stack.
6780 <Title>Parameters & usage</Title>
6783 The ingress qdisc itself does not require any parameters. It differs from
6784 other qdiscs in that it does not occupy the root of a device. Attach it like
6788 # tc qdisc add dev eth0 ingress
6791 This allows you to have other, sending, qdiscs on your device besides the
6796 For a contrived example how the ingress qdisc could be used, see the
6804 <Sect1 id="lartc.adv-qdisc.red">
6805 <Title>Random Early Detection (RED)</Title>
6808 This section is meant as an introduction to backbone routing, which often
6809 involves <100 megabit bandwidths, which requires a different approach than
6810 your ADSL modem at home.
6814 The normal behaviour of router queues on the Internet is called tail-drop.
6815 Tail-drop works by queueing up to a certain amount, then dropping all traffic
6816 that 'spills over'. This is very unfair, and also leads to retransmit
6817 synchronization. When retransmit synchronization occurs, the sudden burst
6818 of drops from a router that has reached its fill will cause a delayed burst
6819 of retransmits, which will over fill the congested router again.
6823 In order to cope with transient congestion on links, backbone routers will
6824 often implement large queues. Unfortunately, while these queues are good for
6825 throughput, they can substantially increase latency and cause TCP
6826 connections to behave very burstily during congestion.
6830 These issues with tail-drop are becoming increasingly troublesome on the
6831 Internet because the use of network unfriendly applications is increasing.
6832 The Linux kernel offers us RED, short for Random Early Detect, also called
6833 Random Early Drop, as that is how it works.
6837 RED isn't a cure-all for this, applications which inappropriately fail to
6838 implement exponential backoff still get an unfair share of the bandwidth,
6839 however, with RED they do not cause as much harm to the throughput and
6840 latency of other connections.
6844 RED statistically drops packets from flows before it reaches its hard
6845 limit. This causes a congested backbone link to slow more gracefully, and
6846 prevents retransmit synchronization. This also helps TCP find its 'fair'
6847 speed faster by allowing some packets to get dropped sooner keeping queue
6848 sizes low and latency under control. The probability of a packet being
6849 dropped from a particular connection is proportional to its bandwidth usage
6850 rather than the number of packets it transmits.
6854 RED is a good queue for backbones, where you can't afford the
6855 complexity of per-session state tracking needed by fairness queueing.
6859 In order to use RED, you must decide on three parameters: Min, Max, and
6860 burst. Min sets the minimum queue size in bytes before dropping will begin,
6861 Max is a soft maximum that the algorithm will attempt to stay under, and
6862 burst sets the maximum number of packets that can 'burst through'.
6866 You should set the min by calculating that highest acceptable base queueing
6867 latency you wish, and multiply it by your bandwidth. For instance, on my
6868 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set
6869 min to 1600 bytes. Setting min too small will degrade throughput and too
6870 large will degrade latency. Setting a small min is not a replacement for
6871 reducing the MTU on a slow link to improve interactive response.
6875 You should make max at least twice min to prevent synchronization. On slow
6876 links with small Min's it might be wise to make max perhaps four or
6877 more times large then min.
6881 Burst controls how the RED algorithm responds to bursts. Burst must be set
6882 larger then min/avpkt. Experimentally, I've found (min+min+max)/(3*avpkt) to
6887 Additionally, you need to set limit and avpkt. Limit is a safety value, after
6888 there are limit bytes in the queue, RED 'turns into' tail-drop. I typical set
6889 limit to eight times max. Avpkt should be your average packet size. 1000
6890 works OK on high speed Internet links with a 1500byte MTU.
6895 URL="http://www.aciri.org/floyd/papers/red/red.html"
6896 >the paper on RED queueing</ULink
6897 > by Sally Floyd and Van Jacobson for technical
6903 <Sect1 id="lartc.adv-qdisc.gred">
6904 <Title>Generic Random Early Detection</Title>
6907 Not a lot is known about GRED. It looks like GRED with several internal
6908 queues, whereby the internal queue is chosen based on the Diffserv tcindex
6909 field. According to a slide found
6910 <ULink URL="http://www.davin.ottawa.on.ca/ols/img22.htm">here</ULink>,
6911 it contains the capabilities of Cisco's 'Distributed Weighted RED', as well
6912 as Dave Clark's RIO.
6916 Each virtual queue can have its own Drop Parameters specified.
6920 FIXME: get Jamal or Werner to tell us more
6925 <Sect1 id="lartc.adv-qdisc.vc-atm">
6926 <Title>VC/ATM emulation</Title>
6929 This is quite a major effort by Werner Almesberger to allow you to build
6930 Virtual Circuits over TCP/IP sockets. A Virtual Circuit is a concept from
6935 For more information, see the <ULink
6936 URL="http://linux-atm.sourceforge.net/"
6937 >ATM on Linux homepage</ULink
6943 <Sect1 id="lartc.adv-qdisc.wrr">
6944 <Title>Weighted Round Robin (WRR)</Title>
6947 This qdisc is not included in the standard kernels but can be downloaded from
6948 <ULink URL="http://wipl-wrr.dkik.dk/wrr/">here</ULink>.
6949 Currently the qdisc is only tested with Linux 2.2 kernels but it will
6950 probably work with 2.4/2.5 kernels too.
6954 The WRR qdisc distributes bandwidth between its classes using the weighted
6955 round robin scheme. That is, like the CBQ qdisc it contains classes
6956 into which arbitrary qdiscs can be plugged. All classes which have sufficient
6957 demand will get bandwidth proportional to the weights associated with the classes.
6958 The weights can be set manually using the <Literal remap="tt">tc</Literal> program. But they
6959 can also be made automatically decreasing for classes transferring much data.
6963 The qdisc has a built-in classifier which assigns packets coming from or
6964 sent to different machines to different classes. Either the MAC or IP and
6965 either source or destination addresses can be used. The MAC address can only
6966 be used when the Linux box is acting as an ethernet bridge, however. The
6967 classes are automatically assigned to machines based on the packets seen.
6971 The qdisc can be very useful at sites such as dorms where a lot of unrelated
6972 individuals share an Internet connection. A set of scripts setting up a
6973 relevant behavior for such a site is a central part of the WRR distribution.
6980 <chapter id="lartc.cookbook"
6981 xreflabel="Cookbook">
6982 <Title>Cookbook</Title>
6985 This section contains 'cookbook' entries which may help you solve problems.
6986 A cookbook is no replacement for understanding however, so try and comprehend
6990 <Sect1 id="lartc.cookbook.sla">
6991 <Title>Running multiple sites with different SLAs</Title>
6994 You can do this in several ways. Apache has some support for this with a
6995 module, but we'll show how Linux can do this for you, and do so for other
6996 services as well. These commands are stolen from a presentation by Jamal
6997 Hadi that's referenced below.
7001 Let's say we have two customers, with http, ftp and streaming audio, and we
7002 want to sell them a limited amount of bandwidth. We do so on the server itself.
7006 Customer A should have at most 2 megabits, customer B has paid for 5
7007 megabits. We separate our customers by creating virtual IP addresses on our
7014 # ip address add 188.177.166.1 dev eth0
7015 # ip address add 188.177.166.2 dev eth0
7021 It is up to you to attach the different servers to the right IP address. All
7022 popular daemons have support for this.
7026 We first attach a CBQ qdisc to eth0:
7029 # tc qdisc add dev eth0 root handle 1: cbq bandwidth 10Mbit cell 8 avpkt 1000 \
7036 We then create classes for our customers:
7042 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 10Mbit rate \
7043 2MBit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
7044 # tc class add dev eth0 parent 1:0 classid 1:2 cbq bandwidth 10Mbit rate \
7045 5Mbit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
7051 Then we add filters for our two classes:
7054 ##FIXME: Why this line, what does it do?, what is a divisor?:
7055 ##FIXME: A divisor has something to do with a hash table, and the number of
7057 # tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 1: u32 divisor 1
7058 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.1
7060 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.2
7071 FIXME: why no token bucket filter? is there a default pfifo_fast fallback
7077 <Sect1 id="lartc.cookbook.synflood-protect"
7078 xreflabel="Protecting your host from SYN floods">
7079 <Title>Protecting your host from SYN floods</Title>
7082 From Alexey's iproute documentation, adapted to netfilter and with more
7083 plausible paths. If you use this, take care to adjust the numbers to
7084 reasonable values for your system.
7088 If you want to protect an entire network, skip this script, which is best
7089 suited for a single host.
7093 It appears that you need the very latest version of the iproute2 tools to
7094 get this to work with 2.4.0.
7102 # sample script on using the ingress capabilities
7103 # this script shows how one can rate limit incoming SYNs
7104 # Useful for TCP-SYN attack protection. You can use
7105 # IPchains to have more powerful additions to the SYN (eg
7106 # in addition the subnet)
7108 #path to various utilities;
7109 #change to reflect yours.
7113 IPTABLES=/sbin/iptables
7116 # tag all incoming SYN packets through $INDEV as mark value 1
7117 ############################################################
7118 $iptables -A PREROUTING -i $INDEV -t mangle -p tcp --syn \
7119 -j MARK --set-mark 1
7120 ############################################################
7122 # install the ingress qdisc on the ingress interface
7123 ############################################################
7124 $TC qdisc add dev $INDEV handle ffff: ingress
7125 ############################################################
7129 # SYN packets are 40 bytes (320 bits) so three SYNs equals
7130 # 960 bits (approximately 1kbit); so we rate limit below
7131 # the incoming SYNs to 3/sec (not very useful really; but
7132 #serves to show the point - JHS
7133 ############################################################
7134 $TC filter add dev $INDEV parent ffff: protocol ip prio 50 handle 1 fw \
7135 police rate 1kbit burst 40 mtu 9k drop flowid :1
7136 ############################################################
7140 echo "---- qdisc parameters Ingress ----------"
7141 $TC qdisc ls dev $INDEV
7142 echo "---- Class parameters Ingress ----------"
7143 $TC class ls dev $INDEV
7144 echo "---- filter parameters Ingress ----------"
7145 $TC filter ls dev $INDEV parent ffff:
7147 #deleting the ingress qdisc
7148 #$TC qdisc del $INDEV ingress
7155 <Sect1 id="lartc.cookbook.icmp-ratelimit">
7156 <Title>Rate limit ICMP to prevent dDoS</Title>
7159 Recently, distributed denial of service attacks have become a major nuisance
7160 on the Internet. By properly filtering and rate limiting your network, you can
7161 both prevent becoming a casualty or the cause of these attacks.
7165 You should filter your networks so that you do not allow non-local IP source
7166 addressed packets to leave your network. This stops people from anonymously
7167 sending junk to the Internet.
7171 Rate limiting goes much as shown earlier. To refresh your memory, our
7178 [The Internet] ---<E3, T3, whatever>--- [Linux router] --- [Office+ISP]
7185 We first set up the prerequisite parts:
7191 # tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000
7192 # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate \
7193 10Mbit allot 1514 prio 5 maxburst 20 avpkt 1000
7199 If you have 100Mbit, or more, interfaces, adjust these numbers. Now you need
7200 to determine how much ICMP traffic you want to allow. You can perform
7201 measurements with tcpdump, by having it write to a file for a while, and
7202 seeing how much ICMP passes your network. Do not forget to raise the
7207 If measurement is impractical, you might want to choose 5% of your available
7208 bandwidth. Let's set up our class:
7211 # tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate \
7212 100Kbit allot 1514 weight 800Kbit prio 5 maxburst 20 avpkt 250 \
7219 This limits at 100Kbit. Now we need a filter to assign ICMP traffic to this
7223 # tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip
7224 protocol 1 0xFF flowid 10:100
7232 <Sect1 id="lartc.cookbook.interactive-prio">
7233 <Title>Prioritizing interactive traffic</Title>
7236 If lots of data is coming down your link, or going up for that matter, and
7237 you are trying to do some maintenance via telnet or ssh, this may not go too
7238 well. Other packets are blocking your keystrokes. Wouldn't it be great if
7239 there were a way for your interactive packets to sneak past the bulk
7240 traffic? Linux can do this for you!
7244 As before, we need to handle traffic going both ways. Evidently, this works
7245 best if there are Linux boxes on both ends of your link, although other
7246 UNIX's are able to do this. Consult your local Solaris/BSD guru for this.
7250 The standard pfifo_fast scheduler has 3 different 'bands'. Traffic in band 0
7251 is transmitted first, after which traffic in band 1 and 2 gets considered.
7252 It is vital that our interactive traffic be in band 0!
7256 We blatantly adapt from the (soon to be obsolete) ipchains HOWTO:
7260 There are four seldom-used bits in the IP header, called the Type of Service
7261 (TOS) bits. They effect the way packets are treated; the four bits are
7262 "Minimum Delay", "Maximum Throughput", "Maximum Reliability" and "Minimum
7263 Cost". Only one of these bits is allowed to be set. Rob van Nieuwkerk, the
7264 author of the ipchains TOS-mangling code, puts it as follows:
7270 Especially the "Minimum Delay" is important for me. I switch it on for
7271 "interactive" packets in my upstream (Linux) router. I'm
7272 behind a 33k6 modem link. Linux prioritizes packets in 3 queues. This
7273 way I get acceptable interactive performance while doing bulk
7274 downloads at the same time.
7280 The most common use is to set telnet & ftp control connections to "Minimum
7281 Delay" and FTP data to "Maximum Throughput". This would be
7282 done as follows, on your upstream router:
7288 # iptables -A PREROUTING -t mangle -p tcp --sport telnet \
7289 -j TOS --set-tos Minimize-Delay
7290 # iptables -A PREROUTING -t mangle -p tcp --sport ftp \
7291 -j TOS --set-tos Minimize-Delay
7292 # iptables -A PREROUTING -t mangle -p tcp --sport ftp-data \
7293 -j TOS --set-tos Maximize-Throughput
7299 Now, this only works for data going from your telnet foreign host to your
7300 local computer. The other way around appears to be done for you, ie, telnet,
7301 ssh & friends all set the TOS field on outgoing packets automatically.
7305 Should you have an application that does not do this, you can always do it
7306 with netfilter. On your local box:
7312 # iptables -A OUTPUT -t mangle -p tcp --dport telnet \
7313 -j TOS --set-tos Minimize-Delay
7314 # iptables -A OUTPUT -t mangle -p tcp --dport ftp \
7315 -j TOS --set-tos Minimize-Delay
7316 # iptables -A OUTPUT -t mangle -p tcp --dport ftp-data \
7317 -j TOS --set-tos Maximize-Throughput
7324 <Sect1 id="lartc.cookbook.squid">
7325 <Title>Transparent web-caching using <application>netfilter</application>,
7326 <application>iproute2</application>, <application>ipchains</application> and
7327 <application>squid</application></Title>
7330 This section was sent in by reader Ram Narula from Internet for Education
7335 The regular technique in accomplishing this in Linux
7336 is probably with use of ipchains AFTER making sure
7337 that the "outgoing" port 80(web) traffic gets routed through
7338 the server running squid.
7342 There are 3 common methods to make sure "outgoing"
7343 port 80 traffic gets routed to the server running squid
7344 and 4th one is being introduced here.
7350 <Term>Making the gateway router do it.</Term>
7353 If you can tell your gateway router to
7354 match packets that has outgoing destination port
7355 of 80 to be sent to the IP address of squid server.
7363 This would put additional load on the router and
7364 some commercial routers might not even support this.
7369 <Term>Using a Layer 4 switch.</Term>
7372 Layer 4 switches can handle this without any problem.
7380 The cost for this equipment is usually very high. Typical
7381 layer 4 switch would normally cost more than
7382 a typical router+good linux server.
7387 <Term>Using cache server as network's gateway.</Term>
7390 You can force ALL traffic through cache server.
7396 This is quite risky because Squid does utilize lots of CPU power which might
7397 result in slower over-all network performance or the server itself might crash and no one on the
7398 network will be able to access the Internet if that occurs.
7403 <Term>Linux+NetFilter router.</Term>
7406 By using NetFilter another technique can be implemented
7407 which is using NetFilter for "mark"ing the packets
7408 with destination port 80 and using iproute2 to
7409 route the "mark"ed packets to the Squid server.
7420 10.0.0.1 naret (NetFilter server)
7421 10.0.0.2 silom (Squid server)
7422 10.0.0.3 donmuang (Router connected to the Internet)
7423 10.0.0.4 kaosarn (other server on network)
7425 10.0.0.0/24 main network
7426 10.0.0.0/19 total network
7436 ------------hub/switch----------
7438 naret silom kaosarn RAS etc.
7441 First, make all traffic pass through naret by making sure it is the default gateway except for silom.
7442 Silom's default gateway has to be donmuang (10.0.0.3) or this would create web traffic loop.
7445 (all servers on my network had 10.0.0.1 as the default gateway which was the former IP address of donmuang router so what I did
7446 was changed the IP address of donmuang to 10.0.0.3 and gave naret ip address of 10.0.0.1)
7452 -setup squid and ipchains
7456 Setup Squid server on silom, make sure it does support transparent caching/proxying, the default port is usually
7457 3128, so all traffic for port 80 has to be redirected to port 3128 locally. This can be done by using ipchains with the following:
7461 silom# ipchains -N allow1
7462 silom# ipchains -A allow1 -p TCP -s 10.0.0.0/19 -d 0/0 80 -j REDIRECT 3128
7463 silom# ipchains -I input -j allow1
7467 Or, in netfilter lingo:
7469 silom# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
7473 (note: you might have other entries as well)
7476 For more information on setting Squid server please refer to Squid FAQ page on <ULink
7477 URL="http://squid.nlanr.net">http://squid.nlanr.net</ULink>).
7480 Make sure ip forwarding is enabled on this server and the default gateway for this server is donmuang router (NOT naret).
7486 -setup iptables and iproute2
7487 -disable icmp REDIRECT messages (if needed)
7494 "Mark" packets of destination port 80 with value 2
7496 naret# iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 \
7497 -j MARK --set-mark 2
7503 Setup iproute2 so it will route packets with "mark" 2 to silom
7505 naret# echo 202 www.out >> /etc/iproute2/rt_tables
7506 naret# ip rule add fwmark 2 table www.out
7507 naret# ip route add default via 10.0.0.2 dev eth0 table www.out
7508 naret# ip route flush cache
7512 If donmuang and naret is on the same subnet then naret should not send out icmp REDIRECT messages.
7513 In this case it is, so icmp REDIRECTs has to be disabled by:
7515 naret# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
7516 naret# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
7517 naret# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
7524 The setup is complete, check the configuration
7530 naret# iptables -t mangle -L
7531 Chain PREROUTING (policy ACCEPT)
7532 target prot opt source destination
7533 MARK tcp -- anywhere anywhere tcp dpt:www MARK set 0x2
7535 Chain OUTPUT (policy ACCEPT)
7536 target prot opt source destination
7539 0: from all lookup local
7540 32765: from all fwmark 2 lookup www.out
7541 32766: from all lookup main
7542 32767: from all lookup default
7544 naret# ip route list table www.out
7545 default via 203.114.224.8 dev eth0
7548 10.0.0.1 dev eth0 scope link
7549 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.1
7550 127.0.0.0/8 dev lo scope link
7551 default via 10.0.0.3 dev eth0
7553 (make sure silom belongs to one of the above lines, in this case
7554 it's the line with 10.0.0.0/24)
7562 <Title>Traffic flow diagram after implementation</Title>
7566 |-----------------------------------------|
7567 |Traffic flow diagram after implementation|
7568 |-----------------------------------------|
7574 -----------------donmuang router---------------------
7579 *destination port 80 traffic=========>(cache) ||
7582 \\===================================kaosarn, RAS, etc.
7587 Note that the network is asymmetric as there is one extra hop on
7588 general outgoing path.
7594 Here is run down for packet traversing the network from kaosarn
7595 to and from the Internet.
7597 For web/http traffic:
7598 kaosarn http request->naret->silom->donmuang->internet
7599 http replies from Internet->donmuang->silom->kaosarn
7601 For non-web/http requests(eg. telnet):
7602 kaosarn outgoing data->naret->donmuang->internet
7603 incoming data from Internet->donmuang->kaosarn
7612 <Sect1 id="lartc.cookbook.mtu-discovery">
7613 <Title>Circumventing Path MTU Discovery issues with per route MTU settings</Title>
7616 For sending bulk data, the Internet generally works better when using larger
7617 packets. Each packet implies a routing decision, when sending a 1 megabyte
7618 file, this can either mean around 700 packets when using packets that are as
7619 large as possible, or 4000 if using the smallest default.
7623 However, not all parts of the Internet support full 1460 bytes of payload
7624 per packet. It is therefore necessary to try and find the largest packet
7625 that will 'fit', in order to optimize a connection.
7629 This process is called 'Path MTU Discovery', where MTU stands for 'Maximum
7634 When a router encounters a packet that's too big too send in one piece, AND
7635 it has been flagged with the "Don't Fragment" bit, it returns an ICMP
7636 message stating that it was forced to drop a packet because of this. The
7637 sending host acts on this hint by sending smaller packets, and by iterating
7638 it can find the optimum packet size for a connection over a certain path.
7642 This used to work well until the Internet was discovered by hooligans who do
7643 their best to disrupt communications. This in turn lead administrators to
7644 either block or shape ICMP traffic in a misguided attempt to improve
7645 security or robustness of their Internet service.
7649 What has happened now is that Path MTU Discovery is working less and less
7650 well and fails for certain routes, which leads to strange TCP/IP sessions
7651 which die after a while.
7655 Although I have no proof for this, two sites who I used to have this problem
7656 with both run Alteon Acedirectors before the affected systems - perhaps
7657 somebody more knowledgeable can provide clues as to why this happens.
7661 <Title>Solution</Title>
7664 When you encounter sites that suffer from this problem, you can disable Path
7665 MTU discovery by setting it manually. Koos van den Hout, slightly edited,
7671 The following problem: I set the mtu/mru of my leased line running ppp to
7672 296 because it's only 33k6 and I cannot influence the queueing on the
7673 other side. At 296, the response to a key press is within a reasonable
7678 And, on my side I have a masqrouter running (of course) Linux.
7682 Recently I split 'server' and 'router' so most applications are run on a
7683 different machine than the routing happens on.
7687 I then had trouble logging into irc. Big panic! Some digging did find
7688 out that I got connected to irc, even showed up as 'connected' on irc
7689 but I did not receive the motd from irc. I checked what could be wrong
7690 and noted that I already had some previous trouble reaching certain
7691 websites related to the MTU, since I had no trouble reaching them when
7692 the MTU was 1500, the problem just showed when the MTU was set to 296.
7693 Since irc servers block about every kind of traffic not needed for their
7694 immediate operation, they also block icmp.
7698 I managed to convince the operators of a webserver that this was the cause
7699 of a problem, but the irc server operators were not going to fix this.
7703 So, I had to make sure outgoing masqueraded traffic started with the lower
7704 mtu of the outside link. But I want local ethernet traffic to have the
7705 normal mtu (for things like nfs traffic).
7712 ip route add default via 10.0.0.1 mtu 296
7716 (10.0.0.1 being the default gateway, the inside address of the
7717 masquerading router)
7722 In general, it is possible to override PMTU Discovery by setting specific
7723 routes. For example, if only a certain subnet is giving problems, this
7728 ip route add 195.96.96.0/24 via 10.0.0.1 mtu 1000
7735 <Sect1 id="lartc.cookbook.mtu-mss">
7736 <Title>Circumventing Path MTU Discovery issues with MSS Clamping
7737 (for ADSL, cable, PPPoE & PPtP users)</Title>
7740 As explained above, Path MTU Discovery doesn't work as well as it should
7741 anymore. If you know for a fact that a hop somewhere in your network has a
7742 limited (<1500) MTU, you cannot rely on PMTU Discovery finding this out.
7746 Besides MTU, there is yet another way to set the maximum packet size, the so
7747 called Maximum Segment Size. This is a field in the TCP Options part of a
7752 Recent Linux kernels, and a few PPPoE drivers (notably, the excellent
7753 Roaring Penguin one), feature the possibility to 'clamp the MSS'.
7757 The good thing about this is that by setting the MSS value, you are telling
7758 the remote side unequivocally 'do not ever try to send me packets bigger
7759 than this value'. No ICMP traffic is needed to get this to work.
7763 The bad thing is that it's an obvious hack - it breaks 'end to end' by
7764 modifying packets. Having said that, we use this trick in many places and it
7769 In order for this to work you need at least iptables-1.2.1a and Linux 2.4.3
7770 or higher. The basic command line is:
7773 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
7779 This calculates the proper MSS for your link. If you are feeling brave, or
7780 think that you know best, you can also do something like this:
7786 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
7792 This sets the MSS of passing SYN packets to 128. Use this if you have VoIP
7793 with tiny packets, and huge http packets which are causing chopping in your
7799 <Sect1 id="lartc.cookbook.ultimate-tc">
7800 <Title>The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads</Title>
7803 Note: This script has recently been upgraded and previously only worked for
7804 Linux clients in your network! So you might want to update if you have
7805 Windows machines or Macs in your network and noticed that they were not able
7806 to download faster while others were uploading.
7810 I attempted to create the holy grail:
7814 <Term>Maintain low latency for interactive traffic at all times</Term>
7817 This means that downloading or uploading files should not disturb SSH or
7818 even telnet. These are the most important things, even 200ms latency is
7819 sluggish to work over.
7823 <Term>Allow 'surfing' at reasonable speeds while up or downloading</Term>
7826 Even though http is 'bulk' traffic, other traffic should not drown it out
7831 <Term>Make sure uploads don't harm downloads, and the other way around</Term>
7834 This is a much observed phenomenon where upstream traffic simply destroys
7839 It turns out that all this is possible, at the cost of a tiny bit of
7840 bandwidth. The reason that uploads, downloads and ssh hurt each other is the
7841 presence of large queues in many domestic access devices like cable or DSL
7846 The next section explains in depth what causes the delays, and how we can
7847 fix them. You can safely skip it and head straight for the script if you
7848 don't care how the magic is performed.
7852 <Title>Why it doesn't work well by default</Title>
7855 ISPs know that they are benchmarked solely on how fast people can download.
7856 Besides available bandwidth, download speed is influenced heavily by packet
7857 loss, which seriously hampers TCP/IP performance. Large queues can help
7858 prevent packet loss, and speed up downloads. So ISPs configure large queues.
7862 These large queues however damage interactivity. A keystroke must first
7863 travel the upstream queue, which may be seconds (!) long and go to your
7864 remote host. It is then displayed, which leads to a packet coming back, which
7865 must then traverse the downstream queue, located at your ISP, before it
7866 appears on your screen.
7870 This HOWTO teaches you how to mangle and process the queue in many ways, but
7871 sadly, not all queues are accessible to us. The queue over at the ISP is
7872 completely off-limits, whereas the upstream queue probably lives inside your
7873 cable modem or DSL device. You may or may not be able to configure it. Most
7878 So, what next? As we can't control either of those queues, they must be
7879 eliminated, and moved to your Linux router. Luckily this is possible.
7886 <Term>Limit upload speed</Term>
7889 By limiting our upload speed to slightly less than the truly available rate,
7890 no queues are built up in our modem. The queue is now moved to Linux.
7894 <Term>Limit download speed</Term>
7897 This is slightly trickier as we can't really influence how fast the internet
7898 ships us data. We can however drop packets that are coming in too fast,
7899 which causes TCP/IP to slow down to just the rate we want. Because we don't
7900 want to drop traffic unnecessarily, we configure a 'burst' size we allow at
7908 Now, once we have done this, we have eliminated the downstream queue totally
7909 (except for short bursts), and gain the ability to manage the upstream queue
7910 with all the power Linux offers.
7914 What remains to be done is to make sure interactive traffic jumps to the
7915 front of the upstream queue. To make sure that uploads don't hurt downloads,
7916 we also move ACK packets to the front of the queue. This is what normally
7917 causes the huge slowdown observed when generating bulk traffic both ways.
7918 The ACKnowledgements for downstream traffic must compete with upstream
7919 traffic, and get delayed in the process.
7923 If we do all this we get the following measurements using an excellent ADSL
7924 connection from xs4all in the Netherlands:
7931 round-trip min/avg/max = 14.4/17.1/21.7 ms
7933 Without traffic conditioner, while downloading:
7934 round-trip min/avg/max = 560.9/573.6/586.4 ms
7936 Without traffic conditioner, while uploading:
7937 round-trip min/avg/max = 2041.4/2332.1/2427.6 ms
7939 With conditioner, during 220kbit/s upload:
7940 round-trip min/avg/max = 15.7/51.8/79.9 ms
7942 With conditioner, during 850kbit/s download:
7943 round-trip min/avg/max = 20.4/46.9/74.0 ms
7945 When uploading, downloads proceed at ~80% of the available speed. Uploads
7946 at around 90%. Latency then jumps to 850 ms, still figuring out why.
7952 What you can expect from this script depends a lot on your actual uplink
7953 speed. When uploading at full speed, there will always be a single packet
7954 ahead of your keystroke. That is the lower limit to the latency you can
7955 achieve - divide your MTU by your upstream speed to calculate. Typical
7956 values will be somewhat higher than that. Lower your MTU for better effects!
7960 Next, two versions of this script, one with Devik's excellent HTB, the other
7961 with CBQ which is in each Linux kernel, unlike HTB. Both are tested and work
7968 <Title>The actual script (CBQ)</Title>
7971 Works on all kernels. Within the CBQ
7972 qdisc we place two Stochastic Fairness Queues that make sure that multiple
7973 bulk streams don't drown each other out.
7977 Downstream traffic is policed using a tc filter containing a Token Bucket
7982 You might improve on this script by adding 'bounded' to the line that starts
7983 with 'tc class add .. classid 1:20'. If you lowered your MTU, also lower the
7984 allot & avpkt numbers!
7992 # The Ultimate Setup For Your Internet Connection At Home
7995 # Set the following values to somewhat less than your actual download
7996 # and uplink speed. In kilobits
8001 # clean existing down- and uplink qdiscs, hide errors
8002 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
8003 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
8009 tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
8011 # shape everything at $UPLINK speed - this prevents huge queues in your
8012 # DSL modem which destroy latency:
8015 tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit \
8016 allot 1500 prio 5 bounded isolated
8018 # high prio class 1:10:
8020 tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit \
8021 allot 1600 prio 1 avpkt 1000
8023 # bulk and default class 1:20 - gets slightly less traffic,
8024 # and a lower priority:
8026 tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit \
8027 allot 1600 prio 2 avpkt 1000
8029 # both get Stochastic Fairness:
8030 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
8031 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
8034 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
8035 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8036 match ip tos 0x10 0xff flowid 1:10
8038 # ICMP (ip protocol 1) in the interactive class 1:10 so we
8039 # can do measurements & impress our friends:
8040 tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \
8041 match ip protocol 1 0xff flowid 1:10
8043 # To speed up downloads while an upload is going on, put ACK packets in
8044 # the interactive class:
8046 tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \
8047 match ip protocol 6 0xff \
8048 match u8 0x05 0x0f at 0 \
8049 match u16 0x0000 0xffc0 at 2 \
8050 match u8 0x10 0xff at 33 \
8053 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
8055 tc filter add dev $DEV parent 1: protocol ip prio 13 u32 \
8056 match ip dst 0.0.0.0/0 flowid 1:20
8058 ########## downlink #############
8059 # slow downloads down to somewhat less than the real speed to prevent
8060 # queuing at our ISP. Tune to see how high you can set it.
8061 # ISPs tend to have *huge* queues to make sure big downloads are fast
8063 # attach ingress policer:
8065 tc qdisc add dev $DEV handle ffff: ingress
8067 # filter *everything* to it (0.0.0.0/0), drop everything that's
8068 # coming in too fast:
8070 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
8071 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
8074 If you want this script to be run by ppp on connect, copy it to
8079 If the last two lines give an error, update your tc tool to a newer version!
8085 <Title>The actual script (HTB)</Title>
8088 The following script achieves all goals using the wonderful HTB queue, see
8089 the relevant chapter. Well worth patching your kernel for!
8094 # The Ultimate Setup For Your Internet Connection At Home
8097 # Set the following values to somewhat less than your actual download
8098 # and uplink speed. In kilobits
8103 # clean existing down- and uplink qdiscs, hide errors
8104 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
8105 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
8109 # install root HTB, point default traffic to 1:20:
8111 tc qdisc add dev $DEV root handle 1: htb default 20
8113 # shape everything at $UPLINK speed - this prevents huge queues in your
8114 # DSL modem which destroy latency:
8116 tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k
8118 # high prio class 1:10:
8120 tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
8123 # bulk & default class 1:20 - gets slightly less traffic,
8124 # and a lower priority:
8126 tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
8129 # both get Stochastic Fairness:
8130 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
8131 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
8133 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
8134 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8135 match ip tos 0x10 0xff flowid 1:10
8137 # ICMP (ip protocol 1) in the interactive class 1:10 so we
8138 # can do measurements & impress our friends:
8139 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8140 match ip protocol 1 0xff flowid 1:10
8142 # To speed up downloads while an upload is going on, put ACK packets in
8143 # the interactive class:
8145 tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
8146 match ip protocol 6 0xff \
8147 match u8 0x05 0x0f at 0 \
8148 match u16 0x0000 0xffc0 at 2 \
8149 match u8 0x10 0xff at 33 \
8152 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
8155 ########## downlink #############
8156 # slow downloads down to somewhat less than the real speed to prevent
8157 # queuing at our ISP. Tune to see how high you can set it.
8158 # ISPs tend to have *huge* queues to make sure big downloads are fast
8160 # attach ingress policer:
8162 tc qdisc add dev $DEV handle ffff: ingress
8164 # filter *everything* to it (0.0.0.0/0), drop everything that's
8165 # coming in too fast:
8167 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
8168 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
8174 If you want this script to be run by ppp on connect, copy it to
8179 If the last two lines give an error, update your tc tool to a newer version!
8185 <sect1 id="lartc.ratelimit.single"><title>Rate limiting a single host or netmask</title>
8187 Although this is described in stupendous details elsewhere and in our manpages, this question gets asked a lot and
8188 happily there is a simple answer that does not need full comprehension of traffic control.
8191 This three line script does the trick:
8195 tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
8197 tc class add dev $DEV parent 1: classid 1:1 cbq rate 512kbit \
8198 allot 1500 prio 5 bounded isolated
8200 tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
8201 match ip dst 195.96.96.97 flowid 1:1
8205 The first line installs a class based queue on your interface, and tells the kernel that for calculations,
8206 it can be assumed to be a 10mbit interface. If you get this wrong, no real harm is done. But getting it right will
8207 make everything more precise.
8210 The second line creates a 512kbit class with some reasonable defaults. For details, see the cbq manpages and
8211 <xref linkend="lartc.qdisc">.
8214 The last line tells which traffic should go to the shaped class. Traffic not matched by this rule is NOT shaped. To make more
8215 complicated matches (subnets, source ports, destination ports), see <xref linkend="lartc.filtering.simple">.
8218 If you changed anything and want to reload the script, execute 'tc qdisc del dev $DEV root' to clean up your existing
8222 The script can further be improved by adding a last optional line 'tc qdisc add dev $DEV parent 1:1 sfq perturb 10'. See
8223 <xref linkend="lartc.sfq"> for details on what this does.
8228 <chapter id="lartc.bridging">
8229 <Title>Building bridges, and pseudo-bridges with Proxy ARP</Title>
8232 Bridges are devices which can be installed in a network without any
8233 reconfiguration. A network switch is basically a many-port bridge. A bridge
8234 is often a 2-port switch. Linux does however support multiple interfaces in
8235 a bridge, making it a true switch.
8239 Bridges are often deployed when confronted with a broken network that needs
8240 to be fixed without any alterations. Because the bridge is a layer-2 device,
8241 one layer below IP, routers and servers are not aware of its existence.
8242 This means that you can transparently block or modify certain packets, or do
8247 Another good thing is that a bridge can often be replaced by a cross cable
8248 or a hub, should it break down.
8252 The bad news is that a bridge can cause great confusion unless it is very
8253 well documented. It does not appear in traceroutes, but somehow packets
8254 disappear or get changed from point A to point B ('this network is
8255 HAUNTED!'). You should also wonder if an organization that 'does not want to
8256 change anything' is doing the right thing.
8260 The Linux 2.4/2.5 bridge is documented on
8261 <ULink URL=" http://bridge.sourceforge.net/">this page</ULink>.
8264 <Sect1 id="lartc.bridging.iptables">
8265 <Title>State of bridging and iptables</Title>
8268 As of Linux 2.4.20, bridging and iptables do not 'see' each other without
8269 help. If you bridge packets from eth0 to eth1, they do not 'pass' by
8270 iptables. This means that you cannot do filtering, or NAT or mangling or
8271 whatever. In Linux 2.5.45 and higher, this is fixed.
8274 You may also see 'ebtables' mentioned which is yet another project - it
8275 allows you to do wild things as MACNAT and 'brouting'. It is truly scary.
8278 <Sect1 id="lartc.bridging.shaping">
8279 <Title>Bridging and shaping</Title>
8282 This does work as advertised. Be sure to figure out which side each
8283 interface is on, otherwise you might be shaping outbound traffic in your
8284 internal interface, which won't work. Use tcpdump if needed.
8289 <Sect1 id="lartc.bridging.proxy-arp">
8290 <Title>Pseudo-bridges with Proxy-ARP</Title>
8293 If you just want to implement a Pseudo-bridge, skip down a few sections
8294 to 'Implementing it', but it is wise to read a bit about how it works in
8299 A Pseudo-bridge works a bit differently. By default, a bridge passes packets
8300 unaltered from one interface to the other. It only looks at the hardware
8301 address of packets to determine what goes where. This in turn means that you
8302 can bridge traffic that Linux does not understand, as long as it has an
8303 hardware address it does.
8307 A 'Pseudo-bridge' works differently and looks more like a hidden router than
8308 a bridge, but like a bridge, it has little impact on network design.
8312 An advantage of the fact that it is not a bridge lies in the fact that
8313 packets really pass through the kernel, and can be filtered, changed,
8314 redirected or rerouted.
8318 A real bridge can also be made to perform these feats, but it needs special
8319 code, like the Ethernet Frame Diverter, or the above mentioned patch.
8323 Another advantage of a pseudo-bridge is that it does not pass packets it
8324 does not understand - thus cleaning your network of a lot of cruft. In cases
8325 where you need this cruft (like SAP packets, or Netbeui), use a real bridge.
8329 <Title>ARP & Proxy-ARP</Title>
8332 When a host wants to talk to another host on the same physical network
8333 segment, it sends out an Address Resolution Protocol packet, which, somewhat
8334 simplified, reads like this 'who has 10.0.0.1, tell 10.0.0.7'. In response
8335 to this, 10.0.0.1 replies with a short 'here' packet.
8339 10.0.0.7 then sends packets to the hardware address mentioned in the 'here'
8340 packet. It caches this hardware address for a relatively long time, and
8341 after the cache expires, it re-asks the question.
8345 When building a Pseudo-bridge, we instruct the bridge to reply to these ARP
8346 packets, which causes the hosts in the network to send its packets to the
8347 bridge. The bridge then processes these packets, and sends them to the
8352 So, in short, whenever a host on one side of the bridge asks for the
8353 hardware address of a host on the other, the bridge replies with a packet
8354 that says 'hand it to me'.
8358 This way, all data traffic gets transmitted to the right place, and always
8359 passes through the bridge.
8365 <Title>Implementing it</Title>
8368 In the bad old days, it used to be possible to instruct the Linux Kernel to
8369 perform 'proxy-ARP' for just any subnet. So, to configure a pseudo-bridge,
8370 you would have to specify both the proper routes to both sides of the bridge
8371 AND create matching proxy-ARP rules. This is bad in that it requires a lot
8372 of typing, but also because it easily allows you to make mistakes which make
8373 your bridge respond to ARP queries for networks it does not know how to
8378 With Linux 2.4/2.5 (and possibly 2.2), this possibility has been withdrawn and
8379 has been replaced by a flag in the /proc directory, called 'proxy_arp'. The
8380 procedure for building a pseudo-bridge is then:
8389 Assign an IP address to both interfaces, the 'left' and the 'right'
8396 Create routes so your machine knows which hosts reside on the left,
8397 and which on the right
8403 Turn on proxy-ARP on both interfaces, echo 1 >
8404 /proc/sys/net/ipv4/conf/ethL/proxy_arp, echo 1 >
8405 /proc/sys/net/ipv4/conf/ethR/proxy_arp, where L and R stand for the numbers
8406 of your interfaces on the left and on the right side
8415 Also, do not forget to turn on the ip_forwarding flag! When converting from
8416 a true bridge, you may find that this flag was turned off as it is not
8417 needed when bridging.
8421 Another thing you might note when converting is that you need to clear the
8422 arp cache of computers in the network - the arp cache might contain old
8423 pre-bridge hardware addresses which are no longer correct.
8427 On a Cisco, this is done using the command 'clear arp-cache', under
8428 Linux, use 'arp -d ip.address'. You can also wait for the cache to expire
8429 manually, which can take rather long.
8432 You can speed this up using the wonderful 'arping' tool, which on many
8433 distributions is part of the 'iputils' package. Using 'arping' you can send
8434 out unsolicited ARP messages so as to update remote arp caches.
8437 This is a very powerful technique that is also used by 'black hats' to
8438 subvert your routing!
8442 On Linux 2.4, you may need to execute
8443 'echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind' before being able to send
8444 out unsolicited ARP messages!
8448 You may also discover that your network was misconfigured if you are/were of
8449 the habit of specifying routes without netmasks. To explain, some versions
8450 of route may have guessed your netmask right in the past, or guessed wrong
8451 without you noticing. When doing surgical routing like described above, it
8452 is *vital* that you check your netmasks!
8461 <chapter id="lartc.dynamic-routing">
8462 <Title>Dynamic routing - OSPF and BGP</Title>
8465 Once your network starts to get really big, or you start to consider 'the
8466 internet' as your network, you need tools which dynamically route your data.
8467 Sites are often connected to each other with multiple links, and more are
8468 popping up all the time.
8472 The Internet has mostly standardized on OSPF and BGP4 (rfc1771).
8473 Linux supports both, by way of <application>gated</application> and
8474 <application>zebra</application>
8478 While currently not within the scope of this document, we would like to
8479 point you to the definitive works:
8489 URL="http://www.cisco.com/univercd/cc/td/doc/cisintwk/idg4/nd2003.htm"
8490 >Designing large-scale IP Internetworks</ULink
8500 "OSPF. The anatomy of an Internet routing protocol"
8501 Addison Wesley. Reading, MA. 1998.
8505 Halabi has also written a good guide to OSPF routing design, but this
8506 appears to have been dropped from the Cisco web site.
8515 "Internet routing architectures"
8516 Cisco Press (New Riders Publishing). Indianapolis, IN. 1997.
8529 URL="http://www.cisco.com/univercd/cc/td/doc/cisintwk/ics/icsbgp4.htm"
8530 >Using the Border Gateway Protocol for interdomain routing</ULink
8535 Although the examples are Cisco-specific, they are remarkably similar
8536 to the configuration language in Zebra :-)
8541 <chapter id="lartc.other"
8542 xreflabel="Other possibilities">
8543 <Title>Other possibilities</Title>
8546 This chapter is a list of projects having to do with advanced Linux routing
8547 & traffic shaping. Some of these links may deserve chapters of their
8548 own, some are documented very well of themselves, and don't need more HOWTO.
8555 <Term>802.1Q VLAN Implementation for Linux <ULink
8556 URL="http://scry.wanfear.com/~greear/vlan.html"
8561 VLANs are a very cool way to segregate your
8562 networks in a more virtual than physical way. Good information on VLANs can
8564 URL="ftp://ftp.netlab.ohio-state.edu/pub/jain/courses/cis788-97/virtual_lans/index.htm"
8566 >. With this implementation, you can have your Linux box talk
8567 VLANs with machines like Cisco Catalyst, 3Com: {Corebuilder, Netbuilder II,
8568 SuperStack II switch 630}, Extreme Ntwks Summit 48, Foundry: {ServerIronXL,
8573 A great HOWTO about VLANs can be found <ULink
8574 URL="http://scry.wanfear.com/~greear/vlan/cisco_howto.html"
8580 Update: has been included in the kernel as of 2.4.14 (perhaps 13).
8584 <Term>Alternate 802.1Q VLAN Implementation for Linux <ULink
8585 URL="http://vlan.sourceforge.net "
8590 Alternative VLAN implementation for linux. This project was started out of
8591 disagreement with the 'established' VLAN project's architecture and coding
8592 style, resulting in a cleaner overall design.
8596 <Term>Linux Virtual Server <ULink
8597 URL="http://www.LinuxVirtualServer.org/"
8602 These people are brilliant. The Linux Virtual Server is a highly scalable and
8603 highly available server built on a cluster of real servers, with the load
8604 balancer running on the Linux operating system. The architecture of the
8605 cluster is transparent to end users. End users only see a single virtual
8610 In short whatever you need to load balance, at whatever level of traffic, LVS
8611 will have a way of doing it. Some of their techniques are positively evil!
8612 For example, they let several machines have the same IP address on a
8613 segment, but turn off ARP on them. Only the LVS machine does ARP - it then
8614 decides which of the backend hosts should handle an incoming packet, and
8615 sends it directly to the right MAC address of the backend server. Outgoing
8616 traffic will flow directly to the router, and not via the LVS machine, which
8617 does therefor not need to see your 5Gbit/s of content flowing to the world,
8618 and cannot be a bottleneck.
8622 The LVS is implemented as a kernel patch in Linux 2.0 and 2.2, but as a
8623 Netfilter module in 2.4/2.5, so it does not need kernel patches! Their 2.4
8624 support is still in early development, so beat on it and give feedback or
8629 <Term>CBQ.init <ULink
8630 URL="ftp://ftp.equinox.gu.net/pub/linux/cbq/"
8635 Configuring CBQ can be a bit daunting, especially if all you want to do is
8636 shape some computers behind a router. CBQ.init can help you configure Linux
8637 with a simplified syntax.
8641 For example, if you want all computers in your 192.168.1.0/24 subnet
8642 (on 10mbit eth1) to be limited to 28kbit/s download speed, put
8643 this in the CBQ.init configuration file:
8649 DEVICE=eth1,10Mbit,1Mbit
8659 By all means use this program if the 'how and why' don't interest you.
8660 We're using CBQ.init in production and it works very well. It can even do
8661 some more advanced things, like time dependent shaping. The documentation is
8662 embedded in the script, which explains why you can't find a README.
8666 <Term>Chronox easy shaping scripts <ULink
8667 URL="http://www.chronox.de"
8672 Stephan Mueller (smueller@chronox.de) wrote two useful scripts, 'limit.conn'
8673 and 'shaper'. The first one allows you to easily throttle a single download
8680 # limit.conn -s SERVERIP -p SERVERPORT -l LIMIT
8686 It works on Linux 2.2 and 2.4/2.5.
8690 The second script is more complicated, and can be used to make lots of
8691 different queues based on iptables rules, which are used to mark packets
8692 which are then shaped.
8696 <Term>Virtual Router
8697 Redundancy Protocol implementation <ULink
8698 URL="http://w3.arobas.net/~jetienne/vrrpd/index.html"
8703 This is purely for redundancy. Two machines with their own IP address and
8704 MAC Address together create a third IP Address and MAC Address, which is
8705 virtual. Originally intended purely for routers, which need constant MAC
8706 addresses, it also works for other servers.
8710 The beauty of this approach is the incredibly easy configuration. No kernel
8711 compiling or patching required, all userspace.
8715 Just run this on all machines participating in a service:
8718 # vrrpd -i eth0 -v 50 10.0.0.22
8724 And you are in business! 10.0.0.22 is now carried by one of your servers,
8725 probably the first one to run the vrrp daemon. Now disconnect that computer
8726 from the network and very rapidly one of the other computers will assume the
8727 10.0.0.22 address, as well as the MAC address.
8731 I tried this over here and had it up and running in 1 minute. For some
8732 strange reason it decided to drop my default gateway, but the -n flag
8737 This is a 'live' fail over:
8743 64 bytes from 10.0.0.22: icmp_seq=3 ttl=255 time=0.2 ms
8744 64 bytes from 10.0.0.22: icmp_seq=4 ttl=255 time=0.2 ms
8745 64 bytes from 10.0.0.22: icmp_seq=5 ttl=255 time=16.8 ms
8746 64 bytes from 10.0.0.22: icmp_seq=6 ttl=255 time=1.8 ms
8747 64 bytes from 10.0.0.22: icmp_seq=7 ttl=255 time=1.7 ms
8753 Not *one* ping packet was lost! Just after packet 4, I disconnected my P200
8754 from the network, and my 486 took over, which you can see from the higher
8763 <chapter id="lartc.further">
8764 <Title>Further reading</Title>
8771 URL="http://snafu.freedom.org/linux2.2/iproute-notes.html"
8772 >http://snafu.freedom.org/linux2.2/iproute-notes.html</ULink
8776 Contains lots of technical information, comments from the kernel
8781 URL="http://www.davin.ottawa.on.ca/ols/"
8782 >http://www.davin.ottawa.on.ca/ols/</ULink
8786 Slides by Jamal Hadi Salim, one of the authors of Linux traffic control
8791 URL="http://defiant.coinet.com/iproute2/ip-cref/"
8792 >http://defiant.coinet.com/iproute2/ip-cref/</ULink
8796 HTML version of Alexeys LaTeX documentation - explains part of iproute2 in
8802 URL="http://www.aciri.org/floyd/cbq.html"
8803 >http://www.aciri.org/floyd/cbq.html</ULink
8807 Sally Floyd has a good page on CBQ, including her original papers. None of
8808 it is Linux specific, but it does a fair job discussing the theory and uses
8810 Very technical stuff, but good reading for those so inclined.
8814 <Term>Differentiated Services on Linux</Term>
8818 URL="ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz"
8820 > by Werner Almesberger, Jamal Hadi Salim and Alexey
8821 Kuznetsov describes DiffServ facilities in the Linux kernel, amongst which
8822 are TBF, GRED, the DSMARK qdisc and the tcindex classifier.
8827 URL="http://ceti.pl/~kravietz/cbq/NET4_tc.html"
8828 >http://ceti.pl/~kravietz/cbq/NET4_tc.html</ULink
8832 Yet another HOWTO, this time in Polish! You can copy/paste command lines
8833 however, they work just the same in every language. The author is
8834 cooperating with us and may soon author sections of this HOWTO.
8839 URL="http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/car.htm"
8840 >IOS Committed Access Rate</ULink
8845 From the helpful folks of Cisco who have the laudable habit of putting
8846 their documentation online. Cisco syntax is different but the concepts are
8847 the same, except that we can do more and do it without routers the price of
8852 <Term>Docum experimental site<ULink
8853 URL="http://www.docum.org"
8858 Stef Coene is busy convincing his boss to sell Linux support, and so he is
8859 experimenting a lot, especially with managing bandwidth. His site has a lot
8860 of practical information, examples, tests and also points out some CBQ/tc bugs.
8865 <Term>TCP/IP Illustrated, volume 1, W. Richard Stevens, ISBN 0-201-63346-9</Term>
8868 Required reading if you truly want to understand TCP/IP. Entertaining as
8878 <chapter id="lartc.ack">
8879 <Title>Acknowledgements </Title>
8883 It is our goal to list everybody who has contributed to this HOWTO, or
8884 helped us demystify how things work. While there are currently no plans
8885 for a Netfilter type scoreboard, we do like to recognize the people who are
8891 <ItemizedList spacing="compact">
8895 <author><firstname>Junk</firstname><surname>Alins</surname></author>
8896 <address><email>juanjo@mat.upc.es</email></address>
8901 <author><firstname>Joe</firstname><surname>Van Andel</surname></author>
8907 <author><firstname>Michael</firstname><othername>T.</othername>
8908 <surname>Babcock</surname></author>
8909 <address><email>mbabcock@fibrespeed.net</email></address>
8916 <author><firstname>Christopher</firstname>
8917 <surname>Barton</surname></author>
8918 <address><email>cpbarton%uiuc.edu</email></address>
8925 <author><firstname>Ard</firstname><surname>van Breemen</surname></author>
8926 <address><email>ard%kwaak.net</email></address>
8931 <author><firstname>Ron</firstname><surname>Brinker</surname></author>
8932 <address><email>service%emcis.com</email></address>
8937 <author><firstname>?ukasz</firstname><surname>Bromirski</surname></author>
8938 <address><email>l.bromirski@mr0vka.eu.org</email></address>
8943 <author><firstname>Lennert</firstname><surname>Buytenhek</surname></author>
8944 <address><email>buytenh@gnu.org</email></address>
8949 <author><firstname>Esteve</firstname><surname>Camps</surname></author>
8950 <address><email>esteve@hades.udg.es</email></address>
8955 <author><firstname>Stef</firstname><surname>Coene</surname></author>
8956 <address><email>stef.coene@docum.org</email></address>
8961 <author><firstname>Don</firstname><surname>Cohen</surname></author>
8962 <address><email>don-lartc%isis.cs3-inc.com</email></address>
8967 <author><firstname>Jonathan</firstname><surname>Corbet</surname></author>
8968 <address><email>lwn%lwn.net</email></address>
8973 <author><firstname>Gerry</firstname><surname>Creager</surname>
8974 <othername>N5JXS</othername></author>
8975 <address><email>gerry%cs.tamu.edu</email></address>
8980 <author><firstname>Marco</firstname><surname>Davids</surname></author>
8981 <address><email>marco@sara.nl</email></address>
8986 <author><firstname>Jonathan</firstname><surname>Day</surname></author>
8987 <address><email>jd9812@my-deja.com</email></address>
8992 <author><firstname>Martin</firstname><surname>Devera</surname>
8993 <othername>aka devik</othername></author>
8994 <address><email>devik@cdi.cz</email></address>
9001 <author><firstname>Stephan</firstname><othername>"Kobold"</othername>
9002 <surname>Gehring</surname></author>
9003 <address><email>Stephan.Gehring@bechtle.de</email></address>
9008 <author><firstname>Jacek</firstname><surname>Glinkowski</surname></author>
9009 <address><email>jglinkow%hns.com</email></address>
9014 <author><firstname>Andrea</firstname><surname>Glorioso</surname></author>
9015 <address><email>sama%perchetopi.org</email></address>
9020 <author><firstname>Nadeem</firstname><surname>Hasan</surname></author>
9021 <address><email>nhasan@usa.net</email></address>
9026 <author><firstname>Erik</firstname><surname>Hensema</surname></author>
9027 <address><email>erik%hensema.xs4all.nl</email></address>
9032 <author><firstname>Vik</firstname><surname>Heyndrickx</surname></author>
9033 <address><email>vik.heyndrickx@edchq.com</email></address>
9038 <author><firstname>Spauldo</firstname><surname>Da Hippie</surname></author>
9039 <address><email>spauldo%usa.net</email></address>
9044 <author><firstname>Koos</firstname><surname>van den Hout</surname></author>
9045 <address><email>koos@kzdoos.xs4all.nl</email></address>
9051 Stefan Huelbrock <shuelbrock%datasystems.de>
9057 Alexander W. Janssen <yalla%ynfonatic.de>
9063 Gareth John <gdjohn%zepler.org>
9069 <author><firstname>Dave</firstname><surname>Johnson</surname></author>
9070 <address><email>dj@www.uk.linux.org</email></address>
9078 Martin Josefsson <gandalf%wlug.westbo.se>
9084 Andi Kleen <ak%suse.de>
9090 Andreas J. Koenig <andreas.koenig%anima.de>
9096 Pawel Krawczyk <kravietz%alfa.ceti.pl>
9102 Amit Kucheria <amitk@ittc.ku.edu>
9108 Edmund Lau <edlau%ucf.ics.uci.edu>
9114 Philippe Latu <philippe.latu%linux-france.org>
9120 Arthur van Leeuwen <arthurvl%sci.kun.nl>
9125 <author><firstname>Jose Luis Domingo</firstname><surname>Lopez</surname>
9127 <address><email>jdomingo@24x7linux.com</email></address>
9133 Jason Lunz <j@cc.gatech.edu>
9139 Stuart Lynne <sl@fireplug.net>
9145 Alexey Mahotkin <alexm@formulabez.ru>
9151 Predrag Malicevic <pmalic@ieee.org>
9156 Patrick McHardy <kaber@trash.net>
9164 Andreas Mohr <andi%lisas.de>
9170 Andrew Morton <akpm@zip.com.au>
9182 Stephan Mueller <smueller@chronox.de>
9188 Togan Muftuoglu <toganm%yahoo.com>
9194 Chris Murray <cmurray@stargate.ca>
9200 Patrick Nagelschmidt <dto%gmx.net>
9206 Ram Narula <ram@princess1.net>
9212 Jorge Novo <jnovo@educanet.net>
9218 Patrik <ph@kurd.nu>
9222 <listitem><para>P?l Osgy?ny <oplab%westel900.net></para></listitem>
9227 Lutz Preßler <Lutz.Pressler%SerNet.DE>
9233 Jason Pyeron <jason%pyeron.com>
9239 Rusty Russell <rusty%rustcorp.com.au>
9245 Mihai RUSU <dizzy%roedu.net>
9251 Jamal Hadi Salim <hadi%cyberus.ca>
9257 Ren? Serral <rserral%ac.upc.es>
9264 David Sauer <davids%penguin.cz>
9270 Sheharyar Suleman Shaikh <sss23@drexel.edu>
9276 Stewart Shields <MourningBlade%bigfoot.com>
9282 Nick Silberstein <nhsilber%yahoo.com>
9288 Konrads Smelkov <konrads@interbaltika.com>
9294 <author><firstname>William</firstname><surname>Stearns</surname></author>
9295 <address><email>wstearns@pobox.com</email></address>
9301 Andreas Steinmetz <ast%domdv.de>
9307 Jason Tackaberry <tack@linux.com>
9313 Charles Tassell <ctassell%isn.net>
9319 Glen Turner <glen.turner%aarnet.edu.au>
9325 Tea Sponsor: Eric Veldhuyzen <eric%terra.nu>
9333 Song Wang <wsong@ece.uci.edu>
9339 <author><firstname>Lazar</firstname><surname>Yanackiev</surname></author>
9340 <address><email>Lyanackiev%gmx.net</email></address>