1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V4.1//EN">
4 <?dbhtml banner-text="Made possible by PowerDNS">
5 <?dbhtml banner-href="http://www.powerdns.com">
8 <Title>Linux Advanced Routing & Traffic Control HOWTO</Title>
11 <FirstName>Bert</FirstName><Surname>Hubert</Surname>
13 <orgname>Netherlabs BV</orgname>
14 <address><email>bert.hubert@netherlabs.nl</email></address>
19 <collabname>Thomas Graf (Section Author)</collabname>
21 <address><email>tgraf%suug.ch</email></address>
27 <collabname>Gregory Maxwell (Section Author)</collabname>
32 <collabname>Remco van Mook (Section Author)</collabname>
34 <address><email>remco@virtu.nl</email></address>
39 <collabname>Martijn van Oosterhout (Section Author)</collabname>
41 <address><email>kleptog@cupid.suninternet.com</email></address>
46 <collabname>Paul B Schroeder (Section Author)</collabname>
48 <address><email>paulsch@us.ibm.com</email></address>
53 <collabname>Jasper Spaans (Section Author)</collabname>
55 <address><email>jasper@spaans.ds9a.nl</email></address>
59 <collabname>Pedro Larroy (Section Author)</collabname>
61 <address><email>piotr%member.fsf.org</email></address>
69 <revnumber role="rcs">$Revision$</revnumber>
70 <date role="rcs">$Date$</date>
71 <revremark>DocBook Edition</revremark>
76 <Para>A very hands-on approach to <application>iproute2</application>,
77 traffic shaping and a bit of <application>netfilter</application>.
83 <chapter id="lartc.dedication">
84 <Title>Dedication</Title>
87 This document is dedicated to lots of people, and is my attempt to do
88 something back. To list but a few:
108 The good folks from Google
114 The staff of Casema Internet
124 <chapter id="lartc.intro">
125 <Title>Introduction</Title>
128 Welcome, gentle reader.
132 This document hopes to enlighten you on how to do more with Linux 2.2/2.4
133 routing. Unbeknownst to most users, you already run tools which allow you to
134 do spectacular things. Commands like <command>route</command> and
135 <command>ifconfig</command> are actually
136 very thin wrappers for the very powerful iproute2 infrastructure.
140 I hope that this HOWTO will become as readable as the ones by Rusty Russell
141 of (amongst other things) netfilter fame.
145 You can always reach us by writing to the <ULink
146 URL="mailto:HOWTO@ds9a.nl"
148 >. However, please consider posting to the mailing
149 list (see the relevant section) if you have questions which are not directly
150 related to this HOWTO. We are no free helpdesk, but we often will answer questions
155 Before losing your way in this HOWTO, if all you want to do is simple
156 traffic shaping, skip everything and head to the <citetitle><xref linkend="lartc.other"></citetitle> chapter, and read about CBQ.init.
159 <Sect1 id="lartc.intro.disclaimer">
160 <Title>Disclaimer & License</Title>
163 This document is distributed in the hope that it will be useful,
164 but WITHOUT ANY WARRANTY; without even the implied warranty of
165 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
169 In short, if your STM-64 backbone breaks down and distributes pornography to
170 your most esteemed customers - it's never our fault. Sorry.
174 Copyright (c) 2002 by bert hubert, Gregory Maxwell, Martijn van
175 Oosterhout, Remco van Mook, Paul B. Schroeder and others. This material may
176 be distributed only subject to the terms and conditions set forth in the
177 Open Publication License, v1.0 or later (the latest version is presently
178 available at http://www.opencontent.org/openpub/).
182 Please freely copy and distribute (sell or give away) this document in any
183 format. It's requested that corrections and/or comments be forwarded to the
188 It is also requested that if you publish this HOWTO in hardcopy that you
189 send the authors some samples for <quote>review purposes</quote> :-)
194 <Sect1 id="lartc.intro.prior">
195 <Title>Prior knowledge</Title>
198 As the title implies, this is the <quote>Advanced</quote> HOWTO.
199 While by no means rocket science, some prior knowledge is assumed.
203 Here are some other references which might help teach you more:
207 <ULink URL="http://netfilter.samba.org/unreliable-guides/networking-concepts-HOWTO/index.html">
208 Rusty Russell's networking-concepts-HOWTO</ULink>
211 <Para>Very nice introduction, explaining what a network is, and how it is
212 connected to other networks.
217 <Term>Linux Networking-HOWTO (Previously the Net-3 HOWTO)</Term>
219 <Para>Great stuff, although very verbose. It teaches you a lot of stuff
220 that's already configured if you are able to connect to the Internet.
221 Should be located in <filename>/usr/doc/HOWTO/NET3-4-HOWTO.txt</filename>
222 but can be also be found
223 <ULink URL="http://www.linuxports.com/howto/networking">online</ULink>.
232 <Sect1 id="lartc.intro.linux">
233 <Title>What Linux can do for you</Title>
236 A small list of things that are possible:
241 <Para>Throttle bandwidth for certain computers
245 <Para>Throttle bandwidth TO certain computers
249 <Para>Help you to fairly share your bandwidth
253 <Para>Protect your network from DoS attacks
257 <Para>Protect the Internet from your customers
261 <Para>Multiplex several servers as one, for load balancing or
262 enhanced availability
266 <Para>Restrict access to your computers
270 <Para>Limit access of your users to other hosts
274 <Para>Do routing based on user id (yes!), MAC address, source IP
275 address, port, type of service, time of day or content
281 Currently, not many people are using these advanced features. This is for
282 several reasons. While the provided documentation is verbose, it is not very
283 hands-on. Traffic control is almost undocumented.
288 <Sect1 id="lartc.intro.houskeeping">
289 <Title>Housekeeping notes</Title>
292 There are several things which should be noted about this document. While I
293 wrote most of it, I really don't want it to stay that way. I am a strong
294 believer in Open Source, so I encourage you to send feedback, updates,
295 patches etcetera. Do not hesitate to inform me of typos or plain old errors.
296 If my English sounds somewhat wooden, please realize that I'm not a native
297 speaker. Feel free to send suggestions.
301 If you feel you are better qualified to maintain a section, or think that
302 you can author and maintain new sections, you are welcome to do so. The SGML
303 of this HOWTO is available via CVS, I very much envision more people
308 In aid of this, you will find lots of FIXME notices. Patches are always
309 welcome! Wherever you find a FIXME, you should know that you are treading in
310 unknown territory. This is not to say that there are no errors elsewhere,
311 but be extra careful. If you have validated something, please let us know so
312 we can remove the FIXME notice.
316 About this HOWTO, I will take some liberties along the road. For example, I
317 postulate a 10Mbit Internet connection, while I know full well that those
323 <Sect1 id="lartc.intro.cvs">
324 <Title>Access, CVS & submitting updates</Title>
327 The canonical location for the HOWTO is
328 <ULink URL="http://www.ds9a.nl/lartc">here</ULink>.
332 We now have anonymous CVS access available to the world at large. This is
333 good in a number of ways. You can easily upgrade to newer versions of this
334 HOWTO and submitting patches is no work at all.
338 Furthermore, it allows the authors to work on the source independently,
343 $ export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
345 CVS password: [enter 'cvs' (without 's)]
347 cvs server: Updating 2.4routing
348 U 2.4routing/lartc.db
352 If you made changes and want to contribute them, run <userinput>
353 cvs -z3 diff -uBb</userinput>,
354 and mail the output to <email>howto@ds9a.nl</email>, we
355 can then integrate it easily. Thanks! Please make sure that you edit the
356 .db file, by the way, the other files are generated from that one.
360 A Makefile is supplied which should help you create postscript, dvi, pdf,
361 html and plain text. You may need to install
362 <application>docbook</application>, <application>docbook-utils</application>,
363 <application>ghostscript</application> and <application>tetex</application>
368 Be careful not to edit 2.4routing.sgml! It contains an older version of the
369 HOWTO. The right file is lartc.db.
373 <Sect1 id="lartc.intro.mlist">
374 <Title>Mailing list</Title>
377 The authors receive an increasing amount of mail about this HOWTO. Because
378 of the clear interest of the community, it has been decided to start a
379 mailinglist where people can talk to each other about Advanced Routing and
380 Traffic Control. You can subscribe to the list
381 <ULink URL="http://mailman.ds9a.nl/mailman/listinfo/lartc">here</ULink>.
385 It should be pointed out that the authors are very hesitant of answering
386 questions not asked on the list. We would like the archive of the list to
387 become some kind of knowledge base. If you have a question, please search
388 the archive, and then post to the mailinglist.
393 <Sect1 id="lartc.intro.layout">
394 <Title>Layout of this document</Title>
397 We will be doing interesting stuff almost immediately, which also means that
398 there will initially be parts that are explained incompletely or are not
399 perfect. Please gloss over these parts and assume that all will become clear.
403 Routing and filtering are two distinct things. Filtering is documented very
404 well by Rusty's HOWTOs, available here:
409 <Para><ULink URL="http://netfilter.samba.org/unreliable-guides/">
410 Rusty's Remarkably Unreliable Guides</ULink>
415 <Para>We will be focusing mostly on what is possible by combining netfilter
423 <chapter id="lartc.iproute2">
424 <Title>Introduction to iproute2</Title>
426 <Sect1 id="lartc.iproute2.why">
427 <Title>Why iproute2?</Title>
430 Most Linux distributions, and most UNIX's, currently use the
431 venerable <command>arp</command>, <command>ifconfig</command> and
432 <command>route</command> commands.
433 While these tools work, they show some unexpected behaviour under Linux 2.2
435 For example, GRE tunnels are an integral part of routing these days, but
436 require completely different tools.
440 With <application>iproute2</application>, tunnels are an integral part of
445 The 2.2 and above Linux kernels include a completely redesigned network
446 subsystem. This new networking code brings Linux performance and a feature
447 set with little competition in the general OS arena. In fact, the new
448 routing, filtering, and classifying code is more featureful than the one
449 provided by many dedicated routers and firewalls and traffic shaping
454 As new networking concepts have been invented, people have found ways to
455 plaster them on top of the existing framework in existing OSes. This
456 constant layering of cruft has lead to networking code that is filled with
457 strange behaviour, much like most human languages. In the past, Linux
458 emulated SunOS's handling of many of these things, which was not ideal.
462 This new framework makes it possible to clearly express features
463 previously beyond Linux's reach.
468 <Sect1 id="lartc.iproute2.tour">
469 <Title>iproute2 tour</Title>
472 Linux has a sophisticated system for bandwidth provisioning called Traffic
473 Control. This system supports various method for classifying, prioritizing,
474 sharing, and limiting both inbound and outbound traffic.
478 We'll start off with a tiny tour of iproute2 possibilities.
483 <Sect1 id="lartc.iproute2.package">
484 <Title>Prerequisites</Title>
487 You should make sure that you have the userland tools installed. This
488 package is called 'iproute' on both RedHat and Debian, and may otherwise be
489 found at <filename>ftp://ftp.inr.ac.ru/ip-routing/iproute2-2.2.4-now-ss??????.tar.gz"</filename>.
494 <ULink URL="ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz">here</ULink>
495 for the latest version.
499 Some parts of iproute require you to have certain kernel options enabled. It
500 should also be noted that all releases of RedHat up to and including 6.2
501 come without most of the traffic control features in the default kernel.
505 RedHat 7.2 has everything in by default.
509 Also make sure that you have netlink support, should you choose to roll your
510 own kernel. Iproute2 needs it.
515 <Sect1 id="lartc.iproute2.explore">
516 <Title>Exploring your current configuration</Title>
519 This may come as a surprise, but iproute2 is already configured! The current
520 commands <command>ifconfig</command> and <command>route</command> are already using the advanced
521 syscalls, but mostly with very default (ie. boring) settings.
525 The <command>ip</command> tool is central, and we'll ask it to display our interfaces
530 <Title><command>ip</command> shows us our links</Title>
533 [ahu@home ahu]$ ip link list
534 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
535 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
536 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
537 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
538 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
539 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
540 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
541 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
542 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
548 Your mileage may vary, but this is what it shows on my NAT router at
549 home. I'll only explain part of the output as not everything is directly
554 We first see the loopback interface. While your computer may function
555 somewhat without one, I'd advise against it. The MTU size (Maximum Transfer
556 Unit) is 3924 octets, and it is not supposed to queue. Which makes sense
557 because the loopback interface is a figment of your kernel's imagination.
561 I'll skip the dummy interface for now, and it may not be present on your
562 computer. Then there are my two physical network interfaces, one at the side
563 of my cable modem, the other one serves my home ethernet segment.
564 Furthermore, we see a ppp0 interface.
568 Note the absence of IP addresses. iproute disconnects the concept of 'links'
569 and 'IP addresses'. With IP aliasing, the concept of 'the' IP address had
570 become quite irrelevant anyhow.
574 It does show us the MAC addresses though, the hardware identifier of our
581 <Title><command>ip</command> shows us our IP addresses</Title>
584 [ahu@home ahu]$ ip address show
585 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
586 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
587 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
588 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
589 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
590 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
591 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
592 inet 10.0.0.1/8 brd 10.255.255.255 scope global eth0
593 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
594 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
595 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
597 inet 212.64.94.251 peer 212.64.94.1/32 scope global ppp0
601 This contains more information. It shows all our addresses, and to which
602 cards they belong. 'inet' stands for Internet (IPv4). There are lots of other
603 address families, but these don't concern us right now.
607 Let's examine eth0 somewhat closer. It says that it is related to the inet
608 address '10.0.0.1/8'. What does this mean? The /8 stands for the number of
609 bits that are in the Network Address. There are 32 bits, so we have 24 bits
610 left that are part of our network. The first 8 bits of 10.0.0.1 correspond
611 to 10.0.0.0, our Network Address, and our netmask is 255.0.0.0.
615 The other bits are connected to this interface, so 10.250.3.13 is directly
616 available on eth0, as is 10.0.0.1 for example.
620 With ppp0, the same concept goes, though the numbers are different. Its
621 address is 212.64.94.251, without a subnet mask. This means that we have a
622 point-to-point connection and that every address, with the exception of
623 212.64.94.251, is remote. There is more information, however. It tells us
624 that on the other side of the link there is, yet again, only one address,
625 212.64.94.1. The /32 tells us that there are no 'network bits'.
629 It is absolutely vital that you grasp these concepts. Refer to the
630 documentation mentioned at the beginning of this HOWTO if you have trouble.
634 You may also note 'qdisc', which stands for Queueing Discipline. This will
635 become vital later on.
641 <Title><command>ip</command> shows us our routes</Title>
644 Well, we now know how to find 10.x.y.z addresses, and we are able to reach
645 212.64.94.1. This is not enough however, so we need instructions on how to
646 reach the world. The Internet is available via our ppp connection, and it
647 appears that 212.64.94.1 is willing to spread our packets around the
648 world, and deliver results back to us.
652 [ahu@home ahu]$ ip route show
653 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
654 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
655 127.0.0.0/8 dev lo scope link
656 default via 212.64.94.1 dev ppp0
660 This is pretty much self explanatory. The first 3 lines of output explicitly
661 state what was already implied by <command>ip address show</command>, the last line
662 tells us that the rest of the world can be found via 212.64.94.1, our
663 default gateway. We can see that it is a gateway because of the word
664 via, which tells us that we need to send packets to 212.64.94.1, and that it
665 will take care of things.
669 For reference, this is what the old <command>route</command> utility shows us:
673 [ahu@home ahu]$ route -n
674 Kernel IP routing table
675 Destination Gateway Genmask Flags Metric Ref Use
677 212.64.94.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0
678 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
679 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
680 0.0.0.0 212.64.94.1 0.0.0.0 UG 0 0 0 ppp0
687 <Sect1 id="lartc.iproute2.arp">
691 ARP is the Address Resolution Protocol as described in
692 <ULink URL="http://www.faqs.org/rfcs/rfc826.html">RFC 826</ULink>.
693 ARP is used by a networked machine to resolve the hardware location/address of
694 another machine on the same
695 local network. Machines on the Internet are generally known by their names
697 addresses. This is how a machine on the foo.com network is able to communicate
698 with another machine which is on the bar.net network. An IP address, though,
699 cannot tell you the physical location of a machine. This is where ARP comes
704 Let's take a very simple example. Suppose I have a network composed of several
705 machines. Two of the machines which are currently on my network are foo
706 with an IP address of 10.0.0.1 and bar with an IP address of 10.0.0.2.
707 Now foo wants to ping bar to see that he is alive, but alas, foo has no idea
708 where bar is. So when foo decides to ping bar he will need to send
710 This ARP request is akin to foo shouting out on the network "Bar (10.0.0.2)!
711 Where are you?" As a result of this every machine on the network will hear
712 foo shouting, but only bar (10.0.0.2) will respond. Bar will then send an
713 ARP reply directly back to foo which is akin
715 "Foo (10.0.0.1) I am here at 00:60:94:E9:08:12." After this simple transaction
716 that's used to locate his friend on the network, foo is able to communicate
717 with bar until he (his arp cache) forgets where bar is (typically after
722 Now let's see how this works.
723 You can view your machines current arp/neighbor cache/table like so:
727 [root@espa041 /home/src/iputils]# ip neigh show
728 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
729 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
733 As you can see my machine espa041 (9.3.76.41) knows where to find espa042
735 espagate (9.3.76.1). Now let's add another machine to the arp cache.
739 [root@espa041 /home/paulsch/.gnome-desktop]# ping -c 1 espa043
740 PING espa043.austin.ibm.com (9.3.76.43) from 9.3.76.41 : 56(84) bytes of data.
741 64 bytes from 9.3.76.43: icmp_seq=0 ttl=255 time=0.9 ms
743 --- espa043.austin.ibm.com ping statistics ---
744 1 packets transmitted, 1 packets received, 0% packet loss
745 round-trip min/avg/max = 0.9/0.9/0.9 ms
747 [root@espa041 /home/src/iputils]# ip neigh show
748 9.3.76.43 dev eth0 lladdr 00:06:29:21:80:20 nud reachable
749 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
750 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
754 As a result of espa041 trying to contact espa043, espa043's hardware
755 address/location has now been added to the arp/neighbor cache.
756 So until the entry for
757 espa043 times out (as a result of no communication between the two) espa041
758 knows where to find espa043 and has no need to send an ARP request.
762 Now let's delete espa043 from our arp cache:
766 [root@espa041 /home/src/iputils]# ip neigh delete 9.3.76.43 dev eth0
767 [root@espa041 /home/src/iputils]# ip neigh show
768 9.3.76.43 dev eth0 nud failed
769 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
770 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud stale
774 Now espa041 has again forgotten where to find espa043 and will need to send
775 another ARP request the next time he needs to communicate with espa043.
776 You can also see from the above output that espagate (9.3.76.1) has been
777 changed to the "stale" state. This means that the location shown is still
778 valid, but it will have to be confirmed at the first transaction to that
786 <chapter id="lartc.rpdb">
787 <Title>Rules - routing policy database</Title>
790 If you have a large router, you may well cater for the needs of different
791 people, who should be served differently. The routing policy database allows
792 you to do this by having multiple sets of routing tables.
796 If you want to use this feature, make sure that your kernel is compiled with
797 the "IP: advanced router" and "IP: policy routing" features.
801 When the kernel needs to make a routing decision, it finds out which table
802 needs to be consulted. By default, there are three tables. The old 'route'
803 tool modifies the main and local tables, as does the ip tool (by default).
806 <Para>The default rules:
810 [ahu@home ahu]$ ip rule list
811 0: from all lookup local
812 32766: from all lookup main
813 32767: from all lookup default
817 This lists the priority of all rules. We see that all rules apply to all
818 packets ('from all'). We've seen the 'main' table before, it is output by
819 <userinput>ip route ls</userinput>, but the 'local' and 'default' table are new.
823 If we want to do fancy things, we generate rules which point to different
824 tables which allow us to override system wide routing rules.
828 For the exact semantics on what the kernel does when there are more matching
829 rules, see Alexey's ip-cref documentation.
832 <Sect1 id="lartc.rpdb.simple">
833 <Title>Simple source policy routing</Title>
836 Let's take a real example once again, I have 2 (actually 3, about time I
837 returned them) cable modems, connected to a Linux NAT ('masquerading')
838 router. People living here pay me to use the Internet. Suppose one of my
839 house mates only visits hotmail and wants to pay less. This is fine with me,
840 but they'll end up using the low-end cable modem.
844 The 'fast' cable modem is known as 212.64.94.251 and is a PPP link to
845 212.64.94.1. The 'slow' cable modem is known by various ip addresses,
846 212.64.78.148 in this example and is a link to 195.96.98.253.
849 <Para>The local table:
853 [ahu@home ahu]$ ip route list table local
854 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
855 local 10.0.0.1 dev eth0 proto kernel scope host src 10.0.0.1
856 broadcast 10.0.0.0 dev eth0 proto kernel scope link src 10.0.0.1
857 local 212.64.94.251 dev ppp0 proto kernel scope host src 212.64.94.251
858 broadcast 10.255.255.255 dev eth0 proto kernel scope link src 10.0.0.1
859 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
860 local 212.64.78.148 dev ppp2 proto kernel scope host src 212.64.78.148
861 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
862 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
866 Lots of obvious things, but things that need to be specified somewhere.
867 Well, here they are. The default table is empty.
870 <Para>Let's view the 'main' table:
874 [ahu@home ahu]$ ip route list table main
875 195.96.98.253 dev ppp2 proto kernel scope link src 212.64.78.148
876 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
877 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
878 127.0.0.0/8 dev lo scope link
879 default via 212.64.94.1 dev ppp0
883 We now generate a new rule which we call 'John', for our hypothetical
884 house mate. Although we can work with pure numbers, it's far easier if we add
885 our tables to /etc/iproute2/rt_tables.
889 # echo 200 John >> /etc/iproute2/rt_tables
890 # ip rule add from 10.0.0.10 table John
892 0: from all lookup local
893 32765: from 10.0.0.10 lookup John
894 32766: from all lookup main
895 32767: from all lookup default
899 Now all that is left is to generate John's table, and flush the route cache:
903 # ip route add default via 195.96.98.253 dev ppp2 table John
904 # ip route flush cache
908 And we are done. It is left as an exercise for the reader to implement this
914 <sect1 id="lartc.rpdb.multiple-links">
915 <title>Routing for multiple uplinks/providers</title>
917 A common configuration is the following, in which there are two providers
918 that connect a local network (or even a single machine) to the big Internet.
924 +-------------+ Provider 1 +-------
926 ___/ \_ +------+-------+ +------------+ |
929 | Local network -----+ Linux router | | Internet
932 \___/ +------+-------+ +------------+ |
934 +-------------+ Provider 2 +-------
936 +------------+ \________
940 There are usually two questions given this setup.
942 <sect2><title>Split access</title>
944 The first is how to route answers to packets coming in over a
945 particular provider, say Provider 1, back out again over that same provider.
948 Let us first set some symbolical names. Let <command>$IF1</command> be the name of the
949 first interface (if1 in the picture above) and <command>$IF2</command> the name of the
950 second interface. Then let <command>$IP1</command> be the IP address associated with
951 <command>$IF1</command> and <command>$IP2</command> the IP address associated with
952 <command>$IF2</command>. Next, let <command>$P1</command> be the IP address of the gateway at
953 Provider 1, and <command>$P2</command> the IP address of the gateway at provider 2.
954 Finally, let <command>$P1_NET</command> be the IP network <command>$P1</command> is in,
955 and <command>$P2_NET</command> the IP network <command>$P2</command> is in.
958 One creates two additional routing tables, say <command>T1</command> and <command>T2</command>.
959 These are added in /etc/iproute2/rt_tables. Then you set up routing in
960 these tables as follows:
964 ip route add $P1_NET dev $IF1 src $IP1 table T1
965 ip route add default via $P1 table T1
966 ip route add $P2_NET dev $IF2 src $IP2 table T2
967 ip route add default via $P2 table T2
970 Nothing spectacular, just build a route to the gateway and build a
971 default route via that gateway, as you would do in the case of a single
972 upstream provider, but put the routes in a separate table per provider.
973 Note that the network route suffices, as it tells you how to find any host
974 in that network, which includes the gateway, as specified above.
977 Next you set up the main routing table. It is a good idea to route
978 things to the direct neighbour through the interface connected to that
979 neighbour. Note the `src' arguments, they make sure the right outgoing IP
983 ip route add $P1_NET dev $IF1 src $IP1
984 ip route add $P2_NET dev $IF2 src $IP2
987 Then, your preference for default route:
990 ip route add default via $P1
993 Next, you set up the routing rules. These actually choose what routing table
994 to route with. You want to make sure that you route out a given
995 interface if you already have the corresponding source address:
998 ip rule add from $IP1 table T1
999 ip rule add from $IP2 table T2
1002 This set of commands makes sure all answers to traffic coming in on a
1003 particular interface get answered from that interface.
1007 Reader Rod Roark notes: 'If $P0_NET is the local network and $IF0 is its interface,
1008 the following additional entries are desirable:
1010 ip route add $P0_NET dev $IF0 table T1
1011 ip route add $P2_NET dev $IF2 table T1
1012 ip route add 127.0.0.0/8 dev lo table T1
1013 ip route add $P0_NET dev $IF0 table T2
1014 ip route add $P1_NET dev $IF1 table T2
1015 ip route add 127.0.0.0/8 dev lo table T2
1017 </para></warning</para>
1019 Now, this is just the very basic setup. It will work for all processes
1020 running on the router itself, and for the local network, if it is
1021 masqueraded. If it is not, then you either have IP space from both providers
1022 or you are going to want to masquerade to one of the two providers. In both
1023 cases you will want to add rules selecting which provider to route out from
1024 based on the IP address of the machine in the local network.
1027 <sect2><title>Load balancing</title>
1029 The second question is how to balance traffic going out over the two providers.
1030 This is actually not hard if you already have set up split access as above.
1033 Instead of choosing one of the two providers as your default route,
1034 you now set up the default route to be a multipath route. In the default
1035 kernel this will balance routes over the two providers. It is done
1036 as follows (once more building on the example in the section on
1040 ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
1041 nexthop via $P2 dev $IF2 weight 1
1044 This will balance the routes over both providers. The <command>weight</command>
1045 parameters can be tweaked to favor one provider over the other.
1048 Note that balancing will not be perfect, as it is route based, and routes
1049 are cached. This means that routes to often-used sites will always
1050 be over the same provider.
1053 Furthermore, if you really want to do this, you probably also want to look
1054 at Julian Anastasov's patches at <ulink url="http://www.ssi.bg/~ja/#routes">http://www.ssi.bg/~ja/#routes
1055 </ulink>, Julian's route patch page. They will make things nicer to work with.
1061 <chapter id="lartc.tunnel">
1062 <Title>GRE and other tunnels</Title>
1065 There are 3 kinds of tunnels in Linux. There's IP in IP tunneling, GRE tunneling and tunnels that live outside the kernel (like, for example PPTP).
1068 <Sect1 id="lartc.tunnel.remarks">
1069 <Title>A few general remarks about tunnels:</Title>
1072 Tunnels can be used to do some very unusual and very cool stuff. They can
1073 also make things go horribly wrong when you don't configure them right.
1074 Don't point your default route to a tunnel device unless you know
1075 <Emphasis>EXACTLY</Emphasis> what you are doing :-). Furthermore, tunneling increases
1076 overhead, because it needs an extra set of IP headers. Typically this is 20
1077 bytes per packet, so if the normal packet size (MTU) on a network is 1500
1078 bytes, a packet that is sent through a tunnel can only be 1480 bytes big.
1079 This is not necessarily a problem, but be sure to read up on IP packet
1080 fragmentation/reassembly when you plan to connect large networks with
1081 tunnels. Oh, and of course, the fastest way to dig a tunnel is to dig at
1087 <Sect1 id="lartc.tunnel.ip-ip">
1088 <Title>IP in IP tunneling</Title>
1091 This kind of tunneling has been available in Linux for a long time. It requires 2 kernel modules,
1092 ipip.o and new_tunnel.o.
1096 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
1097 So we have network A:
1102 netmask 255.255.255.0
1106 <Para>The router has address 172.16.17.18 on network C.
1109 <Para>and network B:
1114 netmask 255.255.255.0
1118 <Para>The router has address 172.19.20.21 on network C.
1122 As far as network C is concerned, we assume that it will pass any packet sent
1123 from A to B and vice versa. You might even use the Internet for this.
1126 <Para>Here's what you do:
1129 <Para>First, make sure the modules are installed:
1137 <Para>Then, on the router of network A, you do the following:
1141 ifconfig tunl0 10.0.1.1 pointopoint 172.19.20.21
1142 route add -net 10.0.2.0 netmask 255.255.255.0 dev tunl0
1145 <Para>And on the router of network B:
1149 ifconfig tunl0 10.0.2.1 pointopoint 172.16.17.18
1150 route add -net 10.0.1.0 netmask 255.255.255.0 dev tunl0
1153 <Para>And if you're finished with your tunnel:
1160 <Para>Presto, you're done. You can't forward broadcast or IPv6 traffic through
1161 an IP-in-IP tunnel, though. You just connect 2 IPv4 networks that normally wouldn't be able to talk to each other, that's all. As far as compatibility goes, this code has been around a long time, so it's compatible all the way back to 1.3 kernels. Linux IP-in-IP tunneling doesn't work with other Operating Systems or routers, as far as I know. It's simple, it works. Use it if you have to, otherwise use GRE.
1166 <Sect1 id="lartc.tunnel.gre">
1167 <Title>GRE tunneling</Title>
1170 GRE is a tunneling protocol that was originally developed by Cisco, and it
1171 can do a few more things than IP-in-IP tunneling. For example, you can also
1172 transport multicast traffic and IPv6 through a GRE tunnel.
1176 In Linux, you'll need the ip_gre.o module.
1180 <Title>IPv4 Tunneling</Title>
1183 Let's do IPv4 tunneling first:
1187 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
1191 So we have network A:
1195 netmask 255.255.255.0
1199 The router has address 172.16.17.18 on network C.
1200 Let's call this network neta (ok, hardly original)
1208 netmask 255.255.255.0
1212 The router has address 172.19.20.21 on network C.
1213 Let's call this network netb (still not original)
1217 As far as network C is concerned, we assume that it will pass any packet sent
1218 from A to B and vice versa. How and why, we do not care.
1221 <Para>On the router of network A, you do the following:
1225 ip tunnel add netb mode gre remote 172.19.20.21 local 172.16.17.18 ttl 255
1227 ip addr add 10.0.1.1 dev netb
1228 ip route add 10.0.2.0/24 dev netb
1232 Let's discuss this for a bit. In line 1, we added a tunnel device, and
1233 called it netb (which is kind of obvious because that's where we want it to
1234 go). Furthermore we told it to use the GRE protocol (mode gre), that the
1235 remote address is 172.19.20.21 (the router at the other end), that our
1236 tunneling packets should originate from 172.16.17.18 (which allows your
1237 router to have several IP addresses on network C and let you decide which
1238 one to use for tunneling) and that the TTL field of the packet should be set
1243 The second line enables the device.
1247 In the third line we gave the newly born interface netb the address
1248 10.0.1.1. This is OK for smaller networks, but when you're starting up a
1249 mining expedition (LOTS of tunnels), you might want to consider using
1250 another IP range for tunneling interfaces (in this example, you could use
1255 In the fourth line we set the route for network B. Note the different notation for the netmask. If you're not familiar with this notation, here's how it works: you write out the netmask in binary form, and you count all the ones. If you don't know how to do that, just remember that 255.0.0.0 is /8, 255.255.0.0 is /16 and 255.255.255.0 is /24. Oh, and 255.255.254.0 is /23, in case you were wondering.
1259 But enough about this, let's go on with the router of network B.
1262 ip tunnel add neta mode gre remote 172.16.17.18 local 172.19.20.21 ttl 255
1264 ip addr add 10.0.2.1 dev neta
1265 ip route add 10.0.1.0/24 dev neta
1268 And when you want to remove the tunnel on router A:
1271 ip link set netb down
1275 Of course, you can replace netb with neta for router B.
1281 <Title>IPv6 Tunneling</Title>
1284 See Section 6 for a short bit about IPv6 Addresses.
1288 On with the tunnels.
1292 Let's assume that you have the following IPv6 network, and you want to connect it to 6bone, or a friend.
1298 Network 3ffe:406:5:1:5:a:2:1/96
1301 Your IPv4 address is 172.16.17.18, and the 6bone router has IPv4 address 172.22.23.24.
1307 ip tunnel add sixbone mode sit remote 172.22.23.24 local 172.16.17.18 ttl 255
1308 ip link set sixbone up
1309 ip addr add 3ffe:406:5:1:5:a:2:1/96 dev sixbone
1310 ip route add 3ffe::/15 dev sixbone
1316 Let's discuss this. In the first line, we created a tunnel device called sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it where to go to (remote) and where to come from (local). TTL is set to maximum, 255. Next, we made the device active (up). After that, we added our own network address, and set a route for 3ffe::/15 (which is currently all of 6bone) through the tunnel.
1320 GRE tunnels are currently the preferred type of tunneling. It's a standard that is also widely adopted outside the Linux community and therefore a Good Thing.
1327 <Sect1 id="lartc.tunnel.userland">
1328 <Title>Userland tunnels</Title>
1331 There are literally dozens of implementations of tunneling outside the kernel. Best known are of course PPP and PPTP, but there are lots more (some proprietary, some secure, some that don't even use IP) and that is really beyond the scope of this HOWTO.
1338 <chapter id="lartc.ipv6-tunnel">
1339 <Title>IPv6 tunneling with Cisco and/or 6bone</Title>
1342 By Marco Davids <marco@sara.nl>
1350 As far as I am concerned, this IPv6-IPv4 tunneling is not per definition
1351 GRE tunneling. You could tunnel IPv6 over IPv4 by means of GRE tunnel devices
1352 (GRE tunnels ANY to IPv4), but the device used here ("sit") only tunnels
1353 IPv6 over IPv4 and is therefore something different.
1356 <Sect1 id="lartc.tunnel-ipv6.addressing">
1357 <Title>IPv6 Tunneling</Title>
1360 This is another application of the tunneling capabilities of Linux. It is
1361 popular among the IPv6 early adopters, or pioneers if you like.
1362 The 'hands-on' example described below is certainly not the only way
1363 to do IPv6 tunneling. However, it is the method that is often used to tunnel
1364 between Linux and a Cisco IPv6 capable router and experience tells us that
1365 this is just the thing many people are after. Ten to one this applies to
1370 A short bit about IPv6 addresses:
1374 IPv6 addresses are, compared to IPv4 addresses, really big: 128 bits
1375 against 32 bits. And this provides us just with the thing we need: many, many
1376 IP-addresses: 340,282,266,920,938,463,463,374,607,431,768,211,465 to be
1377 precise. Apart from this, IPv6 (or IPng, for IP Next Generation) is supposed
1378 to provide for smaller routing tables on the Internet's backbone routers,
1379 simpler configuration of equipment, better security at the IP level and
1380 better support for QoS.
1384 An example: 2002:836b:9820:0000:0000:0000:836b:9886
1388 Writing down IPv6 addresses can be quite a burden. Therefore, to make
1389 life easier there are some rules:
1398 Don't use leading zeroes. Same as in IPv4.
1405 Use colons to separate every 16 bits or two bytes.
1412 When you have lots of consecutive zeroes,
1413 you can write this down as ::. You can only do this once in an
1414 address and only for quantities of 16 bits, though.
1423 The address 2002:836b:9820:0000:0000:0000:836b:9886 can be written down
1424 as 2002:836b:9820::836b:9886, which is somewhat friendlier.
1428 Another example, the address 3ffe:0000:0000:0000:0000:0020:34A1:F32C can be
1429 written down as 3ffe::20:34A1:F32C, which is a lot shorter.
1433 IPv6 is intended to be the successor of the current IPv4. Because it
1434 is relatively new technology, there is no worldwide native IPv6 network
1435 yet. To be able to move forward swiftly, the 6bone was introduced.
1439 Native IPv6 networks are connected to each other by encapsulating the IPv6
1440 protocol in IPv4 packets and sending them over the existing IPv4 infrastructure
1441 from one IPv6 site to another.
1445 That is precisely where the tunnel steps in.
1449 To be able to use IPv6, we should have a kernel that supports it. There
1450 are many good documents on how to achieve this. But it all comes down to
1457 Get yourself a recent Linux distribution, with suitable glibc.
1463 Then get yourself an up-to-date kernel source.
1469 If you are all set, then you can go ahead and compile an IPv6 capable
1476 Go to /usr/src/linux and type:
1488 Choose "Networking Options"
1494 Select "The IPv6 protocol", "IPv6: enable EUI-64 token format", "IPv6:
1495 disable provider based addresses"
1501 HINT: Don't go for the 'module' option. Often this won't work well.
1505 In other words, compile IPv6 as 'built-in' in your kernel.
1506 You can then save your config like usual and go ahead with compiling
1511 HINT: Before doing so, consider editing the Makefile:
1512 EXTRAVERSION = -x ; --> ; EXTRAVERSION = -x-IPv6
1516 There is a lot of good documentation about compiling and installing
1517 a kernel, however this document is about something else. If you run into
1518 problems at this stage, go and look for documentation about compiling a
1519 Linux kernel according to your own specifications.
1523 The file /usr/src/linux/README might be a good start.
1524 After you accomplished all this, and rebooted with your brand new kernel,
1525 you might want to issue an '/sbin/ifconfig -a' and notice the brand
1526 new 'sit0-device'. SIT stands for Simple Internet Transition. You may give
1527 yourself a compliment; you are now one major step closer to IP, the Next
1532 Now on to the next step. You want to connect your host, or maybe even
1533 your entire LAN to another IPv6 capable network. This might be the "6bone"
1534 that is setup especially for this particular purpose.
1538 Let's assume that you have the following IPv6 network: 3ffe:604:6:8::/64 and
1539 you want to connect it to 6bone, or a friend. Please note that the /64
1540 subnet notation works just like with regular IP addresses.
1544 Your IPv4 address is 145.100.24.181 and the 6bone router has IPv4 address
1549 # ip tunnel add sixbone mode sit remote 145.100.1.5 [local 145.100.24.181 ttl 255]
1550 # ip link set sixbone up
1551 # ip addr add 3FFE:604:6:7::2/126 dev sixbone
1552 # ip route add 3ffe::0/16 dev sixbone
1556 Let's discuss this. In the first line, we created a tunnel device called
1557 sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it
1558 where to go to (remote) and where to come from (local). TTL is set to
1563 Next, we made the device active (up). After that, we added our own network
1564 address, and set a route for 3ffe::/15 (which is currently all of 6bone)
1565 through the tunnel. If the particular machine you run this on is your IPv6
1566 gateway, then consider adding the following lines:
1570 # echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
1571 # /usr/local/sbin/radvd
1575 The latter, radvd is -like zebra- a router advertisement daemon, to
1576 support IPv6's autoconfiguration features. Search for it with your favourite
1577 search-engine if you like.
1578 You can check things like this:
1582 # /sbin/ip -f inet6 addr
1586 If you happen to have radvd running on your IPv6 gateway and boot your
1587 IPv6 capable Linux on a machine on your local LAN, you would be able to
1588 enjoy the benefits of IPv6 autoconfiguration:
1592 # /sbin/ip -f inet6 addr
1593 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue inet6 ::1/128 scope host
1595 3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
1596 inet6 3ffe:604:6:8:5054:4cff:fe01:e3d6/64 scope global dynamic
1597 valid_lft forever preferred_lft 604646sec inet6 fe80::5054:4cff:fe01:e3d6/10
1602 You could go ahead and configure your bind for IPv6 addresses. The A
1603 type has an equivalent for IPv6: AAAA. The in-addr.arpa's equivalent is:
1604 ip6.int. There's a lot of information available on this topic.
1608 There is an increasing number of IPv6-aware applications available,
1609 including secure shell, telnet, inetd, Mozilla the browser, Apache the
1610 webserver and a lot of others. But this is all outside the scope of this
1611 Routing document ;-)
1615 On the Cisco side the configuration would be something like this:
1620 description IPv6 tunnel
1622 no ip directed-broadcast
1623 ipv6 address 3FFE:604:6:7::1/126
1624 tunnel source Serial0
1625 tunnel destination 145.100.24.181
1628 ipv6 route 3FFE:604:6:8::/64 Tunnel1
1631 But if you don't have a Cisco at your disposal, try one of the many
1632 IPv6 tunnel brokers available on the Internet. They are willing to configure
1633 their Cisco with an extra tunnel for you. Mostly by means of a friendly
1634 web interface. Search for "ipv6 tunnel broker" on your favourite search engine.
1641 <chapter id="lartc.ipsec">
1642 <Title>IPSEC: secure IP over the Internet</Title>
1645 There are two kinds of IPSEC available for Linux these days. For 2.2
1646 and 2.4, there is FreeS/WAN, which was the first major implementation. They
1648 have <ULink URL="http://www.freeswan.org/">an official site</ulink> and <ulink url="http://www.freeswan.ca">
1649 an unofficial one</ulink> that is actually maintained. FreeS/WAN has traditionally not been merged with
1650 the mainline kernel for a number of reasons. Most often mentioned are 'political' issues with Americans
1651 working on crypto tainting its exportability. Furthermore, it does not integrate too well with the Linux kernel,
1652 leading it to be a bad candidate for actual merging.
1655 Additionally, <ulink
1656 url="http://www.edlug.ed.ac.uk/archive/Sep2002/msg00244.html">many</ulink> parties <ulink
1657 url="http://lists.freeswan.org/pipermail/design/2002-November/003901.html">have voiced
1658 worries</ulink> about the quality of the code. To setup FreeS/WAN, a lot of
1660 url="http://www.freeswan.ca/docs/freeswan-1.99/doc/index.html">documentation</ulink>
1661 is <ulink url="http://www.freeswan.org/doc.html">available</ulink>.
1664 As of Linux 2.5.47, there is a native IPSEC implementation in the kernel. It was written by Alexey Kuznetsov and
1665 Dave Miller, inspired by the work of the USAGI IPv6 group. With its merge, James Morris' CrypoAPI also became
1666 part of the kernel - it does the actual crypting.
1669 This HOWTO will only document the 2.5+ version of IPSEC. FreeS/WAN is recommended for Linux 2.4 users for now, but be aware
1670 that its configuration will differ from the native IPSEC. In related
1671 news, there are now <ulink
1672 url="http://gondor.apana.org.au/~herbert/freeswan/">patches</ulink> to make the FreeS/WAN userspace code work with
1673 the native Linux IPSEC.
1676 As of 2.5.49, IPSEC works without further patches.
1681 Userspace tools appear to be available <ulink
1682 url="http://sourceforge.net/projects/ipsec-tools">here</ulink>.
1683 There are multiple programs available, the one linked here is based on
1687 When compiling your kernel, be sure to turn on 'PF_KEY', 'AH', 'ESP' and
1688 everything in the CryptoAPI!
1693 The author of this chapter is a complete IPSEC nitwit! If you find the inevitable mistakes, please email
1694 bert hubert <email>ahu@ds9a.nl</email>.
1699 First, we'll show how to manually setup secure communication between
1700 two hosts. A large part of this process can also be automated, but
1701 here we'll do it by hand so as to acquaint ourselves with what is going on
1705 Feel free to skip the following section if you are only interested
1706 in automatic keying but be aware that some understanding of manual keying is
1709 <sect1 id="lartc.ipsec.intro"><title>Intro with Manual Keying</title>
1711 IPSEC is a complicated subject. A lot of information is available online, this HOWTO will concentrate on getting you
1712 up and running and explaining the basic principles. All examples are
1713 based on Racoon as found on the link above.
1718 Many iptables configurations drop IPSEC packets! To pass IPSEC, use: 'iptables -A xxx -p 50 -j ACCEPT' and 'iptables -A xxx -p 51 -j ACCEPT'
1723 IPSEC offers a secure version of the Internet Protocol. Security in this context means two different things: encryption and authentication.
1724 A naive vision of security offers only encryption but it can easily be shown that is insufficient - you may be communicating encyphered,
1725 but no guarantee is offered that the remote party is the one you expect it to be.
1728 IPSEC supports 'Encapsulated Security Payload' (ESP) for encryption and 'Authentication Header' (AH) for authenticating the remote partner.
1729 You can configure both of them, or decided to do only either.
1732 Both ESP and AH rely on security associations. A security association (SA) consists of a source, a destination and an instruction. A sample
1733 authentication SA may look like this:
1735 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1737 This says 'traffic going from 10.0.0.11 to 10.0.0.216 that needs an AH can be signed using HMAC-MD5 using secret 1234567890123456'. This instruction
1738 is labelled with SPI ('Security Parameter Index') id '15700', more about that later.
1739 The interesting bit about SAs is that they are symmetrical. Both sides of a conversation share exactly the same SA, it is not mirrored on the
1740 other side. Do note however that there is no 'autoreverse' rule - this SA only describes a possible authentication from 10.0.0.11 to
1741 10.0.0.216. For two-way traffic, two SAs are needed.
1746 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1748 This says 'traffic going from 10.0.0.11 to 10.0.0.216 that needs encryption can be encyphered using 3des-cbc with key 123456789012123456789012'. The
1752 So far, we've seen that SAs describe possible instructions, but do not in fact describe policy as to when these need to be used. In fact,
1753 there could be an arbitrary number of nearly identical SAs with only differing SPI ids. Incidentally, SPI stands for Security Parameter Index.
1754 To do actual crypto, we need to describe a policy. This policy can include things as 'use ipsec if available' or 'drop traffic unless we have ispec'.
1757 A typical simple Security Policy (SP) looks like this:
1759 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1760 esp/transport//require
1761 ah/transport//require;
1763 If entered on host 10.0.0.216, this means that all traffic going out to 10.0.0.11 must be encrypted
1764 and be wrapped in an AH authenticating header. Note that this does not describe which SA is to be used,
1765 that is left as an exercise for the kernel to determine.
1768 In other words, a Security Policy specifies WHAT we want; a Security
1769 Association describes HOW we want it.
1772 Outgoing packets are labelled with the SA SPI ('the how') which the
1773 kernel used for encryption and authentication so the remote can
1774 lookup the corresponding verification and decryption instruction.
1778 What follows is a very simple configuration for talking from host 10.0.0.216 to 10.0.0.11 using
1779 encryption and authentication. Note that the reverse path is plaintext in this first version and that
1780 this configuration should not be deployed.
1786 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1787 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1789 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1790 esp/transport//require
1791 ah/transport//require;
1795 On host 10.0.0.11, the same Security Associations, no Security Policy:
1798 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1799 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1803 With the above configuration in place (these files can be executed if 'setkey' is installed in /sbin),
1804 'ping 10.0.0.11' from 10.0.0.216 looks like this using tcpdump:
1806 22:37:52 10.0.0.216 > 10.0.0.11: AH(spi=0x00005fb4,seq=0xa): ESP(spi=0x00005fb5,seq=0xa) (DF)
1807 22:37:52 10.0.0.11 > 10.0.0.216: icmp: echo reply
1809 Note how the ping back from 10.0.0.11 is indeed plainly visible. The forward ping cannot be read by tcpdump
1810 of course, but it does show the Security Parameter Index of AH and ESP, which tells 10.0.0.11 how to
1811 verify the authenticity of our packet and how to decrypt it.
1814 A few things must be mentioned however. The configuration above is shown in a lot of IPSEC examples and it is very dangerous.
1815 The problem is that the above contains policy on how 10.0.0.216 should treat packets going to 10.0.0.11, and that it explains how 10.0.0.11
1816 should treat those packets but it does NOT instruct 10.0.0.11 to discard unauthenticated or unencrypted traffic!
1819 Anybody can now insert spoofed and completely unencrypted data and 10.0.0.11 will accept it. To remedy the above, we need an incoming
1820 Security Policy on 10.0.0.11, as follows:
1823 spdadd 10.0.0.216 10.0.0.11 any -P IN ipsec
1824 esp/transport//require
1825 ah/transport//require;
1827 This instructs 10.0.0.11 that any traffic coming to it from 10.0.0.216 is required to have valid ESP and AH.
1830 Now, to complete this configuration, we need return traffic to be encrypted and authenticated as well of course. The full configuration on
1838 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1839 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1842 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1843 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1845 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
1846 esp/transport//require
1847 ah/transport//require;
1849 spdadd 10.0.0.11 10.0.0.216 any -P in ipsec
1850 esp/transport//require
1851 ah/transport//require;
1863 add 10.0.0.11 10.0.0.216 ah 15700 -A hmac-md5 "1234567890123456";
1864 add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";
1867 add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
1868 add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";
1871 spdadd 10.0.0.11 10.0.0.216 any -P out ipsec
1872 esp/transport//require
1873 ah/transport//require;
1875 spdadd 10.0.0.216 10.0.0.11 any -P in ipsec
1876 esp/transport//require
1877 ah/transport//require;
1882 Note that in this example we used identical keys for both directions of traffic. This is not in any way required however.
1885 To examine the configuration we just created, execute <command>setkey -D</command>, which shows the Security Associations or
1886 <command>setkey -DP</command> which shows the configured policies.
1889 <sect1 id="lartc.ipsec.automatic.keying"><title>Automatic keying</title>
1891 In the previous section, encryption was configured using simple shared secrets. In other words, to remain secure,
1892 we need to transfer our encryption configuration over a trusted channel. If we were to configure the remote host
1893 over telnet, any third party would know our shared secret and the setup would not be secure.
1896 Furthermore, because the secret is shared, it is not a secret. The remote can't do a lot with our secret, but we do
1897 need to make sure that we use a different secret for communicating with all our partners. This requires a large number of keys,
1898 if there are 10 parties, this needs at least 50 different secrets.
1901 Besides the symmetric key problem, there is also the need for key rollover. If a third party manages to sniff enough traffic,
1902 it may be in a position to reverse engineer the key. This is prevented by moving to a new key every once in a while but that is
1903 a process that needs to be automated.
1906 Another problem is that with manual keying as described above we exactly define the algorithms and key lengths used, something
1907 that requires a lot of coordination with the remote party. It is desirable to be able to have the ability to describe a
1908 broader key policy such as 'We can do 3DES and Blowfish with at least the following key lengths'.
1911 To solve these isses, IPSEC provides Internet Key Exchange to automatically exchange randomly generated keys which are
1912 transmitted using asymmetric encryption technology, according to negotiated algorithm details.
1915 The Linux 2.5 IPSEC implementation works with the KAME 'racoon' IKE
1916 daemon. As of 9 November, the racoon version in Alexey's iptools
1917 distribution can be compiled, although you may need to remove
1918 #include <net/route.h> in two files. Alternatively, I've supplied a
1919 <ulink url="http://ds9a.nl/ipsec/racoon.bz2">precompiled version</ulink>.
1924 IKE needs access to UDP port 500, be sure that iptables does
1929 <sect2 id="lartc.ipsec.keying.theory"><title>Theory</title>
1931 As explained before, automatic keying does a lot of the work
1932 for us. Specifically, it creates Security Associations on the fly. It does
1933 not however set policy for us, which is as it should be.
1936 So, to benefit from IKE, setup a policy, but do not supply any
1937 SAs. If the kernel discovers that there is an IPSEC policy, but no Security
1938 Association, it will notify the IKE daemon, which then goes to work on
1939 trying to negotiate one.
1942 Reiterating, a Security Policy specifies WHAT we want; a Security
1943 Association describes HOW we want it. Using automatic keying lets us get
1944 away with only specifying what we want.
1947 <sect2 id="lartc.ipsec.automatic.keying.example"><title>Example</title>
1949 Kame racoon comes with a grand host of options, most of which have
1950 very fine default values, so we don't need to touch them. As described
1951 above, the operator needs to define a Security Policy, but no Security
1952 Associations. We leave their negotiation to the IKE daemon.
1955 In this example, 10.0.0.11 and 10.0.0.216 are once again going to
1956 setup secure communications, but this time with help from racoon. For
1957 simplicity this configuration will be using pre-shared keys, the
1958 dreaded 'shared secrets'. X.509 certificates are discussed in a separate
1959 section, see <xref linkend="lartc.ipsec.x509">.
1962 going to stick to almost the default configuration, identical on both hosts:
1966 path pre_shared_key "/usr/local/etc/racoon/psk.txt";
1970 exchange_mode aggressive,main;
1972 situation identity_only;
1974 my_identifier address;
1976 lifetime time 2 min; # sec,min,hour
1978 proposal_check obey; # obey, strict or claim
1981 encryption_algorithm 3des;
1982 hash_algorithm sha1;
1983 authentication_method pre_shared_key;
1991 lifetime time 2 min;
1992 encryption_algorithm 3des ;
1993 authentication_algorithm hmac_sha1;
1994 compression_algorithm deflate ;
1999 Lots of settings - I think yet more can be removed to get closer to
2000 the default configuration. A few noteworthy things. We've configured two
2001 anonymous settings which hold for all remotes, making further configuration
2002 easy. There is no need for per-host stanzas here, unless we really want
2006 Furthermore, we've set it up such that we identify ourselves based
2007 on our IP address ('my_identifier address'), and declare that we can do
2008 3des, sha1, and that we will be using a pre-shared key, located in psk.txt.
2011 In psk.txt, we now setup two entries, which do differ on both hosts.
2014 10.0.0.216 password2
2020 Make sure these files are owned by root, and set to mode 0600,
2021 racoon will not trust their contents otherwise. Note that these files are
2022 mirrors from eachother.
2025 Now we are ready to setup our desired policy, which is simple
2026 enough. On host 10.0.0.216:
2032 spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
2033 esp/transport//require;
2035 spdadd 10.0.0.11 10.0.0.216 any -P in ipsec
2036 esp/transport//require;
2044 spdadd 10.0.0.11 10.0.0.216 any -P out ipsec
2045 esp/transport//require;
2047 spdadd 10.0.0.216 10.0.0.11 any -P in ipsec
2048 esp/transport//require;
2050 Note how again these policies are mirrored.
2053 We are now ready to launch racoon! Once launched, the moment we try
2054 to telnet from 10.0.0.11 to 10.0.0.216, or the other way around, racoon
2055 will start negotiating:
2057 12:18:44: INFO: isakmp.c:1689:isakmp_post_acquire(): IPsec-SA
2058 request for 10.0.0.11 queued due to no phase1 found.
2059 12:18:44: INFO: isakmp.c:794:isakmp_ph1begin_i(): initiate new
2060 phase 1 negotiation: 10.0.0.216[500]<=>10.0.0.11[500]
2061 12:18:44: INFO: isakmp.c:799:isakmp_ph1begin_i(): begin Aggressive mode.
2062 12:18:44: INFO: vendorid.c:128:check_vendorid(): received Vendor ID:
2064 12:18:44: NOTIFY: oakley.c:2037:oakley_skeyid(): couldn't find
2065 the proper pskey, try to get one by the peer's address.
2066 12:18:44: INFO: isakmp.c:2417:log_ph1established(): ISAKMP-SA
2067 established 10.0.0.216[500]-10.0.0.11[500] spi:044d25dede78a4d1:ff01e5b4804f0680
2068 12:18:45: INFO: isakmp.c:938:isakmp_ph2begin_i(): initiate new phase 2
2069 negotiation: 10.0.0.216[0]<=>10.0.0.11[0]
2070 12:18:45: INFO: pfkey.c:1106:pk_recvupdate(): IPsec-SA established:
2071 ESP/Transport 10.0.0.11->10.0.0.216 spi=44556347(0x2a7e03b)
2072 12:18:45: INFO: pfkey.c:1318:pk_recvadd(): IPsec-SA established:
2073 ESP/Transport 10.0.0.216->10.0.0.11 spi=15863890(0xf21052)
2077 If we now run setkey -D, which shows the Security Associations, they
2080 10.0.0.216 10.0.0.11
2081 esp mode=transport spi=224162611(0x0d5c7333) reqid=0(0x00000000)
2082 E: 3des-cbc 5d421c1b d33b2a9f 4e9055e3 857db9fc 211d9c95 ebaead04
2083 A: hmac-sha1 c5537d66 f3c5d869 bd736ae2 08d22133 27f7aa99
2084 seq=0x00000000 replay=4 flags=0x00000000 state=mature
2085 created: Nov 11 12:28:45 2002 current: Nov 11 12:29:16 2002
2086 diff: 31(s) hard: 600(s) soft: 480(s)
2087 last: Nov 11 12:29:12 2002 hard: 0(s) soft: 0(s)
2088 current: 304(bytes) hard: 0(bytes) soft: 0(bytes)
2089 allocated: 3 hard: 0 soft: 0
2090 sadb_seq=1 pid=17112 refcnt=0
2091 10.0.0.11 10.0.0.216
2092 esp mode=transport spi=165123736(0x09d79698) reqid=0(0x00000000)
2093 E: 3des-cbc d7af8466 acd4f14c 872c5443 ec45a719 d4b3fde1 8d239d6a
2094 A: hmac-sha1 41ccc388 4568ac49 19e4e024 628e240c 141ffe2f
2095 seq=0x00000000 replay=4 flags=0x00000000 state=mature
2096 created: Nov 11 12:28:45 2002 current: Nov 11 12:29:16 2002
2097 diff: 31(s) hard: 600(s) soft: 480(s)
2098 last: hard: 0(s) soft: 0(s)
2099 current: 231(bytes) hard: 0(bytes) soft: 0(bytes)
2100 allocated: 2 hard: 0 soft: 0
2101 sadb_seq=0 pid=17112 refcnt=0
2103 As are the Security Policies we configured ourselves:
2105 10.0.0.11[any] 10.0.0.216[any] tcp
2107 esp/transport//require
2108 created:Nov 11 12:28:28 2002 lastused:Nov 11 12:29:12 2002
2109 lifetime:0(s) validtime:0(s)
2110 spid=3616 seq=5 pid=17134
2112 10.0.0.216[any] 10.0.0.11[any] tcp
2114 esp/transport//require
2115 created:Nov 11 12:28:28 2002 lastused:Nov 11 12:28:44 2002
2116 lifetime:0(s) validtime:0(s)
2117 spid=3609 seq=4 pid=17134
2121 <sect3><title>Problems and known defects</title>
2123 If this does not work, check that all configuration files
2124 are owned by root, and can only be read by root. To start racoon on the
2125 foreground, use '-F'. To force it to read a certain configuration file,
2126 instead of at the compiled location, use '-f'. For staggering amounts of
2127 detail, add a 'log debug;' statement to racoon.conf.
2131 <sect2 id="lartc.ipsec.x509"><title>Automatic keying using X.509 certificates</title>
2133 As mentioned before, the use of shared secrets is hard because they
2134 aren't easily shared and once shared, are no longer secret. Luckily, there
2135 is asymmetric encryption technology to help resolve this.
2138 If each IPSEC participant makes a public and a private key, secure
2139 communications can be setup by both parties publishing their public key, and
2143 Building a key is relatively easy, although it requires some work.
2144 The following is based on the 'openssl' tool.
2146 <sect3><title>Building an X.509 certificate for your host</title>
2148 OpenSSL has a lot of infrastructure for keys that may or may not be
2149 signed by certificate authorities. Right now, we need to circumvent all that
2150 infrastructure and practice some good old Snake Oil security, and do without
2151 a certificate authority.
2154 First we issue a 'certificate request' for our host, called
2157 $ openssl req -new -nodes -newkey rsa:1024 -sha1 -keyform PEM -keyout \
2158 laptop.private -outform PEM -out request.pem
2160 This asks us some questions:
2162 Country Name (2 letter code) [AU]:NL
2163 State or Province Name (full name) [Some-State]:.
2164 Locality Name (eg, city) []:Delft
2165 Organization Name (eg, company) [Internet Widgits Pty Ltd]:Linux Advanced
2166 Routing & Traffic Control
2167 Organizational Unit Name (eg, section) []:laptop
2168 Common Name (eg, YOUR name) []:bert hubert
2169 Email Address []:ahu@ds9a.nl
2171 Please enter the following 'extra' attributes
2172 to be sent with your certificate request
2173 A challenge password []:
2174 An optional company name []:
2176 It is left to your own discretion how completely you want to fill
2177 this out. You may or may not want to put your hostname in there, depending
2178 on your security needs. In this example, we have.
2181 We'll now 'self sign' this request:
2183 $ openssl x509 -req -in request.pem -signkey laptop.private -out \
2186 subject=/C=NL/L=Delft/O=Linux Advanced Routing & Traffic \
2187 Control/OU=laptop/CN=bert hubert/Email=ahu@ds9a.nl
2190 The 'request.pem' file can now be discarded.
2193 Repeat this procedure for all hosts you need a key for. You can
2194 distribute the '.public' file with impunity, but keep the '.private' one
2198 <sect3><title>Setting up and launching</title>
2200 Once we have a public and a private key for our hosts we can tell
2204 We return to our previous configuration and the two hosts, 10.0.0.11
2205 ('upstairs') and 10.0.0.216 ('laptop').
2208 To the <filename>racoon.conf</filename> file on 10.0.0.11, we add:
2210 path certificate "/usr/local/etc/racoon/certs";
2214 exchange_mode aggressive,main;
2215 my_identifier asn1dn;
2216 peers_identifier asn1dn;
2218 certificate_type x509 "upstairs.public" "upstairs.private";
2220 peers_certfile "laptop.public";
2222 encryption_algorithm 3des;
2223 hash_algorithm sha1;
2224 authentication_method rsasig;
2229 This tells racoon that certificates are to be found in
2230 <filename>/usr/local/etc/racoon/certs/</filename>. Furthermore, it contains
2231 configuration items specific for remote 10.0.0.216.
2234 The 'asn1dn' lines tell racoon that the identifier for both the
2235 local and remote ends are to be extracted from the public keys. This is the
2236 'subject=/C=NL/L=Delft/O=Linux Advanced Routing & Traffic
2237 Control/OU=laptop/CN=bert hubert/Email=ahu@ds9a.nl' output from above.
2240 The <command>certificate_type</command> line configures the local
2241 public and private key. The <command>peers_certfile</command> statement
2242 configures racoon to read the public key of the remote peer from the file
2243 <filename>laptop.public</filename>.
2246 The <command>proposal</command> stanza is unchanged from what we've
2247 seen earlier, with the exception that the
2248 <command>authentication_method</command> is now <command>rsasig</command>,
2249 indicating the use of RSA public/private keys for authentication.
2252 The addition to the configuration of 10.0.0.216 is nearly identical, except for the
2255 path certificate "/usr/local/etc/racoon/certs";
2259 exchange_mode aggressive,main;
2260 my_identifier asn1dn;
2261 peers_identifier asn1dn;
2263 certificate_type x509 "laptop.public" "laptop.private";
2265 peers_certfile "upstairs.public";
2268 encryption_algorithm 3des;
2269 hash_algorithm sha1;
2270 authentication_method rsasig;
2277 Now that we've added these statements to both hosts, we only need to
2278 move the key files in place. The 'upstairs' machine needs
2279 <filename>upstairs.private</filename>, <filename>upstairs.public</filename>,
2280 and <filename>laptop.public</filename> in
2281 <filename>/usr/local/etc/racoon/certs</filename>. Make sure that this
2282 directory is owned by root and has mode 0700 or racoon may refuse to read
2286 The 'laptop' machine needs
2287 <filename>laptop.private</filename>, <filename>laptop.public</filename>,
2288 and <filename>upstairs.public</filename> in
2289 <filename>/usr/local/etc/racoon/certs</filename>. In other words, each host
2290 needs its own public and private key and additionally, the public key of the
2294 Verify that a Security Policy is in place (execute the 'spdadd' lines in
2295 <xref linkend="lartc.ipsec.automatic.keying.example">). Then launch racoon and everything should
2299 <sect3><title>How to setup tunnels securely</title>
2301 To setup secure communications with a remote party, we must exchange
2302 public keys. While the public key does not need to be kept a secret, on the
2303 contrary, it is very important to be sure that it is in fact the unaltered
2304 key. In other words, you need to be certain there is no 'man in the middle'.
2307 To make this easy, OpenSSL provides the 'digest' command:
2309 $ openssl dgst upstairs.public
2310 MD5(upstairs.public)= 78a3bddafb4d681c1ca8ed4d23da4ff1
2314 Now all we need to do is verify if our remote partner sees the same
2315 digest. This might be done by meeting in real life or perhaps over the
2316 phone, making sure the number of the remote party was not in fact sent over
2317 the same email containing the key!
2320 Another way of doing this is the use of a Trusted Third Party which
2321 runs a Certificate Authority. This CA would then sign your key, which we've
2322 done ourselves above.
2329 <sect1 id="lartc.ipsec.tunnel"><title>IPSEC tunnels</title>
2331 So far, we've only seen IPSEC in so called 'transport' mode where both endpoints understand IPSEC directly. As this is often not
2332 the case, it may be necessary to have only routers understand IPSEC, and have them do the work for the hosts behind them.
2333 This is called 'tunnel mode'.
2336 Setting this up is a breeze. To tunnel all traffic to 130.161.0.0/16 from 10.0.0.216 via 10.0.0.11, we issue the following on
2343 add 10.0.0.216 10.0.0.11 esp 34501
2345 -E 3des-cbc "123456789012123456789012";
2347 spdadd 10.0.0.0/24 130.161.0.0/16 any -P out ipsec
2348 esp/tunnel/10.0.0.216-10.0.0.11/require;
2350 Note the '-m tunnel', it is vitally important! This first configures an ESP encryption SA between our tunnel endpoints,
2351 10.0.0.216 and 10.0.0.11.
2354 Next the actual tunnel is configured. It instructs the kernel to encrypt all traffic it has to route from 10.0.0.0/24 to
2355 130.161.0.0. Furthermore, this traffic then has to be shipped to 10.0.0.11.
2358 10.0.0.11 also needs some configuration:
2364 add 10.0.0.216 10.0.0.11 esp 34501
2366 -E 3des-cbc "123456789012123456789012";
2368 spdadd 10.0.0.0/24 130.161.0.0/16 any -P in ipsec
2369 esp/tunnel/10.0.0.216-10.0.0.11/require;
2371 Note that this is exactly identical, except for the change from '-P out' to '-P in'. As with earlier examples,
2372 we've now only configured traffic going one way. Completing the other half of the tunnel is left as an
2373 exercise for the reader.
2376 Another name for this setup is 'proxy ESP', which is somewhat clearer.
2381 The IPSEC tunnel needs to have IP Forwarding enabled in the kernel!
2386 <sect1 id="lartc.ipsec.other"><title>Other IPSEC software</title>
2388 Thomas Walpuski reports that he wrote a patch to make OpenBSD isakpmd work with Linux 2.5 IPSEC.
2389 Furthermore, the main isakpmd CVS repository now contains this code!
2390 Some notes are <ulink
2391 url="http://bender.thinknerd.de/~thomas/IPsec/isakmpd-linux.html">on his page</ulink>.
2395 isakpmd is quite different from racoon mentioned above but many
2396 people like it. It can be found <ulink
2397 url="http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/isakmpd/">here</ulink>.
2398 Read more about OpenBSD CVS <ulink
2399 url="http://www.openbsd.org/anoncvs.html">here</ulink>. Thomas also made a
2401 url="http://bender.thinknerd.de/~thomas/IPsec/isakmpd.tgz">tarball</ulink>
2402 available for those uncomfortable with CVS or patch.
2406 Furthermore, there are patches to make the FreeS/WAN userspace tools
2407 work with the native Linux 2.5 IPSEC, you can find them <ulink
2408 url="http://gondor.apana.org.au/~herbert/freeswan/">here</ulink>.
2410 <sect1 id="lartc.ipsec.interop"><title>IPSEC interoperation with other systems</title>
2414 <sect2 id="lartc.ipsec.interop.win32"><title>Windows</title>
2416 Andreas Jellinghaus <aj@dungeon.inka.de> reports: "win2k: it works. pre_shared key with ip address for authentication (I don't
2417 think windows supports fqdn or userfqdn strings). Certificates should also work, didn't
2423 <sect2 id="lartc.ipsec.interop.checkpoint"><title> Check Point VPN-1
2426 Peter Bieringer reports:
2428 Here are some results (tunnel mode only tested, auth=SHA1):
2433 AES-192: not supported by CP VPN-1
2435 CAST* : not supported by used Linux kernel
2437 Tested version: FP4 aka R54 aka w/AI
2441 More information <ulink
2442 url="http://www.fw-1.de/aerasec/ng/vpn-racoon/CP-VPN1-NG-Linux-racoon.html">here</ulink>.
2448 <chapter id="lartc.multicast">
2449 <Title>Multicast routing</Title>
2452 FIXME: Editor Vacancy!
2456 The Multicast-HOWTO is ancient (relatively-speaking) and may be inaccurate
2457 or misleading in places, for that reason.
2461 Before you can do any multicast routing, you need to configure the Linux
2462 kernel to support the type of multicast routing you want to do. This, in
2463 turn, requires you to decide what type of multicast routing you expect to
2464 be using. There are essentially four "common" types - DVMRP (the Multicast
2465 version of the RIP unicast protocol), MOSPF (the same, but for OSPF), PIM-SM
2466 ("Protocol Independent Multicasting - Sparse Mode", which assumes that users
2467 of any multicast group are spread out, rather than clumped) and PIM-DM (the
2468 same, but "Dense Mode", which assumes that there will be significant clumps
2469 of users of the same multicast group).
2473 In the Linux kernel, you will notice that these options don't appear. This is
2474 because the protocol itself is handled by a routing application, such as
2475 Zebra, mrouted, or pimd. However, you still have to have a good idea of which
2476 you're going to use, to select the right options in the kernel.
2480 For all multicast routing, you will definitely need to enable "multicasting"
2481 and "multicast routing". For DVMRP and MOSPF, this is sufficient. If you are
2482 going to use PIM, you must also enable PIMv1 or PIMv2, depending on whether
2483 the network you are connecting to uses version 1 or 2 of the PIM protocol.
2487 Once you have all that sorted out, and your new Linux kernel compiled, you
2488 will see that the IP protocols listed, at boot time, now include IGMP. This
2489 is a protocol for managing multicast groups. At the time of writing, Linux
2490 supports IGMP versions 1 and 2 only, although version 3 does exist and has
2491 been documented. This doesn't really affect us that much, as IGMPv3 is still
2492 new enough that the extra capabilities of IGMPv3 aren't going to be that
2493 much use. Because IGMP deals with groups, only the features present in the
2494 simplest version of IGMP over the entire group are going to be used. For the
2495 most part, that will be IGMPv2, although IGMPv1 is sill going to be
2500 So far, so good. We've enabled multicasting. Now, we have to tell the Linux
2501 kernel to actually do something with it, so we can start routing. This means
2502 adding the Multicast virtual network to the router table:
2506 ip route add 224.0.0.0/4 dev eth0
2510 (Assuming, of course, that you're multicasting over eth0! Substitute the
2511 device of your choice, for this.)
2515 Now, tell Linux to forward packets...
2519 echo 1 > /proc/sys/net/ipv4/ip_forward
2523 At this point, you may be wondering if this is ever going to do anything. So,
2524 to test our connection, we ping the default group, 224.0.0.1, to see if anyone
2525 is alive. All machines on your LAN with multicasting enabled <Emphasis>should</Emphasis>
2526 respond, but nothing else. You'll notice that none of the machines that
2527 respond have an IP address of 224.0.0.1. What a surprise! :) This is a group
2528 address (a "broadcast" to subscribers), and all members of the group will
2529 respond with their own address, not the group address.
2537 At this point, you're ready to do actual multicast routing. Well, assuming
2538 that you have two networks to route between.
2547 <chapter id="lartc.qdisc">
2548 <Title>Queueing Disciplines for Bandwidth Management</Title>
2551 Now, when I discovered this, it <Emphasis>really</Emphasis> blew me away. Linux 2.2/2.4
2552 comes with everything to manage bandwidth in ways comparable to high-end
2553 dedicated bandwidth management systems.
2557 Linux even goes far beyond what Frame and ATM provide.
2560 <Para>Just to prevent confusion, <command>tc</command> uses the following
2561 rules for bandwith specification:
2563 <literallayout class='monospaced'>
2564 mbps = 1024 kbps = 1024 * 1024 bps => byte/s
2565 mbit = 1024 kbit => kilo bit/s.
2566 mb = 1024 kb = 1024 * 1024 b => byte
2567 mbit = 1024 kbit => kilo bit.
2570 Internally, the number is stored in bps and b.
2573 <Para>But when <command>tc</command> prints the rate, it uses following :
2576 <literallayout class='monospaced'>
2577 1Mbit = 1024 Kbit = 1024 * 1024 bps => byte/s
2580 <Sect1 id="lartc.qdisc.explain">
2581 <Title>Queues and Queueing Disciplines explained</Title>
2584 With queueing we determine the way in which data is <Emphasis>SENT</Emphasis>.
2585 It is important to realise that we can only shape data that we transmit.
2589 With the way the Internet works, we have no direct control of what people
2590 send us. It's a bit like your (physical!) mailbox at home. There is no way
2591 you can influence the world to modify the amount of mail they send you,
2592 short of contacting everybody.
2596 However, the Internet is mostly based on TCP/IP which has a few features
2597 that help us. TCP/IP has no way of knowing the capacity of the network
2598 between two hosts, so it just starts sending data faster and faster ('slow
2599 start') and when packets start getting lost, because there is no room to
2600 send them, it will slow down. In fact it is a bit smarter than this, but
2601 more about that later.
2605 This is the equivalent of not reading half of your mail, and hoping that
2606 people will stop sending it to you. With the difference that it works for
2611 If you have a router and wish to prevent certain hosts within your network
2612 from downloading too fast, you need to do your shaping on the *inner* interface
2613 of your router, the one that sends data to your own computers.
2617 You also have to be sure you are controlling the bottleneck of the link.
2618 If you have a 100Mbit NIC and you have a router that has a 256kbit link,
2619 you have to make sure you are not sending more data than your router can
2620 handle. Otherwise, it will be the router who is controlling the link and
2621 shaping the available bandwith. We need to 'own the queue' so to speak, and
2622 be the slowest link in the chain. Luckily this is easily possible.
2627 <Sect1 id="lartc.qdisc.classless">
2628 <Title>Simple, classless Queueing Disciplines</Title>
2631 As said, with queueing disciplines, we change the way data is sent.
2632 Classless queueing disciplines are those that, by and large accept data and
2633 only reschedule, delay or drop it.
2637 These can be used to shape traffic for an entire interface, without any
2638 subdivisions. It is vital that you understand this part of queueing before
2639 we go on the classful qdisc-containing-qdiscs!
2643 By far the most widely used discipline is the pfifo_fast qdisc - this is the
2644 default. This also explains why these advanced features are so robust. They
2645 are nothing more than 'just another queue'.
2649 Each of these queues has specific strengths and weaknesses. Not all of them
2650 may be as well tested.
2654 <Title>pfifo_fast</Title>
2657 This queue is, as the name says, First In, First Out, which means that no
2658 packet receives special treatment. At least, not quite. This queue has 3 so
2659 called 'bands'. Within each band, FIFO rules apply. However, as long as
2660 there are packets waiting in band 0, band 1 won't be processed. Same goes
2661 for band 1 and band 2.
2665 The kernel honors the so called Type of Service flag of packets, and takes
2666 care to insert 'minimum delay' packets in band 0.
2670 Do not confuse this classless simple qdisc with the classful PRIO one!
2671 Although they behave similarly, pfifo_fast is classless and you cannot add
2672 other qdiscs to it with the tc command.
2676 <Title>Parameters & usage</Title>
2679 You can't configure the pfifo_fast qdisc as it is the hardwired default.
2680 This is how it is configured by default:
2684 <Term>priomap</Term>
2687 Determines how packet priorities, as assigned by the kernel, map to bands.
2688 Mapping occurs based on the TOS octet of the packet, which looks like this:
2695 +-----+-----+-----+-----+-----+-----+-----+-----+
2697 | PRECEDENCE | TOS | MBZ |
2699 +-----+-----+-----+-----+-----+-----+-----+-----+
2705 The four TOS bits (the 'TOS field') are defined as:
2708 Binary Decimcal Meaning
2709 -----------------------------------------
2710 1000 8 Minimize delay (md)
2711 0100 4 Maximize throughput (mt)
2712 0010 2 Maximize reliability (mr)
2713 0001 1 Minimize monetary cost (mmc)
2714 0000 0 Normal Service
2720 As there is 1 bit to the right of these four bits, the actual value of the
2721 TOS field is double the value of the TOS bits. Tcpdump -v -v shows you the
2722 value of the entire TOS field, not just the four bits. It is the value you
2723 see in the first column of this table:
2729 TOS Bits Means Linux Priority Band
2730 ------------------------------------------------------------
2731 0x0 0 Normal Service 0 Best Effort 1
2732 0x2 1 Minimize Monetary Cost 1 Filler 2
2733 0x4 2 Maximize Reliability 0 Best Effort 1
2734 0x6 3 mmc+mr 0 Best Effort 1
2735 0x8 4 Maximize Throughput 2 Bulk 2
2736 0xa 5 mmc+mt 2 Bulk 2
2737 0xc 6 mr+mt 2 Bulk 2
2738 0xe 7 mmc+mr+mt 2 Bulk 2
2739 0x10 8 Minimize Delay 6 Interactive 0
2740 0x12 9 mmc+md 6 Interactive 0
2741 0x14 10 mr+md 6 Interactive 0
2742 0x16 11 mmc+mr+md 6 Interactive 0
2743 0x18 12 mt+md 4 Int. Bulk 1
2744 0x1a 13 mmc+mt+md 4 Int. Bulk 1
2745 0x1c 14 mr+mt+md 4 Int. Bulk 1
2746 0x1e 15 mmc+mr+mt+md 4 Int. Bulk 1
2752 Lots of numbers. The second column contains the value of the relevant four
2753 TOS bits, followed by their translated meaning. For example, 15 stands for a
2754 packet wanting Minimal Monetary Cost, Maximum Reliability, Maximum
2755 Throughput AND Minimum Delay. I would call this a 'Dutch Packet'.
2759 The fourth column lists the way the Linux kernel interprets the TOS bits, by
2760 showing to which Priority they are mapped.
2764 The last column shows the result of the default priomap. On the command line,
2765 the default priomap looks like this:
2768 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
2774 This means that priority 4, for example, gets mapped to band number 1. The
2775 priomap also allows you to list higher priorities (> 7) which do not
2776 correspond to TOS mappings, but which are set by other means.
2780 This table from RFC 1349 (read it for more details) tells you how
2781 applications might very well set their TOS bits:
2784 TELNET 1000 (minimize delay)
2786 Control 1000 (minimize delay)
2787 Data 0100 (maximize throughput)
2789 TFTP 1000 (minimize delay)
2792 Command phase 1000 (minimize delay)
2793 DATA phase 0100 (maximize throughput)
2796 UDP Query 1000 (minimize delay)
2798 Zone Transfer 0100 (maximize throughput)
2800 NNTP 0001 (minimize monetary cost)
2804 Requests 0000 (mostly)
2805 Responses <same as request> (mostly)
2811 <Term>txqueuelen</Term>
2814 The length of this queue is gleaned from the interface configuration, which
2815 you can see and set with ifconfig and ip. To set the queue length to 10,
2816 execute: ifconfig eth0 txqueuelen 10
2820 You can't set this parameter with tc!
2831 <Title>Token Bucket Filter</Title>
2834 The Token Bucket Filter (TBF) is a simple qdisc that only passes packets
2835 arriving at a rate which is not exceeding some administratively set rate, but
2836 with the possibility to allow short bursts in excess of this rate.
2840 TBF is very precise, network- and processor friendly. It should be your
2841 first choice if you simply want to slow an interface down!
2845 The TBF implementation consists of a buffer (bucket), constantly filled by
2846 some virtual pieces of information called tokens, at a specific rate (token
2847 rate). The most important parameter of the bucket is its size, that is the
2848 number of tokens it can store.
2852 Each arriving token collects one incoming data packet from the data queue
2853 and is then deleted from the bucket. Associating this algorithm
2854 with the two flows -- token and data, gives us three possible scenarios:
2863 The data arrives in TBF at a rate that's <Emphasis>equal</Emphasis> to the rate
2864 of incoming tokens. In this case each incoming packet has its matching token
2865 and passes the queue without delay.
2872 The data arrives in TBF at a rate that's <Emphasis>smaller</Emphasis> than the
2873 token rate. Only a part of the tokens are deleted at output of each data packet
2874 that's sent out the queue, so the tokens accumulate, up to the bucket size.
2875 The unused tokens can then be used to send data at a speed that's exceeding the
2876 standard token rate, in case short data bursts occur.
2883 The data arrives in TBF at a rate <Emphasis>bigger</Emphasis> than the token rate.
2884 This means that the bucket will soon be devoid of tokens, which causes the
2885 TBF to throttle itself for a while. This is called an 'overlimit situation'.
2886 If packets keep coming in, packets will start to get dropped.
2895 The last scenario is very important, because it allows to
2896 administratively shape the bandwidth available to data that's passing
2901 The accumulation of tokens allows a short burst of overlimit data to be
2902 still passed without loss, but any lasting overload will cause packets to be
2903 constantly delayed, and then dropped.
2907 Please note that in the actual implementation, tokens correspond to bytes,
2912 <Title>Parameters & usage</Title>
2915 Even though you will probably not need to change them, tbf has some knobs
2916 available. First the parameters that are always available:
2920 <Term>limit or latency</Term>
2923 Limit is the number of bytes that can be queued waiting for tokens to become
2924 available. You can also specify this the other way around by setting the
2925 latency parameter, which specifies the maximum amount of time a packet can
2926 sit in the TBF. The latter calculation takes into account the size of the
2927 bucket, the rate and possibly the peakrate (if set).
2931 <Term>burst/buffer/maxburst</Term>
2934 Size of the bucket, in bytes. This is the maximum amount of bytes that
2935 tokens can be available for instantaneously. In general, larger shaping
2936 rates require a larger buffer. For 10mbit/s on Intel, you need at least
2937 10kbyte buffer if you want to reach your configured rate!
2941 If your buffer is too small, packets may be dropped because more tokens
2942 arrive per timer tick than fit in your bucket.
2949 A zero-sized packet does not use zero bandwidth. For ethernet, no packet
2950 uses less than 64 bytes. The Minimum Packet Unit determines the minimal
2951 token usage for a packet.
2958 The speedknob. See remarks above about limits!
2965 If the bucket contains tokens and is allowed to empty, by default it does so
2966 at infinite speed. If this is unacceptable, use the following parameters:
2973 <Term>peakrate</Term>
2976 If tokens are available, and packets arrive, they are sent out immediately
2977 by default, at 'lightspeed' so to speak. That may not be what you want,
2978 especially if you have a large bucket.
2982 The peakrate can be used to specify how quickly the bucket is allowed to be
2983 depleted. If doing everything by the book, this is achieved by releasing a
2984 packet, and then wait just long enough, and release the next. We calculated
2985 our waits so we send just at peakrate.
2989 However, due to the default 10ms timer resolution of Unix, with 10.000 bits
2990 average packets, we are limited to 1mbit/s of peakrate!
2994 <Term>mtu/minburst</Term>
2997 The 1mbit/s peakrate is not very useful if your regular rate is more than
2998 that. A higher peakrate is possible by sending out more packets per
2999 timertick, which effectively means that we create a second bucket!
3003 This second bucket defaults to a single packet, which is not a bucket at
3008 To calculate the maximum possible peakrate, multiply the configured mtu by
3009 100 (or more correctly, HZ, which is 100 on Intel, 1024 on Alpha).
3018 <Title>Sample configuration</Title>
3021 A simple but *very* useful configuration is this:
3024 # tc qdisc add dev ppp0 root tbf rate 220kbit latency 50ms burst 1540
3030 Ok, why is this useful? If you have a networking device with a large queue,
3031 like a DSL modem or a cable modem, and you talk to it over a fast device,
3032 like over an ethernet interface, you will find that uploading absolutely
3033 destroys interactivity.
3037 This is because uploading will fill the queue in the modem, which is
3038 probably *huge* because this helps actually achieving good data throughput
3039 uploading. But this is not what you want, you want to have the queue not too
3040 big so interactivity remains and you can still do other stuff while sending
3045 The line above slows down sending to a rate that does not lead to a queue in
3046 the modem - the queue will be in Linux, where we can control it to a limited
3051 Change 220kbit to your uplink's *actual* speed, minus a few percent. If you
3052 have a really fast modem, raise 'burst' a bit.
3059 <Sect2 id="lartc.sfq">
3060 <Title>Stochastic Fairness Queueing</Title>
3063 Stochastic Fairness Queueing (SFQ) is a simple implementation of the fair
3064 queueing algorithms family. It's less accurate than others, but it also
3065 requires less calculations while being almost perfectly fair.
3069 The key word in SFQ is conversation (or flow), which mostly corresponds to a
3070 TCP session or a UDP stream. Traffic is divided into a pretty large number
3071 of FIFO queues, one for each conversation. Traffic is then sent in a round
3072 robin fashion, giving each session the chance to send data in turn.
3076 This leads to very fair behaviour and disallows any single conversation from
3077 drowning out the rest. SFQ is called 'Stochastic' because it doesn't really
3078 allocate a queue for each session, it has an algorithm which divides traffic
3079 over a limited number of queues using a hashing algorithm.
3083 Because of the hash, multiple sessions might end up in the same bucket, which
3084 would halve each session's chance of sending a packet, thus halving the
3085 effective speed available. To prevent this situation from becoming
3086 noticeable, SFQ changes its hashing algorithm quite often so that any two
3087 colliding sessions will only do so for a small number of seconds.
3091 It is important to note that SFQ is only useful in case your actual outgoing
3092 interface is really full! If it isn't then there will be no queue on your
3093 linux machine and hence no effect. Later on we will describe how to combine
3094 SFQ with other qdiscs to get a best-of-both worlds situation.
3098 Specifically, setting SFQ on the ethernet interface heading to your
3099 cable modem or DSL router is pointless without further shaping!
3103 <Title>Parameters & usage</Title>
3106 The SFQ is pretty much self tuning:
3110 <Term>perturb</Term>
3113 Reconfigure hashing once this many seconds. If unset, hash will never be
3114 reconfigured. Not recommended. 10 seconds is probably a good value.
3118 <Term>quantum</Term>
3121 Amount of bytes a stream is allowed to dequeue before the next queue gets a
3122 turn. Defaults to 1 maximum sized packet (MTU-sized). Do not set below the
3130 The total number of packets that will be queued by this SFQ (after that it
3131 starts dropping them).
3141 <Title>Sample configuration</Title>
3144 If you have a device which has identical link speed and actual available
3145 rate, like a phone modem, this configuration will help promote fairness:
3148 # tc qdisc add dev ppp0 root sfq perturb 10
3150 qdisc sfq 800c: dev ppp0 quantum 1514b limit 128p flows 128/1024 perturb 10sec
3151 Sent 4812 bytes 62 pkts (dropped 0, overlimits 0)
3157 The number 800c: is the automatically assigned handle number, limit means
3158 that 128 packets can wait in this queue. There are 1024 hashbuckets
3159 available for accounting, of which 128 can be active at a time (no more
3160 packets fit in the queue!) Once every 10 seconds, the hashes are
3170 <Sect1 id="lartc.qdisc.advice">
3171 <Title>Advice for when to use which queue</Title>
3174 Summarizing, these are the simple queues that actually manage traffic by
3175 reordering, slowing or dropping packets.
3179 The following tips may help in choosing which queue to use. It mentions some
3180 qdiscs described in the
3181 <citetitle><xref linkend="lartc.adv-qdisc"></citetitle> chapter.
3187 To purely slow down outgoing traffic, use the Token Bucket Filter. Works up
3188 to huge bandwidths, if you scale the bucket.
3194 If your link is truly full and you want to make sure that no single session
3195 can dominate your outgoing bandwidth, use Stochastical Fairness Queueing.
3201 If you have a big backbone and know what you are doing, consider Random
3202 Early Drop (see Advanced chapter).
3208 To 'shape' incoming traffic which you are not forwarding, use the Ingress
3209 Policer. Incoming shaping is called 'policing', by the way, not 'shaping'.
3215 If you *are* forwarding it, use a TBF on the interface you are forwarding
3216 the data to. Unless you want to shape traffic that may go out over several
3217 interfaces, in which case the only common factor is the incoming interface.
3218 In that case use the Ingress Policer.
3224 If you don't want to shape, but only want to see if your interface is so
3225 loaded that it has to queue, use the pfifo queue (not pfifo_fast). It lacks
3226 internal bands but does account the size of its backlog.
3231 Finally - you can also do <quote>social shaping</quote>.
3232 You may not always be able to use technology to achieve what you want.
3233 Users experience technical constraints as hostile.
3234 A kind word may also help with getting your bandwidth to be divided right!
3241 <Sect1 id="lartc.qdisc.terminology">
3242 <Title>Terminology</Title>
3245 To properly understand more complicated configurations it is necessary to
3246 explain a few concepts first. Because of the complexity and the relative
3247 youth of the subject, a lot of different words are used when people in fact
3248 mean the same thing.
3252 The following is loosely based on
3253 <filename>draft-ietf-diffserv-model-06.txt</filename>,
3254 <citetitle>An Informal Management Model for Diffserv Routers</citetitle>.
3255 It can currently be found at
3256 <ulink url="http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt">
3257 http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt
3262 Read it for the strict definitions of the terms used.
3266 <Term>Queueing Discipline (qdisc)</Term>
3269 An algorithm that manages the queue of a device, either incoming (ingress)
3270 or outgoing (egress).
3274 <Term>root qdisc</Term>
3277 The root qdisc is the qdisc attached to the device.
3281 <Term>Classless qdisc</Term>
3284 A qdisc with no configurable internal subdivisions.
3288 <Term>Classful qdisc</Term>
3291 A classful qdisc contains multiple classes. Some of these classes contains a
3292 further qdisc, which may again be classful, but need not be. According to
3293 the strict definition, pfifo_fast *is* classful, because it contains three
3294 bands which are, in fact, classes. However, from the user's configuration
3295 perspective, it is classless as the classes can't be touched with the tc
3300 <Term>Classes</Term>
3303 A classful qdisc may have many classes, each of which is internal to the
3304 qdisc. A class, in turn, may have several classes added to it. So a class
3305 can have a qdisc as parent or an other class.
3307 A leaf class is a class with no child classes. This class has 1 qdisc attached
3308 to it. This qdisc is responsible to send the data from that class. When
3309 you create a class, a fifo qdisc is attached to it. When you add a child class,
3310 this qdisc is removed.
3311 For a leaf class, this fifo qdisc can be replaced with
3312 an other more suitable qdisc. You can even replace this fifo qdisc with a
3313 classful qdisc so you can add extra classes.
3317 <Term>Classifier</Term>
3320 Each classful qdisc needs to determine to which class it needs to send a
3321 packet. This is done using the classifier.
3328 Classification can be performed using filters. A filter contains a number of
3329 conditions which if matched, make the filter match.
3333 <Term>Scheduling</Term>
3336 A qdisc may, with the help of a classifier, decide that some packets need to
3337 go out earlier than others. This process is called Scheduling, and is
3338 performed for example by the pfifo_fast qdisc mentioned earlier. Scheduling
3339 is also called 'reordering', but this is confusing.
3343 <Term>Shaping</Term>
3346 The process of delaying packets before they go out to make traffic confirm
3347 to a configured maximum rate. Shaping is performed on egress. Colloquially,
3348 dropping packets to slow traffic down is also often called Shaping.
3352 <Term>Policing</Term>
3355 Delaying or dropping packets in order to make traffic stay below a
3356 configured bandwidth. In Linux, policing can only drop a packet and not
3357 delay it - there is no 'ingress queue'.
3361 <Term>Work-Conserving</Term>
3364 A work-conserving qdisc always delivers a packet if one is available. In
3365 other words, it never delays a packet if the network adaptor is ready to
3366 send one (in the case of an egress qdisc).
3370 <Term>non-Work-Conserving</Term>
3373 Some queues, like for example the Token Bucket Filter, may need to hold on
3374 to a packet for a certain time in order to limit the bandwidth. This means
3375 that they sometimes refuse to pass a packet, even though they have one
3383 Now that we have our terminology straight, let's see where all these things
3393 +---------------+-----------------------------------------+
3395 | -------> IP Stack |
3400 | | / ----------> Forwarding -> |
3405 | | Egress /--qdisc2--\ |
3406 --->->Ingress Classifier ---qdisc3---- | ->
3407 | Qdisc \__qdisc4__/ |
3410 +----------------------------------------------------------+
3413 Thanks to Jamal Hadi Salim for this ASCII representation.
3417 The big block represents the kernel. The leftmost arrow represents traffic
3418 entering your machine from the network. It is then fed to the Ingress
3419 Qdisc which may apply Filters to a packet, and decide to drop it. This
3420 is called 'Policing'.
3424 This happens at a very early stage, before it has seen a lot of the kernel.
3425 It is therefore a very good place to drop traffic very early, without
3426 consuming a lot of CPU power.
3430 If the packet is allowed to continue, it may be destined for a local
3431 application, in which case it enters the IP stack in order to be processed,
3432 and handed over to a userspace program. The packet may also be forwarded
3433 without entering an application, in which case it is destined for egress.
3434 Userspace programs may also deliver data, which is then examined and
3435 forwarded to the Egress Classifier.
3439 There it is investigated and enqueued to any of a number of qdiscs. In the
3440 unconfigured default case, there is only one egress qdisc installed, the
3441 pfifo_fast, which always receives the packet. This is called 'enqueueing'.
3445 The packet now sits in the qdisc, waiting for the kernel to ask for
3446 it for transmission over the network interface. This is called 'dequeueing'.
3450 This picture also holds in case there is only one network adaptor - the
3451 arrows entering and leaving the kernel should not be taken too literally.
3452 Each network adaptor has both ingress and egress hooks.
3457 <Sect1 id="lartc.qdisc.classful">
3458 <Title>Classful Queueing Disciplines</Title>
3461 Classful qdiscs are very useful if you have different kinds of traffic which
3462 should have differing treatment. One of the classful qdiscs is called 'CBQ',
3463 'Class Based Queueing' and it is so widely mentioned that people identify
3464 queueing with classes solely with CBQ, but this is not the case.
3468 CBQ is merely the oldest kid on the block - and also the most complex one.
3469 It may not always do what you want. This may come as something of a shock
3470 to many who fell for the 'sendmail effect', which teaches us that any
3471 complex technology which doesn't come with documentation must be the best
3476 More about CBQ and its alternatives shortly.
3480 <Title>Flow within classful qdiscs & classes</Title>
3483 When traffic enters a classful qdisc, it needs to be sent to any of the
3484 classes within - it needs to be 'classified'. To determine what to do with a
3485 packet, the so called 'filters' are consulted. It is important to know that
3486 the filters are called from within a qdisc, and not the other way around!
3490 The filters attached to that qdisc then return with a decision, and the
3491 qdisc uses this to enqueue the packet into one of the classes. Each subclass
3492 may try other filters to see if further instructions apply. If not, the
3493 class enqueues the packet to the qdisc it contains.
3497 Besides containing other qdiscs, most classful qdiscs also perform shaping.
3498 This is useful to perform both packet scheduling (with SFQ, for example) and
3499 rate control. You need this in cases where you have a high speed
3500 interface (for example, ethernet) to a slower device (a cable modem).
3504 If you were only to run SFQ, nothing would happen, as packets enter &
3505 leave your router without delay: the output interface is far faster than
3506 your actual link speed. There is no queue to schedule then.
3512 <Title>The qdisc family: roots, handles, siblings and parents</Title>
3515 Each interface has one egress 'root qdisc'. By default, it is the earlier mentioned
3516 classless pfifo_fast queueing discipline. Each qdisc and class is assigned a
3517 handle, which can be used by later configuration statements to refer to that
3518 qdisc. Besides an egress qdisc, an interface may also have an ingress qdisc ,
3519 which polices traffic coming in.
3523 The handles of these qdiscs consist of two parts, a major number and a minor
3524 number : <major>:<minor>. It is customary to name the root qdisc '1:', which
3525 is equal to '1:0'. The minor number of a qdisc is always 0.
3529 Classes need to have the same major number as their parent. This major number
3530 must be unique within a egress or ingress setup. The minor number must be
3531 unique within a qdisc and his classes.
3535 <Title>How filters are used to classify traffic </Title>
3538 Recapping, a typical hierarchy might look like this:
3548 1:10 1:11 1:12 child classes
3554 10:1 10:2 12:1 12:2 leaf classes
3560 But don't let this tree fool you! You should *not* imagine the kernel to be
3561 at the apex of the tree and the network below, that is just not the case.
3562 Packets get enqueued and dequeued at the root qdisc, which is the only thing
3563 the kernel talks to.
3567 A packet might get classified in a chain like this:
3571 1: -> 1:1 -> 1:12 -> 12: -> 12:2
3575 The packet now resides in a queue in a qdisc attached to class 12:2. In this
3576 example, a filter was attached to each 'node' in the tree, each choosing a
3577 branch to take next. This can make sense. However, this is also possible:
3585 In this case, a filter attached to the root decided to send the packet
3592 <Title>How packets are dequeued to the hardware</Title>
3595 When the kernel decides that it needs to extract packets to send to the
3596 interface, the root qdisc 1: gets a dequeue request, which is passed to
3597 1:1, which is in turn passed to 10:, 11: and 12:, each of which queries its
3598 siblings, and tries to dequeue() from them. In this case, the kernel needs to
3599 walk the entire tree, because only 12:2 contains a packet.
3603 In short, nested classes ONLY talk to their parent qdiscs, never to an
3604 interface. Only the root qdisc gets dequeued by the kernel!
3608 The upshot of this is that classes never get dequeued faster than their
3609 parents allow. And this is exactly what we want: this way we can have SFQ in
3610 an inner class, which doesn't do any shaping, only scheduling, and have a
3611 shaping outer qdisc, which does the shaping.
3619 <Title>The PRIO qdisc</Title>
3622 The PRIO qdisc doesn't actually shape, it only subdivides traffic based on
3623 how you configured your filters. You can consider the PRIO qdisc a kind
3624 of pfifo_fast on steroids, whereby each band is a separate class instead of
3629 When a packet is enqueued to the PRIO qdisc, a class is chosen based on the
3630 filter commands you gave. By default, three classes are created. These
3631 classes by default contain pure FIFO qdiscs with no internal
3632 structure, but you can replace these by any qdisc you have available.
3636 Whenever a packet needs to be dequeued, class :1 is tried first. Higher
3637 classes are only used if lower bands all did not give up a packet.
3641 This qdisc is very useful in case you want to prioritize certain kinds of
3642 traffic without using only TOS-flags but using all the power of the tc
3643 filters. You can also add an other qdisc to the 3 predefined classes,
3644 whereas pfifo_fast is limited to simple fifo qdiscs.
3648 Because it doesn't actually shape, the same warning as for SFQ holds: either
3649 use it only if your physical link is really full or wrap it inside a
3650 classful qdisc that does shape. The latter holds for almost all cable modems
3655 In formal words, the PRIO qdisc is a Work-Conserving scheduler.
3659 <Title>PRIO parameters & usage</Title>
3662 The following parameters are recognized by tc:
3669 Number of bands to create. Each band is in fact a class. If you change this
3670 number, you must also change:
3674 <Term>priomap</Term>
3677 If you do not provide tc filters to classify traffic, the PRIO qdisc looks
3678 at the TC_PRIO priority to decide how to enqueue traffic.
3682 This works just like with the pfifo_fast qdisc mentioned earlier, see there
3687 The bands are classes, and are called major:1 to major:3 by default, so if
3688 your PRIO qdisc is called 12:, tc filter traffic to 12:1 to grant it more
3693 Reiterating, band 0 goes to minor number 1! Band 1 to minor number 2, etc.
3699 <Title>Sample configuration</Title>
3702 We will create this tree:
3711 10: 20: 30: qdiscs qdiscs
3719 Bulk traffic will go to 30:, interactive traffic to 20: or 10:.
3726 # tc qdisc add dev eth0 root handle 1: prio
3727 ## This *instantly* creates classes 1:1, 1:2, 1:3
3729 # tc qdisc add dev eth0 parent 1:1 handle 10: sfq
3730 # tc qdisc add dev eth0 parent 1:2 handle 20: tbf rate 20kbit buffer 1600 limit 3000
3731 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
3737 Now let's see what we created:
3740 # tc -s qdisc ls dev eth0
3741 qdisc sfq 30: quantum 1514b
3742 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
3744 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3745 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
3747 qdisc sfq 10: quantum 1514b
3748 Sent 132 bytes 2 pkts (dropped 0, overlimits 0)
3750 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3751 Sent 174 bytes 3 pkts (dropped 0, overlimits 0)
3754 As you can see, band 0 has already had some traffic, and one packet was sent
3755 while running this command!
3759 We now do some bulk data transfer with a tool that properly sets TOS flags,
3760 and take another look:
3763 # scp tc ahu@10.0.0.11:./
3764 ahu@10.0.0.11's password:
3765 tc 100% |*****************************| 353 KB 00:00
3766 # tc -s qdisc ls dev eth0
3767 qdisc sfq 30: quantum 1514b
3768 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
3770 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3771 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
3773 qdisc sfq 10: quantum 1514b
3774 Sent 2230 bytes 31 pkts (dropped 0, overlimits 0)
3776 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3777 Sent 389140 bytes 326 pkts (dropped 0, overlimits 0)
3780 As you can see, all traffic went to handle 30:, which is the lowest priority
3781 band, just as intended. Now to verify that interactive traffic goes to
3782 higher bands, we create some interactive traffic:
3788 # tc -s qdisc ls dev eth0
3789 qdisc sfq 30: quantum 1514b
3790 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
3792 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
3793 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
3795 qdisc sfq 10: quantum 1514b
3796 Sent 14926 bytes 193 pkts (dropped 0, overlimits 0)
3798 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
3799 Sent 401836 bytes 488 pkts (dropped 0, overlimits 0)
3805 It worked - all additional traffic has gone to 10:, which is our highest
3806 priority qdisc. No traffic was sent to the lowest priority, which previously
3807 received our entire scp.
3815 <Title>The famous CBQ qdisc</Title>
3818 As said before, CBQ is the most complex qdisc available, the most hyped, the
3819 least understood, and probably the trickiest one to get right. This is not
3820 because the authors are evil or incompetent, far from it, it's just that the
3821 CBQ algorithm isn't all that precise and doesn't really match the way Linux
3826 Besides being classful, CBQ is also a shaper and it is in that aspect that
3827 it really doesn't work very well. It should work like this. If you try to
3828 shape a 10mbit/s connection to 1mbit/s, the link should be idle 90% of the
3829 time. If it isn't, we need to throttle so that it IS idle 90% of the time.
3833 This is pretty hard to measure, so CBQ instead derives the idle time from
3834 the number of microseconds that elapse between requests from the hardware
3835 layer for more data. Combined, this can be used to approximate how full or
3836 empty the link is.<!--which is combined with which?-->
3840 This is rather tortuous and doesn't always arrive at proper results. For
3841 example, what if the actual link speed of an interface that is not really
3842 able to transmit the full 100mbit/s of data, perhaps because of a badly
3843 implemented driver? A PCMCIA network card will also never achieve 100mbit/s
3844 because of the way the bus is designed - again, how do we calculate the idle
3849 It gets even worse if we consider not-quite-real network devices like PPP
3850 over Ethernet or PPTP over TCP/IP. The effective bandwidth in that case is
3851 probably determined by the efficiency of pipes to userspace - which is huge.
3855 People who have done measurements discover that CBQ is not always very
3856 accurate and sometimes completely misses the mark.
3860 In many circumstances however it works well. With the documentation provided
3861 here, you should be able to configure it to work well in most cases.
3865 <Title>CBQ shaping in detail</Title>
3868 As said before, CBQ works by making sure that the link is idle just long
3869 enough to bring down the real bandwidth to the configured rate. To do so, it
3870 calculates the time that should pass between average packets.
3874 During operations, the effective idletime is measured using an exponential
3875 weighted moving average (EWMA), which considers recent packets to be
3876 exponentially more important than past ones. The UNIX loadaverage is
3877 calculated in the same way.
3881 The calculated idle time is subtracted from the EWMA measured one, the
3882 resulting number is called 'avgidle'. A perfectly loaded link has an avgidle
3883 of zero: packets arrive exactly once every calculated interval.
3887 An overloaded link has a negative avgidle and if it gets too negative, CBQ
3888 shuts down for a while and is then 'overlimit'.
3892 Conversely, an idle link might amass a huge avgidle, which would then allow
3893 infinite bandwidths after a few hours of silence. To prevent this, avgidle is
3898 If overlimit, in theory, the CBQ could throttle itself for exactly the
3899 amount of time that was calculated to pass between packets, and then pass
3900 one packet, and throttle again. But see the 'minburst' parameter below.
3904 These are parameters you can specify in order to configure shaping:
3911 Average size of a packet, measured in bytes. Needed for calculating maxidle,
3912 which is derived from maxburst, which is specified in packets.
3916 <Term>bandwidth</Term>
3919 The physical bandwidth of your device, needed for idle time
3927 The time a packet takes to be transmitted over a device may grow in steps,
3928 based on the packet size. An 800 and an 806 size packet may take just as long
3929 to send, for example - this sets the granularity. Most often set to '8'.
3930 Must be an integral power of two.
3934 <Term>maxburst</Term>
3937 This number of packets is used to calculate maxidle so that when avgidle is
3938 at maxidle, this number of average packets can be burst before avgidle drops
3939 to 0. Set it higher to be more tolerant of bursts. You can't set maxidle
3940 directly, only via this parameter.
3944 <Term>minburst</Term>
3947 As mentioned before, CBQ needs to throttle in case of overlimit. The ideal
3948 solution is to do so for exactly the calculated idle time, and pass 1
3949 packet. For Unix kernels, however, it is generally hard to schedule events
3950 shorter than 10ms, so it is better to throttle for a longer period, and then
3951 pass minburst packets in one go, and then sleep minburst times longer.
3955 The time to wait is called the offtime. Higher values of minburst lead to
3956 more accurate shaping in the long term, but to bigger bursts at millisecond
3961 <Term>minidle</Term>
3964 If avgidle is below 0, we are overlimits and need to wait until avgidle will
3965 be big enough to send one packet. To prevent a sudden burst from shutting
3966 down the link for a prolonged period of time, avgidle is reset to minidle if
3971 Minidle is specified in negative microseconds, so 10 means that avgidle is
3979 Minimum packet size - needed because even a zero size packet is padded
3980 to 64 bytes on ethernet, and so takes a certain time to transmit. CBQ needs
3981 to know this to accurately calculate the idle time.
3988 Desired rate of traffic leaving this qdisc - this is the 'speed knob'!
3995 Internally, CBQ has a lot of fine tuning. For example, classes which are
3996 known not to have data enqueued to them aren't queried. Overlimit classes
3997 are penalized by lowering their effective priority. All very smart &
4004 <Title>CBQ classful behaviour</Title>
4007 Besides shaping, using the aforementioned idletime approximations, CBQ also
4008 acts like the PRIO queue in the sense that classes can have differing
4009 priorities and that lower priority numbers will be polled before the higher
4014 Each time a packet is requested by the hardware layer to be sent out to the
4015 network, a weighted round robin process ('WRR') starts, beginning with the
4016 lower-numbered priority classes.
4020 These are then grouped and queried if they have data available. If so, it is
4021 returned. After a class has been allowed to dequeue a number of bytes, the
4022 next class within that priority is tried.
4026 The following parameters control the WRR process:
4033 When the outer CBQ is asked for a packet to send out on the interface, it
4034 will try all inner qdiscs (in the classes) in turn, in order of
4035 the 'priority' parameter. Each time a class gets its turn, it can only send out
4036 a limited amount of data. 'Allot' is the base unit of this amount. See
4037 the 'weight' parameter for more information.
4044 The CBQ can also act like the PRIO device. Inner classes with higher priority
4045 are tried first and as long as they have traffic, other classes are not
4049 <!--It is rather confusing between high/low "priority" and
4050 "priorigy number" around here.
4051 How about using large/small for the latter?-->
4056 Weight helps in the Weighted Round Robin process. Each class gets a chance
4057 to send in turn. If you have classes with significantly more bandwidth than
4058 other classes, it makes sense to allow them to send more data in one round
4063 A CBQ adds up all weights under a class, and normalizes them, so you can use
4064 arbitrary numbers: only the ratios are important. People have been
4065 using 'rate/10' as a rule of thumb and it appears to work well. The renormalized
4066 weight is multiplied by the 'allot' parameter to determine how much data can
4067 be sent in one round.
4074 Please note that all classes within an CBQ hierarchy need to share the same
4081 <Title>CBQ parameters that determine link sharing & borrowing</Title>
4084 Besides purely limiting certain kinds of traffic, it is also possible to
4085 specify which classes can borrow capacity from other classes or, conversely,
4093 <Term>Isolated/sharing</Term>
4096 A class that is configured with 'isolated' will not lend out bandwidth to
4097 sibling classes. Use this if you have competing or mutually-unfriendly
4098 agencies on your link who do not want to give each other freebies.
4102 The control program tc also knows about 'sharing', which is the reverse
4107 <Term>bounded/borrow</Term>
4110 A class can also be 'bounded', which means that it will not try to borrow
4111 bandwidth from sibling classes. tc also knows about 'borrow', which is the
4112 reverse of 'bounded'.
4116 A typical situation might be where you have two agencies on your link which
4117 are both 'isolated' and 'bounded', which means that they are really limited
4118 to their assigned rate, and also won't allow each other to borrow.
4122 Within such an agency class, there might be other classes which are allowed
4129 <Title>Sample configuration</Title>
4137 1:3 1:4 leaf classes
4144 This configuration limits webserver traffic to 5mbit and SMTP traffic to 3
4145 mbit. Together, they may not get more than 6mbit. We have a 100mbit NIC and
4146 the classes may borrow bandwidth from each other.
4149 # tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 100Mbit \
4151 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 100Mbit \
4152 rate 6Mbit weight 0.6Mbit prio 8 allot 1514 cell 8 maxburst 20 \
4156 This part installs the root and the customary 1:1 class. The 1:1 class is
4157 bounded, so the total bandwidth can't exceed 6mbit.
4161 As said before, CBQ requires a *lot* of knobs. All parameters are explained
4162 above, however. The corresponding HTB configuration is lots simpler.
4168 # tc class add dev eth0 parent 1:1 classid 1:3 cbq bandwidth 100Mbit \
4169 rate 5Mbit weight 0.5Mbit prio 5 allot 1514 cell 8 maxburst 20 \
4171 # tc class add dev eth0 parent 1:1 classid 1:4 cbq bandwidth 100Mbit \
4172 rate 3Mbit weight 0.3Mbit prio 5 allot 1514 cell 8 maxburst 20 \
4179 These are our two leaf classes. Note how we scale the weight with the configured
4180 rate. Both classes are not bounded, but they are connected to class 1:1
4181 which is bounded. So the sum of bandwith of the 2 classes will never be
4182 more than 6mbit. The classids need to be within the same major number as
4183 the parent qdisc, by the way!
4189 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
4190 # tc qdisc add dev eth0 parent 1:4 handle 40: sfq
4196 Both classes have a FIFO qdisc by default. But we replaced these with an SFQ
4197 queue so each flow of data is treated equally.
4200 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
4201 sport 80 0xffff flowid 1:3
4202 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
4203 sport 25 0xffff flowid 1:4
4209 These commands, attached directly to the root, send traffic to the right
4214 Note that we use 'tc class add' to CREATE classes within a qdisc, but that
4215 we use 'tc qdisc add' to actually add qdiscs to these classes.
4219 You may wonder what happens to traffic that is not classified by any of the
4220 two rules. It appears that in this case, data will then be processed within
4221 1:0, and be unlimited.
4225 If SMTP+web together try to exceed the set limit of 6mbit/s, bandwidth will
4226 be divided according to the weight parameter, giving 5/8 of traffic to the
4227 webserver and 3/8 to the mail server.
4231 With this configuration you can also say that webserver traffic will always
4232 get at minimum 5/8 * 6 mbit = 3.75 mbit.
4238 <Title>Other CBQ parameters: split & defmap</Title>
4241 As said before, a classful qdisc needs to call filters to determine
4242 which class a packet will be enqueued to.
4246 Besides calling the filter, CBQ offers other options, defmap & split.
4247 This is pretty complicated to understand, and it is not vital. But as this
4248 is the only known place where defmap & split are properly explained, I'm
4253 As you will often want to filter on the Type of Service field only, a special
4254 syntax is provided. Whenever the CBQ needs to figure out where a packet
4255 needs to be enqueued, it checks if this node is a 'split node'. If so, one
4256 of the sub-qdiscs has indicated that it wishes to receive all packets with
4257 a certain configured priority, as might be derived from the TOS field, or
4258 socket options set by applications.
4262 The packets' priority bits are and-ed with the defmap field to see if a match
4263 exists. In other words, this is a short-hand way of creating a very fast
4264 filter, which only matches certain priorities. A defmap of ff (hex) will
4265 match everything, a map of 0 nothing. A sample configuration may help make
4272 # tc qdisc add dev eth1 root handle 1: cbq bandwidth 10Mbit allot 1514 \
4273 cell 8 avpkt 1000 mpu 64
4275 # tc class add dev eth1 parent 1:0 classid 1:1 cbq bandwidth 10Mbit \
4276 rate 10Mbit allot 1514 cell 8 weight 1Mbit prio 8 maxburst 20 \
4280 Standard CBQ preamble. I never get used to the sheer amount of numbers
4285 Defmap refers to TC_PRIO bits, which are defined as follows:
4291 TC_PRIO.. Num Corresponds to TOS
4292 -------------------------------------------------
4293 BESTEFFORT 0 Maximize Reliablity
4294 FILLER 1 Minimize Cost
4295 BULK 2 Maximize Throughput (0x8)
4297 INTERACTIVE 6 Minimize Delay (0x10)
4304 The TC_PRIO.. number corresponds to bits, counted from the right. See the
4305 pfifo_fast section for more details how TOS bits are converted to
4310 Now the interactive and the bulk classes:
4316 # tc class add dev eth1 parent 1:1 classid 1:2 cbq bandwidth 10Mbit \
4317 rate 1Mbit allot 1514 cell 8 weight 100Kbit prio 3 maxburst 20 \
4318 avpkt 1000 split 1:0 defmap c0
4320 # tc class add dev eth1 parent 1:1 classid 1:3 cbq bandwidth 10Mbit \
4321 rate 8Mbit allot 1514 cell 8 weight 800Kbit prio 7 maxburst 20 \
4322 avpkt 1000 split 1:0 defmap 3f
4328 The 'split qdisc' is 1:0, which is where the choice will be made. C0 is
4329 binary for 11000000, 3F for 00111111, so these two together will match
4330 everything. The first class matches bits 7 & 6, and thus corresponds
4331 to 'interactive' and 'control' traffic. The second class matches the rest.
4335 Node 1:0 now has a table like this:
4352 For additional fun, you can also pass a 'change mask', which indicates
4353 exactly which priorities you wish to change. You only need to use this if you
4354 are running 'tc class change'. For example, to add best effort traffic to
4355 1:2, we could run this:
4361 # tc class change dev eth1 classid 1:2 cbq defmap 01/01
4367 The priority map at 1:0 now looks like this:
4387 FIXME: did not test 'tc class change', only looked at the source.
4395 <Title>Hierarchical Token Bucket </Title>
4398 Martin Devera (<devik>) rightly realised that CBQ is complex and does
4399 not seem optimized for many typical situations. His Hierarchical approach is
4400 well suited for setups where you have a fixed amount of bandwidth which you
4401 want to divide for different purposes, giving each purpose a guaranteed
4402 bandwidth, with the possibility of specifying how much bandwidth can be
4407 HTB works just like CBQ but does not resort to idle time calculations to
4408 shape. Instead, it is a classful Token Bucket Filter - hence the name. It
4409 has only a few parameters, which are well documented on his
4411 URL="http://luxik.cdi.cz/~devik/qos/htb/"
4417 As your HTB configuration gets more complex, your configuration scales
4418 well. With CBQ it is already complex even in simple cases! HTB3 (check
4419 <ulink url="http://luxik.cdi.cz/~devik/qos/htb/">its homepage</ulink> for
4420 details on HTB versions) is now part of the official kernel sources
4421 (from 2.4.20-pre1 and 2.5.31 onwards). However, maybe you still need to
4422 get a HTB3 patched version of 'tc': HTB kernel and userspace parts must
4423 be the same major version, or 'tc' will not work with HTB.
4428 If you already have a modern kernel, or are in a position to patch your
4429 kernel, by all means consider HTB.
4434 <Title>Sample configuration</Title>
4437 Functionally almost identical to the CBQ sample configuration above:
4443 # tc qdisc add dev eth0 root handle 1: htb default 30
4445 # tc class add dev eth0 parent 1: classid 1:1 htb rate 6mbit burst 15k
4447 # tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit burst 15k
4448 # tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 6mbit burst 15k
4449 # tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 6mbit burst 15k
4455 The author then recommends SFQ for beneath these classes:
4458 # tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
4459 # tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
4460 # tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10
4466 Add the filters which direct traffic to the right classes:
4469 # U32="tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32"
4470 # $U32 match ip dport 80 0xffff flowid 1:10
4471 # $U32 match ip sport 25 0xffff flowid 1:20
4474 And that's it - no unsightly unexplained numbers, no undocumented
4479 HTB certainly looks wonderful - if 10: and 20: both have their guaranteed
4480 bandwidth, and more is left to divide, they borrow in a 5:3 ratio, just as
4485 Unclassified traffic gets routed to 30:, which has little bandwidth of its
4486 own but can borrow everything that is left over. Because we chose SFQ
4487 internally, we get fairness thrown in for free!
4496 <Sect1 id="lartc.qdisc.filters">
4497 <Title>Classifying packets with filters</Title>
4500 To determine which class shall process a packet, the so-called 'classifier
4501 chain' is called each time a choice needs to be made. This chain consists of
4502 all filters attached to the classful qdisc that needs to decide.
4505 <Para>To reiterate the tree, which is not a tree:
4521 When enqueueing a packet, at each branch the filter chain is consulted for a
4522 relevant instruction. A typical setup might be to have a filter in 1:1 that
4523 directs a packet to 12: and a filter on 12: that sends the packet to 12:2.
4527 You might also attach this latter rule to 1:1, but you can make efficiency
4528 gains by having more specific tests lower in the chain.
4532 You can't filter a packet 'upwards', by the way. Also, with HTB, you should
4533 attach all filters to the root!
4537 And again - packets are only enqueued downwards! When they are dequeued,
4538 they go up again, where the interface lives. They do NOT fall off the end of
4539 the tree to the network adaptor!
4543 <Title>Some simple filtering examples</Title>
4546 As explained in the Classifier chapter, you can match on literally anything,
4547 using a very complicated syntax. To start, we will show how to do the
4548 obvious things, which luckily are quite easy.
4552 Let's say we have a PRIO qdisc called '10:' which contains three classes, and
4553 we want to assign all traffic from and to port 22 to the highest priority
4554 band, the filters would be:
4560 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
4561 ip dport 22 0xffff flowid 10:1
4562 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
4563 ip sport 80 0xffff flowid 10:1
4564 # tc filter add dev eth0 protocol ip parent 10: prio 2 flowid 10:2
4570 What does this say? It says: attach to eth0, node 10: a priority 1 u32
4571 filter that matches on IP destination port 22 *exactly* and send it to band
4572 10:1. And it then repeats the same for source port 80. The last command says
4573 that anything unmatched so far should go to band 10:2, the next-highest
4578 You need to add 'eth0', or whatever your interface is called, because each
4579 interface has a unique namespace of handles.
4583 To select on an IP address, use this:
4586 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
4587 match ip dst 4.3.2.1/32 flowid 10:1
4588 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
4589 match ip src 1.2.3.4/32 flowid 10:1
4590 # tc filter add dev eth0 protocol ip parent 10: prio 2 \
4597 This assigns traffic to 4.3.2.1 and traffic from 1.2.3.4 to the highest
4598 priority queue, and the rest to the next-highest one.
4602 You can concatenate matches, to match on traffic from 1.2.3.4 and from port
4606 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 match ip src 4.3.2.1/32 \
4607 match ip sport 80 0xffff flowid 10:1
4614 <Sect2 id="lartc.filtering.simple">
4615 <Title>All the filtering commands you will normally need</Title>
4618 Most shaping commands presented here start with this preamble:
4621 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 ..
4624 These are the so called 'u32' matches, which can match on ANY part of a
4629 <Term>On source/destination address</Term>
4632 Source mask 'match ip src 1.2.3.0/24', destination mask 'match ip dst
4633 4.3.2.0/24'. To match a single host, use /32, or omit the mask.
4637 <Term>On source/destination port, all IP protocols</Term>
4640 Source: 'match ip sport 80 0xffff', destination: 'match ip dport 80 0xffff'
4644 <Term>On ip protocol (tcp, udp, icmp, gre, ipsec)</Term>
4647 Use the numbers from /etc/protocols, for example, icmp is 1: 'match ip
4652 <Term>On fwmark</Term>
4655 You can mark packets with either ipchains or iptables and have that mark
4656 survive routing across interfaces. This is really useful to for example only
4657 shape traffic on eth1 that came in on eth0. Syntax:
4659 # tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 6 fw flowid 1:1
4661 Note that this is not a u32 match!
4665 You can place a mark like this:
4668 # iptables -A PREROUTING -t mangle -i eth0 -j MARK --set-mark 6
4671 The number 6 is arbitrary.
4675 If you don't want to understand the full tc filter syntax, just use
4676 iptables, and only learn to select on fwmark. You can also have iptables
4677 print basic statistics that will help you debug your rules.
4678 The following command will show you all the rules that mark packages
4679 in the mangle table, also how many packages and bytes have matched.
4682 # iptables -L -t mangle -n -v
4689 <Term>On the TOS field</Term>
4692 To select interactive, minimum delay traffic:
4695 # tc filter add dev ppp0 parent 1:0 protocol ip prio 10 u32 \
4696 match ip tos 0x10 0xff \
4700 Use 0x08 0xff for bulk traffic.
4707 For more filtering commands, see the Advanced Filters chapter.
4713 <Sect1 id="lartc.imq">
4714 <Title>The Intermediate queueing device (IMQ)</Title>
4717 The Intermediate queueing device is not a qdisc but its usage is tightly bound
4718 to qdiscs. Within linux, qdiscs are attached to network devices and everything
4719 that is queued to the device is first queued to the qdisc. From this concept,
4720 two limitations arise:
4726 Only egress shaping is possible (an ingress qdisc exists, but its
4727 possibilities are very limited compared to classful qdiscs).
4732 A qdisc can only see traffic of one interface, global limitations can't be
4739 IMQ is there to help solve those two limitations. In short, you can put
4740 everything you choose in a qdisc. Specially marked packets get intercepted
4741 in netfilter NF_IP_PRE_ROUTING and NF_IP_POST_ROUTING hooks and pass through
4742 the qdisc attached to an imq device. An iptables target is used for marking
4747 This enables you to do ingress shaping as you can just mark packets coming in from somewhere and/or treat interfaces as classes to set global limits.
4748 You can also do lots of other stuff like just putting your http traffic in a
4749 qdisc, put new connection requests in a qdisc, ...
4753 <Title>Sample configuration</Title>
4756 The first thing that might come to mind is use ingress shaping to give yourself
4757 a high guaranteed bandwidth. ;)
4758 Configuration is just like with any other interface:
4761 tc qdisc add dev imq0 root handle 1: htb default 20
4763 tc class add dev imq0 parent 1: classid 1:1 htb rate 2mbit burst 15k
4765 tc class add dev imq0 parent 1:1 classid 1:10 htb rate 1mbit
4766 tc class add dev imq0 parent 1:1 classid 1:20 htb rate 1mbit
4768 tc qdisc add dev imq0 parent 1:10 handle 10: pfifo
4769 tc qdisc add dev imq0 parent 1:20 handle 20: sfq
4771 tc filter add dev imq0 parent 10:0 protocol ip prio 1 u32 match \
4772 ip dst 10.0.0.230/32 flowid 1:10
4775 In this example u32 is used for classification. Other classifiers should work as
4777 Next traffic has to be selected and marked to be enqueued to imq0.
4780 iptables -t mangle -A PREROUTING -i eth0 -j IMQ --todev 0
4788 The IMQ iptables targets is valid in the PREROUTING and POSTROUTING chains of
4789 the mangle table. It's syntax is
4792 IMQ [ --todev n ] n : number of imq device
4795 An ip6tables target is also provided.
4799 Please note traffic is not enqueued when the target is hit but afterwards.
4800 The exact location where traffic enters the imq device depends on the
4801 direction of the traffic (in/out).
4802 These are the predefined netfilter hooks used by iptables:
4805 enum nf_ip_hook_priorities {
4806 NF_IP_PRI_FIRST = INT_MIN,
4807 NF_IP_PRI_CONNTRACK = -200,
4808 NF_IP_PRI_MANGLE = -150,
4809 NF_IP_PRI_NAT_DST = -100,
4810 NF_IP_PRI_FILTER = 0,
4811 NF_IP_PRI_NAT_SRC = 100,
4812 NF_IP_PRI_LAST = INT_MAX,
4819 For ingress traffic, imq registers itself with NF_IP_PRI_MANGLE + 1 priority
4820 which means packets enter the imq device directly after the mangle PREROUTING
4821 chain has been passed.
4825 For egress imq uses NF_IP_PRI_LAST which honours the fact that packets dropped
4826 by the filter table won't occupy bandwidth.
4830 The patches and some more information can be found at the
4832 URL="http://luxik.cdi.cz/~patrick/imq/"
4842 <chapter id="lartc.loadshare">
4843 <Title>Load sharing over multiple interfaces</Title>
4846 There are several ways of doing this. One of the easiest and straightforward
4847 ways is 'TEQL' - "True" (or "trivial") link equalizer. Like most things
4848 having to do with queueing, load sharing goes both ways. Both ends of a link
4849 may need to participate for full effect.
4853 Imagine this situation:
4859 +-------+ eth1 +-------+
4861 'network 1' ----| A | | B |---- 'network 2'
4863 +-------+ eth2 +-------+
4869 A and B are routers, and for the moment we'll assume both run Linux. If
4870 traffic is going from network 1 to network 2, router A needs to distribute
4871 the packets over both links to B. Router B needs to be configured to accept
4872 this. Same goes the other way around, when packets go from network 2 to
4873 network 1, router B needs to send the packets over both eth1 and eth2.
4877 The distributing part is done by a 'TEQL' device, like this (it couldn't be
4884 # tc qdisc add dev eth1 root teql0
4885 # tc qdisc add dev eth2 root teql0
4886 # ip link set dev teql0 up
4892 Don't forget the 'ip link set up' command!
4896 This needs to be done on both hosts. The device teql0 is basically a
4897 roundrobbin distributor over eth1 and eth2, for sending packets. No data
4898 ever comes in over an teql device, that just appears on the 'raw' eth1 and
4903 But now we just have devices, we also need proper routing. One way to do
4904 this is to assign a /31 network to both links, and a /31 to the teql0 device
4912 # ip addr add dev eth1 10.0.0.0/31
4913 # ip addr add dev eth2 10.0.0.2/31
4914 # ip addr add dev teql0 10.0.0.4/31
4923 # ip addr add dev eth1 10.0.0.1/31
4924 # ip addr add dev eth2 10.0.0.3/31
4925 # ip addr add dev teql0 10.0.0.5/31
4931 Router A should now be able to ping 10.0.0.1, 10.0.0.3 and 10.0.0.5 over the
4932 2 real links and the 1 equalized device. Router B should be able to ping
4933 10.0.0.0, 10.0.0.2 and 10.0.0.4 over the links.
4937 If this works, Router A should make 10.0.0.5 its route for reaching network
4938 2, and Router B should make 10.0.0.4 its route for reaching network 1. For
4939 the special case where network 1 is your network at home, and network 2 is
4940 the Internet, Router A should make 10.0.0.5 its default gateway.
4943 <Sect1 id="lartc.loadshare.caveats">
4944 <Title>Caveats</Title>
4947 Nothing is as easy as it seems. eth1 and eth2 on both router A and B need to
4948 have return path filtering turned off, because they will otherwise drop
4949 packets destined for ip addresses other than their own:
4955 # echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
4956 # echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
4962 Then there is the nasty problem of packet reordering. Let's say 6 packets
4963 need to be sent from A to B - eth1 might get 1, 3 and 5. eth2 would then do
4964 2, 4 and 6. In an ideal world, router B would receive this in order, 1, 2,
4965 3, 4, 5, 6. But the possibility is very real that the kernel gets it like
4966 this: 2, 1, 4, 3, 6, 5. The problem is that this confuses TCP/IP. While not
4967 a problem for links carrying many different TCP/IP sessions, you won't be
4968 able to bundle multiple links and get to ftp a single file lots faster,
4969 except when your receiving or sending OS is Linux, which is not easily
4970 shaken by some simple reordering.
4974 However, for lots of applications, link load balancing is a great idea.
4978 <Sect1 id="lartc.loadshare.other">
4979 <Title>Other possibilities</Title>
4981 William Stearns has used an advanced tunneling setup to achieve good use of
4982 multiple, unrelated, internet connections together. It can be found on
4984 URL="http://www.stearns.org/tunnel/">his tunneling page</ULink>.
4987 The HOWTO may feature more about this in the future.
4992 <chapter id="lartc.netfilter">
4993 <Title>Netfilter & iproute - marking packets</Title>
4996 So far we've seen how iproute works, and netfilter was mentioned a few
4997 times. This would be a good time to browse through <ULink
4998 URL="http://netfilter.samba.org/unreliable-guides/"
4999 >Rusty's Remarkably Unreliable Guides</ULink
5002 URL="http://netfilter.filewatcher.org/"
5008 Netfilter allows us to filter packets, or mangle their headers. One special
5009 feature is that we can mark a packet with a number. This is done with the
5010 --set-mark facility.
5014 As an example, this command marks all packets destined for port 25, outgoing
5021 # iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 25 \
5022 -j MARK --set-mark 1
5028 Let's say that we have multiple connections, one that is fast (and
5029 expensive, per megabyte) and one that is slower, but flat fee. We would most
5030 certainly like outgoing mail to go via the cheap route.
5034 We've already marked the packets with a '1', we now instruct the routing
5035 policy database to act on this:
5041 # echo 201 mail.out >> /etc/iproute2/rt_tables
5042 # ip rule add fwmark 1 table mail.out
5044 0: from all lookup local
5045 32764: from all fwmark 1 lookup mail.out
5046 32766: from all lookup main
5047 32767: from all lookup default
5053 Now we generate a route to the slow but cheap link in the mail.out table:
5056 # /sbin/ip route add default via 195.96.98.253 dev ppp0 table mail.out
5062 And we are done. Should we want to make exceptions, there are lots of ways
5063 to achieve this. We can modify the netfilter statement to exclude certain
5064 hosts, or we can insert a rule with a lower priority that points to the main
5065 table for our excepted hosts.
5069 We can also use this feature to honour TOS bits by marking packets with a
5070 different type of service with different numbers, and creating rules to act
5071 on that. This way you can even dedicate, say, an ISDN line to interactive
5076 Needless to say, this also works fine on a host that's doing NAT
5081 IMPORTANT: We received a report that MASQ and SNAT at least collide
5082 with marking packets. Rusty Russell explains it in
5084 URL="http://lists.samba.org/pipermail/netfilter/2000-November/006089.html"
5085 >this posting</ULink
5086 >. Turn off the reverse path filter to make it work
5091 Note: to mark packets, you need to have some options enabled in your
5098 IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
5099 IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
5100 IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK) [Y/n/?]
5106 See also the <xref linkend="lartc.cookbook.squid"> in the
5107 <citetitle><xref linkend="lartc.cookbook"></citetitle>.
5112 <chapter id="lartc.adv-filter"
5113 xreflabel="Advanced filters for (re-)classifying packets">
5114 <Title>Advanced filters for (re-)classifying packets</Title>
5117 As explained in the section on classful queueing disciplines, filters are
5118 needed to classify packets into any of the sub-queues. These filters are
5119 called from within the classful qdisc.
5123 Here is an incomplete list of classifiers available:
5130 Bases the decision on how the firewall has marked the packet. This can be
5131 the easy way out if you don't want to learn tc filter syntax. See the
5132 Queueing chapter for details.
5139 Bases the decision on fields within the packet (i.e. source IP address, etc)
5146 Bases the decision on which route the packet will be routed by
5150 <Term>rsvp, rsvp6</Term>
5153 Routes packets based on <ULink
5154 URL="http://www.isi.edu/div7/rsvp/overview.html"
5157 on networks you control - the Internet does not respect RSVP.
5161 <Term>tcindex</Term>
5164 Used in the DSMARK qdisc, see the relevant section.
5171 Note that in general there are many ways in which you can classify packet
5172 and that it generally comes down to preference as to which system you wish
5177 Classifiers in general accept a few arguments in common. They are listed
5178 here for convenience:
5185 <Term>protocol</Term>
5188 The protocol this classifier will accept. Generally you will only be
5189 accepting only IP traffic. Required.
5196 The handle this classifier is to be attached to. This handle must be
5197 an already existing class. Required.
5204 The priority of this classifier. Lower numbers get tested first.
5211 This handle means different things to different filters.
5218 All the following sections will assume you are trying to shape the traffic
5219 going to <Literal remap="tt">HostA</Literal>. They will assume that the root class has been
5220 configured on 1: and that the class you want to send the selected traffic to
5224 <Sect1 id="lartc.adv-filter.u32">
5225 <Title>The <option>u32</option> classifier</Title>
5228 The U32 filter is the most advanced filter available in the current
5229 implementation. It entirely based on hashing tables, which make it
5230 robust when there are many filter rules.
5234 In its simplest form the U32 filter is a list of records, each
5235 consisting of two fields: a selector and an action. The selectors,
5236 described below, are compared with the currently processed IP packet
5237 until the first match occurs, and then the associated action is performed.
5238 The simplest type of action would be directing the packet into defined
5243 The command line of <Literal remap="tt">tc filter</Literal> program, used to configure the filter,
5244 consists of three parts: filter specification, a selector and an action.
5245 The filter specification can be defined as:
5251 tc filter add dev IF [ protocol PROTO ]
5252 [ (preference|priority) PRIO ]
5259 The <Literal remap="tt">protocol</Literal> field describes protocol that the filter will be
5260 applied to. We will only discuss case of <Literal remap="tt">ip</Literal> protocol. The
5261 <Literal remap="tt">preference</Literal> field (<Literal remap="tt">priority</Literal> can be used alternatively)
5262 sets the priority of currently defined filter. This is important, since
5263 you can have several filters (lists of rules) with different priorities.
5264 Each list will be passed in the order the rules were added, then list with
5265 lower priority (higher preference number) will be processed. The <Literal remap="tt">parent</Literal>
5266 field defines the CBQ tree top (e.g. 1:0), the filter should be attached
5271 The options described above apply to all filters, not only U32.
5275 <Title>U32 selector </Title>
5278 The U32 selector contains definition of the pattern, that will be matched
5279 to the currently processed packet. Precisely, it defines which bits are
5280 to be matched in the packet header and nothing more, but this simple
5281 method is very powerful. Let's take a look at the following examples,
5282 taken directly from a pretty complex, real-world filter:
5288 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
5289 match u32 00100000 00ff0000 at 0 flowid 1:10
5295 For now, leave the first line alone - all these parameters describe
5296 the filter's hash tables. Focus on the selector line, containing
5297 <Literal remap="tt">match</Literal> keyword. This selector will match to IP headers, whose
5298 second byte will be 0x10 (0010). As you can guess, the 00ff number is
5299 the match mask, telling the filter exactly which bits to match. Here
5300 it's 0xff, so the byte will match if it's exactly 0x10. The <Literal remap="tt">at</Literal>
5301 keyword means that the match is to be started at specified offset (in
5302 bytes) -- in this case it's beginning of the packet. Translating all
5303 that to human language, the packet will match if its Type of Service
5304 field will have `low delay' bits set. Let's analyze another rule:
5310 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
5311 match u32 00000016 0000ffff at nexthdr+0 flowid 1:10
5317 The <Literal remap="tt">nexthdr</Literal> option means next header encapsulated in the IP packet,
5318 i.e. header of upper-layer protocol. The match will also start here
5319 at the beginning of the next header. The match should occur in the
5320 second, 32-bit word of the header. In TCP and UDP protocols this field
5321 contains packet's destination port. The number is given in big-endian
5322 format, i.e. older bits first, so we simply read 0x0016 as 22 decimal,
5323 which stands for SSH service if this was TCP. As you guess, this match
5324 is ambiguous without a context, and we will discuss this later.
5328 Having understood all the above, we will find the following selector
5329 quite easy to read: <Literal remap="tt">match c0a80100 ffffff00 at 16</Literal>. What we
5330 got here is a three byte match at 17-th byte, counting from the IP
5331 header start. This will match for packets with destination address
5332 anywhere in 192.168.1/24 network. After analyzing the examples, we
5333 can summarize what we have learned.
5339 <Title>General selectors</Title>
5342 General selectors define the pattern, mask and offset the pattern
5343 will be matched to the packet contents. Using the general selectors
5344 you can match virtually any single bit in the IP (or upper layer)
5345 header. They are more difficult to write and read, though, than
5346 specific selectors that described below. The general selector syntax
5353 match [ u32 | u16 | u8 ] PATTERN MASK at [OFFSET | nexthdr+OFFSET]
5359 One of the keywords <Literal remap="tt">u32</Literal>, <Literal remap="tt">u16</Literal> or <Literal remap="tt">u8</Literal> specifies
5360 length of the pattern in bits. PATTERN and MASK should follow, of length
5361 defined by the previous keyword. The OFFSET parameter is the offset,
5362 in bytes, to start matching. If <Literal remap="tt">nexthdr+</Literal> keyword is given,
5363 the offset is relative to start of the upper layer header.
5371 Packet will match to this rule, if its time to live (TTL) is 64.
5372 TTL is the field starting just after 8-th byte of the IP header.
5375 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
5376 match u8 64 0xff at 8 \
5384 The following matches all TCP packets which have the ACK bit set:
5387 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
5388 match ip protocol 6 0xff \
5389 match u8 0x10 0xff at nexthdr+13 \
5396 Use this to match ACKs on packets smaller than 64 bytes:
5399 ## match acks the hard way,
5401 ## IP header length 0x5(32 bit words),
5402 ## IP Total length 0x34 (ACK + 12 bytes of TCP options)
5403 ## TCP ack set (bit 5, offset 33)
5404 # tc filter add dev ppp14 parent 1:0 protocol ip prio 10 u32 \
5405 match ip protocol 6 0xff \
5406 match u8 0x05 0x0f at 0 \
5407 match u16 0x0000 0xffc0 at 2 \
5408 match u8 0x10 0xff at 33 \
5415 This rule will only match TCP packets with ACK bit set, and no further
5416 payload. Here we can see an example of using two selectors, the final result
5417 will be logical AND of their results. If we take a look at TCP header
5418 diagram, we can see that the ACK bit is second older bit (0x10) in the 14-th
5419 byte of the TCP header (<Literal remap="tt">at nexthdr+13</Literal>). As for the second
5420 selector, if we'd like to make our life harder, we could write <Literal remap="tt">match u8
5421 0x06 0xff at 9</Literal> instead of using the specific selector <Literal remap="tt">protocol
5422 tcp</Literal>, because 6 is the number of TCP protocol, present in 10-th byte of
5423 the IP header. On the other hand, in this example we couldn't use any
5424 specific selector for the first match - simply because there's no specific
5425 selector to match TCP ACK bits.
5426 <!--orders & extractionis seem to be confusing in this paragraph-->
5430 The filter below is a modified version of the filter above. The difference is, that it
5431 doesn't check the ip header length. Why? Because the filter above does only work on 32
5438 tc filter add dev ppp14 parent 1:0 protocol ip prio 10 u32 \
5439 match ip protocol 6 0xff \
5440 match u8 0x10 0xff at nexthdr+13 \
5441 match u16 0x0000 0xffc0 at 2 \
5451 <Title>Specific selectors</Title>
5454 The following table contains a list of all specific selectors
5455 the author of this section has found in the <Literal remap="tt">tc</Literal> program
5456 source code. They simply make your life easier and increase readability
5457 of your filter's configuration.
5461 FIXME: table placeholder - the table is in separate file ,,selector.html''
5465 FIXME: it's also still in Polish :-(
5469 FIXME: must be sgml'ized
5479 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
5480 match ip tos 0x10 0xff \
5487 FIXME: tcp dport match does not work as described below:
5491 The above rule will match packets which have the TOS field set to 0x10.
5492 The TOS field starts at second byte of the packet and is one byte big,
5493 so we could write an equivalent general selector: <Literal remap="tt">match u8 0x10 0xff
5494 at 1</Literal>. This gives us hint to the internals of U32 filter -- the
5495 specific rules are always translated to general ones, and in this
5496 form they are stored in the kernel memory. This leads to another conclusion
5497 -- the <Literal remap="tt">tcp</Literal> and <Literal remap="tt">udp</Literal> selectors are exactly the same
5498 and this is why you can't use single <Literal remap="tt">match tcp dport 53 0xffff</Literal>
5499 selector to match TCP packets sent to given port -- they will also
5500 match UDP packets sent to this port. You must remember to also specify
5501 the protocol and end up with the following rule:
5507 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
5508 match tcp dport 53 0xffff \
5509 match ip protocol 0x6 0xff \
5519 <Sect1 id="lartc.adv-filter.route">
5520 <Title>The <option>route</option> classifier</Title>
5523 This classifier filters based on the results of the routing tables. When a
5524 packet that is traversing through the classes reaches one that is marked
5525 with the "route" filter, it splits the packets up based on information in
5527 <!--a packet <-> the packets-->
5533 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 route
5539 Here we add a route classifier onto the parent node 1:0 with priority 100.
5540 When a packet reaches this node (which, since it is the root, will happen
5541 immediately) it will consult the routing table. If the packet matches, it will
5542 be send to the given class and have a priority of 100. Then, to finally
5543 kick it into action, you add the appropriate routing entry:
5547 The trick here is to define 'realm' based on either destination or source.
5548 The way to do it is like this:
5554 # ip route add Host/Network via Gateway dev Device realm RealmNumber
5560 For instance, we can define our destination network 192.168.10.0 with a realm
5567 # ip route add 192.168.10.0/24 via 192.168.10.1 dev eth1 realm 10
5573 When adding route filters, we can use realm numbers to represent the
5574 networks or hosts and specify how the routes match the filters.
5580 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
5581 route to 10 classid 1:10
5587 The above rule matches the packets going to the network 192.168.10.0.
5591 Route filter can also be used to match source routes. For example, there is
5592 a subnetwork attached to the Linux router on eth2.
5598 # ip route add 192.168.2.0/24 dev eth2 realm 2
5599 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
5600 route from 2 classid 1:2
5606 Here the filter specifies that packets from the subnetwork 192.168.2.0
5607 (realm 2) will match class id 1:2.
5612 <Sect1 id="lartc.adv-filter.policing">
5613 <Title>Policing filters</Title>
5616 To make even more complicated setups possible, you can have filters that
5617 only match up to a certain bandwidth. You can declare a filter either to entirely
5618 cease matching above a certain rate, or not to match only the bandwidth
5619 exceeding a certain rate.
5623 So if you decided to police at 4mbit/s, but 5mbit/s of traffic is present,
5624 you can stop matching either the entire 5mbit/s, or only not match 1mbit/s,
5625 and do send 4mbit/s to the configured class.
5629 If bandwidth exceeds the configured rate, you can drop a packet, reclassify
5630 it, or see if another filter will match it.
5634 <Title>Ways to police</Title>
5637 There are basically two ways to police. If you compiled the kernel
5638 with 'Estimators', the kernel can measure for each filter how much traffic
5639 it is passing, more or less. These estimators are very easy on the CPU, as
5640 they simply count 25 times per second how many data has been passed, and
5641 calculate the bitrate from that.
5645 The other way works again via a Token Bucket Filter, this time living within
5646 your filter. The TBF only matches traffic UP TO your configured bandwidth,
5647 if more is offered, only the excess is subject to the configured overlimit
5652 <Title>With the kernel estimator</Title>
5655 This is very simple and has only one parameter: avrate. Either the flow
5656 remains below avrate, and the filter classifies the traffic to the classid
5657 configured, or your rate exceeds it in which case the specified action is
5658 taken, which is 'reclassify' by default.
5662 The kernel uses an Exponential Weighted Moving Average for your bandwidth
5663 which makes it less sensitive to short bursts.
5669 <Title>With Token Bucket Filter</Title>
5672 Uses the following parameters:
5678 burst/buffer/maxburst
5705 Which behave mostly identical to those described in the Token Bucket Filter
5706 section. Please note however that if you set the mtu of a TBF policer too
5707 low, *no* packets will pass, whereas the egress TBF qdisc will just pass
5712 Another difference is that a policer can only let a packet pass, or drop it.
5713 It cannot hold it in order to delay it.
5721 <Title>Overlimit actions</Title>
5724 If your filter decides that it is overlimit, it can take 'actions'.
5725 Currently, four actions are available:
5729 <Term>continue</Term>
5732 Causes this filter not to match, but perhaps other filters will.
5739 This is a very fierce option which simply discards traffic exceeding a
5740 certain rate. It is often used in the ingress policer and has limited uses.
5741 For example, you may have a name server that falls over if offered more than
5742 5mbit/s of packets, in which case an ingress filter could be used to make
5743 sure no more is ever offered.
5747 <Term>Pass/OK</Term>
5750 Pass on traffic ok. Might be used to disable a complicated filter, but leave
5755 <Term>reclassify</Term>
5758 Most often comes down to reclassification to Best Effort. This is the
5768 <Title>Examples</Title>
5771 The only real example known is mentioned in the 'Protecting your host
5772 from SYN floods' section.
5776 Limit incoming icmp traffic to 2kbit, drop packets
5781 tc filter add dev $DEV parent ffff: \
5782 protocol ip prio 20 \
5783 u32 match ip protocol 1 0xff \
5784 police rate 2kbit buffer 10k drop \
5789 Limit packets to a certain size (i.e. all packets
5790 with a length greater than 84 bytes will get dropped):
5794 tc filter add dev $DEV parent ffff: \
5795 protocol ip prio 20 \
5797 police mtu 84 drop \
5803 This method can be used to drop all packets:
5807 tc filter add dev $DEV parent ffff: \
5808 protocol ip prio 20 \
5809 u32 match ip protocol 1 0xff \
5815 It actually drops icmp packets greater-than 1 byte. While packets with
5816 a size of 1 byte are possible in theory, you will not find these in a real network.
5823 <Sect1 id="lartc.adv-filter.hashing">
5824 <Title>Hashing filters for very fast massive filtering</Title>
5827 If you have a need for thousands of rules, for example if you have a lot of
5828 clients or computers, all with different QoS specifications, you may find
5829 that the kernel spends a lot of time matching all those rules.
5833 By default, all filters reside in one big chain which is matched in
5834 descending order of priority. If you have 1000 rules, 1000 checks may be
5835 needed to determine what to do with a packet.
5839 Matching would go much quicker if you would have 256 chains with each four
5840 rules - if you could divide packets over those 256 chains, so that the right
5845 Hashing makes this possible. Let's say you have 1024 cable modem customers in
5846 your network, with IP addresses ranging from 1.2.0.0 to 1.2.3.255, and each
5847 has to go in another bin, for example 'lite', 'regular' and 'premium'. You
5848 would then have 1024 rules like this:
5854 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5856 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5859 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5860 1.2.3.254 classid 1:3
5861 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5862 1.2.3.255 classid 1:2
5868 To speed this up, we can use the last part of the IP address as a 'hash
5869 key'. We then get 256 tables, the first of which looks like this:
5872 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5874 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5876 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5878 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5885 The next one starts like this:
5888 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
5896 This way, only four checks are needed at most, two on average.
5900 Configuration is pretty complicated, but very worth it by the time you have
5901 this many rules. First we make a filter root, then we create a table with
5905 # tc filter add dev eth1 parent 1:0 prio 5 protocol ip u32
5906 # tc filter add dev eth1 parent 1:0 prio 5 handle 2: protocol ip u32 divisor 256
5912 Now we add some rules to entries in the created table:
5918 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5919 match ip src 1.2.0.123 flowid 1:1
5920 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5921 match ip src 1.2.1.123 flowid 1:2
5922 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5923 match ip src 1.2.3.123 flowid 1:3
5924 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
5925 match ip src 1.2.4.123 flowid 1:2
5928 This is entry 123, which contains matches for 1.2.0.123, 1.2.1.123,
5929 1.2.2.123, 1.2.3.123, and sends them to 1:1, 1:2, 1:3 and 1:2 respectively.
5930 Note that we need to specify our hash bucket in hex, 0x7b is 123.
5934 Next create a 'hashing filter' that directs traffic to the right entry in
5938 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 800:: \
5939 match ip src 1.2.0.0/16 \
5940 hashkey mask 0x000000ff at 12 \
5944 Ok, some numbers need explaining. The default hash table is called 800:: and
5945 all filtering starts there. Then we select the source address, which lives
5946 as position 12, 13, 14 and 15 in the IP header, and indicate that we are
5947 only interested in the last part. This will be sent to hash table 2:, which we
5952 It is quite complicated, but it does work in practice and performance will
5953 be staggering. Note that this example could be improved to the ideal case
5954 where each chain contains 1 filter!
5960 <Sect1 id="lartc.adv-filter.ipv6">
5961 <Title>Filtering IPv6 Traffic</Title>
5964 <Title>How come that IPv6 tc filters do not work?</Title>
5967 The Routing Policy Database (RPDB) replaced the IPv4 routing and
5968 addressing structure within the Linux Kernel which lead to all the
5969 wonderful features this HOWTO describes. Unfortunately, the IPv6
5970 structure within Linux was implemented outside of this core structure.
5971 Although they do share some facilities, the essential RPDB structure
5972 does not particpate in or with the IPv6 addressing and routing
5977 This will change for sure, we just have to wait a little longer.
5981 FIXME: Any ideas if someone is working on this? Plans?
5987 <Title>Marking IPv6 packets using ip6tables</Title>
5990 ip6tables is able to mark a packet and assign a number to it:
5994 # ip6tables -A PREROUTING -i eth0 -t mangle -p tcp -j MARK --mark 1
5998 But still, this will not help because the packet will not pass through the
6005 <Title>Using the u32 selector to match IPv6 packet</Title>
6008 IPv6 is normally encapsulated in a SIT tunnel and transported
6009 over IPv4 networks. See section IPv6 Tunneling for information on
6010 howto setup such a tunnel. This allows us to filter on the IPv4 packets
6011 holding the IPv6 packets as payload.
6015 The following filter matches all IPv6 encapsulated in IPv4 packets:
6019 # tc filter add dev $DEV parent 10:0 protocol ip prio 10 u32 \
6020 match ip protocol 41 0xff flowid 42:42
6024 Let's carry on with that. Assume your IPv6 packets get sent out
6025 over IPv4 and these packets have no options set. One could use
6026 the following filter to match ICMPv6 in IPv6 in IPv4 with no options.
6027 0x3a (58) is the Next-Header type for ICMPv6.
6031 # tc filter add dev $DEV parent 10:0 protocol ip prio 10 u32 \
6032 match ip protocol 41 0xff \
6033 match u8 0x05 0x0f at 0 \
6034 match u8 0x3a 0xff at 26 \
6039 Matching the destination IPv6 address is a bit more work. The following
6040 filter matches on the destination address
6041 3ffe:202c:ffff:32:230:4fff:fe08:358d:
6045 # tc filter add dev $DEV parent 10:0 protocol ip prio 10 u32 \
6046 match ip protocol 41 0xff \
6047 match u8 0x05 0x0f at 0 \
6048 match u8 0x3f 0xff at 44 \
6049 match u8 0xfe 0xff at 45 \
6050 match u8 0x20 0xff at 46 \
6051 match u8 0x2c 0xff at 47 \
6052 match u8 0xff 0xff at 48 \
6053 match u8 0xff 0xff at 49 \
6054 match u8 0x00 0xff at 50 \
6055 match u8 0x32 0xff at 51 \
6056 match u8 0x02 0xff at 52 \
6057 match u8 0x30 0xff at 53 \
6058 match u8 0x4f 0xff at 54 \
6059 match u8 0xff 0xff at 55 \
6060 match u8 0xfe 0xff at 56 \
6061 match u8 0x08 0xff at 57 \
6062 match u8 0x35 0xff at 58 \
6063 match u8 0x8d 0xff at 59 \
6068 The same technique can be used to match subnets. For example 2001::
6072 # tc filter add dev $DEV parent 10:0 protocol ip prio 10 u32 \
6073 match ip protocol 41 0xff \
6074 match u8 0x05 0x0f at 0 \
6075 match u8 0x20 0xff at 28 \
6076 match u8 0x01 0xff at 29 \
6086 <chapter id="lartc.kernel">
6087 <Title>Kernel network parameters </Title>
6091 The kernel has lots of parameters which
6092 can be tuned for different circumstances. While, as usual, the default
6093 parameters serve 99% of installations very well, we don't call this the
6094 Advanced HOWTO for the fun of it!
6098 The interesting bits are in /proc/sys/net, take a look there. Not everything
6099 will be documented here initially, but we're working on it.
6103 In the meantime you may want to have a look at the Linux-Kernel sources;
6104 read the file Documentation/filesystems/proc.txt. Most of the
6105 features are explained there.
6112 <Sect1 id="lartc.kernel.rpf"
6113 xreflabel="Reverse Path Filtering">
6114 <Title>Reverse Path Filtering</Title>
6117 By default, routers route everything, even packets which 'obviously' don't
6118 belong on your network. A common example is private IP space escaping onto
6119 the Internet. If you have an interface with a route of 195.96.96.0/24 to it,
6120 you do not expect packets from 212.64.94.1 to arrive there.
6124 Lots of people will want to turn this feature off, so the kernel hackers
6125 have made it easy. There are files in /proc where you can tell
6126 the kernel to do this for you. The method is called "Reverse Path
6127 Filtering". Basically, if the reply to a packet wouldn't go out the
6128 interface this packet came in, then this is a bogus packet and should be
6133 The following fragment will turn this on for all current and future
6140 # for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
6141 > echo 2 > $i
6148 Going by the example above, if a packet arrived on the Linux router on eth1
6149 claiming to come from the Office+ISP subnet, it would be dropped. Similarly,
6150 if a packet came from the Office subnet, claiming to be from somewhere
6151 outside your firewall, it would be dropped also.
6152 <!--it's not clear what Office+ISP subnet is and where eth1 is at this point.-->
6156 The above is full reverse path filtering. The default is to only filter
6157 based on IPs that are on directly connected networks. This is because the
6158 full filtering breaks in the case of asymmetric routing (where packets come
6159 in one way and go out another, like satellite traffic, or if you have
6160 dynamic (bgp, ospf, rip) routes in your network. The data comes down
6161 through the satellite dish and replies go back through normal land-lines).
6165 If this exception applies to you (and you'll probably know if it does) you
6166 can simply turn off the rp_filter on the interface where the
6167 satellite data comes in. If you want to see if any packets are being
6168 dropped, the log_martians file in the same directory will tell
6169 the kernel to log them to your syslog.
6175 # echo 1 >/proc/sys/net/ipv4/conf/<interfacename>/log_martians
6181 FIXME: is setting the conf/{default,all}/* files enough? - martijn
6186 <Sect1 id="lartc.kernel.obscure">
6187 <Title>Obscure settings</Title>
6190 Ok, there are a lot of parameters which can be modified. We try to list them
6191 all. Also documented (partly) in Documentation/ip-sysctl.txt.
6195 Some of these settings have different defaults based on whether you
6196 answered 'Yes' to 'Configure as router and not host' while compiling your
6201 Oskar Andreasson also has a page on all these flags and it appears to be
6202 better than ours, so also check
6203 <ulink url="http://ipsysctl-tutorial.frozentux.net/">
6204 http://ipsysctl-tutorial.frozentux.net/</ulink>.
6208 <Title>Generic ipv4</Title>
6211 As a generic note, most rate limiting features don't work on loopback, so
6212 don't test them locally. The limits are supplied in 'jiffies', and are
6213 enforced using the earlier mentioned token bucket filter.
6217 The kernel has an internal clock which runs at 'HZ' ticks (or 'jiffies') per
6218 second. On Intel, 'HZ' is mostly 100. So setting a *_rate file to, say 50,
6219 would allow for 2 packets per second. The token bucket filter is also
6220 configured to allow for a burst of at most 6 packets, if enough tokens have
6225 Several entries in the following list have been copied from
6226 /usr/src/linux/Documentation/networking/ip-sysctl.txt, written by Alexey
6227 Kuznetsov <kuznet@ms2.inr.ac.ru> and Andi Kleen <ak@muc.de>
6231 <Term>/proc/sys/net/ipv4/icmp_destunreach_rate</Term>
6234 If the kernel decides that it can't deliver a packet, it will drop it, and
6235 send the source of the packet an ICMP notice to this effect.
6239 <Term>/proc/sys/net/ipv4/icmp_echo_ignore_all</Term>
6242 Don't act on echo packets at all. Please don't set this by default, but if
6243 you are used as a relay in a DoS attack, it may be useful.
6247 <Term>/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts [Useful]</Term>
6250 If you ping the broadcast address of a network, all hosts are supposed to
6251 respond. This makes for a dandy denial-of-service tool. Set this to 1 to
6252 ignore these broadcast messages.
6256 <Term>/proc/sys/net/ipv4/icmp_echoreply_rate</Term>
6259 The rate at which echo replies are sent to any one destination.
6263 <Term>/proc/sys/net/ipv4/icmp_ignore_bogus_error_responses</Term>
6266 Set this to ignore ICMP errors caused by hosts in the network reacting badly
6267 to frames sent to what they perceive to be the broadcast address.
6271 <Term>/proc/sys/net/ipv4/icmp_paramprob_rate</Term>
6274 A relatively unknown ICMP message, which is sent in response to incorrect
6275 packets with broken IP or TCP headers. With this file you can control the
6276 rate at which it is sent.
6280 <Term>/proc/sys/net/ipv4/icmp_timeexceed_rate</Term>
6283 This is the famous cause of the 'Solaris middle star' in traceroutes. Limits
6284 the rate of ICMP Time Exceeded messages sent.
6288 <Term>/proc/sys/net/ipv4/igmp_max_memberships</Term>
6291 Maximum number of listening igmp (multicast) sockets on the host.
6292 FIXME: Is this true?
6296 <Term>/proc/sys/net/ipv4/inet_peer_gc_maxtime</Term>
6299 FIXME: Add a little explanation about the inet peer storage?
6301 Miximum interval between garbage collection passes. This interval is in
6302 effect under low (or absent) memory pressure on the pool. Measured in
6307 <Term>/proc/sys/net/ipv4/inet_peer_gc_mintime</Term>
6310 Minimum interval between garbage collection passes. This interval is in
6311 effect under high memory pressure on the pool. Measured in jiffies.
6315 <Term>/proc/sys/net/ipv4/inet_peer_maxttl</Term>
6318 Maximum time-to-live of entries. Unused entries will expire after this
6319 period of time if there is no memory pressure on the pool (i.e. when the
6320 number of entries in the pool is very small). Measured in jiffies.
6324 <Term>/proc/sys/net/ipv4/inet_peer_minttl</Term>
6327 Minimum time-to-live of entries. Should be enough to cover fragment
6328 time-to-live on the reassembling side. This minimum time-to-live
6329 is guaranteed if the pool size is less than inet_peer_threshold.
6330 Measured in jiffies.
6334 <Term>/proc/sys/net/ipv4/inet_peer_threshold</Term>
6337 The approximate size of the INET peer storage. Starting from this threshold
6338 entries will be thrown aggressively. This threshold also determines
6339 entries' time-to-live and time intervals between garbage collection passes.
6340 More entries, less time-to-live, less GC interval.
6344 <Term>/proc/sys/net/ipv4/ip_autoconfig</Term>
6347 This file contains the number one if the host received its IP configuration by
6348 RARP, BOOTP, DHCP or a similar mechanism. Otherwise it is zero.
6352 <Term>/proc/sys/net/ipv4/ip_default_ttl</Term>
6355 Time To Live of packets. Set to a safe 64. Raise it if you have a huge
6356 network. Don't do so for fun - routing loops cause much more damage that
6357 way. You might even consider lowering it in some circumstances.
6361 <Term>/proc/sys/net/ipv4/ip_dynaddr</Term>
6364 You need to set this if you use dial-on-demand with a dynamic interface
6365 address. Once your demand interface comes up, any local TCP sockets which haven't seen replies will be rebound to have the right address. This solves the problem that the
6366 connection that brings up your interface itself does not work, but the
6371 <Term>/proc/sys/net/ipv4/ip_forward</Term>
6374 If the kernel should attempt to forward packets. Off by default.
6378 <Term>/proc/sys/net/ipv4/ip_local_port_range</Term>
6381 Range of local ports for outgoing connections. Actually quite small by
6382 default, 1024 to 4999.
6386 <Term>/proc/sys/net/ipv4/ip_no_pmtu_disc</Term>
6389 Set this if you want to disable Path MTU discovery - a technique to
6390 determine the largest Maximum Transfer Unit possible on your path. See also
6391 the section on Path MTU discovery in the
6392 <citetitle><xref linkend="lartc.cookbook"></citetitle> chapter.
6396 <Term>/proc/sys/net/ipv4/ipfrag_high_thresh</Term>
6399 Maximum memory used to reassemble IP fragments. When
6400 ipfrag_high_thresh bytes of memory is allocated for this purpose,
6401 the fragment handler will toss packets until ipfrag_low_thresh
6406 <Term>/proc/sys/net/ipv4/ip_nonlocal_bind</Term>
6409 Set this if you want your applications to be able to bind to an address
6410 which doesn't belong to a device on your system. This can be useful when
6411 your machine is on a non-permanent (or even dynamic) link, so your services
6412 are able to start up and bind to a specific address when your link is down.
6416 <Term>/proc/sys/net/ipv4/ipfrag_low_thresh</Term>
6419 Minimum memory used to reassemble IP fragments.
6423 <Term>/proc/sys/net/ipv4/ipfrag_time</Term>
6426 Time in seconds to keep an IP fragment in memory.
6430 <Term>/proc/sys/net/ipv4/tcp_abort_on_overflow</Term>
6433 A boolean flag controlling the behaviour under lots of incoming connections.
6434 When enabled, this causes the kernel to actively send RST packets when a
6435 service is overloaded.
6439 <Term>/proc/sys/net/ipv4/tcp_fin_timeout</Term>
6442 Time to hold socket in state FIN-WAIT-2, if it was closed by our side. Peer
6443 can be broken and never close its side, or even died unexpectedly. Default
6444 value is 60sec. Usual value used in 2.2 was 180 seconds, you may restore it,
6445 but remember that if your machine is even underloaded WEB server, you risk
6446 to overflow memory with kilotons of dead sockets, FIN-WAIT-2 sockets are
6447 less dangerous than FIN-WAIT-1, because they eat maximum 1.5K of memory, but
6448 they tend to live longer. Cf. tcp_max_orphans.
6452 <Term>/proc/sys/net/ipv4/tcp_keepalive_time</Term>
6455 How often TCP sends out keepalive messages when keepalive is enabled.
6461 <Term>/proc/sys/net/ipv4/tcp_keepalive_intvl</Term>
6464 How frequent probes are retransmitted, when a probe isn't acknowledged.
6466 Default: 75 seconds.
6470 <Term>/proc/sys/net/ipv4/tcp_keepalive_probes</Term>
6473 How many keepalive probes TCP will send, until it decides that the
6474 connection is broken.
6478 Multiplied with tcp_keepalive_intvl, this gives the time a link can be
6479 non-responsive after a keepalive has been sent.
6483 <Term>/proc/sys/net/ipv4/tcp_max_orphans</Term>
6486 Maximal number of TCP sockets not attached to any user file handle, held by
6487 system. If this number is exceeded orphaned connections are reset
6488 immediately and warning is printed. This limit exists only to prevent simple
6489 DoS attacks, you _must_ not rely on this or lower the limit artificially,
6490 but rather increase it (probably, after increasing installed memory), if
6491 network conditions require more than default value, and tune network
6492 services to linger and kill such states more aggressively. Let me remind you
6493 again: each orphan eats up to 64K of unswappable memory.
6497 <Term>/proc/sys/net/ipv4/tcp_orphan_retries</Term>
6500 How may times to retry before killing TCP connection, closed by our side.
6501 Default value 7 corresponds to 50sec-16min depending on RTO. If your machine
6502 is a loaded WEB server, you should think about lowering this value, such
6503 sockets may consume significant resources. Cf. tcp_max_orphans.
6507 <Term>/proc/sys/net/ipv4/tcp_max_syn_backlog</Term>
6510 Maximal number of remembered connection requests, which still did not
6511 receive an acknowledgment from connecting client. Default value is 1024 for
6512 systems with more than 128Mb of memory, and 128 for low memory machines. If
6513 server suffers of overload, try to increase this number. Warning! If you
6514 make it greater than 1024, it would be better to change TCP_SYNQ_HSIZE in
6515 include/net/tcp.h to keep TCP_SYNQ_HSIZE*16<=tcp_max_syn_backlog and to
6520 <Term>/proc/sys/net/ipv4/tcp_max_tw_buckets</Term>
6523 Maximal number of timewait sockets held by system simultaneously. If this
6524 number is exceeded time-wait socket is immediately destroyed and warning is
6525 printed. This limit exists only to prevent simple DoS attacks, you _must_
6526 not lower the limit artificially, but rather increase it (probably, after
6527 increasing installed memory), if network conditions require more than
6532 <Term>/proc/sys/net/ipv4/tcp_retrans_collapse</Term>
6535 Bug-to-bug compatibility with some broken printers.
6536 On retransmit try to send bigger packets to work around bugs in
6541 <Term>/proc/sys/net/ipv4/tcp_retries1</Term>
6544 How many times to retry before deciding that something is wrong
6545 and it is necessary to report this suspicion to network layer.
6546 Minimal RFC value is 3, it is default, which corresponds
6547 to 3sec-8min depending on RTO.
6551 <Term>/proc/sys/net/ipv4/tcp_retries2</Term>
6554 How may times to retry before killing alive TCP connection.
6556 URL="http://www.ietf.org/rfc/rfc1122.txt"
6559 says that the limit should be longer than 100 sec.
6560 It is too small number. Default value 15 corresponds to 13-30min
6565 <Term>/proc/sys/net/ipv4/tcp_rfc1337</Term>
6568 This boolean enables a fix for 'time-wait assassination hazards in tcp', described
6569 in RFC 1337. If enabled, this causes the kernel to drop RST packets for
6570 sockets in the time-wait state.
6576 <Term>/proc/sys/net/ipv4/tcp_sack</Term>
6579 Use Selective ACK which can be used to signify that specific packets are
6580 missing - therefore helping fast recovery.
6584 <Term>/proc/sys/net/ipv4/tcp_stdurg</Term>
6587 Use the Host requirements interpretation of the TCP urg pointer
6590 Most hosts use the older BSD interpretation, so if you turn this on
6591 Linux might not communicate correctly with them.
6597 <Term>/proc/sys/net/ipv4/tcp_syn_retries</Term>
6600 Number of SYN packets the kernel will send before giving up on the new
6605 <Term>/proc/sys/net/ipv4/tcp_synack_retries</Term>
6608 To open the other side of the connection, the kernel sends a SYN with a
6609 piggybacked ACK on it, to acknowledge the earlier received SYN. This is part
6610 2 of the threeway handshake. This setting determines the number of SYN+ACK
6611 packets sent before the kernel gives up on the connection.
6615 <Term>/proc/sys/net/ipv4/tcp_timestamps</Term>
6618 Timestamps are used, amongst other things, to protect against wrapping
6619 sequence numbers. A 1 gigabit link might conceivably re-encounter a previous
6620 sequence number with an out-of-line value, because it was of a previous
6621 generation. The timestamp will let it recognize this 'ancient packet'.
6625 <Term>/proc/sys/net/ipv4/tcp_tw_recycle</Term>
6628 Enable fast recycling TIME-WAIT sockets. Default value is 1.
6629 It should not be changed without advice/request of technical experts.
6633 <Term>/proc/sys/net/ipv4/tcp_window_scaling</Term>
6636 TCP/IP normally allows windows up to 65535 bytes big. For really fast
6637 networks, this may not be enough. The window scaling options allows for
6638 almost gigabyte windows, which is good for high bandwidth*delay products.
6647 <Title>Per device settings</Title>
6650 DEV can either stand for a real interface, or for 'all' or 'default'.
6651 Default also changes settings for interfaces yet to be created.
6655 <Term>/proc/sys/net/ipv4/conf/DEV/accept_redirects</Term>
6658 If a router decides that you are using it for a wrong purpose (ie, it needs
6659 to resend your packet on the same interface), it will send us a ICMP
6660 Redirect. This is a slight security risk however, so you may want to turn it
6661 off, or use secure redirects.
6665 <Term>/proc/sys/net/ipv4/conf/DEV/accept_source_route</Term>
6668 Not used very much anymore. You used to be able to give a packet a list of
6669 IP addresses it should visit on its way. Linux can be made to honor this IP
6674 <Term>/proc/sys/net/ipv4/conf/DEV/bootp_relay</Term>
6677 Accept packets with source address 0.b.c.d with destinations not to this host
6678 as local ones. It is supposed that a BOOTP relay daemon will catch and forward
6683 The default is 0, since this feature is not implemented yet (kernel version
6688 <Term>/proc/sys/net/ipv4/conf/DEV/forwarding</Term>
6691 Enable or disable IP forwarding on this interface.
6695 <Term>/proc/sys/net/ipv4/conf/DEV/log_martians</Term>
6699 <citetitle><xref linkend="lartc.kernel.rpf"></citetitle>.
6703 <Term>/proc/sys/net/ipv4/conf/DEV/mc_forwarding</Term>
6706 If we do multicast forwarding on this interface
6710 <Term>/proc/sys/net/ipv4/conf/DEV/proxy_arp</Term>
6713 If you set this to 1, this interface will respond to ARP requests for
6714 addresses the kernel has routes to. Can be very useful when building 'ip
6715 pseudo bridges'. Do take care that your netmasks are very correct before
6716 enabling this! Also be aware that the rp_filter, mentioned elsewhere, also
6717 operates on ARP queries!
6721 <Term>/proc/sys/net/ipv4/conf/DEV/rp_filter</Term>
6725 <citetitle><xref linkend="lartc.kernel.rpf"></citetitle>.
6729 <Term>/proc/sys/net/ipv4/conf/DEV/secure_redirects</Term>
6732 Accept ICMP redirect messages only for gateways, listed in default gateway
6733 list. Enabled by default.
6737 <Term>/proc/sys/net/ipv4/conf/DEV/send_redirects</Term>
6740 If we send the above mentioned redirects.
6744 <Term>/proc/sys/net/ipv4/conf/DEV/shared_media</Term>
6747 If it is not set the kernel does not assume that different subnets on this
6748 device can communicate directly. Default setting is 'yes'.
6752 <Term>/proc/sys/net/ipv4/conf/DEV/tag</Term>
6764 <Title>Neighbor policy</Title>
6767 Dev can either stand for a real interface, or for 'all' or 'default'.
6768 Default also changes settings for interfaces yet to be created.
6772 <Term>/proc/sys/net/ipv4/neigh/DEV/anycast_delay</Term>
6775 Maximum for random delay of answers to neighbor solicitation messages in
6776 jiffies (1/100 sec). Not yet implemented (Linux does not have anycast support
6781 <Term>/proc/sys/net/ipv4/neigh/DEV/app_solicit</Term>
6784 Determines the number of requests to send to the user level ARP daemon. Use 0
6789 <Term>/proc/sys/net/ipv4/neigh/DEV/base_reachable_time</Term>
6792 A base value used for computing the random reachable time value as specified
6797 <Term>/proc/sys/net/ipv4/neigh/DEV/delay_first_probe_time</Term>
6800 Delay for the first time probe if the neighbor is reachable. (see
6805 <Term>/proc/sys/net/ipv4/neigh/DEV/gc_stale_time</Term>
6808 Determines how often to check for stale ARP entries. After an ARP entry is
6809 stale it will be resolved again (which is useful when an IP address migrates
6810 to another machine). When ucast_solicit is greater than 0 it first tries to
6811 send an ARP packet directly to the known host When that fails and
6812 mcast_solicit is greater than 0, an ARP request is broadcast.
6816 <Term>/proc/sys/net/ipv4/neigh/DEV/locktime</Term>
6819 An ARP/neighbor entry is only replaced with a new one if the old is at least
6820 locktime old. This prevents ARP cache thrashing.
6824 <Term>/proc/sys/net/ipv4/neigh/DEV/mcast_solicit</Term>
6827 Maximum number of retries for multicast solicitation.
6831 <Term>/proc/sys/net/ipv4/neigh/DEV/proxy_delay</Term>
6834 Maximum time (real time is random [0..proxytime]) before answering to an ARP
6835 request for which we have an proxy ARP entry. In some cases, this is used to
6836 prevent network flooding.
6840 <Term>/proc/sys/net/ipv4/neigh/DEV/proxy_qlen</Term>
6843 Maximum queue length of the delayed proxy arp timer. (see proxy_delay).
6847 <Term>/proc/sys/net/ipv4/neigh/DEV/retrans_time</Term>
6850 The time, expressed in jiffies (1/100 sec), between retransmitted Neighbor
6851 Solicitation messages. Used for address resolution and to determine if a
6852 neighbor is unreachable.
6856 <Term>/proc/sys/net/ipv4/neigh/DEV/ucast_solicit</Term>
6859 Maximum number of retries for unicast solicitation.
6863 <Term>/proc/sys/net/ipv4/neigh/DEV/unres_qlen</Term>
6866 Maximum queue length for a pending arp request - the number of packets which
6867 are accepted from other layers while the ARP address is still resolved.
6876 <Title>Routing settings</Title>
6882 <Term>/proc/sys/net/ipv4/route/error_burst and /proc/sys/net/ipv4/route/error_cost</Term>
6885 This parameters are used to limit the warning messages written to the kernel
6886 log from the routing code. The higher the error_cost factor is, the fewer
6887 messages will be written. Error_burst controls when messages will be dropped.
6888 The default settings limit warning messages to one every five seconds.
6892 <Term>/proc/sys/net/ipv4/route/flush</Term>
6895 Writing to this file results in a flush of the routing cache.
6899 <Term>/proc/sys/net/ipv4/route/gc_elasticity</Term>
6902 Values to control the frequency and behavior of the garbage collection
6903 algorithm for the routing cache. This can be important for when doing
6904 fail over. At least gc_timeout seconds will elapse before Linux will skip
6905 to another route because the previous one has died. By default set to 300,
6906 you may want to lower it if you want to have a speedy fail over.
6911 URL="http://mailman.ds9a.nl/pipermail/lartc/2002q1/002667.html"
6913 > by Ard van Breemen.
6917 <Term>/proc/sys/net/ipv4/route/gc_interval</Term>
6920 See /proc/sys/net/ipv4/route/gc_elasticity.
6924 <Term>/proc/sys/net/ipv4/route/gc_min_interval</Term>
6927 See /proc/sys/net/ipv4/route/gc_elasticity.
6931 <Term>/proc/sys/net/ipv4/route/gc_thresh</Term>
6934 See /proc/sys/net/ipv4/route/gc_elasticity.
6938 <Term>/proc/sys/net/ipv4/route/gc_timeout</Term>
6941 See /proc/sys/net/ipv4/route/gc_elasticity.
6945 <Term>/proc/sys/net/ipv4/route/max_delay</Term>
6948 Maximum delay for flushing the routing cache.
6952 <Term>/proc/sys/net/ipv4/route/max_size</Term>
6955 Maximum size of the routing cache. Old entries will be purged once the cache
6956 reached has this size.
6960 <Term>/proc/sys/net/ipv4/route/min_adv_mss</Term>
6967 <Term>/proc/sys/net/ipv4/route/min_delay</Term>
6970 Minimum delay for flushing the routing cache.
6974 <Term>/proc/sys/net/ipv4/route/min_pmtu</Term>
6981 <Term>/proc/sys/net/ipv4/route/mtu_expires</Term>
6988 <Term>/proc/sys/net/ipv4/route/redirect_load</Term>
6991 Factors which determine if more ICMP redirects should be sent to a specific
6992 host. No redirects will be sent once the load limit or the maximum number of
6993 redirects has been reached.
6997 <Term>/proc/sys/net/ipv4/route/redirect_number</Term>
7000 See /proc/sys/net/ipv4/route/redirect_load.
7004 <Term>/proc/sys/net/ipv4/route/redirect_silence</Term>
7007 Timeout for redirects. After this period redirects will be sent again, even if
7008 this has been stopped, because the load or number limit has been reached.
7020 <chapter id="lartc.adv-qdisc">
7021 <Title>Advanced & less common queueing disciplines</Title>
7024 Should you find that you have needs not addressed by the queues mentioned
7025 earlier, the kernel contains some other more specialized queues mentioned here.
7028 <Sect1 id="lartc.adv-qdisc.bfifo-pfifo">
7029 <Title><literal>bfifo</literal>/<literal>pfifo</literal></Title>
7032 These classless queues are even simpler than pfifo_fast in that they lack
7033 the internal bands - all traffic is really equal. They have one important
7034 benefit though, they have some statistics. So even if you don't need shaping
7035 or prioritizing, you can use this qdisc to determine the backlog on your
7040 pfifo has a length measured in packets, bfifo in bytes.
7044 <Title>Parameters & usage</Title>
7053 Specifies the length of the queue. Measured in bytes for bfifo, in packets
7054 for pfifo. Defaults to the interface txqueuelen (see pfifo_fast chapter)
7055 packets long or txqueuelen*mtu bytes for bfifo.
7065 <Sect1 id="lartc.adv-qdisc.csz">
7066 <Title>Clark-Shenker-Zhang algorithm (CSZ)</Title>
7069 This is so theoretical that not even Alexey (the main CBQ author) claims to
7070 understand it. From his source:
7075 David D. Clark, Scott Shenker and Lixia Zhang
7076 <citetitle>Supporting Real-Time Applications in an Integrated Services Packet
7077 Network: Architecture and Mechanism</citetitle>.
7081 As I understand it, the main idea is to create WFQ flows for each guaranteed
7082 service and to allocate the rest of bandwith to dummy flow-0. Flow-0
7083 comprises the predictive services and the best effort traffic; it is handled
7084 by a priority scheduler with the highest priority band allocated for
7085 predictive services, and the rest --- to the best effort packets.
7089 Note that in CSZ flows are NOT limited to their bandwidth. It is supposed
7090 that the flow passed admission control at the edge of the QoS network and it
7091 doesn't need further shaping. Any attempt to improve the flow or to shape it
7092 to a token bucket at intermediate hops will introduce undesired delays and
7097 At the moment CSZ is the only scheduler that provides true guaranteed
7098 service. Another schemes (including CBQ) do not provide guaranteed delay and
7103 Does not currently seem like a good candidate to use, unless you've read and
7104 understand the article mentioned.
7110 <Sect1 id="lartc.adv-qdisc.dsmark"
7112 <Title>DSMARK</Title>
7116 <author><firstname>Esteve</firstname><surname>Camps</surname></author>
7117 <address><email>marvin@grn.es</email></address>
7118 This text is an extract from my thesis on
7119 <citetitle>QoS Support in Linux</citetitle>, September 2000.
7123 <Para>Source documents:
7129 <ULink URL="ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz">
7130 Draft-almesberger-wajhak-diffserv-linux-01.txt</ULink>.
7134 <Para>Examples in iproute2 distribution.
7139 <ULink URL="http://www.qosforum.com/white-papers/qosprot_v3.pdf">
7140 White Paper-QoS protocols and architectures</ULink> and
7141 <ULink URL="http://www.qosforum.com/docs/faq">
7142 IP QoS Frequently Asked Questions</ULink> both by
7143 <citetitle>Quality of Service Forum</citetitle>.
7149 This chapter was written by Esteve Camps <esteve@hades.udg.es>.
7153 <Title>Introduction</Title>
7156 First of all, it would be a great idea for you to read RFCs
7157 written about this (RFC2474, RFC2475, RFC2597 and RFC2598) at
7158 <ULink URL="http://www.ietf.org/html.charters/diffserv-charter.html">
7159 IETF DiffServ working Group web site</ULink> and
7160 <ULink URL="http://diffserv.sf.net/">
7161 Werner Almesberger web site</ULink>
7162 (he wrote the code to support Differentiated Services on Linux).
7168 <Title>What is Dsmark related to?</Title>
7171 Dsmark is a queueing discipline that offers the capabilities needed in
7172 Differentiated Services (also called DiffServ or, simply, DS). DiffServ is
7173 one of two actual QoS architectures (the other one is called Integrated
7174 Services) that is based on a value carried by packets in the DS field of the
7179 One of the first solutions in IP designed to offer some QoS level was
7180 the Type of Service field (TOS byte) in IP header. By changing that value,
7181 we could choose a high/low level of throughput, delay or reliability.
7182 But this didn't provide sufficient flexibility to the needs of new
7183 services (such as real-time applications, interactive applications and
7184 others). After this, new architectures appeared. One of these was DiffServ
7185 which kept TOS bits and renamed DS field.
7191 <Title>Differentiated Services guidelines</Title>
7194 Differentiated Services is group-oriented. I mean, we don't know anything
7195 about flows (this will be the Integrated Services purpose); we know about
7196 flow aggregations and we will apply different behaviours depending on which
7197 aggregation a packet belongs to.
7201 When a packet arrives to an edge node (entry node to a DiffServ domain)
7202 entering to a DiffServ Domain we'll have to policy, shape and/or mark those
7203 packets (marking refers to assigning a value to the DS field. It's just like the
7204 cows :-) ). This will be the mark/value that the internal/core nodes on our
7205 DiffServ Domain will look at to determine which behaviour or QoS level
7210 As you can deduce, Differentiated Services involves a domain on which
7211 all DS rules will have to be applied. In fact you can think I
7212 will classify all the packets entering my domain. Once they enter my
7213 domain they will be subjected to the rules that my classification dictates
7214 and every traversed node will apply that QoS level.
7218 In fact, you can apply your own policies into your local domains, but some
7219 <Emphasis>Service Level Agreements</Emphasis> should be considered when connecting to
7224 At this point, you maybe have a lot of questions. DiffServ is more than I've
7225 explained. In fact, you can understand that I can not resume more than 3
7226 RFCs in just 50 lines :-).
7232 <Title>Working with Dsmark</Title>
7235 As the DiffServ bibliography specifies, we differentiate boundary nodes and
7236 interior nodes. These are two important points in the traffic path. Both
7237 types perform a classification when the packets arrive. Its result may be
7238 used in different places along the DS process before the packet is released
7239 to the network. It's just because of this that the diffserv code supplies an
7240 structure called sk_buff, including a new field called skb->tc_index
7241 where we'll store the result of initial classification that may be used in
7242 several points in DS treatment.
7246 The skb->tc_index value will be initially set by the DSMARK qdisc,
7247 retrieving it from the DS field in IP header of every received packet.
7248 Besides, cls_tcindex classifier will read all or part of skb->tcindex
7249 value and use it to select classes.
7253 But, first of all, take a look at DSMARK qdisc command and its parameters:
7256 ... dsmark indices INDICES [ default_index DEFAULT_INDEX ] [ set_tc_index ]
7259 What do these parameters mean?
7265 <Emphasis remap="bf">indices</Emphasis>: size of table of (mask,value) pairs. Maximum value is 2ˆn, where n>=0.
7271 <Emphasis remap="bf">Default_index</Emphasis>: the default table entry index if classifier finds no match.
7277 <Emphasis remap="bf">Set_tc_index</Emphasis>: instructs dsmark discipline to retrieve the DS field and store it onto skb->tc_index.
7283 Let's see the DSMARK process.
7289 <Title>How SCH_DSMARK works.</Title>
7292 This qdisc will apply the next steps:
7298 If we have declared set_tc_index option in qdisc command, DS field is retrieved and stored onto
7299 skb->tc_index variable.
7305 Classifier is invoked. The classifier will be executed and it will return a class ID that will be stored in
7306 skb->tc_index variable. If no filter matches are found, we consider the default_index option to determine the
7307 classId to store. If neither set_tc_index nor default_index has been declared results may be
7314 After been sent to internal qdiscs where you can reuse the result of the filter, the classid returned by
7315 the internal qdisc is stored into skb->tc_index. We will use this value in the future to index a mask-
7316 value table. The final result to assign to the packet will be that resulting from next operation:
7319 New_Ds_field = ( Old_DS_field & mask ) | value
7328 Thus, new value will result from "anding" ds_field and mask values and next, this result "ORed" with
7329 value parameter. See next diagram to understand all this process:
7338 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
7340 | -- If you declare set_tc_index, we set DS | | <-----May change
7341 | value into skb->tc_index variable | |O DS field
7343 +-|-+ +------+ +---+-+ Internal +-+ +---N|-----|----+
7344 | | | | tc |--->| | |--> . . . -->| | | D| | |
7345 | | |----->|index |--->| | | Qdisc | |---->| v | |
7346 | | | |filter|--->| | | +---------------+ | ---->(mask,value) |
7347 -->| O | +------+ +-|-+--------------^----+ / | (. , .) |
7348 | | | ^ | | | | (. , .) |
7349 | | +----------|---------|----------------|-------|--+ (. , .) |
7350 | | sch_dsmark | | | | |
7351 +-|------------|---------|----------------|-------|------------------+
7352 | | | <- tc_index -> | |
7353 | |(read) | may change | | <--------------Index to the
7354 | | | | | (mask,value)
7355 v | v v | pairs table
7356 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ->
7363 How to do marking? Just change the mask and value of the class you want to remark. See next line of code:
7366 tc class change dev eth0 classid 1:1 dsmark mask 0x3 value 0xb8
7369 This changes the (mask,value) pair in hash table, to remark packets belonging to class 1:1.You have to "change" this values
7370 because of default values that (mask,value) gets initially (see table below).
7374 Now, we'll explain how TC_INDEX filter works and how fits into this. Besides, TCINDEX filter can be
7375 used in other configurations rather than those including DS services.
7381 <Title>TC_INDEX Filter</Title>
7384 This is the basic command to declare a TC_INDEX filter:
7387 ... tcindex [ hash SIZE ] [ mask MASK ] [ shift SHIFT ]
7388 [ pass_on | fall_through ]
7389 [ classid CLASSID ] [ police POLICE_SPEC ]
7393 Next, we show the example used to explain TC_INDEX operation mode. Pay attention to bolded words:
7396 tc qdisc add dev eth0 handle 1:0 root dsmark indices 64 <Emphasis remap="bf">set_tc_index</Emphasis>
7398 tc filter add dev eth0 parent 1:0 protocol ip prio 1 tcindex <Emphasis remap="bf">mask 0xfc shift 2</Emphasis>
7400 tc qdisc add dev eth0 parent 1:0 handle 2:0 cbq bandwidth 10Mbit cell 8 avpkt 1000 mpu 64
7402 # EF traffic class
7404 tc class add dev eth0 parent 2:0 classid 2:1 cbq bandwidth 10Mbit rate 1500Kbit avpkt 1000 prio 1 bounded isolated allot 1514 weight 1 maxburst 10
7406 # Packet fifo qdisc for EF traffic
7408 tc qdisc add dev eth0 parent 2:1 pfifo limit 5
7410 tc filter add dev eth0 parent 2:0 protocol ip prio 1 <Emphasis remap="bf">handle 0x2e</Emphasis> tcindex <Emphasis remap="bf">classid 2:1 pass_on</Emphasis>
7414 (This code is not complete. It's just an extract from EFCBQ example included in iproute2 distribution).
7418 First of all, suppose we receive a packet marked as EF . If you read RFC2598, you'll see that DSCP
7419 recommended value for EF traffic is 101110. This means that DS field will be 10111000 (remember that
7420 less significant bits in TOS byte are not used in DS) or 0xb8 in hexadecimal codification.
7428 +---+ +-------+ +---+-+ +------+ +-+ +-------+
7429 | | | | | | | |FILTER| +-+ +-+ | | | |
7430 | |----->| MASK | -> | | | -> |HANDLE|->| | | | -> | | -> | |
7431 | | . | =0xfc | | | | |0x2E | | +----+ | | | | |
7432 | | . | | | | | +------+ +--------+ | | | |
7433 | | . | | | | | | | | |
7434 -->| | . | SHIFT | | | | | | | |-->
7435 | | . | =2 | | | +----------------------------+ | | |
7436 | | | | | | CBQ 2:0 | | |
7437 | | +-------+ +---+--------------------------------+ | |
7439 | +-------------------------------------------------------------+ |
7441 +-------------------------------------------------------------------------+
7448 The packet arrives, then, set with 0xb8 value at DS field. As we explained before, dsmark qdisc identified
7449 by 1:0 id in the example, retrieves DS field and store it in skb->tc_index variable.
7450 Next step in the example will correspond to the filter associated to this qdisc (second line in the example).
7451 This will perform next operations:
7454 Value1 = skb->tc_index & MASK
7455 Key = Value1 >> SHIFT
7461 In the example, MASK=0xFC and SHIFT=2.
7464 Value1 = 10111000 & 11111100 = 10111000
7465 Key = 10111000 >> 2 = 00101110 -> 0x2E in hexadecimal
7471 The returned value will correspond to a qdisc internal filter handle (in the example, identifier 2:0). If a
7472 filter with this id exists, policing and metering conditions will be verified (in case that filter includes this)
7473 and the classid will be returned (in our example, classid 2:1) and stored in skb->tc_index variable.
7477 But if any filter with that identifier is not found, the result will depend on fall_through flag declaration. If so,
7478 value key is returned as classid. If not, an error is returned and process continues with the rest filters. Be
7479 careful if you use fall_through flag; this can be done if a simple relation exists between values
7481 of skb->tc_index variable and class id's.
7482 <!--what "this" here means?-->
7486 The latest parameters to comment on are hash and pass_on. The first one
7487 relates to hash table size. Pass_on will be used to indicate that if no classid
7488 equal to the result of this filter is found, try next filter.
7489 The default action is fall_through (look at next table).
7493 Finally, let's see which possible values can be set to all this TCINDEX parameters:
7496 TC Name Value Default
7497 -----------------------------------------------------------------
7498 Hash 1...0x10000 Implementation dependent
7499 Mask 0...0xffff 0xffff
7501 Fall through / Pass_on Flag Fall_through
7502 Classid Major:minor None
7509 This kind of filter is very powerful. It's necessary to explore all possibilities. Besides, this filter is not only used in DiffServ configurations.
7510 You can use it as any other kind of filter.
7514 I recommend you to look at all DiffServ examples included in iproute2 distribution. I promise I will try to
7515 complement this text as soon as I can. Besides, all I have explained is the result of a lot of tests.
7516 I would thank you tell me if I'm wrong in any point.
7523 <Sect1 id="lartc.adv-qdisc.ingress">
7524 <Title>Ingress qdisc</Title>
7527 All qdiscs discussed so far are egress qdiscs. Each interface however can
7528 also have an ingress qdisc which is not used to send packets
7529 out to the network adaptor. Instead, it allows you to apply tc filters to
7530 packets coming in over the interface, regardless of whether they have a local
7531 destination or are to be forwarded.
7535 As the tc filters contain a full Token Bucket Filter implementation, and are
7536 also able to match on the kernel flow estimator, there is a lot of
7537 functionality available. This effectively allows you to police incoming
7538 traffic, before it even enters the IP stack.
7542 <Title>Parameters & usage</Title>
7545 The ingress qdisc itself does not require any parameters. It differs from
7546 other qdiscs in that it does not occupy the root of a device. Attach it like
7550 # tc qdisc add dev eth0 ingress
7553 This allows you to have other, sending, qdiscs on your device besides the
7558 For a contrived example how the ingress qdisc could be used, see the
7566 <Sect1 id="lartc.adv-qdisc.red">
7567 <Title>Random Early Detection (RED)</Title>
7570 This section is meant as an introduction to the queuing at backbone networks, which often
7571 involves >100 megabit bandwidths, which requires a different approach than
7572 your ADSL modem at home.
7576 The normal behaviour of router queues on the Internet is called tail-drop.
7577 Tail-drop works by queueing up to a certain amount, then dropping all traffic
7578 that 'spills over'. This is very unfair, and also leads to retransmit
7579 synchronization. When retransmit synchronization occurs, the sudden burst
7580 of drops from a router that has reached its fill will cause a delayed burst
7581 of retransmits, which will over fill the congested router again.
7585 In order to cope with transient congestion on links, backbone routers will
7586 often implement large queues. Unfortunately, while these queues are good for
7587 throughput, they can substantially increase latency and cause TCP
7588 connections to behave very burstily during congestion.
7592 These issues with tail-drop are becoming increasingly troublesome on the
7593 Internet because the use of network unfriendly applications is increasing.
7594 The Linux kernel offers us RED, short for Random Early Detect, also called
7595 Random Early Drop, as that is how it works.
7599 RED isn't a cure-all for this, applications which inappropriately fail to
7600 implement exponential backoff still get an unfair share of the bandwidth,
7601 however, with RED they do not cause as much harm to the throughput and
7602 latency of other connections.
7606 RED statistically drops packets from flows before it reaches its hard
7607 limit. This causes a congested backbone link to slow more gracefully, and
7608 prevents retransmit synchronization. This also helps TCP find its 'fair'
7609 speed faster by allowing some packets to get dropped sooner keeping queue
7610 sizes low and latency under control. The probability of a packet being
7611 dropped from a particular connection is proportional to its bandwidth usage
7612 rather than the number of packets it transmits.
7616 RED is a good queue for backbones, where you can't afford the
7617 complexity of per-session state tracking needed by fairness queueing.
7621 In order to use RED, you must decide on three parameters: Min, Max, and
7622 burst. Min sets the minimum queue size in bytes before dropping will begin,
7623 Max is a soft maximum that the algorithm will attempt to stay under, and
7624 burst sets the maximum number of packets that can 'burst through'.
7628 You should set the min by calculating that highest acceptable base queueing
7629 latency you wish, and multiply it by your bandwidth. For instance, on my
7630 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set
7631 min to 1600 bytes. Setting min too small will degrade throughput and too
7632 large will degrade latency. Setting a small min is not a replacement for
7633 reducing the MTU on a slow link to improve interactive response.
7637 You should make max at least twice min to prevent synchronization. On slow
7638 links with small Min's it might be wise to make max perhaps four or
7639 more times large then min.
7643 Burst controls how the RED algorithm responds to bursts. Burst must be set
7644 larger then min/avpkt. Experimentally, I've found (min+min+max)/(3*avpkt) to
7649 Additionally, you need to set limit and avpkt. Limit is a safety value, after
7650 there are limit bytes in the queue, RED 'turns into' tail-drop. I typical set
7651 limit to eight times max. Avpkt should be your average packet size. 1000
7652 works OK on high speed Internet links with a 1500byte MTU.
7657 URL="http://www.aciri.org/floyd/papers/red/red.html"
7658 >the paper on RED queueing</ULink
7659 > by Sally Floyd and Van Jacobson for technical
7665 <Sect1 id="lartc.adv-qdisc.gred">
7666 <Title>Generic Random Early Detection</Title>
7669 Not a lot is known about GRED. It looks like GRED with several internal
7670 queues, whereby the internal queue is chosen based on the Diffserv tcindex
7671 field. According to a slide found
7672 <ULink URL="http://www.davin.ottawa.on.ca/ols/img22.htm">here</ULink>,
7673 it contains the capabilities of Cisco's 'Distributed Weighted RED', as well
7674 as Dave Clark's RIO.
7678 Each virtual queue can have its own Drop Parameters specified.
7682 FIXME: get Jamal or Werner to tell us more
7687 <Sect1 id="lartc.adv-qdisc.vc-atm">
7688 <Title>VC/ATM emulation</Title>
7691 This is quite a major effort by Werner Almesberger to allow you to build
7692 Virtual Circuits over TCP/IP sockets. A Virtual Circuit is a concept from
7697 For more information, see the <ULink
7698 URL="http://linux-atm.sourceforge.net/"
7699 >ATM on Linux homepage</ULink
7705 <Sect1 id="lartc.adv-qdisc.wrr">
7706 <Title>Weighted Round Robin (WRR)</Title>
7709 This qdisc is not included in the standard kernels but can be downloaded from
7710 <ULink URL="http://wipl-wrr.dkik.dk/wrr/">here</ULink>.
7711 Currently the qdisc is only tested with Linux 2.2 kernels but it will
7712 probably work with 2.4/2.5 kernels too.
7716 The WRR qdisc distributes bandwidth between its classes using the weighted
7717 round robin scheme. That is, like the CBQ qdisc it contains classes
7718 into which arbitrary qdiscs can be plugged. All classes which have sufficient
7719 demand will get bandwidth proportional to the weights associated with the classes.
7720 The weights can be set manually using the <Literal remap="tt">tc</Literal> program. But they
7721 can also be made automatically decreasing for classes transferring much data.
7725 The qdisc has a built-in classifier which assigns packets coming from or
7726 sent to different machines to different classes. Either the MAC or IP and
7727 either source or destination addresses can be used. The MAC address can only
7728 be used when the Linux box is acting as an ethernet bridge, however. The
7729 classes are automatically assigned to machines based on the packets seen.
7733 The qdisc can be very useful at sites such as dorms where a lot of unrelated
7734 individuals share an Internet connection. A set of scripts setting up a
7735 relevant behavior for such a site is a central part of the WRR distribution.
7742 <chapter id="lartc.cookbook"
7743 xreflabel="Cookbook">
7744 <Title>Cookbook</Title>
7747 This section contains 'cookbook' entries which may help you solve problems.
7748 A cookbook is no replacement for understanding however, so try and comprehend
7752 <Sect1 id="lartc.cookbook.sla">
7753 <Title>Running multiple sites with different SLAs</Title>
7756 You can do this in several ways. Apache has some support for this with a
7757 module, but we'll show how Linux can do this for you, and do so for other
7758 services as well. These commands are stolen from a presentation by Jamal
7759 Hadi that's referenced below.
7763 Let's say we have two customers, with http, ftp and streaming audio, and we
7764 want to sell them a limited amount of bandwidth. We do so on the server itself.
7768 Customer A should have at most 2 megabits, customer B has paid for 5
7769 megabits. We separate our customers by creating virtual IP addresses on our
7776 # ip address add 188.177.166.1 dev eth0
7777 # ip address add 188.177.166.2 dev eth0
7783 It is up to you to attach the different servers to the right IP address. All
7784 popular daemons have support for this.
7788 We first attach a CBQ qdisc to eth0:
7791 # tc qdisc add dev eth0 root handle 1: cbq bandwidth 10Mbit cell 8 avpkt 1000 \
7798 We then create classes for our customers:
7804 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 10Mbit rate \
7805 2MBit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
7806 # tc class add dev eth0 parent 1:0 classid 1:2 cbq bandwidth 10Mbit rate \
7807 5Mbit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
7813 Then we add filters for our two classes:
7816 ##FIXME: Why this line, what does it do?, what is a divisor?:
7817 ##FIXME: A divisor has something to do with a hash table, and the number of
7819 # tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 1: u32 divisor 1
7820 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.1
7822 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.2
7833 FIXME: why no token bucket filter? is there a default pfifo_fast fallback
7839 <Sect1 id="lartc.cookbook.synflood-protect"
7840 xreflabel="Protecting your host from SYN floods">
7841 <Title>Protecting your host from SYN floods</Title>
7844 From Alexey's iproute documentation, adapted to netfilter and with more
7845 plausible paths. If you use this, take care to adjust the numbers to
7846 reasonable values for your system.
7850 If you want to protect an entire network, skip this script, which is best
7851 suited for a single host.
7855 It appears that you need the very latest version of the iproute2 tools to
7856 get this to work with 2.4.0.
7864 # sample script on using the ingress capabilities
7865 # this script shows how one can rate limit incoming SYNs
7866 # Useful for TCP-SYN attack protection. You can use
7867 # IPchains to have more powerful additions to the SYN (eg
7868 # in addition the subnet)
7870 #path to various utilities;
7871 #change to reflect yours.
7875 IPTABLES=/sbin/iptables
7878 # tag all incoming SYN packets through $INDEV as mark value 1
7879 ############################################################
7880 $iptables -A PREROUTING -i $INDEV -t mangle -p tcp --syn \
7881 -j MARK --set-mark 1
7882 ############################################################
7884 # install the ingress qdisc on the ingress interface
7885 ############################################################
7886 $TC qdisc add dev $INDEV handle ffff: ingress
7887 ############################################################
7891 # SYN packets are 40 bytes (320 bits) so three SYNs equals
7892 # 960 bits (approximately 1kbit); so we rate limit below
7893 # the incoming SYNs to 3/sec (not very useful really; but
7894 #serves to show the point - JHS
7895 ############################################################
7896 $TC filter add dev $INDEV parent ffff: protocol ip prio 50 handle 1 fw \
7897 police rate 1kbit burst 40 mtu 9k drop flowid :1
7898 ############################################################
7902 echo "---- qdisc parameters Ingress ----------"
7903 $TC qdisc ls dev $INDEV
7904 echo "---- Class parameters Ingress ----------"
7905 $TC class ls dev $INDEV
7906 echo "---- filter parameters Ingress ----------"
7907 $TC filter ls dev $INDEV parent ffff:
7909 #deleting the ingress qdisc
7910 #$TC qdisc del $INDEV ingress
7917 <Sect1 id="lartc.cookbook.icmp-ratelimit">
7918 <Title>Rate limit ICMP to prevent dDoS</Title>
7921 Recently, distributed denial of service attacks have become a major nuisance
7922 on the Internet. By properly filtering and rate limiting your network, you can
7923 both prevent becoming a casualty or the cause of these attacks.
7927 You should filter your networks so that you do not allow non-local IP source
7928 addressed packets to leave your network. This stops people from anonymously
7929 sending junk to the Internet.
7933 Rate limiting goes much as shown earlier. To refresh your memory, our
7940 [The Internet] ---<E3, T3, whatever>--- [Linux router] --- [Office+ISP]
7947 We first set up the prerequisite parts:
7953 # tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000
7954 # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate \
7955 10Mbit allot 1514 prio 5 maxburst 20 avpkt 1000
7961 If you have 100Mbit, or more, interfaces, adjust these numbers. Now you need
7962 to determine how much ICMP traffic you want to allow. You can perform
7963 measurements with tcpdump, by having it write to a file for a while, and
7964 seeing how much ICMP passes your network. Do not forget to raise the
7969 If measurement is impractical, you might want to choose 5% of your available
7970 bandwidth. Let's set up our class:
7973 # tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate \
7974 100Kbit allot 1514 weight 800Kbit prio 5 maxburst 20 avpkt 250 \
7981 This limits at 100Kbit. Now we need a filter to assign ICMP traffic to this
7985 # tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip
7986 protocol 1 0xFF flowid 10:100
7994 <Sect1 id="lartc.cookbook.interactive-prio">
7995 <Title>Prioritizing interactive traffic</Title>
7998 If lots of data is coming down your link, or going up for that matter, and
7999 you are trying to do some maintenance via telnet or ssh, this may not go too
8000 well. Other packets are blocking your keystrokes. Wouldn't it be great if
8001 there were a way for your interactive packets to sneak past the bulk
8002 traffic? Linux can do this for you!
8006 As before, we need to handle traffic going both ways. Evidently, this works
8007 best if there are Linux boxes on both ends of your link, although other
8008 UNIX's are able to do this. Consult your local Solaris/BSD guru for this.
8012 The standard pfifo_fast scheduler has 3 different 'bands'. Traffic in band 0
8013 is transmitted first, after which traffic in band 1 and 2 gets considered.
8014 It is vital that our interactive traffic be in band 0!
8018 We blatantly adapt from the (soon to be obsolete) ipchains HOWTO:
8022 There are four seldom-used bits in the IP header, called the Type of Service
8023 (TOS) bits. They effect the way packets are treated; the four bits are
8024 "Minimum Delay", "Maximum Throughput", "Maximum Reliability" and "Minimum
8025 Cost". Only one of these bits is allowed to be set. Rob van Nieuwkerk, the
8026 author of the ipchains TOS-mangling code, puts it as follows:
8031 Especially the "Minimum Delay" is important for me. I switch it on for
8032 "interactive" packets in my upstream (Linux) router. I'm
8033 behind a 33k6 modem link. Linux prioritizes packets in 3 queues. This
8034 way I get acceptable interactive performance while doing bulk
8035 downloads at the same time.
8040 The most common use is to set telnet & ftp control connections to "Minimum
8041 Delay" and FTP data to "Maximum Throughput". This would be
8042 done as follows, on your upstream router:
8048 # iptables -A PREROUTING -t mangle -p tcp --sport telnet \
8049 -j TOS --set-tos Minimize-Delay
8050 # iptables -A PREROUTING -t mangle -p tcp --sport ftp \
8051 -j TOS --set-tos Minimize-Delay
8052 # iptables -A PREROUTING -t mangle -p tcp --sport ftp-data \
8053 -j TOS --set-tos Maximize-Throughput
8059 Now, this only works for data going from your telnet foreign host to your
8060 local computer. The other way around appears to be done for you, ie, telnet,
8061 ssh & friends all set the TOS field on outgoing packets automatically.
8065 Should you have an application that does not do this, you can always do it
8066 with netfilter. On your local box:
8072 # iptables -A OUTPUT -t mangle -p tcp --dport telnet \
8073 -j TOS --set-tos Minimize-Delay
8074 # iptables -A OUTPUT -t mangle -p tcp --dport ftp \
8075 -j TOS --set-tos Minimize-Delay
8076 # iptables -A OUTPUT -t mangle -p tcp --dport ftp-data \
8077 -j TOS --set-tos Maximize-Throughput
8084 <Sect1 id="lartc.cookbook.squid">
8085 <Title>Transparent web-caching using <application>netfilter</application>,
8086 <application>iproute2</application>, <application>ipchains</application> and
8087 <application>squid</application></Title>
8090 This section was sent in by reader Ram Narula from Internet for Education
8095 The regular technique in accomplishing this in Linux
8096 is probably with use of ipchains AFTER making sure
8097 that the "outgoing" port 80(web) traffic gets routed through
8098 the server running squid.
8102 There are 3 common methods to make sure "outgoing"
8103 port 80 traffic gets routed to the server running squid
8104 and 4th one is being introduced here.
8110 <Term>Making the gateway router do it.</Term>
8113 If you can tell your gateway router to
8114 match packets that has outgoing destination port
8115 of 80 to be sent to the IP address of squid server.
8123 This would put additional load on the router and
8124 some commercial routers might not even support this.
8129 <Term>Using a Layer 4 switch.</Term>
8132 Layer 4 switches can handle this without any problem.
8140 The cost for this equipment is usually very high. Typical
8141 layer 4 switch would normally cost more than
8142 a typical router+good linux server.
8147 <Term>Using cache server as network's gateway.</Term>
8150 You can force ALL traffic through cache server.
8156 This is quite risky because Squid does utilize lots of CPU power which might
8157 result in slower over-all network performance or the server itself might crash and no one on the
8158 network will be able to access the Internet if that occurs.
8163 <Term>Linux+NetFilter router.</Term>
8166 By using NetFilter another technique can be implemented
8167 which is using NetFilter for "mark"ing the packets
8168 with destination port 80 and using iproute2 to
8169 route the "mark"ed packets to the Squid server.
8180 10.0.0.1 naret (NetFilter server)
8181 10.0.0.2 silom (Squid server)
8182 10.0.0.3 donmuang (Router connected to the Internet)
8183 10.0.0.4 kaosarn (other server on network)
8185 10.0.0.0/24 main network
8186 10.0.0.0/19 total network
8196 ------------hub/switch----------
8198 naret silom kaosarn RAS etc.
8201 First, make all traffic pass through naret by making sure it is the default gateway except for silom.
8202 Silom's default gateway has to be donmuang (10.0.0.3) or this would create web traffic loop.
8205 (all servers on my network had 10.0.0.1 as the default gateway which was the former IP address of donmuang router so what I did
8206 was changed the IP address of donmuang to 10.0.0.3 and gave naret ip address of 10.0.0.1)
8212 -setup squid and ipchains
8216 Setup Squid server on silom, make sure it does support transparent caching/proxying, the default port is usually
8217 3128, so all traffic for port 80 has to be redirected to port 3128 locally. This can be done by using ipchains with the following:
8221 silom# ipchains -N allow1
8222 silom# ipchains -A allow1 -p TCP -s 10.0.0.0/19 -d 0/0 80 -j REDIRECT 3128
8223 silom# ipchains -I input -j allow1
8227 Or, in netfilter lingo:
8229 silom# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
8233 (note: you might have other entries as well)
8236 For more information on setting Squid server please refer to Squid FAQ page on <ULink
8237 URL="http://squid.nlanr.net">http://squid.nlanr.net</ULink>).
8240 Make sure ip forwarding is enabled on this server and the default gateway for this server is donmuang router (NOT naret).
8246 -setup iptables and iproute2
8247 -disable icmp REDIRECT messages (if needed)
8254 "Mark" packets of destination port 80 with value 2
8256 naret# iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 \
8257 -j MARK --set-mark 2
8263 Setup iproute2 so it will route packets with "mark" 2 to silom
8265 naret# echo 202 www.out >> /etc/iproute2/rt_tables
8266 naret# ip rule add fwmark 2 table www.out
8267 naret# ip route add default via 10.0.0.2 dev eth0 table www.out
8268 naret# ip route flush cache
8272 If donmuang and naret is on the same subnet then naret should not send out icmp REDIRECT messages.
8273 In this case it is, so icmp REDIRECTs has to be disabled by:
8275 naret# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
8276 naret# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
8277 naret# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
8284 The setup is complete, check the configuration
8290 naret# iptables -t mangle -L
8291 Chain PREROUTING (policy ACCEPT)
8292 target prot opt source destination
8293 MARK tcp -- anywhere anywhere tcp dpt:www MARK set 0x2
8295 Chain OUTPUT (policy ACCEPT)
8296 target prot opt source destination
8299 0: from all lookup local
8300 32765: from all fwmark 2 lookup www.out
8301 32766: from all lookup main
8302 32767: from all lookup default
8304 naret# ip route list table www.out
8305 default via 203.114.224.8 dev eth0
8308 10.0.0.1 dev eth0 scope link
8309 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.1
8310 127.0.0.0/8 dev lo scope link
8311 default via 10.0.0.3 dev eth0
8313 (make sure silom belongs to one of the above lines, in this case
8314 it's the line with 10.0.0.0/24)
8322 <Title>Traffic flow diagram after implementation</Title>
8326 |-----------------------------------------|
8327 |Traffic flow diagram after implementation|
8328 |-----------------------------------------|
8334 -----------------donmuang router---------------------
8339 *destination port 80 traffic=========>(cache) ||
8342 \\===================================kaosarn, RAS, etc.
8347 Note that the network is asymmetric as there is one extra hop on
8348 general outgoing path.
8352 Here is run down for packet traversing the network from kaosarn
8353 to and from the Internet.
8358 <Term>For web/http traffic</Term>
8361 kaosarn http request->naret->silom->donmuang->internet
8362 http replies from Internet->donmuang->silom->kaosarn
8368 <Term>For non-web/http requests(eg. telnet)</Term>
8371 kaosarn outgoing data->naret->donmuang->internet
8372 incoming data from Internet->donmuang->kaosarn
8383 <Sect1 id="lartc.cookbook.mtu-discovery">
8384 <Title>Circumventing Path MTU Discovery issues with per route MTU settings</Title>
8387 For sending bulk data, the Internet generally works better when using larger
8388 packets. Each packet implies a routing decision, when sending a 1 megabyte
8389 file, this can either mean around 700 packets when using packets that are as
8390 large as possible, or 4000 if using the smallest default.
8394 However, not all parts of the Internet support full 1460 bytes of payload
8395 per packet. It is therefore necessary to try and find the largest packet
8396 that will 'fit', in order to optimize a connection.
8400 This process is called 'Path MTU Discovery', where MTU stands for 'Maximum
8405 When a router encounters a packet that's too big too send in one piece, AND
8406 it has been flagged with the "Don't Fragment" bit, it returns an ICMP
8407 message stating that it was forced to drop a packet because of this. The
8408 sending host acts on this hint by sending smaller packets, and by iterating
8409 it can find the optimum packet size for a connection over a certain path.
8413 This used to work well until the Internet was discovered by hooligans who do
8414 their best to disrupt communications. This in turn lead administrators to
8415 either block or shape ICMP traffic in a misguided attempt to improve
8416 security or robustness of their Internet service.
8420 What has happened now is that Path MTU Discovery is working less and less
8421 well and fails for certain routes, which leads to strange TCP/IP sessions
8422 which die after a while.
8426 Although I have no proof for this, two sites who I used to have this problem
8427 with both run Alteon Acedirectors before the affected systems - perhaps
8428 somebody more knowledgeable can provide clues as to why this happens.
8432 <Title>Solution</Title>
8435 When you encounter sites that suffer from this problem, you can disable Path
8436 MTU discovery by setting it manually. Koos van den Hout, slightly edited,
8442 The following problem: I set the mtu/mru of my leased line running ppp to
8443 296 because it's only 33k6 and I cannot influence the queueing on the
8444 other side. At 296, the response to a key press is within a reasonable
8449 And, on my side I have a masqrouter running (of course) Linux.
8453 Recently I split 'server' and 'router' so most applications are run on a
8454 different machine than the routing happens on.
8458 I then had trouble logging into irc. Big panic! Some digging did find
8459 out that I got connected to irc, even showed up as 'connected' on irc
8460 but I did not receive the motd from irc. I checked what could be wrong
8461 and noted that I already had some previous trouble reaching certain
8462 websites related to the MTU, since I had no trouble reaching them when
8463 the MTU was 1500, the problem just showed when the MTU was set to 296.
8464 Since irc servers block about every kind of traffic not needed for their
8465 immediate operation, they also block icmp.
8469 I managed to convince the operators of a webserver that this was the cause
8470 of a problem, but the irc server operators were not going to fix this.
8474 So, I had to make sure outgoing masqueraded traffic started with the lower
8475 mtu of the outside link. But I want local ethernet traffic to have the
8476 normal mtu (for things like nfs traffic).
8483 ip route add default via 10.0.0.1 mtu 296
8487 (10.0.0.1 being the default gateway, the inside address of the
8488 masquerading router)
8493 In general, it is possible to override PMTU Discovery by setting specific
8494 routes. For example, if only a certain subnet is giving problems, this
8499 ip route add 195.96.96.0/24 via 10.0.0.1 mtu 1000
8506 <Sect1 id="lartc.cookbook.mtu-mss">
8507 <Title>Circumventing Path MTU Discovery issues with MSS Clamping
8508 (for ADSL, cable, PPPoE & PPtP users)</Title>
8511 As explained above, Path MTU Discovery doesn't work as well as it should
8512 anymore. If you know for a fact that a hop somewhere in your network has a
8513 limited (<1500) MTU, you cannot rely on PMTU Discovery finding this out.
8517 Besides MTU, there is yet another way to set the maximum packet size, the so
8518 called Maximum Segment Size. This is a field in the TCP Options part of a
8523 Recent Linux kernels, and a few PPPoE drivers (notably, the excellent
8524 Roaring Penguin one), feature the possibility to 'clamp the MSS'.
8528 The good thing about this is that by setting the MSS value, you are telling
8529 the remote side unequivocally 'do not ever try to send me packets bigger
8530 than this value'. No ICMP traffic is needed to get this to work.
8534 The bad thing is that it's an obvious hack - it breaks 'end to end' by
8535 modifying packets. Having said that, we use this trick in many places and it
8540 In order for this to work you need at least iptables-1.2.1a and Linux 2.4.3
8541 or higher. The basic command line is:
8544 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
8550 This calculates the proper MSS for your link. If you are feeling brave, or
8551 think that you know best, you can also do something like this:
8557 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
8563 This sets the MSS of passing SYN packets to 128. Use this if you have VoIP
8564 with tiny packets, and huge http packets which are causing chopping in your
8570 <Sect1 id="lartc.cookbook.ultimate-tc">
8571 <Title>The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads</Title>
8574 Note: This script has recently been upgraded and previously only worked for
8575 Linux clients in your network! So you might want to update if you have
8576 Windows machines or Macs in your network and noticed that they were not able
8577 to download faster while others were uploading.
8581 I attempted to create the holy grail:
8585 <Term>Maintain low latency for interactive traffic at all times</Term>
8588 This means that downloading or uploading files should not disturb SSH or
8589 even telnet. These are the most important things, even 200ms latency is
8590 sluggish to work over.
8594 <Term>Allow 'surfing' at reasonable speeds while up or downloading</Term>
8597 Even though http is 'bulk' traffic, other traffic should not drown it out
8602 <Term>Make sure uploads don't harm downloads, and the other way around</Term>
8605 This is a much observed phenomenon where outgress traffic simply destroys
8610 It turns out that all this is possible, at the cost of a tiny bit of
8611 bandwidth. The reason that uploads, downloads and ssh hurt each other is the
8612 presence of large queues in many domestic access devices like cable or DSL
8617 The next section explains in depth what causes the delays, and how we can
8618 fix them. You can safely skip it and head straight for the script if you
8619 don't care how the magic is performed.
8623 <Title>Why it doesn't work well by default</Title>
8626 ISPs know that they are benchmarked solely on how fast people can download.
8627 Besides available bandwidth, download speed is influenced heavily by packet
8628 loss, which seriously hampers TCP/IP performance. Large queues can help
8629 prevent packet loss, and speed up downloads. So ISPs configure large queues.
8633 These large queues however damage interactivity. A keystroke must first
8634 travel the upstream queue, which may be seconds (!) long and go to your
8635 remote host. It is then displayed, which leads to a packet coming back, which
8636 must then traverse the downstream queue, located at your ISP, before it
8637 appears on your screen.
8641 This HOWTO teaches you how to mangle and process the queue in many ways, but
8642 sadly, not all queues are accessible to us. The queue over at the ISP is
8643 completely off-limits, whereas the upstream queue probably lives inside your
8644 cable modem or DSL device. You may or may not be able to configure it. Most
8649 So, what next? As we can't control either of those queues, they must be
8650 eliminated, and moved to your Linux router. Luckily this is possible.
8657 <Term>Limit upload speed</Term>
8660 By limiting our upload speed to slightly less than the truly available rate,
8661 no queues are built up in our modem. The queue is now moved to Linux.
8665 <Term>Limit download speed</Term>
8668 This is slightly trickier as we can't really influence how fast the internet
8669 ships us data. We can however drop packets that are coming in too fast,
8670 which causes TCP/IP to slow down to just the rate we want. Because we don't
8671 want to drop traffic unnecessarily, we configure a 'burst' size we allow at
8679 Now, once we have done this, we have eliminated the downstream queue totally
8680 (except for short bursts), and gain the ability to manage the upstream queue
8681 with all the power Linux offers.
8685 What remains to be done is to make sure interactive traffic jumps to the
8686 front of the upstream queue. To make sure that uploads don't hurt downloads,
8687 we also move ACK packets to the front of the queue. This is what normally
8688 causes the huge slowdown observed when generating bulk traffic both ways.
8689 The ACKnowledgements for downstream traffic must compete with upstream
8690 traffic, and get delayed in the process.
8694 If we do all this we get the following measurements using an excellent ADSL
8695 connection from xs4all in the Netherlands:
8702 round-trip min/avg/max = 14.4/17.1/21.7 ms
8704 Without traffic conditioner, while downloading:
8705 round-trip min/avg/max = 560.9/573.6/586.4 ms
8707 Without traffic conditioner, while uploading:
8708 round-trip min/avg/max = 2041.4/2332.1/2427.6 ms
8710 With conditioner, during 220kbit/s upload:
8711 round-trip min/avg/max = 15.7/51.8/79.9 ms
8713 With conditioner, during 850kbit/s download:
8714 round-trip min/avg/max = 20.4/46.9/74.0 ms
8716 When uploading, downloads proceed at ~80% of the available speed. Uploads
8717 at around 90%. Latency then jumps to 850 ms, still figuring out why.
8723 What you can expect from this script depends a lot on your actual uplink
8724 speed. When uploading at full speed, there will always be a single packet
8725 ahead of your keystroke. That is the lower limit to the latency you can
8726 achieve - divide your MTU by your upstream speed to calculate. Typical
8727 values will be somewhat higher than that. Lower your MTU for better effects!
8731 Next, two versions of this script, one with Devik's excellent HTB, the other
8732 with CBQ which is in each Linux kernel, unlike HTB. Both are tested and work
8739 <Title>The actual script (CBQ)</Title>
8742 Works on all kernels. Within the CBQ
8743 qdisc we place two Stochastic Fairness Queues that make sure that multiple
8744 bulk streams don't drown each other out.
8748 Downstream traffic is policed using a tc filter containing a Token Bucket
8753 You might improve on this script by adding 'bounded' to the line that starts
8754 with 'tc class add .. classid 1:20'. If you lowered your MTU, also lower the
8755 allot & avpkt numbers!
8763 # The Ultimate Setup For Your Internet Connection At Home
8766 # Set the following values to somewhat less than your actual download
8767 # and uplink speed. In kilobits
8772 # clean existing down- and uplink qdiscs, hide errors
8773 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
8774 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
8780 tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
8782 # shape everything at $UPLINK speed - this prevents huge queues in your
8783 # DSL modem which destroy latency:
8786 tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit \
8787 allot 1500 prio 5 bounded isolated
8789 # high prio class 1:10:
8791 tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit \
8792 allot 1600 prio 1 avpkt 1000
8794 # bulk and default class 1:20 - gets slightly less traffic,
8795 # and a lower priority:
8797 tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit \
8798 allot 1600 prio 2 avpkt 1000
8800 # both get Stochastic Fairness:
8801 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
8802 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
8805 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
8806 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8807 match ip tos 0x10 0xff flowid 1:10
8809 # ICMP (ip protocol 1) in the interactive class 1:10 so we
8810 # can do measurements & impress our friends:
8811 tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \
8812 match ip protocol 1 0xff flowid 1:10
8814 # To speed up downloads while an upload is going on, put ACK packets in
8815 # the interactive class:
8817 tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \
8818 match ip protocol 6 0xff \
8819 match u8 0x05 0x0f at 0 \
8820 match u16 0x0000 0xffc0 at 2 \
8821 match u8 0x10 0xff at 33 \
8824 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
8826 tc filter add dev $DEV parent 1: protocol ip prio 13 u32 \
8827 match ip dst 0.0.0.0/0 flowid 1:20
8829 ########## downlink #############
8830 # slow downloads down to somewhat less than the real speed to prevent
8831 # queuing at our ISP. Tune to see how high you can set it.
8832 # ISPs tend to have *huge* queues to make sure big downloads are fast
8834 # attach ingress policer:
8836 tc qdisc add dev $DEV handle ffff: ingress
8838 # filter *everything* to it (0.0.0.0/0), drop everything that's
8839 # coming in too fast:
8841 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
8842 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
8845 If you want this script to be run by ppp on connect, copy it to
8850 If the last two lines give an error, update your tc tool to a newer version!
8856 <Title>The actual script (HTB)</Title>
8859 The following script achieves all goals using the wonderful HTB queue, see
8860 the relevant chapter. Well worth patching your kernel for!
8865 # The Ultimate Setup For Your Internet Connection At Home
8868 # Set the following values to somewhat less than your actual download
8869 # and uplink speed. In kilobits
8874 # clean existing down- and uplink qdiscs, hide errors
8875 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
8876 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
8880 # install root HTB, point default traffic to 1:20:
8882 tc qdisc add dev $DEV root handle 1: htb default 20
8884 # shape everything at $UPLINK speed - this prevents huge queues in your
8885 # DSL modem which destroy latency:
8887 tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k
8889 # high prio class 1:10:
8891 tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
8894 # bulk & default class 1:20 - gets slightly less traffic,
8895 # and a lower priority:
8897 tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
8900 # both get Stochastic Fairness:
8901 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
8902 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
8904 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
8905 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8906 match ip tos 0x10 0xff flowid 1:10
8908 # ICMP (ip protocol 1) in the interactive class 1:10 so we
8909 # can do measurements & impress our friends:
8910 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
8911 match ip protocol 1 0xff flowid 1:10
8913 # To speed up downloads while an upload is going on, put ACK packets in
8914 # the interactive class:
8916 tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
8917 match ip protocol 6 0xff \
8918 match u8 0x05 0x0f at 0 \
8919 match u16 0x0000 0xffc0 at 2 \
8920 match u8 0x10 0xff at 33 \
8923 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
8926 ########## downlink #############
8927 # slow downloads down to somewhat less than the real speed to prevent
8928 # queuing at our ISP. Tune to see how high you can set it.
8929 # ISPs tend to have *huge* queues to make sure big downloads are fast
8931 # attach ingress policer:
8933 tc qdisc add dev $DEV handle ffff: ingress
8935 # filter *everything* to it (0.0.0.0/0), drop everything that's
8936 # coming in too fast:
8938 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
8939 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
8945 If you want this script to be run by ppp on connect, copy it to
8950 If the last two lines give an error, update your tc tool to a newer version!
8956 <sect1 id="lartc.ratelimit.single"><title>Rate limiting a single host or netmask</title>
8958 Although this is described in stupendous details elsewhere and in our manpages, this question gets asked a lot and
8959 happily there is a simple answer that does not need full comprehension of traffic control.
8962 This three line script does the trick:
8966 tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
8968 tc class add dev $DEV parent 1: classid 1:1 cbq rate 512kbit \
8969 allot 1500 prio 5 bounded isolated
8971 tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
8972 match ip dst 195.96.96.97 flowid 1:1
8976 The first line installs a class based queue on your interface, and tells the kernel that for calculations,
8977 it can be assumed to be a 10mbit interface. If you get this wrong, no real harm is done. But getting it right will
8978 make everything more precise.
8981 The second line creates a 512kbit class with some reasonable defaults. For details, see the cbq manpages and
8982 <xref linkend="lartc.qdisc">.
8985 The last line tells which traffic should go to the shaped class. Traffic not matched by this rule is NOT shaped. To make more
8986 complicated matches (subnets, source ports, destination ports), see <xref linkend="lartc.filtering.simple">.
8989 If you changed anything and want to reload the script, execute 'tc qdisc del dev $DEV root' to clean up your existing
8993 The script can further be improved by adding a last optional line 'tc qdisc add dev $DEV parent 1:1 sfq perturb 10'. See
8994 <xref linkend="lartc.sfq"> for details on what this does.
8998 <sect1 id="lartc.cookbook.fullnat.intro"><title>Example of a full nat solution with QoS</title>
9001 <email>piotr%member.fsf.org</email>. Here I'm describing a common set up where we have lots of users in a private network connected to the Internet trough a Linux router with a public ip address that is doing network address translation (NAT). I use this QoS setup to give access to the Internet to 198 users in a university dorm, in which I live and I'm netadmin of. The users here do heavy use of peer to peer programs, so proper traffic control is a must. I hope this serves as a practical example for all interested lartc readers.
9005 At first I make a practical approach with step by step configuration, and in the end I explain how to make the process automatic at bootime. The network to which this example applies is a private LAN connected to the Internet through a Linux router which has one public ip address. Extending it to several public ip address should be very easy, a couple of iptables rules should be added.
9006 In order to get things working we need:
9009 <Term>Linux 2.4.18 or higher kernel version installed</Term>
9012 If you use 2.4.18 you will have to apply HTB patch available here.
9017 <Term>iproute</Term>
9020 Also ensure the "tc" binary is HTB ready, a precompiled binary is distributed with HTB.
9025 <Term>iptables</Term>
9036 <Title>Let's begin optimizing that scarce bandwidth</Title>
9038 First we set up some qdiscs in which we will classify the traffic. We create a htb qdisc with 6 classes with ascending priority. Then we have classes that will always get allocated rate, but can use the unused bandwidth that other classes don't need. Recall that classes with higher priority ( i.e with a lower prio number ) will get excess of bandwith allocated first. Our connection is 2Mb down 300kbits/s up Adsl. I use 240kbit/s as ceil rate just because it's the higher I can set it before latency starts to grow, due to buffer filling in whatever place between us and remote hosts. This parameter should be timed experimentally, raising and lowering it while observing latency between some near hosts.
9041 Adjust CEIL to 75% of your upstream bandwith limit by now, and where I use eth0, you should use the interface which has a public Internet address. To begin our example execute the following in a root shell:
9044 tc qdisc add dev eth0 root handle 1: htb default 15
9045 tc class add dev eth0 parent 1: classid 1:1 htb rate ${CEIL}kbit ceil ${CEIL}kbit
9046 tc class add dev eth0 parent 1:1 classid 1:10 htb rate 80kbit ceil 80kbit prio 0
9047 tc class add dev eth0 parent 1:1 classid 1:11 htb rate 80kbit ceil ${CEIL}kbit prio 1
9048 tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbit ceil ${CEIL}kbit prio 2
9049 tc class add dev eth0 parent 1:1 classid 1:13 htb rate 20kbit ceil ${CEIL}kbit prio 2
9050 tc class add dev eth0 parent 1:1 classid 1:14 htb rate 10kbit ceil ${CEIL}kbit prio 3
9051 tc class add dev eth0 parent 1:1 classid 1:15 htb rate 30kbit ceil ${CEIL}kbit prio 3
9052 tc qdisc add dev eth0 parent 1:12 handle 120: sfq perturb 10
9053 tc qdisc add dev eth0 parent 1:13 handle 130: sfq perturb 10
9054 tc qdisc add dev eth0 parent 1:14 handle 140: sfq perturb 10
9055 tc qdisc add dev eth0 parent 1:15 handle 150: sfq perturb 10
9057 We have just created a htb tree with one level depth. Something like this:
9064 +---------------------------------------+
9066 +---------------------------------------+
9068 +----+ +----+ +----+ +----+ +----+ +----+
9069 |1:10| |1:11| |1:12| |1:13| |1:14| |1:15|
9070 +----+ +----+ +----+ +----+ +----+ +----+
9074 <Term>classid 1:10 htb rate 80kbit ceil 80kbit prio 0</Term>
9077 This is the highest priority class. The packets in this class will have the lowest delay and would get the excess of bandwith first so it's a good idea to limit the ceil rate to this class. We will send through this class the following packets that benefit from low delay, such as interactive traffic: <emphasis>ssh, telnet, dns, quake3, irc, and packets with the SYN flag</emphasis>.
9083 <Term>classid 1:11 htb rate 80kbit ceil ${CEIL}kbit prio 1</Term>
9086 Here we have the first class in which we can start to put bulk traffic. In my example I have traffic from the local web server and requests for web pages: source port 80, and destination port 80 respectively.
9091 <Term>classid 1:12 htb rate 20kbit ceil ${CEIL}kbit prio 2</Term>
9094 In this class I will put traffic with Maximize-Throughput TOS bit set and the rest of the traffic that goes from <emphasis>local processes</emphasis> on the router to the Internet. So the following classes will only have traffic that is <quote>routed through</quote> the box.
9099 <Term>classid 1:13 htb rate 20kbit ceil ${CEIL}kbit prio 2</Term>
9102 This class is for the traffic of other NATed machines that need higher priority in their bulk traffic.
9108 <Term>classid 1:14 htb rate 10kbit ceil ${CEIL}kbit prio 3</Term>
9111 Here goes mail traffic (SMTP,pop3...) and packets with Minimize-Cost TOS bit set.
9116 <Term>classid 1:15 htb rate 30kbit ceil ${CEIL}kbit prio 3</Term>
9119 And finally here we have bulk traffic from the NATed machines behind the router. All kazaa, edonkey, and others will go here, in order to not interfere with other services.
9132 <Title>Classifying packets</Title>
9134 We have created the qdisc setup but no packet classification has been made, so now all outgoing packets are going out in class 1:15 ( because we used: tc qdisc add dev eth0 root handle 1: htb <emphasis>default 15</emphasis> ). Now we need to tell which packets go where. This is the most important part.
9138 Now we set the filters so we can classify the packets with iptables. I really prefer to do it with iptables, because they are very flexible and you have packet count for each rule. Also with the RETURN target packets don't need to traverse all rules. We execute the following commands:
9140 tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 1 fw classid 1:10
9141 tc filter add dev eth0 parent 1:0 protocol ip prio 2 handle 2 fw classid 1:11
9142 tc filter add dev eth0 parent 1:0 protocol ip prio 3 handle 3 fw classid 1:12
9143 tc filter add dev eth0 parent 1:0 protocol ip prio 4 handle 4 fw classid 1:13
9144 tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 5 fw classid 1:14
9145 tc filter add dev eth0 parent 1:0 protocol ip prio 6 handle 6 fw classid 1:15
9147 We have just told the kernel that packets that have a specific FWMARK value ( handle x fw ) go in the specified class ( classid x:x). Next you will see how to mark packets with iptables.
9151 First you have to understand how packet traverse the filters with iptables:
9153 +------------+ +---------+ +-------------+
9154 Packet -| PREROUTING |--- routing-----| FORWARD |-------+-------| POSTROUTING |- Packets
9155 input +------------+ decision +-Â-------+ | +-------------+ out
9157 +-------+ +--------+
9158 | INPUT |---- Local process -| OUTPUT |
9159 +-------+ +--------+
9162 I assume you have all your tables created and with default policy ACCEPT ( -P ACCEPT ) if you haven't poked with iptables yet, It should be ok by default. Ours private network is a class B with address 172.17.0.0/16 and public ip is 212.170.21.172
9166 Next we instruct the kernel to <emphasis>actually do NAT</emphasis>, so clients in the private network can start talking to the outside.
9169 echo 1 > /proc/sys/net/ipv4/ip_forward
9170 iptables -t nat -A POSTROUTING -s 172.17.0.0/255.255.0.0 -o eth0 -j SNAT --to-source 212.170.21.172
9173 Now check that packets are flowing through 1:15:
9176 tc -s class show dev eth0
9182 You can start marking packets adding rules to the PREROUTING chain in the mangle table.
9185 iptables -t mangle -A PREROUTING -p icmp -j MARK --set-mark 0x1
9186 iptables -t mangle -A PREROUTING -p icmp -j RETURN
9189 Now you should be able to see packet count increasing when pinging from machines within the private network to some site on the Internet. Check packet count increasing in 1:10
9191 tc -s class show dev eth0
9193 We have done a -j RETURN so packets don't traverse all rules. Icmp packets won't match other rules below RETURN. Keep that in mind.
9194 Now we can start adding more rules, lets do proper TOS handling:
9197 iptables -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j MARK --set-mark 0x1
9198 iptables -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j RETURN
9199 iptables -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j MARK --set-mark 0x5
9200 iptables -t mangle -A PREROUTING -m tos --tos Minimize-Cost -j RETURN
9201 iptables -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j MARK --set-mark 0x6
9202 iptables -t mangle -A PREROUTING -m tos --tos Maximize-Throughput -j RETURN
9205 Now prioritize ssh packets:
9207 iptables -t mangle -A PREROUTING -p tcp -m tcp --sport 22 -j MARK --set-mark 0x1
9208 iptables -t mangle -A PREROUTING -p tcp -m tcp --sport 22 -j RETURN
9210 A good idea is to prioritize packets to begin tcp connections, those with SYN flag set:
9212 iptables -t mangle -I PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1
9213 iptables -t mangle -I PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j RETURN
9217 When we are done adding rules to PREROUTING in mangle, we terminate the PREROUTING table with:
9218 <!-- sport/dport 80 examples? -->
9220 iptables -t mangle -A PREROUTING -j MARK --set-mark 0x6
9222 So previously unmarked traffic goes in 1:15. In fact this last step is unnecessary since default class was 1:15, but I will mark them in order to be consistent with the whole setup, and furthermore it's useful to see the counter in that rule.
9226 It will be a good idea to do the same in the OUTPUT rule, so repeat those commands with -A OUTPUT instead of PREROUTING. ( s/PREROUTING/OUTPUT/ ). Then traffic generated locally (on the Linux router) will also be classified. I finish OUTPUT chain with -j MARK --set-mark 0x3 so local traffic has higher priority.
9232 <Title>Improving our setup</Title>
9234 Now we have all our setup working. Take time looking at the graphs, and watching where your bandwith is spent and how do you want it. Doing that for lots of hours, I finally got the Internet connection working really well. Otherwise continuous timeouts and nearly zero allotment of bandwith to newly created tcp connections will occur.
9237 If you find that some classes are full most of the time it would be a good idea to attach another queueing discipline to them so bandwith sharing is more fair:
9239 tc qdisc add dev eth0 parent 1:13 handle 130: sfq perturb 10
9240 tc qdisc add dev eth0 parent 1:14 handle 140: sfq perturb 10
9241 tc qdisc add dev eth0 parent 1:15 handle 150: sfq perturb 10
9247 <Title>Making all of the above start at boot</Title>
9249 It sure can be done in many ways. In mine, I have a shell script in /etc/init.d/packetfilter that accepts [start | stop | stop-tables | start-tables | reload-tables] it configures qdiscs and loads needed kernel modules, so it behaves much like a daemon. The same script loads iptables rules from /etc/network/iptables-rules which can be saved with iptables-save and restored with iptables-restore.
9260 <chapter id="lartc.bridging">
9261 <Title>Building bridges, and pseudo-bridges with Proxy ARP</Title>
9264 Bridges are devices which can be installed in a network without any
9265 reconfiguration. A network switch is basically a many-port bridge. A bridge
9266 is often a 2-port switch. Linux does however support multiple interfaces in
9267 a bridge, making it a true switch.
9271 Bridges are often deployed when confronted with a broken network that needs
9272 to be fixed without any alterations. Because the bridge is a layer-2 device,
9273 one layer below IP, routers and servers are not aware of its existence.
9274 This means that you can transparently block or modify certain packets, or do
9279 Another good thing is that a bridge can often be replaced by a cross cable
9280 or a hub, should it break down.
9284 The bad news is that a bridge can cause great confusion unless it is very
9285 well documented. It does not appear in traceroutes, but somehow packets
9286 disappear or get changed from point A to point B ('this network is
9287 HAUNTED!'). You should also wonder if an organization that 'does not want to
9288 change anything' is doing the right thing.
9292 The Linux 2.4/2.5 bridge is documented on
9293 <ULink URL=" http://bridge.sourceforge.net/">this page</ULink>.
9296 <Sect1 id="lartc.bridging.iptables">
9297 <Title>State of bridging and iptables</Title>
9300 As of Linux 2.4.20, bridging and iptables do not 'see' each other without
9301 help. If you bridge packets from eth0 to eth1, they do not 'pass' by
9302 iptables. This means that you cannot do filtering, or NAT or mangling or
9303 whatever. In Linux 2.5.45 and higher, this is fixed.
9306 You may also see 'ebtables' mentioned which is yet another project - it
9307 allows you to do wild things as MACNAT and 'brouting'. It is truly scary.
9311 <Sect1 id="lartc.bridging.shaping">
9312 <Title>Bridging and shaping</Title>
9315 This does work as advertised. Be sure to figure out which side each
9316 interface is on, otherwise you might be shaping outbound traffic in your
9317 internal interface, which won't work. Use tcpdump if needed.
9322 <Sect1 id="lartc.bridging.proxy-arp">
9323 <Title>Pseudo-bridges with Proxy-ARP</Title>
9326 If you just want to implement a Pseudo-bridge, skip down a few sections
9327 to 'Implementing it', but it is wise to read a bit about how it works in
9332 A Pseudo-bridge works a bit differently. By default, a bridge passes packets
9333 unaltered from one interface to the other. It only looks at the hardware
9334 address of packets to determine what goes where. This in turn means that you
9335 can bridge traffic that Linux does not understand, as long as it has an
9336 hardware address it does.
9340 A 'Pseudo-bridge' works differently and looks more like a hidden router than
9341 a bridge, but like a bridge, it has little impact on network design.
9345 An advantage of the fact that it is not a bridge lies in the fact that
9346 packets really pass through the kernel, and can be filtered, changed,
9347 redirected or rerouted.
9351 A real bridge can also be made to perform these feats, but it needs special
9352 code, like the Ethernet Frame Diverter, or the above mentioned patch.
9356 Another advantage of a pseudo-bridge is that it does not pass packets it
9357 does not understand - thus cleaning your network of a lot of cruft. In cases
9358 where you need this cruft (like SAP packets, or Netbeui), use a real bridge.
9362 <Title>ARP & Proxy-ARP</Title>
9365 When a host wants to talk to another host on the same physical network
9366 segment, it sends out an Address Resolution Protocol packet, which, somewhat
9367 simplified, reads like this 'who has 10.0.0.1, tell 10.0.0.7'. In response
9368 to this, 10.0.0.1 replies with a short 'here' packet.
9372 10.0.0.7 then sends packets to the hardware address mentioned in the 'here'
9373 packet. It caches this hardware address for a relatively long time, and
9374 after the cache expires, it re-asks the question.
9378 When building a Pseudo-bridge, we instruct the bridge to reply to these ARP
9379 packets, which causes the hosts in the network to send its packets to the
9380 bridge. The bridge then processes these packets, and sends them to the
9385 So, in short, whenever a host on one side of the bridge asks for the
9386 hardware address of a host on the other, the bridge replies with a packet
9387 that says 'hand it to me'.
9391 This way, all data traffic gets transmitted to the right place, and always
9392 passes through the bridge.
9398 <Title>Implementing it</Title>
9401 In the bad old days, it used to be possible to instruct the Linux Kernel to
9402 perform 'proxy-ARP' for just any subnet. So, to configure a pseudo-bridge,
9403 you would have to specify both the proper routes to both sides of the bridge
9404 AND create matching proxy-ARP rules. This is bad in that it requires a lot
9405 of typing, but also because it easily allows you to make mistakes which make
9406 your bridge respond to ARP queries for networks it does not know how to
9411 With Linux 2.4/2.5 (and possibly 2.2), this possibility has been withdrawn and
9412 has been replaced by a flag in the /proc directory, called 'proxy_arp'. The
9413 procedure for building a pseudo-bridge is then:
9422 Assign an IP address to both interfaces, the 'left' and the 'right'
9429 Create routes so your machine knows which hosts reside on the left,
9430 and which on the right
9436 Turn on proxy-ARP on both interfaces, echo 1 >
9437 /proc/sys/net/ipv4/conf/ethL/proxy_arp, echo 1 >
9438 /proc/sys/net/ipv4/conf/ethR/proxy_arp, where L and R stand for the numbers
9439 of your interfaces on the left and on the right side
9448 Also, do not forget to turn on the ip_forwarding flag! When converting from
9449 a true bridge, you may find that this flag was turned off as it is not
9450 needed when bridging.
9454 Another thing you might note when converting is that you need to clear the
9455 arp cache of computers in the network - the arp cache might contain old
9456 pre-bridge hardware addresses which are no longer correct.
9460 On a Cisco, this is done using the command 'clear arp-cache', under
9461 Linux, use 'arp -d ip.address'. You can also wait for the cache to expire
9462 manually, which can take rather long.
9465 You can speed this up using the wonderful 'arping' tool, which on many
9466 distributions is part of the 'iputils' package. Using 'arping' you can send
9467 out unsolicited ARP messages so as to update remote arp caches.
9470 This is a very powerful technique that is also used by 'black hats' to
9471 subvert your routing!
9475 On Linux 2.4, you may need to execute
9476 'echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind' before being able to send
9477 out unsolicited ARP messages!
9481 You may also discover that your network was misconfigured if you are/were of
9482 the habit of specifying routes without netmasks. To explain, some versions
9483 of route may have guessed your netmask right in the past, or guessed wrong
9484 without you noticing. When doing surgical routing like described above, it
9485 is *vital* that you check your netmasks!
9494 <chapter id="lartc.dynamic-routing">
9495 <Title>Dynamic routing - OSPF and BGP</Title>
9498 Once your network starts to get really big, or you start to consider 'the
9499 internet' as your network, you need tools which dynamically route your data.
9500 Sites are often connected to each other with multiple links, and more are
9501 popping up all the time.
9505 The Internet has mostly standardized on OSPF (RFC 2328) and BGP4 (RFC 1771).
9506 Linux supports both, by way of <application>gated</application> and
9507 <application>zebra</application>.
9511 While currently not within the scope of this document, we would like to
9512 point you to the definitive works:
9522 URL="http://www.cisco.com/univercd/cc/td/doc/cisintwk/idg4/nd2003.htm"
9523 >Designing large-scale IP Internetworks</ULink
9533 "OSPF. The anatomy of an Internet routing protocol"
9534 Addison Wesley. Reading, MA. 1998.
9538 Halabi has also written a good guide to OSPF routing design, but this
9539 appears to have been dropped from the Cisco web site.
9548 "Internet routing architectures"
9549 Cisco Press (New Riders Publishing). Indianapolis, IN. 1997.
9562 URL="http://www.cisco.com/univercd/cc/td/doc/cisintwk/ics/icsbgp4.htm"
9563 >Using the Border Gateway Protocol for interdomain routing</ULink
9568 Although the examples are Cisco-specific, they are remarkably similar
9569 to the configuration language in Zebra :-)
9572 <Sect1 id="lartc.dynamic-routing.ospf">
9575 <FirstName>Pedro</FirstName><Surname>Larroy Tovar</Surname>
9578 <email>piotr%member.fsf.org</email>
9582 </sect1info><Title>Setting up OSPF with Zebra</Title>
9585 Please, let <ulink url="mailto:piotr%member.fsf.org">me</ulink> know if any of the following information is not accurate or if you have any suggestions.
9586 <ulink url="http://www.zebra.org">Zebra</ulink> is a great dynamic routing software written by Kunihiro Ishiguro, Toshiaki Takada and Yasuhiro Ohara. With Zebra, setting up OSPF is fast an simple, but in practice there's a lot of parameters to tune if you have very specific needs. OSPF stands for Open Shortest Path First, and some of its principal features are:
9589 <Term>Hierachical</Term>
9592 Networks are grouped by <emphasis>areas</emphasis>, which are interconnected by a <emphasis>backbone area</emphasis> which will be designated as <emphasis>area 0</emphasis>. All traffic goes through area 0, and all the routers in area 0 have routing information about all the other areas.
9598 <Term>Short convergence</Term>
9601 Routes are propagated very fast, compared with RIP, for example.
9606 <Term>Bandwith efficient</Term>
9609 Uses multicasting instead of broadcasting, so it doesn't flood other hosts with routing information that may not be of interest for them, thus reducing network overhead. Also, <emphasis>Internal Routers</emphasis> (those which only have interfaces in one area) don't have routing information about other areas. Routers with interfaces in more than one area are called <emphasis>Area Border Routers</emphasis>, and hold topological information about the areas they are connected to.
9614 <Term>Cpu intensive</Term>
9617 OSPF is based on Dijkstra's <ulink url="http://www.soi.wide.ad.jp/class/99007/slides/13/07.html">Shortest Path First algorithm</ulink>, which is expensive compared to other routing algorithms. But really is not that bad, since the Shortest Path is only
9618 calculated for each area, also for small to medium sized networks this won't be an issue, and you won't even notice.
9623 <Term>Link state</Term>
9626 OSPF counts with the special characteristics of networks and interfaces, such as bandwith, link failures, and monetary cost.
9631 <Term>Open protocol and GPLed software</Term>
9634 OSPF is an open protocol, and Zebra is GPL software, which has obvious advantages over propietary software and protocols.
9641 <Sect2 id="lartc.dynamic-routing.ospf.prereq">
9642 <Title>Prerequisites</Title>
9648 <Term>Linux Kernel:</Term>
9651 Compiled with CONFIG_NETLINK_DEV and CONFIG_IP_MULTICAST (I am not sure if anything more is also needed).
9656 <Term>Iproute</Term>
9666 Get it with your favorite package manager or from <ulink url="http://www.zebra.org">http://www.zebra.org</ulink>.
9672 <Sect2 id="lartc.dynamic-routing.ospf.zebracfg">
9673 <Title>Configuring Zebra</Title>
9675 Let's take this network as an example:
9677 ----------------------------------------------------
9680 | Area 0 100BaseTX Switched |
9681 | Backbone Ethernet |
9682 ----------------------------------------------------
9686 |100BaseTX |100BaseTX |100BaseTX |100BaseTX
9688 --------- ------------ ----------- ----------------
9689 |R Omega| |R Atlantis| |R Legolas| |R Frodo |
9690 --------- ------------ ----------- ----------------
9693 |2MbDSL/ATM |100BaseTX |10BaseT |10BaseT |10BaseT
9694 ------------ ------------------------------------ -------------------------------
9695 | Internet | | 172.17.0.0/16 Area 1 | | 192.168.1.0/24 wlan Area 2|
9696 ------------ | Student network (dorm) | | barcelonawireless |
9697 ------------------------------------ -------------------------------
9699 Don't be afraid by this diagram, zebra does most of the work automatically, so it won't take any work to put all the routes up with zebra. It would be painful to maintain all those routes by hand in a day to day basis. The most important thing you must make clear, is the network topology. And take special care with Area 0, since it's the most important.
9700 First configure zebra, editing zebra.conf and adapt it to your needs:
9706 ! Interface's description.
9709 ! description test of desc.
9714 ! Static default route
9716 ip route 0.0.0.0/0 212.170.21.129
9718 log file /var/log/zebra/zebra.log
9720 In Debian, I will also have to edit /etc/zebra/daemons so they start at boot:
9725 Now we have to edit ospfd.conf if you are still running IPV4 or ospf6d.conf if you run IPV6. My ospfd.conf looks like:
9732 network 192.168.0.0/24 area 0
9733 network 172.17.0.0/16 area 1
9736 log file /var/log/zebra/ospfd.log
9738 Here we instruct ospf about our network topology.
9742 <Sect2 id="lartc.dynamic-routing.ospf.running">
9743 <Title>Running Zebra</Title>
9745 Now, we have to start Zebra; either by hand by typing "zebra -d" or with some script like "/etc/init.d/zebra start". Then carefully watching the ospdfd logs we should see something like:
9747 2002/12/13 22:46:24 OSPF: interface 192.168.0.1 join AllSPFRouters Multicast group.
9748 2002/12/13 22:46:34 OSPF: SMUX_CLOSE with reason: 5
9749 2002/12/13 22:46:44 OSPF: SMUX_CLOSE with reason: 5
9750 2002/12/13 22:46:54 OSPF: SMUX_CLOSE with reason: 5
9751 2002/12/13 22:47:04 OSPF: SMUX_CLOSE with reason: 5
9752 2002/12/13 22:47:04 OSPF: DR-Election[1st]: Backup 192.168.0.1
9753 2002/12/13 22:47:04 OSPF: DR-Election[1st]: DR 192.168.0.1
9754 2002/12/13 22:47:04 OSPF: DR-Election[2nd]: Backup 0.0.0.0
9755 2002/12/13 22:47:04 OSPF: DR-Election[2nd]: DR 192.168.0.1
9756 2002/12/13 22:47:04 OSPF: interface 192.168.0.1 join AllDRouters Multicast group.
9757 2002/12/13 22:47:06 OSPF: DR-Election[1st]: Backup 192.168.0.2
9758 2002/12/13 22:47:06 OSPF: DR-Election[1st]: DR 192.168.0.1
9759 2002/12/13 22:47:06 OSPF: Packet[DD]: Negotiation done (Slave).
9760 2002/12/13 22:47:06 OSPF: nsm_change_status(): scheduling new router-LSA origination
9761 2002/12/13 22:47:11 OSPF: ospf_intra_add_router: Start
9763 Ignore the SMUX_CLOSE message by now, since it's about SNMP. We can see that 192.168.0.1 is the <emphasis>Designated Router</emphasis> and 192.168.0.2 is the <emphasis>Backup Designated Router</emphasis>
9767 We can also interact with the zebra or the ospfd interface by executing:
9769 <prompt>$ </prompt>telnet localhost zebra
9770 <prompt>$ </prompt>telnet localhost ospfd
9773 Let's see how to view if the routes are propagating, log into zebra and type:
9776 root@atlantis:~# telnet localhost zebra
9778 Connected to atlantis.
9779 Escape character is '^]'.
9781 Hello, this is zebra (version 0.92a).
9782 Copyright 1996-2001 Kunihiro Ishiguro.
9784 User Access Verification
9787 atlantis> show ip route
9788 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
9789 B - BGP, > - selected route, * - FIB route
9791 K>* 0.0.0.0/0 via 192.168.0.1, eth1
9792 C>* 127.0.0.0/8 is directly connected, lo
9793 O 172.17.0.0/16 [110/10] is directly connected, eth0, 06:21:53
9794 C>* 172.17.0.0/16 is directly connected, eth0
9795 O 192.168.0.0/24 [110/10] is directly connected, eth1, 06:21:53
9796 C>* 192.168.0.0/24 is directly connected, eth1
9797 atlantis> show ip ospf border-routers
9798 ============ OSPF router routing table =============
9799 R 192.168.0.253 [10] area: (0.0.0.0), ABR
9800 via 192.168.0.253, eth1
9801 [10] area: (0.0.0.1), ABR
9802 via 172.17.0.2, eth0
9804 Or with iproute directly:
9806 root@omega:~# ip route
9807 212.170.21.128/26 dev eth0 proto kernel scope link src 212.170.21.172
9808 192.168.0.0/24 dev eth1 proto kernel scope link src 192.168.0.1
9809 172.17.0.0/16 via 192.168.0.2 dev eth1 proto zebra metric 20
9810 default via 212.170.21.129 dev eth0 proto zebra
9813 We can see the zebra routes, that weren't there before. It's really nice to see routes appearing just a few seconds after you start zebra and ospfd. You can check connectivity to other hosts with ping. Zebra routes are automatic, you can just add another router to the network, configure zebra, and voila!
9819 tcpdump -i eth1 ip[9] == 89
9821 To capture OSPF packets for analysis. OSPF ip protocol number is 89, and the protocol field is the 9th octet on the ip header.
9825 OSPF has a lot of tunable parameters, specially for large networks. In further ampliations of the howto we will show some methodologies for fine tunning OSPF.
9830 <Sect1 id="lartc.dynamic-routing.bgp">
9831 <Title>Setting up BGP4 with Zebra</Title>
9834 The Border Gateway Protocol Version 4 (BGP4) is a dynamic routing
9835 protocol described in RFC 1771. It allows the distribution of
9836 reachability information, i.e. routing tables, to other BGP4
9837 enabled nodes. It can either be used as EGP or IGP, in EGP mode
9838 each node must have its own Autonomous System (AS) number.
9839 BGP4 supports Classless Inter Domain Routing (CIDR) and route
9840 aggregation (merge multiple routes into one).
9843 <Sect2 id="lartc.dynamic-routing.bgp.netmap">
9844 <Title>Network Map (Example)</Title>
9847 The following network map is used for further examples. AS 1 and 50
9848 have more neighbors but we only need to configure 1 and 50 as our
9849 neighbor. The nodes itself communicate over tunnels in this example
9850 but that is not a must.
9854 Note: The AS numbers used in this example are reserved, please
9855 get your own AS from RIPE if you set up official peerings.
9859 --------------------
9860 | 192.168.23.12/24 |
9862 --------------------
9866 ------------------ ------------------
9867 | 192.168.1.1/24 |-------| 10.10.1.1/16 |
9868 | AS: 1 | | AS: 50 |
9869 ------------------ ------------------
9874 <Sect2 id="lartc.dynamic-routing.bgp.config">
9875 <Title>Configuration (Example)</Title>
9878 The following configuration is written for node 192.168.23.12/24,
9879 it is easy to adapt it for the other nodes.
9883 It starts with some general stuff like hostname, passwords and
9894 ! enable password (super user mode)
9898 log file /var/log/zebra/bgpd.log
9900 ! debugging: be verbose (can be removed afterwards)
9904 debug bgp keepalives
9909 Access list, used to limit the redistribution to
9910 private networks (RFC 1918).
9915 access-list local_nets permit 192.168.0.0/16
9916 access-list local_nets permit 172.16.0.0/12
9917 access-list local_nets permit 10.0.0.0/8
9918 access-list local_nets deny any
9922 Next step is to do the per AS configuration:
9929 ! IP address of the router
9930 bgp router-id 192.168.23.12
9932 ! announce our own network to other neighbors
9933 network 192.168.23.0/24
9935 ! advertise all connected routes (= directly attached interfaces)
9936 redistribute connected
9938 ! advertise kernel routes (= manually inserted routes)
9943 Every 'router bgp' block contains a list of neighbors to which
9944 the router is connected to:
9948 neighbor 192.168.1.1 remote-as 1
9949 neighbor 192.168.1.1 distribute-list local_nets in
9950 neighbor 10.10.1.1 remote-as 50
9951 neighbor 10.10.1.1 distribute-list local_nets in
9957 <Title>Checking Configuration</Title>
9960 Note: vtysh is a multiplexer and connects all the Zebra interfaces
9965 anakin# sh ip bgp summary
9966 BGP router identifier 192.168.23.12, local AS number 23
9967 2 BGP AS-PATH entries
9968 0 BGP community entries
9970 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
9971 10.10.0.1 4 50 35 40 0 0 0 00:28:40 1
9972 192.168.1.1 4 1 27574 27644 0 0 0 03:26:04 14
9974 Total number of neighbors 2
9976 anakin# sh ip bgp neighbors 10.10.0.1
9977 BGP neighbor is 10.10.0.1, remote AS 50, local AS 23, external link
9978 BGP version 4, remote router ID 10.10.0.1
9979 BGP state = Established, up for 00:29:01
9985 Let's see which routes we got from our neighbors:
9989 anakin# sh ip ro bgp
9990 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
9991 B - BGP, > - selected route, * - FIB route
9993 B>* 172.16.0.0/14 [20/0] via 192.168.1.1, tun0, 2d10h19m
9994 B>* 172.30.0.0/16 [20/0] via 192.168.1.1, tun0, 10:09:24
9995 B>* 192.168.5.10/32 [20/0] via 192.168.1.1, tun0, 2d10h27m
9996 B>* 192.168.5.26/32 [20/0] via 192.168.1.1, tun0, 10:09:24
9997 B>* 192.168.5.36/32 [20/0] via 192.168.1.1, tun0, 2d10h19m
9998 B>* 192.168.17.0/24 [20/0] via 192.168.1.1, tun0, 3d05h07m
9999 B>* 192.168.17.1/32 [20/0] via 192.168.1.1, tun0, 3d05h07m
10000 B>* 192.168.32.0/24 [20/0] via 192.168.1.1, tun0, 2d10h27m
10011 <chapter id="lartc.other"
10012 xreflabel="Other possibilities">
10013 <Title>Other possibilities</Title>
10016 This chapter is a list of projects having to do with advanced Linux routing
10017 & traffic shaping. Some of these links may deserve chapters of their
10018 own, some are documented very well of themselves, and don't need more HOWTO.
10025 <Term>802.1Q VLAN Implementation for Linux <ULink
10026 URL="http://scry.wanfear.com/~greear/vlan.html"
10031 VLANs are a very cool way to segregate your
10032 networks in a more virtual than physical way. Good information on VLANs can
10034 URL="ftp://ftp.netlab.ohio-state.edu/pub/jain/courses/cis788-97/virtual_lans/index.htm"
10036 >. With this implementation, you can have your Linux box talk
10037 VLANs with machines like Cisco Catalyst, 3Com: {Corebuilder, Netbuilder II,
10038 SuperStack II switch 630}, Extreme Ntwks Summit 48, Foundry: {ServerIronXL,
10043 A great HOWTO about VLANs can be found <ULink
10044 URL="http://scry.wanfear.com/~greear/vlan/cisco_howto.html"
10050 Update: has been included in the kernel as of 2.4.14 (perhaps 13).
10054 <Term>Alternate 802.1Q VLAN Implementation for Linux <ULink
10055 URL="http://vlan.sourceforge.net "
10060 Alternative VLAN implementation for linux. This project was started out of
10061 disagreement with the 'established' VLAN project's architecture and coding
10062 style, resulting in a cleaner overall design.
10066 <Term>Linux Virtual Server <ULink
10067 URL="http://www.LinuxVirtualServer.org/"
10072 These people are brilliant. The Linux Virtual Server is a highly scalable and
10073 highly available server built on a cluster of real servers, with the load
10074 balancer running on the Linux operating system. The architecture of the
10075 cluster is transparent to end users. End users only see a single virtual
10080 In short whatever you need to load balance, at whatever level of traffic, LVS
10081 will have a way of doing it. Some of their techniques are positively evil!
10082 For example, they let several machines have the same IP address on a
10083 segment, but turn off ARP on them. Only the LVS machine does ARP - it then
10084 decides which of the backend hosts should handle an incoming packet, and
10085 sends it directly to the right MAC address of the backend server. Outgoing
10086 traffic will flow directly to the router, and not via the LVS machine, which
10087 does therefore not need to see your 5Gbit/s of content flowing to the world,
10088 and cannot be a bottleneck.
10092 The LVS is implemented as a kernel patch in Linux 2.0 and 2.2, but as a
10093 Netfilter module in 2.4/2.5, so it does not need kernel patches! Their 2.4
10094 support is still in early development, so beat on it and give feedback or
10099 <Term>CBQ.init <ULink
10100 URL="ftp://ftp.equinox.gu.net/pub/linux/cbq/"
10105 Configuring CBQ can be a bit daunting, especially if all you want to do is
10106 shape some computers behind a router. CBQ.init can help you configure Linux
10107 with a simplified syntax.
10111 For example, if you want all computers in your 192.168.1.0/24 subnet
10112 (on 10mbit eth1) to be limited to 28kbit/s download speed, put
10113 this in the CBQ.init configuration file:
10119 DEVICE=eth1,10Mbit,1Mbit
10123 RULE=192.168.1.0/24
10129 By all means use this program if the 'how and why' don't interest you.
10130 We're using CBQ.init in production and it works very well. It can even do
10131 some more advanced things, like time dependent shaping. The documentation is
10132 embedded in the script, which explains why you can't find a README.
10136 <Term>Chronox easy shaping scripts <ULink
10137 URL="http://www.chronox.de"
10142 Stephan Mueller (smueller@chronox.de) wrote two useful scripts, 'limit.conn'
10143 and 'shaper'. The first one allows you to easily throttle a single download
10144 session, like this:
10150 # limit.conn -s SERVERIP -p SERVERPORT -l LIMIT
10156 It works on Linux 2.2 and 2.4/2.5.
10160 The second script is more complicated, and can be used to make lots of
10161 different queues based on iptables rules, which are used to mark packets
10162 which are then shaped.
10167 <Term>Virtual Router
10168 Redundancy Protocol implementation (
10169 <ULink URL="http://off.net/~jme/vrrpd/">site1</ULink>,
10170 <ULink URL="http://www.imagestream.com/VRRP.html">site2</ULink>
10174 FIXME: This link died, anybody know where it went?
10177 This is purely for redundancy. Two machines with their own IP address and
10178 MAC Address together create a third IP Address and MAC Address, which is
10179 virtual. Originally intended purely for routers, which need constant MAC
10180 addresses, it also works for other servers.
10184 The beauty of this approach is the incredibly easy configuration. No kernel
10185 compiling or patching required, all userspace.
10189 Just run this on all machines participating in a service:
10192 # vrrpd -i eth0 -v 50 10.0.0.22
10198 And you are in business! 10.0.0.22 is now carried by one of your servers,
10199 probably the first one to run the vrrp daemon. Now disconnect that computer
10200 from the network and very rapidly one of the other computers will assume the
10201 10.0.0.22 address, as well as the MAC address.
10205 I tried this over here and had it up and running in 1 minute. For some
10206 strange reason it decided to drop my default gateway, but the -n flag
10211 This is a 'live' fail over:
10217 64 bytes from 10.0.0.22: icmp_seq=3 ttl=255 time=0.2 ms
10218 64 bytes from 10.0.0.22: icmp_seq=4 ttl=255 time=0.2 ms
10219 64 bytes from 10.0.0.22: icmp_seq=5 ttl=255 time=16.8 ms
10220 64 bytes from 10.0.0.22: icmp_seq=6 ttl=255 time=1.8 ms
10221 64 bytes from 10.0.0.22: icmp_seq=7 ttl=255 time=1.7 ms
10227 Not *one* ping packet was lost! Just after packet 4, I disconnected my P200
10228 from the network, and my 486 took over, which you can see from the higher
10236 <ULink URL="http://slava.local.nsys.by/projects/tc_config/">(site)</ULink>
10240 tc_config is set of scripts for linux 2.4+ traffic control
10241 configuration on RedHat systems and (hopefully) derivatives
10242 (linux 2.2.X with ipchains is obsotete).
10243 Uses cbq qdisc as root one, and sfq qdisc at leafs.
10246 Includes snmp_pass utility for getting stats on traffic control via snmp.
10257 <chapter id="lartc.further">
10258 <Title>Further reading</Title>
10265 URL="http://snafu.freedom.org/linux2.2/iproute-notes.html"
10266 >http://snafu.freedom.org/linux2.2/iproute-notes.html</ULink
10270 Contains lots of technical information, comments from the kernel
10275 URL="http://www.davin.ottawa.on.ca/ols/"
10276 >http://www.davin.ottawa.on.ca/ols/</ULink
10280 Slides by Jamal Hadi Salim, one of the authors of Linux traffic control
10285 URL="http://defiant.coinet.com/iproute2/ip-cref/"
10286 >http://defiant.coinet.com/iproute2/ip-cref/</ULink
10290 HTML version of Alexey's LaTeX documentation - explains part of iproute2 in
10296 URL="http://www.aciri.org/floyd/cbq.html"
10297 >http://www.aciri.org/floyd/cbq.html</ULink
10301 Sally Floyd has a good page on CBQ, including her original papers. None of
10302 it is Linux specific, but it does a fair job discussing the theory and uses
10304 Very technical stuff, but good reading for those so inclined.
10308 <Term>Differentiated Services on Linux</Term>
10312 URL="ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz"
10314 > by Werner Almesberger, Jamal Hadi Salim and Alexey
10315 Kuznetsov describes DiffServ facilities in the Linux kernel, amongst which
10316 are TBF, GRED, the DSMARK qdisc and the tcindex classifier.
10321 URL="http://ceti.pl/~kravietz/cbq/NET4_tc.html"
10322 >http://ceti.pl/~kravietz/cbq/NET4_tc.html</ULink
10326 Yet another HOWTO, this time in Polish! You can copy/paste command lines
10327 however, they work just the same in every language. The author is
10328 cooperating with us and may soon author sections of this HOWTO.
10333 URL="http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/car.htm"
10334 >IOS Committed Access Rate</ULink
10339 From the helpful folks of Cisco who have the laudable habit of putting
10340 their documentation online. Cisco syntax is different but the concepts are
10341 the same, except that we can do more and do it without routers the price of
10346 <Term>Docum experimental site<ULink
10347 URL="http://www.docum.org"
10352 Stef Coene is busy convincing his boss to sell Linux support, and so he is
10353 experimenting a lot, especially with managing bandwidth. His site has a lot
10354 of practical information, examples, tests and also points out some CBQ/tc bugs.
10360 <Term>TCP/IP Illustrated, volume 1, W. Richard Stevens, ISBN 0-201-63346-9</Term>
10363 Required reading if you truly want to understand TCP/IP. Entertaining as
10370 <Term>Policy Routing Using Linux, Matthew G. Marsh, ISBN 0-672-32052-5</Term>
10373 A introduction to policy routing with lots of examples.
10379 <Term>Internet QoS: Architectures and Mechanisms for Quality of Service,
10380 Zheng Wang, ISBN 1-55860-608-4</Term>
10383 Hardcover textbook covering topics
10384 related to Quality of Service. Good for understanding basic concepts.
10393 <chapter id="lartc.ack">
10394 <Title>Acknowledgements </Title>
10398 It is our goal to list everybody who has contributed to this HOWTO, or
10399 helped us demystify how things work. While there are currently no plans
10400 for a Netfilter type scoreboard, we do like to recognize the people who are
10406 <ItemizedList spacing="compact">
10410 <author><firstname>Junk</firstname><surname>Alins</surname></author>
10411 <address><email>juanjo@mat.upc.es</email></address>
10416 <author><firstname>Joe</firstname><surname>Van Andel</surname></author>
10422 <author><firstname>Michael</firstname><othername>T.</othername>
10423 <surname>Babcock</surname></author>
10424 <address><email>mbabcock@fibrespeed.net</email></address>
10431 <author><firstname>Christopher</firstname>
10432 <surname>Barton</surname></author>
10433 <address><email>cpbarton%uiuc.edu</email></address>
10439 <author><firstname>Peter</firstname>
10440 <surname>Bieringer</surname></author>
10441 <address><email>pb:bieringer.de</email></address>
10447 <author><firstname>Adam</firstname>
10448 <surname>Burke</surname></author>
10449 <address><email>aburke%crg.ee.uct.ac.za</email></address>
10456 <author><firstname>Ard</firstname><surname>van Breemen</surname></author>
10457 <address><email>ard%kwaak.net</email></address>
10462 <author><firstname>Ron</firstname><surname>Brinker</surname></author>
10463 <address><email>service%emcis.com</email></address>
10468 <author><firstname>?ukasz</firstname><surname>Bromirski</surname></author>
10469 <address><email>l.bromirski@mr0vka.eu.org</email></address>
10474 <author><firstname>Lennert</firstname><surname>Buytenhek</surname></author>
10475 <address><email>buytenh@gnu.org</email></address>
10480 <author><firstname>Esteve</firstname><surname>Camps</surname></author>
10481 <address><email>esteve@hades.udg.es</email></address>
10487 <author><firstname>Ricardo Javier</firstname><surname>Cardenes</surname></author>
10488 <address><email>ricardo%conysis.com</email></address>
10494 <author><firstname>Nelson</firstname><surname>Castillo</surname></author>
10495 <address><email>arhuaco%yahoo.com</email></address>
10501 <author><firstname>Stef</firstname><surname>Coene</surname></author>
10502 <address><email>stef.coene@docum.org</email></address>
10507 <author><firstname>Don</firstname><surname>Cohen</surname></author>
10508 <address><email>don-lartc%isis.cs3-inc.com</email></address>
10513 <author><firstname>Jonathan</firstname><surname>Corbet</surname></author>
10514 <address><email>lwn%lwn.net</email></address>
10519 <author><firstname>Gerry</firstname><surname>Creager</surname>
10520 <othername>N5JXS</othername></author>
10521 <address><email>gerry%cs.tamu.edu</email></address>
10526 <author><firstname>Marco</firstname><surname>Davids</surname></author>
10527 <address><email>marco@sara.nl</email></address>
10532 <author><firstname>Jonathan</firstname><surname>Day</surname></author>
10533 <address><email>jd9812@my-deja.com</email></address>
10538 <author><firstname>Martin</firstname><surname>Devera</surname>
10539 <othername>aka devik</othername></author>
10540 <address><email>devik@cdi.cz</email></address>
10546 <author><firstname>Hannes</firstname><surname>Ebner</surname>
10548 <address><email>he%fli4l.de</email></address>
10554 <author><firstname>Derek</firstname><surname>Fawcus</surname>
10556 <address><email>dfawcus%cisco.com</email></address>
10562 <author><firstname>David</firstname><surname>Fries</surname>
10564 <address><email>dfries%mail.win.org</email></address>
10571 <author><firstname>Stephan</firstname><othername>"Kobold"</othername>
10572 <surname>Gehring</surname></author>
10573 <address><email>Stephan.Gehring@bechtle.de</email></address>
10578 <author><firstname>Jacek</firstname><surname>Glinkowski</surname></author>
10579 <address><email>jglinkow%hns.com</email></address>
10585 <author><firstname>Andrea</firstname><surname>Glorioso</surname></author>
10586 <address><email>sama%perchetopi.org</email></address>
10592 <author><firstname>Thomas</firstname><surname>Graf</surname></author>
10593 <address><email>tgraf%suug.ch</email></address>
10600 <author><firstname>Sandy</firstname><surname>Harris</surname></author>
10601 <address><email>sandy%storm.ca</email></address>
10606 <author><firstname>Nadeem</firstname><surname>Hasan</surname></author>
10607 <address><email>nhasan@usa.net</email></address>
10612 <author><firstname>Erik</firstname><surname>Hensema</surname></author>
10613 <address><email>erik%hensema.xs4all.nl</email></address>
10618 <author><firstname>Vik</firstname><surname>Heyndrickx</surname></author>
10619 <address><email>vik.heyndrickx@edchq.com</email></address>
10624 <author><firstname>Spauldo</firstname><surname>Da Hippie</surname></author>
10625 <address><email>spauldo%usa.net</email></address>
10630 <author><firstname>Koos</firstname><surname>van den Hout</surname></author>
10631 <address><email>koos@kzdoos.xs4all.nl</email></address>
10638 Stefan Huelbrock <shuelbrock%datasystems.de>
10644 <author><firstname>Ayotunde</firstname><surname>Itayemi</surname></author>
10645 <address><email>aitayemi:metrong.com</email></address>
10653 Alexander W. Janssen <yalla%ynfonatic.de>
10659 Andreas Jellinghaus <aj%dungeon.inka.de>
10665 Gareth John <gdjohn%zepler.org>
10671 <author><firstname>Dave</firstname><surname>Johnson</surname></author>
10672 <address><email>dj@www.uk.linux.org</email></address>
10680 Martin Josefsson <gandalf%wlug.westbo.se>
10686 Andi Kleen <ak%suse.de>
10692 Andreas J. Koenig <andreas.koenig%anima.de>
10698 Pawel Krawczyk <kravietz%alfa.ceti.pl>
10704 Amit Kucheria <amitk@ittc.ku.edu>
10710 <author><firstname>Pedro</firstname><surname>Larroy</surname></author>
10711 <address><email>piotr%member.fsf.org</email></address>
10715 Chapter 15, section 10: Example of a full nat solution with QoS
10721 Chapter 17, section 1: Setting up OSPF with Zebra
10730 Edmund Lau <edlau%ucf.ics.uci.edu>
10736 Philippe Latu <philippe.latu%linux-france.org>
10741 Arthur van Leeuwen <arthurvl%sci.kun.nl>
10747 <author><firstname>Jose Luis Domingo</firstname><surname>Lopez</surname>
10749 <address><email>jdomingo@24x7linux.com</email></address>
10756 <author><firstname>Robert</firstname><surname>Lowe</surname>
10758 <address><email>robert.h.lowe@lawrence.edu</email></address>
10765 Jason Lunz <j@cc.gatech.edu>
10771 Stuart Lynne <sl@fireplug.net>
10777 Alexey Mahotkin <alexm@formulabez.ru>
10783 Predrag Malicevic <pmalic@ieee.org>
10788 Patrick McHardy <kaber@trash.net>
10796 Andreas Mohr <andi%lisas.de>
10801 <para>James Morris <jmorris@intercode.com.au>
10807 Andrew Morton <akpm%zip.com.au>
10819 Stephan Mueller <smueller@chronox.de>
10825 Togan Muftuoglu <toganm%yahoo.com>
10832 Chris Murray <cmurray@stargate.ca>
10838 Takeo NAKANO <nakano@apm.seikei.ac.jp>
10845 Patrick Nagelschmidt <dto%gmx.net>
10851 Ram Narula <ram@princess1.net>
10857 Jorge Novo <jnovo@educanet.net>
10863 Patrik <ph@kurd.nu>
10867 <listitem><para>P?l Osgy?ny <oplab%westel900.net></para></listitem>
10872 Lutz Preßler <Lutz.Pressler%SerNet.DE>
10878 Jason Pyeron <jason%pyeron.com>
10884 Rod Roark <rod%sunsetsystems.com>
10890 Pavel Roskin <proski@gnu.org>
10897 Rusty Russell <rusty%rustcorp.com.au>
10903 Mihai RUSU <dizzy%roedu.net>
10909 Rob Pitman <rob%pitman.co.za>
10917 Jamal Hadi Salim <hadi%cyberus.ca>
10923 Ren? Serral <rserral%ac.upc.es>
10930 David Sauer <davids%penguin.cz>
10936 Sheharyar Suleman Shaikh <sss23@drexel.edu>
10942 Stewart Shields <MourningBlade%bigfoot.com>
10948 Nick Silberstein <nhsilber%yahoo.com>
10954 Konrads Smelkov <konrads@interbaltika.com>
10960 <author><firstname>William</firstname><surname>Stearns</surname></author>
10961 <address><email>wstearns@pobox.com</email></address>
10967 Andreas Steinmetz <ast%domdv.de>
10973 Matthew Strait <straitm%mathcs.carleton.edu>
10979 Jason Tackaberry <tack@linux.com>
10985 Charles Tassell <ctassell%isn.net>
10990 <para>Jason Thomas <jason5intology.com.au>
10995 Glen Turner <glen.turner%aarnet.edu.au>
11001 Tea Sponsor: Eric Veldhuyzen <eric%terra.nu>
11007 Thomas Walpuski <thomas%bender.thinknerd.de>
11014 Song Wang <wsong@ece.uci.edu>
11020 Frank v Waveren <fvw@var.cx>
11027 <author><firstname>Chris</firstname><surname>Wilson</surname></author>
11028 <address><email>chris@netservers.co.uk</email></address>
11034 <author><firstname>Lazar</firstname><surname>Yanackiev</surname></author>
11035 <address><email>Lyanackiev%gmx.net</email></address>