1 <!doctype linuxdoc system
>
8 <!-- Title information -->
10 <title>Linux Advanced Routing
& Traffic Control HOWTO
11 <author>Netherlabs BV (bert hubert
<bert.hubert@netherlabs.nl
>)
&nl;
12 Gregory Maxwell
<greg@linuxpower.cx
> &nl;
13 Remco van Mook
<remco@virtu.nl
> &nl;
14 Martijn van Oosterhout
<kleptog@cupid.suninternet.com
> &nl;
15 Paul B Schroeder
<paulsch@us.ibm.com
> &nl;
16 Jasper Spaans
<jasper@spaans.ds9a.nl
> &nl;
20 A very hands-on approach to iproute2, traffic shaping and a bit of netfilter
23 <!-- Table of contents -->
26 <!-- Begin the document -->
30 This document is dedicated to lots of people, and is my attempt to do
31 something back. To list but a few:
35 <item>Alexey N. Kuznetsov
36 <item>The good folks from Google
37 <item>The staff of Casema Internet
42 Welcome, gentle reader.
44 This document hopes to enlighten you on how to do more with Linux
2.2/
2.4
45 routing. Unbeknownst to most users, you already run tools which allow you to
46 do spectacular things. Commands like 'route' and 'ifconfig' are actually
47 very thin wrappers for the very powerful iproute2 infrastructure.
49 I hope that this HOWTO will become as readable as the ones by Rusty Russell
50 of (amongst other things) netfilter fame.
52 You can always reach us by writing to the
<url name=
"HOWTO team"
53 url=
"mailto:HOWTO@ds9a.nl">. However, please consider posting to the mailing
54 list (see the relevant section) if you have questions which are not directly
55 related to this HOWTO.
57 Before losing your way in this HOWTO, if all you want to do is simple
58 traffic shaping, skip everything and head to the 'Other possibilties'
59 chapter, and read about CBQ.init.
62 <sect1>Disclaimer
& License
64 This document is distributed in the hope that it will be useful,
65 but WITHOUT ANY WARRANTY; without even the implied warranty of
66 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
68 In short, if your STM-
64 backbone breaks down and distributes pornography to
69 your most esteemed customers - it's never our fault. Sorry.
71 Copyright (c)
2001 by bert hubert, Gregory Maxwell, Martijn van
72 Oosterhout, Remco can Mook, Paul B. Schroeder and others. This material may
73 be distributed only subject to the terms and conditions set forth in the
74 Open Publication License, v1.0 or later (the latest version is presently
75 available at http://www.opencontent.org/openpub/).
77 Please freely copy and distribute (sell or give away) this document in any
78 format. It's requested that corrections and/or comments be fowarded to the
81 It is also requested that if you publish this HOWTO in hardcopy that you
82 send the authors some samples for 'review purposes' :-)
84 <sect1>Prior knowledge
86 As the title implies, this is the 'Advanced' HOWTO. While by no means rocket
87 science, some prior knowledge is assumed.
89 Here are some other references which might help teach you more:
92 url=
"http://netfilter.samba.org/unreliable-guides/networking-concepts-HOWTO/index.html"
93 name=
"Rusty Russell's networking-concepts-HOWTO"></tag>
94 Very nice introduction, explaining what a network is, and how it is
95 connected to other networks
96 <tag>Linux Networking-HOWTO (Previously the Net-
3 HOWTO)
</tag>
97 Great stuff, although very verbose. It teaches you a lot of stuff that's
98 already configured if you are able to connect to the Internet.
99 Should be located in
<file>/usr/doc/HOWTO/NET3-
4-HOWTO.txt
</file> but can be also be found
100 <url url=
"http://www.linuxports.com/howto/networking"
104 <sect1>What Linux can do for you
106 A small list of things that are possible:
109 <item>Throttle bandwidth for certain computers
110 <item>Throttle bandwidth TO certain computers
111 <item>Help you to fairly share your bandwidth
112 <item>Protect your network from DoS attacks
113 <item>Protect the Internet from your customers
114 <item>Multiplex several servers as one, for load balancing or
115 enhanced availability
116 <item>Restrict access to your computers
117 <item>Limit access of your users to other hosts
118 <item>Do routing based on user id (yes!), MAC address, source IP
119 address, port, type of service, time of day or content
122 Currently, not many people are using these advanced features. This is for
123 several reasons. While the provided documentation is verbose, it is not very
124 hands-on. Traffic control is almost undocumented.
125 <sect1>Housekeeping notes
127 There are several things which should be noted about this document. While I
128 wrote most of it, I really don't want it to stay that way. I am a strong
129 believer in Open Source, so I encourage you to send feedback, updates,
130 patches etcetera. Do not hesitate to inform me of typos or plain old errors.
131 If my English sounds somewhat wooden, please realise that I'm not a native
132 speaker. Feel free to send suggestions.
134 If you feel to you are better qualified to maintain a section, or think that
135 you can author and maintain new sections, you are welcome to do so. The SGML
136 of this HOWTO is available via CVS, I very much envision more people
139 In aid of this, you will find lots of FIXME notices. Patches are always
140 welcome! Wherever you find a FIXME, you should know that you are treading in
141 unknown territory. This is not to say that there are no errors elsewhere,
142 but be extra careful. If you have validated something, please let us know so
143 we can remove the FIXME notice.
145 About this HOWTO, I will take some liberties along the road. For example, I
146 postulate a
10Mbit Internet connection, while I know full well that those
148 <sect1>Access, CVS
& submitting updates
150 The canonical location for the HOWTO is
<url
151 url=
"http://www.ds9a.nl/lartc" name=
"here">.
153 We now have anonymous CVS access available to the world at large. This is
154 good in a number of ways. You can easily upgrade to newer versions of this
155 HOWTO and submitting patches is no work at all.
157 Furthermore, it allows the authors to work on the source independently,
161 $ export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
163 CVS password: [enter 'cvs' (without 's)]
165 cvs server: Updating
2.4routing
166 U
2.4routing/
2.4routing.sgml
169 If you spot an error, or want to add something, just fix it locally, and run
170 cvs -z3 diff -uBb, and send the result to us.
172 A Makefile is supplied which should help you create postscript, dvi, pdf,
173 html and plain text. You may need to install sgml-tools, ghostscript and
174 tetex to get all formats.
179 The authors receive an increasing amount of mail about this HOWTO. Because
180 of the clear interest of the community, it has been decided to start a
181 mailinglist where people can talk to each other about Advanced Routing and
182 Traffic Control. You can subscribe to the list
183 <url url=
"http://mailman.ds9a.nl/mailman/listinfo/lartc" name=
"here">.
185 It should be pointed out that the authors are very hesitant of answering
186 questions not asked on the list. We would like the archive of the list to
187 become some kind of knowledge base. If you have a question, please search
188 the archive, and then post to the mailinglist.
190 <sect1>Layout of this document
192 We will be doing interesting stuff almost immediately, which also means that
193 there will initially be parts that are explained incompletely or are not
194 perfect. Please gloss over these parts and assume that all will become clear.
196 Routing and filtering are two distinct things. Filtering is documented very
197 well by Rusty's HOWTOs, available here:
200 <item><url url=
"http://netfilter.samba.org/unreliable-guides/"
201 name=
"Rusty's Remarkably Unreliable Guides">
204 We will be focusing mostly on what is possible by combining netfilter and
206 <sect>Introduction to iproute2
209 Most Linux distributions, and most UNIX's, currently use the
210 venerable 'arp', 'ifconfig' and 'route' commands. While these tools work,
211 they show some unexpected behaviour under Linux
2.2 and up. For example, GRE
212 tunnels are an integral part of routing these days, but require completely
215 With iproute2, tunnels are an integral part of the tool set.
217 The
2.2 and above Linux kernels include a completely redesigned network
218 subsystem. This new networking code brings Linux performance and a feature
219 set with little competition in the general OS arena. In fact, the new
220 routing, filtering, and classifying code is more featureful than the one
221 provided by many dedicated routers and firewalls and traffic shaping
224 As new networking concepts have been invented, people have found ways to
225 plaster them on top of the existing framework in existing OSes. This
226 constant layering of cruft has lead to networking code that is filled with
227 strange behaviour, much like most human languages. In the past, Linux
228 emulated SunOS's handling of many of these things, which was not ideal.
230 This new framework makes it possible to clearly express features
231 previously beyond Linux's reach.
235 Linux has a sophisticated system for bandwidth provisioning called Traffic
236 Control. This system supports various method for classifying, prioritizing,
237 sharing, and limiting both inbound and outbound traffic.
240 We'll start off with a tiny tour of iproute2 possibilities.
243 You should make sure that you have the userland tools installed. This
244 package is called 'iproute' on both RedHat and Debian, and may otherwise be
245 found at
<tt>ftp://ftp.inr.ac.ru/ip-routing/iproute2-
2.2.4-now-ss??????.tar.gz
"</tt>.
247 You can also try <url name="here
" url="ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz
">
248 for the latest version.
250 Some parts of iproute require you to have certain kernel options enabled. It
251 should also be noted that all releases of RedHat up to and including 6.2
252 come without most of the traffic control features in the default kernel.
254 RedHat 7.2 has everything in by default.
256 Also make sure that you have netlink support, should you choose to roll your
257 own kernel. Iproute2 needs it.
259 <sect1>Exploring your current configuration
261 This may come as a surprise, but iproute2 is already configured! The current
262 commands <tt>ifconfig</tt> and <tt>route</tt> are already using the advanced
263 syscalls, but mostly with very default (ie. boring) settings.
265 The <tt>ip</tt> tool is central, and we'll ask it to display our interfaces
267 <sect2><tt>ip</tt> shows us our links
270 [ahu@home ahu]$ ip link list
271 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
272 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
273 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
274 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
275 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
276 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
277 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
278 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
279 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
283 <p>Your mileage may vary, but this is what it shows on my NAT router at
284 home. I'll only explain part of the output as not everything is directly
287 We first see the loopback interface. While your computer may function
288 somewhat without one, I'd advise against it. The MTU size (Maximum Transfer
289 Unit) is 3924 octets, and it is not supposed to queue. Which makes sense
290 because the loopback interface is a figment of your kernel's imagination.
292 I'll skip the dummy interface for now, and it may not be present on your
293 computer. Then there are my two physical network interfaces, one at the side
294 of my cable modem, the other one serves my home ethernet segment.
295 Furthermore, we see a ppp0 interface.
297 Note the absence of IP addresses. iproute disconnects the concept of 'links'
298 and 'IP addresses'. With IP aliasing, the concept of 'the' IP address had
299 become quite irrelevant anyhow.
301 It does show us the MAC addresses though, the hardware identifier of our
303 <sect2><tt>ip</tt> shows us our IP addresses
306 [ahu@home ahu]$ ip address show
307 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
308 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
309 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
310 2: dummy: <BROADCAST,NOARP> mtu 1500 qdisc noop
311 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
312 3: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1400 qdisc pfifo_fast qlen 100
313 link/ether 48:54:e8:2a:47:16 brd ff:ff:ff:ff:ff:ff
314 inet 10.0.0.1/8 brd 10.255.255.255 scope global eth0
315 4: eth1: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 100
316 link/ether 00:e0:4c:39:24:78 brd ff:ff:ff:ff:ff:ff
317 3764: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 10
319 inet 212.64.94.251 peer 212.64.94.1/32 scope global ppp0
322 This contains more information. It shows all our addresses, and to which
323 cards they belong. 'inet' stands for Internet (IPv4). There are lots of other
324 address families, but these don't concern us right now.
326 Let's examine eth0 somewhat closer. It says that it is related to the inet
327 address '10.0.0.1/8'. What does this mean? The /8 stands for the number of
328 bits that are in the Network Address. There are 32 bits, so we have 24 bits
329 left that are part of our network. The first 8 bits of 10.0.0.1 correspond
330 to 10.0.0.0, our Network Address, and our netmask is 255.0.0.0.
332 The other bits are connected to this interface, so 10.250.3.13 is directly
333 available on eth0, as is 10.0.0.1 for example.
335 With ppp0, the same concept goes, though the numbers are different. Its
336 address is 212.64.94.251, without a subnet mask. This means that we have a
337 point-to-point connection and that every address, with the exception of
338 212.64.94.251, is remote. There is more information, however. It tells us
339 that on the other side of the link there is, yet again, only one address,
340 212.64.94.1. The /32 tells us that there are no 'network bits'.
342 It is absolutely vital that you grasp these concepts. Refer to the
343 documentation mentioned at the beginning of this HOWTO if you have trouble.
345 You may also note 'qdisc', which stands for Queueing Discipline. This will
346 become vital later on.
348 <sect2><tt>ip</tt> shows us our routes
350 Well, we now know how to find 10.x.y.z addresses, and we are able to reach
351 212.64.94.1. This is not enough however, so we need instructions on how to
352 reach the world. The Internet is available via our ppp connection, and it
353 appears that 212.64.94.1 is willing to spread our packets around the
354 world, and deliver results back to us.
357 [ahu@home ahu]$ ip route show
358 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
359 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
360 127.0.0.0/8 dev lo scope link
361 default via 212.64.94.1 dev ppp0
364 This is pretty much self explanatory. The first 4 lines of output explicitly
365 state what was already implied by <tt>ip address show</tt>, the last line
366 tells us that the rest of the world can be found via 212.64.94.1, our
367 default gateway. We can see that it is a gateway because of the word
368 via, which tells us that we need to send packets to 212.64.94.1, and that it
369 will take care of things.
371 For reference, this is what the old 'route' utility shows us:
373 [ahu@home ahu]$ route -n
374 Kernel IP routing table
375 Destination Gateway Genmask Flags Metric Ref Use
377 212.64.94.1 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0
378 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
379 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
380 0.0.0.0 212.64.94.1 0.0.0.0 UG 0 0 0 ppp0
385 ARP is the Address Resolution Protocol as described in
386 <url url="http://www.faqs.org/rfcs/rfc826.html
" name="RFC
826">.
387 ARP is used by a networked machine to resolve the hardware location/address of
388 another machine on the same
389 local network. Machines on the Internet are generally known by their names
391 addresses. This is how a machine on the foo.com network is able to communicate
392 with another machine which is on the bar.net network. An IP address, though,
393 cannot tell you the physical location of a machine. This is where ARP comes
396 Let's take a very simple example. Suppose I have a network composed of several
397 machines. Two of the machines which are currently on my network are foo
398 with an IP address of 10.0.0.1 and bar with an IP address of 10.0.0.2.
399 Now foo wants to ping bar to see that he is alive, but alas, foo has no idea
400 where bar is. So when foo decides to ping bar he will need to send
402 This ARP request is akin to foo shouting out on the network "Bar (
10.0.0.2)!
403 Where are you?
" As a result of this every machine on the network will hear
404 foo shouting, but only bar (10.0.0.2) will respond. Bar will then send an
405 ARP reply directly back to foo which is akin
407 "Foo (
10.0.0.1) I am here at
00:
60:
94:E9:
08:
12.
" After this simple transaction
408 that's used to locate his friend on the network, foo is able to communicate
409 with bar until he (his arp cache) forgets where bar is (typically after
412 Now let's see how this works.
413 You can view your machines current arp/neighbor cache/table like so:
415 [root@espa041 /home/src/iputils]# ip neigh show
416 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
417 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
420 As you can see my machine espa041 (9.3.76.41) knows where to find espa042
422 espagate (9.3.76.1). Now let's add another machine to the arp cache.
425 [root@espa041 /home/paulsch/.gnome-desktop]# ping -c 1 espa043
426 PING espa043.austin.ibm.com (9.3.76.43) from 9.3.76.41 : 56(84) bytes of data.
427 64 bytes from 9.3.76.43: icmp_seq=0 ttl=255 time=0.9 ms
429 --- espa043.austin.ibm.com ping statistics ---
430 1 packets transmitted, 1 packets received, 0% packet loss
431 round-trip min/avg/max = 0.9/0.9/0.9 ms
433 [root@espa041 /home/src/iputils]# ip neigh show
434 9.3.76.43 dev eth0 lladdr 00:06:29:21:80:20 nud reachable
435 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
436 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud reachable
439 As a result of espa041 trying to contact espa043, espa043's hardware
440 address/location has now been added to the arp/neighbor cache.
441 So until the entry for
442 espa043 times out (as a result of no communication between the two) espa041
443 knows where to find espa043 and has no need to send an ARP request.
445 Now let's delete espa043 from our arp cache:
448 [root@espa041 /home/src/iputils]# ip neigh delete 9.3.76.43 dev eth0
449 [root@espa041 /home/src/iputils]# ip neigh show
450 9.3.76.43 dev eth0 nud failed
451 9.3.76.42 dev eth0 lladdr 00:60:08:3f:e9:f9 nud reachable
452 9.3.76.1 dev eth0 lladdr 00:06:29:21:73:c8 nud stale
455 Now espa041 has again forgotten where to find espa043 and will need to send
456 another ARP request the next time he needs to communicate with espa043.
457 You can also see from the above output that espagate (9.3.76.1) has been
458 changed to the "stale
" state. This means that the location shown is still
459 valid, but it will have to be confirmed at the first transaction to that
462 <sect>Rules - routing policy database
464 If you have a large router, you may well cater for the needs of different
465 people, who should be served differently. The routing policy database allows
466 you to do this by having multiple sets of routing tables.
468 If you want to use this feature, make sure that your kernel is compiled with
469 the "IP: advanced router
" and "IP: policy routing
" features.
471 When the kernel needs to make a routing decision, it finds out which table
472 needs to be consulted. By default, there are three tables. The old 'route'
473 tool modifies the main and local tables, as does the ip tool (by default).
477 [ahu@home ahu]$ ip rule list
478 0: from all lookup local
479 32766: from all lookup main
480 32767: from all lookup default
483 This lists the priority of all rules. We see that all rules apply to all
484 packets ('from all'). We've seen the 'main' table before, it is output by
485 <tt>ip route ls</tt>, but the 'local' and 'default' table are new.
487 If we want to do fancy things, we generate rules which point to different
488 tables which allow us to override system wide routing rules.
490 For the exact semantics on what the kernel does when there are more matching
491 rules, see Alexey's ip-cref documentation.
493 <sect1>Simple source policy routing
495 Let's take a real example once again, I have 2 (actually 3, about time I
496 returned them) cable modems, connected to a Linux NAT ('masquerading')
497 router. People living here pay me to use the Internet. Suppose one of my
498 house mates only visits hotmail and wants to pay less. This is fine with me,
499 but they'll end up using the low-end cable modem.
501 The 'fast' cable modem is known as 212.64.94.251 and is a PPP link to
502 212.64.94.1. The 'slow' cable modem is known by various ip addresses,
503 212.64.78.148 in this example and is a link to 195.96.98.253.
507 [ahu@home ahu]$ ip route list table local
508 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
509 local 10.0.0.1 dev eth0 proto kernel scope host src 10.0.0.1
510 broadcast 10.0.0.0 dev eth0 proto kernel scope link src 10.0.0.1
511 local 212.64.94.251 dev ppp0 proto kernel scope host src 212.64.94.251
512 broadcast 10.255.255.255 dev eth0 proto kernel scope link src 10.0.0.1
513 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
514 local 212.64.78.148 dev ppp2 proto kernel scope host src 212.64.78.148
515 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
516 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
519 Lots of obvious things, but things that need to be specified somewhere.
520 Well, here they are. The default table is empty.
522 Let's view the 'main' table:
524 [ahu@home ahu]$ ip route list table main
525 195.96.98.253 dev ppp2 proto kernel scope link src 212.64.78.148
526 212.64.94.1 dev ppp0 proto kernel scope link src 212.64.94.251
527 10.0.0.0/8 dev eth0 proto kernel scope link src 10.0.0.1
528 127.0.0.0/8 dev lo scope link
529 default via 212.64.94.1 dev ppp0
532 We now generate a new rule which we call 'John', for our hypothetical
533 house mate. Although we can work with pure numbers, it's far easier if we add
534 our tables to <file>/etc/iproute2/rt_tables</file>.
537 # echo 200 John >> /etc/iproute2/rt_tables
538 # ip rule add from 10.0.0.10 table John
540 0: from all lookup local
541 32765: from 10.0.0.10 lookup John
542 32766: from all lookup main
543 32767: from all lookup default
546 Now all that is left is to generate John's table, and flush the route cache:
548 # ip route add default via 195.96.98.253 dev ppp2 table John
549 # ip route flush cache
552 And we are done. It is left as an exercise for the reader to implement this
555 <sect1>Routing for multiple uplinks/providers
557 A common configuration is the following, in which there are two providers
558 that connect a local network (or even a single machine) to the big Internet.
564 +-------------+ Provider 1 +-------
566 ___/ \_ +------+-------+ +------------+ |
569 | Local network -----+ Linux router | | Internet
572 \___/ +------+-------+ +------------+ |
574 +-------------+ Provider 2 +-------
576 +------------+ \________
579 There are usually two questions given this setup.
583 The first is how to route answers to packets coming in over a
584 particular provider, say Provider 1, back out again over that same provider.
586 Let us first set some symbolical names. Let <tt>$IF1</tt> be the name of the
587 first interface (if1 in the picture above) and <tt>$IF2</tt> the name of the
588 second interface. Then let <tt>$IP1</tt> be the IP address associated with
589 <tt>$IF1</tt> and <tt>$IP2</tt> the IP address associated with
590 <tt>$IF2</tt>. Next, let <tt>$P1</tt> be the IP address of the gateway at
591 Provider 1, and <tt>$P2</tt> the IP address of the gateway at provider 2.
592 Finally, let <tt>$P1_NET</tt> be the IP network <tt>$P1</tt> is in,
593 and <tt>$P2_NET</tt> the IP network <tt>$P2</tt> is in.
595 One creates two additional routing tables, say <tt>T1</tt> and <tt>T2</tt>.
596 These are added in /etc/iproute2/rt_tables. Then you set up routing in
597 these tables as follows:
600 ip route add $P1_NET dev $IF1 src $IP1 table T1
601 ip route add default via $P1 table T1
602 ip route add $P2_NET dev $IF2 src $IP2 table T2
603 ip route add default via $P2 table T2
606 Nothing spectacular, just build a route to the gateway and build a
607 default route via that gateway, as you would do in the case of a single
608 upstream provider, but put the routes in a seperate table per provider.
609 Note that the network route suffices, as it tells you how to find any host
610 in that network, which includes the gateway, as specified above.
612 Next you set up the main routing table. It is a good idea to route
613 things to the direct neighbour through the interface connected to that
614 neighbour. Note the `src' arguments, they make sure the right outgoing IP
618 ip route add $P1_NET dev $IF1 src $IP1
619 ip route add $P2_NET dev $IF2 src $IP2
622 Then, your preference for default route:
625 ip route add default via $P1
628 Next, you set up the routing rules. These actually choose what routing table
629 to route with. You want to make sure that you route out a given
630 interface if you already have the corresponding source address:
633 ip rule add from $IP1 table T1
634 ip rule add from $IP2 table T2
637 This set of commands makes sure all answers to traffic coming in on a
638 particular interface get answered from that interface.
640 Now, this is just the very basic setup. It will work for all processes
641 running on the router itself, and for the local network, if it is
642 masqueraded. If it is not, then you either have IP space from both providers
643 or you are going to want to masquerade to one of the two providers. In both
644 cases you will want to add rules selecting which provider to route out from
645 based on the IP address of the machine in the local network.
647 <sect2>Load balancing
649 The second question is how to balance traffic going out over the two providers.
650 This is actually not hard if you already have set up split access as above.
652 Instead of choosing one of the two providers as your default route,
653 you now set up the default route to be a multipath route. In the default
654 kernel this will balance routes over the two providers. It is done
655 as follows (once more building on the example in the section on
659 ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
660 nexthop via $P2 dev $IF2 weight 1
663 This will balance the routes over both providers. The <tt>weight</tt>
664 parameters can be tweaked to favor one provider over the other.
666 Note that balancing will not be perfect, as it is route based, and routes
667 are cached. This means that routes to often-used sites will always
668 be over the same provider.
670 Furthermore, if you really want to do this, you probably also want to look
671 at Julian Anastasov's patches at <url
672 url="http://www.linuxvirtualserver.org/~julian/#routes
" name="Julian's route
673 patch page
">. They will make things nicer to work with.
675 <sect>GRE and other tunnels
677 There are 3 kinds of tunnels in Linux. There's IP in IP tunneling, GRE tunneling and tunnels that live outside the kernel (like, for example PPTP).
678 <sect1>A few general remarks about tunnels:
680 Tunnels can be used to do some very unusual and very cool stuff. They can
681 also make things go horribly wrong when you don't configure them right.
682 Don't point your default route to a tunnel device unless you know
683 <bf>exactly</bf> what you are doing :-). Furthermore, tunneling increases
684 overhead, because it needs an extra set of IP headers. Typically this is 20
685 bytes per packet, so if the normal packet size (MTU) on a network is 1500
686 bytes, a packet that is sent through a tunnel can only be 1480 bytes big.
687 This is not necessarily a problem, but be sure to read up on IP packet
688 fragmentation/reassembly when you plan to connect large networks with
689 tunnels. Oh, and of course, the fastest way to dig a tunnel is to dig at
692 <sect1>IP in IP tunneling
694 This kind of tunneling has been available in Linux for a long time. It requires 2 kernel modules,
695 ipip.o and new_tunnel.o.
697 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
698 So we have network A:
702 netmask 255.255.255.0
705 The router has address 172.16.17.18 on network C.
710 netmask 255.255.255.0
713 The router has address 172.19.20.21 on network C.
715 As far as network C is concerned, we assume that it will pass any packet sent
716 from A to B and vice versa. You might even use the Internet for this.
720 First, make sure the modules are installed:
726 Then, on the router of network A, you do the following:
728 ifconfig tunl0 10.0.1.1 pointopoint 172.19.20.21
729 route add -net 10.0.2.0 netmask 255.255.255.0 dev tunl0
731 And on the router of network B:
733 ifconfig tunl0 10.0.2.1 pointopoint 172.16.17.18
734 route add -net 10.0.1.0 netmask 255.255.255.0 dev tunl0
736 And if you're finished with your tunnel:
740 Presto, you're done. You can't forward broadcast or IPv6 traffic through
741 an IP-in-IP tunnel, though. You just connect 2 IPv4 networks that normally wouldn't be able to talk to each other, that's all. As far as compatibility goes, this code has been around a long time, so it's compatible all the way back to 1.3 kernels. Linux IP-in-IP tunneling doesn't work with other Operating Systems or routers, as far as I know. It's simple, it works. Use it if you have to, otherwise use GRE.
745 GRE is a tunneling protocol that was originally developed by Cisco, and it
746 can do a few more things than IP-in-IP tunneling. For example, you can also
747 transport multicast traffic and IPv6 through a GRE tunnel.
749 In Linux, you'll need the ip_gre.o module.
751 <sect2>IPv4 Tunneling
753 Let's do IPv4 tunneling first:
755 Let's say you have 3 networks: Internal networks A and B, and intermediate network C (or let's say, Internet).
757 So we have network A:
760 netmask 255.255.255.0
763 The router has address 172.16.17.18 on network C.
764 Let's call this network neta (ok, hardly original)
769 netmask 255.255.255.0
772 The router has address 172.19.20.21 on network C.
773 Let's call this network netb (still not original)
775 As far as network C is concerned, we assume that it will pass any packet sent
776 from A to B and vice versa. How and why, we do not care.
778 On the router of network A, you do the following:
780 ip tunnel add netb mode gre remote 172.19.20.21 local 172.16.17.18 ttl 255
782 ip addr add 10.0.1.1 dev netb
783 ip route add 10.0.2.0/24 dev netb
786 Let's discuss this for a bit. In line 1, we added a tunnel device, and
787 called it netb (which is kind of obvious because that's where we want it to
788 go). Furthermore we told it to use the GRE protocol (mode gre), that the
789 remote address is 172.19.20.21 (the router at the other end), that our
790 tunneling packets should originate from 172.16.17.18 (which allows your
791 router to have several IP addresses on network C and let you decide which
792 one to use for tunneling) and that the TTL field of the packet should be set
795 The second line enables the device.
797 In the third line we gave the newly born interface netb the address
798 10.0.1.1. This is OK for smaller networks, but when you're starting up a
799 mining expedition (LOTS of tunnels), you might want to consider using
800 another IP range for tunneling interfaces (in this example, you could use
803 <p>In the fourth line we set the route for network B. Note the different notation for the netmask. If you're not familiar with this notation, here's how it works: you write out the netmask in binary form, and you count all the ones. If you don't know how to do that, just remember that 255.0.0.0 is /8, 255.255.0.0 is /16 and 255.255.255.0 is /24. Oh, and 255.255.254.0 is /23, in case you were wondering.
805 But enough about this, let's go on with the router of network B.
807 ip tunnel add neta mode gre remote 172.16.17.18 local 172.19.20.21 ttl 255
809 ip addr add 10.0.2.1 dev neta
810 ip route add 10.0.1.0/24 dev neta
812 And when you want to remove the tunnel on router A:
814 ip link set netb down
817 Of course, you can replace netb with neta for router B.
819 <sect2>IPv6 Tunneling
821 See Section 6 for a short bit about IPv6 Addresses.
825 Let's assume that you have the following IPv6 network, and you want to connect it to 6bone, or a friend.
828 Network 3ffe:406:5:1:5:a:2:1/96
830 Your IPv4 address is 172.16.17.18, and the 6bone router has IPv4 address 172.22.23.24.
833 ip tunnel add sixbone mode sit remote 172.22.23.24 local 172.16.17.18 ttl 255
834 ip link set sixbone up
835 ip addr add 3ffe:406:5:1:5:a:2:1/96 dev sixbone
836 ip route add 3ffe::/15 dev sixbone
839 Let's discuss this. In the first line, we created a tunnel device called sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it where to go to (remote) and where to come from (local). TTL is set to maximum, 255. Next, we made the device active (up). After that, we added our own network address, and set a route for 3ffe::/15 (which is currently all of 6bone) through the tunnel.
841 GRE tunnels are currently the preferred type of tunneling. It's a standard that is also widely adopted outside the Linux community and therefore a Good Thing.
843 <sect1>Userland tunnels
845 There are literally dozens of implementations of tunneling outside the kernel. Best known are of course PPP and PPTP, but there are lots more (some proprietary, some secure, some that don't even use IP) and that is really beyond the scope of this HOWTO.
847 <sect>IPv6 tunneling with Cisco and/or 6bone
849 By Marco Davids <marco@sara.nl>
853 As far as I am concerned, this IPv6-IPv4 tunneling is not per definition
854 GRE tunneling. You could tunnel IPv6 over IPv4 by means of GRE tunnel devices
855 (GRE tunnels ANY to IPv4), but the device used here ("sit
") only tunnels
856 IPv6 over IPv4 and is therefore something different.
858 <sect1>IPv6 Tunneling
860 This is another application of the tunneling capabilities of Linux. It is
861 popular among the IPv6 early adopters, or pioneers if you like.
862 The 'hands-on' example described below is certainly not the only way
863 to do IPv6 tunneling. However, it is the method that is often used to tunnel
864 between Linux and a Cisco IPv6 capable router and experience tells us that
865 this is just the thing many people are after. Ten to one this applies to
868 A short bit about IPv6 addresses:
870 IPv6 addresses are, compared to IPv4 addresses, really big: 128 bits
871 against 32 bits. And this provides us just with the thing we need: many, many
872 IP-addresses: 340,282,266,920,938,463,463,374,607,431,768,211,465 to be
873 precise. Apart from this, IPv6 (or IPng, for IP Next Generation) is supposed
874 to provide for smaller routing tables on the Internet's backbone routers,
875 simpler configuration of equipment, better security at the IP level and
876 better support for QoS.
878 An example: 2002:836b:9820:0000:0000:0000:836b:9886
880 Writing down IPv6 addresses can be quite a burden. Therefore, to make
881 life easier there are some rules:
885 Don't use leading zeroes. Same as in IPv4.
887 <item>Use colons to separate every 16 bits or two bytes.
889 <item>When you have lots of consecutive zeroes,
890 you can write this down as ::. You can only do this once in an
891 address and only for quantities of 16 bits, though.
894 The address 2002:836b:9820:0000:0000:0000:836b:9886 can be written down
895 as 2002:836b:9820::836b:9886, which is somewhat friendlier.
897 Another example, the address 3ffe:0000:0000:0000:0000:0020:34A1:F32C can be
898 written down as 3ffe::20:34A1:F32C, which is a lot shorter.
900 IPv6 is intended to be the successor of the current IPv4. Because it
901 is relatively new technology, there is no worldwide native IPv6 network
902 yet. To be able to move forward swiftly, the 6bone was introduced.
904 Native IPv6 networks are connected to each other by encapsulating the IPv6
905 protocol in IPv4 packets and sending them over the existing IPv4 infrastructure
906 from one IPv6 site to another.
908 That is precisely where the tunnel steps in.
910 To be able to use IPv6, we should have a kernel that supports it. There
911 are many good documents on how to achieve this. But it all comes down to
914 <item>Get yourself a recent Linux distribution, with suitable glibc.
915 <item>Then get yourself an up-to-date kernel source.
917 If you are all set, then you can go ahead and compile an IPv6 capable
920 <item>Go to /usr/src/linux and type:
921 <item>make menuconfig
922 <item>Choose "Networking Options
"
923 <item>Select "The IPv6 protocol
", "IPv6: enable EUI-
64 token format
", "IPv6:
924 disable provider based addresses
"
926 HINT: Don't go for the 'module' option. Often this won't work well.
928 In other words, compile IPv6 as 'built-in' in your kernel.
929 You can then save your config like usual and go ahead with compiling
932 HINT: Before doing so, consider editing the Makefile:
933 EXTRAVERSION = -x ; --> ; EXTRAVERSION = -x-IPv6
935 There is a lot of good documentation about compiling and installing
936 a kernel, however this document is about something else. If you run into
937 problems at this stage, go and look for documentation about compiling a
938 Linux kernel according to your own specifications.
940 The file /usr/src/linux/README might be a good start.
941 After you acomplished all this, and rebooted with your brand new kernel,
942 you might want to issue an '/sbin/ifconfig -a' and notice the brand
943 new 'sit0-device'. SIT stands for Simple Internet Transition. You may give
944 yourself a compliment; you are now one major step closer to IP, the Next
947 Now on to the next step. You want to connect your host, or maybe even
948 your entire LAN to another IPv6 capable network. This might be the "6bone
"
949 that is setup especially for this particular purpose.
951 Let's assume that you have the following IPv6 network: 3ffe:604:6:8::/64 and
952 you want to connect it to 6bone, or a friend. Please note that the /64
953 subnet notation works just like with regular IP adresses.
955 Your IPv4 address is 145.100.24.181 and the 6bone router has IPv4 address
958 # ip tunnel add sixbone mode sit remote 145.100.1.5 [local 145.100.24.181 ttl 255]
959 # ip link set sixbone up
960 # ip addr add 3FFE:604:6:7::2/126 dev sixbone
961 # ip route add 3ffe::0/16 dev sixbone
964 Let's discuss this. In the first line, we created a tunnel device called
965 sixbone. We gave it mode sit (which is IPv6 in IPv4 tunneling) and told it
966 where to go to (remote) and where to come from (local). TTL is set to
969 Next, we made the device active (up). After that, we added our own network
970 address, and set a route for 3ffe::/15 (which is currently all of 6bone)
971 through the tunnel. If the particular machine you run this on is your IPv6
972 gateway, then consider adding the following lines:
975 # echo 1 >/proc/sys/net/ipv6/conf/all/forwarding
976 # /usr/local/sbin/radvd
978 The latter, radvd is -like zebra- a router advertisement daemon, to
979 support IPv6's autoconfiguration features. Search for it with your favourite
980 search-engine if you like.
981 You can check things like this:
984 # /sbin/ip -f inet6 addr
987 If you happen to have radvd running on your IPv6 gateway and boot your
988 IPv6 capable Linux on a machine on your local LAN, you would be able to
989 enjoy the benefits of IPv6 autoconfiguration:
991 # /sbin/ip -f inet6 addr
992 1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue inet6 ::1/128 scope host
994 3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
995 inet6 3ffe:604:6:8:5054:4cff:fe01:e3d6/64 scope global dynamic
996 valid_lft forever preferred_lft 604646sec inet6 fe80::5054:4cff:fe01:e3d6/10
1000 You could go ahead and configure your bind for IPv6 addresses. The A
1001 type has an equivalent for IPv6: AAAA. The in-addr.arpa's equivalent is:
1002 ip6.int. There's a lot of information available on this topic.
1004 There is an increasing number of IPv6-aware applications available,
1005 including secure shell, telnet, inetd, Mozilla the browser, Apache the
1006 websever and a lot of others. But this is all outside the scope of this
1007 Routing document ;-)
1009 On the Cisco side the configuration would be something like this:
1013 description IPv6 tunnel
1015 no ip directed-broadcast
1017 ipv6 address 3FFE:604:6:7::1/126
1018 tunnel source Serial0
1019 tunnel destination 145.100.24.181
1022 ipv6 route 3FFE:604:6:8::/64 Tunnel1
1024 But if you don't have a Cisco at your disposal, try one of the many
1025 IPv6 tunnel brokers available on the Internet. They are willing to configure
1026 their Cisco with an extra tunnel for you. Mostly by means of a friendly
1027 web interface. Search for "ipv6 tunnel broker
" on your favourite search engine.
1029 <sect>IPsec: secure IP over the Internet
1031 FIXME: editor vacancy.
1032 In the meantime, see: <url url="http://www.freeswan.org/
" name="The
1033 FreeS/WAN project
">. Another IPSec implementation for Linux is Cerberus,
1034 by NIST. However, their web pages have not been updated in over a year,
1035 and their version tended to trail well behind the current Linux kernel.
1036 USAGI, an alternative IPv6 implementation for Linux, also includes an
1037 IPSec implementation, but that might only be for IPv6.
1039 <sect>Multicast routing
1041 FIXME: Editor Vacancy!
1043 The Multicast-HOWTO is ancient (relatively-speaking) and may be inaccurate
1044 or misleading in places, for that reason.
1046 Before you can do any multicast routing, you need to configure the Linux
1047 kernel to support the type of multicast routing you want to do. This, in
1048 turn, requires you to decide what type of multicast routing you expect to
1049 be using. There are essentially four "common
" types - DVMRP (the Multicast
1050 version of the RIP unicast protocol), MOSPF (the same, but for OSPF), PIM-SM
1051 ("Protocol Independent Multicasting - Sparse Mode
", which assumes that users
1052 of any multicast group are spread out, rather than clumped) and PIM-DM (the
1053 same, but "Dense Mode
", which assumes that there will be significant clumps
1054 of users of the same multicast group).
1056 In the Linux kernel, you will notice that these options don't appear. This is
1057 because the protocol itself is handled by a routing application, such as
1058 Zebra, mrouted, or pimd. However, you still have to have a good idea of which
1059 you're going to use, to select the right options in the kernel.
1061 For all multicast routing, you will definitely need to enable "multicasting
"
1062 and "multicast routing
". For DVMRP and MOSPF, this is sufficient. If you are
1063 going to use PIM, you must also enable PIMv1 or PIMv2, depending on whether
1064 the network you are connecting to uses version 1 or 2 of the PIM protocol.
1066 Once you have all that sorted out, and your new Linux kernel compiled, you
1067 will see that the IP protocols listed, at boot time, now include IGMP. This
1068 is a protocol for managing multicast groups. At the time of writing, Linux
1069 supports IGMP versions 1 and 2 only, although version 3 does exist and has
1070 been documented. This doesn't really affect us that much, as IGMPv3 is still
1071 new enough that the extra capabilities of IGMPv3 aren't going to be that
1072 much use. Because IGMP deals with groups, only the features present in the
1073 simplest version of IGMP over the entire group are going to be used. For the
1074 most part, that will be IGMPv2, although IGMPv1 is sill going to be
1077 So far, so good. We've enabled multicasting. Now, we have to tell the Linux
1078 kernel to actually do something with it, so we can start routing. This means
1079 adding the Multicast virtual network to the router table:
1081 ip route add 224.0.0.0/4 dev eth0
1083 (Assuming, of course, that you're multicasting over eth0! Substitute the
1084 device of your choice, for this.)
1086 Now, tell Linux to forward packets...
1088 echo 1 > /proc/sys/net/ipv4/ip_forward
1090 At this point, you may be wondering if this is ever going to do anything. So,
1091 to test our connection, we ping the default group, 224.0.0.1, to see if anyone
1092 is alive. All machines on your LAN with multicasting enabled <em>should</em>
1093 respond, but nothing else. You'll notice that none of the machines that
1094 respond have an IP address of 224.0.0.1. What a surprise! :) This is a group
1095 address (a "broadcast
" to subscribers), and all members of the group will
1096 respond with their own address, not the group address.
1100 At this point, you're ready to do actual multicast routing. Well, assuming
1101 that you have two networks to route between.
1105 <sect>Queueing Disciplines for Bandwidth Management
1107 Now, when I discovered this, it <em>really</em> blew me away. Linux 2.2/2.4
1108 comes with everything to manage bandwidth in ways comparable to high-end
1109 dedicated bandwidth management systems.
1111 Linux even goes far beyond what Frame and ATM provide.
1113 Just to prevent confusion, tc uses the following rules for bandwith
1116 mbps = 1024 kbps = 1024 * 1024 bps => byte/s
1117 mbit = 1024 kbit => kilo bit/s.
1118 mb = 1024 kb = 1024 * 1024 b => byte
1119 mbit = 1024 kbit => kilo bit.
1121 Internally, the number is stored in bps and b.
1123 But when tc prints the rate, it uses following :
1125 1Mbit = 1024 Kbit = 1024 * 1024 bps => bit/s
1128 <sect1>Queues and Queueing Disciplines explained
1130 With queueing we determine the way in which data is <em>sent</em>. It is
1131 important to realise that we can only shape data that we transmit.
1133 With the way the Internet works, we have no direct control of what people
1134 send us. It's a bit like your (physical!) mailbox at home. There is no way
1135 you can influence the world to modify the amount of mail they send you,
1136 short of contacting everybody.
1138 However, the Internet is mostly based on TCP/IP which has a few features
1139 that help us. TCP/IP has no way of knowing the capacity of the network
1140 between two hosts, so it just starts sending data faster and faster ('slow
1141 start') and when packets start getting lost, because there is no room to
1142 send them, it will slow down. In fact it is a bit smarter than this, but
1143 more about that later.
1145 This is the equivalent of not reading half of your mail, and hoping that
1146 people will stop sending it to you. With the difference that it works for
1149 If you have a router and wish to prevent certain hosts within your network
1150 from downloading too fast, you need to do your shaping on the *inner* interface
1151 of your router, the one that sends data to your own computers.
1153 You also have to be sure you are controlling the bottleneck of the link.
1154 If you have a 100Mbit NIC and you have a router that has a 256kbit link,
1155 you have to make sure you are not sending more data than your router can
1156 handle. Othewise, it will be the router who is controlling the link and
1157 shaping the available bandwith. We need to 'own the queue' so to speak, and
1158 be the slowest link in the chain. Luckily this is easily possible.
1160 <sect1>Simple, classless Queueing Disciplines
1162 As said, with queueing disciplines, we change the way data is sent.
1163 Classless queueing disciplines are those that, by and large accept data and
1164 only reschedule, delay or drop it.
1166 These can be used to shape traffic for an entire interface, without any
1167 subdivisions. It is vital that you understand this part of queueing before
1168 we go on the the classful qdisc-containing-qdiscs!
1170 By far the most widely used discipline is the pfifo_fast qdisc - this is the
1171 default. This also explains why these advanced features are so robust. They
1172 are nothing more than 'just another queue'.
1174 Each of these queues has specific strengths and weaknesses. Not all of them
1175 may be as well tested.
1179 This queue is, as the name says, First In, First Out, which means that no
1180 packet receives special treatment. At least, not quite. This queue has 3 so
1181 called 'bands'. Within each band, FIFO rules apply. However, as long as
1182 there are packets waiting in band 0, band 1 won't be processed. Same goes
1183 for band 1 and band 2.
1185 The kernel honors the so called Type of Service flag of packets, and takes
1186 care to insert 'minimum delay' packets in band 0.
1188 Do not confuse this classless simple qdisc with the classful PRIO one!
1189 Although they behave similarly, pfifo_fast is classless and you cannot add
1190 other qdiscs to it with the tc command.
1192 <sect3>Parameters & usage
1194 You can't configure the pfifo_fast qdisc as it is the hardwired default.
1195 This is how it is configured by default:
1198 Determines how packet priorities, as assigned by the kernel, map to bands.
1199 Mapping occurs based on the TOS octet of the packet, which looks like this:
1203 +-----+-----+-----+-----+-----+-----+-----+-----+
1205 | PRECEDENCE | TOS | MBZ |
1207 +-----+-----+-----+-----+-----+-----+-----+-----+
1210 The four TOS bits (the 'TOS field') are defined as:
1212 Binary Decimcal Meaning
1213 -----------------------------------------
1214 1000 8 Minimize delay (md)
1215 0100 4 Maximize throughput (mt)
1216 0010 2 Maximize reliability (mr)
1217 0001 1 Minimize monetary cost (mmc)
1218 0000 0 Normal Service
1221 As there is 1 bit to the right of these four bits, the actual value of the
1222 TOS field is double the value of the TOS bits. Tcpdump -v -v shows you the
1223 value of the entire TOS field, not just the four bits. It is the value you
1224 see in the first column of this table:
1227 TOS Bits Means Linux Priority Band
1228 ------------------------------------------------------------
1229 0x0 0 Normal Service 0 Best Effort 1
1230 0x2 1 Minimize Monetary Cost 1 Filler 2
1231 0x4 2 Maximize Reliability 0 Best Effort 1
1232 0x6 3 mmc+mr 0 Best Effort 1
1233 0x8 4 Maximize Throughput 2 Bulk 2
1234 0xa 5 mmc+mt 2 Bulk 2
1235 0xc 6 mr+mt 2 Bulk 2
1236 0xe 7 mmc+mr+mt 2 Bulk 2
1237 0x10 8 Minimize Delay 6 Interactive 0
1238 0x12 9 mmc+md 6 Interactive 0
1239 0x14 10 mr+md 6 Interactive 0
1240 0x16 11 mmc+mr+md 6 Interactive 0
1241 0x18 12 mt+md 4 Int. Bulk 1
1242 0x1a 13 mmc+mt+md 4 Int. Bulk 1
1243 0x1c 14 mr+mt+md 4 Int. Bulk 1
1244 0x1e 15 mmc+mr+mt+md 4 Int. Bulk 1
1247 Lots of numbers. The second column contains the value of the relevant four
1248 TOS bits, followed by their translated meaning. For example, 15 stands for a
1249 packet wanting Minimal Montetary Cost, Maximum Reliability, Maximum
1250 Throughput AND Minimum Delay. I would call this a 'Dutch Packet'.
1252 The fourth column lists the way the Linux kernel interprets the TOS bits, by
1253 showing to which Priority they are mapped.
1255 The last column shows the result of the default priomap. On the commandline,
1256 the default priomap looks like this:
1258 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
1261 This means that priority 4, for example, gets mapped to band number 1. The
1262 priomap also allows you to list higher priorities (> 7) which do not
1263 correspond to TOS mappings, but which are set by other means.
1265 This table from RFC 1349 (read it for more details) tells you how
1266 applications might very well set their TOS bits:
1268 TELNET 1000 (minimize delay)
1270 Control 1000 (minimize delay)
1271 Data 0100 (maximize throughput)
1273 TFTP 1000 (minimize delay)
1276 Command phase 1000 (minimize delay)
1277 DATA phase 0100 (maximize throughput)
1280 UDP Query 1000 (minimize delay)
1282 Zone Transfer 0100 (maximize throughput)
1284 NNTP 0001 (minimize monetary cost)
1288 Requests 0000 (mostly)
1289 Responses <same as request> (mostly)
1292 <tag>txqueuelen</tag>
1293 The length of this queue is gleaned from the interface configuration, which
1294 you can see and set with ifconfig and ip. To set the queue length to 10,
1295 execute: ifconfig eth0 txqueuelen 10
1297 You can't set this parameter with tc!
1299 <sect2>Token Bucket Filter
1301 The Token Bucket Filter (TBF) is a simple qdisc that only passes packets
1302 arriving at a rate which is not exceeding some administratively set rate, but
1303 with the possibility to allow short bursts in excess of this rate.
1305 TBF is very precise, network- and processor friendly. It should be your
1306 first choice if you simply want to slow an interface down!
1308 The TBF implementation consists of a buffer (bucket), constantly filled by
1309 some virtual pieces of information called tokens, at a specific rate (token
1310 rate). The most important parameter of the bucket is its size, that is the
1311 number of tokens it can store.
1313 Each arriving token collects one incoming data packet from the data queue
1314 and is then deleted from the bucket. Associating this algorithm
1315 with the two flows -- token and data, gives us three possible scenarios:
1318 <item> The data arrives in TBF at a rate that's <em>equal</em> to the rate
1319 of incoming tokens. In this case each incoming packet has its matching token
1320 and passes the queue without delay.
1322 <item> The data arrives in TBF at a rate that's <em>smaller</em> than the
1323 token rate. Only a part of the tokens are deleted at output of each data packet
1324 that's sent out the queue, so the tokens accumulate, up to the bucket size.
1325 The unused tokens can then be used to send data a a speed that's exceeding the
1326 standard token rate, in case short data bursts occur.
1328 <item> The data arrives in TBF at a rate <em>bigger</em> than the token rate.
1329 This means that the bucket will soon be devoid of tokens, which causes the
1330 TBF to throttle itself for a while. This is called an 'overlimit situation'.
1331 If packets keep coming in, packets will start to get dropped.
1334 The last scenario is very important, because it allows to
1335 administratively shape the bandwidth available to data that's passing
1338 The accumulation of tokens allows a short burst of overlimit data to be
1339 still passed without loss, but any lasting overload will cause packets to be
1340 constantly delayed, and then dropped.
1342 Please note that in the actual implementation, tokens correspond to bytes,
1344 <sect3>Parameters & usage
1346 Even though you will probably not need to change them, tbf has some knobs
1347 available. First the parameters that are always available:
1349 <tag>limit or latency</tag>
1350 Limit is the number of bytes that can be queued waiting for tokens to become
1351 available. You can also specify this the other way around by setting the
1352 latency parameter, which specifies the maximum amount of time a packet can
1353 sit in the TBF. The latter calculation takes into account the size of the
1354 bucket, the rate and possibly the peakrate (if set).
1356 <tag>burst/buffer/maxburst</tag>
1357 Size of the bucket, in bytes. This is the maximum amount of bytes that
1358 tokens can be available for instantaneously. In general, larger shaping
1359 rates require a larger buffer. For 10mbit/s on Intel, you need at least
1360 10kbyte buffer if you want to reach your configured rate!
1362 If your buffer is too small, packets may be dropped because more tokens
1363 arrive per timer tick than fit in your bucket.
1365 A zero-sized packet does not use zero bandwidth. For ethernet, no packet
1366 uses less than 64 bytes. The Minimum Packet Unit determines the minimal
1367 token usage for a packet.
1369 The speedknob. See remarks above about limits!
1372 If the bucket contains tokens and is allowed to empty, by default it does so
1373 at infinite speed. If this is unacceptable, use the following parameters:
1377 If tokens are available, and packets arrive, they are sent out immediately
1378 by default, at 'lightspeed' so to speak. That may not be what you want,
1379 especially if you have a large bucket.
1381 The peakrate can be used to specify how quickly the bucket is allowed to be
1382 depleted. If doing everything by the book, this is achieved by releasing a
1383 packet, and then wait just long enough, and release the next. We calculated
1384 our waits so we send just at peakrate.
1386 However, due to de default 10ms timer resolution of Unix, with 10.000 bits
1387 average packets, we are limited to 1mbit/s of peakrate!
1389 <tag>mtu/minburst</tag>
1390 The 1mbit/s peakrate is not very useful if your regular rate is more than
1391 that. A higher peakrate is possible by sending out more packets per
1392 timertick, which effectively means that we create a second bucket!
1394 This second bucket defaults to a single packet, which is not a bucket at
1397 To calculate the maximum possible peakrate, multiply the configured mtu by
1398 100 (or more correctly, HZ, which is 100 on intel, 1024 on Alpha).
1401 <sect3>Sample configuration
1403 A simple but *very* useful configuration is this:
1405 # tc qdisc add dev ppp0 root tbf rate 220kbit latency 50ms burst 1540
1408 Ok, why is this useful? If you have a networking device with a large queue,
1409 like a DSL modem or a cablemodem, and you talk to it over a fast device,
1410 like over an ethernet interface, you will find that uploading absolutely
1411 destroys interactivity.
1413 This is because uploading will fill the queue in the modem, which is
1414 probably *huge* because this helps actually achieving good data throughput
1415 uploading. But this is not what you want, you want to have the queue not too
1416 big so interactivity remains and you can still do other stuff while sending
1419 The line above slows down sending to a rate that does not lead to a queue in
1420 the modem - the queue will be in Linux, where we can control it to a limited
1423 Change 220kbit to your uplink's *actual* speed, minus a few percent. If you
1424 have a really fast modem, raise 'burst' a bit.
1425 <sect2>Stochastic Fairness Queueing
1427 Stochastic Fairness Queueing (SFQ) is a simple implementation of the fair
1428 queueing algorithms family. It's less accurate than others, but it also
1429 requires less calculations while being almost perfectly fair.
1431 The key word in SFQ is conversation (or flow), which mostly corresponds to a
1432 TCP session or a UDP stream. Traffic is divided into a pretty large number
1433 of FIFO queues, one for each conversation. Traffic is then sent in a round
1434 robin fashion, giving each session the chance to send data in turn.
1436 This leads to very fair behaviour and disallows any single conversation from
1437 drowning out the rest. SFQ is called 'Stochastic' because it doesn't really
1438 allocate a queue for each session, it has an algorithm which divides traffic
1439 over a limited number of queues using a hashing algorithm.
1441 Because of the hash, multiple sessions might end up in the same bucket, which
1442 would halve each session's chance of sending a packet, thus halving the
1443 effective speed available. To prevent this situation from becoming
1444 noticeable, SFQ changes its hashing algorithm quite often so that any two
1445 colliding sessions will only do so for a small number of seconds.
1447 It is important to note that SFQ is only useful in case your actual outgoing
1448 interface is really full! If it isn't then there will be no queue on your
1449 linux machine and hence no effect. Later on we will describe how to combine
1450 SFQ with other qdiscs to get a best-of-both worlds situation.
1452 Specifically, setting SFQ on the ethernet interface heading to your
1453 cablemodem or DSL router is pointless without further shaping!
1454 <sect3>Parameters & usage
1456 The SFQ is pretty much selftuning:
1459 Reconfigure hashing once this many seconds. If unset, hash will never be
1460 reconfigured. Not recommended. 10 seconds is probably a good value.
1462 Amount of bytes a stream is allowed to dequeue before the next queue gets a
1463 turn. Defaults to 1 maximum sized packet (MTU-sized). Do not set below the
1466 <sect3>Sample configuration
1468 If you have a device which has identical link speed and actual available
1469 rate, like a phone modem, this configuration will help promote fairness:
1471 # tc qdisc add dev ppp0 root sfq perturb 10
1473 qdisc sfq 800c: dev ppp0 quantum 1514b limit 128p flows 128/1024 perturb 10sec
1474 Sent 4812 bytes 62 pkts (dropped 0, overlimits 0)
1477 The number 800c: is the automatically assigned handle number, limit means
1478 that 128 packets can wait in this queue. There are 1024 hashbuckets
1479 available for accounting, of which 128 can be active at a time (no more
1480 packets fit in the queue!) Once every 10 seconds, the hashes are
1483 <sect1>Advice for when to use which queue
1485 Summarizing, these are the simple queues that actually manage traffic by
1486 reordering, slowing or dropping packets.
1488 The following tips may help in chosing which queue to use. It mentions some
1489 qdiscs described in the 'Advanced & less common queueing disciplines'.
1493 To purely slow down outgoing traffic, use the Token Bucket Filter. Works up
1494 to huge bandwidths, if you scale the bucket.
1496 If your link is truly full and you want to make sure that no single session
1497 can dominate your outgoing bandwidth, use Stochastical Fairness Queueing.
1499 If you have a big backbone and know what you are doing, consider Random
1500 Early Drop (see Advanced chapter).
1502 To 'shape' incoming traffic which you are not forwarding, use the Ingress
1503 Policer. Incoming shaping is called 'policing', by the way, not 'shaping'.
1505 If you *are* forwarding it, use a TBF on the interface you are forwarding
1506 the data to. Unless you want to shape traffic that may go out over several
1507 interfaces, in which case the only common factor is the incoming interface.
1508 In that case use the Ingress Policer.
1510 If you don't want to shape, but only want to see if your interface is so
1511 loaded that it has to queue, use the pfifo queue (not pfifo_fast). It lacks
1512 internal bands but does account the size of its backlog.
1514 Finally - you can also do 'social shaping'. You may not always be able to
1515 use technology to achieve what you want. Users experience technical
1516 constraints as hostile. A kind word may also help with getting your
1517 bandwidth to be divided right!
1521 To properly understand more complicated configurations it is necessary to
1522 explain a few concepts first. Because of the complexity and he relative
1523 youth of the subject, a lot of different words are used when people in fact
1524 mean the same thing.
1526 The following is loosely based on draft-ietf-diffserv-model-06.txt, 'An
1527 Informal Management Model for Diffserv Routers'. It can currently be found
1528 at http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt.
1530 Read it for the strict definitions of the terms used.
1532 <tag>Queueing Discipline</tag>
1533 An algorithm that manages the queue of a device, either incoming (ingress)
1534 or outgoing (egress).
1535 <tag>Classless qdisc</tag>
1536 A qdisc with no configurable internal subdivisions.
1537 <tag>Classful qdisc</tag>
1538 A classful qdisc contains multiple classes. Each of these classes contains a
1539 further qdisc, which may again be classful, but need not be. According to
1540 the strict definition, pfifo_fast *is* classful, because it contains three
1541 bands which are, in fact, classes. However, from the user's configuration
1542 perspective, it is classless as the classes can't be touched with the tc
1545 A classful qdisc may have many classes, which each are internal to the
1546 qdisc. Each of these classes may contain a real qdisc.
1547 <tag>Classifier</tag>
1548 Each classful qdisc needs to determine to which class it needs to send a
1549 packet. This is done using the classifier.
1551 Classification can be performed using filters. A filter contains a number of
1552 conditions which if matched, make the filter match.
1553 <tag>Scheduling</tag>
1554 A qdisc may, with the help of a classifier, decide that some packets need to
1555 go out earlier than others. This process is called Scheduling, and is
1556 performed for example by the pfifo_fast qdisc mentioned earlier. Scheduling
1557 is also called 'reordering', but this is confusing.
1559 The process of delaying packets before they go out to make traffic confirm
1560 to a configured maximum rate. Shaping is performed on egress. Colloquially,
1561 dropping packets to slow traffic down is also often called Shaping.
1563 Delaying or dropping packets in order to make traffic stay below a
1564 configured bandwidth. In Linux, policing can only drop a packet and not
1565 delay it - there is no 'ingress queue'.
1566 <tag>Work-Conserving</tag>
1567 A work-conserving qdisc always delivers a packet if one is available. In
1568 other words, it never delays a packet if the network adaptor is ready to
1569 send one (in the case of an egress qdisc).
1570 <tag>non-Work-Conserving</tag>
1571 Some queues, like for example the Token Bucket Filter, may need to hold on
1572 to a packet for a certain time in order to limit the bandwidth. This means
1573 that they sometimes refuse to give up a packet, even though they have one
1577 Now that we have our terminology straight, let's see where all these things
1584 +---------------+-----------------------------------------+
1586 | -------> IP Stack |
1591 | | / ----------> Forwarding -> |
1596 | | Egress /--qdisc2--\ |
1597 --->->Ingress Classifier ---qdisc3---- | ->
1598 | Qdisc \__qdisc4__/ |
1601 +----------------------------------------------------------+
1603 Thanks to Jamal Hadi Salim for this ascii representation.
1605 The big block represents the kernel. The leftmost arrow represents traffic
1606 entering your machine from the network. It is then fed to the Ingress
1607 Qdisc which may apply Filters to a packet, and decide to drop it. This
1608 is called 'Policing'.
1610 This happens at a very early stage, before it has seen a lot of the kernel.
1611 It is therefore a very good place to drop traffic very early, without
1612 consuming a lot of CPU power.
1614 If the packet is allowed to continue, it may be destined for a local
1615 application, in which case it enters the IP stack in order to be processed,
1616 and handed over to a userspace program. The packet may also be forwarded
1617 without entering an application, in which case it is destined for egress.
1618 Userspace programs may also deliver data, which is then examined and
1619 forwarded to the Egress Classifier.
1621 There it is investigated and enqueued to any of a number of qdiscs. In the
1622 unconfigured default case, there is only one egress qdisc installed, the
1623 pfifo_fast, which always receives the packet. This is called 'enqueueing'.
1625 The packet now sits in the qdisc, waiting for the kernel to ask for
1626 it for transmission over the network interface. This is called 'dequeueing'.
1628 This picture also holds in case there is only one network adaptor - the
1629 arrows entering and leaving the kernel should not be taken too literally.
1630 Each network adaptor has both ingress and egress hooks.
1632 <sect1>Classful Queueing Disciplines
1634 Classful qdiscs are very useful if you have different kinds of traffic which
1635 should have differing treatment. One of the classful qdiscs is called 'CBQ'
1636 , 'Class Based Queueing' and it is so widely mentioned that people identify
1637 queueing with classes solely with CBQ, but this is not the case.
1639 CBQ is merely the oldest kid on the block - and also the most complex one.
1640 It may not always do what you want. This may come as something of a shock
1641 to many who fell for the 'sendmail effect', which teaches us that any
1642 complex technology which doesn't come with documentation must be the best
1645 More about CBQ and its alternatives shortly.
1646 <sect2>Flow within classful qdiscs & classes
1648 When traffic enters a classful qdisc, it needs to be sent to any of the
1649 classes within - it needs to be 'classified'. To determine what to do with a
1650 packet, the so called 'filters' are consulted. It is important to know that
1651 the filters are called from within a qdisc, and not the other way around!
1653 The filters attached to that qdisc then return with a decision, and the
1654 qdisc uses this to enqueue the packet into one of the classes. Each subclass
1655 may try other filters to see if further instructions apply. If not, the
1656 class enqueues the packet to the qdisc it contains.
1658 Besides containing other qdiscs, most classful qdiscs also perform shaping.
1659 This is useful to perform both packet scheduling (with SFQ, for example) and
1660 rate control. You need this in cases where you have a high speed
1661 interface (for example, ethernet) to a slower device (a cable modem).
1663 If you were only to run SFQ, nothing would happen, as packets enter &
1664 leave your router without delay: the output interface is far faster than
1665 your actual link speed. There is no queue to schedule then.
1667 <sect2>The qdisc family: roots, handles, siblings and parents
1669 Each interface has one egress 'root qdisc', by default the earlier mentioned
1670 classless pfifo_fast queueing discipline. Each qdisc can be assigned a
1671 handle, which can be used by later configuration statements to refer to that
1672 qdisc. Besides an egress qdisc, an interface may also have an ingress, which
1673 polices traffic coming in.
1675 The handles of these qdiscs consist of two parts, a major number and a minor
1676 number. It is habitual to name the root qdisc '1:', which is equal to '1:0'.
1677 The minor number of a qdisc is always 0.
1679 Classes need to have the same major number as their parent.
1680 <sect3>How filters are used to classify traffic
1682 Recapping, a typical hierarchy might look like this:
1695 But don't let this tree fool you! You should *not* imagine the kernel to be
1696 at the apex of the tree and the network below, that is just not the case.
1697 Packets get enqueued and dequeued at the root qdisc, which is the only thing
1698 the kernel talks to.
1700 A packet might get classified in a chain like this:
1702 1: -> 1:1 -> 12: -> 12:2
1704 The packet now resides in a queue in a qdisc attached to class 12:2. In this
1705 example, a filter was attached to each 'node' in the tree, each chosing a
1706 branch to take next. This can make sense. However, this is also possible:
1710 In this case, a filter attached to the root decided to send the packet
1713 <sect3>How packets are dequeued to the hardware
1715 When the kernel decides that it needs to extract packets to send to the
1716 interface, the root qdisc 1: gets a dequeue request, which is passed to
1717 1:1, which is in turn passed to 10:, 11: and 12:, which each query their
1718 siblings, and try to dequeue() from them. In this case, the kernel needs to
1719 walk the entire tree, because only 12:2 contains a packet.
1721 In short, nested classes ONLY talk to their parent qdiscs, never to an
1722 interface. Only the root qdisc gets dequeued by the kernel!
1724 The upshot of this is that classes never get dequeued faster than their
1725 parents allow. And this is exactly what we want: this way we can have SFQ in
1726 an inner class, which doesn't do any shaping, only scheduling, and have a
1727 shaping outer qdisc, which does the shaping.
1728 <sect2>The PRIO qdisc
1730 The PRIO qdisc doesn't actually shape, it only subdivides traffic based on
1731 how you configured your filters. You can consider the PRIO qdisc a kind
1732 of pfifo_fast on stereoids, whereby each band is a separate class instead of
1735 When a packet is enqueued to the PRIO qdisc, a class is chosen based on the
1736 filter commands you gave. By default, three classes are created. These
1737 classes by default contain pure FIFO qdiscs with no internal
1738 structure, but you can replace these by any qdisc you have available.
1740 Whenever a packet needs to be dequeued, class :1 is tried first. Higher
1741 classes are only used if lower bands all did not give up a packet.
1743 This qdisc is very useful in case you want to prioritize certain kinds of
1744 traffic without using only TOS-flags but using all the power of the tc
1745 filters. It can also contain more all qdiscs, whereas pfifo_fast is limited
1746 to simple fifo qdiscs.
1748 Because it doesn't actually shape, the same warning as for SFQ holds: either
1749 use it only if your physical link is really full or wrap it inside a
1750 classful qdisc that does shape. The last holds for almost all cablemodems
1753 In formal words, the PRIO qdisc is a Work-Conserving scheduler.
1754 <sect3>PRIO parameters & usage
1756 The following parameters are recognized by tc:
1759 Number of bands to create. Each band is in fact a class. If you change this
1760 number, you must also change:
1762 If you do not provide tc filters to classify traffic, the PRIO qdisc looks
1763 at the TC_PRIO priority to decide how to enqueue traffic.
1765 This works just like with the pfifo_fast qdisc mentioned earlier, see there
1768 The bands are classes, and are called major:1 to major:3 by default, so if
1769 your PRIO qdisc is called 12:, tc filter traffic to 12:1 to grant it more
1772 Reiterating, band 0 goes to minor number 1! Band 1 to minor number 2, etc.
1773 <sect3>Sample configuration
1775 We will create this tree:
1786 Bulk traffic will go to 30:, interactive traffic to 20: or 10:.
1790 # tc qdisc add dev eth0 root handle 1: prio
1791 ## This *instantly* creates classes 1:1, 1:2, 1:3
1793 # tc qdisc add dev eth0 parent 1:1 handle 10: sfq
1794 # tc qdisc add dev eth0 parent 1:2 handle 20: tbf rate 20kbit buffer 1600 limit 3000
1795 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
1798 Now lets's see what we created:
1800 # tc -s qdisc ls dev eth0
1801 qdisc sfq 30: quantum 1514b
1802 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
1804 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
1805 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
1807 qdisc sfq 10: quantum 1514b
1808 Sent 132 bytes 2 pkts (dropped 0, overlimits 0)
1810 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
1811 Sent 174 bytes 3 pkts (dropped 0, overlimits 0)
1813 As you can see, band 0 has already had some traffic, and one packet was sent
1814 while running this command!
1816 We now do some bulk data transfer with a tool that properly sets TOS flags,
1817 and take another look:
1819 # scp tc ahu@10.0.0.11:./
1820 ahu@10.0.0.11's password:
1821 tc 100% |*****************************| 353 KB 00:00
1822 # tc -s qdisc ls dev eth0
1823 qdisc sfq 30: quantum 1514b
1824 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
1826 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
1827 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
1829 qdisc sfq 10: quantum 1514b
1830 Sent 2230 bytes 31 pkts (dropped 0, overlimits 0)
1832 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
1833 Sent 389140 bytes 326 pkts (dropped 0, overlimits 0)
1835 As you can see, all traffic went to handle 30:, which is the lowest priority
1836 band, just as intended. Now to verify that interactive traffic goes to
1837 higher bands, we create some interactive traffic:
1840 # tc -s qdisc ls dev eth0
1841 qdisc sfq 30: quantum 1514b
1842 Sent 384228 bytes 274 pkts (dropped 0, overlimits 0)
1844 qdisc tbf 20: rate 20Kbit burst 1599b lat 667.6ms
1845 Sent 2640 bytes 20 pkts (dropped 0, overlimits 0)
1847 qdisc sfq 10: quantum 1514b
1848 Sent 14926 bytes 193 pkts (dropped 0, overlimits 0)
1850 qdisc prio 1: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
1851 Sent 401836 bytes 488 pkts (dropped 0, overlimits 0)
1854 It worked - all additional traffic has gone to 10:, which is our highest
1855 priority qdisc. No traffic was sent to the lowest priority, which previously
1856 received our entire scp.
1858 <sect2>The famous CBQ qdisc
1860 As said before, CBQ is the most complex qdisc available, the most hyped, the
1861 least understood, and probably the trickiest one to get right. This is not
1862 because the authors are evil or incompetent, far from it, it's just that the
1863 CBQ algorithm isn't all that precise and doesn't really match the way Linux
1866 Besides being classful, CBQ is also a shaper and it is in that aspect that
1867 it really doesn't work very well. It should work like this. If you try to
1868 shape a 10mbit/s connection to 1mbit/s, the link should be idle 90% of the
1869 time. If it isn't, we need to throttle so that it IS idle 90% of the time.
1871 This is pretty hard to measure, so CBQ instead derives the idle time from
1872 the number of microseconds that elapse between requests from the hardware
1873 layer for more data. Combined, this can be used to approximate how full or
1876 This is rather circumspect and doesn't always arrive at proper results. For
1877 example, what if the actual link speed of an interface that is not really
1878 able to transmit the full 100mbit/s of data, perhaps because of a badly
1879 implemented driver? A PCMCIA network card will also never achieve 100mbit/s
1880 because of the way the bus is designed - again, how do we calculate the idle
1883 It gets even worse if we consider not-quite-real network devices like PPP
1884 over Ethernet or PPTP over TCP/IP. The effective bandwidth in that case is
1885 probably determined by the efficiency of pipes to userspace - which is huge.
1887 People who have done measurements discover that CBQ is not always very
1888 accurate and sometimes completely misses the mark.
1890 In many circumstances however it works well. With the documentation provided
1891 here, you should be able to configure it to work well in most cases.
1892 <sect3>CBQ shaping in detail
1894 As said before, CBQ works by making sure that the link is idle just long
1895 enough to bring down the real bandwidth to the configured rate. To do so, it
1896 calculates the time that should pass between average packets.
1898 During operations, the effective idletime is measured using an exponential
1899 weighted moving average (EWMA), which considers recent packets to be
1900 exponentially more important than past ones. The unix loadaverage is
1901 calculated in the same way.
1903 The calculated idle time is substracted from the EWMA measured one, the
1904 resulting number is called 'avgidle'. A perfectly loaded link has an avgidle
1905 of zero: packets arrive exactly once every calculated interval.
1907 An overloaded link has a negative avgidle and if it gets too negative, CBQ
1908 shuts down for a while and is then 'overlimit'.
1910 Conversely, an idle link might amass a huge avgidle, which would then allow
1911 infinite bandwidths after a few hours of silence. To prevent this, avgidle is
1914 If overlimit, in theory, the CBQ could throttle itself for exactly the
1915 amount of time that was calculated to pass between packets, and then pass
1916 one packet, and throttle again. But see the 'minburst' parameter below.
1918 These are parameters you can specify in order to configure shaping:
1921 Average size of a packet, measured in bytes. Needed for calculating maxidle,
1922 which is derived from maxburst, which is specified in packets.
1923 <tag>bandwidth</tag>
1924 The physical bandwidth of your device, needed for idle time
1927 The time a packet takes to be transmitted over a device may grow in steps,
1928 based on the packet size. An 800 and an 806 size packet may take just as long
1929 to send, for example - this sets the granularity. Most often set to '8'.
1930 Must be an integral power of two.
1932 This number of packets is used to calculate maxidle so that when avgidle is
1933 at maxidle, this number of average packets can be burst before avgidle drops
1934 to 0. Set it higher to be more tolerant of bursts. You can't set maxidle
1935 directly, only via this parameter.
1937 As mentioned before, CBQ needs to throttle in case of overlimit. The ideal
1938 solution is to do so for exactly the calculated idle time, and pass 1
1939 packet. However, Unix kernels generally have a hard time scheduling events
1940 shorter than 10ms, so it is better to throttle for a longer period, and then
1941 pass minburst packets in one go, and then sleep minburst times longer.
1943 The time to wait is called the offtime. Higher values of minburst lead to
1944 more accurate shaping in the long term, but to bigger bursts at millisecond
1947 If avgidle is below 0, we are overlimits and need to wait until avgidle will
1948 be big enough to send one packet. To prevent a sudden burst from shutting
1949 down the link for a prolonged period of time, avgidle is reset to minidle if
1952 Minidle is specified in negative microseconds, so 10 means that avgidle is
1955 Mininum packet size - needed because even a zero size packet is padded
1956 to 64 bytes on ethernet, and so takes a certain time to transmit. CBQ needs
1957 to know this to accurately calculate the idle time.
1959 Desired rate of traffic leaving this qdisc - this is the 'speed knob'!
1962 Internally, CBQ has a lot of finetuning. For example, classes which are
1963 known not to have data enqueued to them aren't queried. Overlimit classes
1964 are penalized by lowering their effective priority. All very smart &
1967 <sect3>CBQ classful behaviour
1969 Besides shaping, using the aforementioned idletime approximations, CBQ also
1970 acts like the PRIO queue in the sense that classes can have differing
1971 priorities and that lower priority numbers will be polled before the higher
1974 Each time a packet is requested by the hardware layer to be sent out to the
1975 network, a weighted round robin process ('WRR') starts, beginning with the
1976 lower priority classes.
1978 These are then grouped and queried if they have data available. If so, it is
1979 returned. After a class has been allowed to dequeue a number of bytes, the
1980 next class within that priority is tried.
1982 The following parameters control the WRR process:
1985 When the outer cbq is asked for a packet to send out on the interface, it
1986 will try all inner qdiscs (in the classes) in turn, in order of
1987 the 'priority' parameter. Each time a class gets its turn, it can only send out
1988 a limited amount of data. 'Allot' is the base unit of this amount. See
1989 the 'weight' parameter for more information.
1992 The CBQ can also act like the PRIO device. Inner classes with lower priority
1993 are tried first and as long as they have traffic, other classes are not
1997 Weight helps in the Weighted Round Robin process. Each class gets a chance
1998 to send in turn. If you have classes with significantly more bandwidth than
1999 other classes, it makes sense to allow them to send more data in one round
2002 A CBQ adds up all weights under a class, and normalizes them, so you can use
2003 arbitrary numbers: only the ratios are important. People have been
2004 using 'rate/10' as a rule of thumb and it appears to work well. The renormalized
2005 weight is multiplied by the 'allot' parameter to determine how much data can
2006 be sent in one round.
2009 Please note that all classes within an CBQ hierarchy need to share the same
2011 <sect3>CBQ parameters that determine link sharing & borrowing
2013 Besides purely limiting certain kinds of traffic, it is also possible to
2014 specify which classes can borrow capacity from other classes or, conversely,
2018 <tag>Isolated/sharing</tag>
2019 A class that is configured with 'isolated' will not lend out bandwidth to
2020 sibling classes. Use this if you have competing or mutually-unfriendly
2021 agencies on your link who do want to give eachother freebies.
2023 The control program tc also knows about 'sharing', which is the reverse
2025 <tag>bounded/borrow</tag>
2026 A class can also be 'bounded', which means that it will not try to borrow
2027 bandwidth from sibling classes. tc also knows about 'borrow', which is the
2028 reverse of 'bounded'.
2030 A typical situation might be where you have two agencies on your link which
2031 are both 'isolated' and 'bounded', which means that they are really limited
2032 to their assigned rate, and also won't allow each other to borrow.
2034 Within such an agency class, there might be other classes which are allowed
2036 <sect3>Sample configuration
2038 This configuration limits webserver traffic to 5mbit and smtp traffic to 3
2039 mbit. Together, they may not get more than 6mbit. We have a 100mbit NIC and
2040 the classes may borrow bandwidth from each other.
2042 # tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 100Mbit \
2044 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 100Mbit \
2045 rate 6Mbit weight 0.6Mbit prio 8 allot 1514 cell 8 maxburst 20 \
2048 This part installs the root and the customary 1:0 class. The 1:1 class is
2049 bounded, so the total bandwidth can't exceed 6mbit.
2051 As said before, CBQ requires a *lot* of knobs. All parameters are explained
2052 above, however. The corresponding HTB configuration is lots simpler.
2055 # tc class add dev eth0 parent 1:1 classid 1:3 cbq bandwidth 100Mbit \
2056 rate 5Mbit weight 0.5Mbit prio 5 allot 1514 cell 8 maxburst 20 \
2058 # tc class add dev eth0 parent 1:1 classid 1:4 cbq bandwidth 100Mbit \
2059 rate 3Mbit weight 0.3Mbit prio 5 allot 1514 cell 8 maxburst 20 \
2063 These are our two classes. Note how we scale the weight with the configured
2064 rate. Both classes are not bounded, but they are connected to class 1:1
2065 which is bounded. So the sum of bandwith of the 2 classes will never be
2066 more than 6mbit. The classid's need to be within the same major number as
2067 the parent CBQ, by the way!
2070 # tc qdisc add dev eth0 parent 1:3 handle 30: sfq
2071 # tc qdisc add dev eth0 parent 1:4 handle 40: sfq
2074 Both classes have a FIFO qdisc by default. But we replaced these with an SFQ
2075 queue so each flow of data is treated equally.
2077 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
2078 sport 80 0xffff flowid 1:3
2079 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip \
2080 sport 25 0xffff flowid 1:4
2083 These commands, attached directly to the root, send traffic to the right
2086 Note that we use 'tc class add' to CREATE classes within a qdisc, but that
2087 we use 'tc qdisc add' to actually add qdiscs to these classes.
2089 You may wonder what happens to traffic that is not classified by any of the
2090 two rules. It appears that in this case, data will then be processed within
2091 1:0, and be unlimited.
2093 If smtp+web together try to exceed the set limit of 6mbit/s, bandwidth will
2094 be divided according to the weight parameter, giving 5/8 of traffic to the
2095 webserver and 3/8 to the mailserver.
2097 With this configuratien you can also say that webserver traffic will always
2098 get at minimum 5/8 * 6 mbit = 3.75 mbit.
2099 <sect3>Other CBQ parameters: split & defmap
2101 As said before, a classful qdisc needs to call filters to determine
2102 which class a packet will be enqueued to.
2104 Besides calling the filter, CBQ offers other options, defmap & split.
2105 This is pretty complicated to understand, and it is not vital. But as this
2106 is the only known place where defmap & split are properly explained, I'm
2109 As you will often want to filter on the Type of Service field only, a special
2110 syntax is provided. Whenever the CBQ needs to figure out where a packet
2111 needs to be enqueued, it checks if this node is a 'split node'. If so, one
2112 of the sub-qdiscs has indicated that it wishes to receive all packets with
2113 a certain configured priority, as might be derived from the TOS field, or
2114 socket options set by applications.
2116 The packets' priority bits are or-ed with the defmap field to see if a match
2117 exists. In other words, this is a short-hand way of creating a very fast
2118 filter, which only matches certain priorities. A defmap of ff (hex) will
2119 match everything, a map of 0 nothing. A sample configuration may help make
2123 # tc qdisc add dev eth1 root handle 1: cbq bandwidth 10Mbit allot 1514 \
2124 cell 8 avpkt 1000 mpu 64
2126 # tc class add dev eth1 parent 1:0 classid 1:1 cbq bandwidth 10Mbit \
2127 rate 10Mbit allot 1514 cell 8 weight 1Mbit prio 8 maxburst 20 \
2130 Standard CBQ preamble. I never get used to the sheer amount of numbers
2133 Defmap refers to TC_PRIO bits, which are defined as follows:
2136 TC_PRIO.. Num Corresponds to TOS
2137 -------------------------------------------------
2138 BESTEFFORT 0 Maximize Reliablity
2139 FILLER 1 Minimize Cost
2140 BULK 2 Maximize Throughput (0x8)
2142 INTERACTIVE 6 Minimize Delay (0x10)
2146 The TC_PRIO.. number corresponds to bits, counted from the right. See the
2147 pfifo_fast section for more details how TOS bits are converted to
2150 Now the interactive and the bulk classes:
2153 # tc class add dev eth1 parent 1:1 classid 1:2 cbq bandwidth 10Mbit \
2154 rate 1Mbit allot 1514 cell 8 weight 100Kbit prio 3 maxburst 20 \
2155 avpkt 1000 split 1:0 defmap c0
2157 # tc class add dev eth1 parent 1:1 classid 1:3 cbq bandwidth 10Mbit \
2158 rate 8Mbit allot 1514 cell 8 weight 800Kbit prio 7 maxburst 20 \
2159 avpkt 1000 split 1:0 defmap 3f
2162 The 'split qdisc' is 1:0, which is where the choice will be made. C0 is
2163 binary for 11000000, 3F for 00111111, so these two together will match
2164 everything. The first class matches bits 7 & 6, and thus corresponds
2165 to 'interactive' and 'control' traffic. The second class matches the rest.
2167 Node 1:0 now has a table like this:
2180 For additional fun, you can also pass a 'change mask', which indicates
2181 exactly which priorities you wish to change. You only need to use this if you
2182 are running 'tc class change'. For example, to add best effort traffic to
2183 1:2, we could run this:
2186 # tc class change dev eth1 classid 1:2 cbq defmap 01/01
2189 The priority map over at 1:0 now looks like this:
2203 FIXME: did not test 'tc class change', only looked at the source.
2204 <sect2>Hierarchical Token Bucket
2206 Martin Devera (<devik>) rightly realised that CBQ is complex and does
2207 not seem optimized for many typical situations. His Hierarchial approach is
2208 well suited for setups where you have a fixed amount of bandwidth which you
2209 want to divide for different purposes, giving each purpose a guaranteed
2210 bandwidth, with the possibility of specifying how much bandwidth can be
2213 HTB works just like CBQ but does not resort to idle time calculations to
2214 shape. Instead, it is a classful Token Bucket Filter - hence the name. It
2215 has only a few parameters, which are well documented on his
2216 <url url="http://luxik.cdi.cz/~devik/qos/htb/
"
2219 As your HTB configuration gets more complex, your configuration scales
2220 well. With CBQ it is already complex even in simple cases! HTB is not yet a
2221 part of the standard kernel, but it should soon be!
2223 If you are in a position to patch your kernel, by all means consider HTB.
2224 <sect3>Sample configuration
2226 Functionally almost identical to the CBQ sample configuration above:
2229 # tc qdisc add dev eth0 root handle 1: htb default 30
2231 # tc class add dev eth0 parent 1: classid 1:1 htb rate 6mbit burst 15k
2233 # tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit burst 15k
2234 # tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 6mbit burst 15k
2235 # tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 6mbit burst 15k
2238 The author then recommends SFQ for beneath these classes:
2240 # tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
2241 # tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
2242 # tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10
2245 Add the filters which direct traffic to the right classes:
2247 # U32="tc filter add dev eth0 protocol ip parent
1:
0 prio
1 u32
"
2248 # $U32 match ip dport 80 0xffff flowid 1:10
2249 # $U32 match ip sport 25 0xffff flowid 1:20
2251 And that's it - no unsightly unexplained numbers, no undocumented
2254 HTB certainly looks wonderful - if 10: and 20: both have their guaranteed
2255 bandwidth, and more is left to divide, they borrow in a 5:3 ratio, just as
2258 Unclassified traffic gets routed to 30:, which has little bandwidth of its
2259 own but can borrow everything that is left over. Because we chose SFQ
2260 internally, we get fairness thrown in for free!
2262 <sect1>Classifying packets with filters
2264 To determine which class shall process a packet, the so-called 'classifier
2265 chain' is called each time a choice needs to be made. This chain consists of
2266 all filters attached to the classful qdisc that needs to decide.
2268 To reiterate the tree, which is not a tree:
2281 When enqueueing a packet, at each branch the filter chain is consulted for a
2282 relevant instruction. A typical setup might be to have a filter in 1:1 that
2283 directs a packet to 12: and a filter on 12: that sends the packet to 12:2.
2285 You might also attach this latter rule to 1:1, but you can make efficiency
2286 gains by having more specific tests lower in the chain.
2288 You can't filter a packet 'upwards', by the way. Also, with HTB, you should
2289 attach all filters to the root!
2291 And again - packets are only enqueued downwards! When they are dequeued,
2292 they go up again, where the interface lives. They do NOT fall off the end of
2293 the tree to the network adaptor!
2295 <sect2>Some simple filtering examples
2297 As explained in the Classifier chapter, you can match on literally anything,
2298 using a very complicated syntax. To start, we will show how to do the
2299 obvious things, which luckily are quite easy.
2301 Let's say we have a PRIO qdisc called '10:' which contains three classes, and
2302 we want to assign all traffic from and to port 22 to the highest priority
2303 band, the filters would be:
2306 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
2307 ip dport 22 0xffff flowid 10:1
2308 # tc filter add dev eth0 protocol ip parent 10: prio 1 u32 match \
2309 ip sport 80 0xffff flowid 10:1
2310 # tc filter add dev eth0 protocol ip parent 10: prio 2 flowid 10:2
2313 What does this say? It says: attach to eth0, node 10: a priority 1 u32
2314 filter that matches on IP destination port 22 *exactly* and send it to band
2315 10:1. And it then repeats the same for source port 80. The last command says
2316 that anything unmatched so far should go to band 10:2, the next-highest
2319 You need to add 'eth0', or whatever your interface is called, because each
2320 interface has a unique namespace of handles.
2322 To select on an IP address, use this:
2324 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
2325 match ip dst 4.3.2.1/32 flowid 10:1
2326 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 \
2327 match ip src 1.2.3.4/32 flowid 10:1
2328 # tc filter add dev eth0 protocol ip parent 10: prio 2 \
2332 This assigns traffic to 4.3.2.1 and traffic from 1.2.3.4 to the highest
2333 priority queue, and the rest to the next-highest one.
2335 You can concatenate matches, to match on traffic from 1.2.3.4 and from port
2338 # tc filter add dev eth0 parent 10:0 protocol ip prio 1 u32 match ip src 4.3.2.1/32
2339 match ip sport 80 0xffff flowid 10:1
2342 <sect2>All the filtering commands you will normally need
2344 Most shaping commands presented here start with this preamble:
2346 # tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 ..
2348 These are the so called 'u32' matches, which can match on ANY part of a
2351 <tag>On source/destination address</tag>
2352 Source mask 'match ip src 1.2.3.0/24', destination mask 'match ip dst
2353 4.3.2.0/24'. To match a single host, use /32, or omit the mask.
2354 <tag>On source/destination port, all IP protocols</tag>
2355 Source: 'match ip sport 80 0xffff', 'match ip dport 0xffff'
2356 <tag>On ip protocol (tcp, udp, icmp, gre, ipsec)</tag>
2357 Use the numbers from /etc/protocols, for example, icmp is 1: 'match ip
2359 <tag>On fwmark</tag>
2360 You can mark packets with either ipchains and have that mark survive routing
2361 across interfaces. This is really useful to for example only shape traffic on
2362 eth1 that came in on eth0. Syntax:
2363 # tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 6 fw flowid 1:1
2364 Note that this is not a u32 match!
2366 You can place a mark like this:
2368 # iptables -A PREROUTING -t mangle -i eth0 -j MARK --set-mark 6
2370 The number 6 is arbitrary.
2372 If you don't want to understand the full tc filter syntax, just use
2373 iptables, and only learn to select on fwmark.
2374 <tag>On the TOS field</tag>
2375 To select interactive, minimum delay traffic:
2377 # tc filter add dev ppp0 parent 1:0 protocol ip prio 10 u32 \
2378 match ip tos 0x10 0xff \
2381 Use 0x08 0xff for bulk traffic.
2384 For more filtering commands, see the Advanced Filters chapter.
2385 <sect>Loadsharing over multiple interfaces
2387 There are several ways of doing this. One of the easiest and straightforward
2388 ways is 'TEQL' - "True
" (or "trivial
") link equalizer. Like most things
2389 having to do with queueing, loadsharing goes both ways. Both ends of a link
2390 may need to participate for full effect.
2392 Imagine this situation:
2395 +-------+ eth1 +-------+
2397 'network 1' ----| A | | B |---- 'network 2'
2399 +-------+ eth2 +-------+
2402 A and B are routers, and for the moment we'll assume both run Linux. If
2403 traffic is going from network 1 to network 2, router A needs to distribute
2404 the packets over both links to B. Router B needs to be configured to accept
2405 this. Same goes the other way around, when packets go from network 2 to
2406 network 1, router B needs to send the packets over both eth1 and eth2.
2408 The distributing part is done by a 'TEQL' device, like this (it couldn't be
2412 # tc qdisc add dev eth1 root teql0
2413 # tc qdisc add dev eth2 root teql0
2414 # ip link set dev teql0 up
2417 Don't forget the 'ip link set up' command!
2419 This needs to be done on both hosts. The device teql0 is basically a
2420 roundrobbin distributor over eth1 and eth2, for sending packets. No data
2421 ever comes in over an teql device, that just appears on the 'raw' eth1 and
2424 But now we just have devices, we also need proper routing. One way to do
2425 this is to assign a /31 network to both links, and a /31 to the teql0 device
2428 FIXME: does this need something like 'nobroadcast'? A /31 is too small to
2429 house a network address and a broadcast address - if this doesn't work as
2430 planned, try a /30, and adjust the ip adresses accordingly. You might even
2431 try to make eth1 and eth2 do without an IP address!
2435 # ip addr add dev eth1 10.0.0.0/31
2436 # ip addr add dev eth2 10.0.0.2/31
2437 # ip addr add dev teql0 10.0.0.4/31
2442 # ip addr add dev eth1 10.0.0.1/31
2443 # ip addr add dev eth2 10.0.0.3/31
2444 # ip addr add dev teql0 10.0.0.5/31
2447 Router A should now be able to ping 10.0.0.1, 10.0.0.3 and 10.0.0.5 over the
2448 2 real links and the 1 equalized device. Router B should be able to ping
2449 10.0.0.0, 10.0.0.2 and 10.0.0.4 over the links.
2451 If this works, Router A should make 10.0.0.5 its route for reaching network
2452 2, and Router B should make 10.0.0.4 its route for reaching network 1. For
2453 the special case where network 1 is your network at home, and network 2 is
2454 the Internet, Router A should make 10.0.0.5 its default gateway.
2458 Nothing is as easy as it seems. eth1 and eth2 on both router A and B need to
2459 have return path filtering turned off, because they will otherwise drop
2460 packets destined for ip addresses other than their own:
2463 # echo 0 > /proc/net/ipv4/conf/eth1/rp_filter
2464 # echo 0 > /proc/net/ipv4/conf/eth2/rp_filter
2467 Then there is the nasty problem of packet reordering. Let's say 6 packets
2468 need to be sent from A to B - eth1 might get 1, 3 and 5. eth2 would then do
2469 2, 4 and 6. In an ideal world, router B would receive this in order, 1, 2,
2470 3, 4, 5, 6. But the possibility is very real that the kernel gets it like
2471 this: 2, 1, 4, 3, 6, 5. The problem is that this confuses TCP/IP. While not
2472 a problem for links carrying many different TCP/IP sessions, you won't be
2473 able to to a bundle multiple links and get to ftp a single file lots faster,
2474 except when your receiving or sending OS is Linux, which is not easily
2475 shaken by some simple reordering.
2477 However, for lots of applications, link loadbalancing is a great idea.
2480 <sect>Netfilter & iproute - marking packets
2482 So far we've seen how iproute works, and netfilter was mentioned a few
2483 times. This would be a good time to browse through <url name="Rusty's Remarkably
2485 url="http://netfilter.samba.org/unreliable-guides/
">. Netfilter itself
2486 can be found <url name="here
"
2487 url="http://netfilter.filewatcher.org/
">.
2489 Netfilter allows us to filter packets, or mangle their headers. One special
2490 feature is that we can mark a packet with a number. This is done with the
2491 --set-mark facility.
2493 As an example, this command marks all packets destined for port 25, outgoing
2497 # iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 25 \
2498 -j MARK --set-mark 1
2501 Let's say that we have multiple connections, one that is fast (and
2502 expensive, per megabyte) and one that is slower, but flat fee. We would most
2503 certainly like outgoing mail to go via the cheap route.
2505 We've already marked the packets with a '1', we now instruct the routing
2506 policy database to act on this:
2509 # echo 201 mail.out >> /etc/iproute2/rt_tables
2510 # ip rule add fwmark 1 table mail.out
2512 0: from all lookup local
2513 32764: from all fwmark 1 lookup mail.out
2514 32766: from all lookup main
2515 32767: from all lookup default
2518 Now we generate the mail.out table with a route to the slow but cheap link:
2520 # /sbin/ip route add default via 195.96.98.253 dev ppp0 table mail.out
2523 And we are done. Should we want to make exceptions, there are lots of ways
2524 to achieve this. We can modify the netfilter statement to exclude certain
2525 hosts, or we can insert a rule with a lower priority that points to the main
2526 table for our excepted hosts.
2528 We can also use this feature to honour TOS bits by marking packets with a
2529 different type of service with different numbers, and creating rules to act
2530 on that. This way you can even dedicate, say, an ISDN line to interactive
2533 Needless to say, this also works fine on a host that's doing NAT
2536 IMPORTANT: We received a report that MASQ and SNAT at least collide
2537 with marking packets. Rusty Russell explains it in
2539 url="http://lists.samba.org/pipermail/netfilter/
2000-November/
006089.html
"
2540 name="this posting
">. Turn off the reverse path filter to make it work
2543 Note: to mark packets, you need to have some options enabled in your
2547 IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
2548 IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
2549 IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK) [Y/n/?]
2552 See also <ref id="SQUID
" name="Transparent web-caching using netfilter, iproute2, ipchains and squid
">
2554 <sect>Advanced filters for (re-)classifying packets
2556 As explained in the section on classful queueing disciplines, filters are
2557 needed to classify packets into any of the sub-queues. These filters are
2558 called from within the classful qdisc.
2560 Here is an incomplete list of classifiers available:
2563 Bases the decision on how the firewall has marked the packet. This can be
2564 the easy way out if you don't want to learn tc filter syntax. See the
2565 Queueing chapter for details.
2568 Bases the decision on fields within the packet (i.e. source IP address, etc)
2571 Bases the decision on which route the packet will be routed by
2573 <tag>rsvp, rsvp6</tag>
2574 Routes packets based on <url
2575 url="http://www.isi.edu/div7/rsvp/overview.html
" name="RSVP
">. Only useful
2576 on networks you control - the Internet does not respect RSVP.
2579 Used in the DSMARK qdisc, see the relevant section.
2582 Note that in general there are many ways in which you can classify packet
2583 and that it generally comes down to preference as to which system you wish
2586 Classifiers in general accept a few arguments in common. They are listed
2587 here for convenience:
2591 The protocol this classifier will accept. Generally you will only be
2592 accepting only IP traffic. Required.
2595 The handle this classifier is to be attached to. This handle must be
2596 an already existing class. Required.
2599 The priority of this classifier. Lower numbers get tested first.
2602 This handle means different things to different filters.
2606 All the following sections will assume you are trying to shape the traffic
2607 going to <tt>HostA</tt>. They will assume that the root class has been
2608 configured on 1: and that the class you want to send the selected traffic to
2612 <sect1>The "u32
" classifier
2614 The U32 filter is the most advanced filter available in the current
2615 implementation. It entirely based on hashing tables, which make it
2616 robust when there are many filter rules.
2618 In its simplest form the U32 filter is a list of records, each
2619 consisting of two fields: a selector and an action. The selectors,
2620 described below, are compared with the currently processed IP packet
2621 until the first match occurs, and then the associated action is performed.
2622 The simplest type of action would be directing the packet into defined
2625 The commandline of <tt>tc filter</tt> program, used to configure the filter,
2626 consists of three parts: filter specification, a selector and an action.
2627 The filter specification can be defined as:
2630 tc filter add dev IF [ protocol PROTO ]
2631 [ (preference|priority) PRIO ]
2635 The <tt>protocol</tt> field describes protocol that the filter will be
2636 applied to. We will only discuss case of <tt>ip</tt> protocol. The
2637 <tt>preference</tt> field (<tt>priority</tt> can be used alternatively)
2638 sets the priority of currently defined filter. This is important, since
2639 you can have several filters (lists of rules) with different priorities.
2640 Each list will be passed in the order the rules were added, then list with
2641 lower priority (higher preference number) will be processed. The <tt>parent</tt>
2642 field defines the CBQ tree top (e.g. 1:0), the filter should be attached
2645 The options decribed above apply to all filters, not only U32.
2649 The U32 selector contains definition of the pattern, that will be matched
2650 to the currently processed packet. Precisely, it defines which bits are
2651 to be matched in the packet header and nothing more, but this simple
2652 method is very powerful. Let's take a look at the following examples,
2653 taken directly from a pretty complex, real-world filter:
2656 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
2657 match u32 00100000 00ff0000 at 0 flowid 1:10
2661 For now, leave the first line alone - all these parameters describe
2662 the filter's hash tables. Focus on the selector line, containing
2663 <tt>match</tt> keyword. This selector will match to IP headers, whose
2664 second byte will be 0x10 (0010). As you can guess, the 00ff number is
2665 the match mask, telling the filter exactly which bits to match. Here
2666 it's 0xff, so the byte will match if it's exactly 0x10. The <tt>at</tt>
2667 keyword means that the match is to be started at specified offset (in
2668 bytes) -- in this case it's beginning of the packet. Translating all
2669 that to human language, the packet will match if its Type of Service
2670 field will have `low delay' bits set. Let's analyze another rule:
2673 # tc filter add dev eth0 protocol ip parent 1:0 pref 10 u32 \
2674 match u32 00000016 0000ffff at nexthdr+0 flowid 1:10
2678 The <tt>nexthdr</tt> option means next header encapsulated in the IP packet,
2679 i.e. header of upper-layer protocol. The match will also start here
2680 at the beginning of the next header. The match should occur in the
2681 second, 32-bit word of the header. In TCP and UDP protocols this field
2682 contains packet's destination port. The number is given in big-endian
2683 format, i.e. older bits first, so we simply read 0x0016 as 22 decimal,
2684 which stands for SSH service if this was TCP. As you guess, this match
2685 is ambigous without a context, and we will discuss this later.
2688 Having understood all the above, we will find the following selector
2689 quite easy to read: <tt>match c0a80100 ffffff00 at 16</tt>. What we
2690 got here is a three byte match at 17-th byte, counting from the IP
2691 header start. This will match for packets with destination address
2692 anywhere in 192.168.1/24 network. After analyzing the examples, we
2693 can summarize what we have learnt.
2695 <sect2>General selectors
2698 General selectors define the pattern, mask and offset the pattern
2699 will be matched to the packet contents. Using the general selectors
2700 you can match virtually any single bit in the IP (or upper layer)
2701 header. They are more difficult to write and read, though, than
2702 specific selectors that described below. The general selector syntax
2706 match [ u32 | u16 | u8 ] PATTERN MASK [ at OFFSET | nexthdr+OFFSET]
2710 One of the keywords <tt>u32</tt>, <tt>u16</tt> or <tt>u8</tt> specifies
2711 length of the pattern in bits. PATTERN and MASK should follow, of length
2712 defined by the previous keyword. The OFFSET parameter is the offset,
2713 in bytes, to start matching. If <tt>nexthdr+</tt> keyword is given,
2714 the offset is relative to start of the upper layer header.
2720 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
2721 match u8 64 0xff at 8 \
2726 Packet will match to this rule, if its time to live (TTL) is 64.
2727 TTL is the field starting just after 8-th byte of the IP header.
2730 # tc filter add dev ppp14 parent 1:0 prio 10 u32 \
2731 match u8 0x10 0xff at nexthdr+13 \
2736 FIXME: it has been pointed out that this syntax does not work currently.
2738 Use this to match ACKs on packets smaller than 64 bytes:
2741 ## match acks the hard way,
2743 ## IP header length 0x5(32 bit words),
2744 ## IP Total length 0x34 (ACK + 12 bytes of TCP options)
2745 ## TCP ack set (bit 5, offset 33)
2746 # tc filter add dev ppp14 parent 1:0 protocol ip prio 10 u32 \
2747 match ip protocol 6 0xff \
2748 match u8 0x05 0x0f at 0 \
2749 match u16 0x0000 0xffc0 at 2 \
2750 match u8 0x10 0xff at 33 \
2756 This rule will only match TCP packets with ACK bit set, and no further
2757 payload. Here we can see an example of using two selectors, the final result
2758 will be logical AND of their results. If we take a look at TCP header
2759 diagram, we can see that the ACK bit is second older bit (0x10) in the 14-th
2760 byte of the TCP header (<tt>at nexthdr+13</tt>). As for the second
2761 selector, if we'd like to make our life harder, we could write <tt>match u8
2762 0x06 0xff at 9</tt> instead of using the specific selector <tt>protocol
2763 tcp</tt>, because 6 is the number of TCP protocol, present in 10-th byte of
2764 the IP header. On the other hand, in this example we couldn't use any
2765 specific selector for the first match - simply because there's no specific
2766 selector to match TCP ACK bits.
2768 <sect2>Specific selectors
2770 The following table contains a list of all specific selectors
2771 the author of this section has found in the <tt>tc</tt> program
2772 source code. They simply make your life easier and increase readability
2773 of your filter's configuration.
2775 FIXME: table placeholder - the table is in separate file ,,selector.html''
2777 FIXME: it's also still in Polish :-(
2779 FIXME: must be sgml'ized
2785 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
2786 match ip tos 0x10 0xff \
2790 FIXME: tcp dst match does not work as described below:
2792 The above rule will match packets which have the TOS field set to 0x10.
2793 The TOS field starts at second byte of the packet and is one byte big,
2794 so we could write an equivalent general selector: <tt>match u8 0x10 0xff
2795 at 1</tt>. This gives us hint to the internals of U32 filter -- the
2796 specific rules are always translated to general ones, and in this
2797 form they are stored in the kernel memory. This leads to another conclusion
2798 -- the <tt>tcp</tt> and <tt>udp</tt> selectors are exactly the same
2799 and this is why you can't use single <tt>match tcp dst 53 0xffff</tt>
2800 selector to match TCP packets sent to given port -- they will also
2801 match UDP packets sent to this port. You must remember to also specify
2802 the protocol and end up with the following rule:
2805 # tc filter add dev ppp0 parent 1:0 prio 10 u32 \
2806 match tcp dst 53 0xffff \
2807 match ip protocol 0x6 0xff \
2814 describe more options
2829 <sect1>The "route
" classifier
2832 This classifier filters based on the results of the routing tables. When a
2833 packet that is traversing through the classes reaches one that is marked
2834 with the "route
" filter, it splits the packets up based on information in
2838 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 route
2841 Here we add a route classifier onto the parent node 1:0 with priority 100.
2842 When a packet reaches this node (which, since it is the root, will happen
2843 immediately) it will consult the routing table and if one matches will
2844 send it to the given class and give it a priority of 100. Then, to finally
2845 kick it into action, you add the appropriate routing entry:
2847 The trick here is to define 'realm' based on either destination or source.
2848 The way to do it is like this:
2851 # ip route add Host/Network via Gateway dev Device realm RealmNumber
2854 For instance, we can define our destination network 192.168.10.0 with a realm
2858 # ip route add 192.168.10.0/24 via 192.168.10.1 dev eth1 realm 10
2861 When adding route filters, we can use realm numbers to represent the
2862 networks or hosts and specify how the routes match the filters.
2865 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
2866 route to 10 classid 1:10
2869 The above rule says packets going to the network 192.168.10.0 match class id
2872 Route filter can also be used to match source routes. For example, there is
2873 a subnetwork attached to the Linux router on eth2.
2876 # ip route add 192.168.2.0/24 dev eth2 realm 2
2877 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 \
2878 route from 2 classid 1:2
2881 Here the filter specifies that packets from the subnetwork 192.168.2.0
2882 (realm 2) will match class id 1:2.
2884 <sect1>Policing filters
2886 To make even more complicated setups possible, you can have filters that
2887 only match up to a certain bandwidth. You can declare a filter to entirely
2888 cease matching above a certain rate, or only to not match only the bandwidth
2889 exceeding a certain rate.
2891 So if you decided to police at 4mbit/s, but 5mbit/s of traffic is present,
2892 you can stop matching either the entire 5mbit/s, or only not match 1mbit/s,
2893 and do send 4mbit/s to the configured class.
2895 If bandwidth exceeds the configured rate, you can drop a packet, reclassify
2896 it, or see if another filter will match it.
2898 <sect2>Ways to police
2900 There are basically two ways to police. If you compiled the kernel
2901 with 'Estimators', the kernel can measure for each filter how much traffic
2902 it is passing, more or less. These estimators are very easy on the CPU, as
2903 they simply count 25 times per second how many data has been passed, and
2904 calculate the bitrate from that.
2906 The other way works again via a Token Bucket Filter, this time living within
2907 your filter. The TBF only matches traffic UP TO your configured bandwidth,
2908 if more is offered, only the excess is subject to the configured overlimit
2911 <sect3>With the kernel estimator
2913 This is very simple and has only one parameter: avrate. Either the flow
2914 remains below avrate, and the filter classifies the traffic to the classid
2915 configured, or your rate exceeds it in which case the specified action is
2916 taken, which is 'reclassify' by default.
2918 The kernel uses an Exponential Weighted Moving Average for your bandwidth
2919 which makes it less sensitive to short bursts.
2921 <sect3>With Token Bucket Filter
2923 Uses the following parameters:
2925 <item>buffer/maxburst
2931 Which behave mostly identical to those described in the Token Bucket Filter
2932 section. Please note however that if you set the mtu of a TBF policer too
2933 low, *no* packets will pass, whereas the egress TBF qdisc will just pass
2936 Another difference is that a policer can only let a packet pass, or drop it.
2937 It cannot delay hold on to it in order to delay it.
2938 <sect2>Overlimit actions
2940 If your filter decides that it is overlimit, it can take 'actions'.
2941 Currently, three actions are available:
2944 Causes this filter not to match, but perhaps other filters will.
2946 This is a very fierce option which simply discards traffic exceeding a
2947 certain rate. It is often used in the ingress policer and has limited uses.
2948 For example, you may have a nameserver that falls over if offered more than
2949 5mbit/s of packets, in which case an ingress filter could be used to make
2950 sure no more is ever offered.
2952 Pass on traffic ok. Might be used to disable a complicated filter, but leave
2954 <tag>reclassify</tag>
2955 Most often comes down to reclassification to Best Effort. This is the
2961 The only real example known is mentioned in the 'Protecting your host
2962 from SYN floods' section.
2964 FIXME: if you have used this, please share your experience with us
2966 <sect1>Hashing filters for very fast massive filtering
2968 If you have a need for thousands of rules, for example if you have a lot of
2969 clients or computers, all with different QoS specifications, you may find
2970 that the kernel spends a lot of time matching all those rules.
2972 By default, all filters reside in one big chain which is matched in
2973 descending order of priority. If you have 1000 rules, 1000 checks may be
2974 needed to determine what to do with a packet.
2976 Matching would go much quicker if you would have 256 chains with each four
2977 rules - if you could divide packets over those 256 chains, so that the right
2980 Hashing makes this possible. Let's say you have 1024 cablemodem customers in
2981 your network, with IP addresses ranging from 1.2.0.0 to 1.2.3.255, and each
2982 has to go in another bin, for example 'lite', 'regular' and 'premium'. You
2983 would then have 1024 rules like this:
2986 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
2988 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
2991 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
2992 1.2.3.254 classid 1:3
2993 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
2994 1.2.3.255 classid 1:2
2997 To speed this up, we can use the last part of the IP address as a 'hash
2998 key'. We then get 256 tables, the first of which looks like this:
3000 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
3002 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
3004 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
3006 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
3010 The next one starts like this:
3012 # tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
3017 This way, only four checks are needed at most, two on average.
3019 Configuration is pretty complicated, but very worth it by the time you have
3020 this many rules. First we make a filter root, then we create a table with
3023 # tc filter add dev eth1 parent 1:0 prio 5 protocol ip u32
3024 # tc filter add dev eth1 parent 1:0 prio 5 handle 2: protocol ip u32 divisor 256
3027 Now we add some rules to entries in the created table:
3030 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
3031 match ip src 1.2.0.123 flowid 1:1
3032 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
3033 match ip src 1.2.1.123 flowid 1:2
3034 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
3035 match ip src 1.2.3.123 flowid 1:3
3036 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
3037 match ip src 1.2.4.123 flowid 1:2
3039 This is entry 123, which contains matches for 1.2.0.123, 1.2.1.123,
3040 1.2.2.123, 1.2.3.123, and sends them to 1:1, 1:2, 1:3 and 1:2 respectively.
3041 Note that we need to specify our hash bucket in hex, 0x7b is 123.
3043 Next create a 'hashing filter' that directs traffic to the right entry in
3046 # tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 800:: \
3047 match ip src 1.2.0.0/16 \
3048 hashkey mask 0x000000ff at 12 \
3051 Ok, some numbers need explaining. The default hash table is called 800:: and
3052 all filtering starts there. Then we select the source address, which lives
3053 as position 12, 13, 14 and 15 in the IP header, and indicate that we are
3054 only interested in the last part. This we send to hash table 2:, which we
3057 It is quite complicated, but it does work in practice and performance will
3058 be staggering. Note that this example could be improved to the ideal case
3059 where each chain contains 1 filter!
3060 <sect>Kernel network parameters
3062 The kernel has lots of parameters which
3063 can be tuned for different circumstances. While, as usual, the default
3064 parameters serve 99% of installations very well, we don't call this the
3065 Advanced HOWTO for the fun of it!
3067 The interesting bits are in /proc/sys/net, take a look there. Not everything
3068 will be documented here initially, but we're working on it.
3070 In the meantime you may want to have a look at the Linux-Kernel sources;
3071 read the file <file>Documentation/filesystems/proc.txt</file>. Most of the
3072 features are explained there.
3076 <sect1>Reverse Path Filtering
3078 By default, routers route everything, even packets which 'obviously' don't
3079 belong on your network. A common example is private IP space escaping onto
3080 the Internet. If you have an interface with a route of 195.96.96.0/24 to it,
3081 you do not expect packets from 212.64.94.1 to arrive there.
3083 Lots of people will want to turn this feature off, so the kernel hackers
3084 have made it easy. There are files in <file>/proc</file> where you can tell
3085 the kernel to do this for you. The method is called "Reverse Path
3086 Filtering
". Basically, if the reply to this packet wouldn't go out the
3087 interface this packet came in, then this is a bogus packet and should be
3090 The following fragment will turn this on for all current and future
3094 # for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
3099 Going by the example above, if a packet arrived on the Linux router on eth1
3100 claiming to come from the Office+ISP subnet, it would be dropped. Similarly,
3101 if a packet came from the Office subnet, claiming to be from somewhere
3102 outside your firewall, it would be dropped also.
3104 The above is full reverse path filtering. The default is to only filter
3105 based on IPs that are on directly connected networks. This is because the
3106 full filtering breaks in the case of asymmetric routing (where packets come
3107 in one way and go out another, like satellite traffic, or if you have
3108 dynamic (bgp, ospf, rip) routes in your network. The data comes down
3109 through the satellite dish and replies go back through normal land-lines).
3111 If this exception applies to you (and you'll probably know if it does) you
3112 can simply turn off the <file>rp_filter</file> on the interface where the
3113 satellite data comes in. If you want to see if any packets are being
3114 dropped, the <file>log_martians</file> file in the same directory will tell
3115 the kernel to log them to your syslog.
3118 # echo 1 >/proc/sys/net/ipv4/conf/<interfacename>/log_martians
3121 FIXME: is setting the conf/{default,all}/* files enough? - martijn
3123 <sect1>Obscure settings
3125 Ok, there are a lot of parameters which can be modified. We try to list them
3126 all. Also documented (partly) in <file>Documentation/ip-sysctl.txt</file>.
3128 Some of these settings have different defaults based on whether you
3129 answered 'Yes' to 'Configure as router and not host' while compiling your
3134 As a generic note, most rate limiting features don't work on loopback, so
3135 don't test them locally. The limits are supplied in 'jiffies', and are
3136 enforced using the earlier mentioned token bucket filter.
3138 The kernel has an internal clock which runs at 'HZ' ticks (or 'jiffies') per
3139 second. On intel, 'HZ' is mostly 100. So setting a *_rate file to, say 50,
3140 would allow for 2 packets per second. The token bucket filter is also
3141 configured to allow for a burst of at most 6 packets, if enough tokens have
3144 Several entries in the following list have been copied from
3145 /usr/src/linux/Documentation/networking/ip-sysctl.txt, written by Alexey
3146 Kuznetsov <kuznet@ms2.inr.ac.ru> and Andi Kleen <ak@muc.de>
3148 <tag>/proc/sys/net/ipv4/icmp_destunreach_rate</tag>
3149 If the kernel decides that it can't deliver a packet, it will drop it, and
3150 send the source of the packet an ICMP notice to this effect.
3151 <tag>/proc/sys/net/ipv4/icmp_echo_ignore_all</tag>
3152 Don't act on echo packets at all. Please don't set this by default, but if
3153 you are used as a relay in a DoS attack, it may be useful.
3154 <tag>/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts [Useful]</tag>
3155 If you ping the broadcast address of a network, all hosts are supposed to
3156 respond. This makes for a dandy denial-of-service tool. Set this to 1 to
3157 ignore these broadcast messages.
3158 <tag>/proc/sys/net/ipv4/icmp_echoreply_rate</tag>
3159 The rate at which echo replies are sent to any one destination.
3160 <tag>/proc/sys/net/ipv4/icmp_ignore_bogus_error_responses</tag>
3161 Set this to ignore ICMP errors caused by hosts in the network reacting badly
3162 to frames sent to what they perceive to be the broadcast address.
3163 <tag>/proc/sys/net/ipv4/icmp_paramprob_rate</tag>
3164 A relatively unknown ICMP message, which is sent in response to incorrect
3165 packets with broken IP or TCP headers. With this file you can control the
3166 rate at which it is sent.
3167 <tag>/proc/sys/net/ipv4/icmp_timeexceed_rate</tag>
3168 This the famous cause of the 'Solaris middle star' in traceroutes. Limits
3169 number of ICMP Time Exceeded messages sent.
3170 <tag>/proc/sys/net/ipv4/igmp_max_memberships</tag>
3171 Maximum number of listening igmp (multicast) sockets on the host.
3172 FIXME: Is this true?
3173 <tag>/proc/sys/net/ipv4/inet_peer_gc_maxtime</tag>
3174 FIXME: Add a little explanation about the inet peer storage?&nl;
3175 Minimum interval between garbage collection passes. This interval is in
3176 effect under low (or absent) memory pressure on the pool. Measured in
3178 <tag>/proc/sys/net/ipv4/inet_peer_gc_mintime</tag>
3179 Minimum interval between garbage collection passes. This interval is in
3180 effect under high memory pressure on the pool. Measured in jiffies.
3181 <tag>/proc/sys/net/ipv4/inet_peer_maxttl</tag>
3182 Maximum time-to-live of entries. Unused entries will expire after this
3183 period of time if there is no memory pressure on the pool (i.e. when the
3184 number of entries in the pool is very small). Measured in jiffies.
3185 <tag>/proc/sys/net/ipv4/inet_peer_minttl</tag>
3186 Minimum time-to-live of entries. Should be enough to cover fragment
3187 time-to-live on the reassembling side. This minimum time-to-live
3188 is guaranteed if the pool size is less than inet_peer_threshold.
3189 Measured in jiffies.
3190 <tag>/proc/sys/net/ipv4/inet_peer_threshold</tag>
3191 The approximate size of the INET peer storage. Starting from this threshold
3192 entries will be thrown aggressively. This threshold also determines
3193 entries' time-to-live and time intervals between garbage collection passes.
3194 More entries, less time-to-live, less GC interval.
3195 <tag>/proc/sys/net/ipv4/ip_autoconfig</tag>
3196 This file contains the number one if the host received its IP configuration by
3197 RARP, BOOTP, DHCP or a similar mechanism. Otherwise it is zero.
3198 <tag>/proc/sys/net/ipv4/ip_default_ttl</tag>
3199 Time To Live of packets. Set to a safe 64. Raise it if you have a huge
3200 network. Don't do so for fun - routing loops cause much more damage that
3201 way. You might even consider lowering it in some circumstances.
3202 <tag>/proc/sys/net/ipv4/ip_dynaddr</tag>
3203 You need to set this if you use dial-on-demand with a dynamic interface
3204 address. Once your demand interface comes up, any local TCP sockets which haven't seen replies will be rebound to have the right address. This solves the problem that the
3205 connection that brings up your interface itself does not work, but the
3207 <tag>/proc/sys/net/ipv4/ip_forward</tag>
3208 If the kernel should attempt to forward packets. Off by default.
3209 <tag>/proc/sys/net/ipv4/ip_local_port_range</tag>
3210 Range of local ports for outgoing connections. Actually quite small by
3211 default, 1024 to 4999.
3212 <tag>/proc/sys/net/ipv4/ip_no_pmtu_disc</tag>
3213 Set this if you want to disable Path MTU discovery - a technique to
3214 determine the largest Maximum Transfer Unit possible on your path. See also
3215 the section on Path MTU discovery in the cookbook chapter.
3216 <tag>/proc/sys/net/ipv4/ipfrag_high_thresh</tag>
3217 Maximum memory used to reassemble IP fragments. When
3218 ipfrag_high_thresh bytes of memory is allocated for this purpose,
3219 the fragment handler will toss packets until ipfrag_low_thresh
3221 <tag>/proc/sys/net/ipv4/ip_nonlocal_bind</tag>
3222 Set this if you want your applications to be able to bind to an address
3223 which doesn't belong to a device on your system. This can be useful when
3224 your machine is on a non-permanent (or even dynamic) link, so your services
3225 are able to start up and bind to a specific address when your link is down.
3226 <tag>/proc/sys/net/ipv4/ipfrag_low_thresh</tag>
3227 Minimum memory used to reassemble IP fragments.
3228 <tag>/proc/sys/net/ipv4/ipfrag_time</tag>
3229 Time in seconds to keep an IP fragment in memory.
3230 <tag>/proc/sys/net/ipv4/tcp_abort_on_overflow</tag>
3231 A boolean flag controlling the behaviour under lots of incoming connections.
3232 When enabled, this causes the kernel to actively send RST packets when a
3233 service is overloaded.
3234 <tag>/proc/sys/net/ipv4/tcp_fin_timeout</tag>
3235 Time to hold socket in state FIN-WAIT-2, if it was closed by our side. Peer
3236 can be broken and never close its side, or even died unexpectedly. Default
3237 value is 60sec. Usual value used in 2.2 was 180 seconds, you may restore it,
3238 but remember that if your machine is even underloaded WEB server, you risk
3239 to overflow memory with kilotons of dead sockets, FIN-WAIT-2 sockets are
3240 less dangerous than FIN-WAIT-1, because they eat maximum 1.5K of memory, but
3241 they tend to live longer. Cf. tcp_max_orphans.
3242 <tag>/proc/sys/net/ipv4/tcp_keepalive_time</tag>
3243 How often TCP sends out keepalive messages when keepalive is enabled. &nl;
3245 <tag>/proc/sys/net/ipv4/tcp_keepalive_intvl</tag>
3246 How frequent probes are retransmitted, when a probe isn't acknowledged. &nl;
3247 Default: 75 seconds.
3248 <tag>/proc/sys/net/ipv4/tcp_keepalive_probes</tag>
3249 How many keepalive probes TCP will send, until it decides that the
3250 connection is broken. &nl;
3251 Default value: 9. &nl;
3252 Multiplied with tcp_keepalive_intvl, this gives the time a link can be
3253 nonresponsive after a keepalive has been sent.
3254 <tag>/proc/sys/net/ipv4/tcp_max_orphans</tag>
3255 Maximal number of TCP sockets not attached to any user file handle, held by
3256 system. If this number is exceeded orphaned connections are reset
3257 immediately and warning is printed. This limit exists only to prevent simple
3258 DoS attacks, you _must_ not rely on this or lower the limit artificially,
3259 but rather increase it (probably, after increasing installed memory), if
3260 network conditions require more than default value, and tune network
3261 services to linger and kill such states more aggressively. Let me remind you
3262 again: each orphan eats up to ~64K of unswappable memory.
3263 <tag>/proc/sys/net/ipv4/tcp_orphan_retries</tag>
3264 How may times to retry before killing TCP connection, closed by our side.
3265 Default value 7 corresponds to ~50sec-16min depending on RTO. If your machine
3266 is a loaded WEB server, you should think about lowering this value, such
3267 sockets may consume significant resources. Cf. tcp_max_orphans.
3268 <tag>/proc/sys/net/ipv4/tcp_max_syn_backlog</tag>
3269 Maximal number of remembered connection requests, which still did not
3270 receive an acknowledgement from connecting client. Default value is 1024 for
3271 systems with more than 128Mb of memory, and 128 for low memory machines. If
3272 server suffers of overload, try to increase this number. Warning! If you
3273 make it greater than 1024, it would be better to change TCP_SYNQ_HSIZE in
3274 include/net/tcp.h to keep TCP_SYNQ_HSIZE*16<=tcp_max_syn_backlog and to
3276 <tag>/proc/sys/net/ipv4/tcp_max_tw_buckets</tag>
3277 Maximal number of timewait sockets held by system simultaneously. If this
3278 number is exceeded time-wait socket is immediately destroyed and warning is
3279 printed. This limit exists only to prevent simple DoS attacks, you _must_
3280 not lower the limit artificially, but rather increase it (probably, after
3281 increasing installed memory), if network conditions require more than
3283 <tag>/proc/sys/net/ipv4/tcp_retrans_collapse</tag>
3284 Bug-to-bug compatibility with some broken printers.
3285 On retransmit try to send bigger packets to work around bugs in
3287 <tag>/proc/sys/net/ipv4/tcp_retries1</tag>
3288 How many times to retry before deciding that something is wrong
3289 and it is necessary to report this suspection to network layer.
3290 Minimal RFC value is 3, it is default, which corresponds
3291 to ~3sec-8min depending on RTO.
3292 <tag>/proc/sys/net/ipv4/tcp_retries2</tag>
3293 How may times to retry before killing alive TCP connection.
3294 <url url="http://www.ietf.org/rfc/rfc1122.txt
" name="RFC
1122">
3295 says that the limit should be longer than 100 sec.
3296 It is too small number. Default value 15 corresponds to ~13-30min
3298 <tag>/proc/sys/net/ipv4/tcp_rfc1337</tag>
3299 This boolean enables a fix for 'time-wait assassination hazards in tcp', described
3300 in RFC 1337. If enabled, this causes the kernel to drop RST packets for
3301 sockets in the time-wait state.&nl;
3303 <tag>/proc/sys/net/ipv4/tcp_sack</tag>
3304 Use Selective ACK which can be used to signify that specific packets are
3305 missing - therefore helping fast recovery.
3306 <tag>/proc/sys/net/ipv4/tcp_stdurg</tag>
3307 Use the Host requirements interpretation of the TCP urg pointer
3309 Most hosts use the older BSD interpretation, so if you turn this on
3310 Linux might not communicate correctly with them. &nl;
3312 <tag>/proc/sys/net/ipv4/tcp_syn_retries</tag>
3313 Number of SYN packets the kernel will send before giving up on the new
3315 <tag>/proc/sys/net/ipv4/tcp_synack_retries</tag>
3316 To open the other side of the connection, the kernel sends a SYN with a
3317 piggybacked ACK on it, to acknowledge the earlier received SYN. This is part
3318 2 of the threeway handshake. This setting determines the number of SYN+ACK
3319 packets sent before the kernel gives up on the connection.
3320 <tag>/proc/sys/net/ipv4/tcp_timestamps</tag>
3321 Timestamps are used, amongst other things, to protect against wrapping
3322 sequence numbers. A 1 gigabit link might conceivably re-encounter a previous
3323 sequence number with an out-of-line value, because it was of a previous
3324 generation. The timestamp will let it recognise this 'ancient packet'.
3325 <tag>/proc/sys/net/ipv4/tcp_tw_recycle</tag>
3326 Enable fast recycling TIME-WAIT sockets. Default value is 1.
3327 It should not be changed without advice/request of technical experts.
3328 <tag>/proc/sys/net/ipv4/tcp_window_scaling</tag>
3329 TCP/IP normally allows windows up to 65535 bytes big. For really fast
3330 networks, this may not be enough. The window scaling options allows for
3331 almost gigabyte windows, which is good for high bandwidth*delay products.
3334 <sect2>Per device settings
3336 DEV can either stand for a real interface, or for 'all' or 'default'.
3337 Default also changes settings for interfaces yet to be created.
3339 <tag>/proc/sys/net/ipv4/conf/DEV/accept_redirects</tag>
3340 If a router decides that you are using it for a wrong purpose (ie, it needs
3341 to resend your packet on the same interface), it will send us a ICMP
3342 Redirect. This is a slight security risk however, so you may want to turn it
3343 off, or use secure redirects.
3344 <tag>/proc/sys/net/ipv4/conf/DEV/accept_source_route</tag>
3345 Not used very much anymore. You used to be able to give a packet a list of
3346 IP addresses it should visit on its way. Linux can be made to honor this IP
3348 <tag>/proc/sys/net/ipv4/conf/DEV/bootp_relay</tag>
3349 Accept packets with source address 0.b.c.d with destinations not to this host
3350 as local ones. It is supposed that a BOOTP relay daemon will catch and forward
3353 The default is 0, since this feature is not implemented yet (kernel version
3355 <tag>/proc/sys/net/ipv4/conf/DEV/forwarding</tag>
3356 Enable or disable IP forwarding on this interface.
3357 <tag>/proc/sys/net/ipv4/conf/DEV/log_martians</tag>
3358 See the section on reverse path filters.
3359 <tag>/proc/sys/net/ipv4/conf/DEV/mc_forwarding</tag>
3360 If we do multicast forwarding on this interface
3361 <tag>/proc/sys/net/ipv4/conf/DEV/proxy_arp</tag>
3362 If you set this to 1, this interface will respond to ARP requests for
3363 addresses the kernel has routes to. Can be very useful when building 'ip
3364 pseudo bridges'. Do take care that your netmasks are very correct before
3365 enabling this! Also be aware that the rp_filter, mentioned elsewhere, also
3366 operates on ARP queries!
3367 <tag>/proc/sys/net/ipv4/conf/DEV/rp_filter</tag>
3368 See the section on reverse path filters.
3369 <tag>/proc/sys/net/ipv4/conf/DEV/secure_redirects</tag>
3370 Accept ICMP redirect messages only for gateways, listed in default gateway
3371 list. Enabled by default.
3372 <tag>/proc/sys/net/ipv4/conf/DEV/send_redirects</tag>
3373 If we send the above mentioned redirects.
3374 <tag>/proc/sys/net/ipv4/conf/DEV/shared_media</tag>
3375 If it is not set the kernel does not assume that different subnets on this
3376 device can communicate directly. Default setting is 'yes'.
3377 <tag>/proc/sys/net/ipv4/conf/DEV/tag</tag>
3382 <sect2> Neighbor policy
3384 Dev can either stand for a real interface, or for 'all' or 'default'.
3385 Default also changes settings for interfaces yet to be created.
3387 <tag>/proc/sys/net/ipv4/neigh/DEV/anycast_delay</tag>
3388 Maximum for random delay of answers to neighbor solicitation messages in
3389 jiffies (1/100 sec). Not yet implemented (Linux does not have anycast support
3391 <tag>/proc/sys/net/ipv4/neigh/DEV/app_solicit</tag>
3392 Determines the number of requests to send to the user level ARP daemon. Use 0
3394 <tag>/proc/sys/net/ipv4/neigh/DEV/base_reachable_time</tag>
3395 A base value used for computing the random reachable time value as specified
3397 <tag>/proc/sys/net/ipv4/neigh/DEV/delay_first_probe_time</tag>
3398 Delay for the first time probe if the neighbor is reachable. (see
3400 <tag>/proc/sys/net/ipv4/neigh/DEV/gc_stale_time</tag>
3401 Determines how often to check for stale ARP entries. After an ARP entry is
3402 stale it will be resolved again (which is useful when an IP address migrates
3403 to another machine). When ucast_solicit is greater than 0 it first tries to
3404 send an ARP packet directly to the known host When that fails and
3405 mcast_solicit is greater than 0, an ARP request is broadcasted.
3406 <tag>/proc/sys/net/ipv4/neigh/DEV/locktime</tag>
3407 An ARP/neighbor entry is only replaced with a new one if the old is at least
3408 locktime old. This prevents ARP cache thrashing.
3409 <tag>/proc/sys/net/ipv4/neigh/DEV/mcast_solicit</tag>
3410 Maximum number of retries for multicast solicitation.
3411 <tag>/proc/sys/net/ipv4/neigh/DEV/proxy_delay</tag>
3412 Maximum time (real time is random [0..proxytime]) before answering to an ARP
3413 request for which we have an proxy ARP entry. In some cases, this is used to
3414 prevent network flooding.
3415 <tag>/proc/sys/net/ipv4/neigh/DEV/proxy_qlen</tag>
3416 Maximum queue length of the delayed proxy arp timer. (see proxy_delay).
3417 <tag>/proc/sys/net/ipv4/neigh/DEV/retrans_time</tag>
3418 The time, expressed in jiffies (1/100 sec), between retransmitted Neighbor
3419 Solicitation messages. Used for address resolution and to determine if a
3420 neighbor is unreachable.
3421 <tag>/proc/sys/net/ipv4/neigh/DEV/ucast_solicit</tag>
3422 Maximum number of retries for unicast solicitation.
3423 <tag>/proc/sys/net/ipv4/neigh/DEV/unres_qlen</tag>
3424 Maximum queue length for a pending arp request - the number of packets which
3425 are accepted from other layers while the ARP address is still resolved.
3427 <tag>Internet QoS: Architectures and Mechanisms for Quality of Service,
3428 Zheng Wang, ISBN 1-55860-608-4</tag> Hardcover textbook covering topics
3429 related to Quality of Service. Good for understanding basic concepts.
3433 <sect2>Routing settings
3436 <tag>/proc/sys/net/ipv4/route/error_burst</tag>
3437 These parameters are used to limit the warning messages written to the kernel
3438 log from the routing code. The higher the error_cost factor is, the fewer
3439 messages will be written. Error_burst controls when messages will be dropped.
3440 The default settings limit warning messages to one every five seconds.
3441 <tag>/proc/sys/net/ipv4/route/error_cost</tag>
3442 These parameters are used to limit the warning messages written to the kernel
3443 log from the routing code. The higher the error_cost factor is, the fewer
3444 messages will be written. Error_burst controls when messages will be dropped.
3445 The default settings limit warning messages to one every five seconds.
3446 <tag>/proc/sys/net/ipv4/route/flush</tag>
3447 Writing to this file results in a flush of the routing cache.
3448 <tag>/proc/sys/net/ipv4/route/gc_elasticity</tag>
3449 Values to control the frequency and behavior of the garbage collection
3450 algorithm for the routing cache. This can be important for when doing
3451 failover. At least gc_timeout seconds will elapse before Linux will skip
3452 to another route because the previous one has died. By default set to 300,
3453 you may want to lower it if you want to have a speedy failover.
3456 url="http://mailman.ds9a.nl/pipermail/lartc/
2002q1/
002667.html
" name="this
3457 post
"> by Ard van Breemen.
3459 <tag>/proc/sys/net/ipv4/route/gc_interval</tag>
3460 See /proc/sys/net/ipv4/route/gc_elasticity.
3461 <tag>/proc/sys/net/ipv4/route/gc_min_interval</tag>
3462 See /proc/sys/net/ipv4/route/gc_elasticity.
3463 <tag>/proc/sys/net/ipv4/route/gc_thresh</tag>
3464 See /proc/sys/net/ipv4/route/gc_elasticity.
3465 <tag>/proc/sys/net/ipv4/route/gc_timeout</tag>
3466 See /proc/sys/net/ipv4/route/gc_elasticity.
3467 <tag>/proc/sys/net/ipv4/route/max_delay</tag>
3468 Delays for flushing the routing cache.
3469 <tag>/proc/sys/net/ipv4/route/max_size</tag>
3470 Maximum size of the routing cache. Old entries will be purged once the cache
3471 reached has this size.
3472 <tag>/proc/sys/net/ipv4/route/min_adv_mss</tag>
3474 <tag>/proc/sys/net/ipv4/route/min_delay</tag>
3475 Delays for flushing the routing cache.
3476 <tag>/proc/sys/net/ipv4/route/min_pmtu</tag>
3478 <tag>/proc/sys/net/ipv4/route/mtu_expires</tag>
3480 <tag>/proc/sys/net/ipv4/route/redirect_load</tag>
3481 Factors which determine if more ICPM redirects should be sent to a specific
3482 host. No redirects will be sent once the load limit or the maximum number of
3483 redirects has been reached.
3484 <tag>/proc/sys/net/ipv4/route/redirect_number</tag>
3485 See /proc/sys/net/ipv4/route/redirect_load.
3486 <tag>/proc/sys/net/ipv4/route/redirect_silence</tag>
3487 Timeout for redirects. After this period redirects will be sent again, even if
3488 this has been stopped, because the load or number limit has been reached.
3492 <sect>Advanced & less common queueing disciplines
3494 Should you find that you have needs not addressed by the queues mentioned
3495 earlier, the kernel contains some other more specialized queues mentioned here.
3498 These classless queues are even simpler than pfifo_fast in that they lack
3499 the internal bands - all traffic is really equal. They have one important
3500 benefit though, they have some statistics. So even if you don't need shaping
3501 or prioritizing, you can use this qdisc to determine the backlog on your
3504 pfifo has a length measured in packets, bfifo in bytes.
3505 <sect2>Parameters & usage
3509 Specifies the length of the queue. Measured in bytes for bfifo, in packets
3510 for pfifo. Defaults to the interface txqueuelen (see pfifo_fast chapter)
3511 packets long or txqueuelen*mtu bytes for bfifo.
3513 <sect1> Clark-Shenker-Zhang algorithm (CSZ)
3515 This is so theoretical that not even Alexey (the main CBQ author) claims to
3516 understand it. From his source:
3518 "David D. Clark, Scott Shenker and Lixia Zhang
3519 Supporting Real-Time Applications in an Integrated Services Packet
3520 Network: Architecture and Mechanism.
3522 As I understand it, the main idea is to create WFQ flows for each guaranteed
3523 service and to allocate the rest of bandwith to dummy flow-
0. Flow-
0
3524 comprises the predictive services and the best effort traffic; it is handled
3525 by a priority scheduler with the highest priority band allocated for
3526 predictive services, and the rest --- to the best effort packets.
3528 Note that in CSZ flows are NOT limited to their bandwidth. It is supposed
3529 that the flow passed admission control at the edge of the QoS network and it
3530 doesn't need further shaping. Any attempt to improve the flow or to shape it
3531 to a token bucket at intermediate hops will introduce undesired delays and
3534 At the moment CSZ is the only scheduler that provides true guaranteed
3535 service. Another schemes (including CBQ) do not provide guaranteed delay and
3538 Does not currently seem like a good canidate to use, unless you've read and
3539 understand the article mentioned.
3542 Esteve Camps Chust <marvin@grn.es>&nl;
3543 This text is an extract from my thesis on "QoS Support in Linux
", September 2000.&nl;
3545 Source documents:&nl;
3547 <item><url url="http://ica1www.epfl.ch/~almesber
" name="Draft-almesberger-wajhak-diffserv-linux-
01.txt
">.
3548 <item>Examples in iproute2 distribution.
3549 <item><url url="http://www.qosforum.com/white-papers/qosprot_v3.pdf
" name="White Paper-QoS protocols and architectures
"> and
3550 <url url="http://www.qosforum.com/docs/faq
" name="IP QoS Frequently Asked Questions
"> both by <em>Quality of Service Forum</em>.
3553 This chapter was written by Esteve Camps <esteve@hades.udg.es>.
3557 First of all, first of all, it would be a great idea for you to read RFCs
3558 written about this (RFC2474, RFC2475, RFC2597 and RFC2598) at <url
3559 url="http://www.ietf.org/html.charters/diffserv-charter.html
" name="IETF
3560 DiffServ working Group web site
"> and <url
3561 url="http://ica1www.epfl.ch/~almesber
" name="Werner Almesberger web site
">
3562 (he wrote the code to support Differentiated Services on Linux).
3564 <sect2>What is Dsmark related to?
3566 Dsmark is a queueing discipline that offers the capabilities needed in
3567 Differentiated Services (also called DiffServ or, simply, DS). DiffServ is
3568 one of two actual QoS architectures (the other one is called Integrated
3569 Services) that is based on a value carried by packets in the DS field of the
3573 One of the first solutions in IP designed to offer some QoS level was
3574 the Type of Service field (TOS byte) in IP header. By changing that value,
3575 we could choose a high/low level of throughput, delay or reliability.
3576 But this didn't provide sufficient flexibility to the needs of new
3577 services (such as real-time applications, interactive applications and
3578 others). After this, new architectures appeared. One of these was DiffServ
3579 which kept TOS bits and renamed DS field.
3580 <sect2>Differentiated Services guidelines
3582 Differentiated Services is group-oriented. I mean, we don't know anything
3583 about flows (this will be the Integrated Services purpose); we know about
3584 flow aggregations and we will apply different behaviours depending on which
3585 aggregation a packet belongs to.
3588 When a packet arrives to an edge node (entry node to a DiffServ domain)
3589 entering to a DiffServ Domain we'll have to policy, shape and/or mark those
3590 packets (marking refers to assigning a value to the DS field. It's just like the
3591 cows :-) ). This will be the mark/value that the internal/core nodes on our
3592 DiffServ Domain will look at to determine which behaviour or QoS level
3596 As you can deduce, Differentiated Services involves a domain on which
3597 all DS rules will have to be applied. In fact you can think &dquot;I
3598 will classify all the packets entering my domain. Once they enter my
3599 domain they will be subjected to the rules that my classification dictates
3600 and every traversed node will apply that QoS level&dquot;.
3602 In fact, you can apply your own policies into your local domains, but some
3603 <em>Service Level Agreements</em> should be considered when connecting to
3607 At this point, you maybe have a lot of questions. DiffServ is more than I've
3608 explained. In fact, you can understand that I can not resume more than 3
3609 RFC's in just 50 lines :-).
3611 <sect2>Working with Dsmark
3614 As the DiffServ bibliography specifies, we differentiate boundary nodes and
3615 interior nodes. These are two important points in the traffic path. Both
3616 types perform a classification when the packets arrive. Its result may be
3617 used in different places along the DS process before the packet is released
3618 to the network. It's just because of this that the diffserv code supplies an
3619 structure called sk_buff, including a new field called skb->tc_index
3620 where we'll store the result of initial classification that may be used in
3621 several points in DS treatment.
3624 The skb->tc_index value will be initially set by the DSMARK qdisc,
3625 retrieving it from the DS field in IP header of every received packet.
3626 Besides, cls_tcindex classifier will read all or part of skb->tcindex
3627 value and use it to select classes.
3630 But, first of all, take a look at DSMARK qdisc command and its parameters:
3632 ... dsmark indices INDICES [ default_index DEFAULT_INDEX ] [ set_tc_index ]
3634 What do these parameters mean?
3636 <item><bf>indices</bf>: size of table of (mask,value) pairs. Maximum value is 2^n, where n>=0.
3637 <item><bf>Default_index</bf>: the default table entry index if classifier finds no match.
3638 <item><bf>Set_tc_index</bf>: instructs dsmark discipline to retrieve the DS field and store it onto skb->tc_index.
3640 Let's see the DSMARK process.
3642 <sect2>How SCH_DSMARK works.
3644 This qdisc will apply the next steps:
3646 <item>If we have declared set_tc_index option in qdisc command, DS field is retrieved and stored onto
3647 skb->tc_index variable.
3648 <item>Classifier is invoked. The classifier will be executed and it will return a class ID that will be stored in
3649 skb->tc_index variable.If no filter matches are found, we consider the default_index option to be the
3650 classId to store. If neither set_tc_index nor default_index has been declared results may be
3652 <item>After been sent to internal qdisc's where you can reuse the result of the filter, the classid returned by
3653 the internal qdisc is stored into skb->tc_index. We will use this value in the future to index a mask-
3654 value table. The final result to assign to the packet will be that resulting from next operation:
3656 New_Ds_field = ( Old_DS_field & mask ) | value
3659 <item>Thus, new value will result from "anding
" ds_field and mask values and next, this result "ORed
" with
3660 value parameter. See next diagram to understand all this process:
3665 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
3667 | -- If you declare set_tc_index, we set DS | | <-----May change
3668 | value into skb->tc_index variable | |O DS field
3670 +-|-+ +------+ +---+-+ Internal +-+ +---N|-----|----+
3671 | | | | tc |--->| | |--> . . . -->| | | D| | |
3672 | | |----->|index |--->| | | Qdisc | |---->| v | |
3673 | | | |filter|--->| | | +---------------+ | ---->(mask,value) |
3674 -->| O | +------+ +-|-+--------------^----+ / | (. , .) |
3675 | | | ^ | | | | (. , .) |
3676 | | +----------|---------|----------------|-------|--+ (. , .) |
3677 | | sch_dsmark | | | | |
3678 +-|------------|---------|----------------|-------|------------------+
3679 | | | <- tc_index -> | |
3680 | |(read) | may change | | <--------------Index to the
3681 | | | | | (mask,value)
3682 v | v v | pairs table
3683 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ->
3688 How to do marking? Just change the mask and value of the class you want to remark. See next line of code:
3690 tc class change dev eth0 classid 1:1 dsmark mask 0x3 value 0xb8
3692 This changes the (mask,value) pair in hash table, to remark packets belonging to class 1:1.You have to "change
" this values
3693 because of default values that (mask,value) gets initially (see table below).
3695 Now, we'll explain how TC_INDEX filter works and how fits into this. Besides, TCINDEX filter can be
3696 used in other configurations rather than those including DS services.
3699 <sect2>TC_INDEX Filter
3701 This is the basic command to declare a TC_INDEX filter:
3704 ... tcindex [ hash SIZE ] [ mask MASK ] [ shift SHIFT ]
3705 [ pass_on | fall_through ]
3706 [ classid CLASSID ] [ police POLICE_SPEC ]
3709 Next, we show the example used to explain TC_INDEX operation mode. Pay attention to bolded words:
3711 tc qdisc add dev eth0 handle 1:0 root dsmark indices 64 <bf>set_tc_index</bf>&nl;
3712 tc filter add dev eth0 parent 1:0 protocol ip prio 1 tcindex <bf>mask 0xfc shift 2</bf>&nl;
3713 tc qdisc add dev eth0 parent 1:0 handle 2:0 cbq bandwidth 10Mbit cell 8 avpkt 1000 mpu 64&nl;
3714 # EF traffic class&nl;
3715 tc class add dev eth0 parent 2:0 classid 2:1 cbq bandwidth 10Mbit rate 1500Kbit avpkt 1000 prio 1 bounded isolated allot 1514 weight 1 maxburst 10&nl;
3716 # Packet fifo qdisc for EF traffic&nl;
3717 tc qdisc add dev eth0 parent 2:1 pfifo limit 5&nl;
3718 tc filter add dev eth0 parent 2:0 protocol ip prio 1 <bf>handle 0x2e</bf> tcindex <bf>classid 2:1 pass_on</bf>&nl;
3720 (This code is not complete. It's just an extract from EFCBQ example included in iproute2 distribution).
3722 First of all, suppose we receive a packet marked as EF . If you read RFC2598, you'll see that DSCP
3723 recommended value for EF traffic is 101110. This means that DS field will be 10111000 (remember that
3724 less signifiant bits in TOS byte are not used in DS) or 0xb8 in hexadecimal codification.
3730 +---+ +-------+ +---+-+ +------+ +-+ +-------+
3731 | | | | | | | |FILTER| +-+ +-+ | | | |
3732 | |----->| MASK | -> | | | -> |HANDLE|->| | | | -> | | -> | |
3733 | | . | =0xfc | | | | |0x2E | | +----+ | | | | |
3734 | | . | | | | | +------+ +--------+ | | | |
3735 | | . | | | | | | | | |
3736 -->| | . | SHIFT | | | | | | | |-->
3737 | | . | =2 | | | +----------------------------+ | | |
3738 | | | | | | CBQ 2:0 | | |
3739 | | +-------+ +---+--------------------------------+ | |
3741 | +-------------------------------------------------------------+ |
3743 +-------------------------------------------------------------------------+
3748 The packet arrives, then, set with 0xb8 value at DS field. As we explained before, dsmark qdisc identified
3749 by 1:0 id in the example, retrieves DS field and store it in skb->tc_index variable.
3750 Next step in the example will correspond to the filter associated to this qdisc (second line in the example).
3751 This will perform next operations:
3754 Value1 = skb->tc_index & MASK
3755 Key = Value1 >> SHIFT
3760 In the example, MASK=0xFC i SHIFT=2.
3763 Value1 = 10111000 & 11111100 = 10111000
3764 Key = 10111000 >> 2 = 00101110 -> 0x2E in hexadecimal
3769 The returned value will correspond to a qdisc interal filter handle (in the example, identifier 2:0). If a
3770 filter with this id exists, policing and metering conditions will be verified (in case that filter includes this)
3771 and the classid will be returned (in our example, classid 2:1) and stored in skb->tc_index variable.
3774 But if any filter with that identifier is found, the result will depend on fall_through flag declaration. If so,
3775 value key is returned as classid. If not, an error is returned and process continues with the rest filters. Be
3776 careful if you use fall_through flag; this can be done if a simple relation exists between values
3777 &nl;of skb->tc_index variable and class id's.
3780 The latest parameters to comment on are hash and pass_on. The first one
3781 relates to hash table size. Pass_on will be used to indicate that if no classid
3782 equal to the result of this filter is found, try next filter.
3783 The default action is fall_through (look at next table).
3786 Finally, let's see which possible values can be set to all this TCINDEX parameters:
3789 TC Name Value Default
3790 -----------------------------------------------------------------
3791 Hash 1...0x10000 Implementation dependent
3792 Mask 0...0xffff 0xffff
3794 Fall through / Pass_on Flag Fall_through
3795 Classid Major:minor None
3800 This kind of filter is very powerful. It's necessary to explore all possibilities. Besides, this filter is not only used in DiffServ configurations.
3801 You can use it as any other kind of filter.
3803 I recommend you to look at all DiffServ examples included in iproute2 distribution. I promise I will try to
3804 complement this text as soon as I can. Besides, all I have explained is the result of a lot of tests.
3805 I would thank you tell me if I'm wrong in any point.
3806 <sect1>Ingress qdisc
3808 All qdiscs discussed so far are egress qdiscs. Each interface however can
3809 also have an ingress qdisc which is not used to send packets
3810 out to the network adaptor. Instead, it allows you to apply tc filters to
3811 packets coming in over the interface, regardless of whether they have a local
3812 destination or are to be forwarded.
3814 As the tc filters contain a full Token Bucket Filter implementation, and are
3815 also able to match on the kernel flow estimator, there is a lot of
3816 functionality available. This effectively allows you to police incoming
3817 traffic, before it even enters the IP stack.
3819 <sect2>Parameters & usage
3821 The ingress qdisc itself does not require any parameters. It differs from
3822 other qdiscs in that it does not occupy the root of a device. Attach it like
3825 # tc qdisc add dev eth0 ingress
3827 This allows you to have other, sending, qdiscs on your device besides the
3830 For a contrived example how the ingress qdisc could be used, see the
3833 <sect1>Random Early Detection (RED)
3835 This section is meant as an introduction to backbone routing, which often
3836 involves <100 megabit bandwidths, which requires a different approach than
3837 your ADSL modem at home.
3839 The normal behaviour of router queues on the Internet is called tail-drop.
3840 Tail-drop works by queueing up to a certain amount, then dropping all traffic
3841 that 'spills over'. This is very unfair, and also leads to retransmit
3842 synchronisation. When retransmit synchronisation occurs, the sudden burst
3843 of drops from a router that has reached its fill will cause a delayed burst
3844 of retransmits, which will over fill the congested router again.
3846 In order to cope with transient congestion on links, backbone routers will
3847 often implement large queues. Unfortunately, while these queues are good for
3848 throughput, they can substantially increase latency and cause TCP
3849 connections to behave very bursty during congestion.
3851 These issues with tail-drop are becoming increasingly troublesome on the
3852 Internet because the use of network unfriendly applications is increasing.
3853 The Linux kernel offers us RED, short for Random Early Detect, also called
3854 Random Early Drop, as that is how it works.
3856 RED isn't a cure-all for this, applications which inappropriately fail to
3857 implement exponential backoff still get an unfair share of the bandwidth,
3858 however, with RED they do not cause as much harm to the throughput and
3859 latency of other connections.
3861 RED statistically drops packets from flows before it reaches its hard
3862 limit. This causes a congested backbone link to slow more gracefully, and
3863 prevents retransmit synchronisation. This also helps TCP find its 'fair'
3864 speed faster by allowing some packets to get dropped sooner keeping queue
3865 sizes low and latency under control. The probability of a packet being
3866 dropped from a particular connection is proportional to its bandwidth usage
3867 rather than the number of packets it transmits.
3869 RED is a good queue for backbones, where you can't afford the
3870 complexity of per-session state tracking needed by fairness queueing.
3872 In order to use RED, you must decide on three parameters: Min, Max, and
3873 burst. Min sets the minimum queue size in bytes before dropping will begin,
3874 Max is a soft maximum that the algorithm will attempt to stay under, and
3875 burst sets the maximum number of packets that can 'burst through'.
3877 You should set the min by calculating that highest acceptable base queueing
3878 latency you wish, and multiply it by your bandwidth. For instance, on my
3879 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set
3880 min to 1600 bytes. Setting min too small will degrade throughput and too
3881 large will degrade latency. Setting a small min is not a replacement for
3882 reducing the MTU on a slow link to improve interactive response.
3884 You should make max at least twice min to prevent synchronisation. On slow
3885 links with small min's it might be wise to make max perhaps four or
3886 more times large then min.
3888 Burst controls how the RED algorithm responds to bursts. Burst must be set
3889 larger then min/avpkt. Experimentally, I've found (min+min+max)/(3*avpkt) to
3892 Additionally, you need to set limit and avpkt. Limit is a safety value, after
3893 there are limit bytes in the queue, RED 'turns into' tail-drop. I typical set
3894 limit to eight times max. Avpkt should be your average packet size. 1000
3895 works okay on high speed Internet links with a 1500byte MTU.
3897 Read <url url="http://www.aciri.org/floyd/papers/red/red.html
"
3898 name="the paper on RED queueing
"> by Sally Floyd and Van Jacobson for technical
3900 <sect1>Generic Random Early Detection
3902 Not a lot is known about GRED. It looks like GRED with several internal
3903 queues, whereby the internal queue is chosen based on the Diffserv tcindex
3904 field. According to a slide found <url
3905 url="http://www.davin.ottawa.on.ca/ols/img22.htm
" name="here
">, it contains
3906 the capabilities of Cisco's 'Distributed Weighted RED', as well as Dave
3909 Each virtual queue can have its own Drop Parameters specified.
3911 FIXME: get Jamal or Werner to tell us more
3913 <sect1>VC/ATM emulation
3915 This is quite a major effort by Werner Almesberger to allow you to build
3916 Virtual Circuits over TCP/IP sockets. A Virtual Circuit is a concept from
3919 For more information, see the <url url="http://linux-atm.sourceforge.net/
"
3920 name="ATM on Linux homepage
">.
3922 <sect1>Weighted Round Robin (WRR)
3924 This qdisc is not included in the standard kernels but can be downloaded from
3925 <url url="http://wipl-wrr.dkik.dk/wrr/
">.
3926 Currently the qdisc is only tested with Linux 2.2 kernels but it will
3927 probably work with 2.4/2.5 kernels too.
3929 The WRR qdisc distributes bandwidth between its classes using the weighted
3930 round robin scheme. That is, like the CBQ qdisc it contains classes
3931 into which arbitrary qdiscs can be plugged. All classes which have sufficient
3932 demand will get bandwidth proportional to the weights associated with the classes.
3933 The weights can be set manually using the <tt>tc</tt> program. But they
3934 can also be made automatically decreasing for classes transferring much data.
3936 The qdisc has a built-in classifier which assigns packets coming from or
3937 sent to different machines to different classes. Either the MAC or IP and
3938 either source or destination addresses can be used. The MAC address can only
3939 be used when the Linux box is acting as an ethernet bridge, however. The
3940 classes are automatically assigned to machines based on the packets seen.
3942 The qdisc can be very useful at sites such as dorms where a lot of unrelated
3943 individuals share an Internet connection. A set of scripts setting up a
3944 relevant behavior for such a site is a central part of the WRR distribution.
3948 This section contains 'cookbook' entries which may help you solve problems.
3949 A cookbook is no replacement for understanding however, so try and comprehend
3952 <sect1>Running multiple sites with different SLAs
3954 You can do this in several ways. Apache has some support for this with a
3955 module, but we'll show how Linux can do this for you, and do so for other
3956 services as well. These commands are stolen from a presentation by Jamal
3957 Hadi that's referenced below.
3959 Let's say we have two customers, with http, ftp and streaming audio, and we
3960 want to sell them a limited amount of bandwidth. We do so on the server itself.
3962 Customer A should have at most 2 megabits, customer B has paid for 5
3963 megabits. We separate our customers by creating virtual IP addresses on our
3967 # ip address add 188.177.166.1 dev eth0
3968 # ip address add 188.177.166.2 dev eth0
3971 It is up to you to attach the different servers to the right IP address. All
3972 popular daemons have support for this.
3974 We first attach a CBQ qdisc to eth0:
3976 # tc qdisc add dev eth0 root handle 1: cbq bandwidth 10Mbit cell 8 avpkt 1000 \
3980 We then create classes for our customers:
3983 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 10Mbit rate \
3984 2MBit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
3985 # tc class add dev eth0 parent 1:0 classid 1:2 cbq bandwidth 10Mbit rate \
3986 5Mbit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
3989 Then we add filters for our two classes:
3991 ##FIXME: Why this line, what does it do?, what is a divisor?:
3992 ##FIXME: A divisor has something to do with a hash table, and the number of
3994 # tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 1: u32 divisor 1
3995 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.1
3997 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.2
4003 FIXME: why no token bucket filter? is there a default pfifo_fast fallback
4006 <sect1>Protecting your host from SYN floods
4008 From Alexey's iproute documentation, adapted to netfilter and with more
4009 plausible paths. If you use this, take care to adjust the numbers to
4010 reasonable values for your system.
4012 If you want to protect an entire network, skip this script, which is best
4013 suited for a single host.
4015 It appears that you need the very latest version of the iproute2 tools to
4016 get this to work with 2.4.0.
4021 # sample script on using the ingress capabilities
4022 # this script shows how one can rate limit incoming SYNs
4023 # Useful for TCP-SYN attack protection. You can use
4024 # IPchains to have more powerful additions to the SYN (eg
4025 # in addition the subnet)
4027 #path to various utilities;
4028 #change to reflect yours.
4032 IPTABLES=/sbin/iptables
4035 # tag all incoming SYN packets through $INDEV as mark value 1
4036 ############################################################
4037 $iptables -A PREROUTING -i $INDEV -t mangle -p tcp --syn \
4038 -j MARK --set-mark 1
4039 ############################################################
4041 # install the ingress qdisc on the ingress interface
4042 ############################################################
4043 $TC qdisc add dev $INDEV handle ffff: ingress
4044 ############################################################
4048 # SYN packets are 40 bytes (320 bits) so three SYNs equals
4049 # 960 bits (approximately 1kbit); so we rate limit below
4050 # the incoming SYNs to 3/sec (not very useful really; but
4051 #serves to show the point - JHS
4052 ############################################################
4053 $TC filter add dev $INDEV parent ffff: protocol ip prio 50 handle 1 fw \
4054 police rate 1kbit burst 40 mtu 9k drop flowid :1
4055 ############################################################
4059 echo "---- qdisc parameters Ingress ----------
"
4060 $TC qdisc ls dev $INDEV
4061 echo "---- Class parameters Ingress ----------
"
4062 $TC class ls dev $INDEV
4063 echo "---- filter parameters Ingress ----------
"
4064 $TC filter ls dev $INDEV parent ffff:
4066 #deleting the ingress qdisc
4067 #$TC qdisc del $INDEV ingress
4069 <sect1>Ratelimit ICMP to prevent dDoS
4071 Recently, distributed denial of service attacks have become a major nuisance
4072 on the Internet. By properly filtering and ratelimiting your network, you can
4073 both prevent becoming a casualty or the cause of these attacks.
4075 You should filter your networks so that you do not allow non-local IP source
4076 addressed packets to leave your network. This stops people from anonymously
4077 sending junk to the Internet.
4079 <!-- FIXME: netfilter one liner. Is there a netfilter one-liner? Martijn -->
4082 Rate limiting goes much as shown earlier. To refresh your memory, our
4086 [The Internet] ---<E3, T3, whatever>--- [Linux router] --- [Office+ISP]
4090 We first set up the prerequisite parts:
4093 # tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000
4094 # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate \
4095 10Mbit allot 1514 prio 5 maxburst 20 avpkt 1000
4098 If you have 100Mbit, or more, interfaces, adjust these numbers. Now you need
4099 to determine how much ICMP traffic you want to allow. You can perform
4100 measurements with tcpdump, by having it write to a file for a while, and
4101 seeing how much ICMP passes your network. Do not forget to raise the
4104 If measurement is impractical, you might want to choose 5% of your available
4105 bandwidth. Let's set up our class:
4107 # tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate \
4108 100Kbit allot 1514 weight 800Kbit prio 5 maxburst 20 avpkt 250 \
4112 This limits at 100Kbit. Now we need a filter to assign ICMP traffic to this
4115 # tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip
4116 protocol 1 0xFF flowid 10:100
4120 <sect1>Prioritizing interactive traffic
4122 If lots of data is coming down your link, or going up for that matter, and
4123 you are trying to do some maintenance via telnet or ssh, this may not go too
4124 well. Other packets are blocking your keystrokes. Wouldn't it be great if
4125 there were a way for your interactive packets to sneak past the bulk
4126 traffic? Linux can do this for you!
4128 As before, we need to handle traffic going both ways. Evidently, this works
4129 best if there are Linux boxes on both ends of your link, although other
4130 UNIX's are able to do this. Consult your local Solaris/BSD guru for this.
4132 The standard pfifo_fast scheduler has 3 different 'bands'. Traffic in band 0
4133 is transmitted first, after which traffic in band 1 and 2 gets considered.
4134 It is vital that our interactive traffic be in band 0!
4136 We blatantly adapt from the (soon to be obsolete) ipchains HOWTO:
4138 There are four seldom-used bits in the IP header, called the Type of Service
4139 (TOS) bits. They effect the way packets are treated; the four bits are
4140 "Minimum Delay
", "Maximum Throughput
", "Maximum Reliability
" and "Minimum
4141 Cost
". Only one of these bits is allowed to be set. Rob van Nieuwkerk, the
4142 author of the ipchains TOS-mangling code, puts it as follows:
4145 Especially the "Minimum Delay
" is important for me. I switch it on for
4146 "interactive
" packets in my upstream (Linux) router. I'm
4147 behind a 33k6 modem link. Linux prioritizes packets in 3 queues. This
4148 way I get acceptable interactive performance while doing bulk
4149 downloads at the same time.
4152 The most common use is to set telnet & ftp control connections to "Minimum
4153 Delay
" and FTP data to "Maximum Throughput
". This would be
4154 done as follows, on your upstream router:
4157 # iptables -A PREROUTING -t mangle -p tcp --sport telnet \
4158 -j TOS --set-tos Minimize-Delay
4159 # iptables -A PREROUTING -t mangle -p tcp --sport ftp \
4160 -j TOS --set-tos Minimize-Delay
4161 # iptables -A PREROUTING -t mangle -p tcp --sport ftp-data \
4162 -j TOS --set-tos Maximize-Throughput
4165 Now, this only works for data going from your telnet foreign host to your
4166 local computer. The other way around appears to be done for you, ie, telnet,
4167 ssh & friends all set the TOS field on outgoing packets automatically.
4169 Should you have an application that does not do this, you can always do it
4170 with netfilter. On your local box:
4173 # iptables -A OUTPUT -t mangle -p tcp --dport telnet \
4174 -j TOS --set-tos Minimize-Delay
4175 # iptables -A OUTPUT -t mangle -p tcp --dport ftp \
4176 -j TOS --set-tos Minimize-Delay
4177 # iptables -A OUTPUT -t mangle -p tcp --dport ftp-data \
4178 -j TOS --set-tos Maximize-Throughput
4181 <sect1>Transparent web-caching using netfilter, iproute2, ipchains and squid
4184 This section was sent in by reader Ram Narula from Internet for Education
4187 The regular technique in accomplishing this in Linux
4188 is probably with use of ipchains AFTER making sure
4189 that the "outgoing
" port 80(web) traffic gets routed through
4190 the server running squid.
4192 There are 3 common methods to make sure "outgoing
"
4193 port 80 traffic gets routed to the server running squid
4194 and 4th one is being introduced here.
4197 <tag>Making the gateway router do it.</tag>
4198 If you can tell your gateway router to
4199 match packets that has outgoing destination port
4200 of 80 to be sent to the IP address of squid server.
4204 This would put additional load on the router and
4205 some commercial routers might not even support this.
4206 <tag>Using a Layer 4 switch.</tag>
4207 Layer 4 switches can handle this without any problem.
4211 The cost for this equipment is usually very high. Typical
4212 layer 4 switch would normally cost more than
4213 a typical router+good linux server.
4214 <tag>Using cache server as network's gateway.</tag>
4215 You can force ALL traffic through cache server.
4219 This is quite risky because Squid does
4220 utilize lots of cpu power which might
4221 result in slower over-all network performance
4222 or the server itself might crash and no one on the
4223 network will be able to access the Internet if
4227 <tag>Linux+NetFilter router.</tag>
4228 By using NetFilter another technique can be implemented
4229 which is using NetFilter for "mark
"ing the packets
4230 with destination port 80 and using iproute2 to
4231 route the "mark
"ed packets to the Squid server.
4239 10.0.0.1 naret (NetFilter server)
4240 10.0.0.2 silom (Squid server)
4241 10.0.0.3 donmuang (Router connected to the Internet)
4242 10.0.0.4 kaosarn (other server on network)
4244 10.0.0.0/24 main network
4245 10.0.0.0/19 total network
4255 ------------hub/switch----------
4257 naret silom kaosarn RAS etc.
4259 First, make all traffic pass through naret by making
4260 sure it is the default gateway except for silom.
4261 Silom's default gateway has to be donmuang (10.0.0.3) or
4262 this would create web traffic loop.
4266 (all servers on my network had 10.0.0.1 as the default gateway
4267 which was the former IP address of donmuang router so what I did
4268 was changed the IP address of donmuang to 10.0.0.3 and gave
4269 naret ip address of 10.0.0.1)
4274 -setup squid and ipchains
4278 Setup Squid server on silom, make sure it does support
4279 transparent caching/proxying, the default port is usually
4280 3128, so all traffic for port 80 has to be redirected to port
4281 3128 locally. This can be done by using ipchains with the following:
4284 silom# ipchains -N allow1
4285 silom# ipchains -A allow1 -p TCP -s 10.0.0.0/19 -d 0/0 80 -j REDIRECT 3128
4286 silom# ipchains -I input -j allow1
4291 Or, in netfilter lingo:
4293 silom# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
4296 (note: you might have other entries as well)
4299 For more information on setting Squid server please refer
4300 to Squid faq page on <url
4301 url="http://squid.nlanr.net
" name="http://squid.nlanr.net
">).
4305 Make sure ip forwarding is enabled on this server and the default
4306 gateway for this server is donmuang router (NOT naret).
4313 -setup iptables and iproute2
4314 -disable icmp REDIRECT messages (if needed)
4318 <item>"Mark
" packets of destination port 80 with value 2
4320 naret# iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 \
4321 -j MARK --set-mark 2
4324 <item>Setup iproute2 so it will route packets with "mark
" 2 to silom
4326 naret# echo 202 www.out >> /etc/iproute2/rt_tables
4327 naret# ip rule add fwmark 2 table www.out
4328 naret# ip route add default via 10.0.0.2 dev eth0 table www.out
4329 naret# ip route flush cache
4333 If donmuang and naret is on the same subnet then
4334 naret should not send out icmp REDIRECT messages.
4335 In this case it is, so icmp REDIRECTs has to be
4338 naret# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
4339 naret# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
4340 naret# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
4346 The setup is complete, check the configuration
4351 naret# iptables -t mangle -L
4352 Chain PREROUTING (policy ACCEPT)
4353 target prot opt source destination
4354 MARK tcp -- anywhere anywhere tcp dpt:www MARK set 0x2
4356 Chain OUTPUT (policy ACCEPT)
4357 target prot opt source destination
4360 0: from all lookup local
4361 32765: from all fwmark 2 lookup www.out
4362 32766: from all lookup main
4363 32767: from all lookup default
4365 naret# ip route list table www.out
4366 default via 203.114.224.8 dev eth0
4369 10.0.0.1 dev eth0 scope link
4370 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.1
4371 127.0.0.0/8 dev lo scope link
4372 default via 10.0.0.3 dev eth0
4374 (make sure silom belongs to one of the above lines, in this case
4375 it's the line with 10.0.0.0/24)
4383 <sect2>Traffic flow diagram after implementation
4387 |-----------------------------------------|
4388 |Traffic flow diagram after implementation|
4389 |-----------------------------------------|
4395 -----------------donmuang router---------------------
4400 *destination port 80 traffic=========>(cache) ||
4403 \\===================================kaosarn, RAS, etc.
4407 Note that the network is asymmetric as there is one extra hop on
4408 general outgoing path.
4411 Here is run down for packet traversing the network from kaosarn
4412 to and from the Internet.
4414 For web/http traffic:
4415 kaosarn http request->naret->silom->donmuang->internet
4416 http replies from Internet->donmuang->silom->kaosarn
4418 For non-web/http requests(eg. telnet):
4419 kaosarn outgoing data->naret->donmuang->internet
4420 incoming data from Internet->donmuang->kaosarn
4423 <sect1>Circumventing Path MTU Discovery issues with per route MTU settings
4425 For sending bulk data, the Internet generally works better when using larger
4426 packets. Each packet implies a routing decision, when sending a 1 megabyte
4427 file, this can either mean around 700 packets when using packets that are as
4428 large as possible, or 4000 if using the smallest default.
4430 However, not all parts of the Internet support full 1460 bytes of payload
4431 per packet. It is therefore necessary to try and find the largest packet
4432 that will 'fit', in order to optimize a connection.
4434 This process is called 'Path MTU Discovery', where MTU stands for 'Maximum
4437 When a router encounters a packet that's too big too send in one piece, AND
4438 it has been flagged with the "Don't Fragment
" bit, it returns an ICMP
4439 message stating that it was forced to drop a packet because of this. The
4440 sending host acts on this hint by sending smaller packets, and by iterating
4441 it can find the optimum packet size for a connection over a certain path.
4443 This used to work well until the Internet was discovered by hooligans who do
4444 their best to disrupt communications. This in turn lead administrators to
4445 either block or shape ICMP traffic in a misguided attempt to improve
4446 security or robustness of their Internet service.
4448 What has happened now is that Path MTU Discovery is working less and less
4449 well and fails for certain routes, which leads to strange TCP/IP sessions
4450 which die after a while.
4452 Although I have no proof for this, two sites who I used to have this problem
4453 with both run Alteon Acedirectors before the affected systems - perhaps
4454 somebody more knowledgeable can provide clues as to why this happens.
4458 When you encounter sites that suffer from this problem, you can disable Path
4459 MTU discovery by setting it manually. Koos van den Hout, slightly edited,
4464 The following problem: I set the mtu/mru of my leased line running ppp to
4465 296 because it's only 33k6 and I cannot influence the queueing on the
4466 other side. At 296, the response to a keypress is within a reasonable
4469 And, on my side I have a masqrouter running (of course) Linux.
4471 Recently I split 'server' and 'router' so most applications are run on a
4472 different machine than the routing happens on.
4474 I then had trouble logging into irc. Big panic! Some digging did find
4475 out that I got connected to irc, even showed up as 'connected' on irc
4476 but I did not receive the motd from irc. I checked what could be wrong
4477 and noted that I already had some previous trouble reaching certain
4478 websites related to the MTU, since I had no trouble reaching them when
4479 the MTU was 1500, the problem just showed when the MTU was set to 296.
4480 Since irc servers block about every kind of traffic not needed for their
4481 immediate operation, they also block icmp.
4483 I managed to convince the operators of a webserver that this was the cause
4484 of a problem, but the irc server operators were not going to fix this.
4486 So, I had to make sure outgoing masqueraded traffic started with the lower
4487 mtu of the outside link. But I want local ethernet traffic to have the
4488 normal mtu (for things like nfs traffic).
4492 ip route add default via 10.0.0.1 mtu 296
4495 (10.0.0.1 being the default gateway, the inside address of the
4496 masquerading router)
4499 In general, it is possible to override PMTU Discovery by setting specific
4500 routes. For example, if only a certain subnet is giving problems, this
4504 ip route add 195.96.96.0/24 via 10.0.0.1 mtu 1000
4506 <sect1>Circumventing Path MTU Discovery issues with MSS Clamping (for ADSL,
4507 cable, PPPoE & PPtP users)
4509 As explained above, Path MTU Discovery doesn't work as well as it should
4510 anymore. If you know for a fact that a hop somewhere in your network has a
4511 limited (<1500) MTU, you cannot rely on PMTU Discovery finding this out.
4513 Besides MTU, there is yet another way to set the maximum packet size, the so
4514 called Maximum Segment Size. This is a field in the TCP Options part of a
4517 Recent Linux kernels, and a few pppoe drivers (notably, the excellent
4518 Roaring Penguin one), feature the possibility to 'clamp the MSS'.
4520 The good thing about this is that by setting the MSS value, you are telling
4521 the remote side unequivocally 'do not ever try to send me packets bigger
4522 than this value'. No ICMP traffic is needed to get this to work.
4524 The bad thing is that it's an obvious hack - it breaks 'end to end' by
4525 modifying packets. Having said that, we use this trick in many places and it
4528 In order for this to work you need at least iptables-1.2.1a and Linux 2.4.3
4529 or higher. The basic commandline is:
4531 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
4534 This calculates the proper MSS for your link. If you are feeling brave, or
4535 think that you know best, you can also do something like this:
4538 # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
4541 This sets the MSS of passing SYN packets to 128. Use this if you have VoIP
4542 with tiny packets, and huge http packets which are causing chopping in your
4544 <sect1>The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads
4546 Note: This script has recently been upgraded and previously only worked for
4547 Linux clients in your network! So you might want to update if you have
4548 Windows machines or Macs in your network and noticed that they were not able
4549 to download faster while others were uploading.
4551 I attempted to create the holy grail:
4553 <tag>Maintain low latency for interfactive traffic at all times</tag>
4554 This means that downloading or uploading files should not disturb SSH or
4555 even telnet. These are the most important things, even 200ms latency is
4556 sluggish to work over.
4557 <tag>Allow 'surfing' at reasonable speeds while up or downloading</tag>
4558 Even though http is 'bulk' traffic, other traffic should not drown it out
4560 <tag>Make sure uploads don't harm downloads, and the other way around</tag>
4561 This is a much observed phenomenon where upstream traffic simply destroys
4564 It turns out that all this is possible, at the cost of a tiny bit of
4565 bandwidth. The reason that uploads, downloads and ssh hurt eachother is the
4566 presence of large queues in many domestic access devices like cable or DSL
4569 The next section explains in depth what causes the delays, and how we can
4570 fix them. You can safely skip it and head straight for the script if you
4571 don't care how the magic is performed.
4572 <sect2>Why it doesn't work well by default
4574 ISPs know that they are benchmarked solely on how fast people can download.
4575 Besides available bandwidth, download speed is influenced heavily by packet
4576 loss, which seriously hampers TCP/IP performance. Large queues can help
4577 prevent packetloss, and speed up downloads. So ISPs configure large queues.
4579 These large queues however damage interactivity. A keystroke must first
4580 travel the upstream queue, which may be seconds (!) long and go to your
4581 remote host. It is then displayed, which leads to a packet coming back, which
4582 must then traverse the downstream queue, located at your ISP, before it
4583 appears on your screen.
4585 This HOWTO teaches you how to mangle and process the queue in many ways, but
4586 sadly, not all queues are accessible to us. The queue over at the ISP is
4587 completely off-limits, whereas the upstream queue probably lives inside your
4588 cable modem or DSL device. You may or may not be able to configure it. Most
4591 So, what next? As we can't control either of those queues, they must be
4592 eliminated, and moved to your Linux router. Luckily this is possible.
4595 <tag>Limit upload speed</tag>
4596 By limiting our upload speed to slightly less than the truly available rate,
4597 no queues are built up in our modem. The queue is now moved to Linux.
4598 <tag>Limit download speed</tag>
4599 This is slightly trickier as we can't really influence how fast the internet
4600 ships us data. We can however drop packets that are coming in too fast,
4601 which causes TCP/IP to slow down to just the rate we want. Because we don't
4602 want to drop traffic unnecessarily, we configure a 'burst' size we allow at
4606 Now, once we have done this, we have eliminated the downstream queue totally
4607 (except for short bursts), and gain the ability to manage the upstream queue
4608 with all the power Linux offers.
4610 What remains to be done is to make sure interactive traffic jumps to the
4611 front of the upstream queue. To make sure that uploads don't hurt downloads,
4612 we also move ACK packets to the front of the queue. This is what normally
4613 causes the huge slowdown observed when generating bulk traffic both ways.
4614 The ACKnowledgements for downstream traffic must compete with upstream
4615 traffic, and get delayed in the process.
4617 If we do all this we get the following measurements using an excellent ADSL
4618 connection from xs4all in the Netherlands:
4622 round-trip min/avg/max = 14.4/17.1/21.7 ms
4624 Without traffic conditioner, while downloading:
4625 round-trip min/avg/max = 560.9/573.6/586.4 ms
4627 Without traffic conditioner, while uploading:
4628 round-trip min/avg/max = 2041.4/2332.1/2427.6 ms
4630 With conditioner, during 220kbit/s upload:
4631 round-trip min/avg/max = 15.7/51.8/79.9 ms
4633 With conditioner, during 850kbit/s download:
4634 round-trip min/avg/max = 20.4/46.9/74.0 ms
4636 When uploading, downloads proceed at ~80% of the available speed. Uploads
4637 at around 90%. Latency then jumps to 850 ms, still figuring out why.
4640 What you can expect from this script depends a lot on your actual uplink
4641 speed. When uploading at full speed, there will always be a single packet
4642 ahead of your keystroke. That is the lower limit to the latency you can
4643 achieve - divide your MTU by your upstream speed to calculate. Typical
4644 values will be somewhat higher than that. Lower your MTU for better effects!
4646 Next, two versions of this script, one with Devik's excellent HTB, the other
4647 with CBQ which is in each Linux kernel, unlike HTB. Both are tested and work
4649 <sect2>The actual script (CBQ)
4651 Works on all kernels. Within the CBQ
4652 qdisc we place two Stochastic Fairness Queues that make sure that multiple
4653 bulk streams don't drown each other out.
4655 Downstream traffic is policed using a tc filter containing a Token Bucket
4658 You might improve on this script by adding 'bounded' to the line that starts
4659 with 'tc class add .. classid 1:20'. If you lowered your MTU, also lower the
4660 allot & avpkt numbers!
4665 # The Ultimate Setup For Your Internet Connection At Home
4668 # Set the following values to somewhat less than your actual download
4669 # and uplink speed. In kilobits
4674 # clean existing down- and uplink qdiscs, hide errors
4675 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
4676 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
4682 tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
4684 # shape everything at $UPLINK speed - this prevents huge queues in your
4685 # DSL modem which destroy latency:
4688 tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit \
4689 allot 1500 prio 5 bounded isolated
4691 # high prio class 1:10:
4693 tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit \
4694 allot 1600 prio 1 avpkt 1000
4696 # bulk and default class 1:20 - gets slightly less traffic,
4697 # and a lower priority:
4699 tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit \
4700 allot 1600 prio 2 avpkt 1000
4702 # both get Stochastic Fairness:
4703 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
4704 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
4707 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
4708 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
4709 match ip tos 0x10 0xff flowid 1:10
4711 # ICMP (ip protocol 1) in the interactive class 1:10 so we
4712 # can do measurements & impress our friends:
4713 tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \
4714 match ip protocol 1 0xff flowid 1:10
4716 # To speed up downloads while an upload is going on, put ACK packets in
4717 # the interactive class:
4719 tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \
4720 match ip protocol 6 0xff \
4721 match u8 0x05 0x0f at 0 \
4722 match u16 0x0000 0xffc0 at 2 \
4723 match u8 0x10 0xff at 33 \
4726 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
4728 tc filter add dev $DEV parent 1: protocol ip prio 13 u32 \
4729 match ip dst 0.0.0.0/0 flowid 1:20
4731 ########## downlink #############
4732 # slow downloads down to somewhat less than the real speed to prevent
4733 # queuing at our ISP. Tune to see how high you can set it.
4734 # ISPs tend to have *huge* queues to make sure big downloads are fast
4736 # attach ingress policer:
4738 tc qdisc add dev $DEV handle ffff: ingress
4740 # filter *everything* to it (0.0.0.0/0), drop everything that's
4741 # coming in too fast:
4743 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
4744 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
4746 If you want this script to be run by ppp on connect, copy it to
4749 If the last two lines give an error, update your tc tool to a newer version!
4750 <sect2>The actual script (HTB)
4752 The following script achieves all goals using the wonderful HTB queue, see
4753 the relevant chapter. Well worth patching your kernel for!
4757 # The Ultimate Setup For Your Internet Connection At Home
4760 # Set the following values to somewhat less than your actual download
4761 # and uplink speed. In kilobits
4766 # clean existing down- and uplink qdiscs, hide errors
4767 tc qdisc del dev $DEV root 2> /dev/null > /dev/null
4768 tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
4772 # install root HTB, point default traffic to 1:20:
4774 tc qdisc add dev $DEV root handle 1: htb default 20
4776 # shape everything at $UPLINK speed - this prevents huge queues in your
4777 # DSL modem which destroy latency:
4779 tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k
4781 # high prio class 1:10:
4783 tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
4786 # bulk & default class 1:20 - gets slightly less traffic,
4787 # and a lower priority:
4789 tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
4792 # both get Stochastic Fairness:
4793 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
4794 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
4796 # TOS Minimum Delay (ssh, NOT scp) in 1:10:
4797 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
4798 match ip tos 0x10 0xff flowid 1:10
4800 # ICMP (ip protocol 1) in the interactive class 1:10 so we
4801 # can do measurements & impress our friends:
4802 tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
4803 match ip protocol 1 0xff flowid 1:10
4805 # To speed up downloads while an upload is going on, put ACK packets in
4806 # the interactive class:
4808 tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
4809 match ip protocol 6 0xff \
4810 match u8 0x05 0x0f at 0 \
4811 match u16 0x0000 0xffc0 at 2 \
4812 match u8 0x10 0xff at 33 \
4815 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20
4818 ########## downlink #############
4819 # slow downloads down to somewhat less than the real speed to prevent
4820 # queuing at our ISP. Tune to see how high you can set it.
4821 # ISPs tend to have *huge* queues to make sure big downloads are fast
4823 # attach ingress policer:
4825 tc qdisc add dev $DEV handle ffff: ingress
4827 # filter *everything* to it (0.0.0.0/0), drop everything that's
4828 # coming in too fast:
4830 tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
4831 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
4834 If you want this script to be run by ppp on connect, copy it to
4837 If the last two lines give an error, update your tc tool to a newer version!
4838 <sect>Building bridges, and pseudo-bridges with Proxy ARP
4840 Bridges are devices which can be installed in a network without any
4841 reconfiguration. A network switch is basically a many-port bridge. A bridge
4842 is often a 2-port switch. Linux does however support multiple interfaces in
4843 a bridge, making it a true switch.
4845 Bridges are often deployed when confronted with a broken network that needs
4846 to be fixed without any alterations. Because the bridge is a layer-2 device,
4847 one layer below IP, routers and servers are not aware of its existence.
4848 This means that you can transparently block or modify certain packets, or do
4851 Another good thing is that a bridge can often be replaced by a cross cable
4852 or a hub, should it break down.
4854 The bad news is that a bridge can cause great confusion unless it is very
4855 well documented. It does not appear in traceroutes, but somehow packets
4856 disappear or get changed from point A to point B ('this network is
4857 HAUNTED!'). You should also wonder if an organization that 'does not want to
4858 change anything' is doing the right thing.
4860 The Linux 2.4/2.5 bridge is documented on
4862 <url url=" http://bridge.sourceforge.net/
" name="this page
">.
4864 <sect1>State of bridging and iptables
4866 As of Linux 2.4.14, bridging and iptables do not 'see' each other without
4867 help. If you bridge packets from eth0 to eth1, they do not 'pass' by
4868 iptables. This means that you cannot do filtering, or NAT or mangling or
4871 There are several projects going on to fix this, the truly right one is by
4872 the author of the Linux 2.4 bridging code, Lennert Buytenhek. He recently
4873 informed us that as of bridge-nf 0.0.2 (see the url above), the code is
4874 stable and usable in production environments. He is now asking the kernel
4875 people if and how the patch can be merged, stay tuned!
4877 <sect1>Bridging and shaping
4879 This does work as advertised. Be sure to figure out which side each
4880 interface is on, otherwise you might be shaping outbound traffic in your
4881 internal interface, which won't work. Use tcpdump if needed.
4883 <sect1>Pseudo-bridges with Proxy-ARP
4885 If you just want to implement a Pseudo-bridge, skip down a few sections
4886 to 'Implementing it', but it is wise to read a bit about how it works in
4889 A Pseudo-bridge works a bit differently. By default, a bridge passes packets
4890 unaltered from one interface to the other. It only looks at the hardware
4891 address of packets to determine what goes where. This in turn means that you
4892 can bridge traffic that Linux does not understand, as long as it has an
4893 hardware address it does.
4895 A 'Pseudo-bridge' works differently and looks more like a hidden router than
4896 a bridge, but like a bridge, it has little impact on network design.
4898 An advantage of the fact that it is not a brige lies in the fact that
4899 packets really pass through the kernel, and can be filtered, changed,
4900 redirected or rerouted.
4902 A real bridge can also be made to perform these feats, but it needs special
4903 code, like the Ethernet Frame Diverter, or the above mentioned patch.
4905 Another advantage of a pseudo-bridge is that it does not pass packets it
4906 does not understand - thus cleaning your network of a lot of cruft. In cases
4907 where you need this cruft (like SAP packets, or Netbeui), use a real bridge.
4908 <sect2>ARP & Proxy-ARP
4910 When a host wants to talk to another host on the same physical network
4911 segment, it sends out an Address Resolution Protocol packet, which, somewhat
4912 simplified, reads like this 'who has 10.0.0.1, tell 10.0.0.7'. In response
4913 to this, 10.0.0.1 replies with a short 'here' packet.
4915 10.0.0.7 then sends packets to the hardware address mentioned in the 'here'
4916 packet. It caches this hardware address for a relatively long time, and
4917 after the cache expires, it reasks the question.
4919 When building a Pseudo-bridge, we instruct the bridge to reply to these ARP
4920 packets, which causes the hosts in the network to send its packets to the
4921 bridge. The brige then processes these packets, and sends them to the
4924 So, in short, whenever a host on one side of the bridge asks for the
4925 hardware address of a host on the other, the bridge replies with a packet
4926 that says 'hand it to me'.
4928 This way, all data traffic gets transmitted to the right place, and always
4929 passes through the bridge.
4930 <sect2>Implementing it
4932 In the bad old days, it used to be possible to instruct the Linux Kernel to
4933 perform 'proxy-ARP' for just any subnet. So, to configure a pseudo-bridge,
4934 you would have to specify both the proper routes to both sides of the bridge
4935 AND create matching proxy-ARP rules. This is bad in that it requires a lot
4936 of typing, but also because it easily allows you to make mistakes which make
4937 your bridge respond to ARP queries for networks it does not know how to
4940 With Linux 2.4/2.5 (and possibly 2.2), this possibility has been withdrawn and
4941 has been replaced by a flag in the /proc directory, called 'proxy_arp'. The
4942 procedure for building a pseudo-bridge is then:
4945 <item>Assign an IP address to both interfaces, the 'left' and the 'right'
4947 <item>Create routes so your machine knows which hosts reside on the left,
4948 and which on the right
4949 <item>Turn on proxy-ARP on both interfaces, echo 1 >
4950 /proc/sys/net/ipv4/conf/ethL/proxy_arp, echo 1 >
4951 /proc/sys/net/ipv4/conf/ethR/proxy_arp, where L and R stand for the numbers
4952 of your interfaces on the left and on the right side
4955 Also, do not forget to turn on the ip_forwarding flag! When converting from
4956 a true bridge, you may find that this flag was turned off as it is not
4957 needed when bridging.
4959 Another thing you might note when converting is that you need to clear the
4960 arp cache of computers in the network - the arp cache might contain old
4961 pre-bridge hardware addresses which are no longer correct.
4963 On a Cisco, this is done using the command 'clear arp-cache', under
4964 Linux, use 'arp -d ip.address'. You can also wait for the cache to expire
4965 manually, which can take rather long.
4967 You may also discover that your network was misconfigured if you are/were of
4968 the habit of specifying routes without netmasks. To explain, some versions
4969 of route may have guessed your netmask right in the past, or guessed wrong
4970 without you noticing. When doing surgical routing like described above, it
4971 is *vital* that you check your netmasks!
4972 <sect>Dynamic routing - OSPF and BGP
4974 Once your network starts to get really big, or you start to consider 'the
4975 internet' as your network, you need tools which dynamically route your data.
4976 Sites are often connected to each other with multiple links, and more are
4977 popping up all the time.
4979 The Internet has mostly standardised on OSPF and BGP4 (rfc1771). Linux
4980 supports both, by way of <tt>gated</tt> and <tt>zebra</tt>
4982 While currently not within the scope of this document, we would like to
4983 point you to the definitive works:
4989 url="http://www.cisco.com/univercd/cc/td/doc/cisintwk/idg4/nd2003.htm
"
4990 name="Designing large-scale IP Internetworks
">
4997 "OSPF. The anatomy of an Internet routing protocol
"
4998 Addison Wesley. Reading, MA. 1998.
5000 Halabi has also written a good guide to OSPF routing design, but this
5001 appears to have been dropped from the Cisco web site.
5007 "Internet routing architectures
"
5008 Cisco Press (New Riders Publishing). Indianapolis, IN. 1997.
5015 url="http://www.cisco.com/univercd/cc/td/doc/cisintwk/ics/icsbgp4.htm
"
5016 name="Using the Border Gateway Protocol for interdomain routing
">
5019 Although the examples are Cisco-specific, they are remarkably similar
5020 to the configuration language in Zebra :-)
5021 <sect>Other possibilities
5023 This chapter is a list of projects having to do with advanced Linux routing
5024 & traffic shaping. Some of these links may deserve chapters of their
5025 own, some are documented very well of themselves, and don't need more HOWTO.
5028 <tag>802.1Q VLAN Implementation for Linux <url url="http://scry.wanfear.com/~greear/vlan.html
"
5029 name="(site)
"></tag>
5031 VLANs are a very cool way to segregate your
5032 networks in a more virtual than physical way. Good information on VLANs can
5034 url="ftp://ftp.netlab.ohio-state.edu/pub/jain/courses/cis788-
97/virtual_lans/index.htm
"
5035 name="here
">. With this implementation, you can have your Linux box talk
5036 VLANs with machines like Cisco Catalyst, 3Com: {Corebuilder, Netbuilder II,
5037 SuperStack II switch 630}, Extreme Ntwks Summit 48, Foundry: {ServerIronXL,
5040 A great HOWTO about VLANs can be found <url
5041 url="http://scry.wanfear.com/~greear/vlan/cisco_howto.html
" name="here
">.
5043 Update: has been included in the kernel as of 2.4.14 (perhaps 13).
5044 <tag>Alternate 802.1Q VLAN Implementation for Linux <url
5045 url="http://vlan.sourceforge.net
"
5046 name="(site)
"></tag>
5047 Alternative VLAN implementation for linux. This project was started out of
5048 disagreement with the 'established' VLAN project's architecture and coding
5049 style, resulting in a cleaner overall design.
5051 <tag>Linux Virtual Server <url url="http://www.LinuxVirtualServer.org/
"
5052 name="(site)
"></tag>
5054 These people are brilliant. The Linux Virtual Server is a highly scalable and
5055 highly available server built on a cluster of real servers, with the load
5056 balancer running on the Linux operating system. The architecture of the
5057 cluster is transparent to end users. End users only see a single virtual
5060 In short whatever you need to loadbalance, at whatever level of traffic, LVS
5061 will have a way of doing it. Some of their techniques are positively evil!
5062 For example, they let several machines have the same IP address on a
5063 segment, but turn off ARP on them. Only the LVS machine does ARP - it then
5064 decides which of the backend hosts should handle an incoming packet, and
5065 sends it directly to the right MAC address of the backend server. Outgoing
5066 traffic will flow directly to the router, and not via the LVS machine, which
5067 does therefor not need to see your 5Gbit/s of content flowing to the world,
5068 and cannot be a bottleneck.
5070 The LVS is implemented as a kernel patch in Linux 2.0 and 2.2, but as a
5071 Netfilter module in 2.4/2.5, so it does not need kernel patches! Their 2.4
5072 support is still in early development, so beat on it and give feedback or
5075 <tag>CBQ.init <url url="ftp://ftp.equinox.gu.net/pub/linux/cbq/
"
5076 name="(site)
"></tag>
5077 Configuring CBQ can be a bit daunting, especially if all you want to do is
5078 shape some computers behind a router. CBQ.init can help you configure Linux
5079 with a simplified syntax.
5081 For example, if you want all computers in your 192.168.1.0/24 subnet
5082 (on 10mbit eth1) to be limited to 28kbit/s download speed, put
5083 this in the CBQ.init configuration file:
5086 DEVICE=eth1,10Mbit,1Mbit
5093 By all means use this program if the 'how and why' don't interest you.
5094 We're using CBQ.init in production and it works very well. It can even do
5095 some more advanced things, like time dependent shaping. The documentation is
5096 embedded in the script, which explains why you can't find a README.
5098 <tag>Chronox easy shaping scripts <url url="http://www.chronox.de
"
5099 name="(site)
"></tag>
5101 Stephan Mueller (smueller@chronox.de) wrote two useful scripts, 'limit.conn'
5102 and 'shaper'. The first one allows you to easily throttle a single download
5106 # limit.conn -s SERVERIP -p SERVERPORT -l LIMIT
5109 It works on Linux 2.2 and 2.4/2.5.
5111 The second script is more complicated, and can be used to make lots of
5112 different queues based on iptables rules, which are used to mark packets
5113 which are then shaped.
5116 Redundancy Protocol implementation <url url="http://w3.arobas.net/~jetienne/vrrpd/index.html
"
5117 name="(site)
"></tag>
5119 This is purely for redundancy. Two machines with their own IP address and
5120 MAC Address together create a third IP Address and MAC Address, which is
5121 virtual. Originally intended purely for routers, which need constant MAC
5122 addresses, it also works for other servers.
5124 The beauty of this approach is the incredibly easy configuration. No kernel
5125 compiling or patching required, all userspace.
5127 Just run this on all machines participating in a service:
5129 # vrrpd -i eth0 -v 50 10.0.0.22
5132 And you are in business! 10.0.0.22 is now carried by one of your servers,
5133 probably the first one to run the vrrp daemon. Now disconnect that computer
5134 from the network and very rapidly one of the other computers will assume the
5135 10.0.0.22 address, as well as the MAC address.
5137 I tried this over here and had it up and running in 1 minute. For some
5138 strange reason it decided to drop my default gateway, but the -n flag
5141 This is a 'live' failover:
5144 64 bytes from 10.0.0.22: icmp_seq=3 ttl=255 time=0.2 ms
5145 64 bytes from 10.0.0.22: icmp_seq=4 ttl=255 time=0.2 ms
5146 64 bytes from 10.0.0.22: icmp_seq=5 ttl=255 time=16.8 ms
5147 64 bytes from 10.0.0.22: icmp_seq=6 ttl=255 time=1.8 ms
5148 64 bytes from 10.0.0.22: icmp_seq=7 ttl=255 time=1.7 ms
5151 Not *one* ping packet was lost! Just after packet 4, I disconnected my P200
5152 from the network, and my 486 took over, which you can see from the higher
5155 <sect>Further reading
5158 <tag><url url="http://snafu.freedom.org/linux2.2/iproute-notes.html
"
5159 name="http://snafu.freedom.org/linux2.2/iproute-notes.html
"></tag>
5160 Contains lots of technical information, comments from the kernel
5161 <tag><url url="http://www.davin.ottawa.on.ca/ols/
"
5162 name="http://www.davin.ottawa.on.ca/ols/
"></tag>
5163 Slides by Jamal Hadi Salim, one of the authors of Linux traffic control
5164 <tag><url url="http://defiant.coinet.com/iproute2/ip-cref/
"
5165 name="http://defiant.coinet.com/iproute2/ip-cref/
"></tag>
5166 HTML version of Alexeys LaTeX documentation - explains part of iproute2 in
5168 <tag><url url="http://www.aciri.org/floyd/cbq.html
"
5169 name="http://www.aciri.org/floyd/cbq.html
"></tag>
5170 Sally Floyd has a good page on CBQ, including her original papers. None of
5171 it is Linux specific, but it does a fair job discussing the theory and uses
5173 Very technical stuff, but good reading for those so inclined.
5175 <tag>Differentiated Services on Linux</tag>
5176 This <url url="ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-
01.txt.gz
"
5177 name="document
"> by Werner Almesberger, Jamal Hadi Salim and Alexey
5178 Kuznetsov describes DiffServ facilities in the Linux kernel, amongst which
5179 are TBF, GRED, the DSMARK qdisc and the tcindex classifyer.
5182 <tag><url url="http://ceti.pl/~kravietz/cbq/NET4_tc.html
"
5183 name="http://ceti.pl/~kravietz/cbq/NET4_tc.html
"></tag>
5184 Yet another HOWTO, this time in Polish! You can copy/paste command lines
5185 however, they work just the same in every language. The author is
5186 cooperating with us and may soon author sections of this HOWTO.
5189 url="http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/car.htm
"
5190 name="IOS Committed Access Rate
"></tag>
5192 From the helpful folks of Cisco who have the laudable habit of putting
5193 their documentation online. Cisco syntax is different but the concepts are
5194 the same, except that we can do more and do it without routers the price of
5197 <tag>Docum experimental site<url url="http://www.docum.org
"
5198 name="(site)
"></tag>
5199 Stef Coene is busy convincing his boss to sell Linux support, and so he is
5200 experimenting a lot, especially with managing bandwidth. His site has a lot
5201 of practical information, examples, tests and also points out some CBQ/tc bugs.
5203 <tag>TCP/IP Illustrated, volume 1, W. Richard Stevens, ISBN 0-201-63346-9</tag>
5204 Required reading if you truly want to understand TCP/IP. Entertaining as
5208 <sect>Acknowledgements
5210 It is our goal to list everybody who has contributed to this HOWTO, or
5211 helped us demystify how things work. While there are currently no plans
5212 for a Netfilter type scoreboard, we do like to recognise the people who are
5216 <item>Juanjo Alins <juanjo%mat.upc.es>
5218 <item>Michael T. Babcock <mbabcock@fibrespeed.net>
5219 <item>Ard van Breemen <ard%kwaak.net>
5220 <item>Ron Brinker <service%emcis.com>
5221 <item>?ukasz Bromirski <L.Bromirski@prosys.com.pl>
5222 <item>Lennert Buytenhek <buytenh@gnu.org>
5223 <item>Esteve Camps <esteve@hades.udg.es>
5224 <item>Stef Coene <stef.coene@docum.org>
5225 <item>Don Cohen <don-lartc%isis.cs3-inc.com>
5226 <item>Jonathan Corbet <lwn%lwn.net>
5227 <item>Gerry Creager N5JXS <gerry%cs.tamu.edu>
5228 <item>Marco Davids <marco@sara.nl>
5229 <item>Jonathan Day <jd9812@my-deja.com>
5230 <item>Martin Devera aka devik <devik@cdi.cz>
5231 <item>Stephan "Kobold
" Gehring <Stephan.Gehring@bechtle.de>
5232 <item>Jacek Glinkowski <jglinkow%hns.com>
5233 <item>Andrea Glorioso <sama%perchetopi.org>
5234 <item>Nadeem Hasan <nhasan@usa.net>
5235 <item>Erik Hensema <erik%hensema.xs4all.nl>
5236 <item>Vik Heyndrickx <vik.heyndrickx@edchq.com>
5237 <item>Spauldo Da Hippie <spauldo%usa.net>
5238 <item>Koos van den Hout <koos@kzdoos.xs4all.nl>
5239 <item>Stefan Huelbrock <shuelbrock%datasystems.de>
5240 <item>Alexander W. Janssen <yalla%ynfonatic.de>
5241 <item>Gareth John <gdjohn%zepler.org>
5242 <item>Martin Josefsson <gandalf%wlug.westbo.se>
5243 <item>Andi Kleen <ak%suse.de>
5244 <item>Andreas J. Koenig <andreas.koenig%anima.de>
5245 <item>Pawel Krawczyk <kravietz%alfa.ceti.pl>
5246 <item>Amit Kucheria <amitk@ittc.ku.edu>
5247 <item>Edmund Lau <edlau%ucf.ics.uci.edu>
5248 <item>Philippe Latu <philippe.latu%linux-france.org>
5249 <item>Arthur van Leeuwen <arthurvl%sci.kun.nl>
5250 <item>Jason Lunz <j@cc.gatech.edu>
5251 <item>Stuart Lynne <sl@fireplug.net>
5252 <item>Alexey Mahotkin <alexm@formulabez.ru>
5253 <item>Predrag Malicevic <pmalic@ieee.org>
5254 <item>Andreas Mohr <andi%lisas.de>
5255 <item>Andrew Morton <akpm@zip.com.au>
5256 <item>Wim van der Most
5257 <item>Stephan Mueller <smueller@chronox.de>
5258 <item>Togan Muftuoglu <toganm%yahoo.com>
5259 <item>Chris Murray <cmurray@stargate.ca>
5260 <item>Patrick Nagelschmidt <dto%gmx.net>
5261 <item>Ram Narula <ram@princess1.net>
5262 <item>Jorge Novo <jnovo@educanet.net>
5263 <item>Patrik <ph@kurd.nu>
5264 <item>Lutz Preßler <Lutz.Pressler%SerNet.DE>
5265 <item>Jason Pyeron <jason%pyeron.com>
5266 <item>Rusty Russell <rusty%rustcorp.com.au>
5267 <item>Mihai RUSU <dizzy%roedu.net>
5268 <item>Jamal Hadi Salim <hadi%cyberus.ca>
5269 <item>David Sauer <davids%penguin.cz>
5270 <item>Sheharyar Suleman Shaikh <sss23@drexel.edu>
5271 <item>Stewart Shields <MourningBlade%bigfoot.com>
5272 <item>Nick Silberstein <nhsilber%yahoo.com>
5273 <item>Konrads Smelkov <konrads@interbaltika.com>
5274 <item>Andreas Steinmetz <ast%domdv.de>
5275 <item>Jason Tackaberry <tack@linux.com>
5276 <item>Charles Tassell <ctassell%isn.net>
5277 <item>Glen Turner <glen.turner%aarnet.edu.au>
5278 <item>Tea Sponsor: Eric Veldhuyzen <eric%terra.nu>
5279 <item>Song Wang <wsong@ece.uci.edu>