2 .\" Copyright (c) 2009, Sun Microsystems, Inc. All Rights Reserved
3 .\" The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License.
4 .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License.
5 .\" When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner]
6 .TH FLOWADM 1M "Feb 14, 2009"
8 flowadm \- administer bandwidth resource control and priority for protocols,
9 services, containers, and virtual machines
13 \fBflowadm show-flow\fR [\fB-pP\fR] [\fB-S\fR] [\fB-s\fR [\fB-i\fR \fIinterval\fR]] [\fB-l\fR \fIlink\fR]
14 [\fB-o\fR \fIfield\fR[,...]] [\fIflow\fR]
19 \fBflowadm add-flow\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-l\fR \fIlink\fR \fB-a\fR \fIattr\fR=\fIvalue\fR[,...]
20 \fB-p\fR \fIprop\fR=\fIvalue\fR[,...] \fIflow\fR
21 \fBflowadm remove-flow\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] {\fB-l\fR \fIlink\fR | \fIflow\fR}
26 \fBflowadm set-flowprop\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-p\fR \fIprop\fR=\fIvalue\fR[,...] \fIflow\fR
27 \fBflowadm reset-flowprop\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] [\fB-p\fR \fIprop\fR[,...]] \fIflow\fR
28 \fBflowadm show-flowprop\fR [\fB-cP\fR] [\fB-l\fR \fIlink\fR] [\fB-o\fR \fIfield\fR[,...]]
29 [\fB-p\fR \fIprop\fR[,...]] [\fIflow\fR]
34 \fBflowadm show-usage\fR [\fB-a\fR] [\fB-d\fR | {\fB-p\fR \fIplotfile\fR \fB-F\fR \fIformat\fR}] [\fB-s\fR \fItime\fR]
35 [\fB-e\fR \fItime\fR] \fB-f\fR \fIfilename\fR [\fIflow\fR]
41 The \fBflowadm\fR command is used to create, modify, remove, and show
42 networking bandwidth and associated resources for a type of traffic on a
46 The \fBflowadm\fR command allows users to manage networking bandwidth resources
47 for a transport, service, or a subnet. The service is specified as a
48 combination of transport and local port. The subnet is specified by its IP
49 address and subnet mask. The command can be used on any type of data link,
50 including physical links, virtual NICs, and link aggregations.
53 A flow is defined as a set of attributes based on Layer 3 and Layer 4 headers,
54 which can be used to identify a protocol, service, or a virtual machine. When a
55 flow is identified based on flow attributes, separate kernel resources
56 including layer 2, 3, and 4 queues, their processing threads, and other
57 resources are uniquely created for it, such that other traffic has minimal or
61 Inbound and outbound packet are matched to flows in a very fast and scalable
62 way, so that limits can be enforced with minimal performance impact.
65 The \fBflowadm\fR command can be used to identify a flow without imposing any
66 bandwidth resource control. This would result in the traffic type getting its
67 own resources and queues so that it is isolated from rest of the networking
68 traffic for more observable and deterministic behavior.
71 \fBflowadm\fR is implemented as a set of subcommands with corresponding
72 options. Options are described in the context of each subcommand.
76 The following subcommands are supported:
80 \fB\fBflowadm show-flow\fR [\fB-pP\fR] [\fB-s\fR [\fB-i\fR \fIinterval\fR]]
81 [\fB-o\fR \fIfield\fR[,...]] [\fB-l\fR \fIlink\fR] [\fIflow\fR]\fR
85 Show flow configuration information (the default) or statistics, either for all
86 flows, all flows on a link, or for the specified \fIflow\fR.
90 \fB\fB-o\fR \fIfield\fR[,...]\fR
94 A case-insensitive, comma-separated list of output fields to display. The field
95 name must be one of the fields listed below, or a special value \fBall\fR, to
96 display all fields. For each flow found, the following fields can be displayed:
104 The name of the flow.
114 The name of the link the flow is on.
124 IP address of the flow. This can be either local or remote depending on how the
131 \fB\fBtransport\fR\fR
135 The name of the layer for protocol to be used.
145 Local port of service for flow.
155 Differentiated services value for flow and mask used with \fBDSFIELD\fR value
156 to state the bits of interest in the differentiated services field of the IP
165 \fB\fB-p\fR, \fB--parseable\fR\fR
169 Display using a stable machine-parseable format.
175 \fB\fB-P\fR, \fB--persistent\fR\fR
179 Display persistent flow property information.
185 \fB\fB-S\fR, \fB--continuous\fR\fR
189 Continuously display network utilization by flow in a manner similar to the way
190 that \fBprstat\fR(1M) displays CPU utilization by process.
196 \fB\fB-s\fR, \fB--statistics\fR\fR
200 Displays flow statistics.
206 \fB\fB-i\fR \fIinterval\fR, \fB--interval\fR=\fIinterval\fR\fR
210 Used with the \fB-s\fR option to specify an interval, in seconds, at which
211 statistics should be displayed. If this option is not specified, statistics are
218 \fB\fB-l\fR \fIlink\fR, \fB--link\fR=\fIlink\fR | \fIflow\fR\fR
222 Display information for all flows on the named link or information for the
231 \fB\fBflowadm add-flow\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-l\fR
232 \fIlink\fR \fB-a\fR \fIattr\fR=\fIvalue\fR[,...] \fB-p\fR
233 \fIprop\fR=\fIvalue\fR[,...] \fIflow\fR\fR
237 Adds a flow to the system. The flow is identified by its flow attributes and
240 As part of identifying a particular flow, its bandwidth resource can be limited
241 and its relative priority to other traffic can be specified. If no bandwidth
242 limit or priority is specified, the traffic still gets its unique layer 2, 3,
243 and 4 queues and processing threads, including NIC hardware resources (when
244 supported), so that the selected traffic can be separated from others and can
245 flow with minimal impact from other traffic.
249 \fB\fB-t\fR, \fB--temporary\fR\fR
253 The changes are temporary and will not persist across reboots. Persistence is
260 \fB\fB-R\fR \fIroot-dir\fR, \fB--root-dir\fR=\fIroot-dir\fR\fR
264 Specifies an alternate root directory where \fBflowadm\fR should apply
271 \fB\fB-l\fR \fIlink\fR, \fB--link\fR=\fIlink\fR\fR
275 Specify the link to which the flow will be added.
281 \fB\fB-a\fR \fIattr\fR=\fIvalue\fR[,...], \fB--attr\fR=\fIvalue\fR\fR
285 A comma-separated list of attributes to be set to the specified values.
291 \fB\fB-p\fR \fIprop\fR=\fIvalue\fR[,...], \fB--prop\fR=\fIvalue\fR[,...]\fR
295 A comma-separated list of properties to be set to the specified values.
303 \fB\fBflowadm remove-flow\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-l\fR
304 {\fIlink\fR | \fIflow\fR}\fR
308 Remove an existing flow identified by its link or name.
312 \fB\fB-t\fR, \fB--temporary\fR\fR
316 The changes are temporary and will not persist across reboots. Persistence is
323 \fB\fB-R\fR \fIroot-dir\fR, \fB--root-dir\fR=\fIroot-dir\fR\fR
327 Specifies an alternate root directory where \fBflowadm\fR should apply
334 \fB\fB-l\fR \fIlink\fR | \fIflow\fR, \fB--link\fR=\fIlink\fR | \fIflow\fR\fR
338 If a link is specified, remove all flows from that link. If a single flow is
339 specified, remove only that flow.
347 \fB\fBflowadm set-flowprop\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-p\fR
348 \fIprop\fR=\fIvalue\fR[,...] \fIflow\fR\fR
352 Set values of one or more properties on the flow specified by name. The
353 complete list of properties can be retrieved using the \fBshow-flow\fR
358 \fB\fB-t\fR, \fB--temporary\fR\fR
362 The changes are temporary and will not persist across reboots. Persistence is
369 \fB\fB-R\fR \fIroot-dir\fR, \fB--root-dir\fR=\fIroot-dir\fR\fR
373 Specifies an alternate root directory where \fBflowadm\fR should apply
374 persistent setting of properties.
380 \fB\fB-p\fR \fIprop\fR=\fIvalue\fR[,...], \fB--prop\fR=\fIvalue\fR[,...]\fR
384 A comma-separated list of properties to be set to the specified values.
392 \fB\fBflowadm reset-flowprop\fR [\fB-t\fR] [\fB-R\fR \fIroot-dir\fR] \fB-p\fR
393 [\fIprop\fR=\fIvalue\fR[,...]] \fIflow\fR\fR
397 Resets one or more properties to their default values on the specified flow. If
398 no properties are specified, all properties are reset. See the
399 \fBshow-flowprop\fR subcommand for a description of properties, which includes
400 their default values.
404 \fB\fB-t\fR, \fB--temporary\fR\fR
408 Specifies that the resets are temporary. Temporary resets last until the next
415 \fB\fB-R\fR \fIroot-dir\fR, \fB--root-dir\fR=\fIroot-dir\fR\fR
419 Specifies an alternate root directory where \fBflowadm\fR should apply
420 persistent setting of properties.
426 \fB\fB-p\fR \fIprop\fR=\fIvalue\fR[,...], \fB--prop\fR=\fIvalue\fR[,...]\fR
430 A comma-separated list of properties to be reset.
438 \fB\fBflowadm show-flowprop\fR [\fB-cP\fR] [\fB-l\fR \fIlink\fR] [\fB-p\fR
439 \fIprop\fR[,...]] [\fIflow\fR]\fR
443 Show the current or persistent values of one or more properties, either for all
444 flows, flows on a specified link, or for the specified flow.
446 By default, current values are shown. If no properties are specified, all
447 available flow properties are displayed. For each property, the following
448 fields are displayed:
456 The name of the flow.
466 The name of the property.
476 The current (or persistent) property value. The value is shown as \fB--\fR
477 (double hyphen), if it is not set, and \fB?\fR (question mark), if the value is
478 unknown. Persistent values that are not set or have been reset will be shown as
479 \fB--\fR and will use the system \fBDEFAULT\fR value (if any).
489 The default value of the property. If the property has no default value,
490 \fB--\fR (double hyphen), is shown.
500 A comma-separated list of the values the property can have. If the values span
501 a numeric range, the minimum and maximum values might be shown as shorthand. If
502 the possible values are unknown or unbounded, \fB--\fR (double hyphen), is
506 Flow properties are documented in the "Flow Properties" section, below.
510 \fB\fB-c\fR, \fB--parseable\fR\fR
514 Display using a stable machine-parseable format.
520 \fB\fB-P\fR, \fB--persistent\fR\fR
524 Display persistent flow property information.
530 \fB\fB-p\fR \fIprop\fR[,...], \fB--prop\fR=\fIprop\fR[,...]\fR
534 A comma-separated list of properties to show.
542 \fB\fBflowadm show-usage\fR [\fB-a\fR] [\fB-d\fR | {\fB-p\fR \fIplotfile\fR
543 \fB-F\fR \fIformat\fR}] [\fB-s\fR \fItime\fR] [\fB-e\fR \fItime\fR]
548 Show the historical network flow usage from a stored extended accounting file.
549 Configuration and enabling of network accounting through \fBacctadm\fR(1M) is
550 required. The default output will be the summary of flow usage for the entire
551 period of time in which extended accounting was enabled.
559 Display all historical network usage for the specified period of time during
560 which extended accounting is enabled. This includes the usage information for
561 the flows that have already been deleted.
571 Display the dates for which there is logging information. The date is in the
572 format \fIDD\fR/\fIMM\fR/\fIYYYY\fR.
578 \fB\fB-F\fR \fIformat\fR\fR
582 Specifies the format of \fIplotfile\fR that is specified by the \fB-p\fR
583 option. As of this release, \fBgnuplot\fR is the only supported format.
589 \fB\fB-p\fR \fIplotfile\fR\fR
593 When specified with \fB-s\fR or \fB-e\fR (or both), outputs flow usage data to
594 a file of the format specified by the \fB-F\fR option, which is required.
600 \fB\fB-s\fR \fItime\fR, \fB-e\fR \fItime\fR\fR
604 Start and stop times for data display. Time is in the format
605 \fIYYYY\fR.\fIMM\fR.\fIDD\fR,\fIhh\fR:\fImm\fR:\fIss\fR.
611 \fB\fB-f\fR \fIfilename\fR\fR
615 Read extended accounting records of network flow usage from \fIfilename\fR.
625 If specified, display the network flow usage only from the named flow.
626 Otherwise, display network usage from all flows.
631 .SS "Flow Attributes"
634 The flow operand that identify a flow in a \fBflowadm\fR command is a
635 comma-separated list of one or more keyword, value pairs from the list below.
639 \fB\fBlocal_ip\fR[\fB/\fR\fIprefix_len\fR]\fR
643 Identifies a network flow by the local IP address. \fIvalue\fR must be a IPv4
644 address in dotted-decimal notation or an IPv6 address in colon-separated
645 notation. \fIprefix_len\fR is optional.
647 If \fIprefix_len\fR is specified, it describes the netmask for a subnet
648 address, following the same notation convention of \fBifconfig\fR(1M) and
649 \fBroute\fR(1M) addresses. If unspecified, the given IP address will be
650 considered as a host address for which the default prefix length for a IPv4
651 address is \fB/32\fR and for IPv6 is \fB/128\fR.
657 \fB\fBremote_ip\fR[\fB/\fR\fIprefix_len\fR]\fR
661 Identifies a network flow by the remote IP address. The syntax is the same as
662 \fBlocal_ip\fR attributes
668 \fB\fBtransport\fR={\fBtcp\fR|\fBudp\fR|\fBsctp\fR|\fBicmp\fR|\fBicmpv6\fR}\fR
672 Identifies a layer 4 protocol to be used. It is typically used in combination
673 with local_port to identify the service that needs special attention.
679 \fB\fBlocal_port\fR\fR
683 Identifies a service specified by the local port.
689 \fB\fBdsfield\fR[\fB:\fR\fIdsfield_mask\fR]\fR
693 Identifies the 8-bit differentiated services field (as defined in RFC 2474).
695 The optional \fIdsfield_mask\fR is used to state the bits of interest in the
696 differentiated services field when comparing with the \fBdsfield\fR value. A
697 \fB0\fR in a bit position indicates that the bit value needs to be ignored and
698 a \fB1\fR indicates otherwise. The mask can range from \fB0x01\fR to
699 \fB0xff\fR. If \fIdsfield_mask\fR is not specified, the default mask \fB0xff\fR
700 is used. Both the \fBdsfield\fR value and mask must be in hexadecimal.
705 The following five types of combinations of attributes are supported:
709 local_ip[/\fIprefixlen\fR]=\fIaddress\fR
710 remote_ip[/\fIprefixlen\fR]=\fIaddress\fR
711 transport={tcp|udp|sctp|icmp|icmpv6}
712 transport={tcp|udp|sctp},local_port=\fIport\fR
713 dsfield=\fIval\fR[:\fIdsfield_mask\fR]
720 On a given link, the combinations above are mutually exclusive. An attempt to
721 create flows of different combinations will fail.
725 There are individual flow restrictions and flow restrictions per zone.
726 .SS "Individual Flow Restrictions"
729 Restrictions on individual flows do not require knowledge of other flows that
730 have been added to the link.
733 An attribute can be listed only once for each flow. For example, the following
734 command is not valid:
738 # \fBflowadm add-flow -l vnic1 -a local_port=80,local_port=8080 httpflow\fR
745 \fBtransport\fR and \fBlocal_port\fR:
748 TCP, UDP, or SCTP flows can be specified with a local port. An ICMP or ICMPv6
749 flow that specifies a port is not allowed. The following commands are valid:
753 # \fBflowadm add-flow -l e1000g0 -a transport=udp udpflow\fR
754 # \fBflowadm add-flow -l e1000g0 -a transport=tcp,local_port=80 \e
762 The following commands are not valid:
766 # \fBflowadm add-flow -l e1000g0 -a local_port=25 flow25\fR
767 # \fBflowadm add-flow -l e1000g0 -a transport=icmpv6,local_port=16 \e
773 .SS "Flow Restrictions Per Zone"
776 Within a zone, no two flows can have the same name. After adding a flow with
777 the link specified, the link will not be required for display, modification, or
778 deletion of the flow.
779 .SS "Flow Properties"
782 The following flow properties are supported. Note that the ability to set a
783 given property to a given value depends on the driver and hardware.
791 Sets the full duplex bandwidth for the flow. The bandwidth is specified as an
792 integer with one of the scale suffixes(\fBK\fR, \fBM\fR, or \fBG\fR for Kbps,
793 Mbps, and Gbps). If no units are specified, the input value will be read as
794 Mbps. The default is no bandwidth limit.
804 Sets the relative priority for the flow. The value can be given as one of the
805 tokens \fBhigh\fR, \fBmedium\fR, or \fBlow\fR. The default is \fBmedium\fR.
810 \fBExample 1 \fRCreating a Policy Around a Mission-Critical Port
813 The command below creates a policy around inbound HTTPS traffic on an HTTPS
814 server so that HTTPS obtains dedicated NIC hardware and kernel TCP/IP
815 resources. The name specified, \fBhttps-1\fR, can be used later to modify or
821 # \fBflowadm add-flow -l bge0 -a transport=TCP,local_port=443 https-1\fR
822 # \fBflowadm show-flow -l bge0\fR
823 FLOW LINK IP ADDR PROTO PORT DSFLD
824 https1 bge0 -- tcp 443 --
830 \fBExample 2 \fRModifying an Existing Policy to Add Bandwidth Resource Control
833 The following command modifies the \fBhttps-1\fR policy from the preceding
834 example. The command adds bandwidth control and give the policy a high
840 # \fBflowadm set-flowprop -p maxbw=500M,priority=high https-1\fR
841 # \fBflowadm show-flow https-1\fR
842 FLOW LINK IP ADDR PROTO PORT DSFLD
843 https1 bge0 -- tcp 443 --
845 # \fBflowadm show-flowprop https-1\fR
846 FLOW PROPERTY VALUE DEFAULT POSSIBLE
847 https-1 maxbw 500 -- --
848 https-1 priority HIGH -- LOW,NORMAL,HIGH
854 \fBExample 3 \fRLimiting the UDP Bandwidth Usage
857 The following command creates a policy for UDP protocol so that it cannot
858 consume more than 100Mbps of available bandwidth. The flow is named
864 # \fBflowadm add-flow -l bge0 -a transport=UDP -p maxbw=100M, \e
865 priority=low limit-udp-1\fR
871 \fBExample 4 \fRShowing Flow Usage
874 Flow usage statistics can be stored using the extended accounting facility,
880 # \fBacctadm -e extended -f /var/log/net.log net\fR
883 Network accounting: active
884 Network accounting file: /var/log/net.log
885 Tracked Network resources: extended
886 Untracked Network resources: none
893 The historical data that was saved can be retrieved in summary form using the
894 \fBshow-usage\fR subcommand of \fBflowadm\fR.
897 \fBExample 5 \fRSetting Policy, Making Use of \fBdsfield\fR Attribute
900 The following command sets a policy for EF PHB (DSCP value of 101110 from RFC
901 2598) with a bandwidth of 500 Mbps and a high priority. The \fBdsfield\fR value
902 for this flow will be \fB0x2e\fR (101110) with the \fBdsfield_mask\fR being
903 \fB0xfc\fR (because we want to ignore the 2 least significant bits).
908 # \fBflowadm add-flow -l bge0 -a dsfield=0x2e:0xfc \e
909 -p maxbw=500M,priority=high efphb-flow\fR
916 Display summary information:
921 # \fBflowadm show-usage -f /var/log/net.log\fR
922 FLOW DURATION IPACKETS RBYTES OPACKETS OBYTES BANDWIDTH
923 flowtcp 100 1031 546908 0 0 43.76 Kbps
924 flowudp 0 0 0 0 0 0.00 Mbps
931 Display dates for which logging information is available:
936 # \fBflowadm show-usage -d -f /var/log/net.log\fR
944 Display logging information for \fBflowtcp\fR starting at 02/19/2008, 10:38:46
945 and ending at 02/19/2008, 10:40:06:
950 # \fBflowadm show-usage -s 02/19/2008,10:39:06 -e 02/19/2008,10:40:06 \e
951 -f /var/log/net.log flowtcp\fR
952 FLOW TIME IPACKETS RBYTES OPACKETS OBYTES BANDWIDTH
953 flowtcp 10:39:06 1 1546 4 6539 3.23 Kbps
954 flowtcp 10:39:26 2 3586 5 9922 5.40 Kbps
955 flowtcp 10:39:46 1 240 1 216 182.40 bps
956 flowtcp 10:40:06 0 0 0 0 0.00 bps
963 Output the same information as above as a plotfile:
968 # \fBflowadm show-usage -s 02/19/2008,10:39:06 -e 02/19/2008,10:40:06 \e
969 -p /home/plot/myplot -F gnuplot -f /var/log/net.log flowtcp\fR
970 # \fBTime tcp-flow\fR
987 All actions were performed successfully.
1003 See \fBattributes\fR(5) for descriptions of the following attributes:
1011 ATTRIBUTE TYPE ATTRIBUTE VALUE
1013 Interface Stability Committed
1019 \fBacctadm\fR(1M), \fBdladm\fR(1M), \fBifconfig\fR(1M), \fBprstat\fR(1M),
1020 \fBroute\fR(1M), \fBattributes\fR(5), \fBdlpi\fR(7P)