Support per-flow queueing in dummynet.

Implement masks on UDP/TCP ports.
Large rewrite of the manpage.

Work supported by Akamba Corp.

2 files changed, 531 insertions, 296 deletions

diff --git a/sbin/ipfw/ipfw.8 b/sbin/ipfw/ipfw.8
index ffe431f..7057f39 100644
--- a/sbin/ipfw/ipfw.8
+++ b/sbin/ipfw/ipfw.8
@@ -6,7 +6,7 @@
 .Os FreeBSD
 .Sh NAME
 .Nm ipfw
-.Nd controlling utility for IP firewall
+.Nd controlling utility for IP firewall and traffic shaper
 .Sh SYNOPSIS
 .Nm ipfw
 .Op Fl q
@@ -27,26 +27,11 @@ flush
 .Oo
 .Fl q
 .Oc
-zero
+{zero|resetlog|delete}
 .Op Ar number ...
 .Nm ipfw
-.Oo
-.Fl q
-.Oc
-resetlog
-.Op Ar number ...
-.Nm ipfw
-delete
-.Ar number ...
-.Nm ipfw
 .Op Fl aftN
-list
-.Op Ar number ...
-.Nm ipfw
-.Oo
-.Fl ftN
-.Oc
-show
+{list|show}
 .Op Ar number ...
 .Nm ipfw
 .Oo
@@ -54,18 +39,113 @@ show
 .Oc
 add
 .Op Ar number
-.Op prob match_probability
-.Ar action 
-.Op log Op Ar logamount Ar number
-.Ar proto
-from
-.Ar src
-to
-.Ar dst
-.Op via Ar name | ipno
-.Op Ar options
+.Ar rule-body 
+.Nm ipfw
+pipe
+.Ar number
+config
+.Ar pipe-config-options
+.Nm ipfw
+pipe {delete|list|show}
+.Op Ar number ...
 .Sh DESCRIPTION
-If used as shown in the first synopsis line, the
+.Nm
+is the user interface for controlling the IPFW firewall and
+.Nm dummynet
+traffic shaper in FreeBSD.
+.Pp
+Each incoming or outgoing packet is passed through the
+.Nm
+rules. In case a host is acting as a gateway, packets
+forwarded by the gateway are processed by
+.Nm
+twice. In case a host is acting as a bridge, packets
+forwarded by the bridge are processed by
+.Nm
+once.
+.Pp
+A firewall configuration is made of a list of numbered rules, which are
+scanned for each packet until a match is
+found and the relevant action is performed. Depending on the
+action and certain system settings, packets can be reinjected
+into the firewall at the rule after the matching one for further
+processing. All rules apply to all interfaces, so it is
+responsibility of the sysadmin to write the ruleset in such
+a way to minimize the number of checks.
+.Pp
+A configuration always includes a
+.Ar DEFAULT
+rule (numbered 65535) which cannot be modified by the programmer
+and always matches packets. The action associated with the
+default rule can be either
+.Ar deny
+or
+.Ar allow
+depending on how the kernel is configured.
+.Pp
+All rules have a few associated counters: a packet count and
+a byte count, a log count, and a timestamp indicating the time
+of the last match. Counters can be visualized or reset with
+.Nm
+commands.
+.Pp
+Rules can be added with the
+.Ar add
+command; deleted individually with the
+.Ar delete
+command, and globally with the
+.Ar flush
+command; visualized, optionally with the content of
+the counters, using the
+.Ar show
+and
+.Ar list
+commands. Finally, counters can be reset with the
+.Ar zero
+and
+.Ar resetlog
+commands.
+.Pp
+The following options are available:
+.Bl -tag -width indent
+.It Fl a
+While listing, show counter values.  See also 
+.Dq show
+command.
+.It Fl f
+Don't ask for confirmation for commands that can cause problems if misused
+(i.e. flush).
+.Ar Note ,
+if there is no tty associated with the process, this is implied.
+.It Fl q
+While adding, zeroing, resetlogging or flushing, be quiet about actions (implies
+.Fl f Ns ).
+This is useful for adjusting rules by executing multiple
+.Nm
+commands in a script
+.Po
+e.g.,
+.Sq sh /etc/rc.firewall
+.Pc ,
+or by processing a file of many
+.Nm 
+rules,
+across a remote login session.  If a flush is performed in normal
+(verbose) mode (with the default kernel configuration), it prints a message.
+Because all rules are flushed, the
+message cannot be delivered to the login session.  This causes the
+remote login session to be closed and the remainder of the ruleset is
+not processed.  Access to the console is required to recover.
+.It Fl t
+While listing, show last match timestamp.
+.It Fl N
+Try to resolve addresses and service names in output.
+.El
+.Pp
+To ease configuration, rules can be put into a file which is processed
+using
+.Nm
+as shown in the first synopsis line. The
 .Ar file
 will be read line by line and applied as arguments to the 
 .Nm
@@ -97,219 +177,57 @@ This allows for flexible configuration files (like conditionalizing
 them on the local hostname) and the use of macros to centralize
 frequently required arguments like IP addresses.
 .Pp
-Each packet that has been received or is about to be sent goes through
-the
+The
 .Nm
-rules. In the case of a host acting as a gateway, packets that are
-forwarded by the host are processed by
+.Ar pipe
+commands are used to configure the traffic shaper, as shown in
+the ``TRAFFIC SHAPER CONFIGURATION'' section below.
+.Pp
+.Sh RULE FORMAT
+The
 .Nm
-twice
-.Po
-once when entering, and once when leaving
-.Pc .
+rule format is the following
+.Pp
+.Op prob Ar match_probability
+.Ar action
+.Op log Op Ar logamount Ar number
+.Ar proto
+from
+.Ar src
+to
+.Ar dst
+.Op interface-spec
+.Op Ar options
+.Pp
 Each packet can be filtered based on the following information that is
 associated with it:
 .Pp
 .Bl -tag -offset indent -compact -width xxxx
-.It Receive Interface Pq Ar recv
-Interface over which the packet was received
-.It Transmit Interface Pq Ar xmit
-Interface over which the packet would be transmitted
-.It Incoming Pq Ar in
-Packet was just received
-.It Outgoing Pq Ar out
-Packet would be transmitted
-.It Source IP Address
-Sender's IP Address
-.It Destination IP Address
-Target's IP Address
-.It Protocol
-IP protocol, including but not limited to IP
-.Pq Ar ip ,
-UDP
-.Pq Ar udp ,
-TCP
-.Pq Ar tcp ,
-or
-ICMP
-.Pq Ar icmp
-.It Source Port
-Sender's UDP or TCP port
-.It Destination Port
-Target's UDP or TCP port
-.It Connection Setup Flag Pq Ar setup
-This packet is a request to setup a TCP connection
-.It Connection Established Flag Pq Ar established
-This packet is part of an established TCP connection
-.It All TCP Flags Pq Ar tcpflags
-One or more of the TCP flags: close connection
-.Pq Ar fin ,
-open connection
-.Pq Ar syn ,
-reset connection
-.Pq Ar rst ,
-push
-.Pq Ar psh ,
-acknowledgment
-.Pq Ar ack ,
-and
-urgent
-.Pq Ar urg
-.It Fragment Flag Pq Ar frag
-This packet is a fragment of an IP packet
-.It IP Options Pq Ar ipoptions
-One or more of the IP options: strict source route
-.Pq Ar ssrr ,
-loose source route
-.Pq Ar lsrr ,
-record route
-.Pq Ar rr ,
-and timestamp
-.Pq Ar ts
-.It ICMP Types Pq Ar icmptypes
-One or more of the ICMP types: echo reply
-.Pq Ar 0 ,
-destination unreachable
-.Pq Ar 3 ,
-source quench
-.Pq Ar 4 ,
-redirect
-.Pq Ar 5 ,
-echo request
-.Pq Ar 8 ,
-router advertisement
-.Pq Ar 9 ,
-router solicitation
-.Pq Ar 10 ,
-time-to-live exceeded
-.Pq Ar 11 ,
-IP header bad
-.Pq Ar 12 ,
-timestamp request
-.Pq Ar 13 ,
-timestamp reply
-.Pq Ar 14 ,
-information request
-.Pq Ar 15 ,
-information reply
-.Pq Ar 16 ,
-address mask request
-.Pq Ar 17 ,
-and address mask reply
-.Pq Ar 18
+.It Transmit and Receive Interface (by name or address)
+.It Direction (Incoming or Outgoing)
+.It Source and Destination IP Address (possibly masked)
+.It Protocol (TCP, UDP, ICMP, etc.)
+.It Source and Destination Port (lists, ranges or masks)
+.It TCP Flags
+.It IP Fragment Flag
+.It IP Options
+.It ICMP Types
+.It User/Group ID of the socket associated with the packet
 .El
 .Pp
 Note that may be dangerous to filter on the source IP address or
 source TCP/UDP port because either or both could easily be spoofed.
 .Pp
-The
-.Nm
-code works by going through the rule-list for each packet
-until a match is found.
-All rules have two associated counters, a packet count and
-a byte count.
-These counters are updated when a packet matches the rule.
-.Pp
-If a rule has the optional
-.Dq prob match_probability
-specifier, where the match_probability is a floating point number
-between 0 and 1, a match is only declared with the specified
-probability. This can be useful for a number of applications
+.Ar prob match_probability
+.Bd -ragged -offset flag
+A match is only declared with the specified
+probability (floating point number between 0 and 1). This can be useful for a number of applications
 such as random packet drop or (in conjunction with
 .Xr dummynet 4
 ) to simulate the effect of multiple paths leading to out-of-order
 packet delivery.
-.Pp
-The rules are ordered by a 
-.Dq line-number
-from 1 to 65534 that is used
-to order and delete rules. Rules are tried in increasing order, and the
-first rule that matches a packet applies.
-Multiple rules may share the same number and apply in
-the order in which they were added.
-.Pp
-If a rule is added without a number, it is numbered 100 higher than the highest
-defined rule number, unless the highest defined rule number is 65435 or
-greater, in which case new rules are given that same number.
-.Pp
-The delete operation deletes the first rule with number
-.Ar number ,
-if any.
-.Pp
-The list command prints out the current rule set.
-.Pp
-The show command is equivalent to 
-.Sq ipfw -a list .
-.Pp
-The zero operation zeroes the counters associated with rule number
-.Ar number .
-.Pp
-The resetlog operation resets the logging counters associated with
-rule number
-.Ar number .
-.Pp
-The flush operation removes all rules.
-.Pp
-Any command beginning with a 
-.Sq # ,
-or being all blank, is ignored.
-.Pp
-One rule is always present:
-.Bd -literal -offset center
-65535 deny all from any to any
 .Ed
 .Pp
-This rule is the default policy, i.e., don't allow anything at all.
-Your job in setting up rules is to modify this policy to match your
-needs.
-.Pp
-However, if the kernel option
-.Dq IPFIREWALL_DEFAULT_TO_ACCEPT
-is active, the rule is instead:
-.Bd -literal -offset center
-65535 allow all from any to any
-.Ed
-.Pp
-This variation lets everything pass through.  This option should only be
-activated in particular circumstances, such as if you use the firewall
-system as an on-demand denial-of-service filter that is normally wide open.
-.Pp
-The following options are available:
-.Bl -tag -width indent
-.It Fl a
-While listing, show counter values.  See also 
-.Dq show
-command.
-.It Fl f
-Don't ask for confirmation for commands that can cause problems if misused
-(i.e. flush).
-.Ar Note ,
-if there is no tty associated with the process, this is implied.
-.It Fl q
-While adding, zeroing, resetlogging or flushing, be quiet about actions (implies
-.Fl f Ns ).
-This is useful for adjusting rules by executing multiple
-.Nm
-commands in a script
-.Po
-e.g.,
-.Sq sh /etc/rc.firewall
-.Pc ,
-or by processing a file of many
-.Nm 
-rules,
-across a remote login session.  If a flush is performed in normal
-(verbose) mode (with the default kernel configuration), it prints a message.
-Because all rules are flushed, the
-message cannot be delivered to the login session.  This causes the
-remote login session to be closed and the remainder of the ruleset is
-not processed.  Access to the console is required to recover.
-.It Fl t
-While listing, show last match timestamp.
-.It Fl N
-Try to resolve addresses and service names in output.
-.El
-.Pp
 .Ar action :
 .Bl -hang -offset flag -width 1234567890123456
 .It Ar allow
@@ -415,27 +333,16 @@ Skip all subsequent rules numbered less than
 The search continues with the first rule numbered
 .Ar number
 or higher.
-.It Ar uid user
-Match all TCP or UDP packets sent by or received for a
-.Ar user .
-A
-.Ar user
-may be matched by name or identification number.
-.It Ar gid group
-Match all TCP or UDP packets sent by or received for a
-.Ar group .
-A
-.Ar group
-may be matched by name or identification number.
 .El
 .Pp
+.Ar log Op Ar logamount Ar number
+.Bd -ragged -offset flag
 If the kernel was compiled with
 .Dv IPFIREWALL_VERBOSE ,
 then when a packet matches a rule with the
 .Ar log
-keyword or a clear/resetlog is performed, a message will be logged to
-.Xr syslogd 8 ,
-or, if that fails, to the console.  If the kernel was compiled with the
+keyword a message will be printed on the console.
+If the kernel was compiled with the
 .Dv IPFIREWALL_VERBOSE_LIMIT
 option, then by default logging will cease after the number
 of packets specified by the option are received for that
@@ -445,47 +352,39 @@ is used, that
 .Ar number
 will be the default logging limit rather than
 .Dv IPFIREWALL_VERBOSE_LIMIT .
-When this limit is reached, the limit and rule number will be logged.
 Logging may then be re-enabled by clearing the logging counter
 or the packet counter for that entry.
 .Pp
-The
-.Xr syslogd 8
-logging and the default log limit are adjustable dynamically through the
+Console logging and the log limit are adjustable dynamically
+through the
 .Xr sysctl 8
 interface in the MIB base of
 .Dv net.inet.ip.fw .
+.Ed
 .Pp
 .Ar proto :
-.Bl -hang -offset flag -width 1234567890123456
-.It Ar ip
-All packets match. The alias
-.Ar all
-has the same effect.
-.It Ar tcp
-Only TCP packets match.
-.It Ar udp
-Only UDP packets match.
-.It Ar icmp
-Only ICMP packets match.
-.It Ar <number|name>
-Only packets for the specified protocol matches (see
+.Bd -ragged -offset flag
+An IP protocol specified by number or name (see
 .Pa /etc/protocols
 for a complete list).
-.El
+The
+.Ar ip
+or
+.Ar all
+keywords mean any protocol will match.
+.Ed
 .Pp
 .Ar src 
 and
 .Ar dst :
-.Bl -hang -offset flag
-.It Ar <address/mask>
-.Op Ar ports
-.El
+.Bd -ragged -offset flag
+.Ar <address/mask> Op Ar ports
 .Pp
 The
 .Em <address/mask>
 may be specified as:
-.Bl -hang -offset flag -width 1234567890123456
+.Pp
+.Bl -hang -offset 0n -width 1234567890123456
 .It Ar ipno
 An ipnumber of the form 1.2.3.4.
 Only this exact ip number match the rule.
@@ -493,7 +392,7 @@ Only this exact ip number match the rule.
 An ipnumber with a mask width of the form 1.2.3.4/24.
 In this case all ip numbers from 1.2.3.0 to 1.2.3.255 will match.
 .It Ar ipno:mask
-An ipnumber with a mask width of the form 1.2.3.4:255.255.240.0.
+An ipnumber with a mask of the form 1.2.3.4:255.255.240.0.
 In this case all ip numbers from 1.2.0.0 to 1.2.15.255 will match.
 .El
 .Pp
@@ -507,9 +406,19 @@ With the TCP and UDP protocols, optional
 may be specified as:
 .Pp
 .Bl -hang -offset flag
-.It Ns {port|port-port} Ns Op ,port Ns Op ,...
+.It Ns {port|port-port|port:mask} Ns Op ,port Ns Op ,...
 .El
 .Pp
+The
+.Ql -
+notation specifies a range of ports (including boundaries).
+.Pp
+The
+.Ql \:
+notation specifies a port and a mask, a match is declared if
+the port number in the packet matches the one in the rule,
+limited to the bits which are set in the mask.
+.Pp
 Service names (from 
 .Pa /etc/services )
 may be used instead of numeric port values.
@@ -533,18 +442,16 @@ specifications.  See the
 .Ar frag
 option for details on matching fragmented packets.
 .Pp
-Rules can apply to packets when they are incoming, or outgoing, or both.
-The
-.Ar in
-keyword indicates the rule should only match incoming packets.
-The
-.Ar out
-keyword indicates the rule should only match outgoing packets.
-.Pp
-To match packets going through a certain interface, specify
-the interface using
-.Ar via :
-.Bl -hang -offset flag -width 1234567890123456
+.Ed
+.Ar interface-spec :
+.Pp
+.Bd -ragged -offset flag
+Some combinations of the following specifiers are allowed:
+.Bl -hang -offset 0n -width 1234567890123456
+.It Ar in
+Only match incoming packets.
+.It Ar out
+Only match outgoing packets.
 .It Ar via ifX
 Packet must be going through interface
 .Ar ifX.
@@ -595,10 +502,10 @@ or
 is invalid.
 .Pp
 A packet may not have a receive or transmit interface: packets originating
-from the local host have no receive interface. while packets destined for
+from the local host have no receive interface, while packets destined for
 the local host have no transmit interface.
+.Ed
 .Pp
-Additional
 .Ar options :
 .Bl -hang -offset flag -width 1234567890123456
 .It frag
@@ -608,15 +515,12 @@ of the datagram.
 may not be used in conjunction with either
 .Ar tcpflags
 or TCP/UDP port specifications.
-.It in
-Match if this packet was on the way in.
-.It out
-Match if this packet was on the way out.
 .It ipoptions Ar spec
 Match if the IP header contains the comma separated list of 
 options specified in
 .Ar spec .
 The supported IP options are:
+.Pp
 .Ar ssrr 
 (strict source route),
 .Ar lsrr 
@@ -639,6 +543,7 @@ Match if the TCP header contains the comma separated list of
 flags specified in
 .Ar spec .
 The supported TCP flags are:
+.Pp
 .Ar fin ,
 .Ar syn ,
 .Ar rst ,
@@ -660,6 +565,116 @@ Match if the ICMP type is in the list
 .Ar types .
 The list may be specified as any combination of ranges
 or individual types separated by commas.
+The supported ICMP types are:
+.Pp
+echo reply
+.Pq Ar 0 ,
+destination unreachable
+.Pq Ar 3 ,
+source quench
+.Pq Ar 4 ,
+redirect
+.Pq Ar 5 ,
+echo request
+.Pq Ar 8 ,
+router advertisement
+.Pq Ar 9 ,
+router solicitation
+.Pq Ar 10 ,
+time-to-live exceeded
+.Pq Ar 11 ,
+IP header bad
+.Pq Ar 12 ,
+timestamp request
+.Pq Ar 13 ,timestamp reply
+.Pq Ar 14 ,
+information request
+.Pq Ar 15 ,
+information reply
+.Pq Ar 16 ,
+address mask request
+.Pq Ar 17 ,
+and address mask reply
+.Pq Ar 18
+.It Ar uid user
+Match all TCP or UDP packets sent by or received for a
+.Ar user .
+A
+.Ar user
+may be matched by name or identification number.
+.It Ar gid group
+Match all TCP or UDP packets sent by or received for a
+.Ar group .
+A
+.Ar group
+may be matched by name or identification number.
+.El
+.Sh TRAFFIC SHAPER CONFIGURATION
+Ipfw is also the user interface for the
+.Xr dummynet 4
+traffic shaper.
+The shaper operates by passing packets to objects called
+.Ar pipes ,
+which emulates a link with given bandwidth, propagation delay,
+queue size and packet loss rate.
+The
+.Nm
+pipe configuration format is the following
+.Pp
+.Ar pipe number config
+.Op bw Ar bandwidth
+.Op queue Ar {slots|size}
+.Op delay Ar delay-ms
+.Op plr Ar loss-probability
+.Op mask Ar {all | {dst-ip|src-ip|dst-port|src-port|proto} bitmask}
+.Op buckets Ar hash-table-size
+.Pp
+The following parameters can be configured for a pipe:
+.Bl -hang -offset flag -width 1234567890
+.It bw Ar bandwidth
+Bandwidth, measured in
+.Ar [K|M]{bit/s|Byte/s} .
+A value of 0 (default) means unlimited bandwidth.
+The unit must follow immediately the number, as in
+.Dl "ipfw pipe 1 config bw 300Kbit/s queue 50KBytes"
+.It delay Ar ms-delay
+propagation delay, measured in milliseconds. The value is rounded
+to the next multiple of the clock tick (typically 10ms, but it is
+good practice to run kernels with "options HZ=1000" to reduce
+the granularity to 1ms or less). Default value is 0, meaning
+no delay.
+.It queue Ar {slots|size}
+queue size, in slots or KBytes. Default value is 50 slots, which
+is the typical queue size for Ethernet devices. Note that for
+slow speed links you should keep the queue size short or your
+traffic might be affected by a significant queueing delay. E.g.
+50  max-sized ethernet packets (1500
+bytes) mean 600Kbit or 20s of queue on a 30Kbit/s pipe.
+Even worse effect can result if you get
+packets from an interface with a much larger MTU e.g. the loopback
+interface with its 16KB packets.
+.It plr packet-loss-rate
+packet loss rate. NN is a floating-point number, with 0 meaning
+no loss, 1 means 100% loss. The loss rate is internally represented
+on 31 bits.
+.It mask Ar mask-specifier
+dummynet allows you to generate per-flow queues
+using a single pipe specification. A flow identifier is constructed
+by masking the IP addresses, ports and protocol types as specified
+in the pipe configuration. Packets with the same ID after masking fall
+into the same queue. Available mask specifiers are a combination
+of the following:
+.Ar dst-ip mask , src-ip mask ,
+.Ar dst-port mask , src-port mask ,
+.Ar proto mask
+or
+.Ar all
+where the latter means all bits in all fields are significant.
+.It buckets Ar NN
+Specifies the size of the hash table used for storing the various queues.
+Default value is 64 controlled by the sysctl variable
+.Ar net.inet.ip.dummynet.hash_size ,
+allowed range is 16 to 1024.
 .El
 .Sh CHECKLIST
 Here are some important points to consider when designing your
@@ -705,7 +720,7 @@ for information on system security levels
 .Pc .
 .Sh PACKET DIVERSION
 A divert socket bound to the specified port will receive all packets diverted
-or tee'd to that port; see
+to that port; see
 .Xr divert 4 .
 If no socket is bound to the destination port, or if the kernel
 wasn't compiled with divert socket support, the packets are dropped.
@@ -737,11 +752,86 @@ or in short form without timestamps:
 This rule diverts all incoming packets from 192.168.2.0/24 to divert port 5000:
 .Pp
 .Dl ipfw divert 5000 all from 192.168.2.0/24 to any in
+.Pp
+The following rules show some of the applications of ipfw and
+dummynet for simulations and the like.
+.Pp
+This rule drops random packets with a probability of 5%
+.Pp
+.Dl "ipfw add prob 0.05 deny ip from any to any in"
+.Pp
+A similar effect can be achieved making use of dummynet pipes:
+.Pp
+.Dl "ipfw add pipe 10 ip from any to any"
+.Dl "ipfw pipe 10 config plr 0.05"
+.Pp
+We can use pipes to artificially limit bandwidth e.g. on a machine
+acting as a router, if we want to limit traffic from local clients
+on 192.168.2.0/24 we do:
+.Pp
+.Dl "ipfw add pipe 1 ip from 192.168.2.0/24 to any out"
+.Dl "ipfw pipe 1 config bw 300Kbit/s queue 50KBytes"
+.Pp
+note that we use the
+.Ql out
+specifier so that the rule is not used twice. Remember in fact
+that ipfw rules are checked both on incoming and outgoing packets.
+.Pp
+Should we like to simulate a bidirectional link with bandwidth
+limitations, the correct way is the following:
+.Pp
+.Dl "ipfw add pipe 1 ip from any to any out"
+.Dl "ipfw add pipe 2 ip from any to any in"
+.Dl "ipfw pipe 1 config bw 64Kbit/s queue 10Kbytes"
+.Dl "ipfw pipe 2 config bw 64Kbit/s queue 10Kbytes"
+.Pp
+The above can be very useful e.g. if you want to see how your fancy
+Web page will look for a residential user which is connected only through
+a slow link.
+You should not use only
+one pipe for both directions, unless you want to simulate a half-duplex
+medium (e.g. appletalk, Ethernet, IRDA).
+It is not necessary that both pipes have the same configuration,
+so we can also simulate asymmetric links.
+.Pp
+Another typical application of the traffic shaper is to introduce some
+delay in the communication. This can affect a lot applications which do
+a lot of Remote Procedure Calls, and where the round-trip-time of the
+connection often becomes a limiting factor much more than bandwidth:
+.Pp
+.Dl "ipfw add pipe 1 ip from any to any out"
+.Dl "ipfw add pipe 2 ip from any to any in"
+.Dl "ipfw pipe 1 config delay 250ms bw 1Mbit/s"
+.Dl "ipfw pipe 2 config delay 250ms bw 1Mbit/s"
+.Pp
+Per-flow queueing can be useful for a variety of purposes. A very
+simple one is counting traffic:
+.Pp
+.Dl "ipfw add pipe 1 tcp from any to any"
+.Dl "ipfw add pipe 1 udp from any to any"
+.Dl "ipfw add pipe 1 ip from any to any"
+.Dl "ipfw pipe 1 config mask all"
+.Pp
+The above set of rules will create queues (and collect statistics)
+for all traffic. Because the pipes have no limitations, the only
+effect is collecting statistics. Note that we need 3 rules, not just
+the last one, because when ipfw tries to match ip packets it will
+not consider ports, so we would not see connections on separate ports
+as different ones.
+.Pp
+A more sophisticated example is limiting the outbound traffic on a net
+with per-host limits, rather than per-network limits:
+.Pp
+.Dl "ipfw add pipe 1 ip from 192.168.2.0/24 to any out"
+.Dl "ipfw add pipe 2 ip from any to 192.168.2.0/24 in"
+.Dl "ipfw pipe 1 config mask src-ip 0x000000ff bw 200Kbit/s queue 20Kbytes"
+.Dl "ipfw pipe 2 config mask dst-ip 0x000000ff bw 200Kbit/s queue 20Kbytes"
 .Sh SEE ALSO
 .Xr cpp 1 ,
 .Xr m4 1 ,
 .Xr divert 4 ,
 .Xr dummynet 4 ,
+.Xr bridge 4 ,
 .Xr ip 4 ,
 .Xr ipfirewall 4 ,
 .Xr protocols 5 ,
@@ -753,6 +843,8 @@ This rule diverts all incoming packets from 192.168.2.0/24 to divert port 5000:
 .Xr syslogd 8 .
 .Sh BUGS
 .Pp
+The syntax has grown over the years and it is not very clean.
+.Pp
 .Em WARNING!!WARNING!!WARNING!!WARNING!!WARNING!!WARNING!!WARNING!!
 .Pp
 This program can put your computer in rather unusable state. When
@@ -778,11 +870,18 @@ going through the rule list.  This may be fixed in a later version.
 .An Ugen J. S. Antsilevich ,
 .An Poul-Henning Kamp ,
 .An Alex Nash ,
-.An Archie Cobbs .
+.An Archie Cobbs ,
+.An Luigi Rizzo .
+.Pp
 API based upon code written by
-.An Daniel Boulet
+Daniel Boulet
 for BSDI.
+.Pp
+Work on dummynet traffic shaper supported by Akamba Corp.
 .Sh HISTORY
 .Nm Ipfw
 first appeared in
 .Fx 2.0 .
+.Nm dummynet
+was introduced in
+.Fx 2.2.8
diff --git a/sbin/ipfw/ipfw.c b/sbin/ipfw/ipfw.c
index aaff7d4..04d02ff 100644
--- a/sbin/ipfw/ipfw.c
+++ b/sbin/ipfw/ipfw.c
@@ -127,6 +127,10 @@ print_port(prot, port, comma)
 	const char *protocol;
 	int printed = 0;
 
+	if (!strcmp(comma,":")) {
+		printf("%s0x%04x", comma, port);
+		return ;
+	}
 	if (do_resolv) {
 		pe = getprotobynumber(prot);
 		if (pe)
@@ -300,6 +304,8 @@ show_ipfw(struct ip_fw *chain, int pcwidth, int bcwidth)
 			print_port(chain->fw_prot, chain->fw_uar.fw_pts[i], comma);
 			if (i==0 && (chain->fw_flg & IP_FW_F_SRNG))
 				comma = "-";
+			else if (i==0 && (chain->fw_flg & IP_FW_F_SMSK))
+				comma = ":";
 			else
 				comma = ",";
 		}
@@ -340,6 +346,8 @@ show_ipfw(struct ip_fw *chain, int pcwidth, int bcwidth)
 			print_port(chain->fw_prot, chain->fw_uar.fw_pts[nsp+i], comma);
 			if (i==0 && (chain->fw_flg & IP_FW_F_DRNG))
 				comma = "-";
+			else if (i==0 && (chain->fw_flg & IP_FW_F_DMSK))
+				comma = ":";
 			else
 				comma = ",";
 		}
@@ -456,13 +464,13 @@ list(ac, av)
 	int pcwidth = 0;
 	int bcwidth = 0;
 	int n, num = 0;
+	int nbytes;
 
 	/* get rules or pipes from kernel, resizing array as necessary */
 	{
 		const int unit = do_pipe ? sizeof(*pipes) : sizeof(*rules);
 		const int ocmd = do_pipe ? IP_DUMMYNET_GET : IP_FW_GET;
 		int nalloc = 0;
-		int nbytes;
 
 		while (num >= nalloc) {
 			nalloc = nalloc * 2 + 200;
@@ -479,19 +487,25 @@ list(ac, av)
 	/* display requested pipes */
 	if (do_pipe) {
 	    u_long rulenum;
+	    void *next_pipe ;
+	    struct dn_pipe *p = (struct dn_pipe *) data;
 
-	    pipes = (struct dn_pipe *) data;
 	    if (ac > 0)
 		rulenum = strtoul(*av++, NULL, 10);
 	    else
 		rulenum = 0 ;
-	    for (n = 0; n < num; n++) {
-		struct dn_pipe *const p = &pipes[n];
+	    for ( ; nbytes > 0 ; p = (struct dn_pipe *)next_pipe ) {
 		double b = p->bandwidth ;
 		char buf[30] ;
 		char qs[30] ;
 		char plr[30] ;
 		int l ;
+		struct dn_flow_queue *q ;
+
+		l = sizeof(*p) + p->rq_elements * sizeof(struct dn_flow_queue) ;
+		next_pipe = (void *)p  + l ;
+		q = (struct dn_flow_queue *)(p+1) ;
+		nbytes -= l ;
 
 		if (rulenum != 0 && rulenum != p->pipe_nr)
 			continue;
@@ -516,9 +530,33 @@ list(ac, av)
 		else
 		    plr[0]='\0';
 
-		printf("%05d: %s %4d ms %s %s -- %d pkts (%d B) %d drops\n",
+		printf("%05d: %s %4d ms %s%s %d queues (%d buckets)\n",
 		    p->pipe_nr, buf, p->delay, qs, plr,
-		    p->r_len, p->r_len_bytes, p->r_drops);
+		    p->rq_elements, p->rq_size);
+		printf("    mask: 0x%02x 0x%08x/0x%04x -> 0x%08x/0x%04x\n",
+		    p->flow_mask.proto,
+		    p->flow_mask.src_ip, p->flow_mask.src_port,
+		    p->flow_mask.dst_ip, p->flow_mask.src_port);
+		for (l = 0 ; l < p->rq_elements ; l++) {
+		    struct in_addr ina ;
+		    struct protoent *pe ;
+
+		    ina.s_addr = htonl(q[l].id.src_ip) ;
+		    printf("    (%d) ", q[l].hash_slot);
+		    pe = getprotobynumber(q[l].id.proto);
+		    if (pe)
+			printf(" %s", pe->p_name);
+		    else
+			printf(" %u", q[l].id.proto);
+		    printf(" %s/%d -> ",
+			inet_ntoa(ina), q[l].id.src_port);
+		    ina.s_addr = htonl(q[l].id.dst_ip) ;
+		    printf("%s/%d\n",
+			inet_ntoa(ina), q[l].id.dst_port);
+		    printf("\t%u pkts %u bytes, tot %qu pkts %qu bytes %u drops\n",
+			q[l].len, q[l].len_bytes,
+			q[l].tot_pkts, q[l].tot_bytes, q[l].drops);
+		}
 	    }
 	    free(data);
 	    return;
@@ -634,6 +672,8 @@ show_usage(const char *fmt, ...)
 "    delay <milliseconds>\n"
 "    queue <size>{packets|Bytes|KBytes}\n"
 "    plr <fraction>\n"
+"    mask {all| [dst-ip|src-ip|dst-port|src-port|proto] <number>}\n"
+"    buckets <number>}\n"
 );
 
 	exit(EX_USAGE);
@@ -777,6 +817,10 @@ lookup_port(const char *arg, int test, int nodash)
 	return(val);
 }
 
+/*
+ * return: 0 normally, 1 if first pair is a range,
+ * 2 if first pair is a port+mask
+ */
 static int
 fill_port(cnt, ptr, off, arg)
 	u_short *cnt, *ptr, off;
@@ -785,10 +829,23 @@ fill_port(cnt, ptr, off, arg)
 	char *s;
 	int initial_range = 0;
 
-	for (s = arg; *s && *s != ',' && *s != '-'; s++) {
+	for (s = arg; *s && *s != ',' && *s != '-' && *s != ':'; s++) {
 		if (*s == '\\' && *(s+1))
 			s++;
 	}
+	if (*s == ':') {
+		*s++ = '\0';
+		if (strchr(arg, ','))
+			errx(EX_USAGE, "port/mask must be first in list");
+		add_port(cnt, ptr, off, *arg ? lookup_port(arg, 0, 0) : 0x0000);
+		arg = s;
+		s = strchr(arg,',');
+		if (s)
+			*s++ = '\0';
+		add_port(cnt, ptr, off, *arg ? lookup_port(arg, 0, 0) : 0xffff);
+		arg = s;
+		initial_range = 2;
+	} else
 	if (*s == '-') {
 		*s++ = '\0';
 		if (strchr(arg, ','))
@@ -1038,6 +1095,77 @@ config_pipe(int ac, char **av)
                     pipe.queue_size = 0 ;
                 }
                 av+=2; ac-=2;
+	    } else if (!strncmp(*av,"buckets",strlen(*av)) ) {
+		pipe.rq_size = strtoul(av[1], NULL, 0);
+		av+=2; ac-=2;
+	    } else if (!strncmp(*av,"mask",strlen(*av)) ) {
+                /* per-flow queue, mask is dst_ip, dst_port,
+                 * src_ip, src_port, proto measured in bits
+                 */
+                u_int32_t a ;
+                u_int32_t *par = NULL ;
+ 
+                pipe.flow_mask.dst_ip = 0 ;
+                pipe.flow_mask.src_ip = 0 ;
+                pipe.flow_mask.dst_port = 0 ;
+                pipe.flow_mask.src_port = 0 ;
+                pipe.flow_mask.proto = 0 ;
+                end = NULL ;
+                av++ ; ac-- ;
+                if (ac >= 1 && !strncmp(*av,"all", strlen(*av)) ) {
+                    /* special case -- all bits are significant */
+                    pipe.flow_mask.dst_ip = ~0 ;
+                    pipe.flow_mask.src_ip = ~0 ;
+                    pipe.flow_mask.dst_port = ~0 ;
+                    pipe.flow_mask.src_port = ~0 ;
+                    pipe.flow_mask.proto = ~0 ;
+                    pipe.flags |= DN_HAVE_FLOW_MASK ;
+                    av++ ; ac-- ;
+                } else {
+                  for (;;) {
+                    if (ac < 1)
+                        break ;
+                    if (!strncmp(*av,"dst-ip", strlen(*av)))
+                        par = &(pipe.flow_mask.dst_ip) ;
+                    else if (!strncmp(*av,"src-ip", strlen(*av)))
+                        par = &(pipe.flow_mask.src_ip) ;
+                    else if (!strncmp(*av,"dst-port", strlen(*av)))
+                        (u_int16_t *)par = &(pipe.flow_mask.dst_port) ;
+                    else if (!strncmp(*av,"src-port", strlen(*av)))
+                        (u_int16_t *)par = &(pipe.flow_mask.src_port) ;
+                    else if (!strncmp(*av,"proto", strlen(*av)))
+                        (u_int8_t *)par = &(pipe.flow_mask.proto) ;
+                    else
+                        break ;
+                    if (ac < 2)
+                        show_usage("mask: %s value missing", *av);
+                    if (*av[1] == '/') {
+                        a = strtoul(av[1]+1, &end, 0);
+                        if (a == 32) /* special case... */
+                            a = ~0 ;
+                        else
+                            a = (1 << a) - 1 ;
+                        fprintf(stderr, " mask is 0x%08x\n", a);
+                    } else
+                        a = strtoul(av[1], &end, 0);
+                    if ( (u_int16_t *)par == &(pipe.flow_mask.src_port) ||
+                         (u_int16_t *)par == &(pipe.flow_mask.dst_port) ) {
+                        if (a >= (1<<16) )
+                            show_usage("mask: %s must be 16 bit, not 0x%08x",
+                                *av, a);
+                        *((u_int16_t *)par) = (u_int16_t) a;
+                    } else if ( (u_int8_t *)par == &(pipe.flow_mask.proto) ) {
+                        if (a >= (1<<8) )
+                            show_usage("mask: %s must be 8 bit, not 0x%08x",
+                                *av, a);
+                        *((u_int8_t *)par) = (u_int8_t) a;
+                    } else
+                        *par = a;
+                    if (a != 0)
+                        pipe.flags |= DN_HAVE_FLOW_MASK ;
+                    av += 2 ; ac -= 2 ;
+                  } /* end for */
+                }
             } else
                 show_usage("unrecognised option ``%s''", *av);
         }
@@ -1230,9 +1358,13 @@ add(ac,av)
 
 	if (ac && (isdigit(**av) || lookup_port(*av, 1, 1) >= 0)) {
 		u_short nports = 0;
+		int retval ;
 
-		if (fill_port(&nports, rule.fw_uar.fw_pts, 0, *av))
+		retval = fill_port(&nports, rule.fw_uar.fw_pts, 0, *av) ;
+		if (retval == 1)
 			rule.fw_flg |= IP_FW_F_SRNG;
+		else if (retval == 2)
+			rule.fw_flg |= IP_FW_F_SMSK;
 		IP_FW_SETNSRCP(&rule, nports);
 		av++; ac--;
 	}
@@ -1253,10 +1385,14 @@ add(ac,av)
 
 	if (ac && (isdigit(**av) || lookup_port(*av, 1, 1) >= 0)) {
 		u_short	nports = 0;
+		int retval ;
 
-		if (fill_port(&nports,
-		    rule.fw_uar.fw_pts, IP_FW_GETNSRCP(&rule), *av))
+		retval = fill_port(&nports,
+		    rule.fw_uar.fw_pts, IP_FW_GETNSRCP(&rule), *av) ;
+		if (retval == 1)
 			rule.fw_flg |= IP_FW_F_DRNG;
+		else if (retval == 2)
+			rule.fw_flg |= IP_FW_F_DMSK;
 		IP_FW_SETNDSTP(&rule, nports);
 		av++; ac--;
 	}