Bring in the most recent version of ipfw and dummynet, developed

and tested over the past two months in the ipfw3-head branch.  This
also happens to be the same code available in the Linux and Windows
ports of ipfw and dummynet.

The major enhancement is a completely restructured version of
dummynet, with support for different packet scheduling algorithms
(loadable at runtime), faster queue/pipe lookup, and a much cleaner
internal architecture and kernel/userland ABI which simplifies
future extensions.

In addition to the existing schedulers (FIFO and WF2Q+), we include
a Deficit Round Robin (DRR or RR for brevity) scheduler, and a new,
very fast version of WF2Q+ called QFQ.

Some test code is also present (in sys/netinet/ipfw/test) that
lets you build and test schedulers in userland.

Also, we have added a compatibility layer that understands requests
from the RELENG_7 and RELENG_8 versions of the /sbin/ipfw binaries,
and replies correctly (at least, it does its best; sometimes you
just cannot tell who sent the request and how to answer).
The compatibility layer should make it possible to MFC this code in a
relatively short time.

Some minor glitches (e.g. handling of ipfw set enable/disable,
and a workaround for a bug in RELENG_7's /sbin/ipfw) will be
fixed with separate commits.

CREDITS:
This work has been partly supported by the ONELAB2 project, and
mostly developed by Riccardo Panicucci and myself.
The code for the qfq scheduler is mostly from Fabio Checconi,
and Marta Carbone and Francesco Magno have helped with testing,
debugging and some bug fixes.

1 files changed, 80 insertions, 62 deletions

diff --git a/sbin/ipfw/ipfw.8 b/sbin/ipfw/ipfw.8
index fc83ecc..d605d5e 100644
--- a/sbin/ipfw/ipfw.8
+++ b/sbin/ipfw/ipfw.8
@@ -6,8 +6,10 @@
 .Os
 .Sh NAME
 .Nm ipfw
-.Nd IP firewall and traffic shaper control program
+.Nd User interface for firewall, traffic shaper, packet scheduler,
+in-kernel NAT.
 .Sh SYNOPSIS
+.Ss FIREWALL CONFIGURATION
 .Nm
 .Op Fl cq
 .Cm add
@@ -26,12 +28,6 @@
 .Op Cm set Ar N
 .Brq Cm delete | zero | resetlog
 .Op Ar number ...
-.Nm
-.Cm enable
-.Brq Cm firewall | altq | one_pass | debug | verbose | dyn_keepalive
-.Nm
-.Cm disable
-.Brq Cm firewall | altq | one_pass | debug | verbose | dyn_keepalive
 .Pp
 .Nm
 .Cm set Oo Cm disable Ar number ... Oc Op Cm enable Ar number ...
@@ -43,8 +39,17 @@
 .Cm set swap Ar number number
 .Nm
 .Cm set show
+.Ss SYSCTL SHORTCUTS
 .Pp
 .Nm
+.Cm enable
+.Brq Cm firewall | altq | one_pass | debug | verbose | dyn_keepalive
+.Nm
+.Cm disable
+.Brq Cm firewall | altq | one_pass | debug | verbose | dyn_keepalive
+.Pp
+.Ss LOOKUP TABLES
+.Nm
 .Cm table Ar number Cm add Ar addr Ns Oo / Ns Ar masklen Oc Op Ar value
 .Nm
 .Cm table Ar number Cm delete Ar addr Ns Op / Ns Ar masklen
@@ -57,17 +62,19 @@
 .Brq Ar number | all
 .Cm list
 .Pp
+.Ss DUMMYNET CONFIGURATION (TRAFFIC SHAPER AND PACKET SCHEDULER)
 .Nm
-.Brq Cm pipe | queue
+.Brq Cm pipe | queue | sched
 .Ar number
 .Cm config
 .Ar config-options
 .Nm
 .Op Fl s Op Ar field
-.Brq Cm pipe | queue
+.Brq Cm pipe | queue | sched
 .Brq Cm delete | list | show
 .Op Ar number ...
 .Pp
+.Ss IN-KERNEL NAT
 .Nm
 .Op Fl q
 .Cm nat
@@ -89,28 +96,27 @@ The
 .Nm
 utility is the user interface for controlling the
 .Xr ipfw 4
-firewall and the
+firewall, the
 .Xr dummynet 4
-traffic shaper in
-.Fx .
+traffic shaper/packet scheduler, and the
+in-kernel NAT services.
 .Pp
-An
-.Nm
-configuration, or
+A firewall configuration, or
 .Em ruleset ,
 is made of a list of
 .Em rules
 numbered from 1 to 65535.
-Packets are passed to
-.Nm
+Packets are passed to the firewall
 from a number of different places in the protocol stack
 (depending on the source and destination of the packet,
-it is possible that
-.Nm
-is invoked multiple times on the same packet).
+it is possible for the firewall to be
+invoked multiple times on the same packet).
 The packet passed to the firewall is compared
-against each of the rules in the firewall
-.Em ruleset .
+against each of the rules in the
+.Em ruleset ,
+in rule-number order
+(multiple rules with the same number are permitted, in which case
+they are processed in order of insertion).
 When a match is found, the action corresponding to the
 matching rule is performed.
 .Pp
@@ -118,9 +124,7 @@ Depending on the action and certain system settings, packets
 can be reinjected into the firewall at some rule after the
 matching one for further processing.
 .Pp
-An
-.Nm
-ruleset always includes a
+A ruleset always includes a
 .Em default
 rule (numbered 65535) which cannot be modified or deleted,
 and matches all packets.
@@ -137,14 +141,14 @@ If the ruleset includes one or more rules with the
 or
 .Cm limit
 option,
-.Nm
-will have a
+the firewall will have a
 .Em stateful
-behaviour, i.e., upon a match it will create dynamic rules matching
-the exact parameters (source and destination addresses and ports)
-of the matching packet.
-.Pp
-These dynamic rules, which have a limited lifetime, are checked
+behaviour, i.e., upon a match it will create
+.Em dynamic rules ,
+i.e. rules that match packets with the same 5-tuple
+(protocol, source and destination addresses and ports)
+as the packet which caused their creation.
+Dynamic rules, which have a limited lifetime, are checked
 at the first occurrence of a
 .Cm check-state ,
 .Cm keep-state
@@ -283,6 +287,7 @@ When listing, show last match timestamp as seconds from the epoch.
 This form can be more convenient for postprocessing by scripts.
 .El
 .Pp
+.Ss LIST OF RULES AND PREPROCESSING
 To ease configuration, rules can be put into a file which is
 processed using
 .Nm
@@ -322,14 +327,16 @@ This allows for flexible configuration files (like conditionalizing
 them on the local hostname) and the use of macros to centralize
 frequently required arguments like IP addresses.
 .Pp
+.Ss TRAFFIC SHAPER CONFIGURATION
 The
 .Nm
-.Cm pipe
+.Cm pipe , queue
 and
-.Cm queue
-commands are used to configure the traffic shaper, as shown in the
+.Cm sched
+commands are used to configure the traffic shaper and packet scheduler.
+See the
 .Sx TRAFFIC SHAPER (DUMMYNET) CONFIGURATION
-Section below.
+Section below for details.
 .Pp
 If the world and the kernel get out of sync the
 .Nm
@@ -362,7 +369,7 @@ have this picture in mind in order to design a correct ruleset.
        |      to devices       |
 .Ed
 .Pp
-As can be noted from the above picture, the number of
+The number of
 times the same packet goes through the firewall can
 vary between 0 and 4 depending on packet source and
 destination, and system configuration.
@@ -421,9 +428,9 @@ Keywords are case-sensitive, whereas arguments may
 or may not be case-sensitive depending on their nature
 (e.g.\& uid's are, hostnames are not).
 .Pp
-In
-.Nm ipfw2
-you can introduce spaces after commas ',' to make
+Some arguments (e.g. port or address lists) are comma-separated
+lists of values.
+In this case, spaces after commas ',' are allowed to make
 the line more readable.
 You can also put the entire
 command (including flags) into a single argument.
@@ -434,9 +441,7 @@ ipfw -q add deny src-ip 10.0.0.0/24, 127.0.0.1/8
 ipfw "-q add deny src-ip 10.0.0.0/24, 127.0.0.1/8"
 .Ed
 .Sh RULE FORMAT
-The format of
-.Nm
-rules is the following:
+The format of firewall rules is the following:
 .Bd -ragged -offset indent
 .Bk -words
 .Op Ar rule_number
@@ -496,7 +501,7 @@ in future forwarding decisions.
 .El
 .Pp
 Note that some of the above information, e.g.\& source MAC or IP addresses and
-TCP/UDP ports, could easily be spoofed, so filtering on those fields
+TCP/UDP ports, can be easily spoofed, so filtering on those fields
 alone might not guarantee the desired results.
 .Bl -tag -width indent
 .It Ar rule_number
@@ -1643,15 +1648,17 @@ because it engages only on packets with source addresses of directly
 connected networks instead of all source addresses.
 .El
 .Sh LOOKUP TABLES
-Lookup tables are useful to handle large sparse address sets,
-typically from a hundred to several thousands of entries.
+Lookup tables are useful to handle large sparse sets of
+addresses or other search keys (e.g. ports, jail IDs).
+In the rest of this section we will use the term ``address''
+to mean any unsigned value of up to 32-bit.
 There may be up to 128 different lookup tables, numbered 0 to 127.
 .Pp
 Each entry is represented by an
 .Ar addr Ns Op / Ns Ar masklen
 and will match all addresses with base
 .Ar addr
-(specified as an IP address or a hostname)
+(specified as an IP address, a hostname or an unsigned integer)
 and mask width of
 .Ar masklen
 bits.
@@ -1669,9 +1676,9 @@ is not specified, it defaults to 0.
 .Pp
 An entry can be added to a table
 .Pq Cm add ,
-removed from a table
-.Pq Cm delete ,
-a table can be examined
+or removed from a table
+.Pq Cm delete .
+A table can be examined
 .Pq Cm list
 or flushed
 .Pq Cm flush .
@@ -1680,7 +1687,7 @@ Internally, each table is stored in a Radix tree, the same way as
 the routing table (see
 .Xr route 4 ) .
 .Pp
-Lookup tables currently support IPv4 addresses only.
+Lookup tables currently support only ports, jail IDs and IPv4 addresses.
 .Pp
 The
 .Cm tablearg
@@ -1838,7 +1845,7 @@ for more examples on how to use dynamic rules.
 .Nm
 is also the user interface for the
 .Nm dummynet
-traffic shaper and network emulator, a subsystem that
+traffic shaper, packet scheduler and network emulator, a subsystem that
 can artificially queue, delay or drop packets
 emulator the behaviour of certain network links
 or queueing systems.
@@ -1858,17 +1865,17 @@ Packets are queued in front of the pipe as they come out from the classifier,
 and then transferred to the pipe according to the pipe's parameters.
 .It Em queue
 A queue
-is an abstraction used to implement the WF2Q+
-(Worst-case Fair Weighted Fair Queueing) policy, which is
-an efficient variant of the WFQ policy.
+is an abstraction used to implement packet scheduling
+using one of several packet scheduling algorithms.
 .Pp
 The queue associates a
 .Em weight
-and a reference pipe to each flow (a flow is a set of packets
+and a reference scheduler to each flow (a flow is a set of packets
 with the same addresses and ports after masking).
-All backlogged flows (i.e., those
-with packets queued) linked to the same pipe share the pipe's
-bandwidth proportionally to their weights.
+A scheduler in turn is connected to a pipe, and arbitrates
+the pipe's bandwidth among backlogged flows according to
+weights and to the features of the scheduling algorithm in use.
+.Pp
 Note that weights are not priorities; a flow with a lower weight
 is still guaranteed to get its fraction of the bandwidth even if a
 flow with a higher weight is permanently backlogged.
@@ -1911,16 +1918,19 @@ mode can be enabled by setting the
 .Xr sysctl 8
 variable to a non-zero value.
 .Pp
-.Ss PIPE AND QUEUE CONFIGURATION
+.Ss PIPE, QUEUE AND SCHEDULER CONFIGURATION
 The
-.Em pipe
-and
+.Em pipe ,
 .Em queue
+and
+.Em scheduler
 configuration commands are the following:
 .Bd -ragged -offset indent
 .Cm pipe Ar number Cm config Ar pipe-configuration
 .Pp
 .Cm queue Ar number Cm config Ar queue-configuration
+.Pp
+.Cm sched Ar number Cm config Ar sched-configuration
 .Ed
 .Pp
 The following parameters can be configured for a pipe:
@@ -2073,6 +2083,14 @@ Specifies the weight to be used for flows matching this queue.
 The weight must be in the range 1..100, and defaults to 1.
 .El
 .Pp
+The following parameters can be configured for a scheduler:
+.Pp
+.Bl -tag -width indent -compact
+.It Cm type Ar {fifo | wf2qp | rr | qfq}
+.El
+.Pp
+plus all the parameters allowed for a pipe.
+.Pp
 Finally, the following parameters can be configured for both
 pipes and queues:
 .Pp