ET/BWMGR is a bandwidth management system that provides a rich set of tools for monitoring and controlling the flow of traffic to and from your network. The System Monitor allows you to view traffic on your network with a variety of criteria, allowing you to learn about the composition of traffic on your network and also to view current usage by devices or protocols. The Firewall allows you to control the type of traffic that is allowed to flow in and out of your network. The bandwidth manager controls the shaping and limiting of traffic flows in your network.

Getting the ET/BWMGR product

The ET/BWMGR is sold as a complete, integrated hardware solution, or as a software image that can be installed on your own hardware.

Downloading the Demo CD

Before you buy the product you'll want to try it. If you want to run the software on your own hardware you'll need to test the hardware first, as sell as obtain the information needed to determine what licensing is required. To download the CD, you'll need to open a sales ticket (see contact us) and provide your IP so we can allow you to download.

Buying an Appliance

There are many advantages to an appliance vs the CD Appliance version of the product. Anyone who's ever used a computer knows that there are so many diferent chipsets, NICs, video cards and memory types that running any os will always result in some small problems. Our appliances run on hardware that we've rigorously tested. We've fixed and tuned drivers, and we have 100s of customers all using the same hardware. If you encounter a problem, we will help figure out what it is, because we don't want our other customers to have the same problem. With our appliances, you get a solid product with full support. Its a piece of mind thats very important for such an important piece of your network.

Another advantage is performance. Tuning a system, and getting the most out of a NIC are critical at the high end of performance. Our systems are optimised to perform on our hardware. In many cases, we've had to modify drivers and subsystems to fix seamlessly into our system image. If you are using your own hardware, you may encounter some quirkiness, which is why its important that you test the hardware before you buy.

The CD Appliance

The CD appliance is an appliance image that is built from the ET/BWGR demo CD. First you download and burn a demo CD. While you're testing, you can just boot off the CD. If you decide to purchase a license, you can use the CD to build an appliance. This allows you to test your hardware before you buy. In order to build a system, you will need to access our update server, which requires a license. Once you receive your license, you will be able to build a complete system on your own hardware. The advantage of a CD appliance over self-installation of the software is that you get all of the correct versions of companion software needed (mySQL, snmp, kernel configuration, driver patches), plus you get to use our update service to download updates directly to your system. Because we know that you are using the corect software versions, we can provide more help. (you are required to buy a subscription to get updates and support). The disadvantage of the CD appliance is that you may have hardware problems; some variation of a NIC that we haven't encountered, some chip that isn't supported well. Luckily you can test our hardware with our CD demo before making a purchase, so you should be able to find problems.

Editing default.cfg

You should not edit default.cfg manually. The file is described in the GUI manual.

Configuring Stats-only Mode

Stats-only mode is used if you just want to gather statistics and/or monitor your network activity. It allows you to plug only 1 ethernet port into a passive hub without putting the system "inline", so that the system cannot and will not affect your network in any way. Note that you must use a hug and not a switch, as switches will filter the traffic so the ET/BWMGR software will not see everything.

To configure stats-only mode, configure the interface that is plugged into the hub as follows:

bwmgr em0 -ifac -statsonly

From the GUI, go into the bandwidth rules screen, click on the interface link and select "Edit Interface". Then check the "stats-only mode" checkbox.

In stats-only mode, all rules must be -global with stats enabled. This allows you to gather stats by IP, MAC or traffic type and store them in the database. You can also use the system monitor in stats-only mode to view activity on your network.

Setting the Outside Interface

In order for the bwmgr to detect protocols and to understand the direction of your traffic, you must tag the interface connected to the outside world as an outside interface. Typically this is the interface connected to the internet. If you have multiple paths, then all interfaces connected to the interface should be set as outside. Protocols are only gathered on the outside interfaces to avoid being counted twice. To set the outside interface, use the -o setting:

bwmgr em0 -ifac -o

Setting Up Bridging

The ET/BWMGR software allows you to set up your system as a transparent MAC layer bridge. Bridging allows you to put a PC with the ET/BWMGR software in between any two segments without having to worry about renumbering your network.

Bridge Types

There are 2 bridges for FreeBSD 7+: the standard bridge and the "fastbridge".

Bridge

The bridge is essentially the bridge which is included with the FreeBSD operating system, with some customization for use with the ET/BWMGR. Its loaded by default when you load the etbwmgr module. There is nothing special to do to load the normal bridge.

FreeBSD 7.0 requires that you create a bridge device. While the OS allows for name flexibility, the ET/BWMGR requires that bridges be named bridge# where '#' is a number from 1 to 8, representing the bridge group number. While you could use ifconfig to create the bridge, the ET/BWMGR implements a command to do it which allows you to set the MAC address:

bwmgr bridge0 create 00:00:00:17:12:11

This is significant because in FreeBSD 7, the bridge MAC address is used rather than the ethernet address for the local address. If you create the bridge with the ifconfig command, it will generate a random address, which has the consequence of giving the system a different address every time the machine is restarted. Using the ET/BWMGR command allows you to assign a static mac address for the system. The number of the bridge is also the bridge group number, which you'll need to add members to the bridge.

To add members to the bridge, you can use bwmgr commands as follows:

bwmgr em0 bridge 0
bwmgr em1 bridge 0

These commands add em0 and em1 to the bridge. If you need to assign an address to the bridge, you assign it to the bridge that you created rather than an interface on the bridge using an ifconfig command.

ifconfig bridge0 192.168.11.1 netmask 255.255.255.0

Members can be added to the bridge in the same way as in previous versions:

bwmgr em0 bridge 0
bwmgr em1 bridge 0

These statements add em0 and em1 to bridge group 0 (bridge0). There are several flags that can be set per interface in FreeBSD 7. The most useful flag is ffwd; short for "Fast Forward". This can be used when there are only 2 interfaces in the group and all traffic needs to be forwarded. Note on a network you may have local traffic (such as a local file server) which is usually "learned" by a bridge and not forwarded. If your setup is purely client<->internet, then you can use fast forwarding for better performance.

Also supported is STP, or "Spanning Tree Protocol". You can turn STP in with the stp flag. Note that STP interacts with other spanning tree bridges in your network. If you choose to use this feature, we do not support the interactiions. We don't guarantee that its compatible in any given network.

Setting Up a FastBridge

The fastbridge is not really a bridge at all; its actually a forwarder that is designed to provide superior performance in systems where there are only 2 interfaces in the traffic flow, AND you are using a separate interface for administration. The fastbridge eliminates the code necessary to support learning and broadcast functions and reduces a layer of lock contention in the OS. The fastbridge is essential for use in 10 gigabit environments. The rules for using a fastbridge are:

There must be only 2 interfaces in the bridge
Only em, igb and ix interfaces are supported
The bridge may not have an address

To use the fastbridge, you must explicitly load the fastbridge driver. When the etbwmgr.ko module is loaded, it will load the regular bridge by default. To load the fastbridge, you'll need to set it in loader.conf:

loader.conf

In the file /boot/loader.conf, add the following line:

if_fastbridge_load="YES"
etbwmgr_load="YES"

Note that the if_fastbridge_load line must appear before the etbwmgr_load line. Otherwise etbwmgr will load the regular bridge and there will be a conflict.

Setting up the fastbridge is the same as the regular bridge. Fast Forwarding is the default and only option. So all you'll need to do is:

ifconfig bridge0 192.168.11.1 netmask 255.255.255.0
bwmgr ix0 bridge 0
bwmgr ix1 bridge 0

Fast Forwarding

Another setting is "Fast Forwarding" mode. Fast forwarding disabled all learning and other filtering and assumes that there are only 2 ports and that all traffic needs to be forwarded. This is a typical setup where the box is between the router and main switch and all traffic is destined for the internet:

bwmgr em0 bridge 1 ffwd

Setting up a Fallback Interface

A fallback can be implemented using a LAGG(4) interface. You can add a LAGG interface to a bridge as follows:

bwmgr em0 bridge 1 ffwd
bwmgr lagg0 bridge 1

This creates a bridge with em0 and lagg0. Suppose you have lagg0 configured with em1 and em2 where em2 was a fallback. This allows em2 to be a fallback to em2 on the bridge. See the FreeBSD documentation for more information on LAGG interfaces:

man lagg

Span Ports

The Bridge also allows for an interface to be configured as a span port. Span ports send traffic to a monitor and is effectively a "tap" on the line. When a port is configured as a span interface, all traffic forwarded on that bridge group will be sent out the span port. Note that span ports are not supported on the fastbridge.

System Configuration Parameters

Setting the Indexing Level

The ET/BWMGR incorporates a powerful algorithm which indexes rules for ultra-fast lookup. There are 4 indexing levels available for non-firewall rules, both of which use the -addr (IP Address) part of the rule as the index. The default, level 4, should be used for most installations. Levels 1,2 and 3 have less granularity but can be used for rulesets which have a lot of subnet rules. Firewall setups with extensive blocks on a wide variety of rules can be made much more efficient by changing the indexing. Indexing can be summarized as follows. Suppose your address rule is A.B.C.D:

Level 4 - Uses A.B.C.D
Level 3 - Uses A.B.C
Level 2 - Uses A.B
Level 1 - Uses A

Rules that have less precision than the index cannot be indexed and are therefore global rules. Global rules much be checked for each packet received. So the rule:

bwmgr em0 -addr 192.168.1.0 -addrmsk 255.255.255.0

can be indexed for levels 1,2 and 3 (full 255 mask) but not for level 4.

Each interface can index at a different level. As stated before, the default is Level 4, so if you are using level 4 you dont need to do anything to set the level. The GUI (and the "rebuild" command) will always generate a command line to set the index level. The syntax is as follows:

bwmgr em0 indexlevel 4

This sets the index level on interface em0 to 4. Note that you MUST do this before adding any rules to the interface. The driver will reject an attempt to change the level if there are rules presently defined.

Starting with v3.3, the firewall rules are also indexed. Firewalls have 5 index level choices. 1,2,3 and 4 are the same as for the regular rulesets, using the IP address. A 3rd choice is available, ports, which uses -port. port indexing is useful if you are firewalling by port rather than by address. Address is more effective, so if you use both ports and addresses, then use you shouldn't use the port indexlevel. You set the firewall indexlevel using the fwlevel keyword:

bwmgr em0 fwindex 1|2|3|4|ports

Analyzing your ruleset

You can check the efficiency of your ruleset with the analyze command:

bwmgr em0 analyze

FW Global Rules: 0
FW Indexes: 0
FW Worst Case: 0
BW Global Rules: 3
BW Indexes: 1
BW Worst Case: 3

This shows that the current ruleset has 3 rules that have to be checked for every packet that arrives. Keeping the number of global rules down is the key to efficiency. The "Worst Case" is the rule which requires the most processing. Worst case is always higher than the number of global rules, which is "every case".

Note that the context of the term "global" is important here. In this context, a global rule is any rule that is not indexed. So all rules that do not have an IP address or MAC address is a global rule. For example, a rule that matches a protocol is a global rule, since there is no way to index it. Secondly, a rule tagged -global is NOT a global rule if it has an IP or MAC address. So in that sense, the Global rules showed here have nothing to do with rules tagged -global in your ruleset.

Setting the maxbuffers parameter

The maxbuffers parameter will put a cap on the number of buffers that the bandwidth manager will use to store managed data in its limiting or prioritization queues. The purpose of this parameter is to keep the bandwidth manager from exhausting your system of network buffers (which may cause it to fail). You can also use it to limit the amount of backup traffic that the bandwidth manager will manage. Typically, this number should be somewhat lower than the number of system buffers you have available for network frames. You can check to see how many buffers are allocated in your running kernel with the following command:

sysctl kern.ipc.nmbclusters

Issuing this command will print the number of buffers allocated in your system. To change the setting, you can either change the NMBCLUSTERS setting in your kernel, or you can set the number of buffers to use at boot time by putting a line like the following in /boot/loader.conf:

kern.ipc.nmbclusters="16000"

Remember that each buffer is 2K, so you can't allocate an infinite number of buffers. Normally Freebsd will allocate 1/3 of RAM to the kernel. Our supplied kernels and kernels in our appliances use an algorithm to determine how much ram to use. Note that buffers are allocated on the fly, so the system won't tell you if there isn't enough RAM to implement your setting. If you set it too high, your system may just blow up when it runs out of memory, so be careful.

After determining the buffer allocation in your system, you can then use "netstat -m" after initial boot to see how many buffers your system uses with no traffic (this will vary depending on what cards and devices you have in your system. Suppose your system uses 700 clusters after boot. You could set your maxbuffers setting to say 15000 (which is less than 16000 - 700).

In Linux, you have no way of knowing how many buffers your system has as the memory pool is dynamic, so its pretty much a crapshoot. Usually a setting of 10000 or so is safe for most installations.

When you issue a "show" command via the command line or look at the main menu in the HTML interface, you will see "maxbuffers" shown in the n1/n2 format. The first number is the "high use" counter..this is not the number in use currently, but the most that were ever required. The second number is your "maxbuffers" setting. You can clear the first number by setting maxbuffers to the same number as you currently have. Its not generally a good idea to change the maxbuffers setting when your system is experiencing a heavy load, but setting it to the same value will only reset the high use counter. So, if you have 8000 buffers allocated in your system, you can issue the command:

bwmgr maxbuffers 16000

to reset the high use counter.

Setting the Stats Period

The stats period is the length of the window used internally to manage per-rule statistics. The default period is 20 seconds. For statistics, the stats period is how often snapshots of statistics are saved. For the most part, this is completely arbitrary. When you view stat for a rule, you will see the usage for the current period and the last stats period. The current period will have a slight error unless looked at exactly on the second. The "last period" will be a snapshot of the last stats-period, and will be an exact usage value for that period.

These are the stats that are used with the "sort by usage" feature in the GUI.

Starting the Bandwidth Manager

You can start the bandwidth manager process with the following syntax::

bwmgr interface start authentication_key

where interface is the interface that you used to generate your serial number with the "bwmgr interface serialize" command and authentication_key is the license key that you received when you purchased your license..

If you have a v3.2 license key, that is you purchased your license before v4.0 was released and you haven't upgraded, you need to use the startv32 command instead of start:

bwmgr interface startv32 v32_authentication_key

Running the ET/BWMGR Demo

Before you purchase a license, you should test your target hardware with out Demo CD. You'll need to run the sofware in "demo mode" by using the word demo as the key. So, for example, to run the bandwidth manager in demo mode, you could use a line similar to the following:

bwmgr bw0 start demo

Demo mode provides full functionality for a gigabit bandwidth license but will only run for 6 hours, provided that your system can contact out server when you start it (ie an internet connection is required). When the demo concludes, a message will be displayed indicating that it has completed. Once the demo completes, traffic will no longer be controlled, and bridging will cease to operate. In order to restart the software, you will have to reboot the system. You will be able to run the full demo for approximately 20 days. If you cannot contact our server or your demo period has expired, the bandwidth manager will start in "Limited" demo mode, which is only 1 hour, disables use of the System Monitor and allows a maximum of 10 rules. Once you have determined that the sofware is installed properly on your hardware, you can obtain a license key from Emerging Technologies that will no longer stop after the demo time period (see the FAQ).

If you are considering a purchase of a 10Mb/s or 2Mb/s software license, you can run a 10 or 2Mb/s demo, which runs at 10Mb/s or 2Mb/s respectively, and will simulate a software license. To run the 10Mb/s demo, use the following:

bwmgr em0 start demo10

for the 2Mb/s demo:

bwmgr em0 start demo2

and for the 100Mb/s demo:

bwmgr em0 start demo100

Obtaining Your System Serial Number

In order to purchase a license, you will need to supply your system "serial number". This is a unique number that is specific to the physical system you are using. When you order the ET/BWMGR software license, you will receive a "key", which is keyed to the serial number, so it can only be used on your physical system. To get the serial number, run the following command:

# bwmgr interface serialize

(where interface is an ethernet interface, such as em0). This will print out 10 hex pairs that is the serial number for your system, such as:

#bwmgr em0 serialize System Serial# 06:9b:10:08:69:03:77:ff:89:50
License Required:
2Mb/s: 2
10Mb/s: 3
45Mb/s: 6
Unlimited: 23

Note that there is a "license level" next to the different types of licenses available. When you order your license, you will need to provide the serial sequence as well as the license level on the order form so that we can generate your license key. When you receive your license key, you MUST start the bandwidth manager on the same interface that you used to generate the serial (em0 in the above example).

Registering your ET/BWMGR License Key

When you receive your license key, you will need to register it before you can use it. Before you register it you should make certain that you've tested with the demo for long enough (several 6 hour tests are recommended) to make sure you don't have a hardware problem. If you find a problem with your hardware, you can get a replacement key, provided that you have not yet registered the license. Once you register your license, you will have to buy a replacement license, and you are only permitted to buy 1 replacement.

To register the license, you will need to be able to contact our server. If you can ping www.etinc.com then you have access to the server. Also, you will have to have UDP ports in the 4000-5000 range available. So if you have a firewall be certain that you "open up" the bandwidth management device to allow the registration information to pass. To register, from the command line do the following:

# bwmgr em0 register license_key

where em0 is the interface that you used to generate the serial number and license_key is the license you purchased from ET. If all goes well, you will receive a message such as "Registration Complete". Once you receive this message, you can reestablish your firewall and you will be able to restart the bandwidth manager without restriction.

If you still have a version 3.2 license key, you'll need to use the registerv32 command:

# bwmgr em0 registerv32 v32_license_key

Note that any time you receive an updated license key, or if you upgrade your OS or change hard drives you will need to re-register your software (you DO NOT need to re-register when upgrading versions of the ET/BWMGR drivers). If you re-register you will need to do it from the same IP address as you originally registered from. If you need to change the IP, you will have to notify ET so that we can authorize the change. Just create a sales or support ticket with your license_key or serial number and tell us you need to re-register with a different IP address.

Stopping the Bandwidth Manager

You can stop the bandwidth manager, which will stop it from limiting traffic with the following command:

bwmgr stop

Note that you shouldn't normally use this command unless there is something critically wrong. If you have bridging enabled, traffic will no longer be forwarded. If you are routing and not bridging, then stopping the bandwidth manager will just stop the functions of the software and revert to normal processing of traffic.

Rule Tutorial - What is a Rule?

A "rule" is a specification which defines what actions should be taken when events occur. In the context of the bandwidth manager, a rule is a description of a type of traffic, and a specification of what action to take when that traffic is encountered. The ET/BWMGR is controlled by "rules", which you will set up to allow the sofware to control your network traffic. Some products call these "policies". We'll use both terms.

Rule Syntax

The bwmgr rule syntax is very extensive, and will be illustrated throught this document in the appropriate section. The base syntax for adding rules can be summarized in the following:

bmgr interface [-x rulenumber] criteria [bandwidth controls] [-priority priority_selector]

where criteria is the match critera and the default priority is "normal". Basic criteria includes the following:

-name                   name (a label) for the rule               
-x                      index (precedence) of the rule
-r                      reverse rule               
-l                      enable logging               
-fw                     firewall rule               
-saddr {ip_address}     source IP address               
-daddr {ip_address}     destination IP address               
-addr {ip_address}      either source or destination IP address
-smaddr{ mac_address}   source MAC address               
-dmaddr {mac_address}   destination MAC address               
-maddr {mac_address}    either source or destination mac address               
-nameaddr {hostname}    name address (virtual host)               
-url {url}              matches URL text for http and ftp requests               
-prot {ip_protocol}     IP protocol (ie TCP, UDP, ICMP)               
-mprot {mac_protocol}   MAC protocol               
-dport {port}           destination port               
-sport {port}           source port               
-port {port}            either source or destination port               
-portrange {start end}  port range from startport to endport               
-priority {priority}    priority rule 1 to 10  (10 is highest)             
-global                 specifies a global rule

Throughout this document the above criteria will be illustrated along with details of their use. First, some basics.

Rule Precedence

Each rule has an "index", which is a number which defines its precedence. The lower the index, the higher the precedence. Rules are processed from low to high. Generally, once a rule "hits", processing stops and the action specified by the rule is carried out, with the exception of global and log-only rules. Use the -x parameter to specify an index:

bwmgr em0 -x 300 ...

specifies an index of 300. If you add a rule without specifying an index, it will automatically select an index for you at the end of your ruleset, with one exception......

Large Rule Indexes

Auto-indexing will not consider rules with indexes of 1 million or greater when determining the next highest index. This allows you to specify rules that MUST be at the end of your ruleset with high indexes that will always be at the end unless you manually place one higher. For example, suppose you have 2 rules:

bwmgr em0 -x 100 -addr 10.1.1.1 -bwboth 256000
bwmgr em0 -x 1000000 -ipprot allip -priority discard

if you were to add a rule without an index specified, the rule would be added at index 200, so your rule 1000000 would still be at the end as required.

Global Rules

Global rules are a new concept in v3.24. Previously, rule index precedence was absolute. With global rules, its possible for a data frame to match multiple rules, which is useful both for statistics and control. Global rules are set with the -global tag. As as example, consider the following:

bwmgr fxp2 -x 100 -ipprot tcp -global
bwmgr fxp2 -x 110 -ipprot telnet -global
bwmgr fxp2 -x 120 -ipprot 10.1.1.1

Now suppose a telnet packet originating from 10.1.1.1 is received. Without global rules, the packet would "hit" on rule 100 and stop there. But since 100 and 110 are global rules, in this case 100, 110 and 120 all will "hit".

Note that is makes no difference where the global rule is located in the ruleset; a global rule will always hit on a match, so the following:

bwmgr fxp2 -x 110 -ipprot telnet -global
bwmgr fxp2 -x 120 -ipprot 10.1.1.1
bwmgr fxp2 -x 300 -ipprot tcp -global

would make no difference; all 3 rules would "hit". Note however the following case:

bwmgr fxp2 -x 300 -addr 10.1.1.0 -addrmsk 255.255.255.0 -global
bwmgr fxp2 -x 310 -addr 10.1.1.1
bwmgr fxp2 -x 320 -ipprot tcp

In this case, the telnet packet would only hit on 300 and 310, since rule 320 is not global. Only the first non-global rule will hit, and it does so by rule precedence.

Naming Entries

Names function as aliases to rule numbers and are used to uniquely identify rules (out of the scope of the interface) for links with other rules. Names must be unique for the entire system. If you will be graphing the data from a particular rule (see the -stats option below) you MUST name it. Also note that names are case insensitive in the database, so if you have rules named "ACME" and "acme" they will be considered the same for graphing (and it will cause problems). Groups must also be named, as must reverse rules. In order to modify or view a particular rule, you need to specify the index number or the name that is assigned to the rule when you create it. To make things easier, you can name entries and use the name to modify or refer to specific rules without having to know the index number. To name an entry, use the -name option:

bwmgr em0 -name GenMotors -x 300 ...

the above entry will associate the name, GenMotors with this rule 300. So, for example, if you wanted to modify this rule in the future, you could simply do the following:

bwmgr em0 -name GenMotors ...criteria...

Note that names must be no more than 16 characters long and cannot include spaces or any other special characters other than a dash or underscore. You also should not use periods.

Specifying Hosts and Network Addresses

You can also specify host addresses and networks with bwmgr to limit the amount of bandwidth that a specific host or network could use. This is done as follows:

bwmgr em0 -addr 207.11.14.1 -bwout 56000 -bwin 56000

This would limit the traffic from and to host 207.11.14.1 to 56000bps. Similarly, an address mask can be used to specify an entire network or sub-network:

bwmgr em0 -addr 207.11.14.0 -addrmsk 255.255.255.0 -bwout 56000 -bwout 56000

This would limit the bandwidth of all of the devices on network 207.11.14 to an aggregate of 56000bps. You can also use host/domain names for specific hosts. For example, to limit traffic of a specific host by name:

bwmgr em0 -addr www.sumdomain.com -bwout 128000 -bwin 128000

Note that if you use internet host.domain names, the name is simply resolved to a host address. So if you do

bwmgr em0 -addr domain.com ...

the rule will be equivalent to

bwmgr em0 -addr 10.1.1.1 ...

assuming that domain.com resolves to 10.1.1.1. No attempt is made to resolve numbers to hosts, so in your configuration files the numeric IP address will be used.

MAC Address Rules

You may want to lmit an entire physical machine, or a customer by his router's MAC address. Assuming that you know the MAC address of the specific machine, you can specify the MAC address using the -maddr option in the same way that you can limit by IP address. For example:

bwmgr em0 -maddr 00:00:c0:a2:ca:bc -bwin 64000 -bwout 64000

The above example would limit all of the traffic to and from the host with MAC address 00:00:00:c0:a2:bc to 64kb/s. MAC rules match all of the traffic from or to the specified MAC address, including non-IP traffic. Alternatively, you can use -smaddr and/or -dmaddr to limit to or from or to control a specific connection. For example:

bwmgr em0 -smaddr 00:00:c0:a2:ca:bc -dmaddr 00:00:c0:01:01:01
bwmgr em0 -maddr 00:00:c0:a2:ca:bc -bwin 64000 -bwout 64000

The above rule pair would limit all of the traffic for MAC 00:00:c0:a2:ca:bc except traffic going from this address to 00:00:c0:01:01:01.

Grouping Bandwidth Entries - Creating a Group

If you want to define a disjoint group of addresses, you can use the "Group" feature to create a group. To define a group, define a named group entry, and then link the other entries to that name of the head entry. For example:

bwmgr em0 -name losers -group -bwboth 128000
bwmgr em0 -addr 207.11.1.4 -bwlink losers
bwmgr em0 -addr 207.11.17.2 -bwlink losers
bwmgr em0 -addr 207.11.17.3 -bwlink losers

the above would create a group with the 3 specified addresses which would all have the same aggregate bandwidth limits, as if they were all in one rule. . Note the use of the -name parameter for the initial (group) entry. You must use the name of the entry when linking to a group head (previous versions allowed the index to be used). Using the name allows for group members to be on different interfaces. Note that if you delete the Group entry the other entries will continue to exist with an unattached link, and they will re-attach when a group rule with the proper name is added. You should avoid deleting group entries when possible, as the remaining rules will work differently (in the above case they will get unlimited bandwidth) when they are not in the group.

More powerful use of groups is discussed in the Group Bandwidth Management section below.

Managing Traffic by Data Type with bwmgr

Traffic can be identified by "traffic type" as well as by previously described IP or MAC addresses. Traffic types can also be specified along with addresses to identify specific traffic sent by or to specific hosts or customers. Traffic types supported are:

Mac Protocol
IP Protocol
TCP or UDP Port
Specials

We will describe each below.

MAC Protocol Rules

The "MAC protocol" is the "ethernet type" in an ethernet frame. While virtually all of your "internet" traffic is IP, you may have many other types of traffic on your LAN. If you have co-located customers, if you are bridging your traffic to remote locations or if you just want to control certain types of LAN traffic, the MAC protocol can be used to identify such traffic. The syntax for specifying MAC traffic is as follows:

bwmgr em0 -mprot Protocol

Where "Protocol" is the hexidecimal representation of the ethernet type (ie 0x800 or 800 for IP, 0x806 or 806 for ARP, etc). The more common MAC types (IP, ARP, IPX) may be specfied by keyword.

There are 2 special types. First is "-mprot vlan", which matches all vlan tagged traffic. Note that such rules will only match vlan encapsulated traffic that contain the vlan header. The second is "-mprot 802.3", which is "typeless" ethernet traffic. Some older protocols (and IPX implementations) may specify the length instead of the protocol type. Specifying 802.3 will match all such traffic, regardless of the higher level protocol.

VLAN (VPN) Rules

You can also Match based on the vlan tag (ie the vlan ID number) using the -vlan option. For example:

bwmgr em0 -vlan 128 -bwboth 256000

will limit traffic containing vlan header info for vlan 128 to 256000. Note that -vlan implies -mprot 0x8100 and that any other setting of -mprot is not valid.

IP Protocol Rules

All IP traffic has an IP protocol. The most common IP protocols are TCP, UDP and ICMP, but there are many more. Most likely, you will not encounter anything different than the "big 3". Other IP protocols are listed in the file /etc/protocols on most unix-like installations.

To match traffic with a specific IP protocol, use the -ipprot (legacy -prot is also supported). The following are examples of how to match "the big 3":

bwmgr em0 -ipprot UDP
bwmgr em0 -ipprot ICMP
bwmgr em0 -ipprot TCP

Note that port-defined and special internally defined protocols can also be specified with the -ipprot setting. These are discussed in the protocol engine section later in this document.

IP TOS Rules

You can use a mask to match bits in the TOS field of ipv4 packets with the -iptos option. Since TOS is used in many different and sometimes non-standard ways, it is your responsibility to set the mask according to whatever scheme is in place on your network. Note also that this HEX value Matches the entire TOS octet, meaning both TOS and priority fields. For example:

bwmgr em0 -iptos 0x18 -gpriority 8

The above rule will set the global priority (explained later) to 8 for any packets containing either the LOW_DELAY (0x10) or THROUGHPUT (0x08) bits. This implies that 0x10, 0x08 and 0x18 and 0x1E will all Match.

IP Protocol Port Rules

While you *can* use ports in rules to match traffic, its more efficient to use protocol mappings than to use ports. Virtually all port matching has been antiquated by the v3.3 protocol engine. You can still use -port to match a port if the protocol isn't defined, but its recommended that you use -ipprot in most cases.

"Special" Protocol Definitions

There are several "special" ip protocols defined that arent really IP protocols at all, but they can be quite useful. "Special" protocols are specified with the -ipprot syntax, although some of them actually combine protocols and ports. They represent special cases that are common and they are not definable by protocol and port alone:

-ipprot tcpconnect	matches tcp/ip packets with SYN set and ACK not set
-ipprot tcpconnect	matches tcp/ip packets with SYN set and ACK not set
-ipprot tcpsyn	matches all tcp/ip packets with SYN set
-ipprot tcpack	matches tcp/ip packets with ACK (but NOT PUSH) set
-ipprot tcppush	matches all tcp/ip packets with PUSH set (ie, that contains data)
-ipprot ipfragments	matches all IP fragments
-ipprot allip	matches all IP traffic
-ipprot ftp	matches ftp command and transfers with port discovery
-ipprot http	matches all http AND https traffic
-ipprot http-text	matches http traffic with a content-type of text
-ipprot http-image	matches http traffic with a content-type of image
-ipprot http-download	matches http traffic with a content-type force-download or octet-stream
-ipprot http-javascript	matches http traffic with a content-type javascript
-ipprot http-flash	matches http traffic with a content-type of flash
-ipprot http-cached	matches all http traffic marked as a "HIT" from a web cache
-ipprot smtp	matches SMTP traffic
-ipprot p2p	matches supported point-to-point protocols (BearShare, Morpheus, Kazaa, etc)*
-ipprot gnutella	matches gnutella based p2p protocols traffic
-ipprot kazaa	matches kazaa based p2p protocols traffic
-ipprot audio_galaxy	matches audiogalaxy transfer requests
-ipprot dc	matches directconnect (dc++) traffic
-ipprot bittorrent	matched bittorrent traffic
-ipprot im	matches instant messaging traffic (MSN, AIM, Yahoo, Match.com)
-ipprot aim	matches AOL Instant Messenger traffic
-ipprot yim	matches Yahoo Instant Messenger traffic
-ipprot msn_im	matches MSN Messenger traffic
-ipprot sip	matches SIP (session initiation protocol) messages
-ipprot voip	matches VoIP (voice over IP) audio streams
-ipprot skype-voip	matches Skype v1 VoIP connections

*These are described in detail in "Advanced Topics" below

Note that user-defined protocols can be added to this internal list of protocols using the "loadprotocols" and /etc/bwmgr-protocols file. See a description of the protocol engine later in this document.

URL Rules

URL rules are special rules that look into HTTP and FTP frames and seek requests for specific pages or files. URL rules specifically match 2 types of traffic: HTTP GET or POST requests for specific URLs and FTP RETR requests. These can be useful in filtering known attacks (ie default.ida or root.exe) and also for controlling downloads of specific pages or specific types of pages. Suppose Microsoft just released IE 12.0. You could limit only downloads with a "-url iesetup12" rule possibly. Or "-url .exe" to limit all excutables.

URL rules with "track" any session that is encountered using the URL. So, for example, if you wanted to limit the download or tarballs, you could use:

bwmgr em0 -url .tgz -bwin 64000 -bwout 64000

This would be all that would be necessary to limit the entire download of any file containing ".tgz". ".mp3", ".mpg", ".avi", ".wmv" are other useful choices.

URL rules may contain spaces. If a url contains spaces, it must be enclosed with quotation marks:

bwmgr em0 -addr 10.1.1.1 -ipprot smtp "foo bar;

Its also notable that url rules can be useful as triggers for firewall deny type rules. Suppose, for example, the old "form.pl" virus were to appear again. You could block any such use with:

bwmgr em0 -fw -url form.pl -priority fw-deny

There is a setting urldecode "URL Decode" that can be enabled to cause the system to read the parameters string as well as the url. So if you wanted to block an exploit such as "page=http://www.url.com", with URL decoding enabled, you could use:

bwmgr em0 -fw -url "page=http" -priority fw-deny

To set url decoding, use the command:

bwmgr urldecode on|off

where on enables and off disables.

Virtual Host Rules

If you are using name virtual hosts you may have many virtual hosts assigned to the same IP address, in which case limiting the IP address may not be flexible enough. You can use the -nameaddr option to specify a host by name. Note that this feature uses the HTTP header information to determine which host is being accessed, which clearly implies that it will only work on http traffic (AND HTTP 1.1 headers are required). For example:

bwmgr em0 -nameaddr www.thishost.com ...

The above rule will match http traffic with a host header specifying www.thishost.com. Note that the entire string must match. So if this host could also be addressed as "thishost.com", you could use "-nameaddr thishost.com" which would match both "www.thishost.com" and "thishost.com", but also "anthingelse.thishost.com". -nameaddr rules are special rules which use internal stream tracking to "keep track" of the traffic associated with a name address rule. There is no need (in versions 3.22n and later) to use reverse rule syntax with these rules.

Enabling and Disabling Rules

You may want to disable a rule temporarily to allow someone into your network or allow someone out. To disable a rule, use the following syntax:

bwmgr em0 disable ####
bwmgr em0 disablefw ####

where #### is the rule number to disable. disablefw is used to disable firewall rules. Note that disable internally flags the rule as disabled, and you must use enable to re-enable the rule (unless you delete it and re-add it of course). To reenable a rule or to change its I/O flags, use the enable keyword:

bwmgr em0 enable 1000
bwmgr em0 enable 2000 i
bwmgr em0 enable 3000 o
bwmgr em0 enable 4000 io

for the above commands, rule 1000 will be enabled as Incoming and Outgoing (the default), rule 2000 as incoming only, rule 3000 as outgoing and rule 4000 as both. There is no danger in enabling a rule which has not been disabled.

Enabling and Disabling Rule by Time of Day

Rules may optionally be given an "enable time" and "disable time", allowing you to specify when the rule should be active. You may want rules that are only active during certain hours of the day, or only during the week. The default setting is for a rule to be active all of the time. To set up other times, use the following syntax:

-timeon XXX-H-0 -timeoff XXX-H-0

where XXX is either ALL (for every day), WEO (for weekends only) or WDO (week days only), and H is the hour of the day in military time (0-23). Only full hour settings are currently supported, so the 3rd parameter must be set to zero. As an example:

bwmgr em0 -x 100 -addr 10.1.1.1 -timeon ALL-8-0 -timeoff ALL-17-0

The above setting would enable this rule at 8AM and disable it at 5PM every day. Note that the XXX setting must be the same for both -timeon and -timeoff. Also note that both must be used, as one has no meaning without the other.

Time management is performed by the bwmgrd daemon, so it must be running in order for rules to be enabled and disabled.

Changing Bandwidth Settings

If you just want to change the bandwidth settings on a rule, you can use the -changebw to indicate that you are changing the bandwidth on an existing bandwidth rule (don't use this on a rule that doesn't have a bandwidth setting). When you issue a rule by name or index, if there is an old rule with the same name it is actually deleted, and then the new rule is added. If you are gathering stats, this can disrupt the counts and cause some information to be lost. The -changebw option will change the -bwin, -bwout, -bwboth, -bwmin and -bwburst settings safely without affecting any other settings or indexes.For example, suppose you had the following rule in place:

bwmgr em0 -x 3000 -addr 100.1.1.1 -ipprot tcp -port www -bwin 64000 -bwout 64000

and you then issued the following:

bwmgr em0 -x 3000 -changebw -bwin 128000 -bwout 128000

the resulting rule would be

bwmgr em0 -x 3000 -addr 100.1.1.1 -ipprot tcp -port www-bwin 128000 -bwout 128000

Note that all of the matching criteria (address, protocol, port) remain.

Notes on Logging

Any rule can be given the "-l" option to enable logging. Note that logging requires that information be written to the system logging system and that the information be written to a file. While logging can be a powerful tool for debugging or intruder detection, you should be aware that excessive logging can and will eat up your systems resources. If, for example, you set a bandwidth rule for a host and enable logging, every single frame that goes to or comes from that host will be logged, which is unacceptable. More acceptable uses of logging might be as follows. Suppose you wanted to know what connections a specific employee of yours was making. You could use the following:

bwmgr em0 -addr 207.14.172.1 -prot tcpconnect -l

The above rule would log all TCP connect requests from host 207.14.172.1.

The information from rules with logging enabled will be sent to LOG_INFO in FreeBSD (kern.info in syslogd) and KERN_DEBUG (kern.debug in syslogd) in LINUX. "Usually", the defaults for these are the file /var/log/messages.

The ET/BWMGR Protocol Engine

Version 3.3. introduces a completly remodeled internal protocol engine. All port to protocol mappings are under user control, and groups of ports are maintained internally in a much more efficient manner than using external rules. This allows you to customize the port to protocols mapping to fit your particular needs.

Internal Protocols:

Many protocols are defined internally, and you can use any internally-defined name in a rule. To get a list of available protocols, use the command:

bwmgr getprot all

This will display all of the protocols defined by the system.

User Defined Protocols: /etc/bwmgr-protocols

Users can define protocols by manipulating the /etc/bwmgr-protocols file. This file is included in your system distribution with many well-known default settings. Using the protocols file, you can define which ports should be mapped to a specific protocol. This allows you to group ports in a much more efficient manner than using groups of rules. The syntax for /etc/bwmgr-protocols is:

port tcp||udp protocol-name

so to define the protocol "myprot" on udp port 11472, you'd use:

11473 udp myprot

Now suppose that ports 11474 and 12800 also were used in this protocols, you could add the lines:

11474 udp myprot
12800 udp myprot

and now when you used the rule:

bwmgr em0 -x 5000 -addr 11.1.1.1 -ipprot myprot -bwprofile lotsabandwidth

the rule would match on ports 11473, 11474 and 12800.

The /etc/user-protocols files is the same format as the bwmgr-protocols file. Its recommended that you define your protocols in the user-protocols file, so that you can use the standard protocols that we include with each release. We'll be updating bwmgr-protocols regularly so any changes you have made with be lost if you want to use our newer file.

The loadprotocols command

To read protocols into the bandwidth manager system, use the loadprotocols command as follows:

bwmgr loadprotocols

This command is included in your rc.bwmgr file when you do a rebuild or when the GUI rebuilds the file. loadprotocols reads 3 files in the following order:

/etc/protocols
/etc/bwmgr-protocols
/etc/user-protocols

/etc/protocols defined your actual IP protocols (the ip_p field in an IP packet). bwmgr-protocols and user-protocols read user-defined port mappings as decribed above. Note that later settings will override previous settings, so if you have:

1000 tcp http

in bwmgr-protocols and

1000 tcp someprot

in user-protocols, port 1000 will be mapped to someprot.

Mapping Agents to Protocols

Anyone hosting a web site knows about crawlers and exploits. The ET/BWMGR allows you to "map" user agents to protocols so you can manage them more easily. Managing user agents in the BWMGR allow you to create central policies for blocking bots, crawlers and exploits without having to create complicated rules for each web site in Apache or by using .htaccess

You can create user agent protocol mappings in the file /etc/bwmgr_agents. Below is an example:

protocol goodbots bots
protocol badbots bots
# syntax
# agent protocol priority
#
mozilla http 0
googlebot goodboots 10
slurp goodbots 10
twiceler badbots 10
gecko http 5

Once you have your file with settings, run the command:

bwmgr loadagents

After running this command, datastreams with the specified user agent will be mapped to the protocol specified. The purpose of the priority is to specify the importance of the string. Take the following agent string:

Mozilla/5.0 (compatible; Yahoo! Slurp/3.0

This might be both Mozilla and Slurp, however Slurp is more specific. So you want to make sure it checks the more specific ones first. By giving Slurp a higher priority, the above agent will map to goodbots instead of http.

To see the currently defined agents, use the command:

bwmgr getagent all

Mapping URLs to Protocols

You can use the same principle describe above to map parts of urls to protocols. This is particularly useful for catching exploits on a network-wide basis.

You can create URL protocol mappings in the file /etc/bwmgr_urls. Below is an example:

protocol exploits
protocol protected
# syntax
# url protocol priority
#
phpmyadmin protected 10
=http exploits 10
passwd exploits 10

Once you have your file with settings, run the command:

bwmgr loadurls

As with agents, the priority is to specify the importance of the string.

To see the currently defined urls, use the command:

bwmgr geturl all

Beginning with v3.2, the ET/BWMGR software incorporates a firewall function which is separate from the bandwidth management and priority rules. Each manageable interface on your system can optionally have a firewall, which can help you to control the flow of traffic to and from your network.

The purpose of a firewall is to decide which packets are allowed to pass through the network and which packets are not allowed. The default is to allow all traffic. So, if you have no firewall rules enabled, all traffic will pass through the firewall. Typically, you will add rules for traffic that you want to "firewall", or filter, from getting into your network. For example, suppose you don't want anyone on the outside world to access your server whose IP address is 207.11.14.1. You could do the following:

bwmgr em0 -fw -addr 207.11.14.1 -priority fw-deny

the above syntax selects interface em0 (which would have to be the interface connected to the outside world), uses the -fw flag to specify a firewall rule, and selects the address as the match criteria. The "priority" for firewall rules can be fw-deny, fw-allow or log-only. log-only rules will be discussed later. For now, you can either set a rule to allow, or to deny passage through the firewall. The above rule will cause all traffic to or from 207.11.14.1 to be discarded. This results in this address being inaccessable from outside of interface em0.

Setting up the Firewall - Theory of Operation

bwmgr em0 -fw -addr 207.11.14.1 -priority fw-deny

Firewall Criteria

Firewall criteria can be selected based on IP address or MAC address (source, destination or either), port, port range or protocol. Combinations are also allowed, so you can do thing like "allow all traffic if the source IP is X and the source MAC is Y, or allow traffic from IP A to IP B but not from IP C to IP B. You can use combinations of rules to achieve your more complex goals. For example, suppose you wanted to deny access to all ports over 1024 on a particular system but allow access to port 4718. You could use the following:

bwmgr em0 -x 99 -addr 207.11.14.2 -port 4718 -priority FW-Allow
bwmgr em0 -x 100 -addr 207.11.14.2 -portrange 1024 65535 -priority FW-Deny

Because rule 99 is before rule 100, it will be checked before rule 100, so you have effectively excluded port 4718 from the range in rule 100.

Firewall rules use the same criteria as all other rules, except that bandwidth management and statistical parameters are not allowed.

Viewing Firewall Rules with bwmgr

Use the "showfw" command with bwmgr to display firewall rules:

bwmgr em0 showfw

Packets Per Second Firewall Rules

A special firewall rule allows you to limit the packets per second (PPS) of any type of traffic to prevent overloading of your network and also to twart DOS (Denial of Service) attacks.When the PPS of the traffic matching a PPS rule exceeds the setting, all addition traffic will be dropped for the current time window. The syntax of a PPS rule is specified with the bwmgr utility with the -ppsin -ppsout -ppsboth options

bwmgr em0 -fw -ppsin 5000 -ppsout 5000 -priority fw-deny

The above setting would limit the PPS incoming to 5000pps and outgoing to 5000pps individually. To limit the combined PPS, use the -ppsboth option instead:

bwmgr em0 -ppsboth 7000

You can also limit the PPS of specific types of traffic, or traffic to or from a specify host. For example:

bwmgr em0 -ipprot ICMP -ppsin 1000

The above rule would limit imcoming ICMP packets to 1000pps, which would help thwart Ping attacks or other ICMP indications caused by other types of attacks.

Positioning of Packets Per Second Rules

PPS firewall rules monitor activity but will not cause a "hit", which means that other rules after them in precedence will continue to be checked. However, if they are placed after other rules that "hit" then they will not be checked. So your PPS rules must be at the beginning of your firewall ruleset, and should be before all other rules.

Packets Per Second Suggestions

You may not know what "safe" or "normal" PPS levels are on your network, so you should first use "benign" pps rules to monitor your network activity before setting your FW-Deny rules. "Benign" rules are PPS rules with a specification of FW-Allow, which means that they dont do anything even if the the specified level is exceeded. Also, when displaying PPS rules with the GUI or with "showfw", the maximum level of packets per second encountered is displayed. By using a benign rule for a few days, you could check to see what the maximum PPS levels are on your network over that time period. For example:

bwmgr em0 -ppsin 50000 -ppsout 50000 -priority fw-allow

sets a rule for monitoring overall packet per second on the interface. You must set the values to a number that is larger than you ever expect to see, as you will not receive counts when it hits this value. When you display this rule, it will display the "PPS-HIGH" value, which is the max number of PPS that were received since enabling the rule.

#bwmgr showfw 1

1 PPS-IN: 50000 PPS-OUT: 50000 PPS-HIGH: (1278/640) Priority: FW-Deny (1234321/0) <IN OUT>

When displaying the rule, it shows the setting and also indicates that the high PPS count encountered was 1278 packets per second incoming, and 640 outgoing. You can use this info to determine the normal high levels present on your network.

Log-only Firewall Rules

Firewall rules can be given a priority of "log-only", which means that a log entry will be written when the rule matches, but subsequent rules will still be checked. It is important to position log-only rules before allow/deny rules, because once an allow or deny rule hits, then no more rules will be checked. A good strategy is to put all of your log rules at the beginning.

Example:

bwmgr em0 -x 10 -fw -prot tcp -port snmp -l -priority log-only

note that you must specify -l to enable logging, it is not automatic.

Reverse Firewall Rules

Reverse firewall rules allow you to use a "trigger rule" to block host addresses based on some action. There rules are described in detail in the reverse rules section below.

Setting up Bandwidth Management Rules

Setting Bandwidth Rules and Limits with the bwmgr utility

Bandwidth managment rules allow you to customize the way that certain traffic flows through your network. The following section describes how to use the bwmgr utility to set up your bandwidth management criteria.

Shared vs Allocated Bandwidth

All bandwidth that is not guaranteed is considered by the ET/BWMGR to be "shared" bandwidth. What this means is that bandwidth that is not used by one rule or policy may be used by any other rule on a demand basis. Guaranteed bandwidth must be allocated for a specific policy and will not be available to other policies.

Allocated Bandwidth vs Minimum Bandwidth

"Allocating" bandwidth is a meaningless concept unless there is a finite pool from which to allocate from. If a rule with a minimum bandwidth allocation is a member of a Group, the bandwidth must be "allocated" from the Group's bandwidth pool, in order to guarantee that other members can't use it. A rule that is not a member of a group will have its bandwidth allocated from the pool of bandwidth available to the Interface. The Group or Interface is said to be the rule's "Container", because it defines the amount of bandwidth available to all of the "members" within its scope. If there is no bandwidth allotment for the rule's Container, then the bandwidth can not be "Guaranteed", because there is no reference to the amount of bandwidth available. In this case, the "guaranteed" bandwidth is simply a minimum bandwidth setting.

Auto-Shaping

Version 3.23 adds an "auto-shaping" feature designed to automatically manage network congestion without any "rules" whatsoever. Auto-shaping can be used alone or in co-ordination with any other settings.

To enable auto-shaping, simply set -autothresh on an interface:

bwmgr em0 -ifac -autothresh 5800000

Typically you would set this to just below your upstream capacity. So if you have 6Mb/s of upstream bandwidth, set the threshold to 5800000. Whenever your network traffic is over the threshold, the auto-shaping mechanism will "kick in" and begin slowing all of the traffic on the network in a very "fair" manner.

Note that auto-shaping is a pacing mechanism only and will not apply any hard limit or delay any traffic. So your bandwidth will likely be higher than the threshold, but your congestion (ie backup traffic, queue depths, etc) will be much less.

Setting Hard Limits - Allocating Shared Bandwidth with a Rule

The simplest type of bandwidth management rule is the hard limit. Hard limits set a ceiling on the amount of data that a host or type of traffic can use, and will regulate it at the ceiling when operating over the limit. This effectively simulates any data rate you choose regardless of the medium. Limits can be set for incoming or outgoing traffic, or for incoming and outgoing combined. For example:

bwmgr em0 -x 500 -addr 100.1.1.1 -bwin 56000 -bwout 128000

This command sets a bandwidth limit of 56000bps for incoming traffic and 128000bps for outgoing traffic. Remember that "incoming" traffic is traffic received on the interface, so if you have a 2 interface system it will be the opposite on the other interface. To set a combined setting, use -bwboth as follows:

bwmgr em0 -x 500 -addr 100.1.1.1 -bwboth 64000

This setting will allow the total of incoming + outgoing traffic to be no more than 64000bps.

Minimum Bandwidth Allocation

"Allocating" bandwidth can be achieved by setting a bandwidth minimum, which defines the minimum data rate that a host or traffic type will get before any action will be taken on the data stream. When the rules container (ie Group tag rule or Interface) has a bandwidth setting, this setting will represent "guaranteed" bandwidth. Without an interface or Group setting, this value represents bandwidth that will be considered high priority. For example:

bwmgr em0 -ipprot udp -port 15100 -bwmin 64000

Suppose you have VoIP running on udp port 15100. The above setting will set the minimum bandwidth to 64kbs for the port, giving it the highest priority until the minimum is exceeded.

bwmgr em0 -ifac -bwin 1544000 -bwout 1544000
bwmgr em0 -ipprot udp -port 15100 -bwmin 64000

With the above 2 rules, the -bwmin represents a bandwidth guarantee out of the 1544000bps allocated for the Interface.

You can mix all of the bandwidth settings in a single rule. Note that -bwmin and -bwburst (discussed below) inherit the "combined" attribute if -bwboth is specified. So, for example, if -bwboth is specified, then -bwmin and -bwburst apply to the combined (incoming + outgoing) total, if -bwboth is not set, then it applied for each direction.

Burst Settings on the ET/BWMGR

The ET/BWMGR bursting feature allows 2 levels of limits for each defined rule. This allows you to throttle traffic in a precise manner only when a specific condition is met. In order to define a burst bandwidth allocation for the rule, you must define two parameters, the burst limit and the burst trigger.

The burst limit is defined by the -bwburst parameter for a rule. The "trigger" is defined by the -bursttrig parameter.

Defining a Burst Trigger

Any rule or interface can be a burst trigger. A "burst trigger" is a bandwidth level threshold that is used as a criteria for determining whether a particular rule is allowed to "burst" to its burst setting, or whether it should be limited to its "normal" setting. You define a particular rule as a trigger by setting a burst threshold for the rule. For example:

bwmgr em0 -ifac -burstthresh 1000000

This sets the interface em0 as a burst trigger. Any rule that defines fxp0 as their burst trigger will use their -bwburst setting when the traffic level on em0 is under 1Mb/s, and their regular bandwidth limit when fxp0 is above 1Mb/s. Similarly:

bwmgr em0 -x 3000 -name fools -group -burstthresh 800000

This rule defines a group with a burst limit of 800000bps. So any rule that defines "fools" as their burst trigger would burst when the traffic level of Group "fools" was under 800000bps. For example:

bwmgr em0 -x 100 -addr 211.14.1.1 -bwin 64000 -bwout 64000 -bwburst 256000 -bursttrig fools

The above rule specifies that traffic to and from address 211.14.1.1 will be controlled at 64000bps if the usage level of the group "fools" is over 800000bps, and that 211.14.1.1 will get 256000bps when "fools" is below 800000bps.

Setting the Maximum Burst Duration

The -burstmax parameter allows you to set a maximum length of time that a rule can burst. This is designed to prevent long downloads or uploads. With a burstmax setting of 10, for example, the rule with allow "bursting" (which is activity over the standard bandwidth setting) for 10 seconds, and then will not allow bursting again until the rule has activity under its normal bandwidth allotment. So with a continuous download, the rule will be limited to its regular bandwidth setting (non-burst) until the transfer is completed.

Changing The Burst Reset (unburst) Period

Rules using the -burstmax setting need a criteria for determining when the traffic has subsided adaquately. The default setting is 10 seconds. What this means is that once a rule has bursted for its max burst period, the traffic must slow below its non-burst limit for at least unburst seconds before its allowed to burst again. To change the unburst setting, use the following:

bwmgr unburst 20

Defining a Burst Limit for a Bandwidth Rule

Each rule can be given a -bwburst setting which defines the maximum amount of bandwith the rule can use when the burst trigger is below its threshold. When traffic levels on the burst trigger are above the threshold, bursting is disabled. When traffic levels are below the threshold, bursting is enabled. Note that starting in v3.21, the "default" burst is wire speed (ie no limit). In previous versions if you didnt specify a -bwburst setting for a rule, the default was to limit to the normal bandwidth setting all the time. Now you must explicitly set the -bwburst. For example:

bwmgr em0 -ifac -burstthresh 1000000
bwmgr em0 -x 800 -addr 100.1.1.1 -bwin 56000 -bwout 56000 -bwburst 512000-bursttrig em0
bwmgr em0 -x 900 -addr 100.1.1.2 -bwin 128000 -bwout 128000
bwmgr em0 -x 1000 -addr 100.1.1.3 -bwin 128000 -bwout 128000 -burstmax 20 -bursttrig em0

The above settings would set the burst threshold to 1000000bps on interface em0. The second setting (-x 800) would set traffic to and from address 100.1.1.1 to 56000bps when traffic on the interface is above 1Mb/s, and to 512000bps when traffic is below 1Mb/s. The third rule (-x 900) sets a hard limit on address 100.1.1.2 to 128000bps (no -bwburst setting). The third rule (-x 1000) allows bursting to wire speed (ie no bandwidth limit), with a below-threshold limit set to 128000bps, and a maximum burst duration of 20 seconds.

Note that bursting is effectively disabled for a rule if no burst trigger is defined.

Setting Asymetrical Burst Rates

If you want to set different burst rates from each direction, you will have to create two rules. Normally you can create a single rule which controls traffic in both directions, but bursting follows the "both" flag, that is its either the same for both directions or it uses a combined value. Typically you are really only controlling traffic in one direction (in for "client" traffic and out for "server" traffic), so it doesnt much matter. But if you want different burst rates in different directions, simply create an incoming rule with the "in" settings, and an outgoing rule for the "out" settings. For example:

bwmgr em0 -o -addr 10.1.1.1 -bwout 128000 -bwburst 512000 -bwmin 32000
bwmgr em0 -i -addr 10.1.1.1 -bwin 512000 -bwburst 768000

The above -i and -o flag tell the ET/BWMGR to only enforce the rule in the appropriate direction.

Setting a Burst Period

The "burst period" is the window within which the BurstManager will gather bandwidth information about a trigger and enable or disable bursting. Traffic usage is continuously averaged over the period. Longer periods may increase the amount of time that rules may burst, as the longer into the period high activity begins, the longer it will take the average to reach the threshold. Note that values under 10 seconds are not recommended.

Interface Level Bandwidth Control

You may want to limit a complete interface to a specific amount of bandwidth, regardless of the traffic type. You can set a bandwidth limit for the entire interface by specifying the -ifac option. For example:

bwmgr em0 -ifac -bwin 800000 -bwout 800000

The above commnad will set an overall bandwidth limit of 800000 in and out for the interface. This limit is absolute (except for pass-thru rules), so no more than 800000 would be allowed to pass regardless of other rules present. This implies that:

bwmgr em0 -ifac -bwboth 800000
bwmgr em0 -addr 10.1.1.1 -bwin 1000000 -bwout 1000000

with the above rules, host 10.1.1.1 would not be able to get more than 800000 even though its individual limit is higher.

Interface bandwidth limits are useful for defining how much actual bandwidth there is available at your network edge. For the purposed of priorities and bandwidth allocation, setting the outside interface (ie the adapter connected to your edge router's side of the network) to the bandwidth you have (such at 45Mb/s if you have a T3), or the maximum bandwidth you want to use. With an interface bandwidth limit, minimum settings become guaranteed bandwidth on the interface. Without an interface limit, you are only guaranteeing access to the local wire, which may not be what you want to do.

Note that if you have no rules on an interface, you could alternatively use:

bwmgr em0 -bwboth 800000

as a single rule which limits all the traffic on the interface without setting an interface limit.

TCP Rate Limiting

TCP rate limiting is a powerful bandwidth management technique that reduces the amount of traffic in your network and potentially slows each TCP session by reducing the TCP receive window. While some products on the market use rate limiting as their primary bandwidth management mechanism, there are undesireable results when windows are dropped too low, as overhead on your network in terms of the number of packets and bandwidth usage can significantly increase. Because of this, the ET/BWMGR uses rate limiting as one of several mechanisms to manage your network without having an adverse effect on overall traffic volumes.

Specifying a Minimum Window Size

When you set a rule with a bandwidth limit, the ET/BWMGR will use rate limiting as well as other techniques to slow the data rate to not exceed what you specify. However most servers do not honor window "granularity" in that they want to only send full windows of data if possible. Because of this, i f you are running a single session (say a download) on an IP address rule, you will see numbers lower than the limit. This is because the proper window setting for a session may be 1800, but the effective window is 1500 because of the full-window issue. If you have many sessions on a rule (such as a NAT'd client or a server), this is exactly what you want, as you want to force the sessions below the limit until a balance occurs. However if you need a single session to be able to attain the full bandwidth limit, you can specify a minimum widow to use so that the sessions will not be pushed below the point at which they can achieve full bandwidth throughput. For example:

bwmgr em0 -x 1000 -addr 10.1.1.1 -bwboth 512000 -tcpwindow 4500

The above rule will tell the rule to use not set a window below 4500 (which is 3 full packets). The ideal window size varies from connection to connection (depending on intrinsic delays), so there is no exact answer to what this setting should be. You can eliminate TCP window manipulation altogether by setting the window to 64000. While this may seem to be a good way to assure that any connection can reach any bandwidth limit specified, it also defeats one of the primary purposes of bandwidth management, which is to reduce the traffic flows in your network to reduce delays, avoid congestion and reduce the need to drop packets, so you need to weight your objectives carefully.

Per Session Bandwidth Management - Rate Limiting

TCP rate limiting effectively is a "per session" method of controlling traffic. Each TCP session within a container is automatically controlled. When you set a bandwidth limit, the default behavior of the software is to use an internal "adaptive window". The adaptive window algorithm attempts to reduce the window based on the traffic flows, setting it just where the traffic flows at the rate specified without excessive backup. The adaptive window will not go below 1 full packet (1500 bytes), so it will not increase the overhead on your network as some rate limiting products do.

You can effectively "shape" traffic without limiting it by creating a container with a bandwidth setting that is just below your available bandwidth. So for example, if you have 3Mb/s of bandwidth, you could rate shape all traffic with :

bwmgr em0 -ipprot tcp -bwlink GroupA -bwin 3000000 -bwout 3000000

While this may seem trivial, it in fact is quite powerful. The rule with Match all tcp traffic and limit it to 3Mb/s. But the effect of the adaptive window management is to slow each session equally to create overall fairness within the usage of the bandwidth limit. It will also maintain minimum flows, reduce the expectation that a drop will occur, and create a scenario with minimum delay for tcp traffic.

Of course window shaping only works on TCP traffic so you'll need other rules to control other traffic. But this illustrates how you can substantially improve your network flows with a minimum of controls.

Global Controls for Rate Limiting

By default, traffic is not shaped or limiting in any way unless you set a specific rule that matches the traffic type. In order to reduce overall traffic on your network, you could set an interface "default" tcp window, which would apply to all TCP traffic passing through the interface.

bwmgr em0 -ifac -tcpwindow 4000

This would establish a default minimum window of 4000 for all data frames passing through em0. This default is used if no other settings apply to the data frame. You can override the interface default with a -tcpwindow setting in an individual rule, however the interface window will also override the internal rate limiting parameters. For example:

bwmgr em0 -ifac -tcpwindow 4000
bwmgr em0 -x 1000 -addr 10.1.1.1 -bwboth 56000 -tcpwindow 1500
bwmgr em0 -x 8000 -addr 11.1.1.1 -bwboth 56000

With these 3 settings, address 10.1.1.1 would be limited to 56000bps and have a minimum TCP window of 1500, while address 11.1.1.1 would be limited to 56000bps and have a minimum window of 4000.

Packets Per Second Bandwidth Rules

Packet per second rules can also be used to control users. PPS rules create a loosely-defined bandwidth limit which adjusts for the possibility that many small packets can use up resources even at low bandwidth levels. Give that a maximum packet is (usually) 1500 bytes and a minimum is usually 60 bytes, you can use a pps rule to give a wide range of capability to an end user or application without giving them unlimited bandwidth. So the following:

bwmgr fxp2 -x 4000 -name AcmeCorp -addr 192.168.17.1 -ppsin 300 -ppsout 300

would limit the throughout of the specified address to 300 packets/second. This translates to a bandwidth limit of between 144000bps and 2.4Mb/s depending on the average packet size of their transactions. Note that there is no burst setting for packets per second rules.

Combining Global Rules for Bandwidth Management

Using global rules, you can set multiple control criteria for traffic that matches multiple rules. For example:

bwmgr fxp2 -x 1000 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwin 1000000 -bwout 1000000 -global
bwmgr fxp2 -x 1200 -addr 10.1.1.17 -bwprofile default

The above allows you to set a limit for a class C without having to create a group. Since rule 1000 is a global rule, it will be enforced in addition to rule 1200 for traffic matching .17. Previous to version 3.24, you could only have done this in a group.

Combined BW and PPS rules

Beginning with 3.24b, rules can have both bandwidth and PPS settings. This allows you to create a single rule that can control by both bandwidth volume and activity level.

bwmgr em1 -x 500 -addr 10.1.1.1 -ppsin 100 -ppsout 100 -bwin 512000 -bwout 512000

The above rule would limit address 10.1.1.1 to the EITHER 100pps or 512000bps. If all packets are 1500 bytes, a setting of 100pps could allow as much as 1.2Mb/s of bandwidth, which may be more than you want. So adding the bandwidth setting will ALSO cap the rule at 512000. This rule allows both criteria to be estabished in a single rule. With 60 byte packets 100pps allows for only 48Kb/s.

Using Bandwidth Profiles

Bandwidth profiles are templates that allow you to apply common settings to any number of rules.

What is a "Profile"

A profile is a bandwidth service description that can be referenced by name. The service components are as follows:

bandwidth in (may be pps also)
bandwidth out (may be pps also)
combined?
burst bandwidth
bandwidth guarantee
burst max (duration)
burst trigger
priority

Each "profile" has a name and a unique identifier, as well as time of day attributes. For Example:

The above profile defines a profile description named "default", which defines a profile of the same name. The Profile ID and Profile name are case-insensitive (ie Default = default). Typically, if the "When" field is "Always", it makes the most sense to have the profile ID the same as the profile name for simplicity. The bandwidth profile for this description is 256,000bps incoming, 256,000bps outgoing and no bursting. Note that all settings in profiles are K (not really Kilos, but 1000s, so 512 is 512000).

Defining Bandwidth Profiles

Profiles are defined from within the HTML GUI. See the ET/BWMGR HTML manual for info on how to define profiles.

Applying Bandwidth Profiles

To apply a profile to a rule, use the -bwprofile specifier.

bwmgr em0 -x 1000 -addr 192.168.17.11 -bwprofile default

The above would apply the settings of the current "default" profile to the rule, as if you had entered them individually. Note: In the CLI, do NOT specify any specific bandwidth settings in the same command with -bwprofile or you may get the wrong results. In the HTML interface, other settings are ignored when a profile is specified.

The Effect of Using Bandwidth Profiles

At first glance, bandwidth profiles might just appear to be a shortcut for entering rules. But they are much more than that. Profiles are a vehicle for managing large rule sets, and also for implementing large-scale SLAs (service level agreements). Firstly, profiles are not only applied when you create the rule. If you change a profile, ALL of the rules which reference the modified profile will be changed accordingly. So if you have hundreds of customers/users who are using the "default" profile, you can modify global behavior of all of them that reference a specific profile in a single operation.

Profile Transitions

Service levels often (and should) have a time component, and different settings may apply at different times of day. Using the "When" setting with profiles, you can define settings for different times of day:

The above settings define "night" and "early" times of day, with different profile settings for each. A profile is "applied" to all of the rules referencing it at the start time, which is 7AM for default_early and 7PM for default_night. Note that the stop time is not used for profile transitions, it is only a reference so that it can be determined if the profile is active at any given time of day. This implies that "holes" in your time settings are simply ignored: if you had set default_early to start at 7AM and end at 1PM, it still would be applied until 7PM because that is when the next transition takes place. Profiles cannot be turned "off", they are simple overidden by other transitioning profiles.

To take the service level implementation to the next level, you can define other settings that only apply on weekends. This requires that you set default_early and default_late to WDO (weekday only), and you define a third profile which references "default" to apply on weekends. For example:

Now the default service level has 3 settings, one for early times during weekdays, one for late times during weekdays, and one for weekends.

Implementing Tiered SLAs with ET/BWMGR profiles

A simple example of a tiered SLA (service level agreement) would involve defining profiles for various service levels, and then applying them to customers/users who subscribe to that level of service.

Using the above simplistic settings, you have 4 distinct levels of service that simplify the adding new customers.

bwmgr em0 -name cust_a -addr 192.168.17.1 -bwprofile default
bwmgr em0 -name cust_b -addr 192.168.17.2 -bwprofile default
bwmgr em0 -name cust_c -addr 192.168.17.3 -bwprofile silver
bwmgr em0 -name cust_d -addr 192.168.17.123 -bwprofile gold

Reverse Rules

"Reverse rules" are basically dynamic "Groups" that are created automatically by selection of a "trigger". The trigger can be any criteria, but is most usually a single packet which signals that a stream which needs to be controlled is being created. For example, suppose you wanted to limit "each host that accesses Server A" to 256kbs. You could use a reverse rule which triggers on a tcp_connect to Server A which would create a limit on that host (and optionally the port). Once this trigger is detected, you then know that a transfer is occuring on a specific address/port combination and a dynamic control could be put in place automatically.

Another example is a WWW server, where you may want to limit hosts accessing the site on a per session basis; that is to limit individual sessions to an address independently, either with a ceiling, a floor or on a shared, equal basis. This allows you so set up a rule which will limit EACH USER accessing a site to either a specific bandwidth or a percentage of the bandwidth dependent on how many users are currently accessing the site. Actually, reverse rules are better than limiting each session. Other products limit each TCP session to a host, but a host many open many simultaneous sessions (such as browsers typically do for graphics and text), so you cannot precisely limit a "session" because you dont know how many sessions there are. Many FTP clients will open multiple sessions to increase throughput, which defeats the "per session" limiting mechanisms. By limiting the source IP address, you are limiting the remote user no matter how many sessions are open.

Suppose for example you had a site with a video feed that you wanted to limit each user to 80K to avoid one user hogging all of your backbone bandwidth. You're not concerned with an upper limit for the entire site, you just don't want any single user to get more than 80K. You could use a reverse limit to achieve this very easily. For example:

bwmgr em0 -name VideoPlanet -daddr 211.14.1.7 -r -bwboth 80000

When a "reverse rule" is accessed (specified with the -r switch as shown above), the bandwidth management software will attempt to "spawn" a new rule in an available rule slot directly before the reverse rule. So, for example, if the name address rule index is 5000, the first dynamically learned rule would be inserted at 4999, then 4998 and so forth. If a slot is occupied by another, non-associated rule, then the new rule will not be created. This implies that you must leave rule space before the reverse rule in order for the rule to be created. If no slots are available, the rule index plus 1 (NOT the next rule, it MUST be 1 index higher than the target rule) will be enforced. For example:

bwmgr -x 4800 -addr 10.11.1.1
bwmgr -x 5000 -name reverse_test -daddr 211.14.1.7 -r -bwboth 56000
bwmgr -x 5001 -daddr 211.14.1.7 -bwboth 56000

The above allows for 199 dynamic entries between 5000 and 4800. Once all of the slots are filled, no more new rules will be created. With the above settings, if more than 199 users were connected to the site at one time, all additional users (for which new rules could not be created), would be limited to a total of 56000 (as rule 5001 would then be enforced). This implies that the 200th user is actually the sum of all of the users over 199. Note that rule 5001 above is also a regular rule, so make certain to give it match criteria, typically the same as the target reverse rule without the reverse specifier.

Note that reverse rules must be named. Reverse rules create pseudo-groups, so the link to the head entry needs to have a name associated to create the proper linkage.

By default, reverse rules only limit the source address of the host accessing the trigger. So with the above rule, if host 192.168.0.1 accessed the server at 211.14.1.7, the following rule would be created:

bwmgr -x 4999 -addr 192.168.0.1 -t -bwboth 56000

If you don't want the entire IP limited, you could specify -usesport or -usedport indicate that the source or destination port should be used. -usesport indicated that the source port should be used, while -usdport specifies the destaination port. For example:

bwmgr -x 5000 -name reverse_test -daddr 211.14.1.7 -usersport -r -bwboth 56000

The above rule tells the ET/BWMGR to also use the source port when creating the dynamic rule. So now if 192.168.0.1 accesses the server, the following might be created:

bwmgr -x 4999 -addr 192.168.0.1 -port 41003 -t -bwboth 56000

The negative of using the port also is that typically, a browser will open up multiple sessions when accessing a web site, so for each access you may see 3 or more rules created. This increases system overhead (as the dynamic rules have to be created and deleted, and you will have more overally rules in the ruleset), and further limits the number of "actual" users you can limit. If you have 1000 slots available, for example, you may only be able to control 300 users, so you'll need to make the proper adjustments.

To control which address is used, there are several controls available. As noted before, the default is to use the source IP address. Below are the controls:

-usesaddr   Use the Source IP Address               
-usedaddr   Use the Destination Address
-usesport   Use the Source Port
-usedport   Use the Destination Port
-useprot    Use the Protocol               
-r          Reverse IP Address Rule               
-rm         Reverse MAC Rule

The -r or -rm controls whether the MAC or IP is used, and the -use?addr controls whether to use the source or destination IP address, and the -use?port control whether or not the port should be included. -useprot is useful for creating reverse rules for pseudo-protocols such as p2p. For example:

bwmgr em0 -x 3000 -ipprot p2p -bwin 64000 -bwout 64000 -usesaddr -useprot

Be careful when using MAC rules that you understand what your traffic looks like, and that you don't block or limit your gateway if thats not your intention.

A consideration with reverse rules is what happens when a dynamic rule cannot be created. The most common reason being that there are no more slots available, or -maxlinks has been reached. What happens is that the packet not acted upon.

Reverse Rule Modifiers

There are several new settings associated primarily with reverse limiting. The first allows finer control of how many users can access a site. This is independent of the discussion above. If you wanted to limit a site to 50 accesses, you could use the following:

bwmgr -x 4800 -addr 10.11.1.1
bwmgr -x 5000 -name reverse_test -addr 211.14.1.7 -r -bwboth 56000 -maxlinks 50

So even though there are 199 slots available before rule 5000, only 50 dynamic rules will be allowed to be created.

A second concept is the duration in which a dynamically created rule will remain active. You don't want rules to continue to be in the table long after a user has gone away. The default is 5 minutes of inactivity before the dynamically created rule will be deleted. Using the -ruletmo switch, you can specify a timeout for each rule. For example

bwmgr em0 -name reverse_test -addr 207.11.14.1 -bwboth 56000 -r -ruletmo 30

The above specifies a timeout of 30 seconds, which may be more appropriate is using ports with a web server (since each page access generally uses a new port).

The normal "timeout" for a rule is an idle timeout. This means that when no matches occur for x seconds, the rule will be removed, where x is the timeout. Of course in a situation where there is continuous traffic, the rule may never timeout. If that is a problem, you can use -fixedtmo instead of -ruletmo. The -fixedtmo setting will cause the rule to be removed x seconds after it is created. For single trigger events (such as access to a port that is later redirected), this may not work properly, as the "event" may not occur again while the session may continue.

Reverse Rule with a Bandwidth Ceiling

The default reverse rule setting (-r) will cause the dynamically created rules to inherit the settings of the root rule. In the case that you wanted all of the "members" created by the rule to share a particular amount of bandwidth, you need to use the -rc setting, which specifies that the bandwidth specified in the reverse trigger rule is to be shared among all of the resulting members. For example:

bwmgr em0 -name reverse_ceiling -ipprot tcpconnect -port ftp -daddr 17.1.1.1 -bwboth 64000 -rc

The above rule specifies that anyone that connects to the ftp port on host/server 17.1.1.1 will be limited to a total of 64000bps.

Be careful not to misuse the -rc setting. This setting is really only useful if the target rule is an event trigger (ie a connect, or a name address), otherwise you will cause a lot of rules to be created when you don't need them. For example:

bwmgr em0 -name bad_setting -port smtp -bwboth 64000 -rc

The above is a bad choice, as it will cause all smtp (mail) to be limited to 64000bps by creating a dynamic rule for each source. However, without the -rc the single rule would do the same thing, without the added overhead associated with adding and deleting dynamic rules.

Balanced Reverse Limiting

"Balanced" reverse limiting allows you to set a "ceiling" on the amount of bandwidth that a reverse rule can use, and attempts to achieve "fairness" in controlling the hosts in the dynamic group. For example, suppose you wanted a site to be limited to 128000bps total but did not want to allow one or two users to dominate the bandwidth? Using balanced reverse limiting, each user accessing a site gets an equal percentage of the available bandwidth. So, if you have only 1 user on the site the user will get 128000bps, if you have 2 they will each get 64000, 3 users they will each get 1/3, etc, etc. The bandwidth will be balanced periodically based on how many users are currently active. To set a balanced ceiling, simply use the -r and the -b switch together:

bwmgr em0 -name Acme -addr 207.11.14.1 -bwboth 128000 -r -b

This will cause the bandwidth on dynamically created rules to be balanced, with each active user receiving an equal portion of the allocation.

A problem may occur if you have many users on a site which has a relatively low bandwidth allocation. With the above, for example, if you had 50 users on the site at one time, each user would only get about 2500bps, which may render the site unusable. Perhaps this is what you want, but there are also some things you can do to tune this. You could use the -maxlinks switch to limit the number of users, this controlling the minimum to (1/maxlinks * bandwidth). Also, if you wanted to allow at least a certain amount of bandwidth no matter how many users were on the site, you could use the -bwmin to specify a floor for the balanced allocation. So, for example:

bwmgr em0 -name Acme -addr 207.11.14.1 -bwboth 128000 -bwmin 24000 -r -b

would balance the bandwidth until the allocation for each rule would be less than 24000, at which point it would set the usable bandwidth at 24000bps for each rule. This guarantees that any users accessing this site would not be limited to less than 24000bps. When you use minimums, however, the overall limit of 128000 is no longer enforced. The amount of bandwidth that could be used by the address specified would be (bwmin * maxlinks). This use of -bwmin is exclusive to reverse rules. It also implies that reverse rules cannot inherit minimum bandwidth settings, only shared bandwidth and burst settings.

Creating Dynamic Groups with Reverse Rules

You can use the dynamic nature of reverse rules to create "dynamic groups". If a reverse rule is a member of a group, then any dynamic rules created from the reverse trigger will also be a member of the group. For example, suppose you had a class C and you wanted to limit each address to a specific setting. You'd have to create a group, and create 254 rules in the group. How boring. Instead, you could do this:

bwmgr em1 -x 3000 -name MyGroup -bwin 1000000 -bwout 1000000 -group
bwmgr em1 -x 3001 -name MyGroup_rev -addr 11.1.1.0 -addrmsk 255.255.255.0 -r -bwin 64000 -bwout 64000 -bwlink MyGroup

The above defines a group at index 3000 that is limited to 1Mb/s. Rule 3001 defines a reverse rule, which is a member of the group MyGroup, which will create a dyanmic rule, which will be a member of the group MyGroup, with the limits specified in reverse rule 3001. Because the reverse rule is a member of the group MyGroup, the dynamic rules will be created before index 3000 (NOT the index of the reverse rule in this case), and they will inherit the group membership of the reverse trigger rule. This simple construct allows up to 254 dynamic rules (each possible address in the 11.1.1.0 network) with a single rule.

Adding a Dynamic Rule from the Command Line

A subtle effect can be achieved by using the -dyn option. The -dyn option allows you to add a rule to a group as though it was being added by a reverse rule. This allows you to control the position of the rule within the ruleset without having the script or user know about all of the rules in the ruleset. Consider a case where you wanted to allow only certain IP addresses access, and you have to following setup:

bwmgr -x 5000 -group allow_list
bwmgr -x 5001 -ipprot allip -priority discard

The strategy here is to add users BEFORE rule 5000 which will be allowed. The following will achieve this:

bwmgr -addr 10.1.1.1 -bwlink allow_list -dyn

This will cause the rule to be added using the first available index before the "allow_list" rule, or 4999 in this case. So you'd have:

bwmgr -x 4999 -addr 10.1.1.1 -bwlink allow_list
bwmgr -x 5000 -group allow_list
bwmgr -x 5001 -ipprot allip -priority discard

You could also use this to add temporary rules by an application, with a command similar to the following:

bwmgr -addr 207.17.1.14 -bwlink allow_list -dyn -t -ruletmo 3600

This command would create a rule in the allow_list that would expire in 1 hour.

Host Name Reverse Rules

As described earlier, the -nameaddr option can be used to Match virtual hosts even though many hosts may use the same IP address. There is a particular problem with virtual hosts which is that only the initial HTTP header contains the virtual host information. Subsequent packets look like regular http traffic going to an IP address shared with many hosts. In order to simplify the management of virtual host traffic, -nameaddr rules are simply specified as reverse triggers for dynamic reverse rules. The -nameaddr rule is used as a trigger to create dynamic rules to hosts accessing the virtual host. For example:

bwmgr em0 -name thishost -nameaddr www.thishost.com -r -bwboth 128000

The above specifies that each IP address accessing host www.thishost.com will be limited to a 128000bps. The -nameaddr tag will be triggered when an http request with the proper host header info is detected. Once a host request has been detected, a temporary, dynamic rule will be spawned limiting the requester. The resulting rule will be a reverse bandwidth rule which by default will inherit the properties of the trigger rule. The -ruletmo and -maxlinks switches may also be used to control name address rules.

Optionally, Groups or the -b option may be used in conjunction with -nameaddr to change the behavior of the creation of the reverse rules. With the -b optoin set, the bandwidth value of the trigger rule would be a ceiling reference, and the dynamic rules will be balanced in the same way as standard reverse balanced rules. For example:

bwmgr em0 -name thishost -nameaddr www.thishost.com -r -bwboth 128000

will cause all of the hosts accessing www.thishost.com to be limited to 128000bbs each, while

bwmgr em0 -name thishost -nameaddr www.thishost.com -r -b -bwmin 32000 -bwboth 128000

will result in each host accessing www.thishost.com to have an equal share of 128000bps with no less than 32000bps.

Reverse Firewall Rules

The ET/BWMGR allows you to define rules which can be set up to signal that some remote host should be blocked from access to your network. Port scanners are an example, as are address probers (applications that attempt to discover hosts and services on your network). By using what we call "reverse firewall rules", you can set up triggers which cause hosts to be blocked for some predetermined length of time when they access a port or address or a specific service on a specific host.

Reverse firewall rules could also be called "reverse discard" rules, because thats what they do. You set up a trigger rule, and when the rule Matches, a dynamic firewall rule is set up that blocks the source address from any further access to your network. For example, recall a virus that caused hosts to sent a specific URL that could infect specific types of servers. The virus caused servers to scan addresses and send a "default.ida" URL, hoping to find hosts to infect. You could easily thwart this with the following:

bwmgr em0 -x 1000 -url default.ida -fw -r -priority discard

The above rule would look for http traffic with a URL of "default.ida". When a Match was encountered, a reverse rule would be created (starting at index 999 and moving down) with the source address of the frame (ie the "sender" of the specified URL) and a priority of FW-Deny (or discard). Any subsequent traffic from that host would be blocked.

To stop an address scanner, select an address on your network that isnt in use (or is only used internally). The theory is that anyone accessing such an address is bad.

bwmgr em0 -x 1000 -addr 209.14.1.1 -r -fw -ruletmo 300 -l -priority discard

this rule would block anyone who attempted to access 209.14.1.1 for 5 minutes. Note that -l is set, so that each access will also be logged.

To thwart a port scanner, you could select a taboo port or range of ports that shouldn't be accessed by outside hosts. pop3 is a popular port scan port:

bwmgr em0 -x 2000 -ipproto tcp -port pop3 -r -fw -priority discard

This rule would cause anyone accessing a pop3 port to be blocked. When using such rule, remember to insert rules before the triggers that specifically allow hosts that are permitted to access the addresses or port, otherwise you will block them also.

Temporary Rules

You can add a temporary rule using the -t option (for adding a learned rule) and the -fixedtmo option.

Using these options together, you create a rule that will be deleted after some specified number of seconds. For example:

bwmgr em0 -x 50000 -t -fixedtmo 300 -addr 10.1.1.1 -bwin 128000 -bwout 128000

The above rule will limit address 10.1.1.1 to 128000 in and out for 5 minutes. After 5 minutes the rule will be deleted. If you wanted to make it an idle timeout (ie the rule would be deleted after 5 minutes of no activity), use -ruletmo instead of -fixedtmo.

Group Bandwidth Management

"Groups" allow you to create rules which act on a number of rules with some common parameters. You create a group by creating a head group rule with a priority of "Group". Of course this is not a priority, but a marker that the rule is the head entry for a group:

bwmgr em0 -name renegades -group

This rule creates the group head, which is not a real rule in that it will never "hit". To add members to the group renegades, use the -bwlink option:

bwmgr em0 -addr 10.1.1.1 -bwlink renegades

The above rule creates a new rule and links it to the group renegades. Of course, a group with no properties, as in the example above, isn't very interesting or useful. By adding properties to the rule, you can create powerful super rules that can increase your flexibility to manage bandwidth.

Setting a Bandwidth Ceiling for a Group

The simplest form of a group rule is the bandwidth limit for a group. This allows you to set a common bandwidth limit that applies to all of the members of the group as if it were a single rule. For example:

bwmgr em0 -name Group0 -bwboth 256000 -group
bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwlink Group0
bwmgr em0 -addr 12.1.7.14 -bwlink Group0
bwmgr em0 -addr 12.1.7.15 -bwlink Group0

The above ruleset will set a bandwidth limit of 256000 bps which will be shared by the 10.1.1 network, plus 2 hosts on the 12.1.7 network. Grouping allows you to aggregate any number of disjoint hosts, network or traffic types and bundle them as if they were one rule. One benefit of this is that they can each have their own statistics, if required. You can also apply other bandwidth properties which will apply to groups as a whole such as -bwmin, -bwin, -bwout and -bwburst.

Setting Bandwidth Limits Within Groups

You can give rules within a group its own properties which will work in concert with the group properties. For example, in the above example, suppose you didn't want any member of the group to use more than 128K of the bandwidth at any time (the default would allow it to use up to the full 256K if none of the other members were active. You could do the following:

bwmgr em0 -name Group0 -bwboth 256000 -group
bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwboth 128000 -bwlink Group0
bwmgr em0 -addr 12.1.7.14 -bwboth 128000 -bwlink Group0
bwmgr em0 -addr 12.1.7.15 -bwboth 128000 -bwlink Group0

The above would allow up to 128000bps for each member, but limit the group to 256000bps as a whole.

Balanced Group Bandwidth

You may want to achieve "fairness" in a group, so that each member will get an equal chunk of bandwidth depending on how many members are active at any time. By using the -b option, you can "balance" the bandwidth across active members in a group. So, for example, if you had a group with 30 members and a bandwidth allotment of 512000, if 2 of them were active they would get 256000 each and if 10 were active they would get 51200 each. You can additionally use the -bwmin to assure the the allocation per user was at least a certain amount (because if all 30 were on each member would have an allocation so low that none of them would be functional). Using the example above again, you could simply set the -b option in the group rule to achieve balancing:

bwmgr em0 -name Group0 -bwboth 256000 -b -priority group
bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwlink Group0
bwmgr em0 -addr 12.1.7.14 -bwlink Group0
bwmgr em0 -addr 12.1.7.15 -bwlink Group0

Simply by adding the -b option, you transform the group into a balanced bandwidth group. When you view the rules, the current setting (based on how many of the rules are currently in use) will be shown. Note that it may not be safe to set the -b option to an existing group, as some required settings in group members may not be initialized properly.

Weighted Balanced Groups

Taking balancing to another level, if any of the group members have a priority other than the default (5 or normal), then the group will be balanced according to "weight". With a large number of active member this will have little effect, however when a few members are active it can have significant effect. For example:

bwmgr em0 -x 100 -name Group0 -bwboth 256000 -priority group
bwmgr em0 -x 200 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwlink Group0
bwmgr em0 -x 300 -addr 12.1.7.14 -bwlink Group0
bwmgr em0 -x 400 -addr 12.1.7.15 -bwlink Group0 -priority 3

In the above example, each member of the group would get the full 256000bps if they were the only active member. If rule numbers 200 and 300 were active, then each would get 128000bps. But if 200 and 400 were both active, 200 would get 5/8s of the bandwidth (the 8 being derived from the sum of the priorities 5 + 3), and rule 400 would get 3/8s. With all 3 active, rules 200 and 300 would get 5/13ths of the bandwidth and rule 400 would get 3/13ths.

Nested Groups (Groups within Groups)

You can define a group within a group, that is a group tag which is a member of another group. This allows you to use a group as if it were a single rule within a group, or allow a group bandwidth setting to be dependent on the activity or settings within another group. You can even have balanced groups withing balanced groups, allowing for full dynamic bandwidth settings on a demand basis. For example

bwmgr em1 -x 1000 -name BigGroup -bwin 1000000 -bwout 1000000 -group -b
bwmgr em1 -x 1010 -addr 10.1.1.1 -bwlink BigGroup
bwmgr em1 -x 1500 -name LittleGroup -bwlink BigGroup -b
bwmgr em1 -x 1510 -addr 11.1.1.1 -bwlink LittleGroup
bwmgr em1 -x 1510 -addr 15.1.1.3 -bwlink LittleGroup

The above creates a balanced group with a bandwidth limit of 1Mb/s in and 1Mb/s out. There are 2 members in the group, IP address 10.1.1.1 and LittleGroup, which has 2 of its own members. The bandwidth of BigGroup will be distributed equally between IP address 10.1.1.1 and LittleGroup as needed, as if LittleGroup was a single rule. Within LittleGroup, the bandwidth will be further distributed to its members.

Excluding Traffic (Pass-thru Rules)

You may wish to completely exclude certain traffic from group or interface statistics and limits, such as the traffic from certain servers. You can exclude traffic by setting a rule with a priority of "pass-thru" which allows traffic to pass and not be counted towards limits. For example:

bwmgr em0 -ifac -bwin 1000000 -bwout 1000000
bwmgr em0 -x 1 -saddr 10.1.1.0 -saddrmsk 255.255.255.0 -daddr 45.1.1.1 -priority pass-thru
bwmgr em0 -x 2 -saddr 45.1.1.1 -daddr 10.1.1.0 -daddrmsk 255.255.255.0 -priority pass-thru

The above will allow only 1Mb/s of traffic through em0, but will not count traffic between host 45.1.1.1 and the 10.1.1.0 class C network. The pass-thru designation excluded the traffic from being counted towards limits in interfaces or groups in which the rule is a member.

Using the Global Ruleset

Starting with v4.0, there is a "global" ruleset that can be created on pseudo interface bw0. Simply add the rules to bw0 instead of the ethernet interface:

bwmgr bw0 -addr 192.168.17.3 -bwprofile default

The global interface is useful if you have more than flow passing through a machine. Say you have 2 of our bypass cards servicing 2 separate gigabit interfaces, but a particular customer's traffic may be on either flow. The global ruleset allows you to manage the address with only one rule, rather than having to create a group as in previous versions.

Direction is an issue with a global ruleset, because traffic may come from all directions and there is no context. To add context, you need to tag which interfaces are "outside", that is, connected to the internet side rather than the client side. You can do this in the GUI, or use the -o setting in the cli:

bwmgr em0 -ifac -o

This would set em0 as an outside interface.

Prioritization

What is Prioritization?

The fastest way to move data is to transmit it as soon as you can. But since only one frame can be transmitted at a time onto a medium, data becomes "queued" up and important data may have to wait until less important data is finished. By "prioritizing" the data, you can reorder the data so that important data is sent first. While this may all seem to be what you want, there are caveats as well. The big issue with prioritizing data is that there is a time penalty to pay for every frame that is prioritized, because you have to hold some amount of data so that you can reorder it according the prescribed priorities. When traffic is not so heavy (that is, when you would be perfectly happy with the throughput if no prioritization is done), then you are introducing unnecessary latency into ALL of your traffic. This is something that you want to avoid.

ET/BWMGR Prioritization

The ET/BWMGR used an internal algorithm to determine when data is to be applied based on the relative priority of the data. Priorities only have meaning when applied to bandwidth settings and allocations. This implies if you have a ruleset that gives data priorities but you have no bandwidth settings, the priorities will have no meaning. When applied in the context of access to bandwidth, data with higher priority will have first access (and be delay less than) data with lower priority.

How to Set Global Data Priorities

The ET/BWMGR has 10 levels of priority (1-10), with 1 being the lowest priority and 10 being the hightest. When data is set to be prioritized, the priority subsystem tries to find a Match in the ruleset. If a Match occurs, then the priority of the data is set to the priority indicated in the Match.

The rule for setting a priority is simple. The first non-normal priority that matches the traffic is the priority of the traffic. For example

bwmgr em0 -global -x 100 -ipprot icmp -priority 1
bwmgr em0 -global -x 110 -ipprot icmp -addr 10.1.1.1 -priority 6
bwmgr em0 -x 120 -addr 10.1.1.1 -bwprofile default

An ICMP packet from 10.1.1.1 would be set to a priority of 1 in the above ruleset, even though it appears that rule 100 is "more" of a match. A match is a match. Since rule 120 doesn't have any priority set, the first global rule is used. If you wanted rule 110 to hit, you should give it a lower index then rule 100. Alternatively:

bwmgr em0 -x 100 -ipprot icmp -priority 1 -global
bwmgr em0 -x 120 -addr 10.1.1.1 -bwprofile default -priority 8

for the above, ICMP traffic for 10.1.1.1 would be set to a priority of 1 as per rule 100, and set to 8 for all other traffic. Additionally:

bwmgr em0 -x 100 -addr 10.1.1.1 -bwprofile default -priority 8
bwmgr em0 -x 110 -ipprot icmp -priority 1 -global

The above ruleset would set an ICMP packet for 10.1.1.1 to have a priority of 8, while other ICMP traffic (for rules with indexes greater than 110) would be set to 1.

Setting Priorities for Bandwidth Rules

All bandwidth rules include are assigned a priority, with the default being NORMAL, or 5. You can optionally set a different priority with the -priority command line option. If a data frame Matches a bandwidth rule, the priority of the data will be temporarily set to that value. If the priority threshold of the interface or the bandwidth limit of the rule has been exceeded, the data will be checked against the Global Priority Ruleset for a Match. If a Match occurs, then the priority of the data will be set to the priortity indication in the Global Priority rule.

Statistics

Showing "Hits" and "Drops"

Any time that a rule Matches or "hits", a counter is incremented. To view a specific rule on the console, use the "show" command:

bwmgr em0 show

The output might look something like this:

100 dstaddr:211.14.17.1/255.255.255.0 prot:UDP BWIN:56000 BWOUT:56000 priority:NORMAL (140/0) <IN OUT>

where the 100 is the index or rule number and the numbers in parenthesis represents the (hits/drops) number of times a packet that matched this criteria was Matched, and also the number of times that this rule resulted in a dropped data frame. .

Saving Stats for a Rule

Saving statistics is generally a function that is integrated with the Graphical Interface, but an overview of how it works will help you to debug problems and also do things manually if you have the need to do so.

There are two command line options which cause statistics for a particular rule to be stored with the bandwidth manager subsystem. The first option, -stats, is a flag that tells the bandwidth manager to store stats for this rule. For example:

bwmgr em0 -x 7000 -addr 10.1.1.1 -stats

This command creates a rule will will store statistics for all traffic to and from host 10.1.1.1. You can view recent stats with the bwmgr command line utility as follows:

bwmgr em0 stats 7000

would display something similar to the following:

7000 daddr:10.1.1.1/255.255.255.255 (LAST-IN) 672bps (LAST-OUT) 672bps (CUR-IN) 723bps (CUR-OUT) 723bps

Note that "LAST-IN" and "LAST-OUT" refers to a snapshot of the previous "stats period", which is setable with the setperiod option, and current what has occured in the time period since the last snapshot was taken.

About the bwmgrd daemon

ET/BWMGR can be used to obtain statistics for customers, networks, traffic types, etc. that can be displayed with the integrated graphing capabilities of the ET/BWMGR. The data can also be made available to an exteral SNMP management systems such as MRTG. SNMP only gathers statistics from interfaces, so ET/BWMGR allows you to create pseudo-interfaces that are used solely to hold and report statistic from bandwidth manager rulesets.

To enable the gathering of statistics, use the -stats option in the rule. This creates an internal structure that will maintain statistics for the specific rule. You MUST give stats rules a name, so that the info can be gathered by bwmgrd and stored. For example:

bwmgr em0 -name ACMEcorp -addr 200.1.1.7 -stats

will gather statistics for all traffic to and from address 200.1.1.7 internally. Note that bwmgrd will look for a file name /usr/local/etc/bwmgr/config/name.cfg where name.cfg in this case would be ACMEcorp.cfg. If it doesnt find the file, then statistics will not be stored in the data base. When you create a rule with stats gathering enabled, whether you are using the command line or the HTML interface, the file will be created automatically. An example file name for the above rule would be:

/usr/local/etc/bwmgr/config/ACMEcorp.cfg

will create the file. If you use the HTML interface, this file will be deleted if you delete the rule, but it will not be deleted if you flush your rules or if you delete the rule using the command line interface, so if you do one of those things dont forget to delete this file, otherwise bwmgrd will complain about there being no internal rule for the file. We recomment that you use the GUI exclusively to add and delete statistical rules.

bwmgrd also handles time management, that is it enables and disables rules if they have -timeon and -timeoff settings. If bwmgrd is not running, then the state of the rule will not change.

stats-only Rules

Stat-only rules are antiquated in v3.24 and are replaced by global rules.

Note that when a normal rule "hits", no more rules are checked. This is done for efficiency and also as a matter of basic policy. Unfortunately, for statistical purposes it may hinder your ability to gather statistics because traffic types overlap. For example, if you wanted to gather stats for the total www traffic on your network and also the www traffic to a particular host, to following would fail:

bwmgr em0 -x 100 -name A -addr 207.11.14.1 -prot tcp -port www -stats
bwmgr em0 -x 200 -name B -prot tcp -prot www -stats

The reason it would fail to give the results you want is because any traffic for host 207.11.14.1 would not be included in the numbers gathered by rule 200 because once rule 100 hits, no other rules could hit with a higher index.

To override this, you can use a global rule with a stats setting.

bwmgr em0 -x 100 -name tcp_all -prot tcp -stats -global
bwmgr em0 -x 200 -name www_all -prot tcp -port www -stats -global
bwmgr em0 -x 300 -name AWC -addr 207.11.14.1 -stats -global
bwmgr em0 -x 5000 -addr 207.11.14.1 -bwboth 250000

The above allows you to limit all of the traffic to host 207.11.14.1 to 250000bps and also gather statistics on specific traffic types, even if the types overlap with other rules. The -stats switch would allow you to store and graph each type above.

Note that stats rules still must have a lower index than any non-global rules. In the above example, any global rule with an index higher than 5000 would not be counted for host 207.11.14.1. As a simple policy, you should segregate your global and bandwidth rules by setting an index below which all rules are non-global to avoid conflicts.

Combining Statistics from 2 or More Rules

For compilation purposes, you may want to combine statistics from 2 or more rules. Suppose for example you wanted to know the sum of your TCP and UDP traffic? You could use the -statscombine parameter to compile them:

bwmgr em0 -name tcp_stats -ipprot tcp -stats -global
bwmgr em0 -name udp_stats -ipprot udp -stats -global -statscombine tcp_stats

the above would work just fine, but you'd have to do some math to get the tcp total, since tcp_stats would contain the sum of tcp and udp. To compile them seperately:

bwmgr em0 -name tcpudp_stats -stats -global -d
bwmgr em0 -name udp_stats -ipprot udp -stats -statscombine tcp_stats
bwmgr em0 -name tcp_stats -ipprot tcp -stats -statscombine tcp_stats

the above would keep UDP stats in udp_stats, TCP stats in tcp_stats and combined stats in tcpudp_stats. Note the the -d flag is used on the "combined only" rule to keep it from hitting on any traffic. Without the -d, the rule would Match all traffic, which isn't what you want.

You can also do this with a group, as stats from group members are aggregated in the group rule.

SNMP and MRTG with ET/BWMGR

The -stats function described above is the most efficient way to gather statistical information for a rule, but the information cannot be exported to be read by an SNMP manager such as MRTG. In order to be able to export the statistical info to an SNMP gathering device, you must create an interface associated with the rule that can be read by SNMP.

To create a statistical interface, use the "-statsdevice devname" syntax (or add a stats device in the GUI). The interface name must be 8 characters or less and end with a number. Suppose that you have a customer named General Dynamics and their IP address is 178.35.1.8. You could create a statistical interface with the following:

bwmgr em0 -addr 178.35.1.8 -statsdevice gendyn0

This would create the interface gendyn0 and store all of the traffic information for this IP address crossing em0 into this interface. You could then track/chart gendyn0 with an SNMP package such as MRTG.

Notes on SNMP

Note that SNMP only reports interfaces that exist at the time that the SNMP daemon is first run. This implies that interfaces created with the bwmgr function must either be created BEFORE running snmpd or that you much kill and restart SNMP after creating interfaces. Also note that sending a HUP signal to SNMP does NOT cause it to recognize new interfaces.

Note that the ET/BWMGR only counts actual IP data (including the IP headers) for its statistics. Ethernet interfaces include the ethernet header (14 bytes). While this is arguably correct for determining bandwidth used on the LAN itself, the ethernet header is not tranmitted by a router so it should not be included in router or internet related statistics. This discrepency may be noticable when using MRTG to map both physical interface statistics and also interface-specific statistics reported by ET/BWMGR. The numbers for ET/BWMGR will be lower because they do not include the hardware header counts.

Viewing Bandwidth Statistics

You can alternatively view stats on the interface with the CLI using the following syntax:

bwmgr em0 stats [rule]

where rule is an optional parameter which can be either the rule name or number.

Changing the Statistical Period

You can change the statistical period described above with the following command:

bwmgr setperiod value

where value is the new period. So, to set the period to 1 minute, you would use the following:

bwmgr setperiod 60

Note that the statistical period is the same as the burst period, so if you are using bursting its likely that its more important to use a period to conform to your bursting requirements than statistical viewing.

Advanced Topics

Setting a Password for use by non-Superusers

For normal operation, many functions of the bwmgr utility can only be run by superuser. It may be desirable to run the utility from a script or from a web application where running as superuser may not be practical or possible. Or you may want to allow selected system users to use the utility while restricting others. By default, there is no password assigned. To add a password, use the following syntax:

bwmgr setpwd password

where password is the password you want to use. The password is stored in the file /etc/bwmgr.pwd in encrypted form. After a password has been set, it may be used to execute commands that are normally only available to a superuser. For example:

bwmgr em0 -x 100 -addr 10.1.1.1 -bwprofile default -pwd password

The syntax "-pwd password" must appear as the final 2 parameters. If it is correct, it will allow any user with execute priviledges to execute any bwmgr command. If the superuser uses the syntax, there is no effect. If a superuser issues a wrong password, the command line utility will print a message "Bad Password" but will continue to execute the command, as no override is required. If a non-superuser issues the command with a bad password, they will get the same "Bad Password" message and another error message indicating that they are not authorized.

To remove the password from the system, you can either delete the file, or issue:

bwmgr clearpwd

Compression

*Note that compression is only supported in FreeBSD.

Compression can be enabled between any 2 systems running v3.23 or later of the ET/BWMGR software. Compression allows data within IP packets to be compressed by one unit running the software and then decompressed by the other, and can result in significantly higher throughputs between the points when not directly connected on a high speed network. Before using compression you should read the compression technology discussion on our web site.

Global Compression Settings:

There are 3 global compression settings that can be set either from the command line or via the HTML interface.

1) Compression Status: Enabled or Disabled
2) Minimum Packet Size to Compress
3) Compression Ratio

By default, compression is disabled. In order to enable compression, all of the global settings must be set with the following syntax:

bwmgr compress enable|disable min_pkt_size ratio

To enable compression with the default settings you would issue:

bwmgr compress enable 90 33%

The enable|disable setting globally enables or disables compression system-wide regardless of other settings. So if two machines are having problems with compression between one another both can simply disable compression and normal traffic flow should follow.

The minimum packet size parameter tells the system not to try to compress any packets under 90 bytes. Because of overhead that can't be compressed and CPU overhead per transaction, its usually not worthwhile to even try to compress smaller packets. Even if some compression can be achieved, the gains are usually minimal and not worth the CPU overhead to compress and decompress them. So the strategy is to simply let small packets through that are less than this setting.

The compression ratio is the minimum ratio that must be achieved in order to forward the packet as a compressed packet. Allowed settings are 10%, 20%, 33% and 50%. You might wonder why you wouldn't want to just set this to 10%, since thats better than nothing. But there is a fallacy in that thinking. As mentioned in the technology discussion, many packets will not be compressable. Graphics in particular, but also many binaries and pre-compressed downloads. By setting this to a higher value, the system will be able to tell sooner that the packet is not compressible, and will save CPU cycles by aborting before wasting more cycles trying to compress. On a low speed link, say 64kbs, it might be worthwhile to use a low compression setting as you probably have much more CPU than you need anyway. On a fast line, like a T3, you might use 50% as you'll only want to compress packets that are very compressible.

Setting Compression Location Rules:

One requirement for compression: for every compressor you must have a decompressor. There are almost no cases where ALL traffic is to be compressed, because the device is accessed locally as well as remotely.

No traffic will be compressed unless there is a location specified in the compression definitition table. To add an entry, use the bwmgr compress syntax as follows:

bwmgr compress add -x index [-saddr network -saddrmsk networkmask] [-daddr network -daddrmsk networkmask] [-d] [-cmprate rate]

-x is the index of the rule. Rules are scanned in order for a "Match".
-saddr is the source network (ie only traffic FROM this network)
-saddrmsk is the network mask for -saddr
-daddr is the destination network (ie only traffic TO this network)
-d is used to explicitly disable compression to this location. This is required to "exclude" traffic to certain networks.
-cmprate is the minimum compression rate. 10%, 20%, 25%, 33% and 50% are allowed values.

Most entries will include a -daddr, which is the address of the remote network that you are sending compressed traffic to. Note that this is not the address of the router or ET/BWMGR box. There is not "route" to the remote box, so you must tell the ET/BWMGR software what networks are on the receiving end of the compressed traffic. Take the case where an ISP has a box and they are supplying compressed traffic to a customer. The customer only has one line, and all of their traffic is sent through the ISP. The customer has been assigned the 200.1.1 class 'C'. Setting for this would be as follows:

For the ISP:

bwmgr compress add -x 1 -daddr 200.1.1.0 -daddrmsk 255.255.255.0

For the Customer:

bwmgr compress add -x 1

Thats all that is required. The ISP rule says to "compress all traffic with a destination address of 200.1.1.x". So any traffic going to the customer's network will be compressed. The customer's rule says "compress everything going out". As long as the customer's ET/BWMGR box doesn't have any other networks in its path, a simple default rule like this will work. But what if the customer had an internal network? Suppose, for example, the bwmgr box had an address of 192.168.17.40. Any traffic from the 192.168 network that accessed the box would be erroneously compressed, because EVERYTHING going out is being compressed. So this won't work. To make it work, we have to add a rule, BEFORE the default rule, to exclude local traffic:

bwmgr compress add -x 1 -daddr 192.168.0.0 -daddrmsk 255.255.0.0 -d
bwmgr compress add -x 8

We add a rule that has compression disabled to the 192.168 network. So the above ruleset tells the ET/BWMGR system to "compress all outgoing traffic UNLESS the traffic is going to the 192.168 network". Also note that the default rule is set to index 8. There is no requirement that they be sequential. Any number between 1 and 2 billion can be used.

To delete a compression rule, use the del keywork and specify the index:

bwmgr compress del -x 2

Using the html interface, its simple to enable or disable compression to any location, and to track your compression statistics. There is no way to view the statistics from the command line. You can, however, view your compression settings with:

bwmgr compress show

If you get into a pickle and can't figure out why something does't work, you can turn on debugging. If you have a lot of compressed traffic it can get pretty spammy, so its best to only use debugging when you can isolate the traffic. To enable debugging, use the following command:

bwmgr compressdebug on

To disable use the same command but use "off" instead of "on". Debug messages are sent to kern.debug, which usually goes to /var/log/messages and probably the console. It can be tuned in syslogd.conf.

Controls for De-compression:

Decompression doesn't require any special rules. If you are only compression in one direction (ie a customer that only has their downstream compressed), then you can just enable compression with no location rules, and it will work just fine. The default behavior for decompression is to decompress when a compressed packet whenever one is detected. However there are cases when you may want to "pass through" a compressed packet, in which case you can exlicitly disable compression to that location, which will also disable decompression. For example, suppose an ISP has a customer who has an ET/BWMGR box, and the customer also wants to put a box at his home on a cable modem and compress his traffic end to end to his home. In order to do this, the ISP will have to disable decompression to and from that location:

Home (207.1.1.1) <-------- Internet <-----------> ISP <--------------------------> Customer (200.1.1.0)

Typically, cable modem customers only have one address, so the compression stream can be defined as a host address. Now the ISPs ruleset would be as follows:

bwmgr compress add -x 1 -saddr 207.1.1.1 -daddr 200.1.1.0 -daddrmsk 255.255.255.0 -d
bwmgr compress add -x 2 -saddr 200.1.1.0 -saddrmsk 255.255.255.0 -daddr 207.1.1.1 -d
bwmgr compress add -x 3 -daddr 200.1.1.0 -daddrmsk 255.255.255.0

The above ruleset says "dont compress or decompress traffic between the 200.1.1 network and 207.1.1.1", but compress everything else going to or coming from 200.1.1.0.

Controlling File Sharing Programs - Gnutella, Kazaa, Audiogalaxy, Morpheus and iMesh

A special filter is available for detecting these file sharing protocols. While most file sharing protocols do act on specific ports, the way that they work requires some special intelligence, which we have built-in the the ET/BWMGR software. The ET/BWMGR software can detect when these protocols are being run on non-standard ports, allowing you to control the traffic without having to manually sniff out the connections.

Controlling p2p Protocols - The p2p protocol Group

Beginning with version 3.22e, a single special protocol type is available for controlling all of the supported protocols. All of the streams associated with p2p are maintained internally, simplifying the management of all of the protocols with one simple construct. To set a bandwidth limit for the protocols, simply use the -ipprot pseudo definition "p2p". For example:

bwmgr em1 -ipprot p2p -bwin 128000 -bwout 128000

The above rule is all thats required to control all of the p2p protocols, no matter how many streams there are. You can also use the p2p protocol definition along with other settings, just like any other protocol. For example:

bwmgr em1 -name groupA -group -bwin 256000 -bwout 256000
bwmgr em1 -addr 10.1.1.1 -ipprot p2p -bwin 64000 -bwout 64000 -bwlink groupA
bwmgr em1 -addr 10.1.1.1 -bwlink groupA

The above settings would limit 10.1.1.1 to 256000bps and limit the p2p bandwidth to 64000bps out of the 256000.

Gnutella, Kazaa and DC

Special protocols for Gnutella, Kazaa and DirectConnect (DC++) are available in case you want to see counters or stats for these, or in the oddball case where you want to have different settings for one or the other. The protocol "p2p" is a "group", so that it matches gnutella, kazaa, DC++ and all other supported p2p protocols. Sometimes there may be some gnutella or kazaa connections that are classified as P2P or incorrectly (for example if you run kazaa on the gnutella port it will be shown as gnutella), so p2p is safer if controlling p2p is an important function.

Building Compound Burst Triggers

The power of using burst triggers to control the burstability on your network can be enhanced by creating compound statistical rules to use as a burst trigger. Remember that you can make any rule a burst trigger simply by giving it a burst threshold. By using global rules and -statscombine, you can build rules that act as trigger based on multiple criteria.

Suppose you have 3 groups of customers, where the burtability of each group is dependent on the traffic usage of the others. So group A gets as much bandwidth as it needs up to 4Mb/s, group B can burst when group A is using less than 50% of the bandwidth, and group C can burst when groups A and B combined are using less than 70% of the bandwidth. Assuming 10Mb/s is available, you could use something like the following:

bwmgr em1 -name GroupAplusB -global -d -burstthresh 2000000
bwmgr em1 -name GroupA -group -bwboth 4000000 -burstthresh 2000000 -statscombine GroupAplusB
bwmgr em1 -name GroupB -group -bwboth 512000 -bwburst 2000000 -bursttrig GroupA -statscombine GroupAplusB
bwmgr em1 -name GroupC -group -bwboth 512000 -bwburst 2000000 -bursttrig GroupAplusB

The above settings will allow GroupA to have 4Mb/s of bandwidth. When GroupA is below 2Mb/s, GroupB will be allowed to burst to 2Mb/s. GroupC will be allowed to burst to 2Mb/s when GroupA and GroupB combined are using less than 2Mb/s. Note that GroupAplusB has the -d flag to disable it from gathering stats normally. Without the -d, GroupAplusB would Match all traffic. You could give it an address that doesn't exist or something else that would cause it to never match, or if you had a 3rd criteria (like a specific host), you could give it that host address without the -d to combine Group A, Group B and the host.

Policy and Source Routing

The ET/BWMGR includes a feature which allows user defined traffic types to be redirected to other hosts or networks with a simple table entry. This "proxy forwarding" feature facilitates both redirecting to cache servers as well as redirecting traffic to user specified gateways on a policy by policy basis. For example:

bwmgr em0 -fw -prot tcp -dport http -proxydev fxp0 -proxyaddr 123.15.11.1

The above example tells the bandwidth manager to forward all http traffic to a device on the same physical wire as em0 with the IP address of 123.15.11.1. This would be a typical setup for a cache server. Note that the syntax is

-proxydev device
-proxyaddr IP Address or MAC Address
-proxyport tcp_port

where device is an ethernet or ET WAN interface on the system. Note that to use the IP_Address the device must be on the specified interface and it must be ARPable from the box. (ie if you can't ping it from the ET/BWMGR box then you can't proxy to it). A MAC address may be used to avoid the ARP process or to forward to a device that is not IP reachable from the ET/BWMGR box (for example if the IP address is not a network defined on the system).

If the target device is an ethernet network you must specify either an IP address or a MAC address. You may use the interface only syntax it the target interface is a serial line using one of our Serial Adapters. For example, if you have a dual HSSI card installed (interfaces ets0 and ets1) you could source route to an interface with the following:

bwmgr em0 -fw -addr 207.11.14.0 -addrmsk 255.255.255.0 -proxydev ets0

This would force all traffic to or from the 207.11.15.0/24 network to ets0 regardless of the routing table entries. This could be used to force specific types of traffic to or away from specific pipes.

You can also redirect traffic to the local machine by specifying a -proxydev of lo0 in FreeBSD or lo in LINUX. This specifies that the data frame should be passed up the IP stack. Typically this could then be acted upon by the internal ip firewall or iptables code within the kernel for further processing.

Source Routing

You can use the proxy feature to route based on source addresses if you have some reason to do so. For example, suppose you have 2 upstream "pipes" but no logical way of telling which packets to go to which host. Suppose that one of those pipes was a T1 and the other is a 10Mb/s fracitonal T3. You certainly don't want to balance the traffic, but you also would like to use the additional 1.5Mb/s of bandwidth for SOMETHING. So you decide that you want to send all of your dial-up traffic to the 1.5Mb/s pipe, since none of those users can do high-speed downloads anyway. Suppose your setup is that em0 is connected to your backbone, em1 goes to the 10Mb/s pipe (the default), and fxp2 is connected to your 1.5Mb/s pipe. Using a proxy rule, you could force all of the traffic from your dial-up (say with addresses 10.1.3.0) to fxp2:

bwmgr em0 -saddr 10.1.3.0 -saddrmsk 255.255.255.0 -proxydev fxp2 -proxymac 00:01:02:03:04:05

Note that the proxymac is the MAC address of the router for the 1.5Mb/s pipe.

Using Bridging and the Proxy Feature with a Squid Cache Server

A popular use of the proxy capabiity is to use an ET/BWMGR system configured for bridging with a Squid Caching Server running on the box. This allows you to create a single system that acts as a transparent proxy and Caching Server. To do this, you would configure your firewall normally as if you were routing between 2 ethernet segments. One caveat of bridging is that the data frames do not pass through the system firewall, so the redirection in your system will not take place normally. But using the proxy feature, you can tell the ET/BWMGR to grab all of the http packets and pass them to the local machines IP stack, where the internal firewall can act on them. For example:

In FreeBSD:

bwmgr em0 -fw -ipprot tcp -dport 80 -proxydev lo0 -i

In LINUX:

bwmgr eth0 -fw -ipprot tcp -dport 80 -proxydev lo -i

The above setting will grab all port 80 traffic and pass it to the local machines IP stack. Normally you will have the machine's firewall redirect the port 80 data frames to port 3128 (or whatever squid is listening on). Of course you will need to place the above rule on the interface where your http requests will be coming in from. Note that if you are bridging in LINUX, the interface in the above rule MUST have an IP address, or your NAT subsystem will panic. So, if your incoming http requests were coming in on eth1, you would have to give the interface a bogus address to fufull the requirement. On FreeBSD this is not required.

To use SQUID on a FreeBSD system, you would additionally need a firewall rule such as the following:

ipfw add [rule #] fwd 127.0.0.1:3128 tcp from any to any 80 in

In LINUX, the iptables command that might be used to implement a transparent proxy with SQUID might be::

iptables -t nat -A PREROUTING -i eth1 -p tcp --destination-port 80 -j REDIRECT --to-port 3128

Note that the above only applies if you are running SQUID on the same physical system as the ET/BWMGR software. Proxying to external system would have different requirements.

Special Considerations for Managing Traffic with a Cache

A web cache (such as squid) will change the way your traffic appears on your network, so you will need to consider this when implementing your management framework. See the white paper on Bandwidth Management in a cached environment.

General Maintenance

Deleting a Bandwidth Limit Entry

To delete a bandwidth limit entry, use the del keyword with the interface and index or name of the entry you want to delete. For example, to delete an entry displayed with index 200, you would use the following:

bwmgr em0 del 200

If the entry is named, then you can use the name

bwmgr em0 del cust0

To delete a firewall entry, use delfw instead of del. To delete a global priority rule, use delpri.

Saving The Rules

You'll want your current rules to be restored when you reboot your system. If you are using the GUI interface, rules are stored in /etc/rc.bwmgr. Whenever you make a change, the rc.bwmgr file is rebuilt.

If you are not using the GUI, you can use the "bwmgr rebuild" command to generate a current view of your ruleset. To save the rules in a file, simply redirect the output to a file.

bwmgr rebuild >/etc/rc.bwmgr

would save the files.

Support

If you Purchased a Software License

We offer free support for the ET/BWMGR software product as long as you hold a current license via a ticketing system that you can access at under the Support tab at www.etinc.com.

Please do not call us on the phone to ask us how a feature works, or why you can't see your graphs and then get angry when we tell you to create a ticket using the form. Before you buy a license, it is expected that you have successfully installed the product on your hardware. The installation of the software on your hardware is completely your responsibility. It is assumed that you know now to build kernels, install and configure software packages, and that you are very familiar with the operating system that you are using. If you can't get the product to work then you should consider buying one of our complete systems. If you encounter problems with your installation, please email a detailed explanation of the problem, what OS and version you are running and we may be able to point you in the right direction. But please don't expect us to walk you through the process.

If you Purchased a Complete System with ET/BWMGR Installed (An Appliance)

If you purchased a complete system from us, all of the integration has been done and once you get it installed with the proper addresses you should be ready to go. If you have problems with the installation, you can call us during normal business hours (EST) and a technician will try to help you. Telephone support is limited to initial installation and critical issues (ie if your system is down and you need immediate assistance). Otherwise please create a support ticket using the form under found under Support.

If You Purchased From a Reseller...

If you purchased the ET/BWMGR from a reseller, whether you purchased a license or a pre-built system, then you must get your support from them.

Bandwidth Limiting Examples

Shopping Cart

0 items

Bestsellers

01.	ET/BWMGR 30 Day Test Key
02.	ET/BWMGR License
03.	Repair Return
04.	ET/R2800 MultiCore Appliance
05.	ET/R2816 Gigabit Appliance
06.	ET/A1600
07.	Premium/Cache Support
08.	ET/R2400 MultiCore Appliance
09.	ET/R2400-L Cache Appliance
10.	Extension to 3 Year Warranty

What's Popular?

ET/R2816 10Gb/s Appliance
$21,995.00

ET/BWMGR User Manual

(Version 4.1 For FreeBSD 7)

Table of Contents

ET/BWMGR Overview

Installation

Trying out the ET/BWMGR Product

ET/BWMGR Configuration

Stats-only Mode

Setting the Outside Interface

Bridging

System Configuration Issues

Starting the Bandwidth Manager

All about Rules and Policies

The ET/BWMGR Protocol Engine

Configuring the Firewall

Controlling Bandwidth Utilization

Using Bandwidth Profiles

Group Bandwidth Management

Prioritization

Statistics

Gathering and Saving Statistics

AutoMgr - Automated Bandwidth Management

Advanced Topics

General Maintenance

Other Topics

HTML GUI Manual (Graphical User Interface)

ET/BWMGR Overview

Getting the ET/BWMGR product

Downloading the Demo CD

Buying an Appliance

The CD Appliance

Editing default.cfg

Configuring Stats-only Mode

Setting the Outside Interface

Setting Up Bridging

Setting Up a FastBridge

Setting up a Fallback Interface

System Configuration Parameters

Setting the Indexing Level

Analyzing your ruleset

Setting the maxbuffers parameter

Setting the Stats Period

Starting the Bandwidth Manager

Running the ET/BWMGR Demo

Obtaining Your System Serial Number

Registering your ET/BWMGR License Key

Stopping the Bandwidth Manager

Rule Tutorial - What is a Rule?

Rule Syntax

Rule Precedence

Large Rule Indexes

Global Rules

Naming Entries

Specifying Hosts and Network Addresses

MAC Address Rules

Grouping Bandwidth Entries - Creating a Group

Managing Traffic by Data Type with bwmgr

MAC Protocol Rules

VLAN (VPN) Rules

IP Protocol Rules

IP TOS Rules

IP Protocol Port Rules

"Special" Protocol Definitions

URL Rules

Virtual Host Rules

Enabling and Disabling Rules

Enabling and Disabling Rule by Time of Day

Changing Bandwidth Settings

Notes on Logging

The ET/BWMGR Protocol Engine

User Defined Protocols: /etc/bwmgr-protocols

The loadprotocols command

Mapping Agents to Protocols

Mapping URLs to Protocols

Setting up the Firewall - Theory of Operation

Firewall Criteria

Viewing Firewall Rules with bwmgr

Packets Per Second Firewall Rules

Positioning of Packets Per Second Rules

Packets Per Second Suggestions