Bandwidth Management Technologies

(by Dennis Baasch, ET/BWMGR architect)
Updated Feb 2005

 

A lot of people have been asking how our ET/BWMGR product compares with other products on the market. Its very easy for even the most knowledgeable of network engineers to be buried in the jargon used when describing bandwidth management technology. Frankly, even I don't understand half of the stuff on rivals Packeteer and Allot's websites. And the reason is simple: its designed to "sound good" in a somewhat deceptive way. Remember that its been said that the definition of Marketing is "tricking people into thinking that you have what they want". Which makes understanding how the technology works extremely important when selecting a product for your needs. This document is designed to illustrate the various bandwidth management technologies in widespread use so as to make the lingo used by various vendors a bit less cryptic.

The Technologies

To begin, lets assume we have some Box that sits between an internet connection and some network, and that the purpose of the Box is to somehow affect the flow of traffic to and from the internet. The Box has two ports, one going to an internet router and another connected to a switch that services any number of networks within the "intranet". All traffic must flow through the Box in order to get from the internet to the intranet, or from the intranet to the internet. Hence, every data frame can potentially be affected by the Box.

Normally, without any sort of bandwidth management enabled, data frames are passed through the Box as quickly as possible. Data frames come in from the Internet and are passed as quickly as possible to the port connected to the intranet, and vice versa. This is how your typical router or switch functions.

Now lets compare how some bandwidth management technologies running on the Box work.

Queuing Algorithms - CBQ and Fair Queuing

Normally, all data to be sent is put into whats called a "queue". Since the connection on one side of the Box may be faster than the connection on the other side, data frames may arrive on one side of the box faster than the other. You dont want the Box to throw away the data, so its put into a "queue" until it can be processed by the slower interface. The data in the queue is then processed sequentially on a first-come, first-serve basis. Typically, it is sent out as fast as the target medium allows, which facilitates the best possible throughput with the lowest possible latency.

CBQ, the most popular of the techniques and the one used in most low-end bandwidth management products (YDI,microtik, turboPIPE, etc), stands for "class based queuing". Its a fairly simple technique where data is categorized into user-defined "classes" and then the queues are maintained for each class. Data can then be sent out according to time schedules and prioritized by processing the queues at specific intervals and/or in order of priority. However, in order to "reorder" data frames according to specified priorities, data frames are forced into "class" queues (which means they must be delayed) and processed at specific intervals, being sent in order of priority. The purpose of this is to assure that higher priority data frames will always be sent out before lower priority frames. Therefore, high priority data cannot be bottlenecked by anything with a lower priority.

The negatives of CBQ is that it introduces latencies (delays) into virtually all traffic that is being managed and that it is mono-directional in that only outgoing traffic can be controlled. It is also not practical to have a very large number of classes (queues) due to excessive overhead and loss of precision, so controlling hundreds of streams (or hosts, as an ISP might want to) will not work well. CBQ processes the defined class queues in a round-robin fashion, thus as the number of defined classes increases the efficiencies of the management decrease.

CBQ works best in a corporate environment where the user has control of both ends of the link, and where there are a few identifiable types of traffic that need to be managed.

"Fair queueing" is a similar technique that attempts to allocate bandwidth based on the usage by individual flows. It attempts to be a bit more intelligent in making choices, and it's based on relative allocations rather than priorities. The problem with fair queuing is that in the vast majority of cases it does not do what you want to do; you don't want "fairness" between flows, you want fairness between your users or customers. "Flows" are simply a property of a larger entity, as each entity can have any number of flows. Also, like CBQ, Fair Queuing works well in environments where there are a small number of users and a small number of definable flows. When you have 1000 defined "pipes" or "flows", for example, its not practical to manage the flows "fairly" between so many queues.

HTB, the latest "craze" in the linux camp, is "yet another" queuing technique with the same problems: it doesn't scale well and its still a queue-based model. HTB addresses some of the precision issues with queuing algorithms, but it is simply not a technique that you can count on for the long run to manage a large network.

TCP Rate Limiting and Window Manipulation

The Packeteer PacketShaper's primary "technology" is TCP rate limiting, which is a technique that "paces" traffic by "faking out" the transmitter by artificially changing the TCP window and pacing ACKs, effectively throttling the traffic. This works well (on TCP traffic only), however the effects of it can be undesireable when used as a primary means of achieving bandwidth management results. There are several fundamental problems with this technology:

First, there is an adaptation period for each session, as the initial window is usually set to be quite large, and most sessions are quite short. The "pace" required for any particular session is not constant. For example to achieve 512kbs on your local network would require a much longer delay and much smaller window than if the client was 20 hops away. Additionally, since traffic on a busy network (lets talk about ISP's here) changes on a moment to moment basis, the window size required to get the proper management will change more rapidly than the devices ability to change all of the windows for all of the sessions that it is managing. Every time your browser clicks a new page, a new session is created generally. It is difficult and inefficient to control these sessions by window manipulation, because they are so short-lived in nature.

The second issue has to do with resources and overhead. With TCP rate limiting each "session" (that is each TCP connection) must be managed individually. Each stream must be monitored, ACKs must be delayed, and packet headers must be rebuilt. This is why devices that heavily utilize TCP rate limiting on every individual traffic stream have much lower capacities in terms of being able to handle traffic loads than other available technologies.

Another problem is that if you try to rate-limit with windows smaller than one full packet, the result is more packets and more traffic on your network. Typically this is so undesireable that its not done. Window sizes of one full packet (typically 1500 bytes) is the minimum that can be used without creating additional network overhead.

Next, and most importantly, session "pacing" by itself doesn't achieve the desired result. Since an end user can have any number of simultaneous sessions, you can't limit a user to a fair "chunk", which is usually whats needed. A browser, for example, can open any number of tcp sessions to the same location, so pacing each session to 128K will not limit the user in the desired way; he will be able to use N times 128k depending on how many sessions he has open at the time.

Lastly, TCP rate limiting doesn't work at all on encrypted data, because the TCP header is hidden from the bandwidth management device. As IPsec becomes utilized more and more, the effectiveness of TCP rate limiting will dwindle proportionately.

Well, after that you may deduce that window manipulation doesn't have any value at all. On the contrary. We have found that TCP window manipulation as a peripheral function is highly effective in "shaping" traffic and reducing the amount of traffic that needs to be queued and managed. The ET/BWMGR uses TCP window manipulation as a fundamental mechanism to reduce flows and improve overall network performance through your network. As an example, consider the following:

Suppose you have a T1 line and a single server. Suppose that 2 remote clients request the same page simultaneously, and that page has 15,000 bytes of information. With a typical TCP window of 16K, the entire page will be sent without requiring an ACK from the client. So as soon as the requests are received, 30,000 bytes of data are sent from the server to your router's queue. At T1 speeds, 193 bytes per millisecond can be transmitted, which means that it will take 155 milliseconds to empty the queue. Any other traffic that is being sent by your network will have to wait until this traffic is sent. So you have over 150ms of latecy in your network at this moment in time.

Now assume that you have a bandwidth management device which is window shaping and it announces a window of 3000 to your server. Now the server can only send out 6000 bytes of data in response to the requests, which will only take 31 milliseconds to transmit. You have cut your network latency by 124ms without any hard limits.

Of course most pages aren't so large, but you also often have more than 2 connections on your network, so the effect is similar. What is important to understand is that bandwidth management devices which do not utilize window manipulation at all cannot reduce your network latency on an overall basis. They will simply shift the latency from one type of traffic to another.

The ET/BWMGR Technology (version 3.24+)

First of all, when we asked one or our competitor's salesman what they knew about the ET/BWMGR, they told us that it was "like any other public domain software you can download from the internet". Perhaps this is what they'd like you to think, but its completely untrue. The ET/BWMGR does not have one line of public code; it is a completely custom product, owned 100% by Emerging Technologies, Inc., and you cannot get it anywhere else. The fact that it runs on FreeBSD unix and LINUX platforms does not make it public domain, nor does it imply that the product is somehow inferior. We chose these operating systems for two reasons: first that they are rather popular, and second because they are both very fast. And you need speed with a product like this.

Since its obvious we dont believe that any of the previously mentioned "technologies" is suitable for large-scale deployment, you can probably guess that the ET/BWMGR doesn't use CBQ, Fair Queuing or TCP rate limiting as its core management technology. We also don't like to use the term "shaper", because its one of those purposely vague words that really doesn't mean anything. People dont need "shapers", they need to manage traffic. We do, however, utilize both priority queueing and TCP rate limiting as perimeter technologies to achieve what we believe is an optimal balance which maximizes capacity and efficiency.

The first concept of our product is common sense. When you connect to the internet with a 28.8K modem, things are slower than when you are on a T1, and you don't lose packets. You don't need to change the window sizes of TCP sessions, because TCP is fundamentally self-throttling. On a slower line, the ACK packets required to "open" the window (and allow more data to be sent) come back slower, so the session moves along more slowly than on a high speed line. Our core technology simply emulates slower access lines for each defined rule (or stream), so the sending machine and the receiving machine see traffic at the same rates they would on a slower line. By monitoring each defined stream (a stream being a host, network or traffic type with a bandwidth limit), traffic of ANY type can be PRECISELY controlled at virtually any bit rate you choose. The bandwidth manager keeps track of how much data has been sent and only allows as much data as you've specified to pass, just as if it were a separate device on a slower line. This method doesn't normally require data to be discarded, unless there is more data in the pipe than is possible to forward as the specified rate without encountering retransmissions. When TCP traffic is retransmitted, it would result in the same data being queued multiple times if the original traffic wasn't dropped.

But limiting the speed of traffic is not that only element of bandwidth management that needs consideration. When congestion occurs, it is desirable for the bandwidth management device to be able to make on-the-fly decisions about what traffic streams should be streamlined and which streams should be throttled further. Drops are avoided whenever possible, but when discards are necessary, the ET/BWMGR drops lowest priority traffic rather than critical data. Because the prioritization is a secondary process, the ET/BWMGR has a higher capacity in terms of the number of classes it can manage because there is a separation of "pure limitation" and prioritization. Basically, the ET/BWMGR gives you the best features of queue-based bandwidth management without the negatives, while supplying more capacity.

The ET/BWMGR utilizes TCP window manipulation to shorten the window, making on-the-fly decisions (we call it "Adaptive Window Management"), thus reducing the backup in the system and reducing the occurrance of packet loss. But, since it is not using rate limiting to achieve the limits, most of the negatives of rate limiting do not occur. Since window management is used only to reduce the flow and not to enforce the rate, windows can be managed "generally", resulting in the desired flow reduction without creating much overhead. In fact, the ET/BWMGR can manage ANY NUMBER OF TCP STREAMS, unlike our competitor's products, allowing efficient and effective management even at full gigabit traffic levels.

"Guaranteeing" Bandwidth

Many vendors tell you that they can "guarantee" bandwidth, but there is no real way to do this with any efficiency on the INTERNET. You can guarantee access to the wire, but you cannot guarantee the throughput of any particular application because you have no way of controlling or predicting what the delays (or packet loss) characteristics of the internet wires between you and your destination.. You can guarantee minimum access to the wire, and guarantee that a particular traffic stream gets a specific "relative" chunk in relation to other defined traffic types. You can make guarantees on private networks (ie corporate networks that do not use internet connections), as the WAN links are just extensions of the LANs. With proper deployment of bandwidth management devices, you can guarantee access (to the bit in most cases) in such private networks. The ET/BWMGR allows you to "allocate" bandwidth to a particular application or customer, or the bandwidth can be shared when it is not immediately needed.

Some products on the market say that they can "guarantee" bandwidth and also "share" it when its not in use. But this is impossible. Bandwidth in networking terms is not a spectrum, in that it only is definable over time; one bit gets sent at a time, you can't chop up the bit time, and you can't chop up a packet. So if bandwidth is "shared", once you relinquish the allocation by beginning to send some other data, you lose the total time period required to transmit that packet (which in ethernet is at least 60*8 bits). In that time, you have lost the ability to "guarantee" the bandwidth. If the bandwidth is needed during that time, it cannot be granted, so you have lost the guarantee. In order to absolutely guarantee bandwidth, you MUST pre-allocate enough time slots for the data and make them exclusively available to that data as long as the allocation exists.

The Similarities

All of the technologies described can limit your traffic. They all can identify specific types of traffic or hosts or networks, and set specific policies to control the traffic. They all can be purchased with HTML interfaces and graphing and reporting capabilites, and you can get them in handsome, 1U boxes that fit nicely in your rack or on a desktop.

ET/BWMGR Auto-Shaping "Magic"

Nothing is really "magic", but the auto-shaping capabilities of products using the ET/BWMGR technology is about as close as it gets. With a single setting you can dramatically reduce the queue depths of your egress router, reduce or eliminate congestion and gradually "slow" each connection until an equilibrium is obtained. It's an advantage that makes our product uniquely suitable for a network of any size.

ET/BWMGR Packets Per Second (PPS) Rules

The ET/BWMGR is the only product we know of that has "packets per second" rules. At first glance, they may not appear to be terribly useful. But they are very powerful. They work on a principal that isn't very scientific, but more of a "seat of the pants" observation of how things work. Most "good" protocols, such as HTTP, ftp, DNS, are fairly well-behaved. Bandwidth hog protocols, such as p2p, behave very badly. PPS rules allow you to slow down a host based on the volume of activity rather than the number of bytes of traffic that a client uses. For example, suppose that you give a client a setting of 50 packets per second incoming. If the client is downloading a file, with the maximum 1500 byte packets, the client could get about (1500 * 8 * 50) 600Kbps throughput, best case. If the client is running a p2p program that is querying clients and pinging servers to see if they are available, the packets are generally very small. Perhaps all of them are 60 bytes. So the bandwidth used in that case might be only (60 * 8 * 50) 24Kbps. So you've achieved a goal of many providers, which is to allow good things to run at higher speeds than bad things, without having to specifically define which things are good and which are bad.

What's REALLY good about PPS rules is that they are CPU-friendly. They allow you to control p2p and other unruly protocols without having to detect and decode them. On a large network (or with a lower-end appliance), you severely diminish the capacity of the box by enabling protocol sniffing, because each and every TCP and/or UDP packet has to be parsed and examined. None of this is required with PPS rules. PPS rules allow you to control "bad" protocols at a much higher volume of traffic than by using other methods.

ET/BWMGR vs Packeteer

The difference between the ET/BWMGR and Packeteer's packetshaper is one of basic philisophy. Their concept is to micro-manage the content on your network, by analyzing your traffic, identifying bottlenecks and then creating policies based on what you think would work better. Our concept is a bit different. By setting general policies for each client and by generally auto-shaping your TCP flows, your network will flow freely without having to set restrictions which may not be suitable when traffic patterns change or different applications are in use. As an example, suppose you were designing a highway. They would say to have a lane for fire trucks and police, and a lane for busses, and a lane for slow-moving vehicles. We, on the other hand, say that you can have an organized, freely-flowing highway by having enough lanes and setting the speed limit to a reasonable value that is likely to keep everyone happy.

Another difference between our products is capacity. In order to decode and classify all of the traffic on your network, you have to parse and examine every packet header for several levels. While this creates some interesting graphing possibilities, it also consumes a tremendous amount of CPU cycles, which diminishes the capacity of the system. For smaller networks this may not matter, but at the high end its the difference between the system being able to handle the traffic with minimal latency and not. Our methods are much more conducive to high-end traffic management, where the mix of protocols and the number of sessions can be difficult to micro-manage.

ET/BMWGR vs Allot NetEnforcer (Fair Queuing)

From Allot's marketing literature you would think that they somehow invented fair queuing, but all they've really done is added a bunch of features to it and spent a lot of investor's money hyping them up. They do have some high-end features for distributed processing, but as for bandwidth management its pretty basic stuff at a very high price. The big advantage that we have over any of Allot's products is that Allot does not do TCP window manipulation, which means that they cannot naturally reduce the congestion on your network. They've taken the perspective of Packeteer, in that they seem focused on controlling specific protocols rather than establishing a free-flowing network environment.

Other Vendors

NOTE: Information about Exinda removed at their request.

Conclusion

At the same price, you might have a hard time choosing between the ET/BWMGR and other high end products on the market. The ET/BWMGR gives you all of the features that you need to manage bandwidth in a highly flexible manner, plus greater capacity for a much more reasonable price. If you are a Linux or FreeBSD house and have bandwidth requirements of 10Mb/s or less, you can purchase a software license for your own hardware and get all of the power of the ET/BWMGR for the price of a good word processing package. If you have a higher end network, our finely tuned and well supported appliances can keep you in control of your network at a fraction of our competitor's prices.

Disclaimer: Competitor's descriptions are general and may not accurately describe newer versions of software. Please check with them for specifics about their products.