Bandwidth Management for ISPs and Universities

(Bandwidth Management Tutorial Using the ET/BWMGR)

The "Goods" on Bandwidth Management

Bandwidth management is one of the most widely misunderstood subjects in modern networking. The reason is simple: the term itself doesn't convey a specific meaning. Terms like "shaping" further confuse the issue. What does "shaping" mean? What does "bandwidth management" mean? Its just a lot of shady terms that sound good but don't really mean anything. In fact, the terminology used is purposefully vague, perhaps so that you can't complain about the performance of whatever it is you're using. What are you going to say? "The product isn't shaping the way we'd hoped, so we'd like our money back". What are the magnitudes of shaping? How many shapers does it take to screw in a lightbulb? Nobody knows.

Bandwidth Management is a lot like economics, because the complexities of how it works are beyond simple logic. For example, a simpleton might think that if you raise taxes the government will have more money, right? Well, not really, because if you raise taxes people have less money to spend, so they buy less which causes people who sell things to have lower incomes, so they hire fewer people and more people are out of work. So effectively, everyone along the line is making less money, and paying less taxes overall, even though the tax rate is higher, so the government gets less money. There is a dynamic present in economics that makes what seems to be quite simple a lot more complicated as you gain more of an understanding of how things REALLY work. With bandwidth management, for example, you might think that giving HTTP high priority will make everyone's browsing experience better. But if you do so, its likely that several other protocols that you've never even heard of will stop working altogether because there is so much HTTP traffic that nothing else with be able to get through. There's a domino effect that you didn't anticipate. You've actually made things worse with what seemed to be a very logical setting, because you didn't quite understand the bigger picture.

What is "Bandwidth"?

The first problem is that the term "bandwidth" is deceiving, because there are no "bands" and it doesn't have any width. Internet "bandwidth" is not a spectrum; traffic streams are one bit at a time. Bandwidth on the internet can only be conceptualized over time, and the amount of time that you talk about can greatly change the user experience. For example, if you say that a user gets 256Kb/second, does that mean every second? Can he get 512Kb/s in one second and none the next? Thats 256Kb/s. Or a megabit one second and then no bandwidth for the next 3 seconds? What about slices of a second?

Luckily, end users don't know what bandwidth is either, so you can usually trick them into thinking you're giving them something good. But as a provider, or someone involved with the management of bandwidth, you have to understand whats really happening. When you talk about "allocating" bandwidth, you have to realize that its not like setting aside a lane on a highway, because there is only one lane on the internet. All of the bits travel up the same pipe. So you are really only allocating a time slot, which can have a variety of properties, and things will work quite differently depending on the amount of time you use as your baseline.

Why "Application-Aware" Bandwidth Management is Backwards

What's truly ironic is that virtually everyone is doing bandwidth management wrong. Our competitors, you've heard of them: Packeteer, Allot, etc, try to tell you to do it the way that best suits their products. The problem is that their products are based on an incorrect premise. They tell you that the way to manage your network is to micro-manage content. They tell you that the way to streamline performance is to decide which protocols are good, and which protocols are bad. They tell you that streamlining good protocols and squashing bad ones will make your network better. The truth is that "application aware" bandwidth management is more about marketing than it is about managing a large network. Its a way to get you to keep buying upgrades, because if there's one thing you can count on its that there will always be more applications next year.

There are many problems with what our competitors tell you to do, and not all of them are technical. First of all, you're not smart enough to do it their way. You don't know enough about how every protocol ever created works. And I've already told you that there are dynamics present that you don't understand. And you don't have time to learn, and if you do it will be a waste of your time, because next week the patterns will be different, and there will new, mutated versions of protocols that you'll have to learn about. Not to mention buying upgrades to manage the new protocols. Of course thats what they want.

There are also philisophical problems with the way our competitors tell you to manage your network if you are an ISP. As an ISP you are just a reseller of bandwidth. You offer "internet service", not just "services that you think are worthy". If someone wants to use his bandwidth allocation to play games or download music, who are you to tell them that they can't? They can't do what they want to do because you can't figure out a better way to fairly allocated your resources? That's pretty lame.

From a technical standpoint, what do you do when the traffic is encrypted or compressed or both? All of a sudden your entire strategy has been defeated. All of your work has been wasted. You can no longer "see" whats going on. You only have big blobs traveling back and forth. Many bandwidth-hog protocols like "p2p" are moving toward encryption and tunneling.

The most important reason that our competitors are wrong is that their way is a short term solution. If you stop abusers from doing one thing, they'll just find something else. You can't manage abusers long term with the protocol method. All you do is fix things for a short time as long as conditions don't change from when you set up the controls.

So What's the RIGHT way to manage a network?

Think of your network as a society and your bandwidth management device as a police force. The obvious way to stop crime is to get all of the criminals, right? No, thats wrong. There are too many potential criminals. If its easy to get away with a crime you will have more and more criminals. And the more criminals you have, the more difficult it is to get them all. Its like a video game where you shoot a bad guy and 2 more appear. They keep coming. You'll never get them all, because there are more of them than there are of you. You'll be constantly fighting a losing battle.

The way to stop crime is to create a deterrent. I just heard that car thefts are down 20% in the USA. Is that because there are fewer criminals? No, its because today's automobiles are more difficult to steal. If you make your bandwidth more difficult to steal, then you won't have to worry about catching people stealing it.

The question is, how do you create a deterrent? Using the crime analogy, you do it by creating an environment that lets the good guys go about their business normally, and that makes it difficult or impossible for the bad guys to do bad things. You can do it by putting a policeman on every corner, and by putting cameras in banks and at ATMs. You put controls in place that make doing bad things hard to do, and that makes it easier to identify those that do them. The concept can be applied to bandwidth management as well. You don't create a fast, stable network by chasing down the "bad guys". You do it by making it difficult to do bad things. You do it by creating overall policies that thwart the effectiveness of doing things that cause problems. And you don't have to "know" what your clients are doing in order to enforce such policies. Because it doesnt matter WHAT they are doing. It only matters how much bandwidth they are using compared to what they are paying for.

Preparing to use the ET/BWMGR

Clearing your Brain

Your first task is to admit to the fact that you don't know as much as you think you do about how protocols work. Maybe you know a lot, which can sometimes be more dangerous than knowing nothing. More likely you have a superficial understanding of how they work. Even if you know exactly how all protocols work (an arrogant stance indeed), you don't understand what the interactions are on a large network. No one does. You can't model it. It changes constantly. The best you can do is generalize, because if you try to nail it down to a specific, you'll be wrong most of the time. So you need to think in generalities.

The worst case is if you've used some other bandwidth management product and you think you know everything. Unfortunately, what you know is defined by terms that likely only apply to the product you have been using. If you think that "defining queues or pipes" or "setting up partitions" is an element of bandwidth management, you really need to clear your brain. A product that makes you do stuff like that isn't user-friendly. Its creators are asking you to conform to a methodology of doing things which is only applicable within the context of that particular product. You don't know anything about bandwidth management; you only know how to use that product, and if you try to apply those concepts to another product with a different methodology, you will be in trouble. They say you can't learn how to cook by following a recipe. You only know how to make one dish. So clear your head. Start fresh.

Defining a Strategy

In our experience, most users of bandwidth management devices don't have a strategy. Most just want to stop abusers from using all of their bandwidth, which is certainly useful and in many cases will, for a short time, make the network run more smoothly. The problem is that today's abuser is only for today, which means that you are never done. There will be someone else tomorrow. The protocol method is the worst choice. If you "limit" one protocol, they'll find something else. The abusers are clever. You don't want to engage them in a daily battle. You'll spend your whole life chasing down these people. If you want to control abusers, limit their IP or MAC address. By controlling ALL of their traffic, no matter what they do, they are under control. Its as simple as that.

Supposing that abusers aren't your target, if you use our competitor's method and sculpt your network by allocating bandwidth and priorities to scores of protocols, you haven't really defined a strategy either. You've created a framework based on a static model which unfortunately only applies if your network happens to match the model all of the time, which is highly unlikely. You may have determined that your network is 75% http traffic, but the dynamic changes constantly. Not day to day, but minute to minute. So any time that the requirement is above 75%, or if the requirement for some other important protocol increases, your model doesn't work, and everyone on your network will suffer the consequences.

No matter what you use your network for, the real goal is almost always the same. In reality everyone has the same goal: to make your network run smoothly without too many restrictions. Consider the case when you have "more than enough" bandwidth to do anything you need to do without any problems. Things work great. You don't have to examine your usage constantly to see what "the problem" is. You don't have to "catch" anyone doing something that you didn't anticipate. You don't have to run into the office in the middle of the night because your network is so slow that your customers can't even get their mail. This IS the primary goal of bandwidth management. If you have no congestion, you have no problems.

The Problem with Priorities

Back when cave men walked the earth, someone thought of something called "priority queuing", which was the first "method" used for bandwidth management. It made sense, although its not clear why so many "products" still use is. I suspect that the reason is that its easy to implement. Any "pretty good programmer" can do it, and it does make things better, particular if you have a smallish network. What prioritization does is take a bunch of traffic, decide that some traffic is more important than some other traffic, and then send the more important traffic first. But if you think about it, the "method" is flawed...

There is one basic problem with "prioritization". In order to prioritize, you have to have backup. You have to have delays before you can use a priority method, and then all you are doing is reordering your traffic to shift the delay from one protocol to another. This may solve some problems, but it doesn't reduce your bottleneck. And then traffic you haven't thought of is being delayed even more, so things may work worse until you figure out what you've missed. Its like taking a pile of garbage and making several smaller piles so you can get rid of the ugliest stuff first. It may look nicer, but you still have garbage, as long as you're making it faster then it can be taken away.

You don't want to transfer the delays from one type of traffic to another, you want to get rid of the delays or at least reduce them. Good bandwidth management is about getting rid of the delays althogether.

Internet Traffic is Largely Self-Throttling

People ask us all of the time how you can control traffic for an entire network in both directions from a single access point. They understand that you can send traffic out at a slower rate than it comes in, but how can the device control incoming traffic?

The answer is simple. Just about all protocols have built-in mechanisms to control flows, because otherwise things wouldn't work at different speeds. Most traffic is largely "self-throttling", because if you slow things down in one direction it slows down overall, adjusting to the environment. With TCP traffic, you can additionally "shrink" the window, which causes less traffic to be sent by the server. By doing this you lessen the amount of traffic that is in your network at any given time, which reduces the latency of ALL protocols.

Even other protocols, UDP based protocols for example, have similar characteristics. UDP traffic is often a predecessor to TCP traffic or some transfer, so by slowing it down you will slow what happens after it. Large chunks of data are usually not sent without some sort of acknowledgement. A server doesnt know the access speed of the client, so it can't just send data at wire speed to the client. Protocols are generally "well-behaved". You have to trust the fact that for the vast majority of cases, bandwidth can be controlled from a single point between the client and the server.

Getting rid of Congestion

Everyone knows what congestion is. Simply put, congestion is when you have more traffic then you have bandwidth. If you have more bandwidth than you need, then you don't need bandwidth management. So if you are reading this then you likely have a problem.

The "big picture" goal of any good bandwidth management strategy is to change the way traffic flows through your network in such a way that congestion is eliminated. Think of the situation when you have more bandwidth than you need. Suppose you have a 100Mb/s pipe and a PC that can only pull down 10Mb/s. Traffic flows freely through your router, it gets sent out as soon as it arrives. You don't need bandwidth management. You don't need to "prioritize" anything, because there is no backup. There is nothing to re-order.

Much of today's congestion is due to larger TCP windows being used in client systems. The larger the window, the more congestion you have. The more sessions, the more congestion. And as I've told you, the more congestion you have, the more delays you have, and the more difficult it is to manage your bandwidth. So the most important function that your bandwidth management device must do is TCP window shaping. Without window shaping you CANNOT reduce the amount of traffic in your router's queues, so the best you can do is shift delays from one user to another. Most products on the market do not properly window shape to reduce congestion.

Controlling Users

If you think about it, what you really want to do is control your users, not your protocols. Each "customer" is paying for something, and as long as no one uses more than they're supposed to, you won't have problems. Imagine a highway where you can force all of the cars to go a certain speed.

Per user management has many advantages. It allows you to properly engineer your network to determine how much bandwidth you need for your customer base, and how much bandwidth you can give to each one. It allows you to tier your offerings, so you can charge more for more bandwidth or higher quality service. In fact, its the exact way that large networks (such as telephone systems) are engineered. You have X number of users using the network at any given time and each is allowed to use Y bandwidth. Now a telephone network is easier to engineer because requirements are fixed, but you need a lot more bandwidth, because even if no-one is talking they are using bandwidth. On a data network, there are normally lots of users doing nothing or very little (reading a web page, composing mail, making a sandwich, etc). Thats where bursting comes in. Personally I don't like the term "bursting", because its inaccurate. But to some extent we're stuck with existing terminology. For the context of this discussion, "bursting" means "temporarily allowing higher speed transfers". Bursting techniques allow you to temporarily allow users to get more bandwidth when conditions permit it. Bursting allows you to dynamically utilize "excess" bandwidth in a fair and manageable manner.

How to implement an effective strategy

The hardest part about implementing an effective bandwidth management strategy is finding a product capable of implementing it. At minimum you'll need a product that has the following features:

• Can handle your traffic levels with a policy for every one of your customers/users
• Implements window shaping in order to reduce the overall amount of traffic that needs to be managed
• Has flexible bursting controls so that end user performance can be maximized and bandwidth can be dynamically shared

It doesn't seem like much, but the vast majority of products on the market cannot do what is needed. Luckily for you, our products can.

The only brain-work (as opposed to grunt-work) is to formulate your bandwidth policies. For the simplest case, you can define a default policy for everyone on your network. The elements of a bandwidth profile are as follows:

• Bandwidth Limits (directional or combined) - The amount of bandwidth a user can always get (but not guaranteed 100% of the time)
• Burst Limit - The maximum amount of bandwidth a user can get no matter what the conditions (again, not guaranteed)
• Minimum Bandwidth - Pre-allocated Bandwidth (guaranteed)
• Max Burst Duration - The maximum amount of time that continuous bursting can occur
• Burst Trigger - The condition which determines whether an end user can burst or not
• packet per second - The maximum packets/second that a user can get at any time

Before trying to implement it on your device, write it down in words. And use words that are quantifiable. If you use terms like "when bandwidth is available" you'll be in trouble. You have to define what that means. Typically you will not use the Minimum Bandwidth setting. In order to guarantee bandwidth, you must permenantly set aside bandwidth that cannot be shared. Thats a more advanced concept that will be explained in one of our more complex case studies.

Defining a Bandwidth Profile

The first thing that needs to be said is that there is no formula for engineering a general purpose data network. Every network is going to have a different mix of "little old ladies" who use their computer to find a recipe now and then and teenagers who download music all day long. The more little old ladies you have the more excess bandwidth you'll have on a regular basis. So you'll need to observe and tune. But that's all you'll have to do. If you are a university, then you have a more easily definable client base. All of your clients are bad. Teenagers are bad. But you can control them.

When defining a default bandwidth profile you have to be reasonable. If you have a T1 and 200 users you can't give each user 512K all of the time. If you want to be mathematical about it, you can multiply your available bandwidth by 4 and divide by the number of users online at any given time. So say that you have 6Mb/s and 200 users online at any given time. That's 24Mb / 200 = 120Kb/s. Using this, lets formulate a bandwidth profile. First, do it in words:

"Each user gets 120Kb/s and can burst to 512K for up to 60 seconds whenever overall network usage is below 5.9Mb/s"

What this means is that your default bandwidth profile will allow each user to always get 120Kbs and will allow 512Kb/s downloads for up to 60 seconds when conditions permit. So your profile would be:

Bandwidth-in = 120K
Bandwidth-out = 120K
Bandwidth-burst = 512K
Burst Max = 60
Burst Trigger = TriggerRule

Now that was pretty easy. Now you might ask why there is no minimum setting? The reason is that you are "assuming" that each user is not going to use 120K all of the time. That's why you can multiply your available bandwidth by 4. If you "guarantee" the bandwidth, you'd have to only allow 1/4th of the bandwidth per user. And thats not enough. Its simply not necessary to guarantee bandwidth if you engineer your network properly. More than likely, each user will be able to get their 120K whenever they want it.

Another element of a bandwidth profile is a packets per second setting. You can use PPS settings to minimize the unfair use of resources. After all, even though you are giving everyone 120Kbs you don't actually want them to use that 100% of the time. If everyone did you'd only have 1/4th the bandwidth that you need. Most normal internet use is not continuous, except when you are downlowning a large file or watching a large format video. Even so, the number of PPS to download a big file is small. 10 PPS allows for a 120Kbs download. But suppose you have 2 customers on the same service, one which is a small drug store and another which is a travel business with 50 users on an internal network. The drug store hardly ever uses their bandwidth, and the travel business has all of their agents online all day long. You really want the travel business to pay more than the drug store. Setting the profile to say 25 packets/second will make the setting more suitable for single users as it will slow things down when there are a lot of connections active. Its also a way to defeat many p2p protocols, which tend to send lots of little packets while discovering servers and sharing directories.

Secondary Topics - AutoShaping

Ok, so there's that word again "shaping". What does that mean? Well, to be truthful, its a bit difficult to explain what the ET/BWMGR AutoShaping feature does. But one thing I can tell you is that it's REALLY cool! Basically, it sets a threshold at which every session on your network is automatically slowed until an equilibrium is reached. Flows are reduced, sessions are paced, all in a very economical way that doesn't require much processing power, and its done in a very fair way. For a network that is only occasionally congested and fairly well-behaved, AutoShaping may be all that's needed to clean things up. Its also a way to manage bursting if you don't want to shut off bursting when your network is running at capacity. For example, the normal mechanism will shut down bursting when your network hits a threshold. If your network is constantly at capacity, this will result in bursting turning on and off as all of the sessions slow down and speed up. Autoshaping can be used to pace the bursting when required, which allows the clients to run faster than they would if bursting is shut down. We'll use Autoshaping in our MiniWISP example to illustrate.

Note on PPS rules

If you run a bridged network with lots of units on the same "network" then you may have a LOT more ARP packets flying around than you realize. For example on a cable network you may have 30pps streaming in on a regular basis. You'll need to either compensate for these in your PPS rules, or better yet not count ARPs as traffic, but placing a static rule at the beginning of your ruleset which matches ARPs.

Case Study - MiniWisp Wireless ISP

Network info:

• 12Mb/s of internet bandwidth
• 800 customers
• 500 IP addresses assigned by DHCP

Goals:

1. Set up a 4 tiered structure to allow for low-usage residential, broadband residential and 2 business offerings
2. Only allow "known" MAC addresses to access the internet through their wireless facilities
3. Monitor the network to control users running parasitic, abusive applications
4. Business Accounts can host a web site and get a static IP, whereas port 80 is blocked on residential

Implementation:

• Set Burst Threshold at 11.5Mb/s
• Set AutoShaping Threshold at 10Mb/s
• Set default policy to deny access.
• Created policy for each MAC in database (script generated initially to default profile)

Policy Definitions:

Default (Basic Residential)

Bandwidth In = 128Kb/s
Bandwidth Out = 56Kb/s
Burst = 512Kb/s
Burst Max = 30 seconds
PPS in = 30
PPS out = 30

Note: Allows for 1.35MB file download in burst period. (Note that the customers can burst to 512K, but can only do sustained downloads of 360K due to the 30pps limit, 30 * 1500 * 8). Limited outbound bandwidth. PPS rules limit overall activity (so that you can price a household with 6 people with laptops on a wireless router the same as one guy with a desktop).

Broadband Residential

Bandwidth In = 256Kb/s
Bandwidth Out = 56Kb/s
Burst = 750Kb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = 80

Note: Allows for download of 25MB file in burst period.

Business Standard

Bandwidth In =512Kb/s
Bandwidth out = 256Kb/s
Burst = 1Mb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = 80

Notes: Allows for greater bandwidth out and higher burst. Port 80 services blocked

Business Premium with Hosting

Bandwidth In =512Kb/s
Bandwidth out = 512KKb/s
Burst = 1Mb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = No Limit

Notes: Allows for higher outbound usage, port 80 services allowed.

Screen Shots of the Set up

The Profiles:

The Policies / Rules:

(click image to view full sized image in new window)

Some Notes on the Setup:

Note that the trigger must be a global rule, and that it could also be used to graph / monitor the interface usage (minus ARPs) by enabling stats on the rule. The port 80 handling rules must also be global. A packet will only be dropped if the FIRST rule that matches is a discard. All port 80 traffic will match rule 900. But for the dogood-hospital, 850 will match first, so the packets will not be dropped. You could also put this logic into the firewall, particularly if you only have a few cases where its allowed. However firewall rules are scanned sequentially, so if you have a lot of rules with IP or MAC addresses its more efficient to put them in the normal ruleset.

A non-wireless ISP (or one that has authentication and doesn't have to worry about unknown clients attaching to their network) can use IP addresses more efficiently rather than MACs. These are also easier to set up using our Do Range feature that allows you to ad any number of rules with sequential IP addresses.

Regarding triggers, you could have different triggers for different types of service. Perhaps you'd prefer to allow your premium customers to burst all the time for their Burst Max setting no matter what the traffic. You could create a trigger that is higher than your available bandwidth, such as 20Mb/s in this case. Since the trigger could never be reached, bursting would be enabled all of the time, so they'd always be able to burst for their max allowed burst period. Of course this would affect the engineering of your network.

Also, don't forget to allow your own hosts and servers if you are using the internal network. Be careful not to put rules here that will allow users to get more bandwidth than they are supposed to. However typically you don't count accessing your servers towards their bandwidth allocations.

Case Study - CityWide Internet

Network info:

• 45Mb/s of internet bandwidth
• 1600 customers
• 1024 IP addresses assigned by DHCP

Goals:

1. Set up a 4 tiered structure to allow for low-usage residential, broadband residential and 2 business offerings
2. Monitor the network to control users running parasitic, abusive applications
3. Business Accounts can host a web site and get a static IP, whereas port 80 is blocked on residential

Implementation:

• Set Burst Threshold at 43Mb/s
• Set AutoShaping Threshold at 38Mb/s
• Use AutoMgr to monitor long term usage for of basic residential clients
• Assign Static IPs to all premium service customers (this can be done with MAC mapping in your DHCP config. You could also create DHCP pools of MAC addresses in pariticular service tiers and then create generic static rules for those IPs).

Policy Definitions:

Default (Basic Residential)

Bandwidth In = 128Kb/s
Bandwidth Out = 56Kb/s
Burst = 512Kb/s
Burst Max = 30 seconds
PPS in = 30
PPS out = 30

Note: Allows for 2MB file download in burst period. Limited outbound bandwidth. PPS rules limit overall activity (so that you can price a household with 6 people with laptops on a wireless router the same as one guy with a desktop).

Broadband Residential

Bandwidth In = 256Kb/s
Bandwidth Out = 56Kb/s
Burst = 750Kb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = 80

Note: Allows for download of 25MB file in burst period. The profile allows longer burst periods simulating continuous broadband service without allowing for continuous use or resources.

Business Standard

Bandwidth In =512Kb/s
Bandwidth out = 256Kb/s
Burst = 1Mb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = 80

Notes: Allows for greater bandwidth out and higher burst. Port 80 services blocked

Business Premium with Hosting

Bandwidth In =512Kb/s
Bandwidth out = 512KKb/s
Burst = 1Mb/s
Burst Max = 240 seconds
PPS In = 80
PPS Out = No Limit

Notes: Allows for higher outbound usage, port 80 services allowed.

Screen Shots of the Set up

The Profiles:

The Policies / Rules

(Click image to see full sized image)