Comcast Service Overview

Comcast advertises their High Speed Internet service as having speeds of up to 1500kb/s downstream and 256kb/s upstream. In reality, the service is 1800/256, but it is not possible to reach those speeds. Most users never even see close to 1500kb/s when they download files. I've noticed that during downloads, the speeds are never consistent. Sometimes they'll spike up to 230kB/s, and then drop to 120kB/s, and then hover somewhere in between. Why the varying speeds? My DSL line doesn't seem to have that problem.

The Problem

The problem is the method that Comcast uses to cap connections. It's easy to say that a connection is capped to 1800/256, but the reality is that capping properly can be a complex issue. Some say that the capping is done by the cable modem, others say that it's done by the CMTS. According to this Cisco document  [edit Dec 22nd, 2004: the link seems to have died, don't bother clicking it], downstream capping is done by the CMTS, and upstream capping is done by the cable modem. To the end user, the location of the capping is irrelevant.

On most Comcast systems, capping is done as follows: Once a user has downloaded 1800kb of data within a 1 second period, no more data is allowed to pass until the second completes. Then data is allowed to pass until the cap is hit again. Whether packets that are over the limit are dropped or just delayed is not known (at least by me). If packets are dropped, there will be a delay as they will need to be sent again. All of this wreaks havoc on the user's download rate. Not all Comcast systems may experience this, and I believe that ATTBI already does traffic shaping on the CMTS, so they have no need for shaping on the user end.

Traffic Policing Versus Traffic Shaping

What Comcast appears to be doing can be described as traffic policing. Their network simply measures the transfer, when the cap is exceeded in a 1 second period, it pauses until the end of that second. A better method is known as traffic shaping. Traffic shaping regulates the data transfer over the course of the entire second, keeping data transfers smooth, enabling the user to get maximum throughput. Cisco CMTS's support traffic shaping, but it doesn't appear that Comcast has it enabled.

The Solution

Since Comcast isn't using traffic shaping, I've decided to try it out myself using FreeBSD and Dummynet. The FreeBSD will act as the natd machine, and will be hooked up directly to the cable modem. There will be a second network card in the machine that will go to the local lan. In this case, it's hooked up via a crossover cable directly to my Windows XP workstation. Once natd is working properly, it's time for the firewall rules.

The rules are as follows:

/sbin/ipfw add prob 1 pipe 1 ip from any to any in via vx0
/sbin/ipfw pipe 1 config bw 1765kbits/s

/sbin/ipfw add prob 1 pipe 2 ip from any to any out via vx0
/sbin/ipfw pipe 2 config bw 255kbits/s

The first two rules limit the data rate for any type of IP data going into my external ethernet card (vx0) to 1765kb/s - which is 35kb/s below Comcast's caps. The second two rules limit the data going out of the card to 255kb/s, which seems to work best in my tests.

With that done, let's see how well it works.

Speed Tests

In order to quickly gauge the effectiveness of the FreeBSD traffic shaping, I used the web based speed test located at phl.speakeasy.net. The server is located 13 hops from my connection, and pings around 15ms. I performed 4 pairs of speed tests over the course of several hours. For the first pair, I tested without shaping, and then immediately again with shaping enabled. For the second test, I started with shaping enabled, and then then immediately tested with it disabled. I repeated the process again later using the pattern above.

The above graph clearly shows the gains made in the speed test by using traffic shaping. Download speed gained over 300kb/s, and became more consistent. Upload speed gained nearly 90kb/s on average. By tweaking my shaping limits further, I was actually able to break the 1700kb/s mark on this speedtest, averaging about 1706kb/s.

One could argue that testing over a period of time of several hours may not give an accurate representation of the effectiveness of traffic shaping. Although my sample data was only over a period of several hours, I have been consistently repeating these results for the last few days.

Real World Testing

While the web based speed tests are a quick way to compare speeds, they may not always give a real world representation of connection performance. Another way to perform connection testing is by measuring raw throughput of one large file. For this test, I downloaded a 64MB test file from ftp1.optonline.net. To help show the results of the traffic shaping for this test, I've taken some screenshots of Analog X's NetStat Live, which measures current, average, and max bandwidth consumption, and also displays data with a graph.

Below you can see some screenshots of NSL while the 64MB test file was downloading. The first two were taken without using traffic shaping. I've highlighted the two sections of importance, the incoming and outgoing bandwidth. There are 3 numbers each for the incoming and outgoing sections. The left most number is the current bandwidth usage. The center is average usage, and the right number is the maximum usage. The graph directly below these numbers is a representation of bandwidth consumption over the last minute or so. Keep in mind the numbers shown in the graph is the raw transfer rate, including TCP/IP overhead.

Looking at the maximum bandwidth number shows us that indeed it is possible to go above the 1800kb/s caps, with bursts of 2367kb/s and 3000kb/s. Unfortunately, when this happens the cable modem/CMTS retalliates and delay/drop packets. This causes our average bandwidth utilization to drop to around the 1600kb/s range, and gives us our peaks and valleys on the graphs. The dropping and queuing of packets for long periods of time have an adverse effect on our download speeds.

Fig 1. - Traffic meter examples without shaping

Compare the above graphs with the one below, taken when downloading the test file with traffic shaping enabled. The current, average, and max utilization are all within 30kb/s of each other. That gives us a consistent download speed, which the graph shows off nicely. Most importantly, the maximum utilization is 20kb/s under the caps, which keep us from facing the wrath of the cable modem/CMTS. As you can see, my average throughput with traffic shaping enabled is 150-200kb/s greater than without it.

Fig 2. - Traffic meter example with shaping enabled

Overall, the results are as follows:

Without any traffic shaping, the 64MB file downloaded in 5:53, an average speed of 185.7kB/s.

With traffic shaping, the 64MB file downloaded in 5:18, an average speed of 206.9kB/s.

While not Earth shattering, it can make a big difference if you tend to download large files, such as Linux (or FreeBSD) ISO images.

Analysis Shortcomings

Conclusion

From what I've seen taking the time to set up traffic shaping is worth the effort. One can never complain about faster downloads, and possibly better gameplay on any of the online games I may be playing at the moment. Another nice feature is that tweaking of individual machines doesn't need to be done, since the FreeBSD natd machine takes care of it for all machines on the lan. Of course, if Comcast would put forth the effort to enable traffic shaping on the CMTS's, none of this would be necessary.

Credits

Traffic shaping to improve performance on CHSI is not my idea. I wasn't even aware of the method that Comcast is using to implement caps until it was brought to my attention on the Broadband Reports Comcast Forum. There have been several discussions on traffic shaping, and that is where the links to the Cisco documents above came from. I'd like to thank the always watchful and clueful users of the Comcast forum for bringing this to my attention.