Brain Murmurs

I’ve been doing a lot of benchmarking to try and characterize our system performance and thought I’d post these numbers to give people an idea of what to expect from Pulse on their handhelds.

Once we had the system up and running, we started measuring performance by logging on with Pulse and using the download speed test provided at bandwidth.com. Our original numbers were quite poor (about 33 kps for an 8700g on an EDGE network), so we started putting together a series of tests to figure out where the bottleneck might be.

We wrote a suite of applications to pound the living daylights out of the mac-to-blackberry Bluetooth connection and the blackberry-to-proxy server network connection. The work was useful for a few reasons: first off, it gave us the absolute best possible performance numbers we could expect to see in Pulse. Secondly, it allowed us to fine tune the settings for those connections and really optimize our performance as much as possible. We were able to dramatically improve our network performance based on the results of the TCP Blaster suite tests. The BlueBlaster results were less helpful in boosting performance, but were eye opening nonetheless.

The results of the tests appear below:

B E N C H M A R K S
Model	Bluetooth Raw Bandwidth	Network Type	Network Raw Bandwidth	Pulse Bandwidth
7290	40kps	GPRS	40 kps	29 kps
8700g	70kps	EDGE	205 kps	68 kps
Pearl	400kps	EDGE	205 kps	111 kps

Analysis

The 7290 is bandwidth limited on both sides with Bluetooth and GPRS speeds of 40 kps. We are currently realizing an actual overall performance of about 29 kps and will continue to see if we can do anything to bring the performance up to the 29 kps limit.

The 8700 is bandwidth limited by its Bluetooth connection, which we haven’t been able to drive any faster than 72 kps. On the other hand, the EDGE network provided excellent TCP throughput with measured bandwidths of up to 205 kps. The current actual Pulse performance is around 68 kps, which indicates that we are probably hitting the limit imposed by the Bluetooth connection.

The Pearl is a monster, supporting a 400 kps Bluetooth connection and the same 205 kps network performance. Since Pulse topped out at around 115 kps it seems likely that, with this model at least, our system is the current bottleneck.

Comments
The poor Bluetooth performance on the 8700 and 7290 models was a real shock, as was the intense bandwidth provided by the Pearl’s Bluetooth implementation. The 8700 looks like a strong candidate for a USB-enabled version of Pulse, as the 70 kps Bluetooth connection appears to be cutting its potential performance down by about 40%. Some additional optimization for the 7290 will probably bring it closer to its 40 kps limit, but switching to USB isn’t going to help as it will then be network limited.

The Pearl’s performance is excellent, but was tested largely to help characterize Pulse’s current capabilities. Since the Pearl already ships with a better performing dial up networking system the numbers are of theoretical value only.

Based on these findings we will most likely add USB support for Pulse over the next month.

Filed under: Pulse by Daniel