Speed of JavaScript framework's DOM selection within mobile widgets

This is the second part of my tests of JavaScript frameworks within mobile widgets. You may find the first part at Speed of JavaScript framework’s DOM selection within mobile widgets

JavaScript frameworks, such as Dojo, ExtJS, jQuery, MooTools, Prototype, and YUI, - to mention only a few - provide a great way for (web) developers to ease development of (web) applications. They make it possible to write code that runs in various environments without the need to make specific changes for different browsers.

As you may know from the first part of my tests, I have been involved with the Vodafone Widgets team for the last couple of weeks. Well, actually it has been several months now. Anyway, after the results of the first tests I wanted to see how JavaScript frameworks perform in a more “real world” situation. Therefore, I created a widget out of the TaskSpeed webpage and ran several hundred tests on various mobile phones that support widgets and are approved by Vodafone.

Tested mobile phones

At the time I carried out the first series of tests a few weeks ago, there were eleven (11) phones available that Vodafone approved for the Vodafone Apps Manager runtime. Today, there are more phones approved by Vodafone, but to be able to compare the results of this series of tests and the first series of tests, I tested the same phones this time than I did the last time.

The tested mobile phones are (in alphabetical order):

Nokia 6110 Communicator
Nokia 6124 classic
Nokia 6210 Navigator
Nokia E66
Nokia N73
Nokia N78
Nokia N85
Nokia N95
Nokia N96
Samsung SGH-i550
A phone that is currently not publicly available; therefore, I can not publish any test results on this phone

In general, every phone that runs on Symbian S60 v3.0 Feature Pack 1 is capable of running the Vodafone Apps Manager. But the mobile phones mentioned above are tested and approved by Vodafone.

Tested JavaScript frameworks

The tested JavaScript frameworks are (in alphabetical order):

Dojo v1.2.3
Dojo v1.3.0
jQuery v1.2.6
jQuery v1.3.2
MooTools v1.2.1
Prototype v1.6.0.3
YUI v2.7.0 Selector

How the tests were executed

If you read my blog post on the first series of tests, you can scroll down to the results, since this chapter is only a slightly modified version of the chapter about how the tests were performed from the previous post, adapted to the TaskSpeed test suite.

To make use of the rendering engine that is available within the Vodafone Apps Manager runtime - instead of the phone’s native browser -, I wrote a widget that is based on the TaskSpeed Test Suite, which is itself based on the SlickSpeed Selectors Test created by the people at mootools.net, with some modifications by Peter Higgins, who is involved with the Dojo team. The code of the TaskSpeed Test Suite is available at GitHub.

There were a couple of things I had to bear in mind. First, because of the limitations in screen size and resolution, the look of the test suite was slightly modified. Furthermore, only one framework at a time instead of several frameworks at once was loaded and tested. This was done because of the memory limitations some of the tested mobile phones have. There is simply not enough memory available to load all frameworks at once. Additionally, I implemented a little menu to make it easier for me to choose from the available framework tests (see screenshot 1 below).

The original TaskSpeed Test Suite makes use of PHP to create the XHTML and parts of the JavaScript dynamically. Within the Vodafone Apps Manager, there is no PHP interpreter available. Therefore, to avoid any delays caused by network traffic while loading the files from a distant server, the code written in PHP was completely replaced by pure XHTML and JavaScript.

Every framework was tested five times on every mobile phone. The average of these five tests per framework and phone was then taken over into the test results. All in all, 385 tests were carried out.

Here is an example of how a test was carried out (the screenshots were taken from a Nokia N95):

First, the main menu is shown, in which one has to choose the framework they want to test. For this demo, the MooTools framework in version 1.2.1 was choosen (this choice was purley random).

Screenshot 1: Mobile Framework Tests 2 - Menu

The (modified) TaskSpeed Test Suite is then loaded.

Screenshot 2: Mobile Framework Tests 2 - Loaded TaskSpeed Suite

After a click on start tests , the different tests are performed. This may take some time, depending on the mobile phone the test is run on

Screenshot 3: Mobile Framework Tests 2 - Tests are running

After all test are finished, the final test result is displayed at the bottom.

Screenshot 4: Mobile Framework Tests 2 - Final results

Results

Below you will find several .pdf-files containing the results of the tests. I grouped the results by moble phone and by framework.

Results by mobile phone

Results by framework

What was true for the first series of tests applies to this series of tests as well: The tests on a mobile phone take a lot - and I mean a lot - more time than they do on a desktop computer or laptop. For example, the tests of the Prototype v1.6.0.3 framework take about 475ms on my MacBook 2.4GHz Core 2 Duo with 2GB RAM, while they take about 692300ms on a Samsung SGH-i550. So again, keep in mind that when you write your own mobile widgets, processors installed into mobile phones are not as powerful as they are in desktop computers or laptops and that there is usually not as much RAM available as on desktop computers or laptops.

Final words

From the different charts it is clearly visible that some frameworks perform (much) better than others. The slowest framework in this series of tests was the Prototype v1.6.0.3 framework, followed by the jQuery v1.2.6 framework. The fastest framework was the Dojo v1.3.0 framework.

Of course, the mobile device plays a significant role when it comes to performance of the different frameworks. The Samsung SGH-i550 was by far the slowest phone, no matter which framework was tested on that device. For example, running the Prototype v1.6.0.3 framework on that phone took more than 11 minutes on average! The fastest phones were the Nokia 6210 Navigator and the Nokia N85, very closely followed by the Nokia E66.

I know, I repeated things from the first series of tests quite often in this post, but I will do it for the final time now: Hopefully, the tested - well established - frameworks will soon be optimized to perform better on mobile phones. After that, I am sure, they will be of great value for the mobile widget developer, just like they are for web developers today.

Speed of JavaScript framework's DOM selection within mobile widgets - part 2