For some background on testing methodology using Android and iOS.
The Development Platforms
- Native (Objective-C 64 bit and Java)
- Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
- Classic Xamarin (64 bit unified beta for iOS)
- Xamarin.Forms (64 bit unified beta for iOS with beta version of Xamarin.Forms, note latest version of the unified API in the beta/alpha channels could not be used as it is not supported by Xamarin.Forms Note: 1.3.1 pre 1 was released Dec 24th so Xamarin.Forms may now work with the version of the unified iOS API in the alpha and beta channel)
The Devices
- iPad Mini (non Retina) running iOS 8.1.1 (12B435)
- ASUS K00F running Android 4.2.2
The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good". But they looked about the same. There were some differences such as Java, Classic Xamarin and Xamarin.Forms rendered the tabs on the top in Android as expected while the JavaScript library showed them on the bottom.
The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch. Each timing was taken ten times and the results were averaged. It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error.
Test 1: Test App Size
The size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times. For Android the size of the APKs was examined. For iOS I looked at Settings to find out how much space the apps took up on disk.
| Development Platform | Size |
|---|---|
| Android | |
| Java | 166kb |
| Cordova | 433kb |
| Classic Xamarin | 3.5mb |
| Xamarin.Forms | 4.7mb |
| iOS | |
| Objective-C (64 bit) | 644kb |
| Cordova | 2.7mb |
| Classic Xamarin | 12.1mb |
| Xamarin.Forms | 16.9mb |
When it comes to application size Xamarin shows the extra size involved in the overhead of the .Net framework. There was an attempt to reduce the size of the deployed Xamarin application by using the "Link SDK assemblies only" setting. I am surprised in a very small application how large the difference is. However, from experience in "real" applications the difference is much less consequential as graphics and frameworks get added to the projects.
Test 2: Load Times
I wanted to see how long it took the application to load into memory. While the initial load time is important, many mobile applications tend to stay in memory so it tends to have a limited impact. For this test I made sure to close all applications before each timing.
| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 1.085 |
| Cordova | 3.978 |
| Classic Xamarin | 1.704 |
| Xamarin.Forms | 2.764 |
| iOS | |
| Objective-C | 1.221 |
| Cordova | 1.715 |
| Classic Xamarin | 1.28 |
| Xamarin.Forms | 1.813 |
In all cases the vendor native technologies loaded the fastest. Classic Xamarin loaded nearly as fast as the native languages. Xamarin.Forms and Cordova had the slowest load times. The Cordova load time on Android was particularly bad while on iOS the load times were close enough to not be a huge factor.
Test 3: Loading a List from Azure Mobile Services
In this test I wanted to look at getting data from an external service so I loaded 1000 records from Azure Mobile Services. For Xamarin iOS 64 bit I had to modify the Azure Mobile Services to be compatible with the unified API. The timings were taken from pushing the button to load the list until the results visibly came back and were displayed on a list on the screen.
Java:
public void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
if (dbConn == null) {
openConnection();
}
ContentValues values = new ContentValues();
values.put("firstName", firstName);
values.put("lastName", lastName + index);
values.put("misc", misc);
dbConn.insertOrThrow(TABLE_NAME, null, values);
}
Objective-C:
- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
NSString *sqlStatement = NULL;
char *errInfo;
*error = nil;
if (dbConn == nil) {
[self openConnection:error];
return;
}
sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
if (result != SQLITE_OK) {
NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
[errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
*error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
}
}
JavaScript:
db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
successCount++;
if (successCount === maxValue) {
$.ui.popup({
title: "Success",
message: "All records written to database",
doneText: "OK",
cancelOnly: false
});
$.ui.unblockUI();
}
}, function (e) {
$.ui.popup({
title: "Error",
message: "An error has occurred adding records: " + e.toString(),
doneText: "OK",
cancelOnly: false
});
$.ui.unblockUI();
return;
});
Xamarin (All Versions):
public void AddRecord(string fName, string lName, int i, string m)
{
if (dbConn == null)
{
OpenConnection();
}
var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};
dbConn.Insert(testRecord);
}
Xamarin Classic Android Alternate:
public void AddRecord(string firstName, string lastName, int index, string misc)
{
if (dbConn == null)
{
OpenConnection();
}
ContentValues values = new ContentValues();
values.Put("firstName", firstName);
values.Put("lastName", lastName + index);
values.Put("misc", misc);
dbConn.InsertOrThrow(TABLE_NAME, null, values);
}
| Development Platform | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 | Test 8 | Test 9 | Test 10 | Test Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Android | |||||||||||
| Java | 22.71 | 17.5 | 18.04 | 17.7 | 18.63 | 20.33 | 2.68 | 2.42 | 2.16 | 2.34 | 2.369 |
| Cordova | 25.99 | 24.76 | 27.05 | 23.3 | 24.06 | 22.86 | 2.12 | 2.02 | 1.94 | 2.48 | 2.149 |
| Classic Xamarin | 34.07 | 27.38 | 32.05 | 38.77 | 29.27 | 34.63 | 1.61 | 1.63 | 1.84 | 1.85 | 1.738 |
| Xamarin.Forms | 1.99 | 1.76 | 2.32 | 1.91 | 1.9 | 1.58 | 1.93 | 2.02 | 2.03 | 1.64 | 1.908 |
| iOS | |||||||||||
| Objective-C | 2.38 | 2.44 | 2.24 | 2.3 | 2.34 | 2.32 | 2.32 | 2.35 | 2.2 | 2.27 | 2.316 |
| Cordova | 3.57 | 2.18 | 2.07 | 1.95 | 1.97 | 2.05 | 2.04 | 1.93 | 2.2 | 1.96 | 2.192 |
| Classic Xamarin | 2 | 1.87 | 1.88 | 2.06 | 1.74 | 1.9 | 1.81 | 1.94 | 1.75 | 1.96 | 1.891 |
| Xamarin.Forms | 2.11 | 2.01 | 2.23 | 1.96 | 1.95 | 2.07 | 2.12 | 2.16 | 2.08 | 2.1 | 2.079 |
In many ways this test is showing how well the Azure Mobile Services libraries perform on the different platforms. Unsurprisingly Xamarin, with it's underpinnings of a .Net implementation, performs the best in this test. I was surprised to see the libraries for the native technologies perform the worst, both on Android and iOS.
I have heard that the Xamarin.Forms lists can perform poorly with large data sets. These results did not show that, at least with lists of up to 1000 records.
Test 4: Prime Number Calculation
In the final of my first series of tests I wanted to try out a CPU intensive operation. I created a Sieve of Eratosthenes on each of the platforms. My first plan was to calculate all prime numbers up to 50,000,000. This required some special handling of the method's array for both Objective-C and JavaScript. In the case of Objective-C I had to malloc memory to support arrays that large. Also for Objective-C and JavaScript I had to initialize the array items to 0. To keep the timings the same I did the array item initialization to 0 on all platforms even though it could have been left out for .Net (and Java I believe). It that was done, the .Net timings would have been even better.
I did end up having to settle for only calculating primes up to 5,000,000. The reason for this is that the JavaScript performed so poorly that I was unwilling to wait for it to complete 10 times.
Java:
private int getPrimesFromSieve(int maxValue)
{
byte[] primes = new byte[maxValue + 1];
for (int i = 0; i <=maxValue; i++)
{
primes[i] = 0;
}
int largestPrimeFound = 1;
for (int i = 2; i <=maxValue; i++)
{
if (primes[i - 1] == 0)
{
primes[i - 1] = 1;
largestPrimeFound = i;
}
int c = 2;
int mul = i*c;
for (; mul <= maxValue;)
{
primes[mul - 1] = 1;
c++;
mul = i*c;
}
}
return largestPrimeFound;
}
Objective-C:
- (int) getPrimesFromSieve: (int) maxValue {
Byte *primes;
primes = (Byte *) malloc(maxValue * sizeof(Byte));
for (int i=1; i<=maxValue; i++)
{
primes[i-1] = 0;
}
int largestPrimeFound;
largestPrimeFound = 1;
for (int i=2; i<=maxValue; i++)
{
if(primes[i-1] == 0)
{
primes[i-1] = 1;
largestPrimeFound = i;
}
int c=2;
int mul = i*c;
for(; mul <= maxValue;)
{
primes[mul-1] = 1;
c++;
mul = i*c;
}
}
return largestPrimeFound;
}
JavaScript:
function getPrimesFromSieve(maxValue) {
var primes = new Uint8Array(new ArrayBuffer(Number(maxValue)));
for (var i = 0; i <=maxValue; i++) {
primes[i] = 0;
}
var largestPrimeFound = 1;
for (i = 2; i <= maxValue; i++) {
if (primes[i - 1] == 0) {
primes[i - 1] = 1;
largestPrimeFound = i;
}
var c = 2;
var mul = i * c;
for (; mul <= maxValue;) {
primes[mul - 1] = 1;
c++;
mul = i * c;
}
}
return largestPrimeFound;
}
Xamarin (All Versions):
public static int GetPrimesFromSieve(int maxValue)
{
var primes = new byte[maxValue + 1];
for (var i = 0; i <=maxValue; i++)
{
primes[i] = 0;
}
var largestPrimeFound = 1;
for (var i = 2; i <=maxValue; i++)
{
if (primes[i - 1] == 0)
{
primes[i - 1] = 1;
largestPrimeFound = i;
}
var c = 2;
var mul = i*c;
for (; mul <= maxValue;)
{
primes[mul - 1] = 1;
c++;
mul = i*c;
}
}
return largestPrimeFound;
}
| Development Platform | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 | Test 8 | Test 9 | Test 10 | Test Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Android | |||||||||||
| Java | 4.31 | 4.31 | 4.2 | 4.33 | 4.39 | 4.37 | 4.32 | 4.45 | 4.34 | 4.4 | 4.342 |
| Cordova | 91.69 | 95 | 94.31 | 94.4 | 94.73 | 94.1 | 94.1 | 91.8 | 93.63 | 97.75 | 94.151 |
| Classic Xamarin | 4.27 | 4.25 | 4.15 | 4.32 | 4.51 | 4.41 | 4.22 | 4.12 | 4.14 | 4.19 | 4.258 |
| Xamarin.Forms | 4.21 | 4.17 | 4.31 | 4.3 | 4.2 | 4.34 | 4.29 | 4.36 | 4.22 | 4.19 | 4.259 |
| iOS | |||||||||||
| Objective-C | 5.04 | 5.49 | 5.38 | 4.86 | 4.8 | 5.02 | 5.03 | 4.83 | 4.84 | 4.85 | 5.014 |
| Cordova | 66.96 | 67.36 | 67.22 | 67.3 | 67.17 | 67.44 | 67.13 | 67.11 | 67.58 | 67.64 | 67.291 |
| Classic Xamarin | 4.41 | 4.42 | 4.35 | 4.34 | 4.49 | 4.37 | 4.17 | 4.27 | 4.39 | 4.28 | 4.349 |
| Xamarin.Forms | 4.51 | 4.33 | 4.31 | 4.31 | 4.33 | 4.4 | 4.41 | 4.4 | 4.33 | 4.46 | 4.379 |
While I was expecting that JavaScript would be slower, I was unprepared for how much worse it was for this type of operation. This would make Cordova problematic for highly CPU bound work were performance is important (I wonder what that means for server side Node.js...). CPU bound work is not as important in many of today's mobile applications but given the history of increased performance for mobile CPUs and what happened in the PC market in the 1990's, it is likely that in the future CPU bound work will be more prevalent in mobile applications.
Having said that, using native HTML commands can be very fast. In the past I've testing loading JSON using HTML commands vs. the JSON.Net library on Xamarin and found them nearly comparable.
I was also surprised that Xamarin performed better than Objective-C by a noticeable amount. It also performed better than Java on Android but by a very marginal amount, well within the .2 second margin of error that manual timings give us.
That's it for my first installment of performance tests. Much of this code was made more difficult for Xamarin due to the flux around the 64 bit Unified iOS API. For next month I'll take a look at loading large JSON strings and perhaps something else.
I hope this is useful or you. If you have any ideas for performance tests I can perform, I'd love to hear them.
Source code for the tests can be found here: Performance Tests Source
