For some background on testing methodology using Android and iOS.
The Development Platforms
- Native (Objective-C 64 bit and Java)
- Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
- Classic Xamarin (64 bit unified beta for iOS)
- Xamarin.Forms (64 bit unified beta for iOS with Xamarin Forms version 1.3.1)
The Devices
- iPad Mini (non Retina) running iOS 8.1.1 (12B435)
- ASUS K00F running Android 4.2.2
The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good". But they looked about the same.
The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch. Each timing was taken ten times and the results were averaged. It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error. In the previous post I showed the 10 individual timings and the average. This time to save me some typing I'm just showing the average of the ten timings.
Test 1: Test App Size
As mentioned in the last set of test, the size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times. For Android the size of the APKs was examined. For iOS I looked ipa files for ad-hoc deployment.
| Development Platform | Size |
|---|---|
| Android | |
| Java | 921kb |
| Cordova | 417kb |
| Classic Xamarin | 2.1mb |
| Xamarin.Forms | 3.4mb |
| iOS | |
| Objective-C (64 bit) | 55kb |
| Cordova | 621kb |
| Classic Xamarin | 2.95mb |
| Xamarin.Forms | 8.17mb |
A few interesting things to look at here. First is the Cordova apk is actually smaller than the vendor native tools on Android. I suspect this has to do with differences in size of the SqLite libraries for both of the platforms. Also while the Xamarin APKs are larger, they are not as large as they were on the first set of tests. For this I used the linking option of Link All Assemblies. This was probably important for Xamarin Forms which is, of course, just a library on top on Classic Xamarin. For whatever reason I was not able to get the Link All Assemblies option to stay with Xamarin iOS and you see the results with Link SDK assemblies only instead.
Test 2: Load Times
Like the last test I verified how long the applications took to load into memory. The results were similar to the set of tests I did previously.
| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 1.044 |
| Cordova | 4.068 |
| Classic Xamarin | 1.689 |
| Xamarin.Forms | 3.032 |
| iOS | |
| Objective-C | 1.14 |
| Cordova | 2.138 |
| Classic Xamarin | 1.204 |
| Xamarin.Forms | 1.928 |
As last time vendor native technologies load the fastest. Xamarin Classic follows closely with Xamarin Forms somewhat after that. In all cases, Cordova is the slowest loading. It is interesting that on iOS Xamarin Forms loaded almost as slowly as the Cordova app. Of course for iOS, nothing really took that long to load.
Test 3: Adding 1,000 records to SqLite
I originally wanted to add 10,000 records but found that too slow on my Android device and also it didn't scroll well under Cordova where my implementation had no auto paging. Offline storage is a common need with mobile applications and SqLite is one of the few solutions that span all platforms.
* I have had a few comments that there are better performing ways to do this test. For example I could have inserted 1,000 records in the same transaction or if I wanted to do it one one record in one transaction at a time, I could have done it asynchronously. Both of these points are undoubtedly true. Through SqLite you can get 1,000 records into the local database in much quicker ways on all platforms than this tests indicates. What this test is doing is looking at synchronously saving a single record in a single implicit transaction and doing that 1,000 times for all platforms. It would probably not be too difficult to do the other tests as suggested and they may be the source of a future post. Thank you to everyone who commented.
Java:
public void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
if (dbConn == null) {
openConnection();
}
ContentValues values = new ContentValues();
values.put("firstName", firstName);
values.put("lastName", lastName + index);
values.put("misc", misc);
dbConn.insertOrThrow(TABLE_NAME, null, values);
}
Objective-C:
- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
NSString *sqlStatement = NULL;
char *errInfo;
*error = nil;
if (dbConn == nil) {
[self openConnection:error];
return;
}
sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
if (result != SQLITE_OK) {
NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
[errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
*error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
}
}JavaScript:
function addRecords() {
$.ui.blockUI();
var maxValue = 999;
var successCount = 0;
var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });
for (var i = 0; i <= maxValue; i++) {
var lastName = "person" + i.toString();
db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
successCount++;
if (successCount === maxValue) {
$.ui.popup({
title: "Success",
message: "All records written to database",
doneText: "OK",
cancelOnly: false
});
$.ui.unblockUI();
}
}, function (e) {
$.ui.popup({
title: "Error",
message: "An error has occurred adding records: " + e.toString(),
doneText: "OK",
cancelOnly: false
});
$.ui.unblockUI();
return;
});
}
}Xamarin (All Versions):
public void AddRecord(string fName, string lName, int i, string m)
{
if (dbConn == null)
{
OpenConnection();
}
var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};
dbConn.Insert(testRecord);
}
Xamarin Android Alternate:
public void AddRecord(string fName, string lName, int i, string m)
{
if (dbConn == null)
{
OpenConnection();
}
var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};
dbConn.Insert(testRecord);
}
| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 18.569 |
| Cordova | 24.126 |
| Classic Xamarin | 32.55 |
| Xamarin.Forms | 30.873 |
| Xamarin Classic Alternate | 18.341 |
| iOS | |
| Objective-C | 8.044 |
| Cordova | 14.944 |
| Classic Xamarin | 8.151 |
| Xamarin.Forms | 8.137 |
Right off the bat you might notice that Xamarin on Android did poorly. Very poorly. Slower than native Android or Cordova by a large margin. One thing I noticed is that in native Android I used the SQLiteOpenHelper but this is only available in the Android API. I used the method I did because it was cross platform and you can see the results on Android. SQLiteOpenHelper is available in the SqLite Xamarin Library for Android and on a hunch I tried it instead and you can see those timings under Xamarin Classic Alternate for Android and the timing were virtually the same as they were for Java. I suspect if I used this for Xamarin.Forms I would have gotten about the same advantages. The lesson from this is just because there is a cross platform version of the API, it doesn't mean it will perform well. Buyer beware.
Test 4: Querying SqLite Records
I wanted to test reading the 1,000 records I just wrote. I only did the 1,000 I originally wrote because I didn't want to deal with paging solutions for Cordova. A 10,000 record test performed very poorly when just creating a 10,000 for HTML table.
Java:
public ArrayList<String> getAllRecords() throws Exception {
if (dbConn == null) {
openConnection();
}
ArrayList<String> returnValue = new ArrayList<String>();
String sqlStatement = "SELECT * FROM " + TABLE_NAME;
Cursor cursor = dbConn.rawQuery(sqlStatement, null);
if (cursor.moveToFirst()) {
do {
returnValue.add(cursor.getString(1) + " " + cursor.getString(2));
} while (cursor.moveToNext());
}
return returnValue;
}
Objective-C:
- (NSMutableArray*)getAllRecords:(NSError**) error {
NSString *sqlStatement = NULL;
NSMutableArray *results;
sqlite3_stmt *sqlResult;
char *errInfo;
*error = nil;
if (dbConn == nil) {
[self openConnection:error];
return nil;
}
sqlStatement = @"SELECT * FROM testTable";
results = [[NSMutableArray alloc] init];
int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
result = sqlite3_prepare_v2(dbConn, [sqlStatement UTF8String], -1, &sqlResult, nil);
if (result == SQLITE_OK) {
while (sqlite3_step(sqlResult)==SQLITE_ROW) {
NSString* firstName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 1)];
NSString* lastName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 2)];
NSString* fullName = [NSString stringWithFormat:@"%@ %@", firstName, lastName];
[results addObject:fullName];
}
sqlite3_finalize(sqlResult);
return results;
} else {
NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
[errorDetail setValue:@"Error loading records" forKey:NSLocalizedDescriptionKey];
*error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
return nil;
}
}function showSqlRecords(sql) {
var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });
db.executeSql(sql, [], function (res) {
try {
var lstRecords = $("#lstRecords");
lstRecords.empty();
var lastValue = res.rows.length;
for (var i = 0; i < lastValue; i++) {
var listItem = "<li>" + res.rows.item(i).firstName + " " + res.rows.item(i).lastName + "</li>";
lstRecords.append(listItem);
}
} catch (err) {
alert(err.toString());
}
});
}Xamarin (All Versions):
public IList<string> GetAllRecords()
{
if (dbConn == null)
{
OpenConnection();
}
var results = (from t in dbConn.Table<TestTable>() select t ).ToList();
var returnList = new List<string>();
foreach (var result in results)
{
returnList.Add(string.Format("{0} {1}", result.firstName, result.lastName));
}
return returnList;
}Xamarin Android Alternate:
public IList<string> GetAllRecords()
{
if (dbConn == null) {
OpenConnection();
}
var returnValue = new List<String>();
var sqlStatement = "SELECT * FROM " + TABLE_NAME;
var results = dbConn.RawQuery(sqlStatement, null);
if (results.MoveToFirst()) {
do
{
returnValue.Add(results.GetString(1) + " " + results.GetString(2));
} while (results.MoveToNext());
}
return returnValue;
}| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 0.551 |
| Cordova | 1.117 |
| Classic Xamarin | 1.144 |
| Xamarin.Forms | 0.89 |
| Xamarin Classic Alternate | 0.601 |
| iOS | |
| Objective-C | 0.792 |
| Cordova | 1.1745 |
| Classic Xamarin | 0.799 |
| Xamarin.Forms | 0.735 |
Reading records in Android was similarly bad for Xamarin on Android until I switched to the SQLiteOpenHelper implementation and then performance came online with the native example. As usual the interpreted JavaScript in Cordova slower but probably won't be a factor unless there is a large operation reading thousands of lines.
Test 5: Writing Lines to a File
I also wanted to test writing lines to a text file and how that performs. For each platform I write 1,000 lines to a file.
Java:
public void writeLineToFile(String line) throws Exception {
if (!textFile.exists()) {
this.createFile();
}
if (fileHandle == null) {
this.openFile();
}
fileHandle.write(line);
}
Objective-C:
- (void)writeLineToFile:(NSError**)error withTextToWrite:(NSData*)textToWrite {
*error = nil;
if (fileHandle == nil) {
[self openFile:error];
}
if (*error == nil) {
[fileHandle seekToEndOfFile];
[fileHandle writeData:textToWrite];
}
}file.createWriter(function (fileWriter) {
fileWriter.seek(fileWriter.length);
var count = 0;
var line;
var message = "Writing line to file at index: ";
var maxLines = 999;
fileWriter.onwriteend = function(evt) {
count += 1;
if (count <= maxLines) {
line = message + count + "\\n";
fileWriter.write(line);
} else {
$.ui.unblockUI();
$.ui.popup({
title: "Success",
message: "All lines written to file.",
doneText: "OK",
cancelOnly: false
});
}
};
line = message + count + "\\n";
fileWriter.write(line);public void WriteLineToFile(String line)
{
if (!File.Exists(filePath))
{
this.CreateFile();
}
if (streamWriter == null)
{
this.OpenFile();
}
streamWriter.WriteLine(line);
}| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 0.504 |
| Cordova | 3.045 |
| Classic Xamarin | 0.658 |
| Xamarin.Forms | 0.715 |
| iOS | |
| Objective-C | 0.835 |
| Cordova | 4.721 |
| Classic Xamarin | 1.217 |
| Xamarin.Forms | 1.17 |
When it comes to writing lines to a text file the vendor native technologies are the undisputed leaders in high performance. Xamain on iOS is close and a little slower on Android. Cordova comes in at 3-4 times slower than the other technologies. Some of this may come from the interpreted looping logic and not the plugin but the reality is that the overhead of the interpreted code that calls into the plug happens if you write one line or a thousand.
Test 6: Reading Lines from a File
Compliment to test 5, reading those 1,000 lines from a file and displaying them on a list.
Java:
public ArrayList<String> readFileContents() throws Exception {
BufferedReader reader = new BufferedReader(new FileReader(textFile));
String line;
ArrayList<String> returnValue = new ArrayList<String>();
while((line = reader.readLine()) != null){
returnValue.add(line);
}
reader.close();
return returnValue;
}
Objective-C:
- (NSArray*)readFileContents:(NSError**)error {
*error = nil;
if (fileHandle == nil) {
[self openFile:error];
}
if (*error == nil) {
[fileHandle seekToFileOffset: 0];
NSData* fileContents = [fileHandle readDataToEndOfFile];
NSString *fileString = [[NSString alloc] initWithData:fileContents encoding:NSUTF8StringEncoding];
return [fileString componentsSeparatedByString:@"\r\n"];
} else {
return nil;
}
}file.file(function(innerFile) {
var reader = new FileReader();
reader.onerror = function(e) {
alert("Error");
}
reader.onloadend = function (e) {
var lines = e.target.result.split("\\n");
var lstFileLines = $("#lstFileLines");
lstFileLines.empty();
if (lines.length > 1) {
for (var i = 0; i < lines.length - 2; i++) {
var listItem = "<li>" + lines[i] + "</li>";
lstFileLines.append(listItem);
}
}
};
reader.readAsText(innerFile);
}, function(ex) {
$.ui.popup({
title: "Error",
message: "Error occurred opening file: " + ex.description,
doneText: "OK",
cancelOnly: false
});Xamarin (All Versions):
public IList<string> ReadFileContents()
{
if (!File.Exists(filePath))
{
this.CreateFile();
}
var returnValue = new List<String>();
using (var streamReader = new StreamReader(filePath))
{
string line;
while ((line = streamReader.ReadLine()) != null)
{
returnValue.Add(line);
}
}
return returnValue;
}| Development Platform | Test Avg. |
|---|---|
| Android | |
| Java | 0.438 |
| Cordova | 1.126 |
| Classic Xamarin | 0.596 |
| Xamarin.Forms | 0.847 |
| iOS | |
| Objective-C | 0.727 |
| Cordova | 1.16 |
| Classic Xamarin | 0.706 |
| Xamarin.Forms | 0.776 |
You want fast, go for native. Xamarin is as good or nearly as good except for Form on Android. This may have more to do with the performance of the list display than the actual reading of the list. That should be comparable to classic Xamarin. Same comments on Cordova as before, it's slower. If this is a factor or not depends a lot about the type of application.
I hope some of you find these comparisons helpful. The lessons I learned from this set of tests were:
- It is self-evident that not all implementations of code even on the same development platform perform the same. Having said that, be careful of wrappers around libraries that have unified APIs to allow for cross platform code. It may not perform as well as the specialized libraries for Android or iOS.
- While it is hard to differentiate slowness in code that comes from it being interpreted and what comes from the libraries they are using, to some extent it doesn't matter. In production apps the Cordova JavaScript code will be interpreted as it calls into plugins like SqLite so both will be a factor.
- The linking settings on Xamarin can have a huge difference in the size of the app. If you are using external libraries where you are not likely to use all the functionality use the Link All Assemblies option if possible.
Thanks everyone.
Source code for the tests can be found here: Performance Tests Source
