Wednesday, February 4, 2015

Mobile Development Platform Performance Part 2 (Native, Cordova, Classic Xamarin, Xamarin.Forms)

This is the second in the series comparing sample test applications using vendor native development technologies (Objective-C, Java), Cordova (using Microsoft's Multi-Device Hybrid Apps), classic Xamarin and Xamarin Forms.  This time I decided to focus on IO using the different frameworks.  For this I used SqLite to save and retrieve records and also just writing and reading from plain text files.

For some background on testing methodology using Android and iOS.

The Development Platforms
- Native (Objective-C 64 bit and Java)
- Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
- Classic Xamarin (64 bit unified beta for iOS)
- Xamarin.Forms (64 bit unified beta for iOS with Xamarin Forms version 1.3.1)

The Devices
- iPad Mini (non Retina) running iOS 8.1.1 (12B435)
- ASUS K00F running Android 4.2.2

The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good".  But they looked about the same.

The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch.  Each timing was taken ten times and the results were averaged.  It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error.  In the previous post I showed the 10 individual timings and the average.  This time to save me some typing I'm just showing the average of the ten timings.

Test 1: Test App Size
As mentioned in the last set of test, the size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times.  For Android the size of the APKs was examined.  For iOS I looked ipa files for ad-hoc deployment.


Development PlatformSize
Android
Java921kb
Cordova417kb
Classic Xamarin2.1mb
Xamarin.Forms3.4mb
iOS
Objective-C (64 bit)55kb
Cordova621kb
Classic Xamarin2.95mb
Xamarin.Forms8.17mb

A few interesting things to look at here.  First is the Cordova apk is actually smaller than the vendor native tools on Android.  I suspect this has to do with differences in size of the SqLite libraries for both of the platforms.  Also while the Xamarin APKs are larger, they are not as large as they were on the first set of tests.  For this I used the linking option of Link All Assemblies.  This was probably important for Xamarin Forms which is, of course, just a library on top on Classic Xamarin.  For whatever reason I was not able to get the Link All Assemblies option to stay with Xamarin iOS and you see the results with Link SDK assemblies only instead.

Test 2: Load Times
Like the last test I verified how long the applications took to load into memory.  The results were similar to the set of tests I did previously.

Development PlatformTest Avg.
Android
Java1.044
Cordova4.068
Classic Xamarin1.689
Xamarin.Forms3.032
iOS
Objective-C1.14
Cordova2.138
Classic Xamarin1.204
Xamarin.Forms1.928

As last time vendor native technologies load the fastest.  Xamarin Classic follows closely with Xamarin Forms somewhat after that.  In all cases, Cordova is the slowest loading.  It is interesting that on iOS Xamarin Forms loaded almost as slowly as the Cordova app.  Of course for iOS, nothing really took that long to load.

Test 3: Adding 1,000 records to SqLite
I originally wanted to add 10,000 records but found that too slow on my Android device and also it didn't scroll well under Cordova where my implementation had no auto paging.  Offline storage is a common need with mobile applications and SqLite is one of the few solutions that span all platforms.

* I have had a few comments that there are better performing ways to do this test.  For example I could have inserted 1,000 records in the same transaction or if I wanted to do it one one record in one transaction at a time, I could have done it asynchronously.  Both of these points are undoubtedly true.  Through SqLite you can get 1,000 records into the local database in much quicker ways on all platforms than this tests indicates.  What this test is doing is looking at synchronously saving a single record in a single implicit transaction and doing that 1,000 times for all platforms.  It would probably not be too difficult to do the other tests as suggested and they may be the source of a future post.  Thank you to everyone who commented.

Java:
public void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ContentValues values = new ContentValues();
    values.put("firstName", firstName);
    values.put("lastName", lastName + index);
    values.put("misc", misc);
    dbConn.insertOrThrow(TABLE_NAME, null, values);
}

Objective-C:
- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
    NSString *sqlStatement = NULL;
    char *errInfo;
    *error = nil;

    if (dbConn == nil) {
        [self openConnection:error];
        return;
    }
    
    sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
    
    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    if (result != SQLITE_OK) {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
    }
}

JavaScript:
function addRecords() {
    $.ui.blockUI();
    var maxValue = 999;
    var successCount = 0;
    var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });

    for (var i = 0; i <= maxValue; i++) {
        var lastName = "person" + i.toString();
        db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
            successCount++;
            if (successCount === maxValue) {
                $.ui.popup({
                    title: "Success",
                    message: "All records written to database",
                    doneText: "OK",
                    cancelOnly: false
                });
                $.ui.unblockUI();
            }
        }, function (e) {
            $.ui.popup({
                title: "Error",
                message: "An error has occurred adding records: " + e.toString(),
                doneText: "OK",
                cancelOnly: false
            });
            $.ui.unblockUI();
            return;
        });
    }
}

Xamarin (All Versions):
public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

Xamarin Android Alternate:
public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

Development PlatformTest Avg.
Android
Java18.569
Cordova24.126
Classic Xamarin32.55
Xamarin.Forms30.873
Xamarin Classic Alternate18.341
iOS
Objective-C8.044
Cordova14.944
Classic Xamarin8.151
Xamarin.Forms8.137
*results in seconds

Right off the bat you might notice that Xamarin on Android did poorly.  Very poorly.  Slower than native Android or Cordova by a large margin.  One thing I noticed is that in native Android I used the SQLiteOpenHelper but this is only available in the Android API.  I used the method I did because it was cross platform and you can see the results on Android.  SQLiteOpenHelper is available in the SqLite Xamarin Library for Android and on a hunch I tried it instead and you can see those timings under Xamarin Classic Alternate for Android and the timing were virtually the same as they were for Java.  I suspect if I used this for Xamarin.Forms I would have gotten about the same advantages.  The lesson from this is just because there is a cross platform version of the API, it doesn't mean it will perform well.  Buyer beware.

Test 4: Querying SqLite Records
I wanted to test reading the 1,000 records I just wrote.  I only did the 1,000 I originally wrote because I didn't want to deal with paging solutions for Cordova.  A 10,000 record test performed very poorly when just creating a 10,000 for HTML table.

Java:
public ArrayList<String> getAllRecords() throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ArrayList<String> returnValue = new ArrayList<String>();
    String sqlStatement = "SELECT * FROM " + TABLE_NAME;
    Cursor cursor = dbConn.rawQuery(sqlStatement, null);

    if (cursor.moveToFirst()) {
        do {
            returnValue.add(cursor.getString(1) + " " + cursor.getString(2));
        } while (cursor.moveToNext());
    }

    return returnValue;
}

Objective-C:
- (NSMutableArray*)getAllRecords:(NSError**) error {
    NSString *sqlStatement = NULL;
    NSMutableArray *results;
    sqlite3_stmt *sqlResult;
    char *errInfo;
    *error = nil;
    
    if (dbConn == nil) {
        [self openConnection:error];
        return nil;
    }
    
    sqlStatement = @"SELECT * FROM testTable";
    results = [[NSMutableArray alloc] init];

    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    result = sqlite3_prepare_v2(dbConn, [sqlStatement UTF8String], -1, &sqlResult, nil);
    
    if (result == SQLITE_OK) {
        while (sqlite3_step(sqlResult)==SQLITE_ROW) {
            NSString* firstName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 1)];
            NSString* lastName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 2)];
            NSString* fullName = [NSString stringWithFormat:@"%@ %@", firstName, lastName];
        
            [results addObject:fullName];
        }
        sqlite3_finalize(sqlResult);
        return results;
    } else {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:@"Error loading records" forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
        return nil;
    }
}

JavaScript:
function showSqlRecords(sql) {
    var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });
    db.executeSql(sql, [], function (res) {
        try {
            var lstRecords = $("#lstRecords");
            lstRecords.empty();
            var lastValue = res.rows.length;
            for (var i = 0; i < lastValue; i++) {
                var listItem = "<li>" + res.rows.item(i).firstName + " " + res.rows.item(i).lastName + "</li>";
                lstRecords.append(listItem);
            }
        } catch (err) {
            alert(err.toString());
        }
    });
}

Xamarin (All Versions):
public IList<string> GetAllRecords()
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var results = (from t in dbConn.Table<TestTable>() select t ).ToList(); 

    var returnList = new List<string>();

    foreach (var result in results)
    {
        returnList.Add(string.Format("{0} {1}", result.firstName, result.lastName));    
    }
    return returnList;
}

Xamarin Android Alternate:
public IList<string> GetAllRecords() 
{
    if (dbConn == null) {
        OpenConnection();
    }

    var returnValue = new List<String>();
    var sqlStatement = "SELECT * FROM " + TABLE_NAME;
    var results = dbConn.RawQuery(sqlStatement, null);

    if (results.MoveToFirst()) {
        do 
        {
            returnValue.Add(results.GetString(1) + " " + results.GetString(2));
        } while (results.MoveToNext());
    }
    return returnValue;
}


Development PlatformTest Avg.
Android
Java0.551
Cordova1.117
Classic Xamarin1.144
Xamarin.Forms0.89
Xamarin Classic Alternate0.601
iOS
Objective-C0.792
Cordova1.1745
Classic Xamarin0.799
Xamarin.Forms0.735
*results in seconds

Reading records in Android was similarly bad for Xamarin on Android until I switched to the SQLiteOpenHelper implementation and then performance came online with the native example.  As usual the interpreted JavaScript in Cordova slower but probably won't be a factor unless there is a large operation reading thousands of lines.

Test 5: Writing Lines to a File
I also wanted to test writing lines to a text file and how that performs.  For each platform I write 1,000 lines to a file.

Java:
public void writeLineToFile(String line) throws Exception {
    if (!textFile.exists()) {
        this.createFile();
    }
    if (fileHandle == null) {
        this.openFile();
    }

    fileHandle.write(line);
}

Objective-C:
- (void)writeLineToFile:(NSError**)error withTextToWrite:(NSData*)textToWrite {
    *error = nil;

    if (fileHandle == nil) {
        [self openFile:error];
    }
    
    if (*error == nil) {
        [fileHandle seekToEndOfFile];
        [fileHandle writeData:textToWrite];
    }
}

JavaScript:
file.createWriter(function (fileWriter) {
    fileWriter.seek(fileWriter.length);
    var count = 0;
    var line;
    var message = "Writing line to file at index: ";
    var maxLines = 999;
    fileWriter.onwriteend = function(evt) {
        count += 1;
        if (count <= maxLines) {
            line = message + count + "\\n";
            fileWriter.write(line);
        } else {
            $.ui.unblockUI();
            $.ui.popup({
                title: "Success",
                message: "All lines written to file.",
                doneText: "OK",
                cancelOnly: false
        });
    }
};
line = message + count + "\\n";
fileWriter.write(line);

Xamarin (All Versions):
public void WriteLineToFile(String line) 
{
    if (!File.Exists(filePath)) 
    {
        this.CreateFile();
    }
    if (streamWriter == null) 
    {
        this.OpenFile();
    }

    streamWriter.WriteLine(line);
}

Development PlatformTest Avg.
Android
Java0.504
Cordova3.045
Classic Xamarin0.658
Xamarin.Forms0.715
iOS
Objective-C0.835
Cordova4.721
Classic Xamarin1.217
Xamarin.Forms1.17
*results in seconds

When it comes to writing lines to a text file the vendor native technologies are the undisputed leaders in high performance.  Xamain on iOS is close and a little slower on Android.  Cordova comes in at 3-4 times slower than the other technologies.  Some of this may come from the interpreted looping logic and not the plugin but the reality is that the overhead of the interpreted code that calls into the plug happens if you write one line or a thousand.

Test 6: Reading Lines from a File
Compliment to test 5, reading those 1,000 lines from a file and displaying them on a list.

Java:
public ArrayList<String> readFileContents() throws Exception {

    BufferedReader reader = new BufferedReader(new FileReader(textFile));
    String line;
    ArrayList<String> returnValue = new ArrayList<String>();

    while((line = reader.readLine()) != null){
        returnValue.add(line);
    }
    reader.close();
    return returnValue;
}

Objective-C:
- (NSArray*)readFileContents:(NSError**)error {
    *error = nil;

    if (fileHandle == nil) {
        [self openFile:error];
    }
    
    if (*error == nil) {
        [fileHandle seekToFileOffset: 0];
        NSData* fileContents = [fileHandle readDataToEndOfFile];
        NSString *fileString = [[NSString alloc] initWithData:fileContents encoding:NSUTF8StringEncoding];
        return [fileString componentsSeparatedByString:@"\r\n"];
        
    } else {
        return nil;
    }
}

JavaScript:
file.file(function(innerFile) {
    var reader = new FileReader();

    reader.onerror = function(e) {
        alert("Error");
    }
    reader.onloadend = function (e) {
        var lines = e.target.result.split("\\n");
        var lstFileLines = $("#lstFileLines");
        lstFileLines.empty();
        if (lines.length > 1) {
            for (var i = 0; i < lines.length - 2; i++) {
                var listItem = "<li>" + lines[i] + "</li>";
                lstFileLines.append(listItem);
            }
        }
    };
    reader.readAsText(innerFile);
}, function(ex) {
    $.ui.popup({
        title: "Error",
        message: "Error occurred opening file: " + ex.description,
        doneText: "OK",
        cancelOnly: false
});

Xamarin (All Versions):
public IList<string> ReadFileContents()
{
    if (!File.Exists(filePath))
    {
        this.CreateFile();
    }

    var returnValue = new List<String>();

    using (var streamReader = new StreamReader(filePath))
    {
        string line;
        while ((line = streamReader.ReadLine()) != null)
        {
            returnValue.Add(line);
        }
    }
    return returnValue;
}


Development PlatformTest Avg.
Android
Java0.438
Cordova1.126
Classic Xamarin0.596
Xamarin.Forms0.847
iOS
Objective-C0.727
Cordova1.16
Classic Xamarin0.706
Xamarin.Forms0.776
*results in seconds

You want fast, go for native.  Xamarin is as good or nearly as good except for Form on Android.  This may have more to do with the performance of the list display than the actual reading of the list.  That should be comparable to classic Xamarin.  Same comments on Cordova as before, it's slower.  If this is a factor or not depends a lot about the type of application.

I hope some of you find these comparisons helpful.  The lessons I learned from this set of tests were:
- It is self-evident that not all implementations of code even on the same development platform perform the same.  Having said that, be careful of wrappers around libraries that have unified APIs to allow for cross platform code.  It may not perform as well as the specialized libraries for Android or iOS.
- While it is hard to differentiate slowness in code that comes from it being interpreted and what comes from the libraries they are using, to some extent it doesn't matter.  In production apps the Cordova JavaScript code will be interpreted as it calls into plugins like SqLite so both will be a factor.
- The linking settings on Xamarin can have a huge difference in the size of the app.  If you are using external libraries where you are not likely to use all the functionality use the Link All Assemblies option if possible.

Thanks everyone.

Source code for the tests can be found here: Performance Tests Source

12 comments:

  1. Hi! Thanks for the tests, they can be useful as a reference :)

    Just a note: I didn't try your code, but I think that in the test "Adding 1,000 records to SqLite" you should wrap the whole process in a single database transaction.
    In this way the code should run very, very faster: probably in less than 1 - 2 seconds you can insert all the records (usually these are the performance that I get with this trick).
    If you don't use a unique transaction for the whole process, Sqlite will open "internally" one for each of the insert, and this cause the bad performance.

    Stefano

    ReplyDelete
    Replies
    1. Stefano,
      I could have done that but it would have been a different test. What is important for this is that I do the same thing for each of the platforms to see the difference in how it performs. I.e. the test I did answers the question of how long would it take to insert one record; now lets do that 1,000 times so I have something large enough to time. The test you are describing would be how long does it take each platform to insert 1,000 records all at once. An interesting but different test.

      I'm not really trying to answer the question of what is the best way to code this but instead how do these operations compare across environments if I do the same thing. If I get time I can create the test you describe to see how each platform compares in adding 1,000 records as a unit.

      Delete
  2. I've converted your tests to B4i / B4A. The results are available here:
    http://www.b4x.com/android/forum/threads/performance-comparison.50298/

    ReplyDelete
    Replies
    1. Excellent thanks! My previous post had a similar set of tests as well. Of course all these tests are device specific. YMMV.

      Delete
  3. Using SQLite.Net.Async (with synchronous calls) yields much better performance even without a transaction. Tested on S4 1000 records inserted in 3.5s versus 15s using the Android SQLiteHelper. Using transaction that number of course goes down to milliseconds.

    ReplyDelete
    Replies
    1. Thanks. I have added some commentary to talk about that set of tests. Thanks for your input.

      Delete
  4. The point of true cross platform is in my mind not to exclude Windows.
    Would have been nice to see the statistics for Windows as well.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. In Test 4 for Xamarin
    You should be able to drop the ToList from the end of
    var results = (from t in dbConn.Table() select t ).ToList();
    Your current code is copying the results again.

    ReplyDelete
    Replies
    1. Looks like you correct. I'm just using the results to enumerate so an IEnumerable would have been just fine.

      Delete