Mobile Development Platform Performance (Native, Cordova, Classic Xamarin, Xamarin.Forms)

Last month I published my Mobile Technology Decision Making White Paper and I feel that goes a long way to helping choose what technology to use in the mobile space.  One question I get asked frequently is how do the different mobile development platforms compare to each other from a performance perspective. While I've heard a lot of anecdotal information I have not seen a lot of formal comparisons.  At Magenic we normally work with native (iOS, Android, WP), Cordova and Xamarin so I wanted to compare these.  I plan on this being the first post on this topic.

For some background on testing methodology using Android and iOS.

The Development Platforms
- Native (Objective-C 64 bit and Java)
- Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
- Classic Xamarin (64 bit unified beta for iOS)
- Xamarin.Forms (64 bit unified beta for iOS with beta version of Xamarin.Forms, note latest version of the unified API in the beta/alpha channels could not be used as it is not supported by Xamarin.Forms Note: 1.3.1 pre 1 was released Dec 24th so Xamarin.Forms may now work with the version of the unified iOS API in the alpha and beta channel)

The Devices
- iPad Mini (non Retina) running iOS 8.1.1 (12B435)
- ASUS K00F running Android 4.2.2

The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good".  But they looked about the same.  There were some differences such as Java, Classic Xamarin and Xamarin.Forms rendered the tabs on the top in Android as expected while the JavaScript library showed them on the bottom.

The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch.  Each timing was taken ten times and the results were averaged.  It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error.

Test 1: Test App Size
The size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times.  For Android the size of the APKs was examined.  For iOS I looked at Settings to find out how much space the apps took up on disk.

Development Platform	Size
Android
Java	166kb
Cordova	433kb
Classic Xamarin	3.5mb
Xamarin.Forms	4.7mb
iOS
Objective-C (64 bit)	644kb
Cordova	2.7mb
Classic Xamarin	12.1mb
Xamarin.Forms	16.9mb

When it comes to application size Xamarin shows the extra size involved in the overhead of the .Net framework.  There was an attempt to reduce the size of the deployed Xamarin application by using the "Link SDK assemblies only" setting.  I am surprised in a very small application how large the difference is.  However, from experience in "real" applications the difference is much less consequential as graphics and frameworks get added to the projects.

Test 2: Load Times
I wanted to see how long it took the application to load into memory.  While the initial load time is important, many mobile applications tend to stay in memory so it tends to have a limited impact.  For this test I made sure to close all applications before each timing.

Development Platform	Test Avg.
Android
Java	1.085
Cordova	3.978
Classic Xamarin	1.704
Xamarin.Forms	2.764
iOS
Objective-C	1.221
Cordova	1.715
Classic Xamarin	1.28
Xamarin.Forms	1.813

In all cases the vendor native technologies loaded the fastest.  Classic Xamarin loaded nearly as fast as the native languages.  Xamarin.Forms and Cordova had the slowest load times.  The Cordova load time on Android was particularly bad while on iOS the load times were close enough to not be a huge factor.

Test 3: Loading a List from Azure Mobile Services
In this test I wanted to look at getting data from an external service so I loaded 1000 records from Azure Mobile Services.  For Xamarin iOS 64 bit I had to modify the Azure Mobile Services to be compatible with the unified API.  The timings were taken from pushing the button to load the list until the results visibly came back and were displayed on a list on the screen.

Java:

public  void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ContentValues values = new ContentValues();
    values.put("firstName", firstName);
    values.put("lastName", lastName + index);
    values.put("misc", misc);
    dbConn.insertOrThrow(TABLE_NAME, null, values);
}

Objective-C:

- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
    NSString *sqlStatement = NULL;
    char *errInfo;
    *error = nil;

    if (dbConn == nil) {
        [self openConnection:error];
        return;
    }
    
    sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
    
    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    if (result != SQLITE_OK) {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
    }
}

JavaScript:

db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
    successCount++;
    if (successCount === maxValue) {
        $.ui.popup({
            title: "Success",
            message: "All records written to database",
            doneText: "OK",
            cancelOnly: false
        });
        $.ui.unblockUI();
    }
}, function (e) {
    $.ui.popup({
        title: "Error",
        message: "An error has occurred adding records: " + e.toString(),
        doneText: "OK",
        cancelOnly: false
    });
    $.ui.unblockUI();
    return;
});

Xamarin (All Versions):

public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

Xamarin Classic Android Alternate:

public  void AddRecord(string firstName, string lastName, int index, string misc) 
{
    if (dbConn == null) 
    {
        OpenConnection();
    }

    ContentValues values = new ContentValues();
    values.Put("firstName", firstName);
    values.Put("lastName", lastName + index);
    values.Put("misc", misc);
    dbConn.InsertOrThrow(TABLE_NAME, null, values);
}

Development Platform	Test 1	Test 2	Test 3	Test 4	Test 5	Test 6	Test 7	Test 8	Test 9	Test 10	Test Avg.
Android
Java	22.71	17.5	18.04	17.7	18.63	20.33	2.68	2.42	2.16	2.34	2.369
Cordova	25.99	24.76	27.05	23.3	24.06	22.86	2.12	2.02	1.94	2.48	2.149
Classic Xamarin	34.07	27.38	32.05	38.77	29.27	34.63	1.61	1.63	1.84	1.85	1.738
Xamarin.Forms	1.99	1.76	2.32	1.91	1.9	1.58	1.93	2.02	2.03	1.64	1.908
iOS
Objective-C	2.38	2.44	2.24	2.3	2.34	2.32	2.32	2.35	2.2	2.27	2.316
Cordova	3.57	2.18	2.07	1.95	1.97	2.05	2.04	1.93	2.2	1.96	2.192
Classic Xamarin	2	1.87	1.88	2.06	1.74	1.9	1.81	1.94	1.75	1.96	1.891
Xamarin.Forms	2.11	2.01	2.23	1.96	1.95	2.07	2.12	2.16	2.08	2.1	2.079

*results in seconds

In many ways this test is showing how well the Azure Mobile Services libraries perform on the different platforms.  Unsurprisingly Xamarin, with it's underpinnings of a .Net implementation, performs the best in this test.  I was surprised to see the libraries for the native technologies perform the worst, both on Android and iOS.

I have heard that the Xamarin.Forms lists can perform poorly with large data sets.  These results did not show that, at least with lists of up to 1000 records.

Test 4: Prime Number Calculation
In the final of my first series of tests I wanted to try out a CPU intensive operation.  I created a Sieve of Eratosthenes on each of the platforms.  My first plan was to calculate all prime numbers up to 50,000,000.  This required some special handling of the method's array for both Objective-C and JavaScript.  In the case of Objective-C I had to malloc memory to support arrays that large.  Also for Objective-C and JavaScript I had to initialize the array items to 0.  To keep the timings the same I did the array item initialization to 0 on all platforms even though it could have been left out for .Net (and Java I believe).  It that was done, the .Net timings would have been even better.

I did end up having to settle for only calculating primes up to 5,000,000.  The reason for this is that the JavaScript performed so poorly that I was unwilling to wait for it to complete 10 times.

Java:

private int getPrimesFromSieve(int maxValue)
{
    byte[] primes = new byte[maxValue + 1];
    for (int i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    int largestPrimeFound = 1;

    for (int i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        int c = 2;
        int mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

Objective-C:

- (int) getPrimesFromSieve: (int) maxValue {
    Byte *primes;
    primes = (Byte *) malloc(maxValue * sizeof(Byte));
    for (int i=1; i<=maxValue; i++)
    {
        primes[i-1] = 0;
    }    

    int largestPrimeFound;
    largestPrimeFound = 1;

    for (int i=2; i<=maxValue; i++)
    {
        if(primes[i-1] == 0)
        {
            primes[i-1] = 1;
            largestPrimeFound = i;
        }

        int c=2;
        int mul = i*c;
        for(; mul <= maxValue;)
        {
            primes[mul-1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

JavaScript:

function getPrimesFromSieve(maxValue) {
    var primes = new Uint8Array(new ArrayBuffer(Number(maxValue)));
    for (var i = 0; i <=maxValue; i++) {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (i = 2; i <= maxValue; i++) {
        if (primes[i - 1] == 0) {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i * c;
        for (; mul <= maxValue;) {
            primes[mul - 1] = 1;
            c++;
            mul = i * c;
        }
    }
    return largestPrimeFound;
}

Xamarin (All Versions):

public static int GetPrimesFromSieve(int maxValue)
{
    var primes = new byte[maxValue + 1];
    for (var i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (var i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

Development Platform	Test 1	Test 2	Test 3	Test 4	Test 5	Test 6	Test 7	Test 8	Test 9	Test 10	Test Avg.
Android
Java	4.31	4.31	4.2	4.33	4.39	4.37	4.32	4.45	4.34	4.4	4.342
Cordova	91.69	95	94.31	94.4	94.73	94.1	94.1	91.8	93.63	97.75	94.151
Classic Xamarin	4.27	4.25	4.15	4.32	4.51	4.41	4.22	4.12	4.14	4.19	4.258
Xamarin.Forms	4.21	4.17	4.31	4.3	4.2	4.34	4.29	4.36	4.22	4.19	4.259
iOS
Objective-C	5.04	5.49	5.38	4.86	4.8	5.02	5.03	4.83	4.84	4.85	5.014
Cordova	66.96	67.36	67.22	67.3	67.17	67.44	67.13	67.11	67.58	67.64	67.291
Classic Xamarin	4.41	4.42	4.35	4.34	4.49	4.37	4.17	4.27	4.39	4.28	4.349
Xamarin.Forms	4.51	4.33	4.31	4.31	4.33	4.4	4.41	4.4	4.33	4.46	4.379

*results in seconds

While I was expecting that JavaScript would be slower, I was unprepared for how much worse it was for this type of operation.  This would make Cordova problematic for highly CPU bound work were performance is important (I wonder what that means for server side Node.js...).  CPU bound work is not as important in many of today's mobile applications but given the history of increased performance for mobile CPUs and what happened in the PC market in the 1990's, it is likely that in the future CPU bound work will be more prevalent in mobile applications.

Having said that, using native HTML commands can be very fast.  In the past I've testing loading JSON using HTML commands vs. the JSON.Net library on Xamarin and found them nearly comparable.

I was also surprised that Xamarin performed better than Objective-C by a noticeable amount.  It also performed better than Java on Android but by a very marginal amount, well within the .2 second margin of error that manual timings give us.

That's it for my first installment of performance tests.  Much of this code was made more difficult for Xamarin due to the flux around the 64 bit Unified iOS API.  For next month I'll take a look at loading large JSON strings and perhaps something else.

I hope this is useful or you.  If you have any ideas for performance tests I can perform, I'd love to hear them.

Source code for the tests can be found here: Performance Tests Source

Windows Azure SQL Databases, Cost Effective or Not?

Last week I gave an Windows Azure presentation at Modern Apps Live in Orlando.  The session itself wasn't well attended and that matches many of my past experiences with Azure presentations.  I spoke to several people afterwards and it seems that much of the disinterest around Azure has to do with its perceived high cost when comparing it to alternatives.

To some extent this is Microsoft's fault, when they came out with the service initially it wasn't all that well priced.  But for the most part that has changed and Azure is a much better deal than people realize, particularly when looking at factors around redundancy.  For example, there are few ways to quickly and cheaply get a redundant T-SQL database up than by using a Windows Azure SQL Database (WASD).

Companies usually have about three options here.  They can use a private cloud (either existing or build a new one), they can use some sort of IAAS provider like Rackspace (or use Azure IAAS), or they can use a PAAS like WASD.

The private cloud cost comparison is harder to quantify as there are so many situational specific factors that need to be understood in the calculation.  It is usually true that for companies who have existing private clouds it is cheaper to leverage a sunk investment than to move to an IAAS or PAAS SQL solution in the short term.  I want to reiterate; in the short term.  That is because in the short term much of the cost involved in creating that private cloud is sunk.  In the long term (five year horizon) that calculation may change as hardware needs to be refreshed and the amount of services  you plan on deploying to your private cloud will influence the amount of hardware you will need to keep up to date to continue your private cloud investment.  This is an area where momentum tends to come into play that is to say, the path of least resistance is for IT managers to continue to refresh their private cloud infrastructure than to come up with a plan for moving a portion of their services into a hybrid cloud.

In general, if you have a private cloud and want to do a long term analysis on the benefits of moving SQL Services to a hybrid or public cloud these are some of the factors you want to keep in mind.

• If I move some of my services to a hybrid/public cloud what will my data transfer services cost me in the cloud vs. how much do they cost me now?  Corollary to this is the question of should I move applications that use the data into the cloud as well to limit that data transfer cost?  Remember, to do an apples to apples comparison here, moving around data on your current infrastructure has an associated cost as well (hardware to deal with the amount of data flowing through the network, dark fiber between facilities, external lines that have a data charge for lighting buildings or employees in the field, etc.)  The cost of data transfer is most likely the greatest single monetary factor in this equation.
• If you move some portion of your services to the cloud, can you do a lower investment in hardware in your private cloud due to the lower need for services in your own facilities?
• What kind of data do you have and what are the risks between maintaining it in different locations?  The common example is that HIPPA generally is not used in public clouds, though Azure has been working on solutions for that too: http://www.eweek.com/c/a/Health-Care-IT/Microsoft-Adds-HIPAA-Compliance-in-Windows-Azure-for-Cloud-Health-Data-446671/
• Do I need to have a solution with geographically diverse databases and if so, what are my requirements behind how often the data needs to be kept in sync?  WASD can be set up in many different geographic regions and kept in sync, over time.  The sync interval can be set as low as every five minutes but due to the hub and spoke model even that may take as long as 10 minutes for data changes to be distributed from one spoke to another.
• How much does your OS licensing cost you?
• Can the applications that use the database use SQL security instead of Windows integrated security?  WASD only supports the SQL security model.
• How much does your infrastructure cost you and what are the opportunity costs associated with having infrastructure personnel working machines with SQL databases instead of the many other mission critical items they could be focusing on?  Do you need the same amount of infrastructure resources, particularly as the company grows, etc.?
• Do you have SLA requirements?  Currently the WASD offer 99.9% monthly uptime.  Many companies require more. Having said that, in my experience very few companies actually maintain greater than 99.9% when doing an analysis of their internal data center operations.  This is an area where companies may have to analyze what their internal capabilities really are.
• What are your redundancy requirements?  This is one where WASD are hard to beat and give a benefit few people recognize.  If you have an instance of a WASD in a Azure data center you are getting that plus three redundant instances; one primary redundant instance and two secondary.  If your main instance, or one (or more) of the redundant instances go down, Azure will use the others to continue uninterrupted service and bring up more redundant instances in the background.  The net effect is that it is very difficult to take down a WASD instance without an entire data center or the Azure service in general going down (which has happened twice that I can recall).

One thing you may notice is missing from my list; that is DBAs.  You will likely need the same number of DBAs if you host your databases in your private cloud, in a public cloud through IAAS or using a PAAS offering like WASD.  We know comparing the cost of a private cloud is highly situational, but what about looking at IAAS vs. a PAAS like WASD?  That is much clearer, and in most cases WASD will be doing IAAS at a company like Rackspace.  If you've determined that moving a SQL Service instance from your internal private cloud to an IAAS service, in many cases it may be less expensive to take it one step further and go to WASD.

Let's look at this comparison.  For information about assumptions, we have about 60 GB of data of which our average outbound data per month is 2000 GB, increasing 10% yearly.  Also for the Rackspace offering we set up two servers is an active/passive SQL cluster to get some level of redundancy but still less than what we get with WASD.  If that's the case, we end up with something like this:

Windows Azure	Year 1	Year 2	Year 3	Year 4	Year 5	Total
Cost WASD/Year	$1,510.56	$1,570.44	$1,630.44	$1,690.32	$1,750.32	$8,152.08
Cost of outbound data	$2,880.00	$3,168.00	$3,484.80	$3,833.28	$4,216.61	$17,582.69
Total	$4,390.56	$4,738.44	$5,115.24	$5,523.60	$5,966.93	$25,734.77

IAAS (Rackspace)	Year 1	Year 2	Year 3	Year 4	Year 5	Total
Cost of server licensing	$155.40	$0.00	$0.00	$180.00	$0.00	$335.40
Cost of SQL licensing	$678.00	$0.00	$0.00	$711.90	$0.00	$1,389.90
Cost of data center	$1,401.60	$1,401.60	$1,401.60	$1,401.60	$1,401.60	$7,008.00
Cost of IT labor	$480.00	$494.40	$509.23	$524.51	$540.24	$2,548.39
Cost of outbound data	$2,880.00	$3,168.00	$3,484.80	$3,833.28	$4,216.61	$17,582.69
Total	$5,595.00	$5,064.00	$5,395.63	$6,651.29	$6,158.45	$28,864.37

Of course there are many factors that can cause these numbers to change.  For example, we can say that the SQL instance doesn't need to be redundant like it is with WASD.  Also Rackspace's outgoing data charges decrease the more data you use while WA does not (perhaps something MS should change to stay competitive).  The point of the exercise is to realize that prima facie the Windows Azure services are not more expensive than competitive offerings.  The services offer other benefits that I have not mentioned here but they are worth doing a serious analysis on, both from a cost and a feature perspective.

If you do, or do not, think Windows Azure services are cost effective, or even if you don't know I'd be interesting in speaking with you.