Deep sea dive

/

Its been more that a few years since i updated my blog. In that time, i have moved to Shanghai, become a father and helped grow a company. I finally got some time to get back to personal projects so i revisited my 3 year old project of learning iOS development and actually making an iPhone App.

In the 3 year that i have been AWOL a lot has changed in iOS development, primarily the announcement of Swift and it being open sourced. Having briefly learned Obj-C years ago, i decided to take a stab at developing an app in Apple's new flagship language.

Swift everywhere

One of the key things that attracted me to learning this new language was the fact that it was Open Source and it was officially supported on Linux. Developing iOS apps in Obj-C always felt a little archaic to me, the syntax felt overly verbose and you would often spend an inordinate amount of time writing or autocompleting message calls that spanned multiple lines which made it difficult to read and debug. The Swift syntax is a lot more modern and descriptive, with the ability to achieve the same end result with a lot fewer lines code. Before i vere too far away from my original point - having a language that was supported both for the frontend (iOS) and backend (Linux) was compelling enough for me to take a stab at learning it.

Learning a new Language

Its been awhile since i have attempted to teach myself a new programming language. The last attempt was Obj-C (over 3 years ago). Everyone learns to code differently and i typically use a range of resources to get started. Making sure you have the development environment is usually the first step - a quick download of Xcode took care of that. In terms of learning resources, i started using the following:

  • Swift Programming Language Book - available on Apple's iBook store
  • iOS Programming: The Big Nerd Ranch Guide - i have a subscription to Safari Books online that gives me access to tonnes of great material
  • www.raywenderlich.com - great tutorials and videos

Apple's idea of creating building REPL functionality into Swift and then adding Swift Playgrounds was genius. I have spent an inordinate amount of time using the Perl and Python interpreters to quickly verify a few lines of code before i embed them in a larger program. I routinely embed a Swift Playground in my Xcode project to quickly test various pieces of functionality. There are also numerous great template Playgrounds that you can download to experiment with various frameworks and libraries.

Digging Deep

The purpose of this blog post was to answer the question, how far do you go to fix a technical problem or bug with your code ?. Most programmers are faced with this problem at one time or another and the answer to this can wildly vary based on the situation. Often times the determining factor is time and motivation.

More often than not, constrained by time, programmers pick a workaround for the bug and at some point in the future come back and do a Root Cause Analysis and fix the actual bug. But i have seen a fair share of programmers who are content with leave the workaround as is because doing the RCA is too hard or they are just lazy.

Having worked with Opensource software for the better part of my career, i often find it unacceptable to live with workarounds for bugs, because having source code available for everything that your code is using empowers you to find and fix the actual bug no matter how deep in the software stack the problem lies. The remainder of this post describes an instance where i was posed with a similar problem, do i use a workaround or do i dig deep.

IBM Kitura

As part of learning Swift, i decided to write an iOS App that fetches some data from a REST based service and displays it. I wanted to try and write the backend and frontend in Swift. For the backend service, i rented a simple Linux VM in Alicloud set about trying to find an appropriate framework to build this REST API endpoint. Currently there are a number of competing frameworks available - Perfect, Vapor, Zewo and Kitura . After initially testing Vapor for a few days, i settled in on using IBM developed Kitura, it seemed the most straightforward to use and setup.

Getting the initial working prototype for the backend service involved nothing more than copy and pasting a few example code snippets, running swift build and then running the resultant executable. This got me a small service that ran on a port of my choosing, waiting for REST requests and the ability respond to them. All of this seemed simple and straightforward.

For the client side i decided against using the native iOS API call (URLSession) and decided to use Alamofire, a very popular networking library written from the ground up in Swift. To begin testing out my server i created a Swift playground with Alamofire library included (as of writing this blog, adding external/third-party libraries to Swift playgrounds is rather tedious, hopefully they improve this in the future). The test code in the playground looked like this

import Alamofire

Alamofire.request("https://server.org:9099/").responseString { response in
    if response.result.isSuccess {
        print("Data : \(response.result.value)")
    }
}

I started out my testing using HTTP as the protocol, but then later switched to using HTTPS. Getting a certificate for my service was a breeze with Letsencrypt.org. Once i got my kitura based server to load and trust the certificates i tried testing with HTTPS enabled, and it worked!. 

Next i proceeded to slowly build my app,  using the code that i had prototyped in the Swift playground. After about a week of coding i began to run early tests in the iOS simulator and I immediately ran into a problem. I would repeatedly see SSL connection errors when my app tried to make connections to the service. Initial debugging lead me to believe that i had somehow incorrectly implemented the Alamofire connection code. Copying and pasting the code back into Swift playground revealed no errors.

I knew that the Swift Playground was not really representative of the iOS environment, so started testing with various other endpoints. Initially i tested using web browsers. I tested against Chrome, Safari and Firefox, all of them were able to successfully connect to  my server end point over SSL and return a simple JSON result. Next i used a few REST testing apps, Rested for macOS and Httper iOS, both are available on the App Store. Both of these Apps worked with my test server.

Finally exhausting all testing options, i resorted to searching online for other cases for which people were reporting similar problems. On the client side i would get the following error message, so all my searches typically returned cases of people running into problems with misconfiguration of their certificates or not having the correct version of openssl installed on the server.

2017-04-10 15:15:56.441396+0800 AlamofireTest[3602:379526] [] nw_coretls_read_one_record tls_handshake_process: [-9824]

2017-04-10 15:15:56.447 AlamofireTest[3602:379535] NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9824)

I tried digging around in the Alamofire and BlueSSL (IBM-Kitura SSL connection implementation) github issue page to see if any users were reporting similar problems. No such luck, there were no reported issues on either page.  Alamofire has been around a while so i suspected that the problem lay in BlueSSL. I had no evidence for a bug report or adequate debugging to understand where the problem lay.

Wireshark to the rescue

Finally frustrated with the lack of proper error reporting and no actionable ways to debug this problem, i fired up trusty ol Wireshark to see if i could analyze what was going on at the protocol level.

After capturing packets from both the iOS simulator and tests from Safari, i got my first real lead. I noticed a difference in the types of Cipher suites advertised by the client.

Safari Supported Cipher Suites

iOS Simulator Supported Cipher Suites

The way TLSv1.2 is supposed to work, is that the client advertises a list of supported cipher suites and the servers is supposed to pick one of them that it prefers, if they cannot agree on a supported share cipher suite, the connection is dropped. So in this case it seemed like my Kitura server possibly did not like the list of cipher suites presented to it by iOS

To better understand why wasn't the server accepting any of the cipher suites presented, i began testing with the openssl utility

openssl s_client -connect server:port -debug

I could further test various subsets of cipher suites to see which ones were supported using the following command.

openssl s_client -connect server:port -debug -cipher 'RSA'

or

openssl s_client -connect server:port -debug -cipher 'ECDHE'

These tests helped me confirm that problem was in the compatibility of the cipher suites between the client and the server (iOS and Kitura)

The 2 questions i needed to answer

  1. Why was iOS advertising such a limited subset of ciphers
  2. Why didn't the Kitura server support the ECDHE family of ciphers

App Transport Security

Answering the first question was relatively easy. A few google searches later I met Apple's App Transport Security(ATS). Starting with iOS 9.0 and OS X v10.11 ATS is enabled by default. This forces all HTTP connections to be defaulted to HTTPS, any connections over HTTP will fail. ATS uses TLS v1.2 and more specifically enforces Perfect Forward Secrecy (PFS). Within PFS, Apple forces the use of Elliptical Curve Diffie-Hellman Ephemeral (ECDHE) key exchange with AES-128 or AES-256 symmetric cipher.

The details can gotten from the following page from Apple 

Below is a exact list of cipher suites that ATS supports. This explains why the iOS simulator was advertising a smaller subset of specific cipher

  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

BlueSSL

Now for the second question. Why doesn't the IBM Kitura frameworks support ATS ?. Initially I passed this off as me not configuring the framework correctly. The Kitura framework is relatively new and is under heavy development by IBM. The SSL implementation for Kitura is provided by the BlueSSL library. Since all of this is opensource and hosted on github, I started by creating an issue on the github repo asking if the developer had more insight.

BlueSSL is just a wrapper around the OS provided SSL libraries (Linux: OpenSSL, MacOS: CommonCrypto) so the code has to deal with different API based on the platform and somehow provide a common set of features across them. The lead maintainer of the project initially suggested that the Linux OpenSSL implementation did not have those cipher suites turned on. That was quickly checked by using the openssl ciphers command.  Not getting further help from the maintainer I decided to dig into the library myself to try and fix it.

After a few days of pouring over the OpenSSL SDK and finding a few blogs, I found that adding ECDHE support to BlueSSL was trivial and only required a single API call in OpenSSL

SSL_CTX_set_ecdh_auto (ctx, 1)

But this call was only supported in OpenSSL v1.0.2 and above, so after upgrading my backend server to a newer version of Ubuntu-Linux , patching BlueSSL with a single line addition, recompiling and rebuilding my server and finally success!!.

Workarounds

I did not really need to fix the BlueSSL service, since just as Apple enabled ATS by default, you could just as easily have it switched off by editing your Info.plist file in your application. Details are available here (ATS Configuration Basics section). Similarly I could have chosen to switch to one of the other backend frameworks(mentioned earlier in the post) that supported ATS.

Wrap Up

I must have spent around 2 weeks working through the bug I discovered. I contributed that fix back to the maintainer and BlueSSL/IBM-Kitura now has ATS support. All of this did not further my goal of developing my iOS App, but it provided me a deep dive into a whole number of areas that I ordinarily would not have gotten.

So to answer the question that I posed earlier in the blog post, how far do you go to fix a technical problem, I guess my answer is as deep as it takes (barring any time/resource constraints). The lessons you learn along the way and the experience gained in investigating and finally fixing deep technical problems is invaluable and is absolutely needed any good developer. Therefore I encourage anyone reading this post to challenge themselves the next time they are posed with a similar problem and - dig deep.