Thanks Ryan, just to reiterate I am not using HTTP/2 and HTTP/2 should have nothing to do with TTFB. I am talking about the initial TLS handshake delaying TTFB. As I said in my findings SSL TTFB is sensitive to geographical distance (which http is not in my tests). I am not disputing HTTP/2 is a good thing :). I am arguing that the HTTPS TLS handshake does create an overhead for TTFB which is not good for users with an international audience.
I'm guessing your VPS is in the USA somewhere. Compare your TTFB, it is 3x higher outside the USA. We can of course all get good TTFB when the server is geographically close 
New York: http://tools.pingdom.com/fpt/#!/rEK04/https://www.ryansmithphotography.com/ Sweden: http://tools.pingdom.com/fpt/#!/cHwKQc/https://www.ryansmithphotography.com/
Now look at the data here for a site without https where the VPS is located in Amsterdam. The TTFB is less affected (we must expect some latency) Sweden: http://tools.pingdom.com/fpt/#!/eb8dOr/www.htpcguides.com New York: http://tools.pingdom.com/fpt/#!/erZS4/www.htpcguides.com
Of course I need to run more tests but this is the overall trend I have found with SSL TTFB overhead along with other users.
