I am talking about the initial TTFB not the overall load time.
My understanding is that http/2 helps with pushing multiple files over a single connection after the TTFB is complete along with the initial handshake, HTTP/2 does not affect TTFB as far as I can tell.
It may have to do with the strong ciphers I am using (official recommended from https://cipherli.st/ to get A+ security rating) or that I am using haproxy with CloudFlare full SSL so encryption is taking place twice (once between the user and CF then between CF and the VPS).
When I use just flexible ssl from CloudFlare the TTFB is much lower presumably because my VPS doesn't have to encrypt anything.
I will try with nginx doing the SSL with the same strong ciphers to see if that reduces TTFB.
@rsmith4321 and @janiosarmento would you mind sharing your URLs so I can examine TTFB, more data would help 