It was on a dedicated box with 32gb ram, apache, mysql, php5.6 w/ opcache... and started getting a bit nasty around ~400 concurrent users on the site (wordpress).
I decided to move it over to a 4gb ram DigitalOcean with easyengine (nginx, phpfpm)... the site loads nicely now, resource usage looks fine:
So far.. it seems like video is loading ok on the box, but it could be better. I will look at doing an RTMP server to serve just the video though.