My customers put 8k users online on a VPS with 2GB RAM, 2 vCPU.
Instead of Super Cache try wp-redis.
And check your theme and plugins; there are some of them which dispatch one (or more) calls to /wp-admin/ajax.php for statics or anything else. All themes form My Theme Shop show this awful behavior. This is evil.