Homeworld: Cataclysm actually made a space fleet RTS scary, superbly well done game. Having a limited fleet, dark atmosphere, and the threat of your own ships quickly turning against you at a moment, combined with the aesthetics of the soundtrack and environment. Plus, quite a few 'out of the dark' attacks (mission 7 especially).