Well, people can think they are larger, and in most cases they would be. But, the last time I checked, the size of a nation's armed forces was inversely proportional to the efficiency that it could combat guerilla warfare.
For example, the Hessians may be a great deal larger than the Phantoms, but they are fighting both the Rheinland Military (when they arent assisting them against the Liberty Navy) and the Corsairs. Why would they want to spend resources fending off attacks that come out of nowhere and dissolve without a trace? The same can be applied to almost any unlawful faction, more so the larger they get. In my mind, the Phantoms have always been small enough to be 'ignored' if you can call neutrality that, but organised enough to initiate some very damaging attacks.
Im sure there would be severe inRP reprisals on parties who didn't see the reasoning of such an argument. Nobody wants to be the target of such reprisals, so why bother making them an enemy that you don't need? What could they gain?