If done flawlessly, I think it could be a good idea. Be a very, very tough faction to get a decent membership for. I mean, I personally have a Xeno character, but there are very few of us (I think). You have already named the reasons why.
Anyone starting it though, would need to understand that it would be under scrutiny from the very begining, and any RP or rule slip-up would probably be dealt with post haste. Just know, that a Xeno faction has more limitations and stringent rules than ANY of the other NPC factions.
<span style="color:#FF0000"><strike>RETIRED</strike></span>
ON MY WAY OUT OF RETIREMENT