It does seem like a grey though. Removing base defenders, even though you aren't explicitly sieging the base, is still technically attacking the base in some capacity. At the least, it's undermining its defences, which are meant to be warding off the siegers.