There are plenty of traitors among us. Secession allows us to separate from in-group opponents and out-group opponents as well.
Think of it this way: we can spend time, effort, and resources, to regulate, and another thousand years trying to breed out their behavior – or we can simply leave them behind and let them outbreed and be lost among the out-groups.
When you make our struggle against colors alone you miss the opportunity that men of all other colors have the same problem we do: eliminating parasites.