Found work around to the problem.
If I change the button to an image box, it will show all the colours correctly.
Just have to use the image as a pseudo button.