I've been playing around with it a bit, and I know the problem

If you close the document en open it again, it doesn't work anymore.
On any new document where it is set as a caption it does.

What does it change to the caption?