I finally got a working solution. As I use dots as a leader between the toc entry and the page number it is possible to pad a very short entry with periods which are coloured white.

These then show up as slightly different dots in the TOC but ensure that the page number lines up OK.