Technically of course it has nothing to do with it being a TOC (therefore it has nothing to do with the customising window). The TOC uses a style. Styles do not inherently have DIFFERENT formatting within the paragraph. You have to apply a character style to the page numbers, or manually change them.

You will have to select the page numbers and change them, either manually or apply a character style.