Your TOC styles should not be based on heading styles. At least not if you mean the styles used in the TOC itself. They should not have outline levels and any styles based on the heading styles will have outline levels.
If I have time before someone else answers your problem, I will try to reproduce it and see what can be done. In the meantime, here are some articles that may help you.