You don't even need literal page numbers for the TOC to work.

You are entering the Header or Footer section and you can't delete page numbers?

If you go to the ribbon under --

Insert >> Header & Footer >> Page Number >> Remove Page Numbers

Other than that, can you attach a small sample where you can't delete.

I can't replicate your problem