Usually this is an issue with a generally applied formatting, i.e., applied to an entire row or column. This applies formatting info to all cells in the row/column, causing Excel to think that all these cells are used (and increasing the workbook size excessively).

What happens if you copy-paste to a new sheet without formatting? Pasting column widths using the right-mouse quick menu also pasts cell formatting. Try using the actual paste-special menu, which has more options.