Paul, you partly right - if you haven't got a big bunch of variables remaining in the memory it won't affect speed significantly. But in times of automation many people aren't aware of this issue but wonder why excel gets slow during working and that's one point (of many) to look at.
Memory assigned is restricted to 512 MB no matter of the size of RAM available. In filling up the "pot" it slows down speed.