"If you right click on the calendar, you can select "delete calendar" and it will just remove the calendar from your Outlook. It will not delete the calendar of the other user"

This doesn't work on all the calendars in the lists. Some we can delete but on most others that option is greyed out. I thought this was because my User either wasn't the owner of those calendars or only had view or contribute permissions in the first place. Given that this is just a list like the Recent Docs list on the Start Menu there really should be an easy way to clear it out.