When the list comes up, use UP and Down arrows to highlight the email address you want to delete and press Delete.

BTW, IE works the same way.