If the entire document is formatted into two columns, this should not happen. The new page is formed by a next-page section break. I expect if you format that page as having the same number of columns, you will be able to merge the two pages.

Delete a Next-Page Section Break that Word won't delete