On the Review tab of the ribbon, click on Reviewing pane. If you are getting the vertical one on open, click on Vertical Pane, which should make it disappear. Save your document, close and reopen.

Did this solve your problem?