It's hard to say exactly what is is. But, if you right mouse click on your arrow and choose More Layout Options, go to the third tab "size" and there is a rotation digital setting, so you should be able to type in a 0 there.

Also, on the first tab, "position," maybe uncheck "Move object with text" and see if that helps with the arrowing around. It may also be defaulting to a position relative to a paragraph rather than a page. And, do you have text wrapping on the object?