If the right column of the mesra is meant to be read first, and then the left column of the mesra is meant to be read second, may I suggest that instead of using "TAB" to create columns, you use the "COLUMNS" feature of Word (sometimes referred to as "newspaper columns.")

Begin with a blank page.
Go to the LAYOUT tab and choose "2 columns."
The paragraph should now be formatted to use full justification (ideally, using a style).
Type the first beyt.
Either use the shortcut key (in my version of Word, COMMAND-SHIFT-RETURN) or else LAYOUT, BREAK, NEW COLUMN.
Type the second beyt.

If this solves the problem of justification, you will probably want to increase the space between the columns. This would involve going back to LAYOUT, but choosing "more columns". There will be a dialog box, where the number of columns should be (already) set to 2, and you will need to change the spacing and confirm it should apply to the WHOLE DOCUMENT, and NOT "this point forward". You should NOT need to change the column widths, Microsoft should be able to narrow the columns by itself if you play with the spacing between.

I must confess that I was unable to verify this possible solution. Evidently, my version of Word knows I am a mostly monolingual American, and the "newspaper columns" would not follow the right-to-left rules even when I switched the language and font.

If this does NOT solve correctly, the other idea I had was to set a style for the left beyt paragraph that involves a "frame" which is a less-used feature of Word.