This was annoying me when I was just working on a Word Template. I just figured it out and am posting it since there were no solutions in any of the results I looked at!

When you create a new table style and then go back to try to modify it, "Modify Table Style" will be greyed out until you apply that style to the table your cursor is in.

Again, the style must be applied to the table, or you need to move your cursor into the table that has your custom table style applied to it, THEN Modify Table Style will become active.

Hope that makes sense and helps someone out there!