This macro works for me. You can assign a keyboard shortcut to run it by setting that shortcut as Alt+)
Code:
Sub CloseBracketsAndFormat()
Dim aRng As Range
Selection.TypeText ")"
Set aRng = Selection.Range
aRng.MoveStartUntil "(", -100
If Len(aRng.Text) > 2 Then
aRng.Style = "styMyStyl"
End If
End Sub
Note that I've chosen to limit the distance backwards that the macro searches for the starting bracket to 100 characters. You can fiddle this number if you want to search more or less characters.