You don't need an OR test for what you've described; you need a LIKE test:
Code:
Sub DeleteLinesStartWith()
Dim i As Long
With ActiveDocument
For i = .Paragraphs.Count To 1 Step -1
With .Paragraphs(i).Range
If LCase(Trim(.Words.First.Text)) Like "[!.•]" Then .Delete
End With
Next i
End With
End Sub