1. Given that you're running a new Word session, you don't need CreateObject; you can use New instead.
2. You need to be specific as to which application your Shape belongs; otherwise it is assumed to belong to the calling app, not the called one.
3. You don't need ActiveWindow.Selection; instead you can tell Word precisely where the pic should go.
Try something along the lines of:
Code:
Dim wdApp As New Word.Application, wdDoc As Word.Document, wdShp As Word.Shape
wdApp.Visible = True
Set wdDoc = wdApp.Documents.Add
Set wdShp = wdDoc.InlineShapes.AddPicture(FileName:="C:\MyPicture.jpg", LinkToFile:=False, _
SaveWithDocument:=True, Range:=wdDoc.Range.Characters.Last).ConvertToShape