Ok, so I figured out how to do this:
Find ^p^w
Replace ^p

Feeling like a champion now

However, now I noticed another problem. After some words on the list, there are empty spaces. I want them gone. Any ideas how to get rid of those?

Edit: I am on fire today! Found a solution to my latest problem:
Find [space]^p
Replace ^p

I hope this s useful for someone else!