You nailed it there - you need to stop trying to please everyone.
Trust me, YOU CAN'T!!!!! (In the beginning when I was doing more of that - I would run in the direction of one suggestion only for that person to RSVP no anyways - grrrr - so I stopped that months back!)
I think you need to go with your heart on what you want for your wedding and accept that some people won't come for the stupidest of reasons, even people very close to you. If they want to be there, they could go!