No matter what you do, people will complain. We anticipated some people not being able to afford it or simply not wanting to shell out the dough. So before we even sent out the Save-the-Date packets, we vowed that whoever came came and we would have a BLAST. And going in with that attitude totally helps. So whenever someone says anything remotely related to the cost, either directly to us or to someone else and it gets back to us, we simply and lightheartedly say that we understand, if they can't make it it's fine, but we just wanted to extend the invite.
I think when you let people know it bothers you, for some reason they continue to run their mouths. Take the power back from them. It's your wedding and in the end make a short list of the people without whom you would not want to get married. When I did that, the only people that absolutely HAD to be there or I wouldn't get married were him, my daughter, his mother and my best friend. That's it. And they're all coming. So it will be perfect.
Be encouraged.