Don't feel guilty, it's your day. If people can't afford to go, then they can't make it.
We decided that we were going to Jamaica for the wedding, WE decided. We invited a small group of friends to join us there, and then we're having a reception when we get home to celebrate with everyone who couldn't make it to Jamaica. So we'll have a group of 20-30 (currently 22 confirmed) in Jamaica, and then a reception for 150-200 when we get back. For us, this seemed like a good balance