It sounds like you're practically writing the experience we went through when we booked our wedding. We had a lot of people say they weren't coming...and they're almost all coming now.
What we did was made a list of the things that were most important to us. For example, can the people who are the closest to us come, is it what we imagined, are we going to be okay with a more intimate wedding. The only really important guests who we'll be missing on our wedding day is my sister and her husband.
At the end of the day, do what is going to make you and your fiance happy, you're marrying each other, not the members of your family. We're still getting a lot of grief over the fact our wedding is happening in Mexico, but not as much as we used to. The closer to the deadline we're getting, the more people are starting to get excited about it.
Good luck!