I totally understand, and i have been upset about this lately as well. i even considered canceling the destination wedding cause quite a few people that i love dearly could not make it.
But when my FI and I sat down and spoke about it, we decided that as much as it sucks that people can;t make it, we have always wanted to get married away, and when it comes right down to it, it is really about us.
So chin up, deep breaths it will all work out.