A lot of us seem to have one problem or another when it comes to other people and our weddings... I know you are having a difficult time and are so stressed out and upset (I think we all are in the same boat). Doing a destination wedding comes with even more possible problems. Without going into great detail we have people not coming that we really hoped would (FI mom and my best friend) and we have people that are mad that we are going because they can't afford it (my father).
But in the end my FI and I decided that we are still going forward even if we are the only two people there because this is what we have always wanted and really the only people that matter are he and I. As long as he shows up that's all that matters! lol. Of course it would be great to share it with all our loved ones and it does make me sad to think about what I am giving up. But I certainly don't want anyone there that doesn't want to be there and for the people that do show up, that is really saying something about them. We are even leaving our children (from previous marriages) behind, not because we don't want them there but because its the only way for us to have the wedding we want without giving up our dreams. We plan to hire a videographer to capture everything so if they do want to watch it they can. Same goes for the family and friends that don't want to go.
Bottom line, its about you and your FI and if I were you I would do without the wedding party because it isn't going to mean anything if its not special to them.
And of course you should still have your party! Just don't invite them!
Keep your chin up, and we are always here to listen!