I have the EXACT same problem. I have quite a few people who we aren't really close to but they have invited themselves. (Several people told us that they can make it... even though we never invited them). Others invited random friends to the wedding without asking us... So like you are, I'm getting the sense that people just want to go on vacation and think this is a good excuse, without thinking that this is a special event for us...
Anyway, I didn't send formal invites to these people. I figure that a verbal "we can come" doesn't hold much weight... and that they won't end up booking. On the other hand, if they are serious, maybe your bridesmaids can help spread the word that you'd rather keep this a more intimate event for closer friends and family.