A tough one! We aren't inviting kids to the wedding or reception, but I've told my friends/family that they are welcome to bring them if they're making it a family vacation and we will make arrangements for babysitting (some are bringing a parent or other sitter with them, but most are viewing the long weekend as an adult getaway). If we weren't having a flower girl, though, I agree that we would have chosen an adults only resort to eliminate the issue.
As a person who's traveled out of town for a dozen weddings the last couple of years, though, I've attended more weddings that were adults-only than not - and those were "destinations" in their own right and often required 3 nights away even though they were in the US! If you're doing a long weekend (vs. a week), I think if you communicate it far in advance to those close to you, you'll be in much better shape. At the end of the day, if you've accepted the fact that some will come and some won't (regardless of kids, destination, timing, etc), hopefully you won't be too disappointed.