Do what will make you feel comfortable. My advice is this is YOUR special day and those invited should be people who care and matter to you and your FI.
We are spending a week in the resort we are getting married inat (to allow time with our guest) and then going to another resort for the HM (which is a complete surprise to me) for 4 days.