it's fine if you say no, at the end it's Your big day and you decide who's invited. There is no an easy way to tell those who ask but a good answer could be that you wanted to keep it intimate to closest family.
it’s fine to limit the guest list to immediate family, you can hosting a separate “cocktail party" at your town. So, you should not feel guilty at all!