You're not alone in this. Many of us BDW brides have been where you are now. As you said, focus on whats most important, you & FI. In the end, when you look back on your day, thats what you will remember. The great time that you had & the fact that you are married!
Also, when I felt the same way & was complaining to my mother, she made me realize that the people that were coming, were the people who I should be thankful for. They made the sacrifice & that should show me how much they cared. I was spending so much time worrying about those that were'nt coming, that I forgot how special those that did come were!
So, again, you'll be fine. Don't worry about those who can't come, backed out, whatever. Think about all the wonderful memories you'll have of your wedding day, spent with friends & family who matter the most. And a group of 20 is much more intimate & manageable than a group of 40-50! I don't know what I would've done if I had to keep up with & entertain 50 guests instead of 30.