Follow your heart.
I am toying with the exact topic right now and have decided that we will invite everyone we would have had it been a "normal" wedding at home; however, I would love to be able to put in some type of caveat that let's people know we would love for them to be part of our day, but completely understand if they can't make it. That way, it's their choice with no pressure.
I would imagine the people who come are going to be the people you would truly want there anyway.
Good luck!!!