We totally had this same issue. Except we aren't limited to 20 people, which makes us look even worse. We chose only immediate family and then friends. No extended family. This is because FI hates one of his aunts and one of his cousins and didn't want them at the wedding...and he was told by his mother he had to choose all or nothing. So we (he) chose nothing. This has caused major drama in the family, but we're just assuming they'll grow up and get over it at some point.
FI and I are definitely doing things the way WE want them, maybe to a fault. But that's just how it is. We don't see why our wedding is such a big deal to so many people...it's just a wedding! But I guess it is a big deal to some. I think most couples make decisions to avoid drama which is probably good. But we didn't. I don't know if this has helped you at all but I just wanted to let you know that if you do decide to only invite friends and immediate family, you won't be the only "evil" bride&groom out there!