We had this issue too. FI has a huge fam and mine is tiny. We just invited my parents and bro (all gp's are gone) and his "first tier" fam. That means only his parents, bro, gma and gpa, and aunts/uncles first cousins. Sounds like a ton but it's better than inviting all 400 of his family members.
We did the "who would I have coffee or a beer with alone" rule. Would either FI or I sit and chat with X if we weren't being forced together by a family event.