I think that if you asked them to be in your wedding, it means they are very special to you. Who cares how many guests you have? I had 3 BM's and 5 (or was it 6?) GM's as well as 2 flowergirls and 1 ring bearer, and we only had about 25 guests. Including the weddnig party! So I don't think there is anything wrong with having the people that you felt should be in it! I also had 4 BM's that couldn't come to Jamaica, but they still participated in everything (shower and whatnot) and wore a special dress to the AHR so they appeared to be a part of the bridal party. I wouldn't change a thing.
Ok, so that was me. Maybe you don't really care what I say (not in a mean way!), and you want to ask people to step down. Who will you have remaining? Siblings? Just wondering, cuz I think it depends on the people on how to go about it. Like if you have a sis and your FI has a bro, and those are the ones you want to remain in the party, I think it would be easy to say "We aren't having the turnout that we originally thought, so we decided that we needed to cut the bridal party down to jsut family." Or whatever.
But honestly, I don't think you HAVE to, unless you WANT to. Hope that all makes sense and maybe even helps you!