Bobinator's got it.
It's not really the "bigs" that kill us, it's the consistent athletic mid-size guard types that are killing us. Hill and Lane are good on-ball point defenders, but neither matches up with mid-size athletic guys. Agee, bless his heart, can actually defend a big reasonably well with his strength, maturity and skill. The only guy we have that can "sort of" match up with the longer athletic types, other than Agee, theoretically, is Vinson.
As it is, if a team has more than one big guard on the court, we're immediately in double-team, rotational scramble pretty much the entire possession. The effect on our already weak defensive rebounding is often catastrophic.