On the 3-3-5... watched it for many years. New Mexico has been working that scheme for over 20 years with considerable success at times. They never had the depth or athletes to really make it successful in big games, but they were always very tough for pro-set offenses.
At UNM, they tended to get run over by big powerful lines. However, that was with UNM athletes. At South Caroline (Joe Lee Dunn) it worked really well and Lou Holtz uses a similar variation of it today in a modified 4-3 set-up that oeprates much the same way. South Carolina does not get run over in the SEC. But all defenses have their weaknesses.
The thing that makes BYU dangerous is that the defense is a perfect match for their offense. It's risky and therefore tends to produce as many turnovers as the offense gives up, so it is a neutralizer. Furthermore, BYU has exceptional athletes, every bit as good as most BCS teams, they really do. That being coupled with this defense means they are probably much more capable than UNM, a school which usually ranks in the top 25 in total team defense and in a league where offense really is has always been a big deal (long 75 play/team games)... that's saying something.
So with Mendenhall has now gone from UNM to BYU, has better athletes (14 with starting Experience, 3 legitimate All-Am prospects), so it's expected that BYU will be very very good defensively.
The trump card is they've never played a real game with it. So we'll see. the Key to making it work is taking the big lineman off their feet to create natural walls and then shooting the gaps with smaller, quicker men who disrupt the backfield. I can tell you UNM when not banged up could really stop even the best running teams, big powerful running teams like Utah and UNLV. I suspect BYU will be hard to run against... but we won't know until tomorrow night. The scary thing is they have great individual cover guys and read/react safeties so it will be hard to throw against them as well. Some of us think their other corner (Brandon Heaney) may be the better of the two and yet it's Gilford (NFL bound for sure) that gets the All-Am honors. Their roverback in the system is a 225 lb. safety that hits like a freight train, Francisco Aaron so it should be interesting to see them play all those stunts and blitzes.
MSU still employs a similar defense and Jackie Sherrill couldn't get it to work against BYU, but then BYU faces it year in and year out. Florida for the same reason has similar offensive success against the Bulldogs every year. But when the players are healthy, it's a very good defense. It puts a lot of people wherever the ball is and forces the QB to make quick decisions, good decisions because any little mistake will be costly.
Finally, UNM, MSU and South Carolina all generate a lot of points off defensive plays. BYU has better athletes, excellent linebackers and lots of depth in the defensive line so they and expect to do the same. Add a potentially powerful top 10 offense which is pretty typical of BYU in historic terms, exceptional special teams (kicking and punting) and BYU really could be either a bad underachieving team (like last year)or a very very good team, much better than the ranked team of a year before. I think the last scenario is very likely from what I am gleaning in fall camp notes.