Ok look for yourself in 2006. We were #27 in the nation on defense Pts/Gm, and #8 in the ACC behind VPI, WF, Miami, BC, Clemson, Louisville, & UVA. The ACC was so bad in those days that there were open discussion on ESPN about should the ACC be dropped from the Power 5. It picked up the moniker the 'All Clown Conference'.
Don't confuse stats with rankings. In 2007 we were #21 in Pts/Gm but ranked #48. That is because we played bunch of weak offenses. You can't be outside of the top few defenses in the ACC and expect to be top 25 nationally, not in a conference that is so weak Wake Forest is your champion.
2007 Team Defense | College Football at Sports-Reference.com