Not necessarily. Say you have 101 returns, and all but one get to the 23 yard line. The remaining return goes for a touchdown. So, if you take the 25 yard line as zero, you've netted negative 125 yards. But, I would argue that even just one touchdown is clearly worth losing 2 yards on your starting field position for the rest of your possessions.
Obviously, this is an extreme example, but if you have a bunch of returns going to the 19-23 yard line, and a few gaining an extra 20+ yards, even if the overall average is short of the 25, it's still worth the risk.