My sister went to UVA and we have made the trip several times. I love Charlottesville and highly recommend going up for the game. We stay just north of town at any one of the chain hotels that they have. The Omni is the best in the area, but it will be full, as will most of the downtown hotels.
If you are driving take Friday off and drive up the day before. It is a hike unless you fly into Richmond or even DC. We usually left Thursday night and stayed in Asheville on Thursday and spent some time around the Biltmore or other attractions on Friday before driving up to Hooville. Once there, check out the Mall (downtown area with great restaurants, bars and shops).
Whatever you do, get your reservations early (for dinner and for rooms).