Supposing that you are looking at the Online & Recent Visitors screen.
If you click the "next" button at the bottom of the screen, you will be taken to the next 20 visitors, and see this URL:
The number "20" at the end of the URL refers to the number of visitors you are seeing. Therefore, if you make this number "500", instead of "20" you will be taken 500 visitors back.
So, if you know that you receive 100 visitors a day, for example, and want to find a day four days back, make the number at the end of the URL 400.
Yes, this is an ad-hoc solution. We hope that for the time-being, this will help, until we are able to program a more efficient way to scroll back.
This tip applies to all visitor tables; online, first-time, returning, search terms, referred?, etc.