I too have done this years ago, in the rain nonetheless, without pulling the motor. It is an achievable goal for a younger, energetic go-getter. These days I wouldn't attempt it, as it's really a pain, and certainly does not offer the ideal cleanliness factors or abilities to check tolerances. However, I'm not suggesting you not do it, as long as you have patience, and are willing to come out from under there for a breather, when you get frustrated. And you do not have to jack the motor up,,, but yes,,, remove the belt by all means!
Since you don't rely on it for a daily driver, do you have the ability/equipment to go on and pull the motor to do it properly? You may find you would benefit more from doing that, and being able to analyze if other things need to be replaced/redone as well. Just something to consider...