DO IT!!!
One tip, if you can find a community college near by that offers it sounds like that may be the perfect environment for you to start. That being said, i don't think you could find a sensei who wouldn't welcome you if you came in and explained yourself to him like you have here. You might get razzed for sitting out a few rounds but nothing mean-spirited. I think the most important thing is the desire you're clearly expressing, and no dojo wouldn't welcome someone with enthusiasm and respect for the art. I think a lot of the injuries people are talking about come from training for a very serious competitive level maybe? I have bad joints (don't ask me why, i'm only 21) but i've never experienced more than some soreness the next day. Unless you have some very serious joint or back problems, the pounding you'll take on the mats is nothing your body can't take.
Aside from the very practical reasons, I think there's something very telling about the fact that the first thing you learn in judo is how to take a fall. You're going to take a pounding, but it's all in the interest of learning. It's like the saying goes, "get thrown 7 times, get up 8" and it sounds to me like you've got the attitude to do just that.