I back mine out of the garage and downhill almost everytime I ride. It is best if you are tall enough to easily reach the ground with both feet, but I recommend that you back out as straight as possible all the way to the street with your hand ready at the brake. And whatever you do, don't get in a hurry.
If you really can't handle or are intimidated by the thought of backing any bike down your driveway safely, I'd honestly recommend either a different bike or a new past time!