I could just see myself explaining to the wife that i did not see the car in front of me apply the brakes because i was messing around with the phone while riding. Everything has its place and time using a phone while riding fits neither of the above. To each his own I guess.
You're absolutely right. We riders get all bent out of shape about people playing with phones in traffic when they're in cars, and not getting worked up about people on bikes doing it is just a huge double standard.
That said, a smartphone on a phone holder on a bike will probably be doing double duty as a GPS unit. That's not at all the same thing as texting and driving. You set the GPS app up while stationary, place the phone in the holder, and then you just ride. The smartphone GPS gives voice instructions on where to turn, which will probably relax you more than trying to be reading maps and every street sign in the vicinity.
So mounting a phone on the bike isn't automatically a safety hazard, in my opinion. It can in fact be a good help. Just as long as you do what any reasonable person does - namely, keep your mitts off it while riding, and only make adjustments or settings changes while stopped.
I have no problem with people using their phones that way in a car, either, for that matter. All distractions are bad, but some are not all that distracting. In fact, talking on the phone (with a headset) is a lot more damaging to traffic safety than a smartphone used as a GPS, in my opinion.