It usually turns out to be weight in one saddle bag. A few extra pounds will cause a pull to one side or the other. And the bike will also follow the crown of the road down hill.
Hell, I've even had a buddy complain that when he wears his wallet the bike pulls left, that turned out to be because his ass hung off to the left side of the seat when his wallet was in his left pocket.
It's either wind, weight, road crown or body position that causes it, just have to narrow it down.
It's not the drive shaft, no way. The drive shaft will make it pull on hard acceleration as the tire breaks loose, that's about it.