I'm not going to come out and say the starter is the only thing to replace but I will say that, in my case, it was the one thing that I can say made the most difference.
I'm not a fan of the "carpet bomb" approach to fixing things. If you change five things at once, and it works, you'll never know which was the fix. As a result, I took quite some time and made one change at a time. That's how I know it was the starter that made the biggest difference. I now run with a single Yuasa battery with no issues. The starter is such an inexpensive, and quick, fix I see no reason not to do it.
Whatever you end up doing, I wish you luck. I know, all too well, how it feels to have a beast in the garage that won't wake up.