You just described it, actually. The big bolt marked with a torque value of 23Nm is the pivot bolt. The one that I marked in red is what attaches all that to the shaft. Those are the only two points of contact between the "gear change pedal" and the bike. Remove the two bolts completely and it will all come off together. The other bolts and nuts are just for adjusting the height of the shifter. If you need even more clearance you can rotate where the clamp attaches to the shaft. As I mentioned, you may want to lay down under the bike and mark the shaft and that clamp with an alignment mark for reference. What I'm calling the "clamp" is what the 8 Nm bolt at the bottom of the diagram goes into. Hope this makes sense.