I don't believe an intelligent skilled rider pushing at the limit is as saddled by age as much as they would be in many other sports. The age will come after the adrenaline dump, during recovery, when it takes ages longer.
It's mostly mental, at least in my case, I am no slower now than 20 years ago. Maybe measurably so on a reaction time tester, but I don't subscribe to that translating to being a slower rider.
With age comes knowledge, comfort and confidence in your skills, bike, and superior judgement. Those things, IMHO, offset any minor degradation of reaction time.
The fastest riders I know are all over 30, most over 40...
That said: The day I have a wreck that's my fault because of physical limitation, not misjudgment, is the day I hang it up.