Every horse I have peaks at age 4, and then declines. This plays out both in their performance and their potential dropping.

Am i overracing them at 2/3, or is this irrelevant?

Is there any way to prevent your horses from declining from Age 5?

If they're flat racers, this generally tends to the way it goes. Sometimes you'll get horses racing for longer but superstars are usually done by the time they're 5.

The thing that can change this a little is the maturity bar - the longer it is, the slower the horse matures. So they'll be better when they're older but still not far past 5.

The upside is that if you're breeding them, the decline in potential doesn't matter, they'll pass on their highest possible potential rating.