Hi Thistle
I'm mostly a road rider, and my Mercian had an expensive repaint (at Mercian) a few years back, so I tend to keep that bike to roads for fear of stone chips. However, I'd ride other, similar bikes on tracks without worrying about it. The frame construction is as strong as anything available, and wheels with 32 or (preferably) 36 spokes are going to stand up to most things.
Where suspension really comes in is in bouncing over boulders and the like, that can knock steering off track or stop you dead. You certainly couldn't do some of the down-hill racing that now goes on using a rigid frame. On the other hand, look at cyclo-cross for an example of racing off road on various terrains - the classic example being the
Three Peaks. That branch of the sport does now allow MTBs, but pre-dates them by decades, and serious competitors still choose
cyclo-cross designs, which are modified conventional bikes.
Or
Paris Roubaix, the classic event in which road bikes are ridden flat out over cobbles that hammer man and machine to pieces. I don't think any photo can do justice to the battering that (most) bikes survive - and these really are lightweight road bikes, with minimal concessions to the surface (wider tyres than normal, and so on). And it's not just professionals, who can get issued with new bikes afterwards - there's an
event for amateur riders too.
You'll notice that the one modification that isn't mentioned in that cyclo-cross article is making the bike stronger, because that's simply not necessary. OK, you probably shouldn't take your lightest racing wheels or ultra-light road frame off road, but carbon frames are certainly used now, and we're talking about 531 steel, which will have no problem (boulders and stupidly steep drops excepted).
Remember, when riding on tracks, to use your arms as part of the suspension system. Relax your arms, do not lock them, and let the bike find the way. The top half of your body will then behave as though you're riding the latest hi-tech MTB forks.