The last point is easy. Operate the trail centres like ski resorts. Buy a day pass, sign the rules and if you are caught breaking them you are banned.
Never work. A good number of people won't pay when they can just join the trail at any point for free. And on most trails it will be impossible to stop them entering at any point - the cost of fencing off the trail would be enormous unless the whole area is fenced off already. Likewise, given the length of most built trails, how would you police them to see if people are breaking the rules?
Nope, the only real way to encourage better behaviour on trails is education and peer pressure. Perhaps if you address the people who run the trails or any clubs or internet fora whose members use these trails, telling them how inconsiderate riding by a few selfish people is putting off the next generation, they might be further moved to emphasise good trail use to the users?
Regarding dangerous or 'bad' trail use, TBH, I would not complain about people going very fast or overtaking unless it's designated a Family Trail. I don't know the trail in question, but on the built trails I use part of the fun is going flat out and testing your skills to the max - that's why people built the trail in the first place. Thus, on my trails if someone comes up fast behind me and passes at speed having given an audible warning, I would not be annoyed, indeed, I would feel some duty to get out of the way if convenient so as not to spoil their ride. On the other hand, I would agree that doing close, unexpected, fast passes on inexperienced children is a bad thing, and this is what I would emphasise if it were me making the complaint. Likewise, if the overtaking rider forced their way past me, or got verbally annoyed because I had no where to go to let him past, then I would say that they were out of order (and in the past other riders have learned the error of their ways the hard way.....I'm more solid on a bike than I look!).
So basically, I think what I'm saying is that to garner more support for your issue within the MTB community - concentrate your complaint on how the chap treated your kids rather than his general 'race pace' speed, as if other MTBers get the impression that you are just complaining about fast riding then they'll lose sympathy. And I hope that your kids get over the shock, that they realise that this chap was not typical of the MTB community, and that they can continue to enjoy their MTBing.