It can be done, and has been tried before. But your product would not be F1 calves, they would be F2. This means they would have, on average, half the hybrid vigor seen in the F1 calves (may not be initially observed because the F1 dams would provide maternal hybrid vigor that did not exist in production of the F1 calves). In addition to the loss in hybrid vigor, you would also have an increase in variability due to the crossbreeding coming from both sides. You will have calves that range in appearance from full Brahman to full Hereford and everything in between.
What is typically done is to take the F1 dams and use a purebred terminal type sire (not Hereford or Brahman). This allows for 100% direct and maternal hybrid vigor and calves that are as uniform as possible for a crossbred.