Yes, the younger they are the lower the stress. I prefer banding 4-8 weeks of age, but Purdue U says after they have a belly full of colostrum is fine. Bulls grow faster, but study done concluded all the early gains from remaining a bull are lost when castrated late and calves castrated younger actually surpass late castrated calves in performance after weaning and in the feedlot when implanted.
I can't say I've noticed a difference except the occasional calf that takes late castration really hard and drags down the average performance of the whole late castration group... especially if it dies, that really hurts the average.
I've never lost a calf from early castration, but I can't say that about bulls I've castrated late.