It is no fun trying to build the herd you want, but it is worth it. We went through some rough years getting to where we are at now as well. We finally have our herd up to 100 head. We have only lost one calf in the last 2 years. It was all worth it. Get a good vitamin / mineral program in place, a good vaccination program, and stick to culling. Your goal should be to get a herd of cows that work for you instead of the other way around. Don't be scared to search the country for genetics that will better your herd. A good bull is half your calf crop and his genetics can run through your cowherd for generations.
I know there are stresses and pains in this industry, I am suffering from them right now. I have a job off the farm, so I have been working nights for the last week on the farm trying to get cows to pasture, herds sorted for bulls, a little seeding, and keeping up with the kids. Hell, last night I finished dropping off the last trailer of cows at 11:45 p.m. and was up to go to work at 6:00 this morning. This will be my routine for the next 5 days till I go on days off and can just farm for a couple days.