He's the kind of person I'd consider an arsehole.. the kind I"d get away from.. it sucks that you have family ties to him, and I don't know what you could say that'll make things any better. It sounds like he's got trust issues somewhere.
I take it he's your sisters husband and your sister is coming to see you? If not I'd probably schedule my horseback rides to coincide with their visits as an easy way out.
For the kids... Offer to have them overnight (OK, that's opening a can of worms perhaps), and give them the farm tour without dad around, I think they'd feel more confident too.
Today my best friend came over for a visit (lives in Taiwan), and has 2 little girls, aged 4 and 7. They wanted to see a cow, so I went and got "Kama", who's a people-loving bred heifer. The girls' grandmother, who doesn't speak any english was with them. Kama did want to skirt around them, so I let her, and I picked up a full bucket of grain for her and put it on the ground. She was well occupied with that. The girls then got to pet her, and grandma got lots of pictures. I had to be a little concerned when the 4 year old had to show her excitement by jumping up and down and clapping her hands, and there was a bit of a language barrier between us so I couldn't do much about it.. Kama was a bit concerned about this little noisy jumping 'thing', but behaved very nicely.
Even though they're animals, they're all different, but they have unique characters... I knew Kama was cool headed around people, that's why I chose her... I think you know your horse well, and your horse knows you.. For him, perhaps a nice dog bit him out of excitement when they were playing.. I don't know.
This might help you though