A few thoughts. Like any electrical circuit, the positive wire carries the voltage and amps to the load (in this case the fence and anything that touches it). The negative "wire" carries the amps back to the source (charger negative terminal). The negative "wire" in this case is the animal's feet, the dirt, the ground rods and clamps and the negative wire to the charger. The path through the ground will be the path of least resistance depending on soil moisture, mud puddles, buried metal, steel panels touching the ground, etc. Is the ground wire from the charger to the ground rods insulated or uninsulated? How is the ground attached to the ground rods? Are the connections corroded? If the ground wire is not insulated or has cracked insulation and there is corrosion between the ground wire and the rods, the current flow from the fence back to the charger is still going to take the path of least resistance. Steel panels setting on dirt may have less resistance than a corroded connection of the ground wire to the rods.
Easiest solution is probably to move the ground rods away from the panels and closer to the charger.