Hopefully someone with more experience with bottle calves will chime in, but maybe because he is three weeks old he does not want to accept the bottle? Have you tried putting it in a bucket and getting him to drink it that way? Have you seen him drinking water? Have you got access to a vet for advice? He might be dehydrated, and an IV of fluids might be in mind if he is that far gone. It might be time to get a vet involved. I would guess, just by the age of the cow, that she just did not give enough for the calf, and it is a case of failure to thrive.
Just a thought, if he is eating calf manna, what about soaking the pellets in milk???