Grant17: Thanks for explaining further Geek4me, that makes it clearer what is going on.
It sounds to me like you could be suffering from heavy packet loss i.e. many of the voice packets are not arriving at the far end of the link. This typically makes phone conversations raspy and extremely difficult to understand. It doesn't really account for the reduced volume though, that is strange. Are you sure the voice quality remains OK, even when the volume is low?
If there is increased network traffic on your base station (as evidenced by the reducing download speed), this could well be the cause of the packet loss.
If possible, you should move your equipment to another area of the city (with a different basestation) and see if you get the same results.
Alternatively try the phone late at night (after midnight) or early in the morning when internet traffic is light and see if that makes a difference.