Phin: ...All the installation information tells you to connect the alarm first in the line.
The reason they recommend this is so that the Alarm Panel can "seize the line". This means that it can interrupt the connection of any telephone devices in the house before going "On Hook" for a short time. Then the modem will go "Off Hook" and dial out. So it will still work if somebody left a phone off the hook somewhere, or they are talking on the phone when the alarm goes off.
At least, that's the way it works with a monitored alarm panel which dials OUT whenever there is a change of state. In your case where you have to dial IN, it is a little bit different because if one of the phones in the house is already "Off Hook" it won't be possible for the alarm to answer.
Phin: ...I think the problem lies in timing issues within the Voip protocol.
Yes, I agree with you there. However, trying to run analogue modems over a digital network is not a good way to be doing things in the 21st century.
It is now the responsibility of alarm panel manufacturers to provide an Ethernet port as alternate access for those homes with Naked DSL -- or no landline at all -- which is going to be increasingly common in the future.