Any help would be appreciated.
Do you have your amp hooked up to your PC via SPDIF or analog 6ch output?
sbiddle: First thing to do is make sure your sound settings are set to 5.1 in your sound card config. Configure this and test it.
Now in AC3filter set the output format to 3/2+SW 5.1CH and make sure the SPDIF box is not ticked.
In the mixer settings check the same setting is there. Enable auto matrix (you may want voice and stereo expand set as well, it's a matter of personal preference). Then in the next window make sure there are no ticks in the SPDIF passthru options.
Now providing you select AC3Filter as the sound device for AC3 you should see the sound mixer showing audio and be hearing something out of your amp.
Remember not all audio is 5.1 - some of the AC3 is only stereo.
sbiddle: You can set the bitrate to whatever you like - it'll just use more CPU resources the higher you go. 16 bit is fine for most purposes.
You should see the levels moving when it's decoding.