How to get the 5.1 output from the PlayStation 3?

If you want Surround Sound, you must use a digital connection. Optical provides 5.1 and HDMI provided both 5.1 and 7.1. Your receiver must have at least an Optical input in order to get 5.1 from the PS3. Once you connect the devices, you need to set the Optical sound connection to Linear PCM (this is in the Video Settings, don't ask why). The receiver will probably not say that DTS is enabled, but it will be playing in the DTS format (this is what Linear PCM does, omits DTS display on your receiver).