Steps to reproduce:
1. Download and open attached project "784179_repro.7z"
2. Make sure VR mode is on ("Edit->Project Settings->Player" and select "Virtual Reality Supported")
3. Open scene "SceneTest" and play it
4. Select camera and notice that ray is drawn off center
5. Turn off VR mode and replay the scene. Now notice that it is drawn straight ahead

Actual result: PhysicsRaycaster is being cast off-center
Expected result: It goes straight ahead

Note: See screenshot:
https://gyazo.com/c6a32b77b4da6d906e8c029dbde6139f VR mode off
https://gyazo.com/f334f4cb93c6b0f554e9ec5a53398463 VR mode on

Tested and reproduced on:
Windows 10
Unity 5.4.0b13

Not reproducible on:
Unity 5.2.4p1, 5.3.4p1