Steps to reproduce:

1) Download attached project 'CardboardTest.zip' and open in Unity.
2) File >> Build Settings >> Player Settings >> Resolution and Presentation >> Make sure Orientation is set to Auto Rotation.
3) Build and run this project on a device.
4) Make sure it opens in a portrait mode.
5) Rotate to landscape(see attached image 'wrong.png').
6) Press home button to leave app on a background and back to app. Problem disappears(see attached image 'good.png').

Reproduced with:
5.2.4f1, 5.3.6p3, 5.4.0p3, 5.5.0b1