Steps to reproduce:
1. Open project
2. Open "Scene" scene
3. Select "Tree" prefab and change layer to any other (not Default)
4. Select Terrain object in hierarchy and set layer to Default (if it's not set)
5. Select Camera and unselect Default layer in culling mask
-- Notice that terrain and all terrain objects aren't rendered in game view
6. Select camera and select every layer in culling mask
7. Play in editor
8. Ray cast with mouse click in game view window on terrain and terrain's objects
-- Notice that ray cast ignores terrain's objects as they aren't on default layer

Results: Terrain objects doesn't inherit collision layer but does inherit layer when using culling mask

Note: Behaviour should be unified in all cases

Reproduced with: 5.4.0b25, 5.3.5p7, 5.2.4f1