Steps to reproduce:
1. Open any project
2. Create a Render Texture
3. Set the texture dimensions to 3D
4. Set the texture size to something large like 1024x1024x512 (or larger if needed), set Anti-aliasing to 8
5. If Unity hasn't crashed yet, select Main Camera and set the render texture field as this new RT you've created.
6. Unity will crash without bug reporter and ghost process will remain eating up CPU

Expected outcome: Unity doesn't crash, if some upper limit is hit, there should be an error message.
Actual outcome: Unity tries to build a texture that's too large and crashes without bug reporter. Background processes remain and can go unnoticed eating up CPU/memory usage

Reproduced on 2017.1.0f1, 2017.2.0b1, 5.6.2p1, 5.5.4p1

Note: Attached "RenderTextureMemory.zip" just has a bunch of Render textures that eat up massive amounts of memory and basically freeze the system.