Steps to repro:
1. Open the attached project and 'TEST' scene;
2. Go to Edit>Preferences>GI Cache and click Clean Cache button;
3. In the Lighting window (Window>Rendering>Lighting) press the arrow next to Generate Lighting button and select "Clear Baked Data";
4. In the Hierarchy select Reflection Probe gameobject;
5. In the Inspector set Resolution to 1024;
6. In the Inspector click Bake button;
7. When the reflection probe baking starts (in the Progress bar at the bottom) start a timer;
8. End a timer when the reflection probe baking is done and remember the timing;
9. Repeat steps 2-4;
10. In the Inspector set Resolution to 2048;
11. Repeat steps 6-8;
12. Compare the timing for baking probes with 1k and 2k cubemap resolution.

Expected result:
Reflection probe baking time increases proportionally to the baking resolution change (~2 times).

Actual result:
Reflection probe baking time increases drastically (by ~4 times) when switching from 1k to 2k cubemaps.

Notes:
- Reproducible in Versions 2021.2.0a6, 2021.1.0b8, 2020.2.6f1, 2020.1.17f1, 2019.4.21f1;
- Reproducible on Windows and OSX.