Reproduction steps:
1. Download and open the submitted project "android.zip"
2. Change the Package name (Edit -> Project Settings -> Player -> Other Settings)
3. Open the Test Runner window (Window -> General -> Test Runner)
4. Make sure that no devices are connected so that the tests fail
5. Press "Run All Tests (Android)"
6. Dismiss the error and observe the package name

Expected result: Package name is reverted back to what it was after step 2
Actual result: Package name is set to com.UnityTestRunner.UnityTestRunner

Reproduces on: 2019.4.15f1, 2020.1.14f1, 2020.2.0b12, 2021.1.0a6

Could not test on: 2018.4.29f1 (Scripts lose references and other compilation errors occur)

Notes:
Could not test on iOS as tests run successfully in the provided reproduction project.