How to reproduce:
1. Open the attached project named "EditorPrefs"
2. Click Examples -> Has Key Example in the Toolbar
3. Press the "Save XyZ as Key" button
4. Notice that the label says that the key exists and has a specific value
5. Press "Delete All" button
6. Observe the label again

Expected results: The label should show that the ref key no longer exists
Actual results: The ref key is not deleted

Reproducible with: 2021.2.0a1, 2021.2.0a3
Does not reproduce with: 2018.4.31f1, 2019.4.20f1
Could not test with: 2020.2.3f1, 2021.1.0b6 (Due to another similar issue regarding DeleteAll)

Notes:
1. On Windows open regedit and look for key: Computer\HKEY_USERS\S-1-5-21-1740753149-379259074-1237386463-1002\Software\Unity Technologies. Notice that it hasn't been deleted