Lightmap Encoding is not exactly an input to light baking. It affects the texture importers for lightmaps, and thus causes lightmaps to be reimported when the setting is changed in the Project Settings window. The setting is relevant primarily after baking.

When you change this setting in said window the internal Lightmapping.OnUpdateLightmapEncoding(BuildTargetGroup) is called immediately after, which, among other actions, will reimport lightmaps. If you want to use reflection to change the lightmap encoding setting, you should also use reflection to call this method like shown in the snippet below.

Doing this before baking, however, does not make much sense, since as soon as you change the project setting, the chosen encoding will be used for all lightmaps in the project. Lightmap encoding being per-bake or per-lightmap is currently not supported.

Snippet of how to call OnUpdateLightmapEncoding:
```
typeof(Lightmapping).GetMethod("OnUpdateLightmapEncoding", BindingFlags.NonPublic | BindingFlags.Static).Invoke(null, new object[] { EditorUserBuildSettings.selectedBuildTargetGroup });
```