This is not a bug, but is in fact how assembly definitions are expected to work. While they can reference things in UnityEngine/UnityEditor by opting into the respective platforms, code that exists in other assemblies needs to be added to the list of dependencies (Assembly Definition References in the Inspector). In this case, the user needs to add a reference to Unity.Mathematics to the newly created assembly. Code that does not exist in an assembly definition (i.e. exists in the default assembly, something like Assembly-CSharp) automatically references all the assemblies in the project, which is why mathematics is accessible when not in an assembly defined via an asmdef.

That said, there is currently a limitation where assemblies imported as a package dependency do not appear in the project. In short, if the user has installed the entities package, it implicitly installs the mathematics package. But because the mathematics package has not been explicitly added, it won't appear in the Project Browser (and respectively, won't appear in the object picker when they try to assign it as an Assembly Definition Reference). As such, the user must do one of two things to reference Unity.Mathematics in their new assembly:

a) Manually edit the asmdef file in a text editor and add "Unity.Mathematics" to the list of "references"
b) Explicitly import the mathematics package via Package Manager (after enabling preview packages to find it) and then assign it as a reference to their new assembly definition via the Inspector