If you're using KVM within a single organization to compartmentalize VMs, but they are all ultimately administered by the same people and toolchain, the threat is much less.
A lot of the extensive security precautions for qemu are intended for a hosting provider threat model where multiple random tenant customers may be using VMs on the same hypervisor bare metal, in which case you absolutely don't want them to be able to escape the vm or read each others' memory.
Which are typically closed source, which is quite a shame.