This. And performance wouldn't be worse than using AVX twice...
But worst case scenario of the OS moving the process to a big core on an illegal instruction or scheduling it to the right core based on a required capabilities system would also be quite acceptable most of the time.
Plus, it'd be great for supporting more specialized cores designed for different purposes and running a single core ISA with extensions for their specific needs. IIRC, there are some ARM chips that have three different kinds of core.