You can use a bit for zero or non-zero and then use bits only for providing the sign to non-zero values, for example. The sign part will be variable length but can probably be made very fast with hardware support.