avcodec/vmdav: return the amount of data that has been unpacked from lz_unpack() (as well as errors)

and setup the bytestream buffer size accordingly

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7fdcc513cd45_229_12.vmd
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>