It is basically a wrapper around av_get_audio_frame_duration(), with a
fallback to AVCodecContext.frame_size. However, that field is set only
when the stream codec context is actually used for encoding or decoding,
which is discouraged.

For muxing, it is generally the responsibility of the caller to set the
packet duration.
For demuxing, if the duration is not stored at the container level, it
should be set by the parser.

Therefore, removing the frame_size fallback should not break any
important case.