lavc/audiotoolboxenc: fix a number of config issues

- size variables were used in a confusing way
- incorrect size var use led to channel layouts not being set properly
- channel layouts were incorrectly mapped for >2-channel AAC
- bitrates not accepted by the encoder were discarded instead of being clamped
- some minor style/indentation fixes