The buffers are created associated with the context, so they should be
destroyed before the context is.  This is enforced by the iHD driver.