This behaviour change caused a regression on our side recently, we might
want to disable the option by default.