-
Ronald S. Bultje authored
This fixes integer multiplication overflows in RGB48 output (vertical) scaling as detected by IOC. What happens is that for certain types of filters (lanczos, spline, bicubic), the intermediate sum of coefficients in the middle of a filter can be larger than the fixed-point equivalent of 1.0, even if the final sum is 1.0. This is fine and we support that. However, at frame edges, initFilter() will merge the coefficients for the off-screen pixels into the top or bottom pixel, such as to emulate edge extension. This means that suddenly, a single coefficient can be larger than the fixed-point equivalent of 1.0, which the vertical scaling routines do not support. Therefore, remove the merging of coefficients for edges for the vertical scaling filter, and instead add edge detection to the scaler itself so that it copies the pointers (not data) for the edges (i.e. it uses line[0] for line[-1] as well), so that a single coefficient is never larger than the fixed-point equivalent of 1.0.
d49352c7