Fix overflow check for multiplication:

- Add special test to detect the case of -1 * INTMAX_MIN
- Protect against elimination of the test division by the optimizer

Garrett Cooper noticed that the overflow checks were incomplete, and Bruce
Evans suggested the use of the "volatile" qualifier to counter the effect
of the undefined behaviour, when the prior multiplication caused overflow,
and he also suggested improvements to the comments.

Reviewed by:	bde
MFC after:	1 week

1 files changed, 16 insertions, 4 deletions

diff --git a/bin/expr/expr.y b/bin/expr/expr.y
index 1856ec8..0ddf399 100644
--- a/bin/expr/expr.y
+++ b/bin/expr/expr.y
@@ -444,14 +444,26 @@ op_minus(struct val *a, struct val *b)
 	return (r);
 }
 
+/*
+ * We depend on undefined behaviour giving a result (in r).
+ * To test this result, pass it as volatile.  This prevents
+ * optimizing away of the test based on the undefined behaviour.
+ */
 void
-assert_times(intmax_t a, intmax_t b, intmax_t r)
+assert_times(intmax_t a, intmax_t b, volatile intmax_t r)
 {
 	/*
-	 * if first operand is 0, no overflow is possible,
-	 * else result of division test must match second operand
+	 * If the first operand is 0, no overflow is possible, 
+	 * else the result of the division test must match the
+	 * second operand.
+	 *
+	 * Be careful to avoid overflow in the overflow test, as
+	 * in assert_div().  Overflow in division would kill us
+	 * with a SIGFPE before getting the test wrong.  In old
+	 * buggy versions, optimization used to give a null test
+	 * instead of a SIGFPE.
 	 */
-	if (a != 0 && r / a != b)
+	if ((a == -1 && b == INTMAX_MIN) || (a != 0 && r / a != b))
 		errx(ERR_EXIT, "overflow");
 }