/* You're not supposed to hit this problem */

For some denormalized long double values, a bug in __hldtoa() (called
from *printf()'s %A format) results in a base 16 digit being rounded
up from 0xf to 0x10.

When this digit is subsequently converted to string format, an index
of 10 reaches past the end of the uppper-case hex/char array, picking
up whatever the code segment happen to contain at that address.

This mostly seem to be some character from the upper half of the
byte range.

When using the %a format instead of %A, the first character past
the end of the lowercase hex/char table happens to be index 0 in
the uppercase hex/char table hextable and therefore the string
representation features a '0', which is supposedly correct.

This leads me to belive that the proper fix _may_ be as simple as
masking all but the lower four bits off after incrementing a hex-digit
in libc/gdtoa/_hdtoa.c:roundup().  I worry however that the upper
bit in 0x10 indicates a carry not carried.

Until das@ or bde@ finds time to visit this issue, extend the
hexdigit arrays with a 17th index containing '?' so that we get a
invalid but consistent and printable output in both %a and %A formats
whenever this bug strikes.

This unmasks the bug in the %a format therefore solving the real
issue may both become easier and more urgent.

Possibly related to:	PR 85080
With help by:		bde@

1 files changed, 2 insertions, 2 deletions

diff --git a/lib/libc/stdio/vfprintf.c b/lib/libc/stdio/vfprintf.c
index c19b069..25ac2d7 100644
--- a/lib/libc/stdio/vfprintf.c
+++ b/lib/libc/stdio/vfprintf.c
@@ -528,8 +528,8 @@ __vfprintf(FILE *fp, const char *fmt0, va_list ap)
 	static char zeroes[PADSIZE] =
 	 {'0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0'};
 
-	static const char xdigs_lower[16] = "0123456789abcdef";
-	static const char xdigs_upper[16] = "0123456789ABCDEF";
+	static const char xdigs_lower[17] = "0123456789abcdef?";
+	static const char xdigs_upper[17] = "0123456789ABCDEF?";
 
 	/*
 	 * BEWARE, these `goto error' on error, and PAD uses `n'.