binutils-gdb/gdb/unittests
Joel Brobecker 4fbb7ccebe Fix stack smashing error during gdb_mpq_write_fixed_point selftest
When building GDB using Ubuntu 20.04's system libgmp and compiler,
running the "maintenance selftest" command triggers the following error:

    | Running selftest gdb_mpq_write_fixed_point.
    | *** stack smashing detected ***: terminated
    | [1]    1092790 abort (core dumped)  ./gdb gdb

This happens while trying to construct an mpq_t object (a rational)
from two integers representing the numerator and denominator.
In our test, the numerator is -8, and the denominator is 1.
The problem was that the rational was constructed using the wrong
function. This is what we were doing prior to this patch:

    mpq_set_ui (v.val, numerator, denominator);

The 'u' in "ui" stands for *unsigned*, which is wrong because
numerator and denominator's type is "int".

As a result of the above, instead of getting a rational value of -8,
we get a rational with a very large positive value (gmp_printf
says "18446744073709551608").

From there, the test performs an operation which is expected to
write this value into a buffer which was not dimensioned to fit
such a number, thus leading GMP into a buffer overflow.
This was verified by applying the formula that GMP's documentation
gives for the required memory buffer size needed during export:

    | When an application is allocating space itself the required size can
    | be determined with a calculation like the following. Since
    | mpz_sizeinbase always returns at least 1, count here will be at
    | least one, which avoids any portability problems with malloc(0),
    | though if z is zero no space at all is actually needed (or written).
    |
    |     numb = 8*size - nail;
    |     count = (mpz_sizeinbase (z, 2) + numb-1) / numb;
    |     p = malloc (count * size);

With the very large number, mpz_sizeinbase returns 66 and thus
the malloc size becomes 16 bytes instead of the 8 we allocated.

This patch fixes the issue by using the correct "set" function.

gdb/ChangeLog:

        * unittests/gmp-utils-selftests.c (write_fp_test): Use mpq_set_si
        instead of mpq_set_ui to initialize our GMP rational.
2020-11-24 06:34:57 +04:00
..
basic_string_view
optional
array-view-selftests.c
child-path-selftests.c
cli-utils-selftests.c
command-def-selftests.c
common-utils-selftests.c
copy_bitwise-selftests.c gdb, gdbserver, gdbsupport: fix leading space vs tabs issues 2020-11-02 10:28:45 -05:00
enum-flags-selftests.c Tweak gdbsupport/valid-expr.h for GCC 6, fix build 2020-09-29 23:48:04 +01:00
environ-selftests.c
filtered_iterator-selftests.c
format_pieces-selftests.c
function-view-selftests.c
gmp-utils-selftests.c Fix stack smashing error during gdb_mpq_write_fixed_point selftest 2020-11-24 06:34:57 +04:00
lookup_name_info-selftests.c
main-thread-selftests.c
memory-map-selftests.c gdb, gdbserver, gdbsupport: fix leading space vs tabs issues 2020-11-02 10:28:45 -05:00
memrange-selftests.c
mkdir-recursive-selftests.c
observable-selftests.c
offset-type-selftests.c
optional-selftests.c
parse-connection-spec-selftests.c
ptid-selftests.c
rsp-low-selftests.c
scoped_fd-selftests.c
scoped_mmap-selftests.c
scoped_restore-selftests.c
search-memory-selftests.c Add simple_search_memory unit tests 2020-10-07 12:07:56 -06:00
string_view-selftests.c
style-selftests.c
tracepoint-selftests.c
tui-selftests.c Use a curses pad for source and disassembly windows 2020-09-27 20:30:32 -06:00
unpack-selftests.c
utils-selftests.c
vec-utils-selftests.c
xml-utils-selftests.c