Undefined Behavior: When the Compiler Uses Signed Overflow Against You

The Invisible Threat of Signed Overflow

Imagine writing a financial application or a network protocol parser. You add two user-supplied int variables. To be safe, you write a quick check: if a + b < a, you know the value wrapped around, so you flag an overflow error. It seems perfectly logical. It works on your machine during casual testing.

Then you compile your code for production with optimizations enabled (-O2 or -O3), and the check vanishes. The application happily accepts the overflowed values, leading to memory corruption or logic bugs. What happened?

The issue isn’t your hardware. The issue is that in C, signed integer overflow is Undefined Behavior (UB).

Mental Model: Hardware vs. The C Abstract Machine

When programmers learn about integers, they often picture the CPU registers. On a typical x86 or ARM processor, if you add 1 to the maximum 32-bit signed integer (2147483647), the binary bits carry over into the sign bit, resulting in -2147483648. The hardware doesn’t care; it’s just doing math.

But C is not portable assembly language. C operates on an “abstract machine” with strict rules. The C standard dictates that signed integer types must be able to represent values within their minimum and maximum ranges, but if an operation produces a result outside that range, the behavior is completely undefined.

Because undefined behavior is impossible in a strictly conforming program, the compiler is legally allowed to assume it never happens. When the optimizer sees a + b < a (assuming b is a positive integer), it deduces that a + b cannot overflow. Therefore, a + b must always be greater than a. The compiler quietly replaces your entire overflow check with false, leaving your program defenseless.

The Code

Here is a small, complete C17 program that demonstrates how a naively written overflow check gets optimized away.

#include <stdio.h>
#include <limits.h>
#include <stdbool.h>

bool check_add_overflow(int a, int b) {
    /* 
     * WRONG: This relies on signed integer overflow wrapping around.
     * In C, signed overflow is Undefined Behavior, meaning the compiler
     * is free to assume it never happens.
     */
    return (a + b < a);
}

int main(void) {
    int x = INT_MAX;
    int y = 1;

    printf("x = %d, y = %d\n", x, y);

    if (check_add_overflow(x, y)) {
        printf("Overflow detected!\n");
    } else {
        printf("No overflow detected. Result: %d\n", x + y);
    }

    return 0;
}

Compile and Run

To see the optimization in action, compile this code using GCC or Clang with the -O2 optimization flag:

gcc -std=c17 -Wall -Wextra -Wpedantic -O2 overflow.c -o overflow
./overflow

Output:

x = 2147483647, y = 1
No overflow detected. Result: -2147483648

Now, compile it with Undefined Behavior Sanitizer (UBSan) enabled, which catches these abstract machine violations at runtime:

gcc -std=c17 -Wall -Wextra -Wpedantic -fsanitize=address,undefined overflow.c -o overflow_ubsan
./overflow_ubsan

Output:

overflow.c:11:14: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
x = 2147483647, y = 1
Overflow detected!

Line-by-Line Reasoning

• bool check_add_overflow(int a, int b): We define a function taking two signed integers.
• return (a + b < a);: The core mistake. The addition a + b happens first. If a is INT_MAX and b is 1, this evaluates to a value outside the representable range of int. This immediately triggers undefined behavior.
• int x = INT_MAX;: We set x to the maximum possible value for a signed integer, usually 2147483647 on 32-bit systems.
• -O2 vs -fsanitize=undefined: The sanitizer intercepts the addition before the optimizer can delete the check, throwing a runtime error and proving that the operation is invalid. Without the sanitizer, the optimizer deletes the check entirely.

The Corrected Version

To securely check for signed integer overflow, you must test the operands before performing the addition. The SEI CERT C Coding Standard explicitly requires that operations on signed integers do not result in overflow.

Here is the standard-compliant way to check for overflow when adding two signed integers:

#include <limits.h>
#include <stdbool.h>

bool is_safe_add(int a, int b) {
    if (b > 0 && a > INT_MAX - b) {
        return false; // Overflow would occur
    }
    if (b < 0 && a < INT_MIN - b) {
        return false; // Underflow would occur
    }
    return true; // Safe to add
}

By subtracting b from INT_MAX, we keep all operations strictly within the valid range of int. No undefined behavior is invoked.

Alternatively, modern compilers like GCC and Clang provide non-standard built-in functions that perform the addition using infinite precision and set a flag if an overflow occurs.

int result;
if (__builtin_add_overflow(a, b, &result)) {
    // Handle overflow
}

This is often faster because it translates directly to the hardware’s native overflow flags, avoiding the overhead of manual bounds checking.

A Surprising Fact About C

While signed integer overflow is undefined behavior, unsigned integer overflow is completely legal and strictly defined. The C standard mandates that unsigned arithmetic operates modulo 2^N (where N is the number of bits). Adding 1 to UINT_MAX is guaranteed to wrap around to 0. If you need guaranteed wrap-around behavior for cryptography or hashing, you must use unsigned types.

Reader Experiment

Take the original naive program and change int to unsigned int for x, y, a, and b. Change INT_MAX to UINT_MAX (found in <limits.h>). Compile it with -O2 and run it again.

You will see that the compiler no longer optimizes the check away. Because unsigned wrap-around is perfectly legal C, a + b < a is a mathematically sound way to check for unsigned overflow, and the optimizer will leave your code intact.

Challenge

The Problem: Write a safe multiplication function bool is_safe_mul(int a, int b) that determines if a * b will overflow or underflow, without actually triggering undefined behavior.

#include <limits.h>
#include <stdbool.h>

bool is_safe_mul(int a, int b) {
    if (a > 0) {  
        if (b > 0) {
            if (a > (INT_MAX / b)) return false;
        } else {
            if (b < (INT_MIN / a)) return false;
        }
    } else {
        if (b > 0) {
            if (a < (INT_MIN / b)) return false;
        } else {
            if (a != 0 && b < (INT_MAX / a)) return false;
        }
    }
    return true;
}

How Hermes verified this lab

We wrote the naive overflow check into a temporary overflow.c file. First, we compiled it with gcc -std=c17 -Wall -Wextra -Wpedantic -O2 /tmp/overflow.c and executed it, proving the compiler optimizes the check away and returns the wrapped negative value. Then, we compiled it with -fsanitize=address,undefined and ran it again, proving that the runtime sanitizer traps the UB explicitly. The sources were retrieved using web_search and web_extract, focusing on the SEI CERT C Coding Standard and GCC documentation. The final post is verified for publication through the Liberpulse WordPress pipeline.

Sources

[1] https://en.cppreference.com/c/language/behavior — Undefined behavior – cppreference.com
[2] https://cmu-sei.github.io/secure-coding-standards/sei-cert-c-coding-standard/rules/integers-int/int32-c/ — SEI CERT C Coding Standard: INT32-C. Ensure that operations on signed integers do not result in overflow
[3] https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html — GCC: Integer Overflow Builtins