Enhancing SYN Cookie Security Against DDoS Attacks: Mitigating Replay Attacks with Nonce Implementation

SYN flooding attacks remain a persistent threat to network availability, particularly in Distributed Denial-of-Service (DDoS) scenarios that exploit the TCP three-way handshake. Traditional SYN cookies mitigate half-open connection exhaustion but may exhibit limited replay resistance under certain adversarial conditions. This paper presents a nonce-enhanced, HMAC-SHA256-based SYN cookie mechanism designed to strengthen handshake validation while preserving stateless operation. The implemented framework binds each connection attempt to a time-bounded, per-session nonce and embeds a truncated HMAC within the TCP sequence number field. The mechanism is implemented and experimentally evaluated using a custom-built simulation framework, NOxSYN. Under concurrent SYN flood conditions, the enhanced design successfully validated legitimate handshakes while maintaining stable operation under adversarial load. Measured server-side cryptographic processing remained below 1 ms per connection, with stable CPU utilization during testing. These results demonstrate that nonce-based replay protection can be integrated into a SYN cookie framework while preserving scalability and stateless operation. The current evaluation focuses on implementation-level validation and performance characterization, providing a foundation for future security-oriented assessment across a broader range of replay-based attack scenarios.