JUNIOR TAITT

IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error

or

%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

The above or similar error message is caused by when a cisco device detects and mitigates an IPsec replay network attack [RFC6479].

“IP Authentication Header” [RFC4302] and “IP Encapsulating Security Payload (ESP)” [RFC4303] define an anti-replay service that employs a sliding window mechanism. The mechanism, when enabled by a receiver, uses an anti-replay window of size W. This window limits how far out of order a packet can be, relative to the packet with the highest sequence number that has been authenticated so far.

To make a long story short, if IPsec receives packets to be sent too late, it drops the packet because it is possible the packet was captured by an malicious person, modified and sent (A.K.A replayed). See RFC6479 for more details.

Our team encountered this issue recently and found it quite vexing and challenging to resolve. We were attempting to promote a server to a domain controller but it failed at various points, hung during the wizard, didn’t find a list of domains, being very slow going between pages after clicking next, etc.

The steps to resolve where:

1. Swapped GNS3 routers terminating the VPN to physical routers terminating the VPN
2. Swapped from L2L FlexVPN to DMVPN
3. Naturally out of desperation, we changed multiple VPN parameters. This is how we work in IT, smash buttons until it works, don’t hate us 🙂
4. Reduce MTU and MSS on routed devices, on cisco routers along the path the following was applied IP mtu 1300 and tcp mss 1260. Issue not resolved.
5. Increase the window size from 64 to 1064. crypto ipsec security-association replay window-size 1024. Issue not resolved
6. Disable the Anti-Replay check (NOT recommended) crypto ipsec security-association replay disable
7. Used VMware workstation instead of Hyper-V

None of these solutions worked but we did notice a change in wireshark.

With anti-replay disabled – Packets were arriving out of order e.g.

With anti-replay enabled by default – Packets were being lost and sent a lot of retransmissions. e.g.

Cisco’s documentation hinted towards potential QoS queuing causing the issue and other internet sources motioned lower MTU along the path, but we already tried changing the MTU.

Either way, after lengthy tests and verifying router and router VPN configuration, were certain the issue was caused by either the server or the the hypervisor (Hyper-V).

This led us to change the mtu on the windows servers:

1. View existing MTU
2. Set new MTU to 1300

Issue solved, no more replay drops and the server was able to promoted to a domain controller across the VPN