IPSec MSS / MTU problems - solution
Shadow Hawkins on Sunday, 25 November 2007 06:01:13
I just wanted to drop a line here with the solution to a sticky problem I ran into recently. I'm using IPSec tunnel mode (not transport) to create a VPN between two IPv6 subnets. The SixXS endpoints create a tunnel between themselves and all traffic from one subnet to the other is encrypted and authenticated. I noticed that some commands, like 'top' would cause SSH sessions over this tunnel to freeze. With IPSec disabled, they worked. I suspected an MTU issue. I was right. For some reason, the ICMPv6 "Message too big" packets were getting lost along the way and/or stuff was getting dropped by IPSec. So the nodes would send out big packets (they're unaware of the IPSec overhead), and things would get messed up. The solution is this: on both firewalls, use the following ip6tables command (if you're running Linux 2.6.21 or later with ip6tables 1.3.8 or newer, use pf or equiv for bsd and hopefully windows already does this): ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu . If you're using tunnel mode ON an endpoint, use POSTROUTING instead of FORWARD. This will cause the firewall to restrict packets to the path MTU, solving the problem even where ICMPv6 doesn't get through the SixXS network (or is dropped b/c of IPSec policies, I don't know which it is). Hope this is a help to somebody searching for answers.
Posting is only allowed when you are logged in.