Nic: JP15-6BONE Tunnel ID: T14046 Email: rtfm@cox.net This machine is functioning as the NAT router and has a public IPv4 IP. It's been online for 320 days and for all of that time this tunnel has worked with no issues. Today it stopped working. There have been no configuration changes made and no one else has access. OS: FreeBSD 7.2-RELEASE AICCU Version: AICCU 2007.01.15-console-kame by Jeroen Massar Interface info: rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 00:c0:49:a6:f8:18 inet6 fe80::2c0:49ff:fea6:f818%rl0 prefixlen 64 scopeid 0x1 inet 10.240.101.1 netmask 0xffffff00 broadcast 10.240.101.255 inet6 2001:4978:12e:a::1 prefixlen 64 media: Ethernet autoselect (100baseTX <full-duplex>) status: active dc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 00:c0:49:b4:72:21 inet6 fe80::2c0:49ff:feb4:7221%dc0 prefixlen 64 scopeid 0x2 inet6 2001:4978:12e:1::fed:1 prefixlen 128 inet 68.3.162.211 netmask 0xfffffe00 broadcast 68.3.163.255 media: Ethernet autoselect (100baseTX <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 tunnel inet 68.3.162.211 --> 216.14.98.22 inet6 fe80::2c0:49ff:fea6:f818%gif0 prefixlen 64 scopeid 0x4 inet6 2001:4978:f:82::2 --> 2001:4978:f:82::1 prefixlen 128 Public IP provided by Cox Communications. Routing table: [root@morpheus ~]# netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 68.3.162.1 UGS 0 182259008 dc0 10.240.101.0/24 link#1 UC 0 0 rl0 10.240.101.1 00:c0:49:a6:f8:18 UHLW 1 106980 lo0 10.240.101.9 20:cf:30:a7:75:a6 UHLW 1 241496 rl0 515 10.240.101.61 e0:cb:4e:a1:c3:17 UHLW 1 49378 rl0 1196 10.240.101.209 00:0c:29:b0:bd:94 UHLW 1 119 rl0 401 10.240.101.230 00:0c:29:6a:d4:b7 UHLW 1 116 rl0 1098 10.240.101.231 00:0c:29:a8:f0:6a UHLW 1 157 rl0 616 10.240.101.255 ff:ff:ff:ff:ff:ff UHLWb 1 26748 rl0 68.3.162.0/23 link#2 UC 0 0 dc0 68.3.162.1 00:1e:be:ff:26:db UHLW 2 0 dc0 1199 68.3.162.211 00:c0:49:b4:72:21 UHLW 1 1116 lo0 68.3.163.108 00:24:e8:02:f2:a7 UHLW 1 6 dc0 1098 127.0.0.1 127.0.0.1 UH 0 34861 lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 => default 2001:4978:f:82::1 UGS gif0 ::1 ::1 UHL lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 2001:4978:f:82::1 link#4 UHL gif0 2001:4978:f:82::2 link#4 UHL lo0 2001:4978:12e:1::fed:1 00:c0:49:b4:72:21 UHL lo0 2001:4978:12e:a::/64 link#1 UC rl0 2001:4978:12e:a::1 00:c0:49:a6:f8:18 UHL lo0 2001:4978:12e:a::61 e0:cb:4e:a1:c3:17 UHLW rl0 2001:4978:12e:a::231 00:0c:29:a8:f0:6a UHLW rl0 fe80::/10 ::1 UGRS lo0 fe80::%rl0/64 link#1 UC rl0 fe80::20c:29ff:fea8:f06a%rl0 00:0c:29:a8:f0:6a UHLW rl0 fe80::2c0:49ff:fea6:f818%rl0 00:c0:49:a6:f8:18 UHL lo0 fe80::c9fb:5157:7d10:895f%rl0 e0:cb:4e:a1:c3:17 UHLW rl0 fe80::%dc0/64 link#2 UC dc0 fe80::2c0:49ff:feb4:7221%dc0 00:c0:49:b4:72:21 UHL lo0 fe80::%lo0/64 fe80::1%lo0 U lo0 fe80::1%lo0 link#3 UHL lo0 fe80::%gif0/64 link#4 UC gif0 fe80::2c0:49ff:fea6:f818%gif0 link#4 UHL lo0 ff01:1::/32 link#1 UC rl0 ff01:2::/32 link#2 UC dc0 ff01:3::/32 ::1 UC lo0 ff01:4::/32 link#4 UC gif0 ff02::/16 ::1 UGRS lo0 ff02::%rl0/32 link#1 UC rl0 ff02::%dc0/32 link#2 UC dc0 ff02::%lo0/32 ::1 UC lo0 ff02::%gif0/32 link#4 UC gif0 Firewall is ipfw. I flushed all rules for a few minutes while running the test because traceroute has to be run with the -I switch to get out of firewall. AICCU Test: [root@morpheus ~]# /usr/local/sbin/sixxs-aiccu test ifconfig: SIOCIFCREATE2: File exists route: writing to routing socket: File exists add net default: gateway 2001:4978:f:82::1: route already in table ####### ####### AICCU Quick Connectivity Test ####### ####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (68.3.162.211) ### This should return so called 'echo replies' ### If it doesn't then check your firewall settings ### Your local endpoint should always be pingable ### It could also indicate problems with your IPv4 stack PING 68.3.162.211 (68.3.162.211): 56 data bytes 64 bytes from 68.3.162.211: icmp_seq=0 ttl=64 time=0.070 ms 64 bytes from 68.3.162.211: icmp_seq=1 ttl=64 time=0.052 ms 96 bytes from sixxs.cx01.chi.bb.your.org (216.14.98.22): Destination Port Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5800 fd28 0 0000 16 29 8659 68.3.162.211 216.14.98.22 64 bytes from 68.3.162.211: icmp_seq=2 ttl=64 time=0.092 ms --- 68.3.162.211 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.052/0.071/0.092/0.016 ms ###### Did this work? [Y/n] y ####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (216.14.98.22) ### These pings should reach the PoP and come back to you ### In case there are problems along the route between your ### host and the PoP this could not return replies ### Check your firewall settings if problems occur PING 216.14.98.22 (216.14.98.22): 56 data bytes 64 bytes from 216.14.98.22: icmp_seq=0 ttl=57 time=66.052 ms 64 bytes from 216.14.98.22: icmp_seq=1 ttl=57 time=64.782 ms 96 bytes from sixxs.cx01.chi.bb.your.org (216.14.98.22): Destination Port Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5800 fd44 0 0000 16 29 863d 68.3.162.211 216.14.98.22 64 bytes from 216.14.98.22: icmp_seq=2 ttl=57 time=64.016 ms --- 216.14.98.22 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 64.016/64.950/66.052/0.840 ms ###### Did this work? [Y/n] y ####### [3/8] Traceroute to the PoP (216.14.98.22) over IPv4 ### This traceroute should reach the PoP ### In case this traceroute fails then you have no connectivity ### to the PoP and this is most probably the problem traceroute to 216.14.98.22 (216.14.98.22), 64 hops max, 40 byte packets 1 10.147.240.1 (10.147.240.1) 8.789 ms 7.402 ms 7.721 ms 2 68.2.8.189 (68.2.8.189) 16.121 ms 8.450 ms 7.756 ms 3 mcdldsrj01-ae2.0.rd.ph.cox.net (70.169.76.225) 8.447 ms 10.117 ms 9.629 ms 4 langbprj01-ae0.0.rd.la.cox.net (68.1.0.232) 21.948 ms 21.882 ms 22.226 ms 5 ae0-50g.cr1.lax1.us.nlayer.net (69.31.127.141) 30.389 ms ae0-50g.cr1.lax1.us.nlayer.net (69.31.127.137) 30.491 ms ae0-50g.cr1.lax1.us.nlayer.net (69.31.127.141) 56.590 ms 6 xe-4-0-0.cr2.ord1.us.nlayer.net (69.22.142.255) 63.843 ms 61.598 ms 61.665 ms 7 ae2-30g.ar1.ord1.us.nlayer.net (69.31.111.138) 89.827 ms 75.874 ms 92.161 ms 8 as19255.ge-0-0-42-106.ar1.ord1.us.nlayer.net (69.31.111.22) 64.135 ms as19255.ge-0-0-42-107.ar1.ord1.us.nlayer.net (69.31.111.26) 64.605 ms as19255.ge-0-0-42-106.ar1.ord1.us.nlayer.net (69.31.111.22) 65.166 ms 9 sixxs.cx01.chi.bb.your.org (216.14.98.22) 65.225 ms 64.007 ms 63.761 ms ###### Did this work? [Y/n] y ###### [4/8] Checking if we can ping IPv6 localhost (::1) ### This confirms if your IPv6 is working ### If ::1 doesn't reply then something is wrong with your IPv6 stack PING6(56=40+8+8 bytes) ::1 --> ::1 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=0 hlim=64 dst=::1%3 time=0.703 ms 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=1 hlim=64 dst=::1%3 time=0.584 ms 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=2 hlim=64 dst=::1%3 time=0.298 ms --- ::1 ping6 statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.298/0.528/0.703/0.170 ms ###### Did this work? [Y/n] y ###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:4978:f:82::2) ### This confirms that your tunnel is configured ### If it doesn't reply then check your interface and routing tables PING6(56=40+8+8 bytes) 2001:4978:f:82::2 --> 2001:4978:f:82::2 16 bytes from 2001:4978:f:82::2: Echo Request 16 bytes from 2001:4978:f:82::2, icmp_seq=0 hlim=64 dst=2001:4978:f:82::2%4 time=0.741 ms 16 bytes from 2001:4978:f:82::2: Echo Request 16 bytes from 2001:4978:f:82::2, icmp_seq=1 hlim=64 dst=2001:4978:f:82::2%4 time=0.312 ms 16 bytes from 2001:4978:f:82::2: Echo Request 16 bytes from 2001:4978:f:82::2, icmp_seq=2 hlim=64 dst=2001:4978:f:82::2%4 time=0.310 ms --- 2001:4978:f:82::2 ping6 statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.310/0.454/0.741/0.203 ms ###### Did this work? [Y/n] y ###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:4978:f:82::1) ### This confirms the reachability of the other side of the tunnel ### If it doesn't reply then check your interface and routing tables ### Don't forget to check your firewall of course ### If the previous test was succesful then this could be both ### a firewalling and a routing/interface problem PING6(56=40+8+8 bytes) 2001:4978:f:82::2 --> 2001:4978:f:82::1 --- 2001:4978:f:82::1 ping6 statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss ###### Did this work? [Y/n] n