Username: Password:

Ticket ID: SIXXS #6758690
Ticket Status: Invalid

PoP: gblon02 - Goscomb Technologies (London)

glonb02 dropped 2 tunnels
[au] Maurice Castro on Tuesday, 03 April 2012 08:18:20
Both my tunnels to the sixes ipv6 network which go via glonb02 have dropped. NIC Handle MCJ9-SIXXS T39369gblon02 - Goscomb Technologiesheartbeat 2a01:348:6:39f::2 T22965gblon02 - Goscomb Technologies118.107.51.2402a01:348:6:25e::2 The two systems run different configurations. The first uses aiccu and heartbeat from a freebsd box and the second runs m0n0wall in a static configuration. The problem appears to be a failure of glonb02 to create tunnels and pass ip6 packets. According to aiccu when run on the heartbeat tunnel: # /usr/local/sbin/sixxs-aiccu test /usr/local/etc/aiccu.conf ifconfig: SIOCIFCREATE2: File exists route: writing to routing socket: File exists add net default: gateway 2a01:348:6:39f::1: route already in table ####### ####### AICCU Quick Connectivity Test ####### ####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (110.232.242.134) ### This should return so called 'echo replies' ### If it doesn't then check your firewall settings ### Your local endpoint should always be pingable ### It could also indicate problems with your IPv4 stack PING 110.232.242.134 (110.232.242.134): 56 data bytes 64 bytes from 110.232.242.134: icmp_seq=0 ttl=64 time=1.231 ms 121 bytes from gblon02.sixxs.net (77.75.104.126): Destination Port Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 7100 0d99 0 0000 27 11 6eab 110.232.242.134 77.75.104.126 UDP: from port 60934, to port 3740 (decimal) 64 bytes from 110.232.242.134: icmp_seq=1 ttl=64 time=1.087 ms 124 bytes from gblon02.sixxs.net (77.75.104.126): Destination Protocol Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 7400 0dc5 0 0000 05 29 9064 110.232.242.134 77.75.104.126 64 bytes from 110.232.242.134: icmp_seq=2 ttl=64 time=0.856 ms --- 110.232.242.134 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.856/1.058/1.231/0.154 ms ###### Did this work? [Y/n] ####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (77.75.104.126) ### These pings should reach the PoP and come back to you ### In case there are problems along the route between your ### host and the PoP this could not return replies ### Check your firewall settings if problems occur PING 77.75.104.126 (77.75.104.126): 56 data bytes 64 bytes from 77.75.104.126: icmp_seq=0 ttl=50 time=351.580 ms 64 bytes from 77.75.104.126: icmp_seq=1 ttl=50 time=350.114 ms 84 bytes from gblon02.sixxs.net (77.75.104.126): Destination Protocol Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 4c00 0dfb 0 0000 05 29 9056 110.232.242.134 77.75.104.126 92 bytes from gblon02.sixxs.net (77.75.104.126): Destination Protocol Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 0dfc 0 0000 05 29 904d 110.232.242.134 77.75.104.126 64 bytes from 77.75.104.126: icmp_seq=2 ttl=50 time=358.087 ms --- 77.75.104.126 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 350.114/353.260/358.087/3.465 ms ###### Did this work? [Y/n] ####### [3/8] Traceroute to the PoP (77.75.104.126) over IPv4 ### This traceroute should reach the PoP ### In case this traceroute fails then you have no connectivity ### to the PoP and this is most probably the problem traceroute to 77.75.104.126 (77.75.104.126), 64 hops max, 40 byte packets 1 loop0.core2.smithst.dcsi.net.au (203.27.231.24) 38.093 ms 46.859 ms 39.998 ms 2 gi0-1-200.core1.smithst.dcsi.net.au (203.27.231.17) 25.852 ms 63.611 ms 38.729 ms 3 gi4-0-2.core1.portmel.aussiebb.net (202.142.143.145) 43.055 ms 53.602 ms 22.528 ms 4 gi5-1.bdr1.portmel.aussiebb.net (180.150.0.149) 37.276 ms 45.148 ms 24.841 ms 5 GigabitEthernet2-11.lon55.Melbourne.telstra.net (165.228.138.149) 44.830 ms * 48.035 ms 6 TenGigabitEthernet0-12-0-2.exi-core1.Melbourne.telstra.net (203.50.80.1) 57.939 ms 60.152 ms 49.691 ms 7 Bundle-Ether12.chw-core2.Sydney.telstra.net (203.50.11.74) 59.833 ms 71.081 ms 59.712 ms 8 Bundle-Ether4.ken-core4.Sydney.telstra.net (203.50.6.41) 64.893 ms 60.177 ms 59.808 ms 9 Bundle-Ether1.pad-gw2.Sydney.telstra.net (203.50.6.29) 59.827 ms 60.650 ms 59.741 ms 10 203.50.13.154 (203.50.13.154) 59.876 ms 61.811 ms 59.885 ms 11 i-0-0-2-0.tlot-core01.bx.reach.com (202.84.140.78) 209.802 ms 219.934 ms 209.811 ms 12 i-1-1.tlot03.bi.reach.com (202.84.251.234) 204.748 ms 211.311 ms 204.906 ms 13 xe-11-2-0.edge1.losangeles6.level3.net (4.68.62.9) 214.746 ms 220.607 ms 214.855 ms 14 vlan80.csw3.LosAngeles1.Level3.net (4.69.144.190) 214.664 ms vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 215.152 ms vlan60.csw1.LosAngeles1.Level3.net (4.69.144.62) 224.985 ms 15 ae-83-83.ebr3.LosAngeles1.Level3.net (4.69.137.41) 229.857 ms ae-73-73.ebr3.LosAngeles1.Level3.net (4.69.137.37) 218.734 ms ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) 240.022 ms 16 ae-3-3.ebr1.SanJose1.Level3.net (4.69.132.9) 249.478 ms 219.749 ms 215.462 ms 17 ae-2-2.ebr2.NewYork1.Level3.net (4.69.135.186) 284.734 ms 285.790 ms 284.861 ms 18 ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34) 284.751 ms ae-82-82.csw3.NewYork1.Level3.net (4.69.148.42) 301.001 ms ae-92-92.csw4.NewYork1.Level3.net (4.69.148.46) 287.889 ms 19 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 296.098 ms 286.034 ms ae-81-81.ebr1.NewYork1.Level3.net (4.69.134.73) 289.796 ms 20 ae-43-43.ebr2.London1.Level3.net (4.69.137.73) 370.853 ms ae-44-44.ebr2.London1.Level3.net (4.69.137.77) 370.964 ms ae-43-43.ebr2.London1.Level3.net (4.69.137.73) 371.546 ms 21 ae-59-224.csw2.London1.Level3.net (4.69.153.142) 348.485 ms ae-58-223.csw2.London1.Level3.net (4.69.153.138) 360.713 ms ae-56-221.csw2.London1.Level3.net (4.69.153.130) 350.772 ms 22 ae-2-52.edge4.London1.Level3.net (4.69.139.106) 360.885 ms * * 23 195.50.122.242 (195.50.122.242) 420.839 ms 352.752 ms 359.761 ms 24 xe-0-0-1.rt0.the.uk.goscomb.net (46.17.60.33) 359.934 ms 360.695 ms 364.638 ms 25 xe-0-1-0.cs0.thw.uk.goscomb.net (93.89.91.14) 349.807 ms 355.839 ms 346.844 ms 26 gblon02.sixxs.net (77.75.104.126) 367.799 ms 358.697 ms 372.674 ms ###### Did this work? [Y/n] ###### [4/8] Checking if we can ping IPv6 localhost (::1) ### This confirms if your IPv6 is working ### If ::1 doesn't reply then something is wrong with your IPv6 stack PING6(56=40+8+8 bytes) ::1 --> ::1 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=0 hlim=64 dst=::1%4 time=3.631 ms 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=1 hlim=64 dst=::1%4 time=2.448 ms 16 bytes from ::1: Echo Request 16 bytes from ::1, icmp_seq=2 hlim=64 dst=::1%4 time=1.289 ms --- ::1 ping6 statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 1.289/2.456/3.631/0.956 ms ###### Did this work? [Y/n] ###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2a01:348:6:39f::2) ### This confirms that your tunnel is configured ### If it doesn't reply then check your interface and routing tables PING6(56=40+8+8 bytes) 2a01:348:6:39f::2 --> 2a01:348:6:39f::2 16 bytes from 2a01:348:6:39f::2: Echo Request 16 bytes from 2a01:348:6:39f::2, icmp_seq=0 hlim=64 dst=2a01:348:6:39f::2%7 time=3.533 ms 16 bytes from 2a01:348:6:39f::2: Echo Request 16 bytes from 2a01:348:6:39f::2, icmp_seq=1 hlim=64 dst=2a01:348:6:39f::2%7 time=2.322 ms 56 bytes from fe80::260:b0ff:feb5:a53a%pcn0: Router Advertisement 16 bytes from 2a01:348:6:39f::2: Echo Request 16 bytes from 2a01:348:6:39f::2, icmp_seq=2 hlim=64 dst=2a01:348:6:39f::2%7 time=2.277 ms --- 2a01:348:6:39f::2 ping6 statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 2.277/2.711/3.533/0.582 ms ###### Did this work? [Y/n] ###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2a01:348:6:39f::1) ### This confirms the reachability of the other side of the tunnel ### If it doesn't reply then check your interface and routing tables ### Don't forget to check your firewall of course ### If the previous test was succesful then this could be both ### a firewalling and a routing/interface problem PING6(56=40+8+8 bytes) 2a01:348:6:39f::2 --> 2a01:348:6:39f::1 --- 2a01:348:6:39f::1 ping6 statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss ###### Did this work? [Y/n] HE looking glass shows glomb02 end of tunnel not present core1.fmt1.he.net> ping ipv6 2a01:348:6:39f::1 numeric count 5 Sending 5, 16-byte ICMPv6 Echo to 2a01:348:6:39f::1 timeout 5000 msec, Hop Limit 64 Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. No reply from remote host.# Entry cached for another 35 seconds. core1.fmt1.he.net> ping ipv6 2a01:348:6:25e::1 numeric count 5 Sending 5, 16-byte ICMPv6 Echo to 2a01:348:6:25e::1 timeout 5000 msec, Hop Limit 64 Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. No reply from remote host.# Entry cached for another 34 seconds. Additional information HE Traceroute core1.fmt1.he.net> traceroute ipv6 2a01:348:6:39f::1 Tracing the route to IPv6 node 2a01:348:6:39f::1 from 1 to 30 hops 1 82 ms 1 ms <1 ms 2001:470:0:2f::2 2 55 ms 38 ms 27 ms 2001:470:0:1b4::2 3 54 ms 83 ms 65 ms 2001:470:0:1af::1 4 71 ms 74 ms 92 ms 2001:470:0:1c6::1 5 151 ms 152 ms 153 ms 2001:470:0:128::2 6 150 ms 142 ms 149 ms 2001:7f8:4::999e:1 7 150 ms 153 ms 157 ms 2a01:348::49:1:1 8 138 ms * * 2a01:348::27:1:1 9 149 ms 164 ms 162 ms 2a01:348:0:4:0:3:1:1 10 147 ms 155 ms 144 ms 2a01:348:0:4:0:3:0:1 11 201 ms 147 ms 149 ms 2a01:348:0:4:0:3:1:1 12 152 ms * 164 ms 2a01:348:0:4:0:3:0:1 13 161 ms 149 ms 161 ms 2a01:348:0:4:0:3:1:1 14 149 ms 139 ms 153 ms 2a01:348:0:4:0:3:0:1 15 145 ms 150 ms 150 ms 2a01:348:0:4:0:3:1:1 16 155 ms 182 ms 161 ms 2a01:348:0:4:0:3:0:1 17 149 ms 149 ms 149 ms 2a01:348:0:4:0:3:1:1 18 212 ms * 361 ms 2a01:348:0:4:0:3:0:1 19 157 ms 156 ms 149 ms 2a01:348:0:4:0:3:1:1 20 151 ms 152 ms * 2a01:348:0:4:0:3:0:1 21 214 ms 159 ms 150 ms 2a01:348:0:4:0:3:1:1 22 151 ms 148 ms * 2a01:348:0:4:0:3:0:1 23 148 ms 186 ms 143 ms 2a01:348:0:4:0:3:1:1 24 145 ms 151 ms 151 ms 2a01:348:0:4:0:3:0:1 25 146 ms 151 ms 153 ms 2a01:348:0:4:0:3:1:1 26 146 ms 149 ms 149 ms 2a01:348:0:4:0:3:0:1 27 149 ms 149 ms 150 ms 2a01:348:0:4:0:3:1:1 28 158 ms 141 ms 174 ms 2a01:348:0:4:0:3:0:1 29 167 ms 157 ms 149 ms 2a01:348:0:4:0:3:1:1 30 227 ms 172 ms 194 ms 2a01:348:0:4:0:3:0:1 IP: Errno(8) Trace Route Failed, no response from target node.# Entry cached for another 33 seconds. core1.fmt1.he.net> traceroute ipv6 2a01:348:6:25e::1 Tracing the route to IPv6 node 2a01:348:6:25e::1 from 1 to 30 hops 1 4 ms <1 ms <1 ms 2001:470:0:2f::2 2 27 ms 53 ms 40 ms 2001:470:0:1b4::2 3 65 ms 68 ms 126 ms 2001:470:0:1af::1 4 69 ms 70 ms 74 ms 2001:470:0:1c6::1 5 149 ms 149 ms 149 ms 2001:470:0:128::2 6 150 ms 200 ms 148 ms 2001:7f8:4::999e:1 7 156 ms 146 ms 149 ms 2a01:348::49:1:1 8 * 151 ms 163 ms 2a01:348::27:1:1 9 198 ms * 181 ms 2a01:348:0:4:0:3:1:1 10 147 ms 149 ms 148 ms 2a01:348:0:4:0:3:0:1 11 149 ms 156 ms 146 ms 2a01:348:0:4:0:3:1:1 12 147 ms * 150 ms 2a01:348:0:4:0:3:0:1 13 161 ms 138 ms 166 ms 2a01:348:0:4:0:3:1:1 14 158 ms 180 ms 160 ms 2a01:348:0:4:0:3:0:1 15 159 ms 149 ms * 2a01:348:0:4:0:3:1:1 16 152 ms 162 ms 173 ms 2a01:348:0:4:0:3:0:1 17 146 ms 142 ms 149 ms 2a01:348:0:4:0:3:1:1 18 150 ms 149 ms 149 ms 2a01:348:0:4:0:3:0:1 19 149 ms 150 ms 150 ms 2a01:348:0:4:0:3:1:1 20 * 151 ms 166 ms 2a01:348:0:4:0:3:0:1 21 158 ms 149 ms 152 ms 2a01:348:0:4:0:3:1:1 22 151 ms 145 ms * 2a01:348:0:4:0:3:0:1 23 * 179 ms 149 ms 2a01:348:0:4:0:3:1:1 24 149 ms 150 ms 149 ms 2a01:348:0:4:0:3:0:1 25 228 ms * 204 ms 2a01:348:0:4:0:3:1:1 26 160 ms 140 ms * 2a01:348:0:4:0:3:0:1 27 166 ms 161 ms 151 ms 2a01:348:0:4:0:3:1:1 28 153 ms 148 ms 146 ms 2a01:348:0:4:0:3:0:1 29 149 ms 149 ms 151 ms 2a01:348:0:4:0:3:1:1 30 148 ms 150 ms * 2a01:348:0:4:0:3:0:1 IP: Errno(8) Trace Route Failed, no response from target node.# Entry cached for another 28 seconds. Both trace routes appear to be looping at glomb02
State change: invalid
[ch] Jeroen Massar SixXS Staff on Tuesday, 03 April 2012 10:13:47
Message is Locked
The state of this ticket has been changed to invalid
glonb02 dropped 2 tunnels
[ch] Jeroen Massar SixXS Staff on Tuesday, 03 April 2012 10:19:56
# /usr/local/sbin/sixxs-aiccu test /usr/local/etc/aiccu.conf
ifconfig: SIOCIFCREATE2: File exists
route: writing to routing socket: File exists
add net default: gateway 2a01:348:6:39f::1: route already in table
One cannot run aiccu-test while another aiccu instance is already running.
####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (110.232.242.134)
...
PING 110.232.242.134 (110.232.242.134): 56 data bytes
64 bytes from 110.232.242.134: icmp_seq=0 ttl=64 time=1.231 ms 121 bytes from gblon02.sixxs.net (77.75.104.126): Destination Port Unreachable You have a ping towards your local IP address and get responses back from a remote host? I suggest you check your system as that is very odd.
###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2a01:348:6:39f::2)
...
PING6(56=40+8+8 bytes) 2a01:348:6:39f::2 --> 2a01:348:6:39f::2
...
56 bytes from fe80::260:b0ff:feb5:a53a%pcn0: Router Advertisement
Why do you see router advertisements on a different interface (pcn0) when pinging a local address on a tunnel (though the kernel likely handles it as loopback)? Can you show the details of how the tunnel is configured? Thus the output of (g)ifconfig (depending on FreeBSD version), routing tables etc?
glonb02 dropped 2 tunnels
[au] Maurice Castro on Tuesday, 03 April 2012 17:36:59
Problem resolved without intervention at my end Failure detected at 15:26 EST Australian time Service restored 18:13 EST Australian time Both the aiccu tunnel and the static tunnel returned to operation. For future reference: The file exists error is probably due to a terminated previous instance of aiccu; if I remember correctly the gif interface was probably not torn down and the routes remained. The second issue (the system is odd) is due to the config: the router is connected by PPP over Ethernet to a bridged WiMax service for its IPv4 connection (pcn0). I also run an IPv4 IPSec link between the 2 sites. I can provide further details in the morning. Please be aware that this service has been operating successfully without modification for many months.
glonb02 dropped 2 tunnels
[au] Maurice Castro on Wednesday, 04 April 2012 01:23:32
pcn0 is the inside interface rl0 is the outside interface
ifconfig -a
pcn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 00:60:b0:b5:a5:3a inet6 fe80::260:b0ff:feb5:a53a%pcn0 prefixlen 64 scopeid 0x1 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 inet6 2a01:348:245::1 prefixlen 64 media: Ethernet autoselect (100baseTX <full-duplex>) status: active rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 00:18:4d:eb:92:07 inet6 fe80::218:4dff:feeb:9207%rl0 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (100baseTX <full-duplex>) status: active plip0: flags=108851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 gre0: flags=9011<UP,POINTOPOINT,LINK0,MULTICAST> metric 0 mtu 1476 tunnel inet 203.30.68.103 --> 118.107.51.240 inet6 fe80::260:b0ff:feb5:a53a%gre0 prefixlen 64 scopeid 0x5 inet 10.0.0.2 --> 10.100.0.1 netmask 0xffffff00 tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492 inet 110.232.242.134 --> 203.27.231.24 netmask 0xffffffff Opened by PID 53865 gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 tunnel inet 110.232.242.134 --> 77.75.104.126 inet6 fe80::260:b0ff:feb5:a53a%gif0 prefixlen 64 scopeid 0x7 inet6 2a01:348:6:39f::2 --> 2a01:348:6:39f::1 prefixlen 128

Please note Posting is only allowed when you are logged in.

Not logged in.
©2001-2014 SixXS - IPv6 Deployment & Tunnel Broker