Tickets - vnhan01 external connectivity massive latency :: SixXS

Ticket ID: SIXXS #10249278
Ticket Status: User

PoP: vnhan01 - NetNam Corporation (Hanoi)

vnhan01 external connectivity massive latency

Shadow Hawkins on Monday, 07 October 2013 14:52:55

I have a heartbeat tunnel connected to vnhan01. I was using AYIYA up until last week, when I reconfigured my router to no longer be behind another NAT device, so protocol-41 became possible. Attempting to ping either IPv4 to 119.17.194.232 (vnhan01's IPv4 tunnel address) or the other end of my tunnel produces odd results: I will eventually get some of my ping replies, and they tend to come through in batches. For example, here's an IPv4 ping from my router:

root@tamachan:~# ping 119.17.194.232 PING 119.17.194.232 (119.17.194.232): 56 data bytes 64 bytes from 119.17.194.232: seq=0 ttl=58 time=11019.971 ms 64 bytes from 119.17.194.232: seq=1 ttl=58 time=10010.870 ms 64 bytes from 119.17.194.232: seq=2 ttl=58 time=8995.667 ms 64 bytes from 119.17.194.232: seq=3 ttl=58 time=7990.163 ms 64 bytes from 119.17.194.232: seq=4 ttl=58 time=6974.631 ms 64 bytes from 119.17.194.232: seq=5 ttl=58 time=5968.927 ms 64 bytes from 119.17.194.232: seq=6 ttl=58 time=4963.252 ms 64 bytes from 119.17.194.232: seq=7 ttl=58 time=3945.743 ms 64 bytes from 119.17.194.232: seq=8 ttl=58 time=2939.927 ms 64 bytes from 119.17.194.232: seq=9 ttl=58 time=1924.595 ms 64 bytes from 119.17.194.232: seq=10 ttl=58 time=918.959 ms 64 bytes from 119.17.194.232: seq=11 ttl=58 time=6480.918 ms 64 bytes from 119.17.194.232: seq=12 ttl=58 time=5463.352 ms 64 bytes from 119.17.194.232: seq=13 ttl=58 time=4457.537 ms 64 bytes from 119.17.194.232: seq=14 ttl=58 time=3451.758 ms 64 bytes from 119.17.194.232: seq=15 ttl=58 time=2436.166 ms 64 bytes from 119.17.194.232: seq=16 ttl=58 time=1430.748 ms 64 bytes from 119.17.194.232: seq=17 ttl=58 time=415.172 ms 64 bytes from 119.17.194.232: seq=18 ttl=58 time=6.050 ms

And an IPv6 attempt:

root@tamachan:~# ping6 2401:e800:100:TUNNELID::1 PING 2401:e800:100:TUNNELID::1 (2401:e800:100:TUNNELID::1): 56 data bytes 64 bytes from 2401:e800:100:TUNNELID::1: seq=7 ttl=64 time=86737.545 ms ^C --- 2401:e800:100:TUNNELID::1 ping statistics --- 176 packets transmitted, 1 packets received, 99% packet loss round-trip min/avg/max = 86737.545/86737.545/86737.545 ms

A tcpdump I had running during the above test tells me I hit Control-C too soon, and that 23 seconds after my last ping outwards, replies 8 through 124 arrived in the space of 20ms. Similarly attempting to ping6 www.google.com.vn:

root@tamachan:~# ping6 www.google.com.vn PING www.google.com.vn (2404:6800:4005:802::1017): 56 data bytes 64 bytes from 2404:6800:4005:802::1017: seq=0 ttl=55 time=23071.247 ms 64 bytes from 2404:6800:4005:802::1017: seq=1 ttl=55 time=22066.994 ms 64 bytes from 2404:6800:4005:802::1017: seq=2 ttl=55 time=21051.415 ms 64 bytes from 2404:6800:4005:802::1017: seq=3 ttl=55 time=20046.059 ms 64 bytes from 2404:6800:4005:802::1017: seq=6 ttl=55 time=17012.457 ms 64 bytes from 2404:6800:4005:802::1017: seq=5 ttl=55 time=18025.163 ms 64 bytes from 2404:6800:4005:802::1017: seq=4 ttl=55 time=19031.581 ms 64 bytes from 2404:6800:4005:802::1017: seq=7 ttl=55 time=16007.312 ms 64 bytes from 2404:6800:4005:802::1017: seq=8 ttl=55 time=14991.250 ms 64 bytes from 2404:6800:4005:802::1017: seq=9 ttl=55 time=13987.477 ms 64 bytes from 2404:6800:4005:802::1017: seq=10 ttl=55 time=12971.917 ms 64 bytes from 2404:6800:4005:802::1017: seq=12 ttl=55 time=10908.421 ms 64 bytes from 2404:6800:4005:802::1017: seq=11 ttl=55 time=11914.909 ms 64 bytes from 2404:6800:4005:802::1017: seq=13 ttl=55 time=9893.462 ms 64 bytes from 2404:6800:4005:802::1017: seq=14 ttl=55 time=8886.742 ms 64 bytes from 2404:6800:4005:802::1017: seq=16 ttl=55 time=6865.322 ms 64 bytes from 2404:6800:4005:802::1017: seq=15 ttl=55 time=7871.790 ms 64 bytes from 2404:6800:4005:802::1017: seq=17 ttl=55 time=5860.685 ms 64 bytes from 2404:6800:4005:802::1017: seq=18 ttl=55 time=4847.318 ms 64 bytes from 2404:6800:4005:802::1017: seq=19 ttl=55 time=3835.465 ms 64 bytes from 2404:6800:4005:802::1017: seq=22 ttl=55 time=810.144 ms 64 bytes from 2404:6800:4005:802::1017: seq=23 ttl=55 time=26.971 ms 64 bytes from 2404:6800:4005:802::1017: seq=24 ttl=55 time=29.489 ms ^C --- www.google.com.vn ping statistics --- 99 packets transmitted, 23 packets received, 76% packet loss round-trip min/avg/max = 26.971/11739.721/23071.247 ms

That said, the same tcpdump I mentioned earlier can see the ping _from_ vnhan01 checking my status every few minutes and replies correctly. Attempting to access virror.hanoilug.org(2401:e800:0:1::21) via port 80 comes through eventually, but with lots of latency. If I connect with my web browser, their little ipv6-test.com widget in the corner sometimes comes up with my IP address, and some times does not. Attempting to access ipv6-test.com(2001:41d0:8:e8ad::1) via port 80 similarly comes through eventually if I use netcat, but my web browser is unwilling to wait long enough for it to complete, and fails the test. A quick look at tracepath suggests it's not a problem between here and there:

root@tamachan:~# tracepath vnhan01.sixxs.net 1: localhost 0.680ms pmtu 1492 1: localhost 21.829ms 1: localhost 22.503ms 2: localhost 25.572ms 3: localhost 28.050ms asymm 5 4: localhost 21.682ms 5: 218.100.10.28 22.270ms 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 6: vnhan01.sixxs.net 18649.781ms reached

root@tamachan:~# tracepath6 vnhan01.sixxs.net 1?: [LOCALHOST] 0.079ms pmtu 1280 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 1: gw-TUNNELID.han-01.vn.sixxs.net 41177.240ms 1: gw-TUNNELID.han-01.vn.sixxs.net 40168.828ms 1: gw-TUNNELID.han-01.vn.sixxs.net 39154.915ms 1: gw-TUNNELID.han-01.vn.sixxs.net 38151.032ms 2: vnhan01.sixxs.net 37161.563ms reached Resume: pmtu 1280 hops 2 back 64

vnhan01 external connectivity massive latency

Jeroen Massar SixXS Staff

on Monday, 07 October 2013 17:39:02

A quick look at tracepath suggests it's not a problem between here and there:

root@tamachan:~# tracepath vnhan01.sixxs.net

1: localhost 0.680ms pmtu 1492

1: localhost 21.829ms

1: localhost 22.503ms

2: localhost 25.572ms

3: localhost 28.050ms asymm 5

4: localhost 21.682ms

How come you have *4* 'localhost' hops before going anywhere else? And how come that you have 22ms already there?

5: 218.100.10.28 22.270ms

6: no reply

7: no reply

8: no reply

9: no reply

10: no reply

11: no reply

No replies for a variety of hops and then:

6: vnhan01.sixxs.net 18649.781ms reached

Hop 6 again!? Sounds rather weird and with rather bad connectivity.

root@tamachan:~# tracepath6 vnhan01.sixxs.net

As your IPv4 is causing problems already, a tunnel will just show more of that and thus any such result is quite useless. The fact that your host thinks the remote end is several hops away though, is rather bad as it should be on the same link and thus it should never try to reach further.

vnhan01 external connectivity massive latency

Shadow Hawkins on Tuesday, 08 October 2013 03:45:08

Ah, sorry, let me do the tracepath with -n. Those are all 'localhost' due to my ISP's dumb name resolver...

root@tamachan:~# tracepath -n vnhan01.sixxs.net 1: 117.4.10.20 0.666ms pmtu 1492 1: 117.4.10.1 21.961ms 1: 117.4.10.1 21.758ms 2: 27.68.241.85 22.028ms 3: 27.68.240.25 23.808ms asymm 5 4: 27.68.240.70 21.602ms 5: 218.100.10.28 24.346ms 6: no reply 6: 119.17.194.232 4713.587ms reached Resume: pmtu 1492 hops 6 back 57

So the first five hops reply happily, then when the 6th packet doesn't return after a short timeout, it sends another packet and marks it no reply. _Later_ that packet comes back, with the TTL indicating it's the packet from 6, so it's output that way. That's what happened before, except tracepath had sent packets 7, 8, 9, 10 and 11 before 6 came back. Here's the equivalent traceroute, however traceroute doesn't indicate when a reply packet comes in after later packets in the path have been sent, making it look like there's non-responsive hops in the path:

root@tamachan:~# traceroute -n vnhan01.sixxs.net traceroute to vnhan01.sixxs.net (119.17.194.232), 30 hops max, 38 byte packets 1 117.4.10.1 3.994 ms 4.277 ms 4.396 ms 2 27.68.241.85 6.276 ms 11.170 ms 5.060 ms 3 27.68.240.25 11.776 ms 11.456 ms 11.763 ms 4 27.68.240.70 5.093 ms 4.271 ms 4.847 ms 5 218.100.10.28 5.118 ms 8.606 ms 4.732 ms 6 * * * 7 * 119.17.194.232 863.943 ms 708.877 ms

When I'm lucky and run my trace during the burst of arriving packets, I get apparently-good results:

root@tamachan:~# tracepath -n vnhan01.sixxs.net 1: 117.4.10.20 0.622ms pmtu 1492 1: 117.4.10.1 21.403ms 1: 117.4.10.1 22.069ms 2: 27.68.241.85 24.969ms 3: 27.68.240.25 37.067ms asymm 5 4: 27.68.240.70 21.738ms 5: 218.100.10.28 22.086ms 6: 119.17.194.232 24.573ms reached Resume: pmtu 1492 hops 6 back 57

traceroute to vnhan01.sixxs.net (119.17.194.232), 30 hops max, 38 byte packets 1 117.4.10.1 4.317 ms 5.198 ms 11.524 ms 2 27.68.241.85 4.738 ms 5.424 ms 5.328 ms 3 27.68.240.25 6.530 ms 11.747 ms 11.458 ms 4 27.68.240.70 5.301 ms 4.686 ms 4.383 ms 5 218.100.10.28 6.617 ms 4.973 ms 5.333 ms 6 * * 119.17.194.232 1689.357 ms

So, it looks to me like the tunnel server vnhan01.sixxs.net is having connectivity problems. I have no trouble accessing any other site, nor do I see these packet-delay-and-burst from the hop immediately before vnhan01.sixxs.net. I also noted, but didn't look too hard at yet, that pretty much every on-line ping6 I tried couldn't reach vnhan01.sixxs.net by ipv6, but since I can reach _out_ via IPv6 (eventually) I suspect this is an administrative block, not a connectivity issue. I understood such issues could be reported through here and would be passed on to the PoP's local administrators? If not, I'll try and chase it up with netnam directly.

vnhan01 external connectivity massive latency

Jeroen Massar SixXS Staff

on Wednesday, 09 October 2013 13:05:31

Traceroute from the other side of the planet:

$ traceroute6 vnhan01.sixxs.net traceroute to vnhan01.sixxs.net (2401:e800::4), 30 hops max, 80 byte packets 1 ge-1-3-0.breda.ipv6.concepts-ict.net (2001:838:1:1::1) 0.597 ms 0.797 ms 0.779 ms 2 2001:838:5:11::2 (2001:838:5:11::2) 0.639 ms 0.637 ms 0.627 ms 3 gi2-11.nikhef.ipv6.concepts-ict.net (2001:838:5:a::1) 1.965 ms 1.984 ms 2.018 ms 4 ge-0.ams-ix.amstnl02.nl.bb.gin.ntt.net (2001:7f8:1::a500:2914:1) 3.015 ms 3.073 ms 3.247 ms 5 ae-2.r22.amstnl02.nl.bb.gin.ntt.net (2001:728:0:2000::3d) 2.125 ms 2.181 ms 2.187 ms 6 as-0.r25.tokyjp05.jp.bb.gin.ntt.net (2001:418:0:2000::16) 345.143 ms 269.186 ms 269.350 ms 7 as-1.r21.tkokhk01.hk.bb.gin.ntt.net (2001:218:0:2000::13e) 317.932 ms 316.440 ms 314.148 ms 8 ae-6.r02.chwahk02.hk.bb.gin.ntt.net (2001:218:0:2000::12a) 312.899 ms 308.278 ms 308.271 ms 9 2001:218:7000:5000::93 (2001:218:7000:5000::93) 330.629 ms 331.678 ms 331.023 ms 10 vnhan01.sixxs.net (2401:e800::4) 352.517 ms 343.649 ms 340.578 ms

As such, all looks fine to me.

I understood such issues could be reported through here and would be passed on to the PoP's local administrators? If not, I'll try and chase it up with netnam directly.

As you are a user of SixXS, your reports should go to us.

vnhan01 external connectivity massive latency

Shadow Hawkins on Friday, 11 October 2013 07:06:44

Yeah, all working fine from my end now too. 2001:218:7000:5000::93 was the host I was getting "unreachable" back from when trying to use web-based IPv6 pings and traceroutes to vnhan01.sixxs.net. So it must have been that machine or its local network link that was in trouble. So consider this resolved as far as I'm concerned.

Please note Posting is only allowed when you are logged in.