|
Ticket ID: SIXXS #6590422 Ticket Status: Remote Problem PoP: gblon02 - Goscomb Technologies (London)
T55649: Recent high packet loss to gateway
![[gb]](/s/countries/gb.gif) Shadow Hawkins on Tuesday, 06 March 2012 15:44:18
Hi
Since around 08:00 this morning I'm seeing significant packet loss when I ping the POP's gateway. Running a quick test from this machine I'm seeing an 8% packet loss followed by 18%. (Wrote this email and then checked again. 18% loss once more).
If you visit <http://f8lure.mouselike.org/index.asp?DoAction=ViewHost&HostName=ipv6.ldml.com> you can see the issue. That URL has links to historic graphs which show that the packet loss increased markedly around 20:00 / 22:00 last night (2012-02-05).
Comparisons to IPv4 traffic can be seen at <http://f8lure.mouselike.org/index.asp?DoAction=ViewHost&HostName=78.105.5.204>. This shows little packet loss.
FWIW, my provider, Be*, has had some significant issues in the recent past but these seem to have been addressed.
Thank you for any assistance / pointers you can offer.
ATB
Simon
<simon@ldml.com>
State change: remoteproblem
The state of this ticket has been changed to remoteproblem
T55649: Recent high packet loss to gateway
You might want to determine where the packet loss is taking place, a ping won't tell you that. Likely though, there is not much we can do about this.
T55649: Recent high packet loss to gateway
Shadow Hawkins on Tuesday, 06 March 2012 16:50:41
Hi
Thank you for the response.
"Remote problem" is not massively descriptive. It would seem that the gateway router my tunnel communicates with (gw-1161.lon-02.gb.sixxs.net) probably is overworked / underpaid and is dropping packets. Is there nothing that can be done to alert someone to this and try to get some remedial work undertaken?
If I'm misunderstanding your response, apologies.
Simon
T55649: Recent high packet loss to gateway
You are missing the part where your IPv6 packets are being sent over IPv4 and that the loss can happen on that path. When you do a IPv6 ping indeed, the next hop will be the PoP and that one will seem to be dropping them while effectively they are being dropped somewhere in IPv4.
gblon02 like every other PoP has a lot more capacity than what is being used, thus that is not the issue.
Is there nothing that can be done to alert someone to this and try to get some remedial work undertaken?
Yes, try to figure out where on the IPv4 path the packets are being dropped, do a IPv4 traceroute. 'mtr' is a handy tool for this. That said though, if a node in the path is applying "QoS" (or other wise said: rate limiting protocols it does not know) then you won't see it with ICMPv4 pings either, thus that makes it hard to determine what it is. Ask your ISP what they do with these packets and you might find out.
T55649: Recent high packet loss to gateway
![[gr]](/s/countries/gr.gif) Shadow Hawkins on Thursday, 12 July 2012 22:58:22
I also have this issue for some time now.
Bellow are some technical details. I'm running the test from the SIXXS termination box using AICCU. Pings were ran at parallel.
IPv6 has packet loss (I'm pinging the tunnel's remote ipv6 address):
$ ping6 -s 1024 2a01:348:6:563::1
--- 2a01:348:6:563::1 ping statistics ---
61 packets transmitted, 40 received, 34% packet loss, time 60141ms
rtt min/avg/max/mdev = 38.655/39.692/42.019/0.562 ms
IPv4 doesn't (I'm pinging gblon02 which is my sixxs remote tunnel endpoint - IOW it is the same box as above):
$ ping -s 1024 77.75.104.126
--- 77.75.104.126 ping statistics ---
59 packets transmitted, 59 received, 0% packet loss, time 60617ms
rtt min/avg/max/mdev = 33.395/34.795/74.788/5.403 ms
tcpdump shows the lost icmp echo replies:
21:52:47.857387 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 30, length 1032
21:52:47.896451 IP 77.75.104.126 > 10.1.1.2: IP6 2a01:348:6:563::1 > 2a01:348:6:563::2: ICMP6, echo reply, seq 30, length 1032
21:52:48.858906 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 31, length 1032
21:52:49.860428 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 32, length 1032
21:52:50.860448 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 33, length 1032
21:52:50.899175 IP 77.75.104.126 > 10.1.1.2: IP6 2a01:348:6:563::1 > 2a01:348:6:563::2: ICMP6, echo reply, seq 33, length 1032
21:52:51.861615 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 34, length 1032
21:52:51.901631 IP 77.75.104.126 > 10.1.1.2: IP6 2a01:348:6:563::1 > 2a01:348:6:563::2: ICMP6, echo reply, seq 34, length 1032
21:52:52.863091 IP 10.1.1.2 > 77.75.104.126: IP6 2a01:348:6:563::2 > 2a01:348:6:563::1: ICMP6, echo request, seq 35, length 1032
21:52:52.902538 IP 77.75.104.126 > 10.1.1.2: IP6 2a01:348:6:563::1 > 2a01:348:6:563::2: ICMP6, echo reply, seq 35, length 1032
Because of that my IPv6 TCP connections are ridiculously slow(er):
IPv6:
100%[===================================================================================>] 134,674 19.2K/s in 6.9s
2012-07-12 21:55:54 (19.2 KB/s) - `vr2.png.19' saved [134674/134674]
IPv4:
100%[===================================================================================>] 134,674 753K/s in 0.2s
2012-07-12 21:56:03 (753 KB/s) - `vr2.png.20' saved [134674/134674]
T55649: Recent high packet loss to gateway
IPv6 has packet loss (I'm pinging the tunnel's remote ipv6 address):
As you are transporting IPv6 over IPv4, if there is loss somewhere on the path between you and the PoP, then that loss is directly translated to loss on IPv6.
As such, you will need to supply a traceroute and maybe a mtr to see where that loss is happening.
IPv4 doesn't (I'm pinging gblon02 which is my sixxs remote tunnel endpoint - IOW it is the same box as above):
It can quite well be that an ISP on the path is doing "QoS", or otherwise stated: dropping packets they do not like.
Very difficult to see how that is happening.
Because of that my IPv6 TCP connections are ridiculously slow(er):
What are the paths taken for both IPv4 and IPv6?
Note also that this can be caused by a lot of other issues; though likely indeed that you have packet loss somewhere between you and the PoP if the above is already showing.
T55649: Recent high packet loss to gateway
Shadow Hawkins on Monday, 16 July 2012 01:57:53
I am pinging my POP with IPv6 and IPv4 and only IPv6 shows packet loss. IOW I am pinging the exact same destination with IPv6 and IPv4 and IPv6 shows significant packet loss. The problem only happens with large packets.
Obviously the IPv6 traceroute only shows one hop. Here are the traceroutes to my POP with IPv4 and IPv6:
$ traceroute -n -q 5 2a01:348:6:563::1 1024
traceroute to 2a01:348:6:563::1 (2a01:348:6:563::1), 30 hops max, 1024 byte packets
1 2a01:348:6:563::1 47.800 ms 56.619 ms 65.312 ms 74.748 ms 84.335 ms
$ traceroute -n -q 5 77.75.104.126 1024
traceroute to 77.75.104.126 (77.75.104.126), 30 hops max, 1024 byte packets
1 10.1.1.1 0.825 ms 0.734 ms 0.647 ms 0.645 ms 0.658 ms
2 78.149.128.1 29.463 ms 38.076 ms 47.237 ms 55.968 ms 65.002 ms
3 78.151.225.209 74.512 ms 83.778 ms 92.506 ms 101.545 ms 110.537 ms
4 78.151.225.236 145.666 ms 78.151.225.216 129.638 ms 78.151.225.208 138.876 ms 78.151.225.212 147.218 ms 78.151.225.204 176.050 ms
5 62.24.240.5 165.014 ms 62.24.240.13 158.742 ms 62.24.240.5 148.483 ms 62.24.240.13 159.613 ms 159.627 ms
6 78.144.1.63 147.845 ms 78.144.1.61 159.868 ms 78.144.1.65 150.955 ms 78.144.1.63 148.955 ms 148.272 ms
7 78.144.0.89 149.815 ms 78.144.0.193 150.638 ms 78.144.0.178 138.749 ms 78.144.0.91 141.761 ms 78.144.1.53 159.345 ms
8 195.66.224.226 143.631 ms 135.926 ms 143.910 ms 145.849 ms 141.793 ms
9 46.17.60.181 139.730 ms 154.642 ms 141.656 ms 150.334 ms 146.739 ms
10 93.89.94.202 151.472 ms 145.826 ms 142.509 ms 143.059 ms 145.603 ms
11 77.75.104.126 135.242 ms 140.365 ms 146.206 ms 142.170 ms 147.139 ms
Here's a more informative IPv6 traceroute to ipv6.google.com (it shows the packet loss):
$ traceroute -n -q 5 ipv6.google.com 1024
traceroute to ipv6.google.com (2a00:1450:4007:802::1010), 30 hops max, 1024 byte packets
1 * 2a01:348:6:563::1 48.227 ms 55.338 ms 64.134 ms 73.597 ms
2 * 2a01:348:0:4:0:3:1:1 91.119 ms 99.820 ms 109.035 ms *
3 2a01:348:0:4:0:3:0:1 128.662 ms 135.871 ms 145.704 ms 154.417 ms 162.867 ms
4 2a01:348::36:1:1 172.592 ms * 138.152 ms * 137.814 ms
5 * 2a01:348::24:1:1 128.945 ms 129.175 ms 118.832 ms 120.123 ms
6 2a01:348::40:1:1 120.757 ms 119.701 ms 120.216 ms 119.437 ms *
7 2a01:348::41:1:1 103.377 ms * * 91.910 ms *
8 * 2a01:348::17:0:1 91.489 ms 91.850 ms 92.092 ms 79.883 ms
9 2a01:348::32:1:1 101.327 ms 75.849 ms 67.592 ms 59.266 ms 51.553 ms
10 2001:7f8:17::3b41:1 39.871 ms 47.782 ms 55.919 ms 63.567 ms 71.842 ms
11 2001:4860::1:0:15f 45.419 ms * 64.789 ms 2001:4860::1:0:3067 46.329 ms 2001:4860::1:0:15f 45.063 ms
12 2001:4860::8:0:2ddf 45.615 ms * 43.541 ms * 2001:4860::8:0:2dde 49.550 ms
13 2001:4860::8:0:3df4 55.553 ms 2001:4860::8:0:3df5 55.258 ms 54.674 ms 56.203 ms 55.226 ms
14 2001:4860::1:0:23 66.969 ms * * * 74.561 ms
15 2001:4860:0:1::d1 72.265 ms 82.751 ms 91.349 ms 100.428 ms 108.889 ms
16 2a00:1450:8000:d::6 117.830 ms 127.554 ms 135.465 ms 144.918 ms 153.368 ms
Here's MTR with small and large packets to ipv6.google.com.
$ mtr -s 1024 -rn -c 20 ipv6.google.com
HOST: efika.hell.gr Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:348:6:563::1 30.0% 20 39.6 39.9 39.5 40.7 0.4
2.|-- 2a01:348:0:4:0:3:1:1 35.0% 20 39.6 39.9 39.2 40.8 0.5
3.|-- 2a01:348:0:4:0:3:0:1 30.0% 20 40.6 40.4 39.6 41.0 0.4
4.|-- 2a01:348::36:1:1 40.0% 20 41.1 40.9 40.1 41.6 0.5
5.|-- 2a01:348::24:1:1 35.0% 20 40.3 40.3 39.4 40.8 0.4
6.|-- 2a01:348::40:1:1 10.0% 20 40.7 40.9 39.7 47.8 1.8
7.|-- 2a01:348::41:1:1 30.0% 20 41.4 41.5 40.3 43.5 1.0
8.|-- 2a01:348::17:0:1 50.0% 20 40.0 39.9 39.1 40.5 0.4
9.|-- 2a01:348::32:1:1 25.0% 20 40.2 40.1 39.5 41.1 0.5
10.|-- 2001:7f8:17::3b41:1 45.0% 20 40.4 45.0 39.8 91.2 15.3
11.|-- 2001:4860::1:0:15f 45.0% 20 65.0 51.2 40.3 100.2 18.2
12.|-- 2001:4860::8:0:2dde 20.0% 20 41.2 47.2 40.8 83.2 11.3
| `|-- 2001:4860::8:0:2ddf
13.|-- 2001:4860::8:0:3df5 55.0% 20 56.7 58.5 55.9 68.3 3.9
| `|-- 2001:4860::8:0:3df4
14.|-- 2001:4860::1:0:23 40.0% 20 79.0 66.2 55.3 116.5 20.5
15.|-- 2001:4860:0:1::d1 15.0% 20 56.2 56.1 55.1 58.3 0.8
16.|-- 2a00:1450:4007:802::1012 31.6% 19 55.2 55.7 55.0 56.5 0.5
$ mtr -rn -c 20 ipv6.google.com
HOST: efika.hell.gr Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:348:6:563::1 0.0% 20 29.9 29.2 28.4 30.0 0.4
2.|-- 2a01:348:0:4:0:3:1:1 0.0% 20 29.3 29.1 28.0 29.8 0.4
3.|-- 2a01:348:0:4:0:3:0:1 0.0% 20 29.1 29.7 28.9 30.4 0.4
4.|-- 2a01:348::36:1:1 0.0% 20 30.0 30.0 29.0 31.3 0.5
5.|-- 2a01:348::24:1:1 0.0% 20 28.8 29.7 28.8 30.7 0.5
6.|-- 2a01:348::40:1:1 0.0% 20 30.1 29.8 29.0 30.8 0.5
7.|-- 2a01:348::41:1:1 0.0% 20 30.5 30.6 29.0 33.3 1.2
8.|-- 2a01:348::17:0:1 0.0% 20 30.3 30.8 28.7 50.0 5.0
9.|-- 2a01:348::32:1:1 0.0% 20 30.3 32.0 28.6 75.2 10.3
10.|-- 2001:7f8:17::3b41:1 0.0% 20 29.8 35.3 29.0 96.9 16.0
11.|-- 2001:4860::1:0:15f 0.0% 20 30.3 47.7 29.8 209.0 44.1
12.|-- 2001:4860::8:0:2ddf 0.0% 20 30.0 45.8 29.4 94.2 22.2
| `|-- 2001:4860::8:0:2dde
13.|-- 2001:4860::8:0:3df5 0.0% 20 44.7 52.0 44.4 114.2 16.7
| `|-- 2001:4860::8:0:3df4
14.|-- 2001:4860::1:0:23 0.0% 20 63.2 50.2 44.2 70.4 8.9
15.|-- 2001:4860:0:1::d1 5.0% 20 45.8 45.0 44.3 45.8 0.5
16.|-- 2a00:1450:4007:802::1012 0.0% 20 44.9 44.9 44.0 45.6 0.4
Pinging my tunnel's IPv6 address from an external host also gives packet loss with 1024 bytes packets:
--- 2a01:348:6:563::2 ping statistics ---
36 packets transmitted, 23 received, 36% packet loss, time 35073ms
rtt min/avg/max/mdev = 140.576/141.679/143.166/0.618 ms
Surprisingly I don't see packet loss when pinging my tunnel's remote address from the same (external) host:
--- 2a01:348:6:563::1 ping statistics ---
57 packets transmitted, 56 received, 1% packet loss, time 56061ms
rtt min/avg/max/mdev = 101.837/102.107/102.929/0.464 ms
My ISP in the UK is Talktalk which may be doing a lot of ****** things. I tried pinging other networks from the 2a01:348:6:563::/64 range (e.g. 562::2, 561::2, ...) and I didn't notice any packet loss. This indicates that indeed it may be another problem of TalkTalk. Maybe they are traffic shaping and giving less bandwidth to 6in4 packets, causing packet loss.
T55649: Recent high packet loss to gateway
I am pinging my POP with IPv6 and IPv4 and only IPv6 shows packet loss. IOW I am pinging the exact same destination with IPv6 and IPv4 and IPv6 shows significant packet loss. The problem only happens with large packets.
Thus then you have to figure out which hop on the IPv4 path is dropping those large packets.
Note that if an ISP or network is performing some kind of QoS (or better said traffic classification and dropping), then it might be that they accept all ICMP but classify the packets used for tunneling differently.
Here's a more informative IPv6 traceroute to ipv6.google.com (it shows the packet loss):
But it does not show at which IPv4 hop this is happening as you obviously already have shown that the packet loss happens between your IPv4 endpoint and the PoPs IPv4 address.
Maybe they are traffic shaping and giving less bandwidth to 6in4 packets, causing packet loss.
This is a very very likely cause especially with a cheap heavily oversubscribed provider like Talk Talk.
|
![[ch]](/s/countries/ch.gif)
on Tuesday, 06 March 2012 16:00:40
Posting is only allowed when you are