|
/48 subnet is not routed
![[ru]](/s/countries/ru.gif) Shadow Hawkins on Wednesday, 01 October 2014 20:38:02
Please help me debug a problem with packets from outside networks addrressed for my /48 subnet (R254313) not reaching their destination.
As I see from traceroute, they are dropped before reaching the PoP.
I have no problems with my tunnel's (T102970) routing to its endpoint or the initial /64 subnet routing.
Here I am testing networks allocated to me by SixXS from a third-party VPS.
Please note the -I switch, you'll get a slightly different result without it.
1. Tracing my tunnel's endpoint.
% sudo traceroute6 -I 2a02:578:5002:62::2
traceroute to 2a02:578:5002:62::2 (2a02:578:5002:62::2), 30 hops max, 80 byte packets
1 2a00:ab00:108::1 (2a00:ab00:108::1) 1.033 ms 1.026 ms 1.023 ms
2 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.276 ms 8.282 ms 8.282 ms
3 router03.msk.ru.edpnet.net (2001:7f8:20:102::246:16) 9.785 ms 9.789 ms 9.788 ms
4 2a02:578:1:48::2 (2a02:578:1:48::2) 16.855 ms 17.015 ms 17.177 ms
5 2a02:578:1:46::1 (2a02:578:1:46::1) 17.288 ms 17.857 ms 17.846 ms
6 ruled01.sixxs.net (2a02:578:5001:2::2) 16.809 ms 16.964 ms 16.942 ms
7 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 16.816 ms 17.134 ms 17.119 ms
8 cl-99.led-01.ru.sixxs.net (2a02:578:5002:62::2) 33.040 ms 33.446 ms 33.437 ms
2. Tracing a host in /48 subnet.
% sudo traceroute6 -I 2a02:578:5418:d1:5054:a2ff:fe00:1
traceroute to 2a02:578:5418:d1:5054:a2ff:fe00:1 (2a02:578:5418:d1:5054:a2ff:fe00:1), 30 hops max, 80 byte packets
1 2a00:ab00:108::1 (2a00:ab00:108::1) 1.257 ms 1.364 ms 1.365 ms
2 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.695 ms 8.707 ms 8.707 ms
3 router03.msk.ru.edpnet.net (2001:7f8:20:102::246:16) 10.001 ms 10.008 ms 10.006 ms
4 2a02:578:1:48::2 (2a02:578:1:48::2) 17.382 ms 17.396 ms 17.709 ms
5 2a02:578:1:46::1 (2a02:578:1:46::1) 18.019 ms 18.025 ms 18.071 ms
6 * * *
7 * * *
8 * * *
From the host 2a02:578:5418:d1:5054:a2ff:fe00:1 (traceroute destination in second example) I can ping some hosts (e.g. www.sixxs.net) but not others (e.g. google.com).
3. Routing table on my tunnel endpoint (a DD-WRT router with 2.6.24 kernel):
# ip -6 r s
2a02:578:5002:62::/64 via :: dev sixxs metric 256 expires 42758045sec
2a02:578:5002:8062::/64 dev br0 metric 256 expires 42758045sec
2a02:578:5002:8062::/64 dev br0 metric 1024 expires 42758046sec
2a02:578:5418:d1::/64 via 2a02:578:5002:8062::2 dev br0 metric 1024 expires 42927936sec
fe80::/64 dev eth0 metric 256 expires 42757956sec
fe80::/64 dev br0 metric 256 expires 42757957sec
fe80::/64 via :: dev sixxs metric 256 expires 42758046sec
fe80::/64 dev vlan1 metric 256 expires 42850367sec
fe80::/64 dev eth1 metric 256 expires 42850367sec
fe80::/64 dev vlan2 metric 256 expires 42850368sec
ff00::/8 dev eth0 metric 256 expires 42757956sec
ff00::/8 dev br0 metric 256 expires 42757957sec
ff00::/8 dev sixxs metric 256 expires 42758046sec
ff00::/8 dev vlan1 metric 256 expires 42850367sec
ff00::/8 dev eth1 metric 256 expires 42850367sec
ff00::/8 dev vlan2 metric 256 expires 42850368sec
default dev sixxs metric 1024 expires 42758046sec
unreachable default dev lo metric -1 error -128):
4. Firewall rules on the tunnel endpoint are added in a script:
MELFORCE_ADDR="2a02:578:5002:8062::2"
EXTIF=sixxs
INTIF=br0
ip6tables -F
ip6tables -A FORWARD -i $EXTIF -o $EXTIF -j DROP
# Limit ICMPv6
ip6tables -A INPUT -p icmpv6 -m limit --limit 600/min -j ACCEPT
ip6tables -A INPUT -p icmpv6 -j DROP
ip6tables -A OUTPUT -p icmpv6 -m limit --limit 600/min -j ACCEPT
ip6tables -A OUTPUT -p icmpv6 -j DROP
ip6tables -A FORWARD -p icmpv6 -m limit --limit 600/min -j ACCEPT
ip6tables -A FORWARD -p icmpv6 -j DROP
# Input rules
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -i $INTIF -j ACCEPT
ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -j DROP
# Forward rules
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -i $EXTIF -d $MELFORCE_ADDR -j ACCEPT
ip6tables -A FORWARD -i $EXTIF -o $INTIF -j DROP
/48 subnet is not routed
2a02:578:5002:8062::/64 dev br0 metric 256 expires 42758045sec 2a02:578:5002:8062::/64 dev br0 metric 1024 expires 42758046sec
Why do you have that route twice?
2a02:578:5418:d1::/64 via 2a02:578:5002:8062::2 dev br0 metric 1024 expires 42927936sec
What is 2a02:578:5002:8062::2 ?
And where do you terminate the rest of 2a02:578:5418::/48 ?
# Limit ICMPv6
You are doing rate limits, hence when the limit is reached some packets will randomly disappear.
ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
State tracking, have fun debugging that...
You might want to add logging to see what is disappearing. Also, checking the active rules is a good idea instead of checking some script that does not reset the state of already installed rules.
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 06:09:54
Jeroen Massar wrote:
Why do you have that route twice?
One was manually added in a startup script, the other was received by kernel from local radvd. Newer radvd would set /proc/sys/net/ipv6/conf/br0/autoconf to "0" and prevent kernel from doing this. I removed the static anyway.
>2a02:578:5418:d1::/64 via 2a02:578:5002:8062::2 dev br0 metric 1024 expires 42927936sec
What is 2a02:578:5002:8062::2 ?
That is where I route part of the /48, it's a desktop PC.
And where do you terminate the rest of 2a02:578:5418::/48 ?
I'm don't need the rest yet, thus I'm not routing it.
> # Limit ICMPv6
You are doing rate limits, hence when the limit is reached some packets will randomly disappear.
Yes, I noticed it too yesterday. I removed the rate-limit, it accepts all of ICMPv6 now.
> ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
State tracking, have fun debugging that...
I'll leave it for now, I didn't experience problems with it yet.
You might want to add logging to see what is disappearing.
I'm not sure what I have to debug exactly. My initial problem was packets addressed for the /48 not reaching the tunnel. I thought it was a routing problem within EDPnet. Can you confirm this?
I can reach host within the /48 from some outside servers but not from others:
(this is a successful example, an ICMP traceroute)
6 2a02:578:1:34::1 (2a02:578:1:34::1) 57.594 ms 57.226 ms 56.936 ms
7 2a02:578:1:25::2 (2a02:578:1:25::2) 56.915 ms 57.236 ms 57.425 ms
8 ruled01.sixxs.net (2a02:578:5001:2::2) 56.758 ms 56.730 ms 56.657 ms
9 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 57.131 ms 57.499 ms 57.411 ms
10 melforce.xtsubasa.org (2a02:578:5002:8062::2) 73.908 ms 73.679 ms 73.429 ms
11 tux1.xtsubasa.org (2a02:578:5418:d1:5054:a2ff:fe00:1) 73.759 ms 73.791 ms 74.021 ms
Also, checking the active rules is a good idea instead of checking some script that does not reset the state of already installed rules.
I'm not sure what you mean. The script executes "ip6tables -F" in the beginning, do I have to do anything else? I work with the rules through this script only, not anywhere else. I posted the script because -L does not display everything, e.g. input/output interface name in a rule...
Anyway, this is the updated routing table:
# ip -6 r s
2a02:578:5002:62::/64 via :: dev sixxs metric 256 expires 42948779sec
2a02:578:5002:8062::/64 dev br0 metric 256 expires 42948779sec
2a02:578:5418:d1::/64 via 2a02:578:5002:8062::2 dev br0 metric 1024 expires 42948780sec
fe80::/64 dev eth0 metric 256 expires 42948690sec
fe80::/64 dev vlan1 metric 256 expires 42948691sec
fe80::/64 dev eth1 metric 256 expires 42948691sec
fe80::/64 dev br0 metric 256 expires 42948691sec
fe80::/64 dev vlan2 metric 256 expires 42948692sec
fe80::/64 via :: dev sixxs metric 256 expires 42948780sec
ff00::/8 dev eth0 metric 256 expires 42948690sec
ff00::/8 dev vlan1 metric 256 expires 42948691sec
ff00::/8 dev eth1 metric 256 expires 42948691sec
ff00::/8 dev br0 metric 256 expires 42948691sec
ff00::/8 dev vlan2 metric 256 expires 42948692sec
ff00::/8 dev sixxs metric 256 expires 42948780sec
default dev sixxs metric 1024 expires 42948780sec
unreachable default dev lo metric -1 error -128
Current rules:
# ip6tables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT icmpv6 ::/0 ::/0
ACCEPT all ::/0 ::/0
ACCEPT all ::/0 ::/0
ACCEPT all ::/0 ::/0 state RELATED,ESTABLISHED
DROP all ::/0 ::/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DROP all ::/0 ::/0
ACCEPT icmpv6 ::/0 ::/0
ACCEPT all ::/0 ::/0 state RELATED,ESTABLISHED
ACCEPT all ::/0 2a02:578:5002:8062::2/128
ACCEPT all ::/0 2a02:578:5418:d1::/64
DROP all ::/0 ::/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT icmpv6 ::/0 ::/0
And the firewall script again:
#!/bin/sh
MELFORCE_ADDR="2a02:578:5002:8062::2"
VIRTUAL_NET="2a02:578:5418:d1::/64"
EXTIF=sixxs
INTIF=br0
ICMP_LIMIT="100/s"
ip6tables -F
ip6tables -A FORWARD -i $EXTIF -o $EXTIF -j DROP
# Limit ICMPv6
ip6tables -A INPUT -p icmpv6 -j ACCEPT
ip6tables -A OUTPUT -p icmpv6 -j ACCEPT
ip6tables -A FORWARD -p icmpv6 -j ACCEPT
# Input rules
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -i $INTIF -j ACCEPT
ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -j DROP
# Forward rules
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -i $EXTIF -d $MELFORCE_ADDR -j ACCEPT
ip6tables -A FORWARD -i $EXTIF -d $VIRTUAL_NET -j ACCEPT
ip6tables -A FORWARD -i $EXTIF -o $INTIF -j DROP
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 15:00:53
I have changed my firewall script using example in SixXS Wiki as a template:
https://www.sixxs.net/wiki/IPv6_Firewalling#Example_script_for_IPv6_stateful_firewall
This is how it looks now:
MELFORCE_ADDR="2a02:578:5002:8062::2"
VIRTUAL_NET="2a02:578:5418:d1::/64"
EXTIF=sixxs
INTIF=br0
# First, delete all:
ip6tables -F
ip6tables -X
# Allow anything on the local link
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o lo -j ACCEPT
# Allow anything out on the internet
ip6tables -A OUTPUT -o $EXTIF -j ACCEPT
# Allow established, related packets back in
ip6tables -A INPUT -i $EXTIF -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow the localnet access us:
ip6tables -A INPUT -i $INTIF -j ACCEPT
ip6tables -A OUTPUT -o $INTIF -j ACCEPT
# Filter all packets that have RH0 headers:
ip6tables -A INPUT -m rt --rt-type 0 -j DROP
ip6tables -A FORWARD -m rt --rt-type 0 -j DROP
ip6tables -A OUTPUT -m rt --rt-type 0 -j DROP
# Allow Link-Local addresses
ip6tables -A INPUT -s fe80::/10 -j ACCEPT
ip6tables -A OUTPUT -s fe80::/10 -j ACCEPT
# Allow ICMPv6 everywhere
ip6tables -I INPUT -p icmpv6 -j ACCEPT
ip6tables -I OUTPUT -p icmpv6 -j ACCEPT
ip6tables -I FORWARD -p icmpv6 -j ACCEPT
# Allow forwarding
ip6tables -A FORWARD -m state --state NEW -i $INTIF -o $EXTIF -j ACCEPT
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
# Exception for melforce
ip6tables -A FORWARD -i $EXTIF -d $MELFORCE_ADDR -j ACCEPT
# Exception for /48 subnet
ip6tables -A FORWARD -i $EXTIF -d $VIRTUAL_NET -j ACCEPT
# Set the default policy
ip6tables -P INPUT DROP
ip6tables -P FORWARD DROP
ip6tables -P OUTPUT DROP
/48 subnet is not routed
Instead of cut & pasting a random example firewall ruleset, what is it that you want to achieve with your firewall?
Also, when you have a DROP somewhere, you might want to LOG them before as long as you have strange issues. Next to of course trying without a firewall at all, as then that cannot be the issue anymore.
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 15:56:49
Jeroen Massar wrote:
Instead of cut & pasting a random example firewall ruleset, what is it that you want to achieve with your firewall?
I use the firewall to have NAT-like security. Main points are:
1. It should accept all outgoing TCP/UDP connections.
2. It should deny all incoming TCP/UDP connections with some exceptions.
3. Should be stateful.
This is more or less what the previous ruleset did and the current, too.
I'm afraid there's no syslog facility on DD-WRT.
I just tried removing all firewall rules and rebooting the thing.
The routing problem was still there:
1. From outside to the allocated /64 subnet (all good):
traceroute to melforce (2a02:578:5002:8062::2), 30 hops max, 80 byte packets
1 2a00:ab00:108::1 (2a00:ab00:108::1) 2.721 ms 11.762 ms 2.658 ms
2 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.599 ms 2a00:ab00:0:15::2 (2a00:ab00:0:15::2) 10.369 ms 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.584 ms
3 router03.msk.ru.edpnet.net (2001:7f8:20:102::246:16) 9.457 ms 11.656 ms 11.632 ms
4 2a02:578:1:48::2 (2a02:578:1:48::2) 18.739 ms 16.740 ms 16.739 ms
5 2a02:578:1:46::1 (2a02:578:1:46::1) 18.675 ms 17.287 ms 17.292 ms
6 ruled01.sixxs.net (2a02:578:5001:2::2) 19.506 ms 16.670 ms 16.655 ms
7 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 19.040 ms 18.999 ms 18.529 ms
8 melforce.xtsubasa.org (2a02:578:5002:8062::2) 35.376 ms 33.803 ms 33.789 ms
2. From outside to the allocated /48 subnet (breaks middleway):
traceroute to tux1 (2a02:578:5418:d1:5054:a2ff:fe00:1), 30 hops max, 80 byte packets
1 2a00:ab00:108::1 (2a00:ab00:108::1) 1.188 ms 1.157 ms 1.119 ms
2 2a00:ab00:0:15::2 (2a00:ab00:0:15::2) 10.247 ms 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.426 ms 8.436 ms
3 router03.msk.ru.edpnet.net (2001:7f8:20:102::246:16) 9.913 ms 11.381 ms 9.870 ms
4 2a02:578:1:48::2 (2a02:578:1:48::2) 19.357 ms 16.573 ms 16.970 ms
5 2a02:578:1:46::1 (2a02:578:1:46::1) 18.818 ms 17.357 ms 19.022 ms
6 * * *
7 * * *
8 * * *
/48 subnet is not routed
I use the firewall to have NAT-like security. Main points are: 1. It should accept all outgoing TCP/UDP connections. 2. It should deny all incoming TCP/UDP connections with some exceptions. 3. Should be stateful.
State is never a good idea, are you really sure you want that?
I'm afraid there's no syslog facility on DD-WRT.
There is "readlog" which sometimes does work and some other times does not work.
2. From outside to the allocated /48 subnet (breaks middleway):
What do the routing tables and firewalls of the involved hops look like?
What does wireshark on those hops show?
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 16:07:54
Jeroen Massar wrote:
State is never a good idea, are you really sure you want that?
I don't really know about the downsides, I'll have to investigate.
What do the routing tables and firewalls of the involved hops look like?
What does wireshark on those hops show?
Do you mean hop 2a02:578:1:46::1 ?
It's not mine, it's EDPnet's...
/48 subnet is not routed
Do you mean hop 2a02:578:1:46::1 ?
On every hop that you control, it is not unlikely that you are routing packets back the wrong way or another.
But check this traceroute:
$ traceroute6 2a02:578:5418:d1:5054:a2ff:fe00:1
traceroute to 2a02:578:5418:d1:5054:a2ff:fe00:1 (2a02:578:5418:d1:5054:a2ff:fe00:1), 30 hops max, 80 byte packets
[..]
11 2a02:578:1:46::1 (2a02:578:1:46::1) 107.629 ms 101.587 ms 102.072 ms
12 ruled01.sixxs.net (2a02:578:5001:2::2) 59.227 ms 59.225 ms 58.918 ms
13 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 59.250 ms 59.281 ms 59.406 ms
14 melforce.xtsubasa.org (2a02:578:5002:8062::2) 76.713 ms 75.780 ms *
15 melforce.xtsubasa.org (2a02:578:5002:8062::2) 74.805 ms !N 75.510 ms !N 75.648 ms !N
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 16:20:49
Jeroen Massar wrote:
> Do you mean hop 2a02:578:1:46::1 ?
On every hop that you control, it is not unlikely that you are routing packets back the wrong way or another.
But even if so I don't understand why ruled01 would not reply in my problem traceroute above.
This is routing table on melforce:
2a02:578:5002:8062::/64 dev br0 proto kernel metric 256
2a02:578:5418:d1::/64 dev br1 metric 1024
fe80::/64 dev br1 proto kernel metric 256
fe80::/64 dev br0 proto kernel metric 256
fe80::/64 dev lan0 proto kernel metric 256
fe80::/64 dev qemu1 proto kernel metric 256
fe80::/64 dev qemu2 proto kernel metric 256
fe80::/64 dev qemu3 proto kernel metric 256
fe80::/64 dev qemu0 proto kernel metric 256
ff00::/8 dev br0 metric 256
ff00::/8 dev br1 metric 256
ff00::/8 dev lan0 metric 256
ff00::/8 dev qemu1 metric 256
ff00::/8 dev qemu2 metric 256
ff00::/8 dev qemu3 metric 256
ff00::/8 dev qemu0 metric 256
default via 2a02:578:5002:8062::1 dev br0 proto static metric 1024
Firewall rules:
# nft -nn list table inet filter
table inet filter {
chain input {
type filter hook input priority 0;
}
chain forward {
type filter hook forward priority 0;
ct state established,related accept
iif br0 oif br1 tcp dport { 80, 81} accept
iif br0 oif br1 ip protocol { tcp, udp} reject
iif br0 oif br1 ip6 nexthdr { tcp, udp} reject
}
chain output {
type filter hook output priority 0;
oif br1 tcp dport { 7001, 4001} reject
}
}
I can check Wireshark later.
/48 subnet is not routed
But even if so I don't understand why ruled01 would not reply in my problem traceroute above.
It likely replies, but the packet just does not make it back at all.
Firewall rules: # nft -nn list table inet filter
No idea what kind of firewall rule setup that is, but does "inet" mean IPv4 and/or IPv6?
ct state established,related accept
State tracking, are you sure that is doing exactly what you want it to do?
Also, which host is this on, and did you try disabling the firewall accepting all traffic?
You have a lot of components in the mix, any of these can be causing problems.
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 16:50:40
Jeroen Massar wrote:
No idea what kind of firewall rule setup that is, but does "inet" mean IPv4 and/or IPv6?
State tracking, are you sure that is doing exactly what you want it to do?
Also, which host is this on, and did you try disabling the firewall accepting all traffic?
You have a lot of components in the mix, any of these can be causing problems.
"inet" family means both IPv4 and IPv6.
This is on host melforce (2a02:578:5002:8062::2).
I changed the rules and removed connection tracking, it's now basically 2 rejects for specific ports:
# nft -nn list table inet filter
table inet filter {
chain input {
type filter hook input priority 0;
}
chain forward {
type filter hook forward priority 0;
iif br0 oif br1 tcp dport { 80, 81} accept
iif br0 oif br1 tcp dport { 7001, 4001} reject
}
chain output {
type filter hook output priority 0;
oif br1 tcp dport { 7001, 4001} reject
}
}
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 16:56:22
I re-did the Wireshark test with ping packets on the above host and it's not capturing anything again.
/48 subnet is not routed
Shadow Hawkins on Friday, 03 October 2014 05:34:13
Finally I tried running tcpdump on the tunnel endpoint.
1. Capturing packets for the host in allocated /64 (hostname melforce) with:
# tcpdump -v -i sixxs dst host 2a02:578:5002:8062::2
I see that something is being captured.
2. Capturing packets for the host in allocated /48 (hostname tux1) with:
# tcpdump -v -i sixxs dst host 2a02:578:5418:d1:5054:a2ff:fe00:1
then doing the ping from outside.
Tried with both local firewall up & down. Nothing is output.
I think I'm out of test options now...
/48 subnet is not routed
# tcpdump -v -i sixxs dst host 2a02:578:5002:8062::2
Looking at the tunnel interface is rarely correct as you will miss out on all tunneled packets and details that might indicate issues there.
Specifying only a single host will also miss out on every other packet which might indicate the actual problem.
/48 subnet is not routed
Shadow Hawkins on Saturday, 04 October 2014 17:34:23
Jeroen Massar wrote:
> # tcpdump -v -i sixxs dst host 2a02:578:5002:8062::2
Looking at the tunnel interface is rarely correct as you will miss out on all tunneled packets and details that might indicate issues there.
Specifying only a single host will also miss out on every other packet which might indicate the actual problem.
Okay, to eliminate this possibility I did a similar experiment catching IPv4 to/from the PoP.
1. To minimize network activity I powered of or unplugged Ethernet cables from all my devices and turned off Wi-Fi on the router (tunnel endpoint). I only left the router and the PC on. This PC does routing between allocated /64 and /48 and I login to router's SSH session from it. Virtual machine in /48 was also up. I logged out from X session and logged into tty.
2. Cleared all ip6tables firewall rules on the router and set default policy to ACCEPT on all chains.
For IPv4 rules I only have what DD-WRT sets as default and some port forwarding.
3. I ran this command on the router (vlan2 connects to WAN):
# tcpdump -v -i vlan2 host 77.109.111.178
4. Ran "ping6 2a02:578:5418:d1:5054:a2ff:fe00:1" on outside host which *has* connectivity to my /48.
tcpdump output looked like this:
21:16:11.580484 IP (tos 0x0, ttl 54, id 16896, offset 0, flags [none], proto IPv6 (41), length 124)
77.109.111.178.static.edpnet.net > 10.2.2.90: ipv6 104
21:16:11.581152 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto IPv6 (41), length 124)
10.2.2.90 > 77.109.111.178.static.edpnet.net: ipv6 104
21:16:12.580870 IP (tos 0x0, ttl 54, id 16896, offset 0, flags [none], proto IPv6 (41), length 124)
77.109.111.178.static.edpnet.net > 10.2.2.90: ipv6 104
21:16:12.581498 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto IPv6 (41), length 124)
10.2.2.90 > 77.109.111.178.static.edpnet.net: ipv6 104
5. Ran the same ping6 on outside host which *has no* connectivity to my /48.
ping6 was running for about 10 seconds and I saw nothing captured during that time.
My IPv4 routing table on DD-WRT just in case:
# ip r s
10.2.2.1 dev vlan2
10.2.2.0/24 dev vlan2 src 10.2.2.90
192.168.1.0/24 dev br0 src 192.168.1.1
169.254.0.0/16 dev br0 src 169.254.255.1
127.0.0.0/8 dev lo
default via 10.2.2.1 dev vlan2
And the interface vlan2:
# ip a s vlan2
9: vlan2@eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc noqueue
link/ether 00:03:2f:39:45:37 brd ff:ff:ff:ff:ff:ff
inet 10.2.2.90/24 brd 10.2.2.255 scope global vlan2
inet6 fe80::203:2fff:fe39:4537/64 scope link
valid_lft forever preferred_lft forever
/48 subnet is not routed
For IPv4 rules I only have what DD-WRT sets as default and some port forwarding.
And likely connection tracking....
Instead of assuming, do check and do not forget to check the normal rules and the '-t nat' ones... though if the connection tracker code is loaded it is effective and will affect connections, hence check the modules too.
# tcpdump -v -i vlan2 host 77.109.111.178
What about other hosts that might give you indications that there are issues, eg by sending you ICMP messages?
21:16:11.580484 IP (tos 0x0, ttl 54, id 16896, offset 0, flags [none], proto IPv6 (41), length 124) 77.109.111.178.static.edpnet.net > 10.2.2.90: ipv6 104
A NATted IP ddress, hello!
How are you enforcing that packets are always NATted towards your host?
5. Ran the same ping6 on outside host which *has no* connectivity to my /48.
What is the traceroute or other details that come along with this host?
/48 subnet is not routed
Shadow Hawkins on Saturday, 04 October 2014 19:22:21
Jeroen Massar wrote:
And likely connection tracking....
Instead of assuming, do check and do not forget to check the normal rules and the '-t nat' ones... though if the connection tracker code is loaded it is effective and will affect connections, hence check the modules too.
My nat tables contain nothing suspicious (given below).
In filter tables I see connection tracking and related modules are loaded. But I think tcpdump outputs everything, filtered or not. Is that right?
Anyway I did this preparation for the next test:
root@runesave:~# ip6tables -P INPUT ACCEPT
root@runesave:~# ip6tables -P OUTPUT ACCEPT
root@runesave:~# ip6tables -P FORWARD ACCEPT
root@runesave:~# ip6tables -F
root@runesave:~# ip6tables -X
root@runesave:~# iptables -F
root@runesave:~# iptables -X
root@runesave:~# rmmod nf_nat_pptp
root@runesave:~# rmmod nf_conntrack_pptp
root@runesave:~# rmmod nf_nat_proto_gre
root@runesave:~# rmmod nf_conntrack_proto_gre
root@runesave:~# rmmod nf_conntrack_ipv6
root@runesave:~# lsmod
Module Size Used by Not tainted
ip6t_rt 4096 0 - Live 0x86624000
ip6table_filter 4096 0 - Live 0x86622000
ip6_tables 12288 2 ip6t_rt,ip6table_filter, Live 0x8660c000
etherip 8192 0 - Live 0x87b74000
ext2 40960 2 - Live 0x874c0000
mbcache 8192 0 - Live 0x87488000
sit 12288 0 - Live 0x80748000
ipv6 266240 21 sit, Live 0x87900000
usb_storage 32768 3 - Live 0x81340000
sd_mod 20480 4 - Live 0x81338000
scsi_wait_scan 416 0 - Live 0x812ee000
scsi_mod 73728 3 usb_storage,sd_mod,scsi_wait_scan, Live 0x81320000
ehci_hcd 32768 0 - Live 0x812b0000
usbcore 106496 3 usb_storage,ehci_hcd, Live 0x812c0000
switch_robo 8192 0 - Live 0x87dcc000
switch_core 8192 1 switch_robo, Live 0x87dd4000
bcm57xx 110592 0 - Live 0x81260000
nat chains:
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT icmp -- anywhere 10.2.2.90 to:192.168.1.1
DNAT tcp -- anywhere 10.2.2.90 tcp dpt:10250 to:192.168.1.2:10250
DNAT udp -- anywhere 10.2.2.90 udp dpt:4444 to:192.168.1.2:4444
DNAT udp -- anywhere 10.2.2.90 udp dpt:6881 to:192.168.1.2:6881
DNAT tcp -- anywhere 10.2.2.90 tcp dpt:13588 to:192.168.1.2:13588
DNAT udp -- anywhere 10.2.2.90 udp dpt:3478 to:192.168.1.20:3478
DNAT udp -- anywhere 10.2.2.90 udp dpt:3479 to:192.168.1.20:3479
DNAT udp -- anywhere 10.2.2.90 udp dpt:3658 to:192.168.1.20:3658
DNAT tcp -- anywhere 10.2.2.90 tcp dpt:https to:192.168.1.20:443
DNAT tcp -- anywhere 10.2.2.90 tcp dpt:5223 to:192.168.1.20:5223
DNAT tcp -- anywhere 10.2.2.90 tcp dpt:15874 to:192.168.1.3:30667
DNAT udp -- anywhere 10.2.2.90 udp dpt:15874 to:192.168.1.3:30667
TRIGGER all -- anywhere 10.2.2.90 [16 bytes of unknown target data]
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- anywhere anywhere to:10.2.2.90
RETURN all -- anywhere anywhere PKTTYPE = broadcast
MASQUERADE all -- 192.168.1.0/24 192.168.1.0/24
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Then I re-did the above test. Still no captures.
> # tcpdump -v -i vlan2 host 77.109.111.178
What about other hosts that might give you indications that there are issues, eg by sending you ICMP messages?
I don't understand.
If there's an error with incoming IPv4 packet with proto 41, ICMP errors will be sent to PoP.
And if there's an error with outgoing IPv4, I'll first see the incoming packet in tcpdump. Am I missing something?
A NATted IP ddress, hello!
How are you enforcing that packets are always NATted towards your host?
My ISP does the translation both ways, I pay them for a personal global IPv4 address. Other clients don't need it and inside the ISP network we all have 10.*.*.*.
> 5. Ran the same ping6 on outside host which *has no* connectivity to my /48.
What is the traceroute or other details that come along with this host?
It was some posts above:
traceroute to tux1.xtsubasa.org (2a02:578:5418:d1:5054:a2ff:fe00:1), 30 hops max, 80 byte packets
1 2a00:ab00:108::1 (2a00:ab00:108::1) 2.061 ms 1.315 ms 2.020 ms
2 2a00:ab00:0:18::2 (2a00:ab00:0:18::2) 8.664 ms 8.658 ms 2a00:ab00:0:15::2 (2a00:ab00:0:15::2) 10.537 ms
3 router03.msk.ru.edpnet.net (2001:7f8:20:102::246:16) 9.938 ms 11.981 ms 11.716 ms
4 2a02:578:1:48::2 (2a02:578:1:48::2) 94.709 ms 94.668 ms 94.673 ms
5 2a02:578:1:46::1 (2a02:578:1:46::1) 19.266 ms 19.244 ms 17.671 ms
6 * * *
7 * * *
8 * * *
You can also use this looking glass:
http://noc.runnet.ru/lg/
If you choose routers msk* and ams* it's ok.
If you choose the other two, it fails.
Target host is tux1.xtsubasa.org as always.
/48 subnet is not routed
My nat tables contain nothing suspicious (given below).
Does not matter if they do, the moment connection tracker is active, packets are subject to it and connections will expire when you don't want them to. Unloading the modules does not solve the problem as there is some code that remains loaded.
There is a FAQ article about this problem.
# iptables -t nat -L
As there are no public IP addresses there, which host does the NAT to the public address?
Note that that devices also is a NAT/connection-tracker and can cause magic issues.
If there's an error with incoming IPv4 packet with proto 41, ICMP errors will be sent to PoP.
Intermediate hops can cause ICMP and other packets too. Hence why one needs to look at it all.
My ISP does the translation both ways, I pay them for a personal global IPv4 address. Other clients don't need it and inside the ISP network we all have 10.*.*.*.
Likely there is your connection tracker issue. Are you 100% sure they translate all packets properly?
It was some posts above:
What is the source address?
/48 subnet is not routed
Shadow Hawkins on Sunday, 05 October 2014 06:02:15
Jeroen Massar wrote:
> My nat tables contain nothing suspicious (given below).
Does not matter if they do, the moment connection tracker is active, packets are subject to it and connections will expire when you don't want them to. Unloading the modules does not solve the problem as there is some code that remains loaded.
There is a FAQ article about this problem.
I know about this case: https://www.sixxs.net/faq/connectivity?faq=conntracking
But since the connectivity to /48 is not restored after outgoing traffic is generated I don't see how it is related.
At the same time /64 is connected and NAT/conntrack should not distinguish between them.
I executed "iptables -t raw -A PREROUTING --proto 41 -j NOTRACK" as suggested there (other confguration has not changed since last post).
> # iptables -t nat -L
As there are no public IP addresses there, which host does the NAT to the public address?
Note that that devices also is a NAT/connection-tracker and can cause magic issues.
I don't know about it, probably some Cisco/Juniper hardware.
> If there's an error with incoming IPv4 packet with proto 41, ICMP errors will be sent to PoP.
Intermediate hops can cause ICMP and other packets too. Hence why one needs to look at it all.
I looked with "tcpdump -i vlan2 icmp", did the ping6 test (same configuration as last time, only the above NOTRACK rule added). Still no captures.
Likely there is your connection tracker issue. Are you 100% sure they translate all packets properly?
Well, if you ask me I'm pretty sure. Let me summarize.
Let's assume that IPv6 routing in EDPnet is correct and the problem packets are routed all the way to my DD-WRT (or at least my ISP). Then we have 2 issues at the same time:
1. When doing the traceroute6 test from various outside hosts in several locations (not all hosts but some) to my /48 destination (not /64), ruled01 does not reply or the reply is always lost.
2. There is a connection tracker issue or some misconfiguration or bug on my side or my ISP's that causes IPv6 packets destined for /48 filtered out on IPv4 level. It also lets packets destined for /64 or tunnel endpoint pass.
Both issues cause problems for /48 and not for /64.
My opinion is that the chances of that happening simultaneously is much closer to 0 than EDPnet routing being incorrect.
What is the source address?
It's 2a00:ab00:108:31:186:97:243:1
/48 subnet is not routed
I don't know about it, probably some Cisco/Juniper hardware.
If you do not control the NAT, proto-41 tunnels are very hard to make work reliably.
Just do not use proto-41 for tunneling in that situation.
/48 subnet is not routed
Shadow Hawkins on Thursday, 09 October 2014 23:39:58
I also did a few reverse tests (ping and traceroute from hosts in /48 and /64 to the host mentioned above plus another problem host).
1. When doing the ping6, I could catch both incoming and outgoing IPv6 on both outside hosts but on tunnel endpoint's IPv4 interface only outgoing packets pass through.
2. Traceoutes:
traceroute #1, from /64 to outside host A
traceroute to elben (2a00:ab00:108:31:186:97:243:1), 30 hops max, 80 byte packets
1 runesave-int-fixed (2a02:578:5002:8062::1) 0.443 ms 0.664 ms 0.664 ms
2 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 16.643 ms 16.839 ms 17.038 ms
3 ruled01.sixxs.net (2a02:578:5001:2::2) 17.360 ms 17.357 ms 17.354 ms
4 2a02:578:5001:2::1 (2a02:578:5001:2::1) 18.862 ms 19.075 ms 19.083 ms
5 2a02:578:1:24::1 (2a02:578:1:24::1) 19.073 ms 19.061 ms 18.836 ms
6 2a02:578:1:48::1 (2a02:578:1:48::1) 26.837 ms 26.380 ms 26.233 ms
7 brz-rt-msk.selectel.ru (2001:7f8:20:101::245:162) 26.612 ms 25.143 ms 24.830 ms
8 2a00:ab00:0:18::1 (2a00:ab00:0:18::1) 32.942 ms 32.940 ms 32.917 ms
9 2a00:ab00:108:31:186:97:243:1 (2a00:ab00:108:31:186:97:243:1) 36.614 ms 35.623 ms 35.816 ms
traceroute #2, from /48 to outside host A
traceroute to elben.xtsubasa.org (2a00:ab00:108:31:186:97:243:1), 30 hops max, 80 byte packets
1 melforce.xtsubasa.org (2a02:578:5002:8062::2) 0.276 ms 0.252 ms 0.265 ms
2 runesave.wolf (2a02:578:5002:8062::1) 0.502 ms 0.700 ms 0.703 ms
3 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 16.962 ms 16.966 ms 17.144 ms
4 ruled01.sixxs.net (2a02:578:5001:2::2) 17.430 ms 17.428 ms 17.423 ms
5 * 2a02:578:5001:2::1 (2a02:578:5001:2::1) 18.957 ms 18.955 ms
6 2a02:578:1:24::1 (2a02:578:1:24::1) 21.274 ms 17.668 ms 17.220 ms
7 2a02:578:1:48::1 (2a02:578:1:48::1) 24.910 ms 24.870 ms 24.859 ms
8 * * *
9 2a00:ab00:0:18::1 (2a00:ab00:0:18::1) 33.803 ms 33.804 ms 33.793 ms
10 * * *
11 * * *
12 * * *
traceroute #3, from /64 to outside host B
traceroute to luna (2a03:b0c0:2:d0::e:2001), 30 hops max, 80 byte packets
1 runesave-int-fixed (2a02:578:5002:8062::1) 0.439 ms 0.698 ms 0.706 ms
2 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 16.743 ms 16.749 ms 17.066 ms
3 ruled01.sixxs.net (2a02:578:5001:2::2) 17.337 ms 17.336 ms 17.332 ms
4 2a02:578:5001:2::1 (2a02:578:5001:2::1) 18.988 ms 18.969 ms 18.980 ms
5 2a02:578:1:25::1 (2a02:578:1:25::1) 163.331 ms 163.332 ms 163.646 ms
6 2a02:578:1:39::1 (2a02:578:1:39::1) 30.305 ms 30.121 ms 29.606 ms
7 2a02:578:1:2f::2 (2a02:578:1:2f::2) 59.247 ms 57.694 ms 58.298 ms
8 2a02:578:1:39::1 (2a02:578:1:39::1) 73.771 ms 73.778 ms 73.777 ms
9 40ge1-3.core1.lon2.he.net (2001:7f8:4::1b1b:1) 96.675 ms 95.070 ms 95.064 ms
10 100ge3-2.core1.ams1.he.net (2001:470:0:2d0::2) 84.269 ms 90.524 ms 90.527 ms
11 2001:7f8:1:0:a500:20:2018:1 (2001:7f8:1:0:a500:20:2018:1) 75.020 ms 75.700 ms 77.500 ms
12 2a03:b0c0:2::502 (2a03:b0c0:2::502) 77.130 ms 77.193 ms 2a03:b0c0:2::702 (2a03:b0c0:2::702) 75.450 ms
13 luna.xtsubasa.org (2a03:b0c0:2:d0::e:2001) 76.693 ms 76.679 ms 76.675 ms
traceroute #4, from /48 to outside host B
traceroute to luna.xtsubasa.org (2a03:b0c0:2:d0::e:2001), 30 hops max, 80 byte packets
1 melforce.xtsubasa.org (2a02:578:5002:8062::2) 0.201 ms 0.169 ms 0.161 ms
2 runesave.wolf (2a02:578:5002:8062::1) 0.542 ms 0.537 ms 0.840 ms
3 gw-99.led-01.ru.sixxs.net (2a02:578:5002:62::1) 17.128 ms 17.135 ms 17.128 ms
4 ruled01.sixxs.net (2a02:578:5001:2::2) 17.240 ms 17.238 ms 17.530 ms
5 2a02:578:5001:2::1 (2a02:578:5001:2::1) 18.366 ms * 18.354 ms
6 2a02:578:1::1 (2a02:578:1::1) 18.679 ms 17.444 ms 17.420 ms
7 * * *
8 2a02:578:1:2f::2 (2a02:578:1:2f::2) 58.921 ms 58.943 ms 58.931 ms
9 * * *
10 40ge1-3.core1.lon2.he.net (2001:7f8:4::1b1b:1) 85.923 ms 86.685 ms 86.943 ms
11 100ge3-2.core1.ams1.he.net (2001:470:0:2d0::2) 85.414 ms 85.375 ms 85.358 ms
12 2001:7f8:1:0:a500:20:2018:1 (2001:7f8:1:0:a500:20:2018:1) 77.139 ms 77.075 ms 78.065 ms
13 2a03:b0c0:2::702 (2a03:b0c0:2::702) 76.112 ms 76.486 ms *
14 * * *
15 * * *
16 * * *
These are reproducible: missing hosts in the middle always miss.
I'm not sure how to interpret this.
Also tried edpnet's looking glass: http://lg.edpnet.net/
For /48 target trace fails with routers in Brussels, Saint-Petersburg and London (the page times out and does not output any traceroute result).
/48 subnet is not routed
Shadow Hawkins on Thursday, 02 October 2014 16:39:55
I checked with Wireshark.
I pinged the host 2a02:578:5418:d1:5054:a2ff:fe00:1 from a server where the traceroute was bad (above).
Incoming pings didn't arrive.
I also pinged the same host from another server where the traceroute was good.
I saw both pings and pongs to the correct address.
|
![[ch]](/s/countries/ch.gif)
on Thursday, 02 October 2014 03:01:45
Posting is only allowed when you are