Forum - SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug) :: SixXS

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Shadow Hawkins on Saturday, 19 October 2013 11:22:27

Hi all, I am using SixXS tunnels (AYIYA) to hook up to the linux boxes of my family members. These tunnels are great to overcome any firewall settings on the client side (which are usually mobile and I can't rely on any local firewall settings) and allow me to login over ssh without using any "Teamviewer" (or similar) service which I do not trust. The tunnels are started manually by the user when I request them to do so only. This year, Swisscom started to rollout IPv6 to end users using 6rd and this broke my connectivity to one of the boxes. Unfortunately, the Linux Network Manager has a bug (links see below) adding a default route with metric 1 (!) to the main routing table when native IPv6 connectivity is offered on the local LAN. The default route added by aiccu (correctly with metric 1024) is therefore never considered and it is impossible to get connectivity over SixXS. I have now implemented an elegant solution on the client linux boxes using policy based routing (This script goes e.g. into /usr/local/etc/aiccu-subnets.sh and you need to enable this in aiccu.conf with the setupscript option):

IFSIXXS=sixxs ## This table name must be MANUALLY added to /etc/iproute2/rt_tables before it can be used TSIXXS=tsixxs MYSIXXS=`/sbin/ip -6 addr show $IFSIXXS | \ sed -ne "s/ *inet6 $[0-9a-f:][0-9a-f:]*$\/[0-9][0-9]* scope global/\1/p"` if ip -6 rule | grep $TSIXXS then # deleting the rule does NOT change the referred table! ip -6 rule del table $TSIXXS fi ip -6 route flush table $TSIXXS ip -6 rule add from $MYSIXXS table $TSIXXS ip -6 route add default dev $IFSIXXS table $TSIXXS

I hope to help anyone suffering from the same problem. It took me quite a while to figure that out properly... Have fun! PS: Links to NM-Bug: ubuntu bug upstream bug

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Jeroen Massar SixXS Staff

on Monday, 21 October 2013 09:27:32

This year, Swisscom started to rollout IPv6 to end users using 6rd and this broke my connectivity to one of the boxes.

Hence why you should ALWAYS disable accepting Router advertisements (accept_ra in sysctl on Linux) on untrusted interfaces / networks not under your control.

Unfortunately, the Linux Network Manager has a bug (links see below) adding a default route with metric 1 (!) to the main routing table when native IPv6 connectivity is offered on the local LAN.

I do not per-se see that as a bug. Local network connectivity should IMHO be preferred over tunneled connectivity. If you do not want that connectivity you should not enable (accept_ra=0) it or de-pref it.

I have now implemented an elegant solution on the client linux boxes using policy based routing

If I get the script correctly, you are changing the table so that when an application selects the source address to be the tunnel endpoint, that the default route to be chosen is the default route over sixxs. This should always be the case, thus what does this change? Also, that still requires the application in question to select the correct source address, which is either application-configuration specific and/or based on destination address (and then the default should be correct too) and/or by luck of the fact that the address was added last. I think you are trying to resolve the wrong problem here...

PS: Links to NM-Bug: ubuntu bug upstream bug

Seems the gnome bugtracker is down, thus little I can see there.

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Shadow Hawkins on Wednesday, 06 November 2013 22:04:13

Hi Jeroen,

Local network connectivity should IMHO be preferred over tunneled connectivity. If you do not want that connectivity you should not enable (accept_ra=0) it or de-pref it.

I agree on this for newly initiated connections, but I disagree for "reply" packets. I would expect if a packet comes in from IP aaa to IP bbb over Interface iii, the reply packet should go out from IP bbb to IP aaa over Interface iii. Please tell me if I am wrong...

Yes.

This should always be the case, thus what does this change?

Actually, the correct source address is chosen. But the kernel selects the wrong outgoing interface. I believe this can only happen by selecting the wrong default route (i.e. the default route seems to be chosen without considering the actual source address).

Also, that still requires the application in question to select the correct source address, which is either application-configuration specific and/or based on destination address (and then the default should be correct too) and/or by luck of the fact that the address was added last.

I fully agree on this - the issue causing me headaches is the outgoing interface selection... Please excuse the lengthy example below, but I needed this to make the problem clear to myself, so I hope it is also clear to my readers...

+--------+ +--------+ +-------------+ | Client | --- v6 Internet --- | Router | --- | IPv6 native | +--------+ | +--------+ | | | | ssh server | +----------+ | | | SixXS TS | ----------------- | IPv6 Tunnel | +----------+ +-------------+

I will do simple ping6 from the Client to the ssh server, once using the "native" address, once the "tunnel" address as destination. I have anonymized all addresses using "sed" using hopefully reasonable replacement strings... For every test, I show the full IPv6 routing table on the ssh server. On the ssh server, I have established ip6tables rules to log the relevant packages to kern.log. This was easier for me than tcpdump. I did 4 tests: 1) No SixXS Tunnel on ssh server yet, ping from client to ssh server (native). 2) SixXS Tunnel established, NO manipulation of the routing tables, ping to both the native and the tunnel address on the ssh server. 3) SixXS Tunnel established, manually removed the "metric 1" route, ping as above. 4) SixXS Tunnel established, additionally set the policy based route, ping as above. You will see that in 2) and 3), the ping reply packet to the SixXS ping has the correct source address, but it goes out via the wrong interface: It goes out via eth2 instead of sixxs. This seems to me to be a bug in the Linux IPv6 implementation, but I am happy to be proven wrong... Test 1: No SixXS Tunnel yet, ping from "client" to "native" Routing Table

------------------ rules ----------------- 0: from all lookup local 32766: from all lookup main ------------------ tables ----------------- .................. local ................. local ::1 via :: dev lo proto none metric 0 local 2001:NATI:VE:SSH:SERVER:: via :: dev lo proto none metric 0 local 2001:NATI:VE:0:4801:8577:e561:4b06 via :: dev lo proto none metric 0 local fe80::LOCA:LE:SSH:SERVER:: via :: dev lo proto none metric 0 ff00::/8 dev eth2 metric 256 .................. main ................. 2001:NATI:VE::/64 dev eth2 proto kernel metric 256 expires 10688sec fe80::/64 dev eth2 proto kernel metric 256 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto static metric 1 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto kernel metric 1024 expires 1687sec

ping6 -c 1 2001:NATI:VE:SSH:SERVER::

Nov 5 23:47:04 omega kernel: [ 1296.873329] INPUT6 IN=eth2 OUT= MAC=DO:NT:CA:RE SRC=2001:CLIE:NT:: DST=2001:NATI:VE:SSH:SERVER:: LEN=104 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=634 SEQ=1 Nov 5 23:47:04 omega kernel: [ 1296.873418] OUTPUT6 IN= OUT=eth2 SRC=2001:NATI:VE:SSH:SERVER:: DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=634 SEQ=1

Test 2: SixXS Tunnel established, NO manipulation of the routing tables Routing Table

------------------ rules ----------------- 0: from all lookup local 32766: from all lookup main ------------------ tables ----------------- .................. local ................. local ::1 via :: dev lo proto none metric 0 local 2001:NATI:VE:SSH:SERVER:: via :: dev lo proto none metric 0 local 2001:NATI:VE:0:4801:8577:e561:4b06 via :: dev lo proto none metric 0 local 2001:1620:SIXX:S::2 via :: dev lo proto none metric 0 local fe80::LOCA:LE:SSH:SERVER:: via :: dev lo proto none metric 0 local fe80::1420:SIXX:S:2 via :: dev lo proto none metric 0 ff00::/8 dev eth2 metric 256 ff00::/8 dev sixxs metric 256 .................. main ................. 2001:NATI:VE::/64 dev eth2 proto kernel metric 256 expires 10716sec 2001:1620:SIXX:S::/64 dev sixxs proto kernel metric 256 fe80::/64 dev eth2 proto kernel metric 256 fe80::/64 dev sixxs proto kernel metric 256 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto static metric 1 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto kernel metric 1024 expires 1716sec default via 2001:1620:SIXX:S::1 dev sixxs metric 1024

ping6 -c 1 2001:NATI:VE:SSH:SERVER::

Nov 5 23:51:49 omega kernel: [ 1581.429319] INPUT6 IN=eth2 OUT= MAC=DO:NT:CA:RE SRC=2001:CLIE:NT:: DST=2001:NATI:VE:SSH:SERVER:: LEN=104 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=14202 SEQ=1 Nov 5 23:51:49 omega kernel: [ 1581.429402] OUTPUT6 IN= OUT=eth2 SRC=2001:NATI:VE:SSH:SERVER:: DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=14202 SEQ=1

ping6 -c 2001:1620:SIXX:S::2

Nov 5 23:52:57 omega kernel: [ 1649.963130] INPUT6 IN=sixxs OUT= MAC= SRC=2001:CLIE:NT:: DST=2001:1620:SIXX:S:0000:0000:0000:0002 LEN=104 TC=0 HOPLIMIT=56 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=17274 SEQ=1 Nov 5 23:52:57 omega kernel: [ 1649.963216] OUTPUT6 IN= OUT=eth2 SRC=2001:1620:SIXX:S:0000:0000:0000:0002 DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=17274 SEQ=1

I believe the last packet to be wrong: It takes the SixXS Tunnel Endpoint Source address, but goes out via eth2 instead of sixxs. Test 3 SixXS Tunnel established, manually removed the "metric 1" route

------------------ rules ----------------- 0: from all lookup local 32766: from all lookup main ------------------ tables ----------------- .................. local ................. local ::1 via :: dev lo proto none metric 0 local 2001:NATI:VE:SSH:SERVER:: via :: dev lo proto none metric 0 local 2001:NATI:VE:0:4801:8577:e561:4b06 via :: dev lo proto none metric 0 local 2001:1620:SIXX:S::2 via :: dev lo proto none metric 0 local fe80::LOCA:LE:SSH:SERVER:: via :: dev lo proto none metric 0 local fe80::1420:SIXX:S:2 via :: dev lo proto none metric 0 ff00::/8 dev eth2 metric 256 ff00::/8 dev sixxs metric 256 .................. main ................. 2001:NATI:VE::/64 dev eth2 proto kernel metric 256 expires 10773sec 2001:1620:SIXX:S::/64 dev sixxs proto kernel metric 256 fe80::/64 dev eth2 proto kernel metric 256 fe80::/64 dev sixxs proto kernel metric 256 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto kernel metric 1024 expires 1773sec default via 2001:1620:SIXX:S::1 dev sixxs metric 1024

ping6 -c 1 2001:NATI:VE:SSH:SERVER::

Nov 5 23:56:12 omega kernel: [ 1844.971287] INPUT6 IN=eth2 OUT= MAC=DO:NT:CA:RE SRC=2001:CLIE:NT:: DST=2001:NATI:VE:SSH:SERVER:: LEN=104 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=28538 SEQ=1 Nov 5 23:56:12 omega kernel: [ 1844.971372] OUTPUT6 IN= OUT=eth2 SRC=2001:NATI:VE:SSH:SERVER:: DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=28538 SEQ=1

ping6 -c 1 2001:1620:SIXX:S::2

Nov 5 23:56:42 omega kernel: [ 1874.301475] INPUT6 IN=sixxs OUT= MAC= SRC=2001:CLIE:NT:: DST=2001:1620:SIXX:S:0000:0000:0000:0002 LEN=104 TC=0 HOPLIMIT=56 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=30586 SEQ=1 Nov 5 23:56:42 omega kernel: [ 1874.301520] OUTPUT6 IN= OUT=eth2 SRC=2001:1620:SIXX:S:0000:0000:0000:0002 DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=30586 SEQ=1

Still wrong: correct source address, wrong outgoing interface Test 4, SixXS Tunnel established, additionally set the policy based route

------------------ rules ----------------- 0: from all lookup local 16383: from 2001:1620:SIXX:S::2 lookup tsixxs 32766: from all lookup main ------------------ tables ----------------- .................. local ................. local ::1 via :: dev lo proto none metric 0 local 2001:NATI:VE:SSH:SERVER:: via :: dev lo proto none metric 0 local 2001:NATI:VE:0:4801:8577:e561:4b06 via :: dev lo proto none metric 0 local 2001:1620:SIXX:S::2 via :: dev lo proto none metric 0 local fe80::LOCA:LE:SSH:SERVER:: via :: dev lo proto none metric 0 local fe80::1420:SIXX:S:2 via :: dev lo proto none metric 0 ff00::/8 dev eth2 metric 256 ff00::/8 dev sixxs metric 256 .................. main ................. 2001:NATI:VE::/64 dev eth2 proto kernel metric 256 expires 10574sec 2001:1620:SIXX:S::/64 dev sixxs proto kernel metric 256 fe80::/64 dev eth2 proto kernel metric 256 fe80::/64 dev sixxs proto kernel metric 256 default via fe80::20d:b9ff:fe17:ed82 dev eth2 proto kernel metric 1024 expires 1574sec default via 2001:1620:SIXX:S::1 dev sixxs metric 1024 .................. tsixxs ................. default dev sixxs metric 1024

ping6 -c 1 2001:NATI:VE:SSH:SERVER::

Nov 5 23:59:44 omega kernel: [ 2057.129892] INPUT6 IN=eth2 OUT= MAC=DO:NT:CA:RE SRC=2001:CLIE:NT:: DST=2001:NATI:VE:SSH:SERVER:: LEN=104 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=37242 SEQ=1 Nov 5 23:59:44 omega kernel: [ 2057.129981] OUTPUT6 IN= OUT=eth2 SRC=2001:NATI:VE:SSH:SERVER:: DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=37242 SEQ=1

ping6 -c 1 2001:1620:SIXX:S::2

Nov 6 00:01:03 omega kernel: [ 2135.220816] INPUT6 IN=sixxs OUT= MAC= SRC=2001:CLIE:NT:: DST=2001:1620:SIXX:S:0000:0000:0000:0002 LEN=104 TC=0 HOPLIMIT=56 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=44154 SEQ=1 Nov 6 00:01:03 omega kernel: [ 2135.220907] OUTPUT6 IN= OUT=sixxs SRC=2001:1620:SIXX:S:0000:0000:0000:0002 DST=2001:CLIE:NT:: LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=44154 SEQ=1

Only now, the correct outgoing interface is chosen. I am happy to receive comments :-) Thanks!

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Jeroen Massar SixXS Staff

on Thursday, 07 November 2013 16:11:57

As stated in the previous reply, you need to disconfigure your other network if you do not want to use it. Which certainly looks like what you are doing. Much easier than creating a lot of rules around it.

This seems to me to be a bug in the Linux IPv6 implementation, but I am happy to be proven wrong...

The Linux kernel is merely forwarding the packet, it does not know what the properties of the source address are, eg that it is allowed over one or the other interface. Hence why in the situation where you have multiple upstreams that filter your address (it can be the case that this does not happen, which means a really bad ISP or that you have arranged that it is allowed), that you need to do source routing.

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Shadow Hawkins on Thursday, 07 November 2013 21:26:36

Hi Jeroen, first of all, thanks for your patience with me.

I believe I have understood now my original misconception: I was dreaming that for locally generated packages (where the source address will be identical to one of the local addresses), the route selection mechanism should prefer routes that send the packet out of the interface having that said address (at least when they have the same metric). But the route selection is based on the destination address only (as long as there is no policy routing). Thanks! PS: No bug in Linux Kernel, no (related) bug in Network-Manager, simply my misconception :-)

SixXS-Tunnel to coexist with native IPv6 using policy based routing (with network manager bug)

Shadow Hawkins on Friday, 15 November 2013 04:16:45

I am too tired right now to follow everything you said and did, but I am using policy-based routing to handle both a SixXS and an HE tunnel. I had one "gotcha" where the default rule was removed when flushing the IPv6 policy table, behavior that didn't seem to happen when flushing IPv4 policy tables. I wrote about it and put my config info here: http://itkia.com/ipv6-policy-routing-linux-gotchas/ . It looks like someone else linked to me and described a situation like yours with an ISP IPv6 address and a tunnel: http://blog.altimos.de/2013/07/ipv6-source-address-routing-with-multiple-uplinks-sixxs/

Please note Posting is only allowed when you are logged in.