Forum - ping6: No buffer space available when pinging PoP endpoint :: SixXS

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Monday, 28 April 2014 12:18:04

Hi all I'm trying to get my tunnel on pfSense working again. Currently, if I'm trying to ping the PoP tunnel endpoint I get:

PING6(56=40+8+8 bytes) 2001:1620:f00:f::2 --> 2001:1620:f00:f::1 ping6: sendmsg: No buffer space available ping6: wrote 2001:1620:f00:f::1 16 chars, ret=-1 ping6: sendmsg: No buffer space available

ifconfig shows:

tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 options=80000<LINKSTATE> inet6 fe80::250:56ff:fea1:7155%tun0 prefixlen 64 scopeid 0xb inet6 fe80::1420:f00:f:2%tun0 prefixlen 64 scopeid 0xb inet6 2001:1620:f00:f::2 --> 2001:1620:f00:f::1 prefixlen 128 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> Opened by PID 1050

uname -a:

FreeBSD f02.techfreak.local 8.3-RELEASE-p15 FreeBSD 8.3-RELEASE-p15 #1: Thu Apr 10 06:15:06 EDT 2014 root@pf2.1.1_amd64.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64

aiccu test:

Tunnel Information for T25921: POP Id : chzrh02 IPv6 Local : 2001:1620:f00:f::2/64 IPv6 Remote : 2001:1620:f00:f::1/64 Tunnel Type : ayiya Adminstate : enabled Userstate : enabled

traceroute PoP:

traceroute to 213.144.148.74 (213.144.148.74), 64 hops max, 52 byte packets 1 * * * 2 217-168-57-45.static.cablecom.ch (217.168.57.45) 13.908 ms 7.551 ms 8.279 ms 3 84.116.202.237 (84.116.202.237) 8.502 ms 8.057 ms 7.941 ms 4 ch-zrh01b-ra1-ae-9-0.aorta.net (84.116.134.22) 8.746 ms 7.461 ms 7.964 ms 5 213.46.171.14 (213.46.171.14) 17.767 ms 13.669 ms 7.987 ms 6 r1zlz1.core.init7.net (77.109.128.210) 32.373 ms 52.661 ms 21.984 ms 7 chzrh02.sixxs.net (213.144.148.74) 15.266 ms 6.999 ms 8.144 ms

aiccu version:

AICCU 2007.01.15-console-kame by Jeroen Massar

I tried disabling the packetfilter (pfctl -d) I have another pfSense box, which works. Tunnel status page says that it receives AICCU heartbeats, but no packets in or out TCPDump shows:

Any ideas on how to troubleshoot that? Thanks! Michel

ping6: No buffer space available when pinging PoP endpoint

Jeroen Massar SixXS Staff

on Monday, 28 April 2014 12:32:01

ping6: sendmsg: No buffer space available

Interesting. I have seen that happen on Linux, but not on FreeBSD yet.

FreeBSD f02.techfreak.local 8.3-RELEASE-p15 FreeBSD 8.3-RELEASE-p15

Side-note: aren't they on 10.x already btw? It should do IPv6 perfectly fine mind you. (Unless there are bugs out there that I am not aware of, which, with FreeBSD-alike stuff is likely, don't use it that much anymore) Though google teaches us that things like this happened before: https://www.sixxs.net/forum/?msg=setup-527467 http://lists.freebsd.org/pipermail/freebsd-current/2010-April/016886.html

TCPDump shows:

That is the *IN*coming packet from the PoP, which your host is expected to reply to. The above indicates that you are not able to send packets outbound though.

Any ideas on how to troubleshoot that?

Provide a lot more details. Routing tables, addresses in which interfaces, firewall (IPv4+IPv6) etc etc. Note that AICCU only configures things, it has no way of knowing that other settings that are made are in conflict with it's operation.

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Monday, 28 April 2014 13:37:05

Jeroen Massar wrote:

> ping6: sendmsg: No buffer space available Interesting. I have seen that happen on Linux, but not on FreeBSD yet.

FreeBSD f02.techfreak.local 8.3-RELEASE-p15 FreeBSD 8.3-RELEASE-p15

TCPDump shows:

That is the *IN*coming packet from the PoP, which your host is expected to reply to. The above indicates that you are not able to send packets outbound though.

Any ideas on how to troubleshoot that?

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Monday, 28 April 2014 13:52:35

Jeroen Massar wrote:

Side-note: aren't they on 10.x already btw?

No, they are moving there, but I'm using the current 2.1.2 version which is still 8.3 based

Though google teaches us that things like this happened before: https://www.sixxs.net/forum/?msg=setup-527467 http://lists.freebsd.org/pipermail/freebsd-current/2010-April/016886.html

Yes, those don't help me very much unfortunately :(

> TCPDump shows: That is the *IN*coming packet from the PoP, which your host is expected to reply to. The above indicates that you are not able to send packets outbound though.

Here's one with my ping... dunno if that helps :)

15:41:59.698689 IP6 2001:1620:f00:f::2 > 2001:1620:f00:f::1: ICMP6, neighbor solicitation, who has 2001:1620:f00:f::1, length 24 0x0000: 6000 0000 0018 3aff 2001 1620 0f00 000f `.....:......... 0x0010: 0000 0000 0000 0002 2001 1620 0f00 000f ................ 0x0020: 0000 0000 0000 0001 8700 a918 0000 0000 ................ 0x0030: 2001 1620 0f00 000f 0000 0000 0000 0001 ................ 15:41:59.708696 IP6 2001:1620:f00:f::2 > 2001:1620:f00:f::1: ICMP6, echo request, seq 7, length 16 0x0000: 6000 0000 0010 3a40 2001 1620 0f00 000f `.....:@........ 0x0010: 0000 0000 0000 0002 2001 1620 0f00 000f ................ 0x0020: 0000 0000 0000 0001 8000 6443 12b4 0007 ..........dC.... 0x0030: 535e 5aa7 000a d043 S^Z....C

As i've disabled the FW, I don't think that FW rules help. Also, there are quite a lot of them :) The important interfaces are:

Routing table is:

Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 84.73.24.1 UGS 0 60761287 em1 62.2.17.60 00:50:56:a1:71:56 UHS 0 13 em1 62.2.17.61 00:50:56:a1:71:56 UHS 0 743 em1 62.2.24.158 00:50:56:a1:71:56 UHS 0 21 em1 62.2.24.162 00:50:56:a1:71:56 UHS 0 0 em1 84.73.24.0/21 link#2 U 0 262498 em1 84.73.24.150 link#2 UHS 0 0 lo0 127.0.0.1 link#7 UH 0 0 lo0 172.16.0.0/24 192.168.50.2 UGS 0 4987 em2 172.16.1.0/24 192.168.50.2 UGS 0 2175 em2 172.16.2.0/24 192.168.50.2 UGS 0 3000 em2 172.16.3.0/24 192.168.50.2 UGS 0 2313 em2 172.16.4.0/24 192.168.50.2 UGS 0 1549 em2 172.16.5.0/24 192.168.50.2 UGS 0 2448 em2 172.16.6.0/24 192.168.50.2 UGS 0 1488172 em2 172.16.50.0/24 192.168.50.2 UGS 0 2198 em2 192.168.0.0/24 192.168.50.2 UGS 0 2767618 em2 192.168.2.0/24 192.168.50.2 UGS 0 1743 em2 192.168.3.0/24 192.168.50.2 UGS 0 43234 em2 192.168.4.0/24 192.168.50.2 UGS 0 1938 em2 192.168.10.0/24 192.168.50.2 UGS 0 5184 em2 192.168.50.0/24 link#3 U 0 318820 em2 192.168.50.3 link#3 UHS 0 0 lo0 192.168.79.0/24 192.168.79.2 UGS 0 0 ovpns1 192.168.79.1 link#10 UHS 0 0 lo0 192.168.79.2 link#10 UH 0 0 ovpns1 192.168.80.0/24 192.168.50.2 UGS 0 35531 em2 192.168.81.0/24 192.168.50.2 UGS 0 9883 em2 192.168.82.0/24 link#1 U 0 54626090 em0 192.168.82.254 link#1 UHS 0 524996 lo0 192.168.83.0/24 192.168.50.2 UGS 0 6529 em2 192.168.84.0/24 192.168.50.2 UGS 0 1274 em2 192.168.96.0/20 192.168.50.2 UGS 0 7271 em2 Internet6: Destination Gateway Flags Netif Expire default 2001:1620:f00:f::1 UGS tun0 ::1 ::1 UH lo0 2001:1620:f00:f::1 2001:1620:f00:f::2 UH tun0 2001:1620:f00:f::2 link#11 UHS lo0 2001:1620:f0a:1::/64 link#1 U em0 2001:1620:f0a:1::ffff link#1 UHS lo0 2001:1620:f0a:ffff::/64 link#3 U em2 2001:1620:f0a:ffff::3 link#3 UHS lo0 2a02:418:3002::/48 2001:1620:f0a:ffff::2 UGS em2 2a02:2528:ff48::/48 00:50:56:a1:71:57 UGS em2 fe80::%em0/64 link#1 U em0 fe80::250:56ff:fea1:7155%em0 link#1 UHS lo0 fe80::%em1/64 link#2 U em1 fe80::250:56ff:fea1:7156%em1 link#2 UHS lo0 fe80::%em2/64 link#3 U em2 fe80::250:56ff:fea1:7157%em2 link#3 UHS lo0 fe80::%lo0/64 link#7 U lo0 fe80::1%lo0 link#7 UHS lo0 fe80::250:56ff:fea1:7155%ovpns1 link#10 UHS lo0 fe80::%tun0/64 link#11 U tun0 fe80::250:56ff:fea1:7155%tun0 link#11 UHS lo0 fe80::1420:f00:f:2%tun0 link#11 UHS lo0 ff01::%em0/32 fe80::250:56ff:fea1:7155%em0 U em0 ff01::%em1/32 fe80::250:56ff:fea1:7156%em1 U em1 ff01::%em2/32 fe80::250:56ff:fea1:7157%em2 U em2 ff01::%lo0/32 ::1 U lo0 ff01::%ovpns1/32 fe80::250:56ff:fea1:7155%ovpns1 U ovpns1 ff01::%tun0/32 fe80::250:56ff:fea1:7155%tun0 U tun0 ff02::%em0/32 fe80::250:56ff:fea1:7155%em0 U em0 ff02::%em1/32 fe80::250:56ff:fea1:7156%em1 U em1 ff02::%em2/32 fe80::250:56ff:fea1:7157%em2 U em2 ff02::%lo0/32 ::1 U lo0 ff02::%ovpns1/32 fe80::250:56ff:fea1:7155%ovpns1 U ovpns1 ff02::%tun0/32 fe80::250:56ff:fea1:7155%tun0 U tun0

I would like to provide all information you need, however I don't know what you need :) (sorry for the other post, I'm still getting used to this forum software :) )

ping6: No buffer space available when pinging PoP endpoint

Jeroen Massar SixXS Staff

on Monday, 28 April 2014 14:57:39

Here's one with my ping... dunno if that helps :)

Depends. Is there a response? Note that tcpdumping the tunnel interface itself is not very useful; that is, if you want to know the packet possibly made it out of the system. (some systems have pre-firewall tcpdump, others after-firewall tcpdump).

As i've disabled the FW, I don't think that FW rules help. Also, there are quite a lot of them :)

Depends on your default policy of course, if that is dropping things, it is not very useful. At least, do check them. Counters and logging are useful things. You are on BSD, thus it might not affect too much; but on Linux for instance adding certain firewall rules adds for instance connection tracking, which are permanently affecting the network stack even after the actual rules are flushed. [..]

2a02:418:3002::/48 2001:1620:f0a:ffff::2 UGS em2

2a02:2528:ff48::/48 00:50:56:a1:71:57 UGS em2

Likely unrelated, but unless you are null-routing these, where do these really go? Are they there on purpose?

I would like to provide all information you need, however I don't know what you need :)

There are these big yellow/orange bars when posting. They contain (repeated) a very big hint.

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Monday, 28 April 2014 18:20:09

I'm sorry for not including every detail in the FAQ in my first post. Let me start over by doing this then. I really appreciate your help.

Always include clear descriptive information about your problem or inquiry.

I currently cannot get my tunnel working on pfSense. It was running at one point in time until about a week ago. I did have more than 50 weeks uptime on the tunnel. One thing that I've changed in about that timeframe (It was before that though), was upgrading pfSense from 2.1 to 2.1.1 and subsequently to 2.1.2 I have done that on 2 of my pfSense boxes. One is still working, one is not though. When I try to ping the PoP endpoint, I get:

PING6(56=40+8+8 bytes) 2001:1620:f00:f::2 --> 2001:1620:f00:f::1 ping6: sendmsg: No buffer space available ping6: wrote 2001:1620:f00:f::1 16 chars, ret=-1 ping6: sendmsg: No buffer space available

Check the "Live Tunnel Status", found by clicking on the Tunnel Details in your User Home, this shows the status of the tunnel as the PoP sees it.

This is the Live Tunnel Status. I don't know what this tells me though, other than what I already know (Tunnel not working)

Live Tunnel Status for T25921 The PoP reports the following status for your tunnel: Tunnel Configuration Tunnel ID T25921 TID 0xf Tunnel Debugging no Inner Us 2001:1620:f00:f::1 Inner Them 2001:1620:f00:f::2 Outer Us 213.144.148.74 Outer Them 84.73.24.150 MTU 1280 Tunnel State up Tunnel Type ayiya AYIYA AF 2 (INET) AYIYA Socket Type 2 (DGRAM (UDP)) AYIYA Protocol 17 (UDP) AYIYA Port Us 5072 AYIYA Port Them 34937 AYIYA Hash 2 (SHA-1) Heartbeat Information (Heartbeat and AYIYA protocols only) Last Heartbeat 2014-04-28 17:13:34 (1398705214; 0 days 00:00:01 ago) Heartbeat Password <password> Tunnel Traffic (last 5 minutes) Packet In 2014-04-25 07:29:35 (1398410975; 3 days 09:44:00 ago) Packets In 0 Octets In 0 Packet Out 2014-04-28 17:13:34 (1398705214; 0 days 00:00:01 ago) Packets Out 13 Octets Out 14300 Tunnel Latency (last 5 minutes) Latency Pkt Sent 13 Latency Pkt Recv 0 Encap.Pkt Too Big 226, last: 2400:cb00:2048:1::be5d:f251 2014-04-25 07:29:35 (1398410975; 3 days 09:44:00 ago) Errors seen for this tunnel Disabled tunnel none Clock Off none Encap.Pkt Send Error none Same In&Out Interface none Wrong Source IPv6 none Wrong Source IPv4 none Packet over uplink none Non-IPv6 Payload none Non-IPv4 Payload none AYIYA Hash Fail none AYIYA-non-AYIYA none AYIYA Invalid Forward none Heartbeat Hash Fail none HB-non-HB none HB Missing IPv4 none HB Sender Mismatch none HB Missing Time none ICMPv4 Errors Received none ICMPv4 Echo Req. Recv. none

When using AICCU, check that you are using the latest version, and of course please specify where the binary comes from, also run it with the 'verbose true' flag and provide the output of that.

Tunnel Information for T25921: POP Id : chzrh02 IPv6 Local : 2001:1620:f00:f::2/64 IPv6 Remote : 2001:1620:f00:f::1/64 Tunnel Type : ayiya Adminstate : enabled Userstate : enabled

Always provide your NIC handle and if applicable the Tunnel or Route IDs you wish to discuss.

FRAG82-RIPE / T25921

Use the email address you have provided in your handle as that is what we use as a contact handle.

I guess this only for the contact form.

Provide details of the setup, type of connections, where NATs are located.

I have a Cablecom Cable-connection. I use a virtual pfSense Box on VMware ESXi 5.5. The cable-connection is attached to VMware using VLAN through 2 switches. There is no nat, the pfSense Box has an external IPv4. Please let me know if you need more information, as I can't think of anything else to write here.

Provide information of your OS type, version and release (ie. uname -a), noting also the distribution name

I use pfSense 2.1.2 which is the latest version as of writing this. uname -a:

FreeBSD f02.techfreak.local 8.3-RELEASE-p15 FreeBSD 8.3-RELEASE-p15 #1: Thu Apr 10 06:15:06 EDT 2014 root@pf2.1.1_amd64.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64

Include full interface, routing and firewall tables.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:a1:71:55 inet 192.168.82.254 netmask 0xffffff00 broadcast 192.168.82.255 inet6 fe80::250:56ff:fea1:7155%em0 prefixlen 64 scopeid 0x1 inet6 2001:1620:f0a:1::ffff prefixlen 64 nd6 options=1<PERFORMNUD> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:a1:71:56 inet 84.73.24.150 netmask 0xfffff800 broadcast 255.255.255.255 inet6 fe80::250:56ff:fea1:7156%em1 prefixlen 64 scopeid 0x2 nd6 options=1<PERFORMNUD> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:a1:71:57 inet 192.168.50.3 netmask 0xffffff00 broadcast 192.168.50.255 inet6 fe80::250:56ff:fea1:7157%em2 prefixlen 64 scopeid 0x3 inet6 2001:1620:f0a:ffff::3 prefixlen 64 nd6 options=1<PERFORMNUD> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:b5:31:dd media: Ethernet autoselect (1000baseT <full-duplex>) status: active plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500 pfsync0: flags=0<> metric 0 mtu 1460 syncpeer: 224.0.0.240 maxupd: 128 syncok: 1 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3<RXCSUM,TXCSUM> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> pflog0: flags=100<PROMISC> metric 0 mtu 33144 enc0: flags=0<> metric 0 mtu 1536 ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet6 fe80::250:56ff:fea1:7155%ovpns1 prefixlen 64 scopeid 0xa inet 192.168.79.1 --> 192.168.79.2 netmask 0xffffffff nd6 options=3<PERFORMNUD,ACCEPT_RTADV> Opened by PID 88243 tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 options=80000<LINKSTATE> inet6 fe80::250:56ff:fea1:7155%tun0 prefixlen 64 scopeid 0xb inet6 fe80::1420:f00:f:2%tun0 prefixlen 64 scopeid 0xb inet6 2001:1620:f00:f::2 --> 2001:1620:f00:f::1 prefixlen 128 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> Opened by PID 1050

Firewall: I have looked at the Firewall logs, and ICMPv6 traffic is registered as PASS in the log:

Apr 28 19:36:35 AICCU_SIXXS PASS [2001:1620:f00:f::1] [2001:1620:f00:f::2] ICMPv6

I have added an explicit rule:

pass in quick on tun0 inet6 proto ipv6-icmp from 2001:1620:f00:f::1 to 2001:1620:f00:f::2 keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"

I also have a reverse rule:

pass in log quick on tun0 inet6 proto ipv6-icmp from any to 2001:1620:f00:f::2 keep state label "USER_RULE: sixxs ping FW"

I also have a general IPv6 ICMP rule:

pass log quick on tun0 inet6 proto ipv6-icmp all keep state label "USER_RULE: Any"

Also, please remember that this used to work, and I did not change rules. For completeness, all rules:

scrub on em1 all fragment reassemble scrub on em0 all fragment reassemble scrub on em2 all fragment reassemble scrub on tun0 all fragment reassemble anchor "relayd/*" all anchor "openvpn/*" all anchor "ipsec/*" all block drop in log inet all label "Default deny rule IPv4" block drop out log inet all label "Default deny rule IPv4" block drop in log inet6 all label "Default deny rule IPv6" block drop out log inet6 all label "Default deny rule IPv6" pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echorep keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echorep keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state pass quick inet6 proto ipv6-icmp all icmp6-type unreach keep state pass quick inet6 proto ipv6-icmp all icmp6-type toobig keep state pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state block drop quick inet proto tcp from any port = 0 to any block drop quick inet proto tcp from any to any port = 0 block drop quick inet proto udp from any port = 0 to any block drop quick inet proto udp from any to any port = 0 block drop quick inet6 proto tcp from any port = 0 to any block drop quick inet6 proto tcp from any to any port = 0 block drop quick inet6 proto udp from any port = 0 to any block drop quick inet6 proto udp from any to any port = 0 block drop quick from <snort2c> to any label "Block snort2c hosts" block drop quick from any to <snort2c> label "Block snort2c hosts" block drop in log quick proto tcp from <sshlockout> to (self) port = ssh label "sshlockout" block drop in log quick proto tcp from <webConfiguratorlockout> to (self) port = https label "webConfiguratorlockout" block drop in quick from <virusprot> to any label "virusprot overload table" block drop in log quick on em1 from <bogons> to any label "block bogon IPv4 networks from WAN" block drop in log quick on em1 from <bogonsv6> to any label "block bogon IPv6 networks from WAN" block drop in on ! em1 inet from 84.73.24.0/21 to any block drop in inet from 84.73.24.150 to any block drop in on em1 inet6 from fe80::250:56ff:fea1:7156 to any block drop in log quick on em1 inet from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8" block drop in log quick on em1 inet from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8" block drop in log quick on em1 inet from 100.64.0.0/10 to any label "Block private networks from WAN block 100.64/10" block drop in log quick on em1 inet from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12" block drop in log quick on em1 inet from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16" block drop in log quick on em1 inet6 from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7" pass in on em1 proto udp from any port = bootps to any port = bootpc keep state label "allow dhcp client out WAN" pass out on em1 proto udp from any port = bootpc to any port = bootps keep state label "allow dhcp client out WAN" block drop in on ! em0 inet6 from 2001:1620:f0a:1::/64 to any block drop in on em0 inet6 from fe80::250:56ff:fea1:7155 to any block drop in inet6 from 2001:1620:f0a:1::ffff to any block drop in on ! em2 inet6 from 2001:1620:f0a:ffff::/64 to any block drop in on em2 inet6 from fe80::250:56ff:fea1:7157 to any block drop in inet6 from 2001:1620:f0a:ffff::3 to any block drop in on ! em0 inet from 192.168.82.0/24 to any block drop in inet from 192.168.82.254 to any block drop in on ! em2 inet from 192.168.50.0/24 to any block drop in inet from 192.168.50.3 to any block drop in log quick on tun0 from <bogons> to any label "block bogon IPv4 networks from AICCU_SIXXS" pass in quick on tun0 inet6 proto udp from fe80::/10 port = dhcpv6-client to fe80::/10 port = dhcpv6-client keep state label "allow dhcpv6 client in AICCU_SIXXS" pass in quick on tun0 proto udp from any port = dhcpv6-server to any port = dhcpv6-client keep state label "allow dhcpv6 client in AICCU_SIXXS" pass out quick on tun0 proto udp from any port = dhcpv6-client to any port = dhcpv6-server keep state label "allow dhcpv6 client out AICCU_SIXXS" block drop in log quick on tun0 from <bogonsv6> to any label "block bogon IPv6 networks from AICCU_SIXXS" block drop in on tun0 inet6 from fe80::250:56ff:fea1:7155 to any block drop in on tun0 inet6 from fe80::1420:f00:f:2 to any block drop in on ! tun0 inet6 from 2001:1620:f00:f::2 to any block drop in inet6 from 2001:1620:f00:f::2 to any block drop in log quick on tun0 inet from 10.0.0.0/8 to any label "Block private networks from AICCU_SIXXS block 10/8" block drop in log quick on tun0 inet from 127.0.0.0/8 to any label "Block private networks from AICCU_SIXXS block 127/8" block drop in log quick on tun0 inet from 100.64.0.0/10 to any label "Block private networks from AICCU_SIXXS block 100.64/10" block drop in log quick on tun0 inet from 172.16.0.0/12 to any label "Block private networks from AICCU_SIXXS block 172.16/12" block drop in log quick on tun0 inet from 192.168.0.0/16 to any label "Block private networks from AICCU_SIXXS block 192.168/16" block drop in log quick on tun0 inet6 from fc00::/7 to any label "Block ULA networks from AICCU_SIXXS block fc00::/7" pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback" pass in on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback" pass out on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback" pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself" pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself" pass out route-to (em1 84.73.24.1) inet from 84.73.24.150 to ! 84.73.24.0/21 flags S/SA keep state allow-opts label "let out anything from firewall host itself" pass in quick on em0 proto tcp from any to (em0) port = https flags S/SA keep state label "anti-lockout rule" pass in quick on em0 proto tcp from any to (em0) port = ssh flags S/SA keep state label "anti-lockout rule" pass in inet all flags S/SA keep state label "NAT REFLECT: Allow traffic to localhost" tagged PFREFLECT anchor "userrules/*" all pass log quick on em1 inet6 proto ipv6-icmp all keep state label "USER_RULE: Any" pass log quick on em0 inet6 proto ipv6-icmp all keep state label "USER_RULE: Any" pass log quick on em2 inet6 proto ipv6-icmp all keep state label "USER_RULE: Any" pass log quick on tun0 inet6 proto ipv6-icmp all keep state label "USER_RULE: Any" pass log quick on openvpn inet6 proto ipv6-icmp all keep state label "USER_RULE: Any" pass in quick on openvpn all flags S/SA keep state label "USER_RULE: OpenVPN TECHFREAK VPN wizard" pass in log quick on em1 reply-to (em1 84.73.24.1) inet proto tcp from any to <FRAGGY> port = 54617 flags S/SA keep state label "USER_RULE: TORRENT" pass in log quick on em1 inet6 proto tcp from any to <FRAGGY> port = 54617 flags S/SA keep state label "USER_RULE: TORRENT" pass in quick on em1 reply-to (em1 84.73.24.1) inet proto tcp from any to <FRAGGY> port = 54617 flags S/SA keep state label "USER_RULE: NAT TORRENT" pass in quick on em1 reply-to (em1 84.73.24.1) inet proto udp from any to 84.73.24.150 port = 1194 keep state label "USER_RULE: OpenVPN TECHFREAK VPN wizard" pass in quick on em1 reply-to (em1 84.73.24.1) inet proto tcp from any to <S16> port = http flags S/SA keep state label "USER_RULE: NAT " pass in quick on em1 reply-to (em1 84.73.24.1) inet proto tcp from any to 192.168.82.18 port = 32400 flags S/SA keep state label "USER_RULE: NAT Plex" pass in quick on em1 reply-to (em1 84.73.24.1) inet proto udp from any to 192.168.82.18 port = 32400 keep state label "USER_RULE: NAT Plex" pass in log quick on em0 inet from <SCTV> to any flags S/SA keep state allow-opts label "USER_RULE: Default allow LAN to any rule SCTV" pass in log quick on em0 inet from 192.168.82.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule" pass in log quick on em0 inet6 from 2001:1620:f0a:1::/64 to any flags S/SA keep state label "USER_RULE: Default allow LAN IPv6 to any rule" pass in quick on em0 inet proto tcp from any to <S16> port = http flags S/SA keep state label "USER_RULE" pass in quick on em0 proto tcp from any to <S16> port = http flags S/SA keep state label "USER_RULE: NAT " pass in log quick on em2 inet proto tcp from <QADROtechLAB> to <S14> port = 8530 flags S/SA keep state label "USER_RULE: WSUS" pass in log quick on em2 inet6 proto tcp from <QADROtechLAB> to <S14> port = 8530 flags S/SA keep state label "USER_RULE: WSUS" pass in log quick on em2 inet proto tcp from <QADROtechLAB> to <S14> port = http flags S/SA keep state label "USER_RULE: WSUS (80)" pass in log quick on em2 inet6 proto tcp from <QADROtechLAB> to <S14> port = http flags S/SA keep state label "USER_RULE: WSUS (80)" pass in log quick on em2 inet proto tcp from <QADROtechLAB> to <FRAGGY> port = 3260 flags S/SA keep state label "USER_RULE: iSCSI" pass in log quick on em2 inet proto tcp from <QADROtechLAB> to <FRAGGY> port = 860 flags S/SA keep state label "USER_RULE: iSCSI" pass in log quick on em2 inet6 proto tcp from <QADROtechLAB> to <FRAGGY> port = 3260 flags S/SA keep state label "USER_RULE: iSCSI" pass in log quick on em2 inet6 proto tcp from <QADROtechLAB> to <FRAGGY> port = 860 flags S/SA keep state label "USER_RULE: iSCSI" pass in log quick on em2 inet proto udp from <QADROtechLAB> to <S11> port = domain keep state label "USER_RULE: DNS" pass in log quick on em2 inet6 proto udp from <QADROtechLAB> to <S11> port = domain keep state label "USER_RULE: DNS" pass in log quick on em2 inet proto udp from 192.168.50.0/24 to <S11> port = domain keep state label "USER_RULE: DNS" pass in quick on em2 inet6 proto ipv6-icmp from <QADROtechLAB> to any keep state label "USER_RULE: ICMPv6" pass in quick on em2 inet proto tcp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port 137:139 flags S/SA keep state label "USER_RULE" pass in quick on em2 inet proto tcp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port = uaac flags S/SA keep state label "USER_RULE" pass in quick on em2 inet proto udp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port 137:139 keep state label "USER_RULE" pass in quick on em2 inet proto udp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port = uaac keep state label "USER_RULE" pass in quick on em2 inet6 proto tcp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port 137:139 flags S/SA keep state label "USER_RULE" pass in quick on em2 inet6 proto tcp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port = uaac flags S/SA keep state label "USER_RULE" pass in quick on em2 inet6 proto udp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port 137:139 keep state label "USER_RULE" pass in quick on em2 inet6 proto udp from <CIFS_ACCESS_FRAGGY> to <FRAGGY> port = uaac keep state label "USER_RULE" pass in quick on em2 inet proto icmp from <QADROtechLAB> to any keep state label "USER_RULE" pass in quick on em2 inet6 proto ipv6-icmp from 2001:1620:f0a:ffff::2 to 2001:1620:f0a:ffff::3 keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View" pass in quick on em2 inet proto icmp from 192.168.50.2 to 192.168.50.3 icmp-type echoreq keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View" pass in log quick on tun0 inet6 proto ipv6-icmp from any to 2001:1620:f00:f::2 keep state label "USER_RULE: sixxs ping FW" pass in log quick on tun0 inet6 proto tcp from any to <FRAGGY> port = 54617 flags S/SA keep state label "USER_RULE: TORRENT" pass in quick on tun0 inet6 proto ipv6-icmp from 2001:1620:f00:f::1 to 2001:1620:f00:f::2 keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View" anchor "tftp-proxy/*" all

Include the list of firewall and anti-virus software you have installed / are running (note that under Windows some 'firewall' tools don't understand IPv6 at all and thus just throw it away, only uninstall helps for those cases)

pfSense is the firewall itself. No other firewall tools / anti-virus tools installed on the box.

Include a IPv4 and IPv6 traceroute to the PoP in question.

traceroute6 to 2001:1620:f00:f::1 (2001:1620:f00:f::1) from 2001:1620:f00:f::2, 64 hops max, 12 byte packets sendto: No buffer space available 1 traceroute6: wrote 2001:1620:f00:f::1 12 chars, ret=-1

(the traceroute does not seem to continue here) Interface Statistics:

AICCU_SIXXS interface (tun0) Status up MAC address 00:00:00:00:00:00 IPv6 Link Local fe80::1420:f00:f:2%tun0 IPv6 address 2001:1620:f00:f::2 Subnet mask IPv6 128 In/out packets 5052/33091 (4.90 MB/6.65 MB) In/out packets (pass) 5052/33091 (4.90 MB/6.65 MB) In/out packets (block) 16/0 (10 KB/0 bytes) In/out errors 0/0 Collisions 0

With heartbeat tunnels, first check clock synchronization or better: use AICCU.

Using AICCU

Check with Wireshark or tcpdumps of the interface over which the tunnel runs. Use -n (numeric) as an option and don't filter returning ICMP which could also come from routers between your endpoint and the PoP and also use -s 1500 so that one gets the full packet.

0x02b0: 5069 6e67 6564 2062 7920 5369 7858 5321 Pinged.by.SixXS! 0x02c0: 0a00 596f 7520 476f 7420 5069 6e67 6564 ..You.Got.Pinged 0x02d0: 2062 7920 5369 7858 5321 0a00 596f 7520 .by.SixXS!..You. 0x02e0: 476f 7420 5069 6e67 6564 2062 7920 5369 Got.Pinged.by.Si 0x02f0: 7858 5321 0a00 596f 7520 476f 7420 5069 xXS!..You.Got.Pi 0x0300: 6e67 6564 2062 7920 5369 7858 5321 0a00 nged.by.SixXS!.. 0x0310: 596f 7520 476f 7420 5069 6e67 6564 2062 You.Got.Pinged.b 0x0320: 7920 5369 7858 5321 0a00 596f 7520 476f y.SixXS!..You.Go 0x0330: 7420 5069 6e67 6564 2062 7920 5369 7858 t.Pinged.by.SixX 0x0340: 5321 0a00 596f 7520 476f 7420 5069 6e67 S!..You.Got.Ping 0x0350: 6564 2062 7920 5369 7858 5321 0a00 596f ed.by.SixXS!..Yo 0x0360: 7520 476f 7420 5069 6e67 6564 2062 7920 u.Got.Pinged.by. 0x0370: 5369 7858 5321 0a00 596f 7520 476f 7420 SixXS!..You.Got. 0x0380: 5069 6e67 6564 2062 7920 5369 7858 5321 Pinged.by.SixXS! 0x0390: 0a00 596f 7520 476f 7420 5069 6e67 6564 ..You.Got.Pinged 0x03a0: 2062 7920 5369 7858 5321 0a00 596f 7520 .by.SixXS!..You. 0x03b0: 476f 7420 5069 6e67 6564 2062 7920 5369 Got.Pinged.by.Si 0x03c0: 7858 5321 0a00 596f 7520 476f 7420 5069 xXS!..You.Got.Pi 0x03d0: 6e67 6564 2062 7920 5369 7858 5321 0a00 nged.by.SixXS!.. 0x03e0: 596f 7520 476f 7420 5069 6e67 6564 2062 You.Got.Pinged.b 0x03f0: 7920 5369 7858 5321 0a00 596f 7520 476f y.SixXS!..You.Go 0x0400: 7420 5069 t.Pi 15:41:59.698689 IP6 2001:1620:f00:f::2 > 2001:1620:f00:f::1: ICMP6, neighbor solicitation, who has 2001:1620:f00:f::1, length 24 0x0000: 6000 0000 0018 3aff 2001 1620 0f00 000f `.....:......... 0x0010: 0000 0000 0000 0002 2001 1620 0f00 000f ................ 0x0020: 0000 0000 0000 0001 8700 a918 0000 0000 ................ 0x0030: 2001 1620 0f00 000f 0000 0000 0000 0001 ................ 15:41:59.708696 IP6 2001:1620:f00:f::2 > 2001:1620:f00:f::1: ICMP6, echo request, seq 7, length 16 0x0000: 6000 0000 0010 3a40 2001 1620 0f00 000f `.....:@........ 0x0010: 0000 0000 0000 0002 2001 1620 0f00 000f ................ 0x0020: 0000 0000 0000 0001 8000 6443 12b4 0007 ..........dC.... 0x0030: 535e 5aa7 000a d043 S^Z....C

I also tried a packet capture using the GUI of pfsense:

19:50:26.185534 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:26.185588 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 128 19:50:27.185210 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:27.185266 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 129 19:50:28.184878 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:28.184941 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 130 19:50:29.184601 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 131 19:50:30.184323 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 132 19:50:31.183934 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 133 19:50:32.183641 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 134 19:50:33.183282 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 135 19:50:34.182910 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:34.182962 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 136 19:50:34.968661 AF IPv6 (28), length 1032: (hlim 64, next-header ICMPv6 (58) payload length: 988) 2001:1620:f00:f::1 > 2001:1620:f00:f::2: [icmp6 sum ok] ICMP6, echo request, length 988, seq 3193 19:50:34.968705 AF IPv6 (28), length 1032: (hlim 64, next-header ICMPv6 (58) payload length: 988) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo reply, length 988, seq 3193 19:50:35.182583 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:35.182627 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 137 19:50:36.182255 AF IPv6 (28), length 68: (hlim 255, next-header ICMPv6 (58) payload length: 24) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2001:1620:f00:f::1 19:50:36.182314 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 138 19:50:37.181972 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 139 19:50:38.181638 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 140 19:50:39.181314 AF IPv6 (28), length 60: (hlim 64, next-header ICMPv6 (58) payload length: 16) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo request, length 16, seq 141

Note this:

19:50:34.968661 AF IPv6 (28), length 1032: (hlim 64, next-header ICMPv6 (58) payload length: 988) 2001:1620:f00:f::1 > 2001:1620:f00:f::2: [icmp6 sum ok] ICMP6, echo request, length 988, seq 3193 19:50:34.968705 AF IPv6 (28), length 1032: (hlim 64, next-header ICMPv6 (58) payload length: 988) 2001:1620:f00:f::2 > 2001:1620:f00:f::1: [icmp6 sum ok] ICMP6, echo reply, length 988, seq 3193

So it seems that a ping from the PoP actually comes through and gets replied to. However, the outgoing packet probably does not really "get out", for a reason that I'm trying to find out here.

The status of the PoP is listed on the PoP Status page, if it is marked down there we are aware of the issue and we will try to resolve it as soon as possible. Additionally other issues are listed in the Ticket Tracker Ticket Tracker.

The status of the PoP is GREEN

We are not your personal helpdesk, thus first read the FAQ and/or use the forum, many cases can already be solved that way, but we do want to have you actually use the tunnel thus do contact us if the problem persists.

I do know that, and I aprreciate the time you take with this. I did google for the problem of course, I have also asked in the pfSense forum for help. However, please also understand that not everyone is an expert in low-level networking, so sometimes we need a little guidance on obtaining the correct information.

Here's one with my ping... dunno if that helps :) Depends. Is there a response? Note that tcpdumping the tunnel interface itself is not very useful; that is, if you want to know the packet possibly made it out of the system. (some systems have pre-firewall tcpdump, others after-firewall tcpdump).

No, I do not get a response. Unfortunately, I don't know what to tcpdump then... If you could give me a hint?

You are right, I think the default behavior really is dropping the packets. I have provided logs / rules with evidence that there are rules in place to properly pass ICMPv6 traffic. If not, please give me a hint on what I should provide. I don't know about the connection tracking though.

2a02:418:3002::/48 2001:1620:f0a:ffff::2 UGS em2 2a02:2528:ff48::/48 00:50:56:a1:71:57 UGS em2 Likely unrelated, but unless you are null-routing these, where do these really go? Are they there on purpose?

They go to my other (secondary) pfsense box, which is used to terminate traffice related to my work (vpn with the datacenter, business-lab) Also note that I am able to reach machines from those other subnets routed through the non-working box:

Tracing route to l1-mizeas-11.domain1.forest1.lab.quadrotech-it.com [2a02:418:30 02:6:3f5:76f5:c426:f68f] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 2001:1620:f0a:1::ffff <--- non-working box 2 <1 ms <1 ms <1 ms 2001:1620:f0a:ffff::2 <--- other pfsense box 3 * * * Request timed out. <--- pf sense box @ work (VPN) 4 42 ms 41 ms 48 ms 2a02:418:3002:6:3f5:76f5:c426:f68f <--- other network @ work (through VPN) Trace complete.

I would like to provide all information you need, however I don't know what you need :) There are these big yellow/orange bars when posting. They contain (repeated) a very big hint.

I apologize again for not include all information in the first place. I did read the (repeated) yellow bars, and I think I did include some reasonable information in my first post.

ping6: No buffer space available when pinging PoP endpoint

Jeroen Massar SixXS Staff

on Monday, 28 April 2014 18:49:53

Last Heartbeat 2014-04-28 17:13:34 (1398705214; 0 days 00:00:01 ago)

At least your host is sending valid heartbeats. Good thing.

Packet In 2014-04-25 07:29:35 (1398410975; 3 days 09:44:00 ago)

It is not sending any (valid) packets.

Encap.Pkt Too Big 226, last: 2400:cb00:2048:1::be5d:f251 2014-04-25 07:29:35 (1398410975; 3 days 09:44:00 ago)

Similar stamp as above, thus that correlates that.

When using AICCU, check that you are using the latest version, and of course please specify where the binary comes from, also run it with the 'verbose true' flag and provide the output of that.

As you quote yourself, version? where does it come from? Though as heartbeats work, likely nothing much wrong with AICCU. Although there is one important detail. When AICCU starts up (thus check the logs) it will report HAS_IFHEAD and NEED_IFHEAD, what is the output of those settings?

I use a virtual pfSense Box on VMware ESXi 5.5. The cable-connection is attached to VMware using VLAN through 2 switches.

While VMWare used to have issues with multicast and IPv6, those problems should be fixed and you are tunneling thus that should not affect that.

There is no nat, the pfSense Box has an external IPv4.

So you are saying the box itself also does not perform NAT? As that is an important thing to note, as that means there is connection tracking and other muckery going on. As a side-note: cablecom.ch allows you to request three (3) IPv4 addresses using DHCP, abuse it while it lasts...

inet 84.73.24.150 netmask 0xfffff800 broadcast 255.255.255.255

That broadcast address is odd. Especially considering the netmask.

62.2.17.60 00:50:56:a1:71:56 UHS 0 13 em1

62.2.17.61 00:50:56:a1:71:56 UHS 0 743 em1

62.2.24.158 00:50:56:a1:71:56 UHS 0 21 em1

62.2.24.162 00:50:56:a1:71:56 UHS 0 0 em1

Strange that you have static routes for those. Likely unrelated but you might want to check where they come from.

2a02:418:3002::/48 2001:1620:f0a:ffff::2 UGS em2

2a02:2528:ff48::/48 00:50:56:a1:71:57 UGS em2

Where do these come from? Is that from OpenVPN (assuming the ovpns is that) ?

Apr 28 19:36:35 AICCU_SIXXS PASS [2001:1620:f00:f::1] [2001:1620:f00:f::2] ICMPv6

That is inbound. That seems to work (no errors in the live log for instance). Your problem is sending packets outbound, at least some times.

Also, please remember that this used to work, and I did not change rules.

You upgraded software, that might break expectations. Also, IPv6 is heavily dependent on multicast, you might just want to do away with the firewall during your tests.

scrub on tun0 all fragment reassemble

Are you sure that fragment reassembly is needed? Let alone have a negative effect.

anchor "relayd/*" all

relayd, one of those tools that can mess with things

anchor "openvpn/*" all

openvpn, another nice one. [.. lots of rules snipped ..] There can be lots of problems with those rules.

pfSense is the firewall itself. No other firewall tools / anti-virus tools installed on the box.

Actually your firewall is either 'pf' or 'ipfw' ;)

1 * * *

Silly cablecom, for one reason or another they don't let the first hop reply properly to ICMP.

In/out packets 5052/33091 (4.90 MB/6.65 MB)

In/out packets (pass) 5052/33091 (4.90 MB/6.65 MB)

In/out packets (block) 16/0 (10 KB/0 bytes)

16 packets blocked, but the counts for in/out + pass are the same. I wonder which 16 packets got blocked but where also passed... Interestingly it claims that you are sending and receiving packets. This while the PoP tells that that is not the case. Hence, your packets must get lost somewhere along the way.

0x02b0: 5069 6e67 6564 2062 7920 5369 7858 5321 Pinged.by.SixXS!

Where is the actual header of the packet, thus source + destination? 15:41:59.698689 IP6 2001:1620:f00:f::2 > 2001:1620:f00:f::1: ICMP6, neighbor solicitation, who has 2001:1620:f00:f::1, length 24 You are looking at the tunnel interface, you have to look at the IPv4 interface where the tunnel is being sent over.

So it seems that a ping from the PoP actually comes through and gets replied to.

However, the outgoing packet probably does not really "get out", for a reason that I'm trying to find out here.

Correct. And this kind-of explains why you have sent/received traffic, at least on the tunneled interface. You'll have to

Unfortunately, I don't know what to tcpdump then... If you could give me a hint?

em1 is the interface that is handling your IPv4 traffic.

You are right, I think the default behavior really is dropping the packets.

I have provided logs / rules with evidence that there are rules in place to properly pass ICMPv6 traffic.

No, it is passing _one kind_ of ICMPv6 traffic. You also need to allow the tunneled traffic (that goes over IPv4).

I don't know about the connection tracking though.

There is an article in the FAQ about this part.

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Monday, 28 April 2014 22:31:02

I think this issue has nothing to do with IPv6 or SixXS at all. I had this issue with IPv4 too. It happened to me when I tried to ping Google Public DNS (8.8.8.8) from the pfSense LiveCD. After I finished installing to writable media, the issue was gone. Maybe it tries to write some swap on the read-only media when pinging.

ping6: No buffer space available when pinging PoP endpoint

Shadow Hawkins on Tuesday, 29 April 2014 05:03:54

I have pfSense installed on a (writeable) harddisk. However, I wanted to collect some info and stuff this morning, and you know what? It suddently works again... I can ping the PoP tunnel endpoint as well as google. Weird! Sorry to waste your time, but thank you very much for the help! I think at least I have learned some things new :)

Please note Posting is only allowed when you are logged in.