Ticket ID: SIXXS #454811
Ticket Status: Resolved
PoP: nlams05 - SURFnet (Amsterdam)
Shadow Hawkins on Tuesday, 05 December 2006 12:38:45
POP seems to be down for more than 4 hours. No one seems to have reported it.
Shadow Hawkins on Tuesday, 05 December 2006 14:46:12
SARA reported a power failure between 0:30 and 0:45 and this system may not yet have been fixed. I got the following Dutch warning from my hosting partner.
Op Dinsdag 05 December 2006, omstreeks 00:30, was er bij Rekencentrum SARA (Een belangrijk knooppunt van internetverbindingen in Amsterdam) een korte stroomstoring. Hierdoor waren diverse systemen bij SARA ontregelt geraakt. Aangezien veel verkeer binnen Nederland over deze lokatie gaat, kan het zijn dat u hier voor een zeer korte tijd last van heeft gehad. Onze verbindingen met SARA zijn reeds automatisch hersteld. Het kan echter gebeuren dat uw provider nog storingen ondervindt.
Jeroen Massar on Tuesday, 05 December 2006 15:05:18
This is indeed what caused the issue, SARA NOC (who manage the Surfnet machines) are looking into it.
State change: confirmed
Jeroen Massar on Tuesday, 05 December 2006 15:04:41
The state of this ticket has been changed to confirmed
Jeroen Massar on Tuesday, 05 December 2006 19:07:22
Due to the abrupt power outage the hardware became affected. Repairs under underway which require a new harddisk (it had been spinning for quite some time and the power outage apparently was fatal). As there is quite some work at SARA at the moment due to other machines also failing hardware is a bit on short supply and the machine will also require a re-install. This has one advantage: it will be upgraded so that we can run the v3a code on it which will finally supply AYIYA connectivity. You might notice that the machine is reachable over both IPv4 and IPv6 at the moment, but it is not routing nor tunneling as the hardware is unstable. Please have patience, the PoP will be restored asap, but do expect a couple of days outage unfortunately.
Jeroen Massar on Wednesday, 06 December 2006 18:32:34
New disks have arrived, new OS is installed, we are proceeding now with installation and further configuration of the PoP and hope to have it online soon and with fully upgraded code thus providing AYIYA functionality.
State change: resolved
Jeroen Massar on Wednesday, 06 December 2006 20:17:00
The state of this ticket has been changed to resolved
Jeroen Massar on Wednesday, 06 December 2006 20:18:14
The nlams05 PoP is operational again, it is correctly routing and handling tunnels again. In case there are still problems, don't hesitate to reply in THIS ticket. Statistics will return also in a few.
Shadow Hawkins on Wednesday, 06 December 2006 22:30:10
Eeeeh. not 100% sure that it's not me, but after re-'connecting' the tunnel a couple of times I still get no connectivity. I noticed that I could ping6 my PoP v6 endpoint, I could traceroute6 there, but I could not traceroute6 anything else, say... noc.sixxs.net. After disabling the IP I normally use for outgoing connections (the one in my subnet, with reverse DNS etc) traceroute6 and ping6 anywhere worked fine. IPv6 connectivity doesn't work on my desktop either, which has an IP from my subnet range. I can ping6 my server from there, though. My routes look fine, after adding the 'external' IP, it shows the route will go over dev sixxs, the outgoing packets stat on ifconfig sixxs increases too, no incomings, no errors. So it looks like routing is working fine again, but subnets aren't. Could this be right or is it really me?
Jeroen Massar on Wednesday, 06 December 2006 22:39:55
Please *SHOW* what exactly is broken, thus show traceroutes etc. See "Reporting Problems" on the contact page for more details.
Shadow Hawkins on Wednesday, 06 December 2006 22:51:05
Well, there is nothing to show really... somewhat shortened info: ip -6 ro 2001:610:600:28d::/64 via :: dev sixxs default via 2001:610:600:28d::1 dev sixxs static aiccu tunnel - 2001:610:600:28d::2/64 ping6 2001:610:600:28d::1 works, +in +out on ifconfig traceroute6 2001:610:600:28d::1 works, +in +out on ifconfig ping6 noc.sixxs.net works, +in +out on ifconfig traceroute6 noc.sixxs.net works, +in +out on ifconfig static aiccu tunnel - 2001:610:600:28d::2/64 and 2001:610:683:1337::1/64 ('primary' IP) ip -6 ro unchanged, except for the extra line: 2001:610:683:1337::/64 via :: dev sixxs ping6 2001:610:600:28d::1 works, +in +out on ifconfig traceroute6 2001:610:600:28d::1 works, +in +out on ifconfig ping6 noc.sixxs.net does not work, 0in +out on ifconfig traceroute6 noc.sixxs.net does not work, 0in +out on ifconfig And, ofcourse... As always with this kind of problems, only after I spend a few minutes typing everything from my server monitor into a textbox, everything magically works again. I guess they brought up the subnet things now, everything is working fine on my server AND on my desktop :)
Jeroen Massar on Wednesday, 06 December 2006 23:02:57
Nothing changed on the PoP since I posted the message that it was working again and that it was marked as being 'up' by the monitoring tools, thus it might have been something different. Please, next time report the information as that helps in these cases to see if there is anything wrong and if there is something wrong where it might be.
Posting is only allowed when you are logged in.