Page 1 of 1
8port DC issue
Posted: Mon Sep 07, 2015 8:47 pm
by LRL
I've got two 8 port DC switches deployed and two 12 port DC switches deployed and until now I haven't had any issues.
Over the weekend I received three e-mails from one of our 8 port switches saying "Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1610612736 C) - from JT Switch (10.247.28.5)". All three e-mail were received within 2 minutes.
right after I received this e-mail I logged into the switch and here are the temp readings from the status page:
Board Temp 41C /106F
CPU Temp55C /131F
PHY Temp57C /135F
Power Supply Board Temp35C /95F
Power Supply Heatsink #1 Temp28C /82F
Power Supply Heatsink #2 Temp28C /82F
Power Supply Controller Temp50C /122F
Our environmental monitor showed the box to be at 79f and our average power usage is 21w. Thoughts on this?
Re: 8port DC issue
Posted: Mon Sep 07, 2015 8:59 pm
by LRL
I forgot, port 5 is a camera and has PoE priority set to 1 so it would naturally be the first port to shutdown. The device was indeed powered off and recovered pretty quick. Below is the log.
- Code: Select all
Sep 5 16:07:37 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1610612736 C)
Sep 5 16:07:38 sSMTP[1563]: Sent mail for *** outbytes=590
Sep 5 16:09:16 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1073741824 C)
Sep 5 16:09:18 sSMTP[1749]: Sent mail for *** outbytes=590
Sep 5 16:09:54 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is -1073741824 C)
Sep 5 16:09:55 sSMTP[1831]: Sent mail for*** outbytes=592
Re: 8port DC issue
Posted: Mon Sep 07, 2015 9:47 pm
by sirhc
lrl wrote: The device was indeed powered off and recovered pretty quick.
Does this mean it recovered on it's own?
Sep 5 16:07:37 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1610612736 C)
Sep 5 16:07:38 sSMTP[1563]: Sent mail for *** outbytes=590
Sep 5 16:09:16 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1073741824 C)
Sep 5 16:09:18 sSMTP[1749]: Sent mail for *** outbytes=590
Sep 5 16:09:54 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is -1073741824 C)
Sep 5 16:09:55 sSMTP[1831]: Sent mail for*** outbytes=592
Obviously those are bogus readings in the log because at the 2 first temperatures nuclear fusion would occur at extremely low gravimetric pressure, and at the 3rd temperature all molecular motion would stop, the question is why?
What is the DC input voltage from the batteries?
Re: 8port DC issue
Posted: Mon Sep 07, 2015 9:55 pm
by LRL
It did indeed recover on it's own. Shortly after these I had a rocket go down and ping watchdog reset it.
Operating temps are up to 55c or 131f, correct? I know I seen a post where you outlined what happen at what temps but I couldn't find it again.
Re: 8port DC issue
Posted: Mon Sep 07, 2015 10:02 pm
by sirhc
LRL wrote:It did indeed recover on it's own. Shortly after these I had a rocket go down and ping watchdog reset it.
Operating temps are up to 55c or 131f, correct? I know I seen a post where you outlined what happen at what temps but I couldn't find it again.
The Rocket going down and recovered by ping watch dog probably is not related to this at all unless you see something in the log that ties them together?
The temperatures are all fine except those
bogus values reported in the log which range from negative BILLION to positive BILLION.
Re: 8port DC issue
Posted: Mon Sep 07, 2015 10:13 pm
by LRL
Nothing really, just shortly after those.
- Code: Select all
Sep 5 16:13:36 pinger: Failure pinging 10.246.204.226, powering off port 6 for 5 seconds
Sep 5 16:13:37 sSMTP[2397]: Sent mail for *** outbytes=515
The switches uptime wasn't affected nor was the backhaul.
We had 0 errors on all the interfaces the day prior and after the even we errors on all interfaces.
- JT status.jpg (174.99 KiB) Viewed 6138 times
- JT Power.jpg (144.29 KiB) Viewed 6138 times
Re: 8port DC issue
Posted: Tue Sep 08, 2015 8:03 am
by sirhc
Hi LRL,
We are looking into this.
We believe those bogus temp readings are a product of an I2C bus error.
We have Eric working on code to compensate for erroneous readings like this and will release a firmware patch soon.
Re: 8port DC issue
Posted: Tue Sep 08, 2015 4:27 pm
by sirhc
Sep 5 16:07:37 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1610612736 C)
Sep 5 16:07:38 sSMTP[1563]: Sent mail for *** outbytes=590
Sep 5 16:09:16 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1073741824 C)
Sep 5 16:09:18 sSMTP[1749]: Sent mail for *** outbytes=590
Sep 5 16:09:54 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is -1073741824 C)
Sep 5 16:09:55 sSMTP[1831]: Sent mail for*** outbytes=592
HEY - LRLWas this the complete log or was this edited?
What I mean is there is 5 entries showing that the same port was disabled 3 times which there should have been entries in between showing the POE was restored before it could disable it again or it should have moved on to the next port.
Eric said he found where he was formating the the number wrong in the log but as I said something seems wrong unless this was an edited log just showing the POE disables?
Also your other readings which were taken right after the event show the Controller temp at a more reasonable range of 50C, so the reading must have been bogus as there is no way the Controller would heat up above 110C to trigger this event then cool back down to 50C so quickly? I am assuming by what you said these events all occurred within a 10 minute span to where it corrected itself and restored POE power?
Re: 8port DC issue
Posted: Thu Sep 10, 2015 5:24 pm
by LRL
The log was for "switch" only and was unedited aside from the e-mail address. No clears or anything left out.
Here is the log "all" only e-mail editied
- Code: Select all
Jan 3 06:42:57 system: starting ntpclient
Sep 5 02:06:49 pinger: starting up with 2 ping watchdogs and 0 scheduled port bounces
Sep 5 16:06:48 kernel: link state changed to 'down' on port 6
Sep 5 16:06:50 kernel: link state changed to 'up' on port 6
Sep 5 16:07:37 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1610612736 C)
Sep 5 16:07:38 sSMTP[1563]: Sent mail for *** (221 2.0.0 **** closing connection) uid=0 username=* outbytes=590
Sep 5 16:07:38 kernel: link state changed to 'down' on port 5
Sep 5 16:07:42 kernel: link state changed to 'up' on port 5
Sep 5 16:07:45 kernel: link state changed to 'down' on port 5
Sep 5 16:07:48 kernel: link state changed to 'up' on port 5
Sep 5 16:07:58 kernel: link state changed to 'down' on port 5
Sep 5 16:08:00 kernel: link state changed to 'up' on port 5
Sep 5 16:09:16 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is 1073741824 C)
Sep 5 16:09:18 sSMTP[1749]: Sent mail for *** (221 2.0.0 **** closing connection) uid=0 username=* outbytes=590
Sep 5 16:09:18 kernel: link state changed to 'down' on port 5
Sep 5 16:09:21 kernel: link state changed to 'up' on port 5
Sep 5 16:09:24 kernel: link state changed to 'down' on port 5
Sep 5 16:09:27 kernel: link state changed to 'up' on port 5
Sep 5 16:09:37 kernel: link state changed to 'down' on port 5
Sep 5 16:09:39 kernel: link state changed to 'up' on port 5
Sep 5 16:09:54 switch[782]: Disabling PoE power on port 5 due to Power Supply Thermal out of specification (controller is -1073741824 C)
Sep 5 16:09:55 sSMTP[1831]: Sent mail for *** (221 2.0.0 **** closing connection) uid=0 username=* outbytes=592
Sep 5 16:09:55 kernel: link state changed to 'down' on port 5
Sep 5 16:09:59 kernel: link state changed to 'up' on port 5
Sep 5 16:10:01 kernel: link state changed to 'down' on port 5
Sep 5 16:10:04 kernel: link state changed to 'up' on port 5
Sep 5 16:10:14 kernel: link state changed to 'down' on port 5
Sep 5 16:10:17 kernel: link state changed to 'up' on port 5
Sep 5 16:13:36 pinger: Failure pinging 10.246.204.226, powering off port 6 for 5 seconds
Sep 5 16:13:37 sSMTP[2397]: Sent mail for *** (221 2.0.0 **** closing connection) uid=0 username=* outbytes=515
Sep 5 16:13:38 kernel: link state changed to 'down' on port 6
Sep 5 16:13:44 kernel: link state changed to 'up' on port 6
Sep 5 16:14:07 kernel: link state changed to 'down' on port 6
Sep 5 16:14:09 kernel: link state changed to 'up' on port 6
Sep 5 16:14:11 kernel: link state changed to 'down' on port 6
Sep 5 16:14:13 kernel: link state changed to 'up' on port 6