Sending low voltage warning email causes WS-12-250-DC reboot

[JayDee]

We use WS-12-250-DCs at a lot of our sites. Our standard configuration is one or more WS-12-250-DCs connected to a series of 4 Deka 8A24 or 8A27 SLA AGM batteries, with an Iota DLS-5413 battery charger also connected to the batteries.

We use the Warning Voltage feature with SMTP email alerts to get notified when the input voltage drops below 50V. When we receive an email alert, we can infer that the grid power is off at the site, and that it is operating on battery power.

I have seen a lot of instances now where the switch and all the attached equipment reboots shorty after we receive the email alert.

After considering electrical causes and coming up empty, I am now convinced that a firmware bug is behind this behavior.

I have been doing tests on the bench, and I find that I can cause the switch to reboot by varying the input voltage, moving it back and forth across the Warning Voltage threshold.

So far I have reproduced this in 1.4.9, 1.5.0, 1.5.1, and 1.5.2. I have not tried other versions.

This only happens when SMTP email alerts are enabled. If email alerts are not enabled, I can vary the input voltage all day long and the switch stays up. I can see the voltage warning events shown in the switch system log, but so long as no emails are sent, everything is fine.

I can also ask the switch to send test emails repeatedly by clicking on the Test button in the configuration page, and everything is fine. The reboot is not triggered.

Sometimes a reboot is triggered on the first crossing. Other times it can take 4 or 5 downward crossings of the threshold before it happens.

Upon rebooting, if the input voltage is still below the threshold the switch immediately tries to send another email alert. Sometimes this will also trigger a reboot. I once saw 3 reboots in a row happen this way.

Another behavior that I sometimes see happen is where it will send a low voltage warning email, and then something hangs up and it will not send another such email until it is rebooted by other means. This seems to happen only in 1.5.1 and 1.5.2. While this behavior does work around the issue somewhat, it is not consistent. There are times where 1.5.1 and 1.5.2 will reboot instead of hanging like this when trying to send email.

One other thing I notice is that after successfully sending a low voltage email (and not rebooting), the input voltage is briefly shown as 399-400V before returning to its correct level.

https://i.imgur.com/WDWKob8.png

I found this thread, where I think the user encountered the same issue: viewtopic.php?f=17&t=3468

Let me know if I can provide any additional data to help pin down the issue.

Thanks.

[Stephen]

Thank you for investigating. Will begin taking a look at this.

[cbl]

We have also seen this issue and reported it here:

viewtopic.php?f=17&t=3914&p=26628#p26628

[bpnz]

We have also seen this issue and reported it here:

viewtopic.php?f=17&t=3914&p=26628#p26628

I can confirm I'm having exactly the same issue with a WS-12-250-DC running 1.4.9 firmware... I have the warning voltage set fairly high (ovely cautious), but lowering it so the warning wasn't triggered fixed it immediately. The sites a 24v site that draws a constant load of up around 7amps.

[Stephen]

Is there any thing else you guys can tell me about the setup on your switches (Specifically the WS-12-250-DC)

I am testing this on WS-12-250-DC with firmware 1.5.2, I have done it dozens of time's and it hasn't rebooted on me yet.

Could anyone with this issue post their configuration and power tab? I'll try to emulate how you have your's setup so I can find this issue.

[JayDee]

Hmm. Here are a couple of screenshots from one of ours that shows the issue: https://imgur.com/a/ZOYEZJM

Note that we are using the "secure" email feature.

[Stephen]

I'm using secure as well, I'm also using the smtp service by gmail.

One difference I see is that your warning voltage is set to 50V mine was at 30 just for testing. I'll try changing the value and see if that make's any difference.

Before it reboots, are there any other log messages that occur that might stand out?

[JayDee]

Before it reboots, are there any other log messages that occur that might stand out?

I sure didn't see any. I tried to connect to the serial console port and look for any interesting output there, but that didn't really work for some reason. Some of the lines came across fine, while others were garbled somehow, and looked like they were missing characters.

Unfortunately the 'dmesg' command in the firmware does not support the --follow option, so I wasn't able to see anything interesting there either.

I'll try and mess with it some more. I'll try connecting to gmail instead of to our mail server and see if that changes anything.

[Stephen]

Sounds good, let me know if that make's any difference.

[JayDee]

Hmm. Well, indeed that does make the difference. I am unable to reproduce the crash when emails are sent via smtp.gmail.com.

I don't administer our mail server so I don't know a lot about it, but it does indicate to me that it runs postfix when I look at the results of the test email.

I'll try to play with it some more. I imagine that capturing packets and looking for a difference there might get tricky what with TLS being involved...