I've got 2 units that are doing this now. I get the following error and can no longer read voltages or temperature via snmp or the main web page, but all other functions seem to be working.
Nov 2 13:07:47 switch[1358]: i2c error: excessive arbitration lost start errors (0x51)
Nov 2 13:07:58 switch[1358]: retrying i2c
Nov 2 13:07:58 switch[1358]: i2c error: excessive arbitration lost start errors (0x51)
Nov 2 13:08:09 switch[1358]: retrying i2c
They repeat about every second. Currently running 1.3.7rc2. Also noticed the CPU's on these 2 units is nearly 100%.
i2c error:
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: i2c error:
Release notes wrote: NOTE:
Our switches are designed to continue to run even when they are damaged. Sometimes the switch can take damage to sensors from a surge from the tower or ground potential shift between the tower and the cabinet damaging the current sensor on a port. Since all the current, voltage, and temp sensors share the same I2C bus this can cause other sensors to report erroneous values. Prior to v1.3.5 we did not do as good of job at alerting the user to out of range readings, therefore after upgrading to v1.3.5 or newer if you are now seeing voltages, watts, or temperatures out of range this may be an indication that your unit took damage in the past and you were unaware of it. This is why I say to always make sure your Tower Ground is Bonded to your Electrical Service ground and to run a ground wire up the tower and to each radios on the tower. I have a good post HERE about proper grounding methods.Another possible cause of "false" out of range readings can be from an incompatible SFP module. Since the SFP cages also share this same I2C bus to relay information about the SFP module to the system some incompatible SFP modules "can" disrupt normal I2C communication. Try removing the SFP module and see if the readings return to acceptable ranges. If you find an SFP module that is causing issue try another brand of SFP module. We are working on a way to eliminate this interference from some SFP module manufacturers and may offer a firmware in the future to fix this issue with these modules.
With that being said we are working on v1.3.7 firmware which will hopefully deal better with possibly damaged boards?
Not saying these are damaged just possible?
I would try power cycling the units to see if the issue goes away first, let us know. Note, when I say power cycle I mean power cycle not a warm software reboot, although you could try a reboot first but a power cycle is needed.
I have seen this happen on boards that are not damaged and a power cycle resolved the issue. This is one thing we are working on for v1.3.7
We will be releasing v1.3.7 soon, until then you may wish to downgrade to v1.3.3 until then?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: i2c error:
You can try v1.3.7rc4
Here is an RC version that fixes this issue, click HERE to download v1.3.7rc4 - RELEASE CANDIDATE
But you may need to power cycle the units.
Here is an RC version that fixes this issue, click HERE to download v1.3.7rc4 - RELEASE CANDIDATE
But you may need to power cycle the units.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
sww - Member
- Posts: 43
- Joined: Tue Aug 25, 2015 12:41 pm
- Location: Roswell, NM
- Has thanked: 0 time
- Been thanked: 10 times
Re: i2c error:
New firmware didn't help. Both of these units are atop hard to reach mountain sites so a hard reboot won't be for a while. I'll admit we probably damaged both of these now that you mention it. One of my idiot techs tried to power a 24v reversed polarity canopy module off a 48 volt powered port and the unit shut down until he power cycled it. The other unit we swapped an 8 port for a 12 porter and another idiot tech couldn't figure out how to match port 1 with port 1, 2 to 2, 3 to 3..etc.
Lesson learned! Let me go yell at somebody. :)
Do you guys offer a repair service?
Lesson learned! Let me go yell at somebody. :)
Do you guys offer a repair service?
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: i2c error:
sww wrote:New firmware didn't help. Both of these units are atop hard to reach mountain sites so a hard reboot won't be for a while. I'll admit we probably damaged both of these now that you mention it. One of my idiot techs tried to power a 24v reversed polarity canopy module off a 48 volt powered port and the unit shut down until he power cycled it. The other unit we swapped an 8 port for a 12 porter and another idiot tech couldn't figure out how to match port 1 with port 1, 2 to 2, 3 to 3..etc.
Lesson learned! Let me go yell at somebody. :)
Do you guys offer a repair service?
LOL - It happens!
Yes we offer repair services.
You should replace the units then request an RMA.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
sww - Member
- Posts: 43
- Joined: Tue Aug 25, 2015 12:41 pm
- Location: Roswell, NM
- Has thanked: 0 time
- Been thanked: 10 times
Re: i2c error:
Both towers need to be visited in the next couple weeks. I'll swap them out and get with you later and looking at the affected ones. Thanks again.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: i2c error:
PS.
Thanks for being honest up front!
Yes we offer a reasonably priced repair service.
I hate it when people send me a switch which is very obviously lightening or user damage such as a blown port from either a dead short cable or from trying to power 24V device with 24VH.
I even had a guy send me a switch that was several months old which he claimed came DOA. When I took it apart to look at it several parts were blown off the board and in their place was large black charred areas so I called him asking if this was possibly a mistake. He was then quite rude saying it must have came that way from the factory.
After explaining I am the factory he made a big fuss and said it came that way, explained how it sat on his shelf for several months then he pulled it to provision it and it would never power up. So I removed the system flash and pulled his config from the switch that came DOA and "never powered up" and emailed him his config which ironically included the coordinates of his tower so I also sent him a Google Earth picture of his tower with his tower config.
Average turn around on RMAs is about 3 weeks. We do not get a lot of RMAs so I let at least 5 or more stack up then take a day and do repairs.
We fully test repaired boards to work 100% and they are then covered under the original warranty period.
Thanks for being honest up front!
Yes we offer a reasonably priced repair service.
I hate it when people send me a switch which is very obviously lightening or user damage such as a blown port from either a dead short cable or from trying to power 24V device with 24VH.
I even had a guy send me a switch that was several months old which he claimed came DOA. When I took it apart to look at it several parts were blown off the board and in their place was large black charred areas so I called him asking if this was possibly a mistake. He was then quite rude saying it must have came that way from the factory.
After explaining I am the factory he made a big fuss and said it came that way, explained how it sat on his shelf for several months then he pulled it to provision it and it would never power up. So I removed the system flash and pulled his config from the switch that came DOA and "never powered up" and emailed him his config which ironically included the coordinates of his tower so I also sent him a Google Earth picture of his tower with his tower config.
Average turn around on RMAs is about 3 weeks. We do not get a lot of RMAs so I let at least 5 or more stack up then take a day and do repairs.
We fully test repaired boards to work 100% and they are then covered under the original warranty period.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
sww - Member
- Posts: 43
- Joined: Tue Aug 25, 2015 12:41 pm
- Location: Roswell, NM
- Has thanked: 0 time
- Been thanked: 10 times
Re: i2c error:
That's a funny story! After 9 years of doing this I could go on and on and on, but you've probably heard them all before.
-
sww - Member
- Posts: 43
- Joined: Tue Aug 25, 2015 12:41 pm
- Location: Roswell, NM
- Has thanked: 0 time
- Been thanked: 10 times
Re: i2c error:
Hey guys, are we sure there's not a chance that a software bug could be causing this? I started to program up a new one out of the box and I'm having the same issue with this one as well. When I first logged into it the first thing I did was update it to 1.3.7rc4. I didn't check the voltages or even what software was on it, but after the update I logged in to program it and it's having the same errors. I promise you on this one we did nothing to it. I hadn't even powered a port on yet and the only thing plugged into it was another switch which my computer is plugged into. I then tried downgrading the software to 1.3.6 just for something to try and no help. Hard reboots don't help either. I did this one myself and feel confident I took every caution I could. Any ideas? I'm using a 24v power supply that came from a Canopy Cluster Management Module for programming and testing.
-
sww - Member
- Posts: 43
- Joined: Tue Aug 25, 2015 12:41 pm
- Location: Roswell, NM
- Has thanked: 0 time
- Been thanked: 10 times
Re: i2c error:
I just fired up another one out of the batch and it's working normally. It's got 1.3.4 on it. Should I try to update it and see what happens? Should I be able to go from 1.3.4 straight to 1.3.7rc4? Or should I be upgrade in steps?
Who is online
Users browsing this forum: No registered users and 44 guests