Slightly Different Subject.. Upgrading the core to 10G!

[slarner]

Slightly off topic but thought you guys might be interested.

In the last 2 years built our own core network which is expanding very fast. Started of with a RB1100. This couldn't hack it so very quickly swapped it out for a Mikrotik 36 Core Cloud core

This has been holding up amazing. With multiple full BGP feeds and IXP connections too. (140 Active BGP sessions) Averaging 20% CPU load with near on 2Gb/s passing through the device.

But the time has come to upgrade to 10G. We are about to light a 160KM of dark fiber. Which means we can deliver fiber to the tower.

As you guys know replacing running equipment is a complete nightmare. We also have a couple of Extreme Switches. I wanted to get these configured up in stacking mode. They now have a 20G back plane which is then connected via 2x SFP+ 10G ports to a new 72 Core Cloud core router.

Here is a couple of pics!



And one with the Extreme Switch. (sorry about the mess of cables!)



Stewart

[sirhc]

WOW - Thats a lot (shit ton) of bandwidth!

[slarner]

WOW - Thats a lot (shit ton) of bandwidth!

Everyone wants more and more bandwidth... We are already pushing all of that 2Gb/s over wireless already!

[lligetfa]

Oh, what a tangled web we weave, when first we start to rack shit up or through moves/changes afterwards. LOL

Spoken in true "do as I say, not as I do" fashion. Lots of my wiring closets end up looking like that. The more cooks there are in the kitchen, the worse it gets.

I try now to use long enough jumpers and wire management guides on the sides of the racks. I also like to leave spaces in the rack so I can slip in a replacement under a unit and then move jumpers with the least disruption.

Years ago I had upgraded to all new HP ProCurve kit and ran into some issues after it all went live. A field tech from HP and I hot-swapped 24 modules on a live network with very minimum disruption. Years later the tangled web grew to where it would not be possible to do that again.

[rebelwireless]

slarner, how do you handle shaping?

also, how do those CCRs run? stable? I've been extremely disappointed in mikrotik in the last year. Their MIPS boards haven't been very stable and new versions of routeros have been flaky.

[Mikea]

@rebelwireless i dont want to hijack this thread but we are using an CCR1009 for PPPoE shapping at headend and it is working without any issue.
0 secs downtime with plenty of CPU for a small wisp with under 1000 clients.
we put in on service replacing an RB1100.

[slarner]

Unfortunately i couldn't tidy it up. That meant more downtime.

The CCR's have been rock solid for us the CCR1036 had 200 days up time with 21% CPU moving around 2Gb/s. This was also doing BGP straight out the the world and internet exchanges.

We are using queue trees with connection and packet marking. We have a plan to replace this with some kind of separate shaping tool maybe bluecoat.

We are a huge mikrotik user for routers. We have no mikrotik wireless kit. I think they have lost the game.

Stewart