10G/40G QSFP switch not working

xfataa

n00b
Joined
Sep 21, 2011
Messages
17
Hello!

I recently scored a used 10G/40G switch on eBay that I want to use for a 10GbE home network setup.

It's a Broadcom BCM956845KQ-02 with 18 QSFP ports on the front and 2 RJ45 ports (labeled RS232 and Ethernet) on the back.
It didn't come with any form of documentation and it seems impossible to find anything online either. I think this might be a prototype or something like that.

Some pictures...

Front:
fsxsw.jpg


Back:
9j2c0w.jpg


My other 10GbE hardware:
  • Brocade 1020 CNA (dual port 10G SFP+ card)
  • Brocade SFP+ transceivers (compatible with the 1020 cards)
  • Brocade active direct attach twinax copper cables
  • Fiber optic patch cables (multi-mode 850nm, OM3)

I can confirm that the combination of my Brocade cards, Brocade transceivers and cables works - both in NIC-to-NIC and NIC-to-switch configurations (I also have a Gigabit switch with 2 SFP+ uplinks that I've tested this with).

However, since the NICs are SFP+ and the Broadcom switch comes with QSFP ports, I need some kind of conversion from the QSFP to the SFP+ form factor.
There are various options: QSFP to SFP+ hybrid cables (direct attach copper), QSFP to 4xSFP+ breakout cables (direct attach copper), QSFP transceivers (very expensive!) with MPO to 8xLC breakout cables (fiber) and finally there's a QSFP to SFP+ adapter that enables you to use SFP+ modules/cables with QSFP gear.

The hybrid DACs I linked aren't an option because they're passive and the Brocade cards only work with active ones.
Since I already had the SFP+ transceivers I went ahead and got those QSFP to SFP+ adapters.

So I powered up the switch, plugged in the QSFP to SFP+ adapter, slid an SFP+ transceiver into it and hooked it up to one of my Brocade cards with appropriate fiber.
Unfortunately, I did not get a link (that would've been too easy, right?).

I figured the switch must be either
  1. broken
  2. not an Ethernet switch
  3. not compatible with the QSFP to SFP+ adapter and/or SFP+ transceiver
  4. not configured properly

Hopefully it is the latter.

Naturally, I tried the RJ45 port on the back labeled "Ethernet", figuring it must be for management porpuses, maybe even a web GUI.
I connected it to my Gigabit switch but unfortunately it didn't get a link either, suppose the port is disabled or something like that.

So my last resort was that RS232/console port. I hooked it up to my PC's COM port with a Cisco console cable (RJ45 to DB9).
I got a response - finally some progress!

Here's what it said during bootup:
Code:
CFE-NTSW-5.1.2 for BCM98548XMC (32bit,SP,BE,PPC)
Build Date: Tue Dec  1 08:04:52 PST 2009 (sdkrel@lc-sj1-3183)
Copyright (C) 2000-2008 Broadcom Corporation.

Initializing Arena.
Initializing PCI. [normal]
PCI[0] bus 0 slot 0/0: unknown vendor 0x1057 product 0x0013 (PowerPC processor, rev 0x21)
PCI[1] bus 0 slot 0/0: unknown vendor 0x1957 product 0x0013 (PowerPC processor, rev 0x21)
PCI[1] bus 2 slot 0/0: Broadcom product 0xb845 (ethernet network, rev 0x05)
Initializing Devices.
CORE clock: 997MHz
CCB clock: 399MHz
BCM98548XMC Board Id: 0x51
CPU type 0x80210022: 998MHz
Total memory: 0x20000000 bytes (512MB)

Total memory used by CFE:  0x1FF38300 - 0x1FFFFF30 (818224)
Initialized Data:          0x1FF79C40 - 0x1FF7CAB0 (11888)
BSS Area:                  0x1FF7CAB0 - 0x1FF7DF30 (5248)
Local Heap:                0x1FF7DF30 - 0x1FFFDF30 (524288)
Stack Area:                0x1FFFDF30 - 0x1FFFFF30 (8192)
Text (code) segment:       0x1FF38300 - 0x1FF7867F (263039)
Boot area (physical):      0x00000000 - 0x00000000
Relocation Factor:         I:20038300 - D:20038300

I'm basically clueless, so I type in help:
Code:
CFE> help
Available commands:

ptable save         Save flash partition table.
ptable clear        Clear flash partition table.
ptable del          Delete flash partition table.
ptable add          Delete flash partition.
ptable show         Show flash partition table.
@                   Boot VxWorks
c                   Change the VxWorks boot string
p                   Parse and display VxWorks boot string
M                   Show or set Ethernet MAC address
reset               Reset the system.
import nvram        Import env variabled to Vxworks NVRAM
pcicw               Write PCI config space
pcicr               Read PCI config space
pciw                Write PCI memory
pcir                Read PCI memory
phy set             Set the value of a PHY register
phy dump            Dump the registers on the PHY
write msr           write MSR register.
write hid0          write HID0 register.
read msr            read MSR register.
read hid0           read HID0 register.
crash               cause an exception
test timer          Test the timer
set date            Set current date
set time            Set current time
set console         Change the active console device
sleep               Wait for some period of time
loop                Loop a command
flash               Update a flash memory device
memtest             Test memory.
f                   Fill contents of memory.
e                   Modify contents of memory.
d                   Dump memory.
u                   Disassemble instructions.
pci dump            Dump PCI configuration space.
reserve             Mark a region of memory as reserved
autoboot            Automatic system bootstrap.
batch               Load a batch file into memory and execute it
go                  Start a previously loaded program.
boot                Load an executable file into memory and execute it
load                Load an executable file into memory without executing it
save                Save a region of memory to a remote file via TFTP
ping                Ping a remote IP host.
arp                 Display or modify the ARP Table
ifconfig            Configure the Ethernet interface
show flash          Display information about a flash device.
show time           Display current time according to RTC
show pci            Display information about PCI buses and devices
show heap           Display information about CFE's heap
show memory         Display the system physical memory map.
show devices        Display information about the installed devices.
sizeenv             Get environment space size information.
clearenv            Clear environment variables.
unsetenv            Delete an environment variable.
printenv            Display the environment variables
setenv              Set an environment variable.
help                Obtain help for CFE commands

For more information about a command, enter 'help command-name'
*** command status = 0

I didn't play around with it any further, as I have neither the experience nor the knowledge regarding VxWorks and the like.

Needless to say, I would love to get it to work, as 10GbE switches tend to be very expensive and I got lucky on this one (provided that it actually works).

I was hoping some of you more knowledgable folks could help me out here :)

Thank you very much in advance and sorry for the long post!
 
It looks like there is no OS on that switch. What you'll most likely need to find out there on the internet is a version of Broadcom FastPath that works with your device, and then have some fun getting it onto the device.

Although, it might not be that hard depending on what kind of flash it's using. Might want to open it up to see if the flash is either soldered on, or is some form of normal flash card.
 
Interesting Switch, searching online is indeed hard. BCM956845KQ brings up nothing, even on broadcoms site.



It looks like its a first gen STRATAXGS switch
http://www.broadcom.com/products/features/BCM56846.php

also I found this
https://www.broadcom.com/collateral/pb/956846KQ_02-PB00-R.pdf

You can find Vxworks for download on various ahem...sites.


I'm curious run the following commands and see what it shows

- ptable show

These might not work
-show ver
-show run
-show ip http

and lastly a @
see if it boots.
 
Last edited:
Thanks for your answers!

I opened the switch up but failed to find a flash card.
There's an "add-on card" (or whatever you call it) that has the 2 RJ45 ports I mentioned in the OP and some memory. See pictures below.



Here's a full shot of the PCB (sorry for the bad quality):



It actually says "BCM956846KQ" on the board (as opposed to "BCM956845KQ" on the front bezel), which matches the product number in the PDF Ruffy linked.
Also, it says BCM56845A4KFRBG/P14 right below the heatsink (can't really see that on my picture), which is similiar to the BCM56846 chip mentioned in the PDF.


You can find Vxworks for download on various ahem...sites.
Won't I need a special version for this device? Or is it "generic"?


I'm curious run the following commands and see what it shows

ptable show:
Code:
Partition Name          Offset          Size
--------------------------------------------
*** command status = 0

As you guessed, the show commands did not work:
Code:
Available "show" commands: flash, time, pci, heap, memory, devices

I ran the available ones (except time):

show flash:
Code:
     show flash [-sectors]

     -sectors     Display sector information

*** command status = -2

I also tried show flash -sectors, same response (suppose it needs a value after -sectors)

show pci:
Code:
PCI[0] bus 0 slot 0/0: unknown vendor 0x1057 product 0x0013 (PowerPC processor, rev 0x21)
PCI[1] bus 0 slot 0/0: unknown vendor 0x1957 product 0x0013 (PowerPC processor, rev 0x21)
PCI[1] bus 2 slot 0/0: Broadcom product 0xb845 (ethernet network, rev 0x05)
*** command status = 0

show heap:
Code:
Total bytes:       524288
Free bytes:        512552
Free nodes:        3
Allocated bytes:   7776
Allocated nodes:   162
Largest free node: 504384
Heap status:       CONSISTENT

*** command status = 0

show memory:
Code:
Range Start  Range End    Range Size     Description
------------ ------------ -------------- --------------------
000000000000-00001FF37FFF (00001FF38000) DRAM (available)
*** command status = 0

show devices:
Code:
Device Name          Description
-------------------  ---------------------------------------------------------
uart0                NS16550 UART at 0xE0004500
uart1                NS16550 UART at 0xE0004600
flash0.os            New CFI flash at F8000000 offset 00000000 size 130048KB
flash0.boot          New CFI flash at F8000000 offset 07F00000 size 1024KB
flash1.os            New CFI flash at F0000000 offset 00000000 size 130048KB
flash1.boot          New CFI flash at F0000000 offset 07F00000 size 1024KB
flash2               New CFI flash at F8000000 size 131072KB
flash3               New CFI flash at F0000000 size 131072KB
BoardId0             PCF8574 on SMBus channel 1 dev 0x27
eeprom0              Microchip 24LC128 EEPROM on SMBus channel 1 dev 0x50
clock0               ST M41T81 RTC on SMBus channel 0 dev 0x68
eth0                 MPC854X TSEC0 at 0xE0024000 PHYBASE 0xE0024000
eth1                 MPC854X TSEC1 at 0xE0025000 PHYBASE 0xE0024000
eth2                 MPC854X TSEC2 at 0xE0026000 PHYBASE 0xE0024000
eth3                 MPC854X TSEC3 at 0xE0027000 PHYBASE 0xE0024000
*** command status = 0

@:
Code:
Error: Invalid boot device ()
Incorrect or invalid fields in VxWorks boot line
*** command status = -1


Let me know if I can run anything else and thanks a lot for your help, it's appreciated!
 
This is an internal test device so you'll never find support for it unless you hang out at their campus in Irvine and rub shoulders with someone from their test lab. Hope you didn't pay more than $50 for it.
 
You're missing a firmware, as others have suggested it's based on vxworks but its specific to your hardware meaning that it doesn't work as a universal operating system. You best bet would be to kindly ask Broadcom for a firmware as you seem to be missing one and instructions on how to install it.
//Danne
 
Got the ethernet management port to work (which was actually embarrassingly easy).

Also, I found this: http://archive.openflow.org/wk/index.php/ImageSetup#The_Broadcom_Reference_Designs
Looks like the firmware can be transferred via TFTP.

Has anybody heard of OpenFlow? Or Indigo?
I think the latter could be used as an alternative to VxWorks, unfortunately my hardware doesn't seem to be supported.

The Pronto/Pica8 3780 switch is supported by Indigo and could be similiar to mine, by the looks of it:
A 48x10Gb system based on the BRCM 56840 Chip

The Indigo site also lists "select Broadcom reference designs using XGS chipsets" as being supported, so maybe there is still hope.


EDIT:
Apparently there's an updated site for the Indigo project: http://www.projectfloodlight.org/indigo/
 
Hi,

The switch is a reference design from Broadcom. There is a powerPC board inside with VxWorks. (If not erased.) CFE is the Common Firmware Environment. (bootloader from Broadcom) When power is connected it should boot VxWorks unless you press Ctrl-C within 3 seconds.(Depends on the environment settings.) With @ you can boot from CFE. Check with p if there are boot parameters.
If you manage to start VxWorks the Broadcom Command Manager (BCM) is started. This will enable you to configure the asic. Configure meaning that you can access the switch more or less a bit above register level. You should be an expert on layer 2 and 3 of the network protocol to use this.
OpenFlow is a protocol to configure switches of different suppliers. If you want to use it you need Broadcom to supply a Hardware Software Interface (HSI) between the OpenFlow agent and the asic. Indigo is an OpenFlow agent you could use to run on the powerPC. You should compile it for VxWorks. You also need an OpenFlow controller to run on whatever you use to control the switch.
All the documentation is classified so you need a NDA with Broadcom.Documentation is several thousend pages and far from complete so you also need support from Broadcom.
OK that's the bad news. The good news is that your switch is working. It shows up on your PCIe list. Even better news is that you have a well paid job if you manage to get OpenFlow running on that switch. I can give you some support but we have a NDA with Braodcom so there is a lot I can't tell you.

Regards,

Rob.
 
Back
Top