At 4Gb, a full packet is about 1 km long, and at 2Gb a full packet is about 2km long! Yes, at any given time 2k of your data is spread from the port to 1km down the cable (as the light travels). Each packet burns 1 buffer, no matter the size of the packet. The buffer credit isn’t released until the packet gets to the receiving switch and the sending switch receives an acknowledgment. So, at 4Gb, you need 2 buffer credit for each km of distance. 1 to fill the 1km pipe to the receiving switch, and 1 waiting for the acknowledgment.
So, for a 10 km fibre path, you need 20 buffer credits, assuming that every packet is a full packet. Even if every data packet is full, there is some inter-switch communication that doesn’t fill a packet. That means that the packets are shorter and take less time to transmit, meaning there are more packets in the pipe. So, the 2 packets per km number is the MINIMUM number you’ll need. And, we’re not counting the time it takes to process the frames on the switches, which is probably pretty minimal.
Also, be aware that on a 48k, there are 1024 buffers per ASIC, out of which 8 buffers per port are reserved. So, if you put a bunch of long distance ISLs all on one ASIC, you might run in into problems because there aren’t enough buffers to go around. If you have a GoldenEye based switch, you only get 700 buffer credits per ASIC, out of which 8 buffers per port are reserved. Use portbuffershow to see your buffer usage per ASIC.
SM
Can you further explain the 48K bit. Let say we are looking at slot 1 which has a total of 32 ports. 16 ports on the top asic, and 16 ports on the bottom asic, right?
So does that mean that the top 16 ports has a max of 1024 buffers, minus the 8 buffers x 16 ports = 128 buffers? When you say “8 buffers per port are reserved” what are they reserved for? So 1024 -128 = 896 buffers left.
In my case I have alot of isl on the top asics of each slot, 8 on average with a 2km run. So each 4GB 2km run would require a min. buffer of 4 correct? 4×8=32, still lots of room to grow. What I find odd is that my port with an array on it has a high tim_txcrd_z of 1237549 after less than 24 hours. Is this normal? Is there a script to gather all the tim_txcrd data per port, without having to type in portstatsshow 1/1, 2/2….
Thank You
Pat Vaughan
It’s been a long time since I looked at this, but I think you’re right.
The 32 Port blades are broken down:
ASIC 1: ports 0-7 and 16-23
ASIC 2: ports 8-15 and 24-31
Each port reserves 8 buffers for it’s normal in and out traffic. That’s all most shortwave ports need because the distance is so short. You’ll see every port uses 8 buffers in portbuffershow
I haven’t been able to completely eradicate buffer exhaustion, but I have been able to limit it. You used to be able to specify a number of buffers per long range port, but now you have to specify kilometers. I spoke with some Brocade technical resources, and they indicated that increasing the distance number for the port only really increases the the buffers reserved. We were originally concerned that it actually increased the power on the laser and may cause damage to the SFP or the cables if it was artificially increased, but that’s not the case. I think that Brocade is probably conservative with their numbers in an effort to keep customers from exhausting all the buffers on an ASIC and causing more wide-spread problems. I would bump up the number of buffers on a couple of ports and see if it helps. Oh, and the distance parameter on both sides must match, so you’ll have to disable the port on both sides to make the change.
Another thing is that distance isn’t the only factor in buffer credits. You may have a device that can’t take the frames as fast as the port delivers them (tape drives are notorious). That makes the switch buffer more data during peak times. Some of the advanced features of the 8Gb ASICs like rate limiting are good for helping that.
As for monitoring, I don’t have anything good. You could get possibly that data into a tool like Nagios via SNMP.