MAX7219 Daisy Chain... soooo close!!!

So I've been posting on the forums a few times now and then looking for guidance on hooking up 60 MAX7219s in a daisy chain configuration. After trying many things, I finally followed CrossRoad's initial advice and put some buffers in the chain for the SPI clock signal. Every 6 chips I'd insert the buffer. This worked great for the first ~24 chips. Once I reached 30 chips and onward, I started to notice that the chips after the last buffer wouldn't work until another set of 6 chips+buffer was connected after it.

I now have all 60 chips hooked up with buffers, and the last 6 chips don't work. I've verified that ICs, power, and buffer for the last 6 chips are working properly, and that the problem is not due to the clock rate. It's just that they are at the far tail end of the SPI chain that they aren't working, and I can't figure out why. Anybody have any ideas what's wrong? Keep in mind only the CLK signal is passed through each buffer. The only thing I can think of is that it may be reflection of some sort?

It really helps if you offer up a schematic.
That said, I assume you have a scope and can tell us what you are observing signal wise.

Try a pull up resistor on the last set of outputs.

Also make sure you have good supply decoupling.

Could also add a buffer 1/2 thru for the chip select line. Could be it's not getting pulled low enough at the very end.

I'll draft up a schematic in the morning, about to head to bed for now. I do have a scope, but not sure how to capture a sample of the clock waveform as I am an oscilloscope noob. I know how to view fully periodic signals, but the clock signal is only periodic within a single SPI.transfer() call and not amongst multiple SPI.transfer() calls... know what I mean?

I have a feeling it is either the CLK line or the CS line, most likely the CLK line. The fact that the separation of chips that work vs chips that dont work is right where the buffer is located (and remember only CLK is going through the buffer) leads me to believe its the CLK line.

Also, every single IC (MAX7219s and buffers) and every single PCB has decoupling capacitors. It's not noise on the power lines. As I mentioned, I verified that the last 6 chips and buffer do work properly by hooking the SPI master right up to the front of that chain's buffer (thus skipping the 54 chips before them), and they work perfectly. So I believe this problem has been isolated to the length of the chain, and not power/power-noise problems.

I know how to view fully periodic signals, but the clock signal is only periodic within a single SPI.transfer() call and not amongst multiple SPI.transfer() calls... know what I mean

You can write some simple code to toggle the line.

So I believe this problem has been isolated to the length of the chain, and not power/power-noise problems.

Many people believe many strange things. There has never been a circuit built that does not have noise, that noise only becomes a problem in the context of the actual setup. So the statement:-

I verified that the last 6 chips and buffer do work properly by hooking the SPI master right up to the front of that chain's buffer (thus skipping the 54 chips before them), and they work perfectly.

Means absolutely nothing.

Did you try the pull up resistor?
Your problem is with signal degradation. Did you put another set of buffers leading nowhere? If that cures it then it is a line termination problem.

I agree that noise is the problem, I was just trying to say that I don't believe it is noise on the VCC pins of the last 6 ICs. I believe its noise on the CLK line and/or degradation of the CS line.

I hooked up the SPI master right to the front of that 6 chip chain at the end to isolate the problem. By doing that test I was able to verify that the last 6 ICs/PCBs and buffer were functioning properly (a.k.a they weren't defective and their connections were good), so I think that does mean something.

I tried a 10k pull-up on the CS line, and 10k pull-downs on CLK and MOSI, but had no luck. Will try a pull-up instead on CLK tonight.

At earlier points in the 60 chip chain, putting another buffer afterwards fixed the problem, so I agree it must havebeen termination. But for these last 6 chips, wiring the CLK line to another empty buffer unfortunately doesn't do the trick.

I haven't tried combining the methods of giving CS a pull-up AND wiring the CLK to one last empty buffer. I'll try that tonight as well.

LarryD:
You can write some simple code to toggle the line.

I can toggle the line, but not nearly as close to the 2MHz that the hardware SPI clock (at SPI_CLK_DIV8) generates. It can't hurt to try, but I'm worried that the problem might not show up or might not be as prominent at the low speed of toggling an IO pin in software. I'll test tonight and let you know if I see anything interesting.

CrossRoads:
Could also add a buffer 1/2 thru for the chip select line. Could be it's not getting pulled low enough at the very end.

I will also try this tonight. You think it might be a capacitance problem there?

I'll draw up a simple schematic sometime today and post for you guys to take a look at. Thanks for all the input so far, much appreciated.

Power supply decoupling not only suppresses noise but reduces the impedance of the power supply.

When you do your test then look at how close to zero the signals go. Also look at the phase difference between the clock at the start of the chain and that at the end.

I can toggle the line, but not nearly as close to the 2MHz that the hardware SPI clock (at SPI_CLK_DIV8) generates.

You can just write a loop that keeps on sending the same SPI package over and over. That will be at full speed.

The other thing to try is see if dropping the SPI speed helps, you have DIV16, DIV32, DIV64, and DIV128 to go at as well.

All of the ICs and PCBs have their own decoupling caps. I'll take a look at the highs and lows and also the clock phase difference tonight.

I did set up a test previously with just looping calls to SPI.transfer(0x00), but the problem is that timing in-between SPI.transfer() calls doesn't match up with the clock period. This makes my scope's trigger freak out and makes it hard to see the signal.

I've also tried it at DIV128 hoping it was just a synchronization problem due to the propagation of delays, but the lower clock speed did not help.

Another thing might be the capacitance of the really long, unbuffered CS line. I'm wondering if capacitance is causing the CS line not to go high fast enough at the end of the chain. If I recall correctly (don't have my code in front of me right now), I don't have any delay between when I raise the CS pin to latch the data and when I start shifting out new data on MOSI. Maybe adding some delay here will give the CS line enough time to rise at the end of the chain before I start shifting new data. Make sense?

I'm also hoping CrossRoad's idea on buffering the CS line halfway through will help.

One question that I still have (from another thread), is why doesn't this system work when I pass all three SPI lines (MOSI, CLK, CS) through each buffer? If I do that, I just get garbage in the registers. It only works if I put CLK through the buffers.

We need to see that schematic. Are your buffers serial or star wired?
I would have thought that all signals need buffering.

is why doesn't this system work when I pass all three SPI lines (MOSI, CLK, CS) through each buffer?

Again schematic needed to see what you mean. It could be that the whole thing is working because of some fluke you have not identified.

The buffers are wired in serial. Would you like an actual schematic, or would a simplified one suffice?

Just put CS on the scope, and it takes ~300ns to settle. Its got some pretty bad oscillation going on, with like a 40% overshoot. I tested it both at the end of the chain (where the chips aren't working) and at the middle of the chain (where they are working), and it looks about the same. Thoughts?

Sounds like a line termination problem. That is what the pull up resistors would help to damp down.
That overshoot is called "ringing" and is caused by the inductance of the line and also by reflections off the impedance discontinuities in the signal path. While a serial wiring of the buffers will reconstitute the signal to a certain extent it will not remove things like the propagation delay the accumulates.

When you measure these points do you move the ground lead of the scope to the board you are measuring? You need to to get a true picture of the signal on each board.

Grumpy Mike, you are certainly right... it was a termination problem. I FINALLY got the whole array of MAX7219s to work. Had to do two things:

  1. Split the chain of 60 MAX7219s into two chains of 30, each with their own CS line (thus halving the length of the CS line).
  2. Added a 100ohm pull-down resistor to the end of each of the two CS lines.

I made them pull downs instead of pull ups because the MAX7219 latches data in on the rising edge of CS, which would cause them all to do weird things when the system first powers up. Do you think the pull downs will work as well as the pull ups you mentioned?

Also, I'm worried about the 100 ohms resistors. 5V @ 100ohms = 50mA draw, which is a few milliamps more than an Arduino Uno DIO pin can source. I tried with a 1K Ohm, but that didn't work (although the wires were pretty long for this resistor since I had it on a breadboard located about 10" from the last MAX7219. I noticed that even the 100 Ohm resistors weren't working well if they were far from the last MAX7219, but once I shorted the wire length they worked great. I wonder if the same will happen for the 1K Ohms, and that way I won't have to worry about drawing such a high current. Thoughts?

As far as line termination goes ground and power are at the same point so pull down is no problem. In fact the IEEE bus specifies both a pull up and pull down resistor in the lines.

Have you tried a 130R resistor that will keep the current down. The other thing you could try is a 100pF capacitor on the buffer inputs along with the resistor, this will in a small way " nutrilise" some of the inductance.