Ground Loops Galore (SPI Application)

I’m working on a project that has ICs daisy chained together through SPI… and for the life of me I cannot get them to communicate robustly. After many hours, I’ve reduced all of my problems to the grounding scheme I’ve used for my project. I’ve attached a simple schematic showing my current wiring scheme so we are all on the same page. I also know I’ve written a lot of text here, so I’ve bolded my questions to make it a little easier haha. Thank you for your patience and help.

So the first thing you might notice is how the ICs are wired to power and ground. I went with this scheme for 2 reasons:

  1. I cannot daisy chain the power for too many ICs. The amount of current builds up with each IC in the chain, and the 22AWG wires and PCB traces cannot support that much current.
  2. I was somewhat lazy, and didn’t want to spend a week creating individual power cables that split off of the main 14AWG branch and route to each of the 60 connected ICs.

This is where my first problem comes in. I believe that splitting the large SPI daisy chain into smaller sub-chains for power are influencing the SPI communications (the data gets garbled) due to different ground potentials at each IC. Is this method of splitting the power bad, and if so, why?

I saw a blog post the other day where a guy had done daisy chain ground wiring exactly the way I have… and he found that by connecting the end of the chain back to the main ground line fixed it. For example, in my schematic connect A and D back up to the main 14AWG ground line. I tried this and the SPI comms improved… but doesn’t this create ground loops?

I’m thinking that I should just bite the bullet and wire an individual ground to each IC from the main branch, but that will take a lot of work. I want to check with some electrical experts before starting so I don’t waste my time. Thoughts?

Now my second problem is grounding the Arduino so that it can communicate with the ICs over the SPI line. Currently, I have the Arduino ground wired to the 14AWG main branch outside of the project enclosure. This does not work; the SPI data gets all garbled. If I open the enclosure and attach the ground to the main branch but very close to the ICs, it does work. Why does the length of ground wire matter so much in this application?

In the current setup, I have to attach an Arduino ground to the main branch because the Arduino is being powered by my Laptop. I understand this creates a ground loop, which I believe may be the answer to my previous question. If I power the Arduino directly from the 14AWG main branch, I believe I can eliminate the ground loop. I’m going to try this when I get home, but do you guys think this will help?

Finally, if powering the Arduino from the same supply as the ICs won’t work, I believe I will need an isolator for the SPI lines. Do you guys know of any isolators good for 1-4MHz SPI?

Thanks again for all of your help guys, and sorry for the long first post. I had so many questions and wanted to make sure you guys had all the details you needed.

  1. I cannot daisy chain the power for too many ICs. The amount of current builds up with each IC in the chain, and the 22AWG wires and PCB traces cannot support that much current.
  2. I was somewhat lazy, and didn't want to spend a week creating individual power cables that split off of the main 14AWG branch and route to each of the 60 connected ICs.

What are A to F? ICs? What type? PCB assemblies? How much current is each one drawing?

The SPI bus includes CLK, MOSI (so output from Arduino only?) and a SS line for each of A to F?

How long are the power and data cable runs?

What decoupling capacitors do you have fitted on A to F?

A-F are MAX7219s on a breakout PCB that passes VCC, GND, CLK, and CS(SS) straight through. Each one can draw up to 330mA.

The SPI bus has a MOSI, CLK, and one SS line. A-F all use the same SS line; the rising edge of SS tells A-F to latch whatever is currently in their registers.

Power and data cable runs are ~3ft each.

Each PCB has two caps. The MAX7219 is powered through a 10uF electrolytic in series with VCC and decoupled with a 0.1uF ceramic.

Thanks.

And you have 60 of the breakouts in total? So, 60 items like A, B, C etc? Or six breakouts each with 10 x MAX7219?

Can you post a link to the breakout schematic. What sort of decoupling does it have on it?

Yes, 60 PCBs in total (only 6 are represented in the schematic). The power is split into chains of 6 instead of the chains of 3 you see in the sample schematic.

I unfortunately don’t have a schematic of the PCBs since they are made god knows where in China, but I’ve attached two pictures to show each side of the 2 layer PCB. I know its not ideal but all i have to work with haha.

I edited my second post to discuss the decoupling and power smoothing caps.

Thanks for all the details. From the photos, it looks like each PCB has the 10uF and 100nF in parallel across Vcc and GND, which makes sense.

No problem. I believe the caps are providing stable power to the MAX7219s, but what I am worried about is the ground as reference to the SPI bus... does that make sense?

So I’m at home, and I’ve powered the arduino from the power supply as well. So now there are no ground loops. However, there is a difference of up to 150mV in the grounds of the Arduino and the IC I am trying to communicate with over SPI. I’ve checked the two grounds with a continuity checker when the circuit was powered off, so why is there such a large potential between them when the circuit is powered on?

I believe this is the ultimate problem. By taking a ground pin from the Arduino and connecting it very close to the ICs, the potential across the grounds drops to zero and the SPI works. Thoughts?

I've powered the arduino from the power supply as well. So now there are no ground loops. However, there is a difference of up to 150mV in the grounds of the Arduino and the IC I am trying to communicate with over SPI.

In this test, at what point did you have the Arduino ground connected to the ground bus?

Were you able to test multiple points on the ground bus? How did the voltage difference vary depending on which chain (of the 10) you tested and how far down that chain you were (from the ground bus)?

Trying to run the SPI lines with no ground is the fault.

You must accompany every logic signal wire with its ground, preferably as a twisted pair if going any distance (20cm or more). At a pinch you can get away with one ground wire shared between SCK/MISO/MOSI, but it must be bundled with those wires in close proximity. This is signal ground and at logic speeds its absolutely vital. The original diagram shows a complete lack of signal ground on the SPI bus.

With a long SPI run you are going to have to consider signal termination too. If you don't know what termination is stick to short SPI runs (50cm max?).

What total current is your supply wiring taking? How long are the cable runs? Are they twisted pair? Have you measured the voltage at each board/module? Do you have a large decoupling capacitor per board (47uF perhaps)?

HackScribble:

At first, I had no Arduino ground pins connected as the Arduino itself was being powered from the power supply (e.g. they were already on the same ground). But as MarkT mentioned, this didn't work at all so I re-attached the Arduino ground pin to one of the chains I was trying to communicate with. Before I did this, however, I measured the voltage differences in ground pins on each chain. I had one probe on the first PCB's ground in the chain (closest to the main 14awg branch), and then measured the ground voltage difference on each PCB in that same chain. The ground voltage difference increased by around 20mV with each PCB in the chain. I think this pretty large difference in ground voltage is the major cause of my problems, so I think I'm just going to have to quit being lazy and do away with the chains, wiring every single PCB's ground directly to the 14awg main branch.

MarkT:

In the schematic I do have a signal ground, but it attaches to the main 14awg branch outside of the enclosure. This doesn't work, but if I attach the signal ground directly to one of the PCBs inside of the enclosure then it works, which makes sense with what you were saying.

I see the importance of what you said about making considerations for the signal ground, but I'm having a tough time wrapping my head around the best way to do that. What do you think is the best way to wire up these grounds? Do you think you can quickly sketch how you would do it?

I checked the shape of the data/clock signals at the beginning and end of the SPI bus with an oscilloscope, and they look good without termination/twisted pairs. There is very low overshoot and almost no ringing, with nice sharp rise times.

Each PCB can draw up to 330mA depending on how many LEDs are on, and there are 60 PCBs for a total of close to 20A. My power supply can only output up to 9A... so I'm going to put a fuse on there and make sure in software that the PCBs never have enough LEDs on to draw over 9A. Also, each PCB has a 10uf decoupling cab for the board, and a 100nF decoupling the MAX7219 IC onboard.

Alright guys, last night I started wiring each individual PCB to the main 14AWG power branch in an effort to equalize the grounds on all PCBs. I’ve attached a picture showing the new connection scheme. Do you guys think this will work?

I’m particularly concerned with how to hook up the SPI signal ground in this case. You can see in the picture how it hooks up to the first PCB in the SPI chain. Do you guys have any thoughts on this?