Tach circuit causes instability/crashing

I'm building a motorcycle tach/speedo project and the tach circuitry has bee giving me some grief. I'm using inturupts to read both the wheel sensor and the tach signal, but when I have the tach actually wired to the circuit I get spurrious triggers on the wheel sensor, and eventually the entire atmega just locks up.

Complete schematic here
Complete source code here

Code is just for completeness, I don't think it's a software issue because it will run fine on the bench with a simulated tach signal for hours.

The tach signal is a roughly 13.5V square wave, that's extremely bouncy and with peaks in the neighborhood of 70V. I'm feeding the signal into the base of an NPN transistor with a simple RC debounce circuit at the collector (connected to 5V system voltage). The RC filter is connected to a Schmidt trigger inverter which is then tied directly to the interrupt pin.

Source signal oscilloscope
Debounced output oscilloscope

That output signal is what goes into the Schmidt trigger. I don't see any big spikes or other noise, and the signal that the Atmega input pin sees is also just as clean. Where could this "interference" be coming from, and how is it "infecting" the rest of the circuit?

I thought it might be picking up interference from the ignition, but with the tach sense wire grounded instead of connected to the tach signal the problem goes away. I thought about opto isolating the tach signal, but since there is only one common ground that wouldn't accomplish anything that the transistor doesn't already do correct? Previous to this arrangement I was using a voltage divider with a Zener diode to control the signal, but that had the same problem only much much worse.

There are a couple of red flags in your schematic that could be leading to the erratic behavior.

Are you sure D1 is connected to pin 2 of of the LM2576? There's no junction dot there in your schematic.

C6 and C7 should be at least 0.1uF (1uF is even better). A 0.01uF decoupling capacitor may be too small (this might fix your issue by itself!)

On U2 and U3, as CMOS devices, you should not leave unused inputs floating. Any unused inputs should be tied to GND (or supply voltage, I prefer GND) to prevent the outputs from oscillating or the device from consuming excessive power.

U3 is powered by 3.3V but you are driving it with 5V inputs. Are you sure that the device can tolerate inputs beyond its power rail?

U2 and U3 should have decoupling capacitors as well.

--
The Basic Motor Driver: simple, inexpensive motor driver for 1 stepper motor or 2 DC motors

D1 is connected, since they both connect directly at the pin tip it seems that eagle doesn't require there to be a junction for the nets to be merged, but I'll make that more clear.

C5 was intended to be the large (.1uF) decoupling cap for the 328, with AVCC and VCC independently decoupled right at the pin with .001uF caps, and then both power nets merged as closely as possible at the .1uF cap before being connected to board power. But that layout didn't come across on the breadboarded version, I may have been extremely optimistic about C5s placement. I'll upsize it and move it closer.

U3 is specifically used as a level shifter to protect the 3.3V SDCard from the 5V outputs. The 3v section of the board isn't actually wired right now.

So if I'm thinking this through right the floating inputs on U2 and it's complete lack of decoupling could potentially cause it to falsely trigger it's output and suck up a lot of current. Alone this would be a localized problem but because the 328 is insufficiently decoupled as well it's causing it to lock up. When the signal lead is disconnected from JP2 the area is electrically quieter and the floating inputs on U2 aren't switching as often, so it's staying below the threshold that causes the 328 to drop out.

Should be a simple (and now so obvious) enough place to start, thanks a lot!

Progress, but no solution yet. Added the decoupling caps and tied down the floating inputs. The circuit now survives cranking (was falsely triggering low power shutdown previously) and no longer seems to have false wheel sensor triggers, but the crashing and unusual behavior still happens. With the tach sense wire disconnected (even if it's floating, not grounded) the problem goes away entirely.

updated schematic (U3 is still not changed, I'm aware)
Photo of the bread boarded version with notations for what each region of the board is and the location of the two power rails (3 new decoupling caps are just photoshopped in)

And for the curious, a short video of the device crashing. The only two parts of the display that are actively being refreshed are the tach numbers in the middle and the tach bar graph on the right. The full screen filling with blue at 18 seconds I can't explain, there are no commands in the code that could do that.
video
--edit--
sorry about the video, it seems to be auto imbedding my link and I'm not sure how to make it stop.

extent:
With the tach sense wire disconnected (even if it's floating, not grounded) the problem goes away entirely.

There is also the possibility that the issue is software related and one probable cause is running out of RAM. Looking briefly at your code, I notice you’re using malloc() and also the string library. To gain better control of memory usage, you should try to avoid using either (calculate worst-case and pre-allocate static RAM accordingly). A quick test where you narrow the code down to RPM only may reveal if the issue is hardware or software related.

I have been a little concerned about mem usage recently, and I've been using the freeMemory() example from playground to watch it, currently it reports just about 1k free when the software is compiled for the 328. I am using malloc, but currently only in the constructors of objects that are instantiated at startup, and never during "runtime".

extern unsigned int __bss_end;
extern unsigned int __heap_start;
extern void *__brkval;

int freeMemory() {
  int free_memory;

  if((int)__brkval == 0)
     free_memory = ((int)&free_memory) - ((int)&__bss_end);
  else
    free_memory = ((int)&free_memory) - ((int)__brkval);
  return free_memory;//-(6*1024);//compensate for the extra SRAM in the Mega
}

I haven't tried cutting it down to just the RPM code, but I have bench tested the code fairly extensively in two different ways. One way that taxes all of the string conversions and the display code (feeding random numbers to the update routines, causing every single frame to re-evaluate everything), and another way that tests the input sampling and averaging routines (hooking up a signal generator to the input pin) Both of these tests have run for hours at a time without issue (and both tests run on both the speedo and tach code simultaneously) If I was leaking anywhere I would expect either of those cases to crash quite quickly.

I'm still fairly convinced that it's a hardware problem, and I think my next test is to start at the 328 input pin and desolder one segment at a time, just to see how much of the input circuitry can be on the board before it starts to effect things (it's only a handful of joints anyway) And I'll start thinking about a cut down RPM only bit of software.

extent:
If I was leaking anywhere I would expect either of those cases to crash quite quickly.

Looking at your video, the mcu doesn’t seem to restart, which would have been a give away for a power or reset/issue (hardware/wiring related). Another lockup possibility is if you trigger interrupts faster than you can handle (such as with a noisy trigger). Your scope traces however suggest this is not the case (unless trigger pulses change prior to lockup). Also, such issues tend to materialize at or around a particular RPM frequency. Looking at your RPM display however it seems more related to running time.

The following link is related to issues with the string class. There are a number of issues here that are difficult to trace (including a malloc() bug). If you use this class for every display refresh cycle you’re likely to run into problems.

http://code.google.com/p/arduino/issues/detail?id=468

wow, the tach signal is really wild. Is that the bike's factory hookup? Does the bike's tach handle it ok?

There's a lot of stuff on the board and a lot of potential for bad connections or high resistance areas where the tach noise could creep in. My tach circuit is very clean by comparison but I still had to put extra caps in because of bike-induced noise.

Is the raw power rail(15v) as messy as the tach? I am not a power supply giant but I would think the inductor might belong on the outside of the regulator, that's where the heavy duty noise would be.

I had lockup problems with a similar circuit and i only got stability when I got rid of everything i could and shortened all the traces. I even got rid of the resonator to just cut down on the wiring.

Firstly without the layout there's no way to know if your decoupling is adequate - decoupling caps need to be physically close to the chips they decouple and with low-impedance connections. I'm personally add 10uF ceramics to strategic places too, 0.1uF isn't enough if large currents are switched (the LCD backlight for instance). Groundplane is pretty important to reduce interference too.

Is the decoupling cap C20 really 1Farad? That's far too large for the switching regulator to handle, the LM2576T datasheet suggests 1000uF, 1000 times smaller.

I'd add about 300pF directly across the tach leads to short out RF noise.

That's the factory tach signal, the bike is from the early 80s, so things are a little more rough and tumble with it in places :stuck_out_tongue:

Where specifically did you have to add extra caps? Your circuit looked fairly simple interface-wise.

The supply rail isn't totally smooth, but it isn't nearly as bad as the tach. On it's own it doesn't present any problem. The inductor is part of the switching supplies reference design. I could possibly add a 2nd inductor on the input, but if it was a basic supply problem I would expect it to effect the micro all the time regardless of it's other hookups.

I was picking up a lot of interference when parts were on a solderless breadboard, I was actually picking up tach pulses (accurately!) from 5 feet away out of thin air. It's why I had to build this circuit as it is now to get rid of the noise problem from doing exactly the same thing. Now that it's built up the problems are no longer happening when the board is not connected to the bike, or when the wheel sensor is hooked up, it's just this one last line that's got me.

I made Pauls string/malloc changes and I'm rerunning the display test while I'm off to work, but it's already been several minutes and it's still running successfully like it always does (even at double normal update rate). I'll see about removing all of the string class objects from the display code.

Now it is possible that a "crash" isn't necessarily the micro hard locking up, if there's something wrong with the serial link it could just be the display not updating anymore. You can kind of see this start happening in the video, where the tach display just dissapears (clear command goes through but the redraw command dosen't) and a little bit later it flicks back on briefly. I am sure that I have had hard-lock situations in the past tho (pre decoupling changes) because the low voltage watchdog function would not fire when power was removed (screen would remain on, and flicker several times as the supercap drained). I should probably put a heartbeat LED somewhere on the board.

Another question - you say the tach input is 13.5V, but how do you think you are clipping that voltage to the 5V supply range? Are you relying on the 74HC14's input protection diodes? That's a recipe for CMOS latch-up.

MarkT:
Firstly without the layout there's no way to know if your decoupling is adequate - decoupling caps need to be physically close to the chips they decouple and with low-impedance connections.

I haven't done a layout yet because I'm still building parts on the breadboard. It is an incomplete picture, but you can see the decoupling cap at the head of the 74HC17 and the two decoupling caps immediately to the sides of each of the 328 supply pins.

MarkT:
Is the decoupling cap C20 really 1Farad?

That's a cap to keep the board alive long enough to properly shut down when power is removed. The 1000uF decoupling cap for the Vreg is immediately next to it in the schematic.[/quote]

MarkT:
I'd add about 300pF directly across the tach leads to short out RF noise.

The stock tach gauge does have a 100pF immediately on it's board, but I'm worried about adversely effecting it's circuitry by whatever I add to my own (both are plugged in at the same time). I suppose that the input diode would prevent that from happening. hmm..

MarkT:
you say the tach input is 13.5V, but how do you think you are clipping that voltage to the 5V supply range?

The tach signal is driving a NPN transistor, which is supplied with 5v. The 74HC14 is driven from the collector of the transistor. I know the axis aren't labeled, but you can see the input signal to the 74HC14 in the OP, it's well contained between the supply rails.

I was picking up a lot of interference when parts were on a solderless breadboard, I was actually picking up tach pulses (accurately!) from 5 feet away out of thin air. It's why I had to build this circuit as it is now to get rid of the noise problem from doing exactly the same thing. Now that it's built up the problems are no longer happening when the board is not connected to the bike, or when the wheel sensor is hooked up, it's just this one last line that's got me.

What about shielding the board? Literally wrap it in foil tied to bike ground.

Well the free air interference went away once I boarded everything and made all the traces shorter than 6" each :p. I did think a bit about changing my sensor cables all to shielded ones but since the general interference has been gone it hasn't seemed really necessary.

Shielding the case I would think more about if I was picking up stray signals with the sense wire floating. It's still routed the same way as when it is connected so if it's acting as an antenna it should be picking up the same interference when disconnected as well. If the problem is being caused by the signal on the wire when connected then wouldn't that bring it right thru any shielding?

Display code has been running for 12 hours straight now, 18 updates/sec and I still can't replicate the issue on the bench. I have excised all of the String objects from my display objects for good measure (and freed up over 100 bytes of ram in the process). Putting the updated code thru it's paces again overnight, and tomorrow I'll poke around with the hardware some more.

So some interesting results so far. I added two heartbeats, one on the display, and one as an independent blinking led. Testing with the tach circut disconnected at the input of the 74HC14 (and that input grounded) I was eventually able to get the display heartbeat to begin skipping beats, but the independent one never had any problems. So it is entirely possible that the current problems are entirely due to the display module getting upset, and the atmega is running just fine.

I tried to add a 1uF cap between the power pins on the display board connecter (which is at the end of an approx. 3" hand braided, unshielded cable) and scoped it and was very surprised at what I saw.

It's picking up a ton of ignition noise. When I removed the cap and left the power lines unaltered it seemed to peak much much lower (but still uncomfortably above my 5.5v max)

For kicks I placed a reverse biased signal diode between the power pins and it seemed to improve the peak somewhat, but obviously this isn't noise from the screen, and it's not an inductive load, so maybe I'm just seeing things?

When I kill the engine this peak goes away entirely, so it's not coming from my power supply. I also noticed that when I moved my hand very close to the cable bundle that the peak all but disappeared as well, what would explain that? Is it possible that this is just being induced by the cheap leads in my DSO Nano?

I'm working on hooking my remaining pins up to a plain parallel LCD display. If it's only this OLED display module that's upset by the interference then that changes things a bit.

You are on the right track with your scope by using it to see the "noise" that your device is dealing with. When you moved your hand and the noise went away, you were shielding the parts with your hand, effectively grounding out the interference. You might want to look up "shielded cables" to see what they look like.

Spark plugs are fired at around 35,000 volts +. If you have a mid 90's Chevy pick-up, the spark plugs fire at around 50,000 volts. My point is that spark plugs generate signals that are a lot like lightning storms that mess-up your Television Set or your radio.

Electronic devices have had to be developed around these problems so that when you are in car and listening to the radio, you do not hear the spark plugs firing over the speakers. I had a 75 Dodge truck with terrible spark noise that would come over the radio. It was like a high pitch buzz that varied with engine speed .

The solutions are numerous, twisted pair wiring, capacitors, inductors, shortened traces, and shielding to name a few.

You will have to experiment,
Mark

Ya, unfortunately I think my scope is causing more problems than it's helping. I hooked the 2nd LCD display up like I mentioned and got it to a point where one of the screens was "crashed" while the other still functioned fine (for a short time :p) confirming that the displays are my biggest problem at the moment. With my test leads hooked up to the power pins of the (new) display I can almost guarantee display corruption within several seconds of regular display updates beginning to take place. With the leads unplugged the system will run for significantly longer before displaying issues (and sometimes not at all)

So I'm taking a step back and going to go get a solid metal project box for this and attempt to bundle the whole thing up and then work backwards from there. I'm assuming that I should ground the shield to the same "ground" that I'm pulling power from and I should keep the shield grounds off the PCB as much as possible (entirely with a shielded box)

I see lots of talk about RF chokes, but that all seems to be in the Mhz range, where the big ignition pulses are all in 10s to 100s of Hz, and with pulsewidths more like those of a 40khz signal. I'm thinking about trying RC filters on all the cable lines, switching to proper twisted pair wiring, and/or TVS diodes.

I guess I was really hoping to avoid lots of that because of all the other bike and scooter projects that seemed to get by with minimal suppression and almost no shielding.

extent:
I see lots of talk about RF chokes, but that all seems to be in the Mhz range, where the big ignition pulses are all in 10s to 100s of Hz, and with pulsewidths more like those of a 40khz signal. I'm thinking about trying RC filters on all the cable lines, switching to proper twisted pair wiring, and/or TVS diodes.

Ignition pulse occur at 10s to 100s of Hz but, the signals produced by them are transients that cover many frequencies. If you could imagine a signal "blast" kinda of like hitting 30 piano keys all at once. It would sound like one bad note but, the note contains many frequencies both high and low Hz.

Mark