Problem with custom Atmega2560 board, cannot use FT232 for Serial comms

Here is the relevant part of the circuit. Does anything that is floating look as if it should be brought high or low?

How do you mean? Neither the ftdi nor the 8u2's firmware do that, they just impose the baudrate sent over usb by the pc on their uart.

Oh, OK then. So when the PC/Mac connects at a certain baud rate, it configures the FT232 to expect that baud rate on the "other side"?

About the theory of overflowing the ftdi's 64 byte buffer. What if you send data only every say 30 msec? (the buffer is flushed over usb after a configurable timeout which is 16msec by default).

I tried a loop sending a letter at a time, with a 100 mS delay, still nothing.

How is the ftdi connected? Is there an auto reset line (DTR or RTS) towards the atmega? Are the other flow ctrl signals connected (toggling cts,... flushes the buffer too)

See above. I presume any pins without a net name are floating.

I just spotted this in the datasheet:

When used in Input Mode, the input pins are pulled to VCCIO via internal 200k? resistors.

Nick, do you have any way to verify your 16MHz crystal? Maybe it's slightly off frequency?

Maybe set the fuse for CLKO and carefully probe pin 9 to confirm the system clock?

Output the ASCII table to Serial1 (serial one). Use a serial-to-USB converter to get the output into your Mac. If that works, the problem is very likely the onboard converter. If that does not work, the problem is very likely the 2560.

hiduino:
Nick, do you have any way to verify your 16MHz crystal? Maybe it's slightly off frequency?

I wrote a sketch that toggles PK4, a pin I can get to:

void setup ()
  {
  noInterrupts ();
  DDRK |= 1 << 4;  // PK4
  
  while (true)
    PINK |= 1 << 4;  // toggle PK4
  }  // end of setup

void loop () { }

The generated code takes 7 clock cycles:

 132:   80 91 06 01     lds     r24, 0x0106   (2)
 136:   80 61           ori     r24, 0x10       ; 16   (1)
 138:   80 93 06 01     sts     0x0106, r24   (2)
 13c:   fa cf           rjmp    .-12            ; 0x132 <setup+0xc>   (2)

Thus it would be 28 cycles for 4 iterations of the loop (4 x 7). That should take 28 * 62.5 nS, that is 1.75 uS.

Measured at 100 MHz it seems to be taking exactly that:

I needed to fire up my RS485 converter as well as the serial-to-USB converter but after doing that, yes the ASCII table appears.

So the FTDI chip is not working then?

I won't quite go that far, because I have uploaded sketches on my Windows PC to it, but not from my Mac. So it works up to a point.

Could it be a power problem? Are you certain the Mac USB port is regulating the voltage to a reasonable range?

Try with a powered hub in the middle.

Recent developments have indicated it is a reset issue.

I uploaded the ASCII table sketch to the board, then I put my multimeter on the Reset pin (via its ICSP cable).

After plugging in the USB cable it reads 4.957V which is about right.

Then activate the serial monitor and it drops to 1.649V. And stays there!

So something is wrong with the way it is interfacing with the FT232 chip, regarding the handling of reset.

Activating the serial monitor is supposed to pulse reset, not hold it low.

And this would account for a lot of things, like why I can't upload sketches, and why I sometimes, but not always, get about 60 bytes of output.

To get 60 bytes of output, the FT232 probably unloads its 62-byte buffer at the moment the processor is reset, and then once the processor resets, it doesn't get any more.

I vaguely recall a few posts on the forum about a reset problem with the Mac that involves the operating system / USB configuration. Could that be the culprit? (Don't ask for details. It's 3 AM here, I've been looking at SQL all night, my brain is on its last leg.)

Did you use DTR or RTS?

DTR.

Are you using Serial Monitor? Have you tried a different terminal application? Maybe the problem is with RXTXcomm.

So something is wrong with the way it is interfacing with the FT232 chip, regarding the handling of reset.

That should be electrically impossible. Regardless of what the FTDI chip does, the cap in the reset line should result in a momentary transition.

IIRC, there was at least one "reference design" published where the schematic showed both resistor and cap in the auto-reset circuitry. Could your board have been based on this schematic, and somehow built with the resistor in place as well? (Um... This was the Arduino duemilanove schematic, which had a resistor on RTS as well as a cap on dtr. the schematic did not make it clear that the resistor was "not populated.")

I absolutely agree. I'll consult with my colleagues. There is something strange about the way that DTR interfaces with the Reset signal. I'm not sure of the purpose of the diode here, would that somehow do it?

Yes I'm using the Serial Monitor. No, I haven't tried a different app. However I don't believe that the FT232 chip should be able to hold the entire board in reset, no matter what input it gets.

Nick that low reset voltage you mentioned when the serial monitor could be the reversal of the cap and resistor on the board. The 2560 and the '28 lines all have an internal reset pull-up,safety, the 'added 10 k (from the Atmel app notes) is for a little extra current 'for safety' I think was the quote.. just a crazy random thought, but what happens to the reset voltage when you close the monitor? Btw the diode is to clamp the reset line at Vcc + .6V so the capacitor has a discharge when DTR goes high besides the protection diodes. I read in some thread a reference to another thread and a partial description of the reason for the diode.. It works for me

Bob

Check your reset capacitor for shorts.

The need for the diode was an obscure hardware bug where a 328P chip with certain pull-up loads on certain I/O pin(s) ( later we determined that one 4.7k ohm pull-up wired to certain input pins would be enough to trigger the symptom) would exhibit a problem where after an upload had completed and the DTR was de-asserted via IDE/AVRDUDE the chip would 'hang-up' and not jump to the loaded sketch. A simple manual reset button push would clear the condition and the sketch would then run. After some trouble shooting by some members and using a scope it was determined that the DTR pulse when turned off by IDE/AVRDUDE would generate a >+10vdc spike into the reset pin apparently putting the chip into partial HV programming mode and latching up until a manual reset was performed.

This was finally repeatable and several arduino 328P based boards, not just the Uno rev2 that if was first noticed by a poster that started the troubleshooting effort. This was not tested at the time on the mega1280 based board as best as I recall.

Anyway once we could recreate the bug any time we wanted and shared the symptom and the diode fix with the arduino folks they studied it for awhile and decided on their own to include the diode in the rev3 Uno board. And looking at the schematic it appears they have added the diode to every AVR based board sense then also at least I see it on the mega2560 board and the Leonard board on a quick schematic check.

Lefty