It doesn't work all the way down to 9600. I don't think the baudrate is the problem. I notice in some boards.txt files I have looked at, it has upload method set as STK500 instead of arduino. Would I need to do this since I am not using a real arduino (which controls reset)?
Nope. That doesn't work either. I am out of ideas for now. At least it works with ISP.
Ahha! It appears that my serial converter IC is not working. I wrote a sketch to say hello world as well as blink the LED in a unique pattern and I get nothing out of the serial port. Hmm...
Anyway, that would probably be the problem. I will have to dig through my parts now and find another serial convert IC.
Update: Nope. That was not it either. I actually just had a bad solder connection. It is now printing to the serial port, but still will not upload from the arduino IDE via serial. GRR!! It does however, talk pretty reliably at 115200, so I should be able to get that upload speed once I get this working.
I noticed in the bootloader source code that it mentions a pin to put the board into bootloader. I set that pin for the pushbutton on my olimex board, and of course THAT doesn't work either. GRRR!!!
I'm giving up for now. Somehow I made some kind of progress. I now get
stk500_getsync() not in sync resp=0x48
instead of
stk500_getsync() not in sync resp=0x00
I have to take a break from this, but when I get back to it, I am going to set up some tests to make sure all the hardware is working like it should.
I've compared the initial bootloader uart sequence on the 1284p/16MHz/3V3 (optiboot) and atmega32/11.0592MHz/3V3 (32 bootloader) while in the same breadboard.
Both are identical, except the 32 does NOT answer the request from PC coming ~700ms after /RST (see below).
The 1284p answers immediately to the first incoming RX char(s) with a TX and it uploads fine then.
The /RST is about 500usec long in my case (via 100nF capacitor from a BT module) and acting twice (it does not matter probably 
)..
RxD is the 32's RXpin, TxD is the 32's TXpin.
PS: I've increased the capacitor to 10u and the /RST pulse is now~50ms, but still no answer to RxD.
So the UART does not respond - because it is not set properly in the bootloader:
Replace:
#elif defined __AVR_ATmega8__
/* m8 */
UBRRH = (((F_CPU/BAUD_RATE)/16)-1)>>8; // set baud rate
UBRRL = (((F_CPU/BAUD_RATE)/16)-1);
UCSRB = (1<<RXEN)|(1<<TXEN); // enable Rx & Tx
UCSRC = (1<<URSEL)|(1<<UCSZ1)|(1<<UCSZ0); // config USART; 8N1
#else
/* m16,m32,m169,m8515,m8535 */
UBRRL = (uint8_t)(F_CPU/(BAUD_RATE*16L)-1);
UBRRH = (F_CPU/(BAUD_RATE*16L)-1) >> 8;
UCSRA = 0x00;
UCSRC = 0x06;
UCSRB = _BV(TXEN)|_BV(RXEN);
#endif
with:
#elif defined __AVR_ATmega8__ || defined __AVR_ATmega32__
/* m8, m32 */
UBRRH = (((F_CPU/BAUD_RATE)/16)-1)>>8; // set baud rate
UBRRL = (((F_CPU/BAUD_RATE)/16)-1);
UCSRB = (1<<RXEN)|(1<<TXEN); // enable Rx & Tx
UCSRC = (1<<URSEL)|(1<<UCSZ1)|(1<<UCSZ0); // config USART; 8N1
#else
/* m16,m169,m8515,m8535 */
UBRRL = (uint8_t)(F_CPU/(BAUD_RATE*16L)-1);
UBRRH = (F_CPU/(BAUD_RATE*16L)-1) >> 8;
UCSRA = 0x00;
UCSRC = 0x06;
UCSRB = _BV(TXEN)|_BV(RXEN);
#endif
BTW - I've just copied mighty1284p core and variant 1:1 and using it with atmega32 (IDE 1.5.2). Needs some mods, probably.. 8)
And below the ready bootloaders:
16MHz/115k2 for 5V operation
11.0592MHz/115k2 for 3V3 operation
8MHz/57k6 for 3V3 operation
LED on pin D0 (PB0)
High fuse = CA or DA
Low fuse = FF
Use at your own risk, provided as is..
ATmegaBOOT_atmega32_16M_115k.hex (4.71 KB)
ATmegaBOOT_atmega32_11M_115k.hex (4.71 KB)
ATmegaBOOT_atmega32_8M_57k.hex (4.71 KB)
Pito, you have those bootloaders working with the arduino IDE? I am still getting the error when trying to transfer over serial. It's back to 0x00 response now, though.
You say that you are using the 1284 variant directly? Hmm... might give that a try. It will need some mods, but the 1284 is very similar to the ATmega32, so probably not much. I'll give that a try.
I've tested with atmega32L @11MHz @3V3 @115k2 upload speed, via Bluetooth, with IDE 1.5.2:
C:\DOCUME~1\pito\LOCALS~1\Temp\build5601785643931788526.tmp/_8_Queens.cpp.hex
Binary sketch size: 3,676 bytes (of a 30,720 byte maximum) - 11% used
D:\ProgramFiles\arduino\arduino-152/hardware/tools/avr/bin/avrdude -CD:\ProgramFiles\arduino\arduino-152/hardware/tools/avr/etc/avrdude.conf -q -q -patmega32 -carduino -PCOM13 -b115200 -D -Uflash:w:C:\DOCUME~1\pito\LOCALS~1\Temp\build5601785643931788526.tmp/_8_Queens.cpp.hex:i
loop= 15225652 solution count= 91
+----------------+
|. . . Q . . . . |
|. . . . . . Q . |
|. . . . Q . . . |
|. Q . . . . . . |
|. . . . . Q . . |
|Q . . . . . . . |
|. . Q . . . . . |
|. . . . . . . Q |
+----------------+
You say that you are using the 1284 variant directly?
Yes, the 1284p and 32 (and 644) are pin compatible. So I took mighty's folders - "core/standard" and "variant/standard", renamed both to atmega32, and placed into 1.5.2. as "variants/atmega32" and "core/atmega32" ..
Added to board.txt, for example:
##############################################################
atmega32_11_115.name=ATmega32L 11MHz 115k2
atmega32_11_115.upload.tool=avrdude
atmega32_11_115.upload.protocol=arduino
atmega32_11_115.upload.maximum_size=30720
atmega32_11_115.upload.speed=115200
atmega32_11_115.bootloader.low_fuses=0xFF
atmega32_11_115.bootloader.high_fuses=0xDA
atmega32_11_115.bootloader.extended_fuses=0x00
atmega32_11_115.bootloader.path=atmega8
atmega32_11_115.bootloader.file=ATmegaBOOT.hex
atmega32_11_115.bootloader.unlock_bits=0x3F
atmega32_11_115.bootloader.lock_bits=0x0F
atmega32_11_115.build.mcu=atmega32
atmega32_11_115.build.f_cpu=11059200L
atmega32_11_115.build.core=atmega32
atmega32_11_115.build.variant=atmega32
Of course it may need some mods, as peripherals differ somehow.. 
Well, maybe I need to try out the Arduino 1.5.2 because none of the combinations are working for me. The only difference I see is that you are using 1.5.2
pito:
Yes, the 1284p and 32 (and 644) are pin compatible. So I took mighty's folders - "core/standard" and "variant/standard", renamed both to atmega32, and placed into 1.5.2. as "variants/atmega32" and "core/atmega32" ..
OK. I did the same thing. I copied the "standard" core and renamed it to atmega32, I took the standard variant and copied it and renamed it atmega32. I made the changes in the boards.txt file as you did (there was only a couple changes from mine.) And it is still not working. I am going to try the newer IDE and maybe even a different USB serial cable. If it is working for you, then it has to be one of those two things.
Once I get it working, I will go through the files and make whatever changes are needed.
Do you have the atmega32' reset connected via a capacitor to your dtr on the usb dongle??
And it is still not working.
Does it compile fine for the new 32 board under 1.5.2? Or you have just problem to upload? After burning in the bootloader the LED (D0) shall flash twice each second. After the first successful upload it flashes twice only after a reset (or at the beginning of an upload). So if it does not flash twice at the upload start, you do not reset the atmega properly (reset is done via a 100nF capacitor from the FTDI dtr/dts or something.. - I do not use FTDI, but BT). During the upload LED does not flash.
I am just starting to set up 1.5.2. When I said it wasn't working, that was with 1.0.3. It compiles fine, no problems. It just wouldn't actually upload. I could upload using programmer just fine though. The bootloader is acting as you describe.
I'll report back in a bit after I try out 1.5.2
There is always a certain amount of effort needed to get the stuff working.. 
And tackling arduino's issues is nothing to compare, for example, with setting up a 2k4 MNP5 modem communication in mid eighties
PS: my atmega32L does not upload @16MHz @115k2 @3V3, a lower crystal freq is needed. The 1284p and 328p work fine @16MHz @115k2 @3V3.
No luck with Arduino 1.5.2 either. So it must be something else with my setup. I uploaded a sketch using the programmer to check the bootloader behavior. When I press reset, it just goes right into my sketch. I don't see it doing the bootloader at all. I wonder if that is the problem.
I changed my sketch to not do anything at all with the LEDs just to be sure. Now on reset the LED never even comes on. Its as if I lose the bootloader when I upload a sketch. But even if I erase is and put in only the bootloader, I still can't upload with the IDE
Burning sketch directly in may erase the bootloader. So burn in the bootloader, you will see the LED flash twice each second. The next step is to verify your usb dongle.
Is Tx, Rx connected properly??
Can you verify the usb serial works with a sketch (both rx and tx)??
What voltages do you use??
Can you verify the crystal frequency?? Burning in sketch with ISP may work with any crystal value. Upload is very sensitive as the 16MHz is 3.7% off the recommended frequency for 115k2. You may decrease the upload speed to ie 38k4.
Does your usb dongle RESET the 32 chip when the upload starts (immediate 2 flashes) ?? - that is very important.. (so you need 4 wires to connect to at32 - GND, RX, TX, RES(via ~100nF cap))
Reset must have a 10k pullup. Good decoupling as well.
pito:
Burning sketch directly in may erase the bootloader. So burn in the bootloader, you will see the LED flash twice each second. The next step is to verify your usb dongle.
Is Tx, Rx connected properly??
Can you verify the usb serial works with a sketch (both rx and tx)??
What voltages do you use??
Does your usb dongle RESET the 32 chip when the upload starts (immediate 2 flashes) ?? - that is very important.. (so you need 4 wires to connect to at32 - GND, RX, TX, RES(via ~100nF cap))
TX/RX are connected properly. I tested that earlier with a sketch that outputs serial and allows me to enter serial. I am using 5V on my board. The board has a MAX232 convert chip to a normal serial port. So I have a USB to serial cable attached to it.
No, it does not reset the board for me. I did attempt to manually reset the board upon upload. Didn't help. What would I need to do to get my cable to reset the board (did I miss some tutorial on that)? I do have a USB to TTL serial cable (FTDI) that I could use.
No, it does not reset the board for me.
You will not be happy without an automatic reset, though.. That is why I asked few times..
http://forums.adafruit.com/viewtopic.php?f=25&t=29807
plus megatons of related topics at this forum..
OK, reading...
I just also tried uploading the bootloader through the IDE and it said it failed (though I didn't see any real error in the debug output.) And the LED stopped blinking. Chip was not bricked though. I think it just erased it and then never programmed it.