What happens for me is that echo blocks until ttyACM1 accepts the data.

Well, that is what I expected to happen for me too when I wrote the script. (And on another pc (linux 3.5 vs. 3.0.0)),  I did actually observe echo blocking). The point I wanted to make is that having no circular buffer, one could temporarily block a program on the pc.

But you are right. Closing and reopening the file every time causes the os to flush the file and this makes echo block. If a program would write to ttyACM1 in a normal way, the os would buffer the output... So it looks like we can indeed live without the buffer in the Due.

[It worked well]

Hi Stimmer,

I found kicking out the circular buffer a great demonstration "out of the box" thinking.

I have tested your implementation with ArduinoISP. I think this is a good real life use scenario of the serial port, with lots of messages of different sizes (though relatively small) in both directions. I powered my leonardo from the due's 3v3 and modified ArduinoISP to use SerialUSB instead of Serial. It worked well: I could upload download the bootloader to/from the leonardo several times.

Nevertheless dropping the serial buffer also has consequences for feeding the due lots of small messages. Each message is received in a separate bank and you only have two of them. Take for example this shell script:

Code:

n=0
while true; do

        echo "$n" > /dev/ttyACM1
        echo $?
        sleep 1
        let "n = $n + 1"
done

I ran this sketch on the due:

Code:

static const int led = 13;
static const int button = 3;

void setup() {
  // initialize serial:
  SerialUSB.begin(9600);
  Serial.begin(115200);
  pinMode(led, OUTPUT);     
  pinMode(button, INPUT);
  digitalWrite(button, HIGH);
}

void loop() {
  // if there's any serial available, read it:
  int i = 0;
  if (digitalRead(button)) {
    while (SerialUSB.available() > 0) {
      char c = SerialUSB.read();
      Serial.write(c);
      i++;
    }
  }
  Serial.print("i=");
  Serial.println(i);
  digitalWrite(led, HIGH);
  delay(500);
  digitalWrite(led, LOW);
  delay(500);
}

If you press the button for a few seconds, some of the messages are lost.
Odd enough the return code of echo is always 0.

I am not sure where the bytes get lost. And maybe it can be prevented with stty calls, but a circular buffer would prevent this. A circular buffer which you fill up with memcpy would also make sense as an implementation.

Update: PeterVH, I tried your fix without the guard but I can't get it to work reliably. It works for 100KB or so, then corruption, then nothing.

I can confirm the behavior. I had only tested with an 8KB file. With an 170K file I have corruption too: it goes well for some time but then each  byte gets sent several times.

I don't understand. I also tried with replacing the guard code with:

Code:

irqflags_t flags = cpu_irq_save();
... body of accept ...
cpu_irq_restore(flags);

The idea was instead of avoiding the isr to enter the routine, to avoid an isr alltogether by disabling interrupts. It does not work either.

The good thing is that I can also confirm the version in github works ok (tested with a 170KB file).

I will do tests with ArduinoISP also.

[UDD_Recv8()]

Woohoo!
I just got back from work and noticed the problem is diagnosed, fixed, tested and committed in git-hub!
Thanks guys.

Nevertheless I still feel we should think about the approach taken to avoid entering accept() from the isr. This is what happens without the guards:
- bytes arrive for the 1st time from the host
- the isr fills up the serial rx buffer, it can't free the fifo because it can only write 511 bytes and it has 512 bytes available.
- the sketch performs SerialUSB.read() so it reads 1 byte.
- so we end up in accept() which calls USBD_Recv(). This function automatically frees the fifo if we read the last byte from it! I think this is where the problem lies. This causes a new irq to arrive while we are still in our loop and did not yet update the head ptr of the rx buffer.

Therefore I tried with the function UDD_Recv8(): it does not automatically free the fifo, so we can finish our loop and call udd_ack_fifocon(CDC_RX) when we are done accessing the state in rx buffer.

I could not see the problem yesterday, but now, after seeing your fix, I can. Following code works without guards:
 

Code:

void Serial_::accept(void)
{
        ring_buffer *buffer = &cdc_rx_buffer;
        uint32_t i = (uint32_t)(buffer->head+1) % CDC_SERIAL_BUFFER_SIZE;

        if (!USBD_Available(CDC_RX))
                return;

        // if we should be storing the received character into the location
        // just before the tail (meaning that the head would advance to the
        // current location of the tail), we're about to overflow the buffer
        // and so we don't write the character or advance the head.
        while (i != buffer->tail) {
                // UDD_Recv8() does not automatically free fifo
                uint32_t c = UDD_Recv8(CDC_RX & 0xF);
                buffer->buffer[buffer->head] = c;
                buffer->head = i;

                if (!USBD_Available(CDC_RX)) {
                        udd_ack_fifocon(CDC_RX);
                        break;
                }
                i = (i + 1) % CDC_SERIAL_BUFFER_SIZE;
        }
}

Safer would be to disable the rx interrupt during accept(). The various low level api's like USBD_Recv() already do something like that via the LockEP class. (though I don't like the idea to use a class for this)

Let us further think about this...

It's not amazingly fast (I am getting a 7MByte file through in about 50s) but it is reliable

Indeed, consuming these bytes one by one is really inefficient. But now we have something reliable, we can think about optimization strategies.
B.t.w if we define CDC_SERIAL_BUFFER_SIZE as 513, would this be noticable? (then we have room to accept an entire fifo bank).
 

[It does not yet work correctly]

The idea is to consume only as much data from the usb controller as you can store. As long as you don't have room for new data, the usb controller must be instructed to reply NACK or NYET to the host (it does that autonomously).

On the sam, the usb controller sends these NAKs/NYETs as long you don't "return" the fifo (or rather the memory bank(s) attached to it) to the the contoller. This is done by the  udd_ack_fifocon(CDC_RX) function. There is an excellent diagram in the data sheet at pg. 1092 (40.5.2.13 Management of OUT Endpoints).

I gave it a try. In USBCore.cpp in USB_ISR():

Code:

...
#ifdef CDC_ENABLED
        if (Is_udd_endpoint_interrupt(CDC_RX))
        {
                udd_ack_out_received(CDC_RX);

                // Handle received bytes
                if (USBD_Available(CDC_RX))  // (this test is probably not needed)
                        SerialUSB.accept();

//              udd_ack_fifocon(CDC_RX);  // <-- don't return the fifo to the controller here,
                                                               // do it from accept() if we could store all bytes
                  ...
        }

In CDC.cpp:

Code:

void Serial_::accept(void)
{
        ring_buffer *buffer = &cdc_rx_buffer;
        uint32_t i = (uint32_t)(buffer->head+1) % CDC_SERIAL_BUFFER_SIZE;

        // if we should be storing the received character into the location
        // just before the tail (meaning that the head would advance to the
        // current location of the tail), we're about to overflow the buffer
        // and so we don't write the character or advance the head.
        while (i != buffer->tail) {
                uint32_t c;
                if (!USBD_Available(CDC_RX)) {
                        udd_ack_fifocon(CDC_RX);
                        break;
                }
                c = USBD_Recv(CDC_RX);
                // c = UDD_Recv8(CDC_RX & 0xF);
                buffer->buffer[buffer->head] = c;
                buffer->head = i;

                i = (i + 1) % CDC_SERIAL_BUFFER_SIZE;
        }
}

In read(), after we consumed some data we must call accept again, thi might return the fifo to the controller:

Code:

int Serial_::read(void)
{
        ring_buffer *buffer = &cdc_rx_buffer;

        // if the head isn't ahead of the tail, we don't have any characters
        if (buffer->head == buffer->tail)
        {
                return -1;
        }
        else
        {
                unsigned char c = buffer->buffer[buffer->tail];
                buffer->tail = (unsigned int)(buffer->tail + 1) % CDC_SERIAL_BUFFER_SIZE;
                if (USBD_Available(CDC_RX))
                        accept();
                return c;
        }
}

It does not yet work correctly: it misses every 512th byte. The best test is to use Stimmer's loopback sketch http://arduino.cc/forum/index.php/topic,132811.msg1000171.html#msg1000171
Cat a file of e.g. 8KB to /dev/ttyACM1. Capture the reply also via cat:
cat /dev/ttyACM1 > txt. A diff shows you miss every 512th byte.

I don't think it is some off by one error in the circular buffer because if I define the CDC_SERIAL_BUFFER_SIZE as 64, I still loose each 512th byte.

I think it is somewhere in the code accessing the rx endpoint, but the low level api's are complex and there is some redundancy in them.

To be continued...
I thought I already share this as a starting point, maybe somebody else sees what is wrong...



 

@cmaglie: Will this release have fixes for the problems referred to by selfonlypath?

I don't find issues on github for these problems. From the "Arduino Due - Serial speed?" thread I infer two problems:

1. Data loss when sending serial data from a pc to the native usb port.

http://arduino.cc/forum/index.php/topic,132811.msg1000171.html#msg1000171
quoting stimmer:

Using any terminal program that can send a reasonably large file (I used gtkterm), send something to the Due at 115200. When connected to the programming port, it echos back fine. On the native port, only short sections at the beginning and end get echoed.

This is because the core (class _Serial) does not make use of usb's built in handshaking mechanism. See also here:
http://arduino.cc/forum/index.php/topic,143871.msg1080544.html#msg1080544

I may try to work out a solution but if a fix is already on its way, I'll await that of coarse.


2. Upload of sketches much (10x) slower than on mega:

http://arduino.cc/forum/index.php/topic,132811.msg1032116.html#msg1032116

In what direction did you have to change the code to make it work?
In 1.0.3, it Should be if (USB_Available(CDC_RX)) or you have an infinite loop. I checked github, the change was comitted. I also just extracted arduino-1.0.3-linux32.tgz again and USBCore.cpp also contains  if (USB_Available(CDC_RX)).

[CDC_]

I noticed that when sending 64 or more bytes from a pc to the Due's native USB port, bytes get lost.
This sketch reproduces it:

Code:

static const int led = 13;

void setup() {
  // initialize serial:
  SerialUSB.begin(9600);
  pinMode(led, OUTPUT);     
}

void loop() {
  // if there's any serial available, read it:
  int i = 0;
  while (SerialUSB.available() > 0) {
    SerialUSB.read();
    i++;
  }
  SerialUSB.println(i);
  digitalWrite(led, HIGH);
  delay(250);
  digitalWrite(led, LOW);
  delay(250);
}

If you send say 80 bytes to the serial port (e.g. simply via an echo to /dev/ttyACM1), the sketch prints series of numbers like this : 0 0 0 63 0 0, which indicates that (at least) 17 bytes are lost.

It can easily be fixed in CDC.cpp: the code defines a ring buffer of size CDC_SERIAL_BUFFER_SIZE (==512). But the code that uses it uses the SERIAL_BUFFER_SIZE macro which is 64. So replacing SERIAL_BUFFER_SIZE with CDC_SERIAL_BUFFER_SIZE in CDC.cpp fixed the problem for me (I wanted to run ArduinoISP which requires receiving 128 byte messages from avrdude).

It will go wrong if you feed the due more than 511 bytes at once. Therefore it would be better to use USB's built in handshaking mechanism (make the device reply with NAK or NYET until it has room to store more incoming data). The same was done for the Leonardo. I could look into how to do that with the due's otg controller, but if someone already knows, that would save time...

I think the vb program does not set the (virtual) DTR signal for the serial port.
Try setting the port's DtrEnable property to True.

...but if I run the ASCIItable sketch I do not see anything in the serial monitor.

I can repeat the behavior on a leonardo with windows 7 SP1, with linux it works correctly.

I know what happens. After you download the sketch, the IDE touches the serial port to remove the magic baudrate. Touching means it briefly opens the sketch's serial port. As a result, the "while (!Serial);" loop exits and the sketch wants to start sending characters. But the port is closed again briefly after this and your characters get lost.

As a work around you can reset the board with the reset button, after this the while (!Serial); loop has the desired effect and chars end up in the serial monitor.

Another work around (though kludgy) is this:

Code:

while (!Serial);  // <-- detects the touch
delay(200);
while (!Serial);  // <-- waits for the real port opening

Btw. It was me who proposed to touch the port after uploading, maybe it should be rolled back.

I just tried out serial comms between processing and the leonardo, myself. On the pc I did not need to do anything special to set DTR/RTS, so apparently processing (or the underlying rxtx) does this for you.

I have a better test sketch (for the leo/micro):

Code:

static const int led = 13;
unsigned char x;

void setup() {
  Serial.begin(9600); // Start serial communication at 9600 bps
  pinMode(led, OUTPUT);
  digitalWrite(led, Serial ? HIGH : LOW);
}

void loop() {
  Serial.write(x & 1);
  digitalWrite(led, Serial ? HIGH : LOW); 
  delay(1000); 
  x++;
}

It alternately sends out 0 and 1. Also, via the arduino led, it lets you observe DTR/RTS: Serial (or rather Serial's boolean operator) returns true when either DTR or RTS are high.

Run this sketch on the leo.
As long as the PC does not open the comm port, the leo's (or micro's) arduino led and the tx led remain dark. The leo runs its loop but all written bytes are dropped by the core.

Then, on the PC, fire up the standard processing example Libraries>serial>SimpleRead.
The arduino led lights up and the tx led flashes.
(and SimpleRead alternately shows a gray or black rectangle).

Note the behavior is different if you run the sketch on an Uno or Due (programming port). For these boards the tx led keeps flashing even if SimpleRead does not run (port not opened by the pc). (though I find that the leo's behavior is ok in this situation).

I tried this on linux 3.0.0-12 (kubuntu) and on windows 7 SP1. On windows I had to dowload the 32 bit version of processing because Serial does not run in 64 bit mode.

Could try whether this basic setup works for you?

I am looking at the pin diagram right now, I do not see DTR or RTS pins... I see RST on the board, but cannot make it which pin it actually is.

You are using the leonardo's virtual com port, so DTR and RTS are no real pins. The pc can however send control messages to "set" or "clear" these "virtual" DTR and RTS signals. And the leonardo's core will only send out data if DTR or RTS are set by the pc.

I am trying to get Analog input into Processing.

So you're processing app should set DTR/RTS. I don't know whether the Processing framework does this for you or how you can do this yourself.

Maybe it is best that you first try to run a simple app that sends out some  serial data and see whether you can receive the bytes from the Arduino-1.0.3 serial monitor. Maybe use wmassano's sketch:http://arduino.cc/forum/index.php/topic,137032.0.html

The leonardo only sends out data if the program/os on the pc sets either DTR or RTS, (or both) high. If you use a program like putty or the serial monitor in the IDE to communicate with the leonardo, there is no problem because these programs set DTR and/or RTS. If you do serial communications from your own program you must make sure yourself to set DTR/RTS.

I have been trying for 4 days now with no lunck.

Can you tell us more about what you try to accomplish? And on which os, version, sp...?
You may observe the same problem as what is described in this thread but it might be something completely different as well.

[//#define HID_ENABLED]

That is great information, Paul. In my opinion, it should be added to the documentation (the minimum SP for windows, the minimum os version for mac). A classic arduino probably works on almost every windows version, for the leonardo, this appears not to be the case.

So the right thing to do for me would be to get SP3. But I should have done that a couple of years ago, now it may not be worth the effort and the risk that this old pc does not survive the upgrade. On the other hand that pc still works well and hapilly deals with my duemilanove, pic18F4550 (also cdc/acm) and the leo's bootloader.

So I thought my Leonardo sketches should work as well if I ran them as "serial only" usb devices (non composite, so no iad needed).

The code allows to do this with only a few ifdefs:

in USBDesc.h:
#define CDC_ENABLED
//#define HID_ENABLED
...
#if defined HID_ENABLED && defined CDC_ENABLED
#define COMPOSITE_DEVICE
#endif

in USBCore.h:

#ifdef COMPOSITE_DEVICE
        IADDescriptor                           iad;    // Only needed on compound device
#endif

in CDC.cpp:

const CDCDescriptor _cdcInterface =
{
#ifdef COMPOSITE_DEVICE
        D_IAD(0,2,CDC_COMMUNICATION_INTERFACE_CLASS,CDC_ABSTRACT_CONTROL_MODEL,1),
#endif
...

I compiled the sketch again and made a .inf file by copying the UNO's in file and adjusting the PID. This made the sketch run on the old xp sp2 box. I also managed to burn hex files via ArduinoISP into an attiny on the same pc.

I hope this does not sound like cursing to the developers who have put lots of effort to make composite devices work. But I believe this idea makes sense:

• it is simpler.
• because of this, it does work on all these old os'es without ugly hacks or spec violations.
• in many occasions you only need either one of the interfaces, not both.
• you win about 1K code size (dropping hid support).

Thanks a lot for sharing this. I started to investigate the problem but could not make sense of it. Windows installed the driver without error but the resulting behavior was really weird.

@wmassano: on your win 7 it does not talk to the serial monitor, but does it blink?