SOLVED: Any harm in sending bytes to serial buffer never accessed?

Projects Programming
1

PS:This paragraph is a note added here by me AFTER several hours of replies coming in. The replies you can read below. Thanks to those replies, my question has been answered. I have gone through my posts, editing them in hopes of making the thread more useful for people who peruse it later. I've also tried to answer some of the questions asked in the past ten hours while this has been active. Answer the questions I couldn't answer properly when I first saw them. Anything [thus] in what follows was added after the discussion was completed.

=======================
A "general tools" question...

I have in mind writing some code that would from time to time send characters to the standard serial port, i.e the place where users could easily see them via the IDE's Serial Monitor.

In all of this, I am hoping for a very general answer, just using the serial port easily monitored by the Serial Monitor. "Just" using the pins which are also connected to the electronics which the Arduino development environment uses for programming the Arduino. I.e. using the pins both for the programming and for my purposes at (almost) the same time.

And when I said "general answer", I was also hoping for something that "should" apply to "most" Arduinos... not need qualifying with things like "But on a Mega, you have to..."

===
In my application, the user wouldn't ordinarily wish to take the trouble to look at the data sent. My poor app would "cry in the wilderness".

If I were doing this, and nothing was taking things OUT of the buffer, wouldn't it eventually become full? [The answer to that would be "no". And thus my concern was misplaced.]

[I was foolishly thinking that the port has "handshaking". It doesn't!... and THAT answers my question. Handshake - Wikipedia ]

Once it was full, I am hoping that if new characters were pushed into the buffer, they would merely push the oldest previously sent ones out of the other end, to disappear forever.

Does anyone anticipate any problems for my app if I were to send things to the buffer that passes them "out" via the serial port? Send them even when nothing was connected to "collect" the output?
[The answer to that was "no". The premise (presence of handshaking) that made me fear there might be a problem was flawed.]

===
Why am I asking? Here's a specific instance...

Suppose the app was a small webserver offering a page with the values seen on sensors attached to the Arduino.

Ordinarily, it would sit quietly reading the sensors, updating the webpage it serves up, and responding to requests for the page. (As done by my "Arduservers" (Arduserver.com, or sheepdogguides.com/ardsrv if you want https))

The system would, however, also send a string to the serial port from time to time. A report on errors encountered? Out-of-expected-range values on sensors? These strings might be of interest to the person administering the server if for some reason the users suspected it wasn't running properly. But ordinarily, in my scenario, the information would never be accessed. Often, there wouldn't even be any hardware connected to the output for the serial buffer.

[Such strings might also be used to "display" things like the name and version of the sketch held in the Arduino, or a guide to what external hardware is needed for it to do what it was created to do. As with the previous example, ordinarily, no one would need to look at that information... but it would be available when wanted.

===
"Of course there wouldn't be any problems" might be your reaction.

I'd be delighted if it were! A few such from experienced people would be welcome.

But maybe there's some problem like, (I don't KNOW... that's why I'm asking!)... like the buffer refusing to accept NEW characters when full? [It won't "get full", if you follow some easy precautions of which I will speak.] Could something in my scenario which could create a blocking event? [Probably not.]

===
And finally, just to save helpful people making sure...

Yes! I do understand! If, say, the buffer is 256 bytes wide, and I read from it for the first time after 356 bytes were sent to it, there will be 100 bytes that have been lost forever.

===
I look forward to your thoughts...

Tom

2

Data is transferred from that buffer at the specified baud rate, regardless of any listener.

So you will only encounter some delay, if you generate more chars.

May be different on Arduinos with native USB interface.

1 Like
3

Hello Tom,

When you write to the serial buffer code hidden from you shovels it out of the serial port in the background. Whether anything receives the data makes no difference, the serial port doesn't know or care. The thing that might cause a problem is that if you write to the buffer faster than it can empty, eventually filling the buffer, then Serial.print and write become blocking, which might slow your other code.

1 Like
4

Ah... Perry said "{in which case} Serial.print and write become blocking."

[THAT was what I was afraid might be an issue. When Perry spoke of the data being shoveled out in the background, I was ASSuming that the output would be happening with handshaking involved. THAT was where I was wrong.]

[When Whandall said earlier "Data is transferred from that buffer at the specified baud rate, regardless of any listener", I didn't consider that it might be the "transferred" that happened regardless... I thought he was warning me that the baud rate would be what the Arduino was set to, regardless of how the receiver might be set,]

= = =
I'm pretty sure I know how to check how many characters are in the buffer. [I do... but "characters in the buffer" is not going to be an issue. I will be explaining why not.]

Is there a way to take things out of it, from the "oldest" end, other than actually reading from the port via some "outside" agency/device? [No. And anyway, they leave "all by themselves, whether something is watching the port and reading what emerges, or not. The "reading" isn't an action controlled by the outside device. It is ALL under the control of the Arduino, and happens whether something is "listening" or not. (You may say "of course!" However, this is NOT always the case, in the wider world. At the level of this discussion, though, it is what always happens in Arduinos.)]

===
Maybe I'll end up doing a "buffer for the buffer" in software, and periodically flushing the buffer, and refilling it with stuff- some new, some recent-but-not-very-old? Sigh. Not elegant! And "not elegant" is often the path to misery. [This was way off of the mark.]

5

But that's independent whether a connected device ( serial monitor ) is actually reading or not. You should never write to the buffer at a higher rate, than the Serial ISR can read ( depending of the baudrate ).

Things are different on Arduinos with native USB interface. Whether you like it or not. Do you use such an Arduino?

1 Like
6

To be clear and to repeat what @MicroBahner said: the buffer filling up is independent of and has nothing to do with whether there is or is not anything connected to or taking any notice of the data coming out of serial Tx. There is nothing in the electronics of the micro-controller that knows or cares what is or is not connected to the port. If the buffer fills up it is because you are writing to it faster than the data can be sent out.

If a tree falls in a forest and no one is around to hear it, does it make a sound?

1 Like
7

the standard size of the buffer is 64 bytes.
Each byte has 10 bits this means at a baudrate of 115200

sending 64 bytes needs 64 * 10 / 115200 = 0,0056 seconds = 5,6 milliseconds
Even if you send 256 bytes this will be finished within 25 milliseconds.

So if you send status-messages once every few seconds this will not slow down code-execution.

What I have not researched for is if the standard TX-buffer has different sizes on different boards.

best regards Stefan

1 Like
8

[Hanging my head in shame! I didn't pay sufficient attention to Perry's "There is nothing in the electronics of the micro-controller that knows or cares what is or is not connected to the port. " That should have made me understand that my thought that there was handshaking was just wrong!

I think I understand some of what you all... thank you!... are telling me.

I'm not sure I've succeeded in conveying what I am wondering about.

I understand that if I CARE about the "outside party" (the entity reading from the Arduino via the serial port) SEEING everything that goes into the buffer, then I must not write to it too quickly. [I'll come to the question of why you mustn't send things too quickly later.]

But I don't. I don't care.

I am quite happy for some of what I send to the buffer to fall off of the edge of the table, be "lost". [EVERYTHING sent to the buffer will "be lost", and very quickly. It passes out of every part if the Arduino almost as soon as it is sent to the buffer. THIS buffer's job is NOT to hold things until something "outside" says "Okay, I'm ready. Send it, if you have something for me." (Such buffers are not unknown, by the way. But not in Arduinos. Not in serial comms as built into the devices. (I'll explain the Arduino buffer's role in a moment.)]

Suppose a buffer of 64 bytes...

If I try to send 100 bytes to it, and NOTHING is taking stuff OUT of the buffer, will the system choke or become sluggish when I try to send the 65th byte? [No... not normally. What "abnormal" would look like is part of the promised "in a moment".]

I'm hoping the oldest character will merely be pushed out of its end of the buffer, by all the other bytes "moving up" to make room for the newcomer? [Moot.]

Apologies to those who already understood that it was THAT which I was asking and feel they've already answered. It's been a while since I a here, and when I was active previously, I was trying to answer as often as I was asking... and I know it can be a frustrating business!

Yes! I could write a small program to test this empirically, but that would only give me the answer for one system.

9

We tried to tell you, that there is ALLWAYS a ISR running that takes the char's out of the buffer. That's independent, wether something is connected to USB. At least as long as we are talking about a standard serial line ( Mega, Nano, UNO ...) and NOT a native USB.

1 Like
11

When you begin Serial, set the baudrate to at least 115200. Input and output buffers get allocated. If you print or write a char, it will be fed to the serial port register and transmitted.

Don't screw with that, in General. Sure you can read/write buffer bytes without telling Serial but when you want to use Serial again, it uses circular buffers (pointers) and can be crashed.

I am quite happy for some of what I send to the buffer to fall off of the edge of the table, be "lost".

Do you care how LONG that takes in lost cpu cycles?

Suppose a buffer of 64 bytes...

If I try to send 100 bytes to it, and NOTHING is taking stuff OUT of the buffer, will the system choke or become sluggish when I try to send the 65th byte?

I'm hoping the oldest character will merely be pushed out of its end of the buffer, by all the other bytes "moving up" to make room for the newcomer?

115200 baud (start bit, 8 data bits, end bit) is 11520 chars/sec is 1388.8 cycles/char to transmit or receive.

If You will print a lot, time when you print to not overfill.
Maybe hang an ATmega168 on SPI, buffer and print through that?

1 Like
12

People reading this thread in the future can just skip over this post]

Thank you, MicroBahner...

Maybe I'm misusing the term "buffer"? Or maybe there are several, and we're each thinking of a different buffer?

I understand that "sent" characters are "gone" from the immediate reach of the code I've written... but surely(?... see what follows) they are "in" the Arduino somewhere? In the Arduino, waiting until an outside agent says "please give me anything that's waiting to go out over the serial line?" At which time they leave the inner parts of the Arduino system, first-in-/first out, and having been read, are now gone, and can't be read from the Arduino again.

Apologies if that's confused. But if it isn't, does what I'm wondering about become more clear?

===
Maybe I overestimate the Arduino's serial port (and point taken that the implementation may be different in different Arduinos.) I'm hoping that there is hand-shaking. RTS/ CTS, and all that.

===
Maybe following everyone's help, I can now answer my own question?....

Answer...? > > >

The buffer is ONLY taking care of baud rate timing issues, and something akin to parallel to serial conversion.

A badly written program might try to send 100 bytes to the 64 byte buffer in less time than it takes for enough of them to "dribble" out the other end. (Code could send bytes TO buffer blindingly fast, relative to the bit rate of a serial data stream, even if the latter, to human sensibilities, seems "fast".)

No handshake.

If the "outside agent" isn't "listening" at the time the code sends something to the serial port, Too Bad! The line was toggled appropriately, as soon as possible upon the code sending the byte, and if no one was "listening" THEN, the byte is gone.

13

If you want to cover all Arduinos, use Serial.avaibleForWrite() to determine if there is sufficient space in the serial output buffer. If not, don't send data. That way you can prevent blocking. This is specifically important with boatds with native USB support that can come to a grinding halt when serial monitor was open and is closed; or if your board works stand alone, a computer gets connected for monitoring and later is disconnected again.

1 Like
14

In my experience it is almost always true that when a sentence contains the word 'surely' then whatever follows is wrong. There is no* handshaking or other means by which bytes wait in the serial port for some external device to ask for them, as soon as they appear in the buffer hidden interrupt driven code shoves them out of the serial port whether or not there is anything there to receive them.

*Not in the scope of your question but it used to be the case that there was some kind of handshake for exactly this purpose but I have not seen it implemented for maybe 30 year or more. It is not implemented in Arduino so far as I am aware.

1 Like
15

Not relevant to Arduino, but hardware and software handshaking protocols still exist for legions of small instruments, serial printers (many POS ticket printers, for example), PLCs, etc. etc. etc. Never underestimate the ability of legacy hardware your project "must use" to mess with your mind with arcane implementations predating Noah.

But in the context of this thread, the OP should not expect any such protocol to be delaying the characters shoved into the buffer dribbling out the serial port regardless of connection status. Set the baud rate high to minimize latency, and move on.

Question for the OP - Why are you sending these characters, if you have no expectation anyone will ever listen? Or is it debugging info which might inform you when trying to get to the bottom of a problem? Or data for a future expansion, for which you are creating functionality you don't expect will ever be needed? In the debug case, I'd devote a pin to a "debugging" flag and only send characters to serial if that pin says to do so.
In the expansion case, perhaps use a #define to add that code "in the future".
C

1 Like
16

There is no 'outside agent'. Only the UART HW of the processor, which will trigger an interrupt when it is free to send the next byte. And there is no handshake on the serial line. The bytes are sent regardless of whether something is connected to receive them.

1 Like
17

This is my final post, written after the patient people who kept at this for me until I woke up, had finally said the things that got me to where I am now: Knowing that what I was worried about Won't Happen. Thank you all again.

(First, in answer to MicroBahner's last post: What I was speaking of as an "outside agent" was just something... a computer running the Arduino IDE, but just displaying data encountered by the Serial Monitor perhaps?... "watching" the output of the serial port, but not part of the Arduino running my sketch.)

= = = = = =
In the course of the conversation, mention was made several times that one way to go wrong when using the serial port is to write to it "too quickly".

Here's an attempt to explain what constitutes "writing too quickly"...

The numbers inside the computer are, of course very different, but the principles in what follows are sound.

And I'm using pseudo-code in my description. Don't imagine that the following would be a viable sketch.

Suppose... bear with me!... that you wanted to send five "X"s out over the serial port, and then pause for 9 seconds. And further suppose that you didn't want to use...

Repeat
Send "XXXXX"
Wait nine seconds
Until false (i.e. do it until the device is unplugged.)

... but wanted to use...

Repeat
Send "X"
Send "X"
Send "X"
Send "X"
Send "X"
Wait nine seconds
Until false (i.e. do it until the device is unplugged.)

Now.. and here is where the numbers are all wrong, scaled up to time intervals that a human can be comfortable with...

Suppose that it takes 0.1 second for "Send "X"" to be finished, as far as your sketch is concerned.

You have to know that the X passes first to the buffer, which was spoken of so much above, and from THERE, with no further trouble to the programmer, it is passed on, as a series of "ons" and "offs", to the output pin of the serial port.

(The buffer will be empty, will have nothing in it, at the start of all this.)

I said "suppose it takes 0.1 second". That, in my explanation, is the time for the character to go INTO the buffer.

Compared to the time for it to go IN, the time for the background processes to pass it OUT, across the pin, as a sequence of "ons and offs", is "long".

Continuing the hypothetical, let's say that it takes 0.8 second for the "X" to pass OUT from the buffer.

===
See the problem? The X's are coming IN faster than they are passing OUT.

But! That's what the buffer is for! The overall system is quite capable of letting your sketch "stuff" 5 X's into the buffer in only 0.1 second for each "stuff". So it would take 0.5 seconds, by the hypothetical numbers, to get to the "Wait nine seconds" instruction.

And, measuring from when the first X arrived, it would be 4 seconds for the system to FINISH passing the 5 X's out.

(That's what my "wait 9 seconds" was about. The "sending out" carries on quite happily even as the sketch carries on to the first thing after the last "send an X to the buffer". But the time we get back to the top of the loop, the first five X's have all been completely sent out over the serial port. The buffer is again empty, and ready to repeat the exercise.

If we tried to send many more Xs before the "Wait", there would be a longer queue of X's waiting to be sent before the sketch reached the "Wait". And the space for X's to queue is not infinite. How could it be?

If we reduced the time of the "wait" below 0.8 second, the same problem would arise because the buffer would be filling faster than it is being emptied.

The good news!...

In real world circumstances, with the real numbers for how these things work, you are very, very unlikely to encounter the problem of the buffer being full any time you try to put something new in it. Unless you deliberately write your program in a way that pushes a LOT of characters towards the serial port in a very short interval. (See things people said along the way to addressing my question, if you want more help with this "not too much too fast" concept.

18

Serial data, then there is any to send, is sent at a more or less constant rate determined by the BAUD rate. The number of characters that can be transmitted per second is roughly the BAUD rate divided by 10. So, at 9600 BAUD, you can send ~960 characters per second. At 115200 BAUD, you can send ~11,000 characters per second. The serial buffer is typically 64 characters. So, even with an empty buffer, if you try to send a 100 character message in a single print statement, the print will block until the first ~36 characters have actually BEEN TRANSMITTED, making enough room to get the rest of the message into the buffer. Any other time you try to print a message that is longer than the number of empty slots in the buffer, the print will block until the last character of the message ha been put into the buffer.

1 Like
19

You have beliefs about Arduino that you base ideas from... where do you get them?

Arduinos have no operating system.

Wire up a 16550 for handshake and a 16 byte buffer.

1 Like
20

115200 baud means 11520 char/sec, 64 chars print in less than 6ms which to the AVR cpu is a long time, 96000 cycles on a RISC chip.

I do process serial bytes even at higher rates, AVRs are more than fast enough.

21

If they can now make any sense out of the replies.

1 Like