how many characters can be "pasted" into the serial buffer input window?
not the serial buffer size on the Arduino
appears to be around 1051 chars
Do you mean the IDE serial monitor? I think you need to specify first if we're talking about 1.* or 2.0 IDE.
I imagine it's a Java textbox so there's no specific limitations except the ones the author might have defined. On a 1.8 IDE I tried 10k characters string and they have been correctly received even when setting 9600 baud (I did that just to stress the buffers) with this pretty straightforward code:
void setup() {
Serial.begin(9600);
}
void loop() {
if (Serial.available())
Serial.write(Serial.read());
}
Why are you asking this?
line 126
line 127
line 128
ine 129
i generated a file with "line1;line2;..." and see problems after line 128 around 1051 chars. in tests with more data, 500 lines, i see repeated text
isn't 9600 bps less stressful than higher rates? lower rates give the code more time to process the data.
i'd like to be able to drop a configuration file into the serial monitor that is used to populate internal data structures that are then written to eeprom read at startup. i'd prefer a single cut&paste
char buf [80];
// -----------------------------------------------------------------------------
void loop()
{
if (Serial.available ()) {
int n = Serial.readBytesUntil (';', buf, sizeof(buf)-1);
buf [n] = '\0';
Serial.println (buf);
}
}
void setup()
{
Serial.begin(9600);
}
How long are your input lines?
A potential problem with your code might be Serial buffer overrun on the Arduino side?
Consider the following. On an Uno, the Arduino serial buffer will be 64 bytes (if memory serves - but that isn't so important as the concept of my point).
So lets say you pass your Arduino a string that is longer than that - for example a string of 200 bytes.
This will be read fine, returned to your buffer which you terminate and then print back to Serial.
First time through, the transmit buffer will be empty, so the first 64 characters of your 200 character string will be sent stored in the transmit buffer, but at this point the print will block - because there will still be 136 characters that couldn't fit into the transmit buffer.
But, while the print is executing, new data will be being received. This is still fine because as characters are being received they will be written to the 64 byte input buffer.
But assuming all things equal, by the time the Arduino will have received 64 new input characters (i.e. filled the input buffer), there will still be about 72 characters waiting to be placed into the output buffer from the first line of input (64 characters will have been sent, another 64 will be in the output buffer waiting to be sent and the print statement will still have 200-64-64 = 72 characters it is still trying to schedule for output). As such, there will be some sort of data loss on the incoming data stream.
Now, you are not reporting data loss in the beginning, and the above is a "contrived example" to illustrate the potential issue, but if you have lots of lines of varying data sizes and some rather long ones then you may enter that situation over time - especially if for some reason there is a delay in processing the data being received at your PC end.
And I do see that you indicated that you are sending data of the form "line1;line 2;" etc.
I don't know for sure if that is what is happening, but it is a possibility to consider.
In situations like this - where I send my Arduino lots of data, I will use some sort of handshaking that lets me know that it has received a record of data (in your case a line of data) before I try to send the next record.
That way, I am less likely to "overrun" the receiving code with a storm of data that is pretty much just "shoved down your Arduino's throat" which is sort of what is happening if you just paste a huge block of text into the serial monitor's input.
Another approach if you just want to check the robustness of the connection is to maybe not echo back every single record that you received, maybe just try printing every 100th record and only if that record is less than say 60 bytes in length (so you don't get into the blocking scenario I outlined above).
Anyway, I hope that maybe gives you something else to maybe consider and try.
Ok, understood. I now report you my tests, just to try and see what happens and why.
I generated a 1000 "lines" on a single row ("line1;line2;...line1000;", total 7893 bytes) and tested on a Wemos D1 Mini with this code (where I got rid of the readBytesUntil):
char buf[80];
byte n = 0;
void setup() {
Serial.begin(115200);
Serial.println();
delay(2000);
Serial.println("READY!");
Serial.println();
}
void loop() {
while (Serial.available ()) {
char c = Serial.read();
if (c != ';') {
buf[n++] = c;
} else {
buf[n] = '\0';
Serial.println(buf);
n = 0;
}
}
}
...and you're (partially) right.
In my case the error comes out always after between line 370 and 380, with the first "l" character missing:
line378
ine379
line380
ine381
line382
after line 510 things become weirder:
line514
ine515
line516ine517
line518
ine519
line520ine521
line522
but on the last 30 lines the output is strangely back to normal:
line966
ine967
line9ine969
line970
line971
line972
...up to...
line998
line999
line1000
Slowing down to 9600 baud the data transfer requires a lot more time, but the issue is exactly identical, so I can say it's not a matter of serial speed!
But with this simpler code:
void setup() {
Serial.begin(115200);
Serial.println();
delay(2000);
Serial.println("READY!");
Serial.println();
}
void loop() {
while (Serial.available ()) {
char c = Serial.read();
if (c == ';') c = '\n';
Serial.write(c);
}
}
the output looks absoluterly perfect!
The only difference is the use of Serial.write() byte per byte, instead of a Serial.println() of the whole buffer string, so I can't actually understand why.
I also made a test using "Putty" serial terminal emulator, and the first sketch posted here it gave me the exact same problem, so I'm pretty sure it has nothing to do with IDE Serial Monitor!
I can just make a guess: this weird behaviour could be somehow caused by the USB-serial device driver (mine is the CH341 "compatible" one, what is yours?) and/or it could have something to do with concurrent serial read and write.
So I tried removing the "Serial.println(buf);" statement from the loop() function and added a consistency check:
int r = 0;
void loop() {
while (Serial.available ()) {
char c = Serial.read();
if (c != ';') {
buf[n++] = c;
} else {
buf[n] = '\0';
n = 0;
++r;
//Serial.println(buf);
if (buf[0] != 'l') {
Serial.print("line"); Serial.print(r);
Serial.println(" IS BAD!");
}
}
}
}
This doesn't report any error! This means to me the most probable issue is when you read and print in the same loop. Don't know why, but it seems to me this is the direction to investigate, but I'm not experienced enough with low level hardware and/or programming to tell you what to do..
Said that, and just for testing at the moment, I think you could avoid printing back to serial the received line and keep writing it inside the EEPROM, and see if it works.
i used the code below to verify the reception of the correct # of chars, up to 900 line, 7999 chars. so it appears the IDE has a reasonable limit, if any and any issues are with the receiving code.
of course the receiving code needs to process faster than the time needed to transmit. one possibility is to simply buffer all the data (~1kchar) before processing
here's an example of the types of command to be processed
CS _ssid NETGEAR33;_pass wittymint609;_host Garrett;_ipAdd 192.168.1.12;_ipAdd 192.168.0.15;U
the single char upper case commands: C, S, U are to clear, dump and update the EEPROM. the multi-char commands preceded with an '_' set string parameters. mulit-char commands terminated with a ';'
unsigned long cnt;
// -----------------------------------------------------------------------------
void loop()
{
if (Serial.available ()) {
char c = Serial.read ();
if ('\n' == c) {
Serial.println (cnt);
cnt = 0;
}
else
cnt++;
}
}
void setup()
{
Serial.begin(9600);
}
Maybe slow down the sender..
AckMe
Should handle your data and maybe it helps..
good luck.. ~q
You mean the configuration file you wanna store on EEPROM is only 1 k bytes long?
In this case, I don't see such a big problem, you don't need to write back anything to serial (my tests said a pure data receiver works fine, we're talking of a 16 MHz processor anyway...) , specially if you use a separate sketch just to store the config data, then load the "real" sketch that makes use of that.
Does that solve your problem?
the idea is to be able to change the configuration of a "node" without changing the code by updating parameters, updating the EEPROM with those parameters and resetting a node.
i'm trying to understand the limits of what can be done simply. i have the above working with relatively small # of parameters and believe i can easily triple the # of commands, but am wondering how far that can go.
processing commands that take a long time can impact reception of serial data, but the longest operation i have in mind is updating the EEPROM and that would be the last command.
capturing a complete set of commands and processing them later is a approach that i believe minimizes processing time. of course slowing the bit-rate allows more processing time of commands.
the configuration file is a set of commands. it needs to be processed at run time. it is not stored in EEPROM, the parameters it updates need to be stored
Don't overlook the fact that println() is adding two characters, and that the data being sent from the computer to the arduino is a continuous unbroken stream of data being sent as fast as the serial line can handle. At some point you will be overrunning the receive buffer while waiting for the transmit buffer.
You might try using a Mega or other Arduino with sufficient memory that the entire data can be read, stored in memory, then sent back after the transmission stops.
my requirement is to not exceed the ability of what i currently have, hence why the question is about the IDE and not the Arduino.
i've worked on data terminals and understand data communication.
The suggestion to use a Mega was more for testing to see if the data was capable of being received correctly, since an UNO might not have sufficient memory to store test data of sufficient size.
Ok, so to do that I think you just need to put the node in a specific "update" state (e.g. doing nothing else, if possible) until the received data ends, or, better, when it receives an "end of data" special command at the end of the string.
I did a similar thing with my photovoltaic inverter monitor, I used UDP packets to send commands and receive data and replies but the concept is the same, like a command "@C" prefix ("Configuration") followed by a sequence of parameters (like the minimum production power and inverter internal temperature to start the fans on, or put the fan in "manual" state, and so on), to be stored in EEPROM. If your project already uses WiFi or Ethernet connection, I suggest you to switch to UDP or TCP/IP instead of plain serial.
I don't know your programming skills, but alternatively, if you know desktop applications programming (e.g. Windows console or Forms) I'd suggest you to create a pretty simple program that reads a specific text file and sends its content one command/row at a time, with a certain interval in between.
I know that, in fact I even tried to raise serial baud rate to check if sending data quicker could get worse. But based on my tests (see above) it seems the results haven't changed between 9660 and 115200 bps.
The goal is not to send them back, but get the configuration commands and execute them or store on EEPROM some parameters. Like the OP said, the command strings can be around 1k bytes long, so it isn't a problem even with a UNO (my tests ran on a Wemos D1 mini, an ESP8266 with a lot more RAM than UNO). IMHO I don't like this approach much, anyway.. 
Changing the baud rate would not affect the problem of the added carriage return and newline, it should occur at the same place because it is the ratio of characters sent to characters being received that determines when a buffer overrun would occur.
thanks, but i have this working for the above command. it can easily be extended. again, it's just a question of how much.
i have a command processor that normally processes commands entered manually thru the serial monitor. i'm just starting to package multiple configuration commands in a file to avoid entering multiple commands by hand
Yep, understood. But this is why I suggested you this (it's what I'd do if I had the same constraints you have):
I'd suggest you to create a pretty simple program that reads a specific text file and sends its content one command/row at a time, with a certain interval in between.
Have you tested to see how long is needed to actually write the date to EEPROM? EEPROM writes can take on the order of milliseconds, which is fairly significant since even 9600 baud is roughly one character per millisecond. Also be careful with testing the EEPROM write time, put() only performs an actual write() if the data differs from the current EEPROM contents.
as i said, it's the last command and doesn't affect the transfer
... and works today
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.