Buffer data from pc and sending from the arduino

Hey people, ill give this thing to you straight.

  1. I have a fairly large text file filled with binary data (0s and 1s only) now i want the arduino or nodemcu ( faster clocks) to save some of the binary data from the text file stored on pc to its own memory( the memory Isn't big enough for the whole file ) and then accordingly pull a pin high or low( or use the pulseln function)

  2. Do the same thing but instead receive the pulses and accordingly saving them to a text file. I understand that it is also enough for it to show them in the serial monitor as there is supposed to be a software (name?)that saves data from the serial monitor.

My project is basically to show how data transfer works, its not meant to be practical but its meant to be simple

One more thing, i cant think of any data Corruption preventing methods, i really want to avoid libraries.
Isnt there any simple way to organise the data or add trailer data blocks as seen on synchronus data transmission?

Another thing, how fast can the arduino pins switch state in respect of this program?

please be as elaborate as possible, thank you

You will have to send your file in blocks using a dedicated application on the PC. Once a block is received and processed by the Arduino, the Arduino can send an acknowledgement back.
2)
Reverse of (1)
3)
You can't prevent data corruption. You can calculate a checksum or crc over a block and add that to the block that you send; the receiver calculates the checksum or crc over the received data and compares the value with the received one. If match, process and acknowlege, else return a error code and the sender can send again.
4)
Speed depends mostly on the baudrate that you use for the serial communication and the exact way that you implement the protocol/software.

sterretje:
You will have to send your file in blocks using a dedicated application on the PC.

Does such an application exist? Or is it upto me to make one?

Wouldnt crc use more processing time between checking file checksum? As the data is binary there is also possibility the crc would get lost or corrupted.
( It struck me last night that the laser im using is modulated at 38kHz, which means it is immune(mostly) to noise. So if i tell the computer that every block carries certain number of bits and if any block.is missing a bit or two then it asks for the block again)

This thing is getting complicated again, i really want to keep it simple. I know how this works, how i want it to work,but i have no idea how im gonna implement this on arduino

I have no idea if such an application exists; you will probably have to write it yourself but google and co are your friends :wink:

CRC consumes a bit more processing power, but is also more reliable in detecting errors than checksum.

You know how many bytes are send in a message, so loosing bytes should be easy to detect; a corrupted crc will result in a failure of the check so you can ask for a resend.

What's your project about? There was no mention of laser in the first post, just data transfer from PC to Arduino and vice versa.

sterretje:
What's your project about? There was no mention of laser in the first post, just data transfer from PC to Arduino and vice versa.

I almost Didn't mention the whole project to avoid complications. My project is basically about transfering data over light. l i f i. sigh
I didnt mention this from the beginning as i saw a lot of people asking about this without any research whatsoever. They basically asked for a final code and diagram.i wanted to avoid that scenario for this post.

Now that it has came up, i would like to share some of my realizations and current problems.

Good stuff

1.there needs to be a receiver and a sender on both of the machines.

2.hardware modulation is better ( I'm sure you know that a ir receiver requires modulated 38khz pulse). Using a 555 timer as the modulator also reduces code complication and cpu usage.

3.it is better that the arduino buffers only 1 packet of bits every time so we Don't have to add different identifiers for each packets. If data Doesn't go through, the arduino sends the whole buffered packet again.

4.The receiver should send confirmation after every packets, not after an error, if it only sends a signal only on error. then if the beam is obstructed the arduino will keep sending data but none will be received. (I know there are many other ways to make this fail, but this is enough for me right now)

Problems

  1. I cant find a proper function that can read data from a file and save it temporarily in the arduino memory,(ram, if possible).

  2. Same as before, a function that saves the received data into ram and then sends it to pc-serial monitor.

  3. I havent tried this out yet, if the data received is considered as arrays with fixed starting and ending, how do i call for the n of array in arduino for (n of array-(fixed start bit + fixed end bit))

  4. Library based digitalWrite function and digitalRead is hella slow, how would i go on and use lower level codes for faster read write functions. I saw some online but they didnt made much sense.please explain a bit.

  5. At what reliable speed can the arduino pins turn high/low?

This is basically the summary of the project in my mind. Please help as much as you can, as far as i can see, no one has gone through this project yet. Atleast there is no documentations

Good grief! No research has been done!

One of the first commercial use for lasers was to extend data communication link between buildings using laser transmission, two way.

All data files are binary! A text file by definition contains only a subset of possible binary values, namely those representing the printable characters and carriage return and line feed. Your file is nothing special and is not a text file if it contains other than the stated characters.

Do you understand the optics necessary to create your laser system, or are you going to use a commercial system?

Paul

kaseftamjid:

  1. I have a fairly large text file filled with binary data (0s and 1s only)

I presume you mean that the text file has a data in the form of '0' and '1' characters - which is not generally what programming people mean when they use the term "binary data". The character '0' has a byte value of decimal 48.

A binary file is different from a text file and in a binary file the data would be stored as byte values 0 and 1

You may find some useful ideas in Serial Input Basics - simple reliable ways to receive data.

The technique in the 3rd example will be the most reliable. It is what I use for Arduino to Arduino and Arduino to PC communication.

You can send data in a compatible format with code like this (or the equivalent in any other programming language)

Serial.print('<'); // start marker
Serial.print(value1);
Serial.print(','); // comma separator
Serial.print(value2);
Serial.println('>'); // end marker

It would be straightforward to extend the code to include a checksum.

This Simple Python - Arduino demo may also be of interest.

...R

Note in advance: I assume that you have a file that contains binary data, not the text 01000011111...

There is no function to read something from a PC. You will have to write (or find) and application. The below implements a serial reading for data from a PC. It reads 32 bytes from the serial port and stores it in RAM. For synchronisation, I've implemented a timeout (1 second, hard coded). It's basically Robin's example reworked.

// result codes for serial receive from PC
enum SERIALRESULT
{
  COMPLETE,       // complete
  IN_PROGRESS,    // waiting for more data
  CRC_ERROR,      // crc error
  TIMEOUT_ERROR,  // timeout error
};


#define BUFFERSIZE 32
uint8_t receivedBytesFromPC[BUFFERSIZE];

void setup()
{
  Serial.begin(2000000);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
}

void loop()
{
  SERIALRESULT sr = readSerial();
  
  switch (sr)
  {
    case COMPLETE:
      // crc check
      // process the packet
      for (uint8_t cnt = 0; cnt < BUFFERSIZE; cnt++)
      {
        if(receivedBytesFromPC[cnt] < 0x10)
        {
          Serial.print(0);
        }
        Serial.print(receivedBytesFromPC[cnt], HEX);
        Serial.println();
      }

      // acknowledge to PC
      Serial.print(sr);
      break;
    case IN_PROGRESS:
      // nothing to do
      break;
    default:
      // tell PC that something failed
      Serial.print(sr);
      break;
  }
}

SERIALRESULT readSerial()
{
  // index in array of received characters
  static uint8_t index = 0;
  static uint32_t startTime;

  // while there is data
  while (Serial.available() > 0)
  {
    // if it's first byte received
    if (index == 0)
    {
      // set the time that this byte was received
      startTime = millis();
    }
    // add to buffer
    receivedBytesFromPC[index++] = Serial.read();

    // if block received
    if (index == BUFFERSIZE)
    {
      index = 0;
      return COMPLETE;
    }
  }

  // if receive in progress and timed out
  if (index != 0 && (millis() - startTime > 1000))
  {
    index = 0;
    return TIMEOUT_ERROR;
  }

  return IN_PROGRESS;
}

You can use to get ideas; tested on a Nano at 2MBit/second

The principle is the same as for (1). Be aware that at 2MBit/s, sending 32 bytes to a PC takes 32 / 200,000 seconds. Don't try to send more than the software buffer size of the buffer in the Serial class (64 bytes) or the program will stall.

Accessing an array element is shown in the above code.

Port manipulation; this is the fastest approach for toggling a pin. Note that it's for AVR chips (e.g. 328P on Uno, Nano).

I think it's a single processor cycle (62.5 ns @16MHz) to set or clear a pin.

sterretje:
Note in advance: I assume that you have a file that contains binary data, not the text 01000011111...

Just to be clear, I was assuming the opposite in Reply #6

Now the OP has both options before him/her

...R

Paul_KD7HB:
Good grief! No research has been done!

One of the first commercial use for lasers was to extend data communication link between buildings using laser transmission, two way.

Paul

Sorry, slight misunderstanding, i meant as a diy thing.

Paul_KD7HB:
All data files are binary! A text file by definition contains only a subset of possible binary values, namely those representing the printable characters and carriage return and line feed. Your file is nothing special and is not a text file if it contains other than the stated characters.

Paul

I dont quite understand the "subset of possible value" i thought all files were binary, i can create a hex file out of them (hex dump?) and then convert it to binary, this thing was pretty much manual, so then i would have a text file that has binary values of another file. For example- an image. It has been said multiple times that the arduino memory is not big enough to send an image. If the data goes through quite okay, then i can manually convert it back to what it was. I would absolutely love nothing more if there was an existing software that did it for me.

Paul_KD7HB:
Do you understand the optics necessary to create your laser system, or are you going to use a commercial system?

Paul

sorry for the fact that im writing everything in one reply, im still considered a bot by the website
I dont know of any available commercial system, multiple google searches didnt come up with any designs for the receiver.
What i planned to do was to place the receiver inside a small box with reflectors inside and a small lense outside. I dont need to have a perfectly focused beam on the ir reciever.

A bye has 8 bits, they can form one of 256 different numeric values. The ASCII set has only 128 possible values, see the ASCII character set table. Of that 128 values, there are 26 numeric values that represent the 26 English upper case letters. There are 26 numeric values that represent the English lower case letters. And there are 10 values that represent the digits 0 through 9. All remaining values are various control characters for a printer. A text file is an ASCII character file with logical lines identified by a carriage return and perhaps a line feed character at the end of the "line".

A binary file can contain any numeric values from 0 to 255, or possibly be all of one numeric value.

You write of using a laser, then you mention using a modulated IR light source. Perhaps you need to explain more clearly what you are designing.

The use of laser communication to allow data communication between campus buildings was something from the 1970's and 1990's. The use of the internet made that leap so easy that no one makes or even has the need for laser communications between buildings any more.

Paul

You people are frigging geniuses, i would've replied specifically to every comment but it would be way longer than this one.

As of explaining what im trying to do is what you people think im trying to do, what people did in 1970-1990. A simple way of showing that visible light can carry data. Not many (if any) people around me know that this tech ever existed, they think that when data has to go somewhere,it happens - so it happens. This is for them, a simplified form of data communication. Binary number system-easy to explain, lights turning on off -easy to explain, a small computer reading data from another computer, easy to explain. As i have stated before,
The objective of this project is to be simple, not practical.

sterretje:
Note in advance: I assume that you have a file that contains binary data, not the text 01000011111...

There is no function to read something from a PC. You will have to write (or find) and application. The below implements a serial reading for data from a PC. It reads 32 bytes from the serial port and stores it in RAM. For synchronisation, I've implemented a timeout (1 second, hard coded). It's basically Robin's example reworked.

// result codes for serial receive from PC

enum SERIALRESULT
{
  COMPLETE,      // complete
  IN_PROGRESS,    // waiting for more data
  CRC_ERROR,      // crc error
  TIMEOUT_ERROR,  // timeout error
};

#define BUFFERSIZE 32
uint8_t receivedBytesFromPC[BUFFERSIZE];

void setup()
{
  Serial.begin(2000000);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
}

void loop()
{
  SERIALRESULT sr = readSerial();
 
  switch (sr)
  {
    case COMPLETE:
      // crc check
      // process the packet
      for (uint8_t cnt = 0; cnt < BUFFERSIZE; cnt++)
      {
        if(receivedBytesFromPC[cnt] < 0x10)
        {
          Serial.print(0);
        }
        Serial.print(receivedBytesFromPC[cnt], HEX);
        Serial.println();
      }

// acknowledge to PC
      Serial.print(sr);
      break;
    case IN_PROGRESS:
      // nothing to do
      break;
    default:
      // tell PC that something failed
      Serial.print(sr);
      break;
  }
}

SERIALRESULT readSerial()
{
  // index in array of received characters
  static uint8_t index = 0;
  static uint32_t startTime;

// while there is data
  while (Serial.available() > 0)
  {
    // if it's first byte received
    if (index == 0)
    {
      // set the time that this byte was received
      startTime = millis();
    }
    // add to buffer
    receivedBytesFromPC[index++] = Serial.read();

// if block received
    if (index == BUFFERSIZE)
    {
      index = 0;
      return COMPLETE;
    }
  }

// if receive in progress and timed out
  if (index != 0 && (millis() - startTime > 1000))
  {
    index = 0;
    return TIMEOUT_ERROR;
  }

return IN_PROGRESS;
}



You can use to get ideas; tested on a Nano at 2MBit/second

2)
The principle is the same as for (1). Be aware that at 2MBit/s, sending 32 bytes to a PC takes 32 / 200,000 seconds. Don't try to send more than the software buffer size of the buffer in the Serial class (64 bytes) or the program will stall.

3)
Accessing an array element is shown in the above code.

4)
[Port manipulation](https://www.arduino.cc/en/Reference/PortManipulation); this is the fastest approach for toggling a pin. Note that it's for AVR chips (e.g. 328P on Uno, Nano).

5)
I think it's a single processor cycle (62.5 ns @16MHz) to set or clear a pin.

Im having a little bit of trouble understanding your code. As in what is it actually reading,

can port manipulation be used with this code

What pins am i limited to?

If i use the tx rx pins can i still use serial functions and port manipulation?

The code reads data from a PC. Once you have master reading a byte from the IR, you can duplicate the code for use with IR.

You can use the code with direct port manipulation; not sure about the rx/tx for writing so I would connect the IR sender to pins of a different port as a precaution (just too lazy to research, use e.g. the analog pins or pins 8..13).

tx/rx are, on most Arduinos, used for the usb communication with a PC as well; don't use them for anything else.

sterretje:
The code reads data from a PC.

So from what i type in the serial monitor?

One more thing, i found this online
https://arduinobasics.blogspot.com/2012/05/reading-from-text-file-and-sending-to.html?m=1
Im not at all good with coding, can someone please analyze this code and tell me if it can fit my purpose?

Yes. Or from what you send using a PC application.

I think ive hit another wall.. The ir reciever(s) i have with me currently right now are not branded, as in i dont know there model number, as they were recycled from a tv. Is there a possibility that the arduino sends the data too fast for the reciever to process? Or should i go with analog 2 pin ir receiver?

I have no udea about IR. But can you please attach your fairly big file so we can have a look at to check if it's text or something else.

about the file.. i originally wanted to send an image but it seems that the conversion of jpg to hex to binary is a bit complicated and impractical. a 7 kb jpg file turns out to be a 60kb++ file. other files are okay, but i originally wanted to send images. the big file filled with binary data i was talking about meant converting a file eg. text to binary through converters,,,,

is there a way to do this without manually converting stuff? as far as i know, most of the time sending devices dont have a really large buffer memory

Why would you want to send a 7kB jpeg file in another format? The code in reply #7 should happily receive that file from a PC in blocks of 32 bytes and do something with it.