Transfer Structures of data Between UNO and ESP8266 using SPI.

Goal Transfer Data Between UNO and ESP8266 using SPI.
I am using Arduino IDE for both the UNO and the ESP8266.
I have established communication using the SPISlave exanples provided by the ESP8266 Library and with some modifications and it works extremely well (removed Delay’s form the Master code UNO side )
(I do not want to use serial communications I am saving this for other purposes)

**Where I need Help: **
The chunk of data that is sent over SPI is always sent is 32 Bytes.

//ESP8266 slave side
void SPISlaveClass::setData(uint8_t * data, size_t len){
    if(len > 32) {
        len = 32; //<<<I believe theis is a Hardware Limit in the ESP8266 using SPI
    }
    hspi_slave_setData(data, len);
}

I have found nothing at this moment to give me proper direction as to how to convert this back to longer char arrays, int’s float’s or other data structures. I have some ideas but I would appreciate examples, sketches, web links, for transferring large amounts of data and or data structures.

My Goal is to: provide a easy way to transfer a chunk of data this could be strings (Not that I use Strings) char int's floats and even structures of data with multiple types.

This is my concept code for UNO: Please advise me on how I can process this in a better way. My demo code uses strings fro ease of visualization, Any Data type and structures is my ultimate goal.

union BufferX {
  byte Byte[164]; // same as uint8_t data type
  char Char[164]; // same as int8_t data type
  unsigned UInt[82]; // same as uint16_t data type
  int Int[82]; // same as int16_t data type
  unsigned long ULong[41]; // same as uint16_t data type
  long Long[41]; // same as int16_t data type
};

union TransferBuffer {
  byte Byte[33]; // same as uint8_t data type
  char Char[33]; // same as int8_t data type
};
TransferBuffer SPI_TransferBuffer; // per code this buffer is exactly 32 bytes
BufferX BufferUNO1;
BufferX BufferUNO2;
void setup() {
  Serial.begin(115200);
  char test[] = "w123456789abcdefx123456789abcdefy123456789abcdefz123456789abcdef@123456789abcdef#123456789abcdef$123456789abcdef^123456789abcdef";
  BufferUNO1.Char[161] = '\0';// Truncate strings for testing
  memcpy(BufferUNO1.Char, test, strlen(test) + 1); // Load the Data to be transfered

  SPI_TransferBuffer.Char[32] = '\0';
  Serial.println("Data to Send:");
  Serial.println(BufferUNO1.Char);
}

void loop() {
  int Recieved = 0;
  while (Recieved < 160) {
    SendBytes(); // This Code is a representation of the ESP8266 Sending Data to the UNO
    Recieved = ReceiveBytes(); // This code is a representation of the UNO receiving data from the ESP8266 
    Serial.print("Recieved Bytes: ");
    Serial.println(Recieved);
    Serial.println(BufferUNO2.Char);
    delay(100);
  }
  Serial.println("Complte");
  Serial.println(BufferUNO2.Char);
  while (1) {};
}


void SendBytes() {
  static int offsetSend = 0;
  setData(BufferUNO1.Byte + offsetSend, 160 - offsetSend );
  offsetSend += 32;
}

int ReceiveBytes() {
  static int offsetReceive = 0;
  byte TempBuffer[32];
  readData(TempBuffer);
  memcpy( BufferUNO2.Byte + offsetReceive, TempBuffer, 32);
  offsetReceive += 32;
  return (offsetReceive);
}


// Code Below this point is for imitation of the SPI Transfer Process



// Slave Code ESP8266
// commented line is the origional code from SPISlave.cpp
void setData(uint8_t * data, size_t len) {
  if (len > 32) {
    len = 32; //<<<I believe theis is a Hardware Limit in the ESP8266 using SPI
  }
  //hspi_slave_setData(data, len);
  memcpy(SPI_TransferBuffer.Byte, data, 32);
}



// Master code for UNO
// Commented lines are part of the code in the ESPSafeMaster.h (I created from the ESPSafeMaster sketch)
void readData(uint8_t * data)
{
  //_pulseSS();
  //SPI.transfer(0x03);
  //SPI.transfer(0x00);
  for (uint8_t i = 0; i < 32; i++) {
    //  data[i] = SPI.transfer(0);
    data[i] = transfer(0);
  }
  //_pulseSS();
}
uint8_t transfer(int x) {
  static int ptr = 0;
  if (ptr == 32) ptr = 0;
  return (SPI_TransferBuffer.Byte[ptr++]);
}

Thanks in advance for any and all suggestions and help :slight_smile:

P.S.
I will provide additional code/schematics upon request but I thought it would be a distraction from the purpose of this post.

I think I would design the struct for my "real" data and then create a byte array that is the same length as the struct. I would then create a union between an instance of the struct and the byte array. And I would iterate over the byte array in 32 byte chunks to send it to the ESP.

I believe this should also work for receiving. If the incoming data is placed in the correct place in the byte array it will automatically be available as the relevant element of the struct.

Or have I completely misunderstood the question?

...R

Robin2: I think I would design the struct for my "real" data and then create a byte array that is the same length as the struct. I would then create a union between an instance of the struct and the byte array. And I would iterate over the byte array in 32 byte chunks to send it to the ESP.

I believe this should also work for receiving. If the incoming data is placed in the correct place in the byte array it will automatically be available as the relevant element of the struct.

Or have I completely misunderstood the question?

...R

So Lets say I wish to receive Data from the ESP's Memory for my LCD and another time Receive Data from The WIFI Cell phone AP or GPS Cords to the web. Both are drastically different in both size and content. you would suggest creating unions for each type of structure? Or could you use the same memory space to handle all this? I'm concerned I could be heading down the path to issues in the future and want to be safe with my choices early on. I need to conserve memory on the UNO I have no such concern with the ESP8266.

a little concept code from your suggestion. Is this what you are suggesting?

typedef struct ConnectToWiFi
{
  char LCDData[64];
  byte IP[4];
  char SSID[32];
  char END[] = "!!!END!!!";
}WiFiConnect;

typedef struct GPSLocation
{
  double Log;
  double Lat;
  char RawData[64];
  char END[] = "!!!END!!!";
}GPSData;
union TransferData
{
  byte * Data;
  WiFiConnect Wifi;
  GPSData GPS;
};
TransferData TransferMe;

//loop would be as follows
void SendBytes() {[color=#222222][/color]
  static int offsetSend = 0;[color=#222222][/color]
  setData(TransferMe.Data + offsetSend, offsetSend - GetTheLengthOfDataAllocated(TransferMe.Data) );
  offsetSend += 32;[color=#222222][/color]
}
GetTheLengthOfDataAllocated(TransferData * Data){
   // does its magic and returns the length :)
  return(Length);
}

a little concept code from your suggestion. Is this what you are suggesting?

Not if you want to upload that to a Mega. The duplicate punctuation does NOT convey more information.

While you list members in one order, the compiler is free to actually arrange the objects in memory however it feels like. Your END member may not actually be at the end.

A union makes things of the same size occupy the same memory. A byte pointer and a WiFiConnect struct are nowhere near the same size.

A struct is a contiguous block of memory. If you have a pointer to a struct, you can pass the data in the struct using just the pointer to the first byte of memory and the number of bytes of memory that the struct uses. No union is necessary.

PaulS: Not if you want to upload that to a Mega. The duplicate punctuation does NOT convey more information.

My C Programming is rusty when it comes to terms like "duplicate punctuation" Could you point out where you are seeing this exactly, Thanks. :)

While you list members in one order, the compiler is free to actually arrange the objects in memory however it feels like. Your END member may not actually be at the end.

Dang, So what I need to do is load the structure into a byte array to transfer. If the compiler can mess up the order then the destination order may not match! So What do you suggest. I need it to be logically readable and be able to recreate it in the destination easily.

A union makes things of the same size occupy the same memory. A byte pointer and a WiFiConnect struct are nowhere near the same size.

Yep :) and this is why I am needing help resolve this. I need a buffer to transfer data as a byte array. I can transfer in 32 byte chunks which my code does very well I have a test routine pounding 32 byte chunks over and back 1000's of times a second and it works great. more than enough bandwidth for my needs. What I am struggling with is how to package a group of data and send it quickly as a packet which will simplify my code... I think :) .

A struct is a contiguous block of memory. If you have a pointer to a struct, you can pass the data in the struct using just the pointer to the first byte of memory and the number of bytes of memory that the struct uses. No union is necessary.

Thank you for this point. I will do this instead. But now your earlier comment is concerning me "the compiler is free to actually arrange the objects [of the structure] in memory however it feels like." Should I allocate an array of bytes large enough to handle the data needed and link the data to it some how. I was hoping the structure would do this without issues. The UNO is where I would like to keep the overall allocation of arrays to a minimum. I would like to use the same space over and over again to transfer all kinds of data. In the ESP8266, I plan on having individual structures to capture the data and share it as needed.

??? Can I access a structure like you would a multi-dimensional array? struct[1] rather than struct.key? What suggestions do you have? Thank you. This information is saving me time :)

My C Programming is rusty when it comes to terms like "duplicate punctuation" Could you point out where you are seeing this exactly,

  char END[] = "!!!END!!!";

I count 6 exclamation points there.

So what I need to do is load the structure into a byte array to transfer.

No. A pointer to the struct points to the first byte of the struct. It is, in effect, a byte array, since memory is nothing but a collection of bytes.

WiFiConnect someStuff;
// Populate the struct

void SomeFunctionToSendData(byte *dataToSend, int lengthOfData)
{
   // Send the data pointed to by dataToSend, until all lengthOfData bytes have been sent
}

   SomeFunctionToSendData(someStuff, sizeof(someStuff));

Since you are limited to sending 32 bytes at a time, SomeFunctionToSendData() needs to send 32 bytes or less, and then increment dataToSend by 32 and decrement lengthOfData by 32. Repeat until lengthOfData is less then 32, then send the remaining bytes (less than 32 of them).

PaulS:

  char END[] = "!!!END!!!";

I count 6 exclamation points there.

lol
I thought you found something critical. Ya I was going to put some kind of unique marker I could search for as i transferred data to know when to stop at the end but your point about “the compiler is free to actually arrange the objects [of the structure] in memory however it feels like.” messed that up and this is just for example to reply to Robin2’s comments. I promise It won’t be in the final code :slight_smile:

No. A pointer to the struct points to the first byte of the struct. It is, in effect, a byte array, since memory is nothing but a collection of bytes.

WiFiConnect someStuff;

// Populate the struct

void SomeFunctionToSendData(byte *dataToSend, int lengthOfData)
{
  // Send the data pointed to by dataToSend, until all lengthOfData bytes have been sent
}

SomeFunctionToSendData(someStuff, sizeof(someStuff));




Since you are limited to sending 32 bytes at a time, SomeFunctionToSendData() needs to send 32 bytes or less, and then increment dataToSend by 32 and decrement lengthOfData by 32. Repeat until lengthOfData is less then 32, then send the remaining bytes (less than 32 of them).

Exactly. But Didn’t we come up with the fact that Structure data can be re-arranged by the compiler. is there a way to fix this to a specific order? This is in the original post and it does what you are mentioning.

void SendBytes() {
  static int offsetSend = 0;
  setData(BufferUNO1.Byte + offsetSend, 160 - offsetSend );
  offsetSend += 32;
}
// With this Code modified from the class
// Code Below this point is for imitation of the SPI Transfer Process

// Slave Code ESP8266
// commented line is the origional code from SPISlave.cpp
void setData(uint8_t * data, size_t len) {
  if (len > 32) {
    len = 32; //<<<I believe theis is a Hardware Limit in the ESP8266 using SPI
  }
  //hspi_slave_setData(data, len);
  memcpy(SPI_TransferBuffer.Byte, data, 32);
}

So now for the tough part. How to properly put it together into a packet and break it out again.
JSON the structure? is there a library for Structure to JSON or something similar/simpler to crunch the structure into something that can be sent.
I can’t assume that the compiler will organize the structure on the UNO and the ESP8266 the same way…
“the compiler is free to actually arrange the objects [of the structure] in memory however it feels like.”

Any Ideas?

I thought you found something critical.

Try uploading that code to a Mega. See if the fact that you can't do so is critical, or not.

But Didn't we come up with the fact that Structure data can be re-arranged by the compiler. is there a way to fix this to a specific order?

Not willy-nilly. If any re-arrangement takes place, it will happen the same way on all compilers.

I don't see why you need to copy the data into another array to send it.

PaulS: Try uploading that code to a Mega. See if the fact that you can't do so is critical, or not. Not willy-nilly.

I'm not going to use the end marker it was just a quick thought char END[] = "!!!END!!!"; found in my structure example I appreciate the point note taken. I checked and the code I am using compiled and works on my mega without any problems. I will be adding to it shortly with your suggestions.

If any re-arrangement takes place, it will happen the same way on all compilers.

I don't see why you need to copy the data into another array to send it.

You have been extremely helpful I have to thank you :) I am going to create a functioning test sketch with your suggestions and post in the next 30 minutes if all goes well. I hope you will have the time to review the post Thank You! Z

Well attempting to run this but it is failing with the transfer of the structure into the function as a char array pointer :confused:
How to convert a structure to a char array? Problems start at line 85 (send)

typedef struct ConnectToWiFi {
  char LCDData[64];
  char List[128] ;
  char SSID[32] ;
};
ConnectToWiFi WiFiConnect;
ConnectToWiFi RWiFiConnect;
typedef struct GPSLocation
{
  double Log ;
  char LonEW;
  double Lat;
  char LatNS;
  char RawData[128];
} ;
GPSLocation GPSData;
GPSLocation RGPSData;
   int offsetSend = 0;
   int offsetReceive = 0;
void PrintWiFiConnect(ConnectToWiFi * WIFI) {
  Serial.print("WiFi Data\nLCDData: ");
  Serial.println(WIFI->LCDData);
  Serial.print("List: ");
  Serial.println(WIFI->List);
  Serial.print("SSID: ");
  Serial.println(WIFI->SSID);

}

void PrintGPSLocation(GPSLocation * GPS) {
  Serial.print("GPS Location\nLon: ");
  Serial.println(GPS->Log);
  Serial.print("LonEW: ");
  Serial.println(GPS->LonEW);
  Serial.print("Lat: ");
  Serial.println(GPS->Log);
  Serial.print("LatNS: ");
  Serial.println(GPS->LatNS);
  Serial.print("RawData: ");
  Serial.println(GPS->RawData);
}

byte SPI_TransferBuffer[33]; // same as uint8_t data type


void setup() {
  //Load the structures for testing

  Serial.begin(115200);
  Serial.println("Data Transfer Test");
  delay(100);
  memcpy(WiFiConnect.LCDData, "Select the SSID", sizeof("Select the SSID"));
  memcpy(WiFiConnect.List, "ThankYou,ArduinoCC,MyWiFi,YourWiFi", sizeof("ThankYou,ArduinoCC,MyWiFi,YourWiFi"));
  memcpy(WiFiConnect.SSID, "ThankYou", sizeof("ThankYou") );
  GPSData.Log = 01131.000;
  GPSData.LonEW = 'E';
  GPSData.Lat = 4807.038;
  GPSData.LatNS = 'N';
  memcpy(GPSData.RawData, "$GPGGA, 123519, 4807.038, N, 01131.000, E, 1, 08, 0.9, 545.4, M, 46.9, M, , * 47", sizeof("$GPGGA, 123519, 4807.038, N, 01131.000, E, 1, 08, 0.9, 545.4, M, 46.9, M, , * 47"));
  SPI_TransferBuffer[32] = '\0';

}
void printSPI_TransferBuffer() {
  Serial.println();
  for (int i = 0; i < 32; i++) {
    Serial.print((char)SPI_TransferBuffer[i]);
  }
  Serial.println();
  for (int i = 0; i < 32; i++) {
    Serial.print((byte)SPI_TransferBuffer[i], HEX);
  }
  Serial.println();
}
void loop() {
  int Recieved = 0;
    offsetSend = 0;
   offsetReceive = 0;
     Serial.println("Data to Send: ");
  PrintWiFiConnect(&WiFiConnect);

  Serial.println();
  Serial.println();
  Serial.println();
  while (Recieved < sizeof(WiFiConnect)) {
    SendBytes((byte) &WiFiConnect);
    Recieved = ReceiveBytes((byte) &RWiFiConnect);
    Serial.print("WiFiConnect SPI_TransferBuffer: ");
    printSPI_TransferBuffer();
    delay(100);
  }
  Serial.println();
    PrintWiFiConnect(&RWiFiConnect);
    delay(2000);
  Serial.println();
  Serial.println();
    PrintGPSLocation(&GPSData);
  Serial.println();
  Serial.println();
  Serial.println();
    while (1) {};
Recieved = 0;
    offsetSend = 0;
   offsetReceive = 0;
  while (Recieved < sizeof(GPSData)) {
    SendBytes((byte *) & GPSData);
    Recieved = ReceiveBytes((byte) &RGPSData);
    Serial.print("GPSData SPI_TransferBuffer: ");
    printSPI_TransferBuffer();
    delay(100);
  }
  Serial.println();
  Serial.println();
  Serial.println();

  Serial.println("Complte");
  PrintWiFiConnect(&RWiFiConnect);
  PrintGPSLocation(&RGPSData);
  while (1) {};
}


void SendBytes(byte * StructurePointer) {

  setData(StructurePointer + offsetSend, 160 - offsetSend );
  offsetSend += 32;
}

int ReceiveBytes(byte * StructurePointer) {

  byte TempBuffer[32];
  readData(TempBuffer);
  memcpy( StructurePointer + offsetReceive, TempBuffer, 32);
  offsetReceive += 32;
  return (offsetReceive);
}


// Code Below this point is for imitation of the SPI Transfer Process



// Slave Code ESP8266
// commented line is the origional code from SPISlave.cpp
void setData(uint8_t * data, size_t len) {
  if (len > 32) {
    len = 32; //<<<I believe theis is a Hardware Limit in the ESP8266 using SPI
  }
  //hspi_slave_setData(data, len);
  memcpy(SPI_TransferBuffer, data, 32);
}



// Master code for UNO
// Commented lines are part of the code in the ESPSafeMaster.h (I created from the ESPSafeMaster sketch)
void readData(uint8_t * data)
{
  //_pulseSS();
  //SPI.transfer(0x03);
  //SPI.transfer(0x00);
  for (uint8_t i = 0; i < 32; i++) {
    //  data[i] = SPI.transfer(0);
    data[i] = transfer(0);
  }
  //_pulseSS();
}
uint8_t transfer(int x) {
  static int ptr = 0;
  if (ptr == 32) ptr = 0;
  return (SPI_TransferBuffer[ptr++]);
}

This is based on an example of how I used a union when receiving serial data from a PC

typedef struct ConnectToWiFi
{
  char LCDData[64];
  byte IP[4];
  char SSID[32];
  char END[] = "!!!END!!!"; // 10 chars
  // 110 bytes total
}WiFiConnect;


union TransferData
{
  WiFiConnect Wifi;
  byte wifiArray[110];
};

TransferData dataToSend;

I don't know, but I assume the Union does not allocate an extra 110 bytes of memory and that it just accesses the same memory using a different name.

If I was sending data using an nRF24 I think, then, I would do it like this

radio.write( &dataToSend, 32 );

followed by

radio.write( &dataToSend + 32, 32 );

Or maybe it needs to be &dataToSend.WifiArray

...R

How to convert a structure to a char array?

YOU DO NOT NEED TO!

Frustrated and not getting anywhere I gave up for the night. After a break I remembered the clean simple functions of EEPROMAnything which accepts any data type including structures! This should accomplish what you are mentioning PaulS and Robin2. I will be tackling this today after church :slight_smile:

To Clarify what my goals are: I am connecting a UNO as master to a ESP8266 as slave through SPI using the Arduino API for programming both the UNO and ESP8266. The ESP8266 has 4 meg of flash memory that could be accessed by the UNO to store anything from Menus for my LCD Display on the UNO to a Web interface. SPI code is small (on the UNO) and hardware driven so it is great! I have accomplished having the two Send 32 bytes back and forth as fast as the could without any errors!
Where I got stuck :slight_smile: is with stuffing data into the 32 byte transfer register (This is part of the library I am using SPISlave.cpp). PaulS You have been an incredible help, I appreciate your advise, I am considering re-writing/adding to or even completely re-creating the Library to access the structure data directly.

setData funciton in SPISlave.cpp:

void SPISlaveClass::setData(uint8_t * data, size_t len)
{
    if(len > 32) {
        len = 32;
    }
    hspi_slave_setData(data, len);
}

Thank you!
Z

Inspiration from a portion of EEPROMAnything.h

template <class T> int EEPROM_writeAnything(int ee, const T& value)
{
    const byte* p = (const byte*)(const void*)&value;
    unsigned int i;
    for (i = 0; i < sizeof(value); i++)
	  EEPROM.write(ee++, *p++);
    return i;
}

template <class T> int EEPROM_readAnything(int ee, T& value)
{
    byte* p = (byte*)(void*)&value;
    unsigned int i;
    for (i = 0; i < sizeof(value); i++)
	  *p++ = EEPROM.read(ee++);
    return i;
}