Writing non-text (binary) data to SD file -- How?

This should be so very simple but it is un-crackable for everyone, it seems. I have read multiple sites and tried multiple examples. They all come down to writing strings to files. That's not practical for real-world usage of data and files.

One needs to just write numbers in their native format to the file and later retrieve them the same way. They need actual binary data I/O. In VB, this is dead simple. You just write the entire array out in one go. It goes in binary format to the file and you read it back the same way -- right into the array. No conversions needed because it plots the whole thing, intact, into the file. Not so in C. Is the SD library even capable of actually writing binary data? Maybe that's the snag...

I have a typedef struct and that is used as the pattern to build an array. I want to write the array data to a file. Can't make it happen. Here are the two closest way I have come to it.

I also will need to read the data back in. If someone can figure out how to write it, maybe include a way to read the data back into the array, too. That would be appreciated. Then I will post the complete, built and working answer so others can write non-string data to an SD file.

If this has been solved and there is a working example of what I want to do, please mention it and I will gladly go start with that example but even the ones that say they will write binary data end up writing strings. I could break the array up into all its pieces, byte by byte and convert them to strings going out and then make them numbers coming back in but that is not a general solution to the problem. And it is ugly and long.

Here are my two examples of code and the error messages I receive:

typedef struct
{
  IPAddress iaddr;    // Little Endian (not sortable)
  IPAddress iaddrBE;  // Big Endian  (sortable)
  long      count;
} IPCallers;

#define   MAX_CALLERS 300
IPCallers MyCallers[MAX_CALLERS];

...

  const size_t BUF_SIZE = sizeof(IPCallers);

  uint8_t buf[BUF_SIZE];  // Taken from the example supposed to write binary data.

  buf = MyCallers[0];  // How???  This is the error line.

  if (dataFile.write(buf, sizeof(buf)) != sizeof(buf)) {
    Serial.println("write failed");
    dataFile.close();
    return;
  }
  
gives: incompatible types in assignment of 'IPCallers' to 'uint8_t [20] {aka unsigned char [20]}'

...or...
  
    const size_t BUF_SIZE = sizeof(IPCallers);

    IPCallers buf[BUF_SIZE];  // Making it a match for what I want to put in.

    buf = MyCallers[0];   // The error line

    if (dataFile.write(buf, sizeof(buf)) != sizeof(buf)) {
      Serial.println("write failed");
      dataFile.close();
      return;
    }
    
gives: incompatible types in assignment of 'IPCallers' to 'IPCallers [20]'

Lots of folks would probably benefit from the solution to this. And my thanks would be legion. I have fought this for a long time and it is time to know how to do it.

Mike

The following edit is for the cute poster who, instead of posting any answer, showed the math to prove I could not possibly run. I am using an ESP8266. MEMORY is NOT A PROBLEM! If that was any problem, it would have already shown up. Don't bother pointing out memory problems. Need answers to the problem only. It is so pleasant to have a big memory space that enables real programs to run, not little toy programs. Somewhat like I have been used to for so many years on much larger machines.

What type of Arduino are you using? If it's a Uno or other 328 based common one I see a bigger problem lurking.

buf = MyCallers[0];  // How???  This is the error line.

You need to cast it.

buf = (uint8_t)MyCallers;  // Name of array is pointer to first element

buf = (uint8_t)(MyCallers + 1);  // Use math to get elements 2 

buf = (uint8_t)(MyCallers + 2);  // or 3

Remember, that first one could have been + 0 if you want to use the same variable you were going to use as an index to that array.

Sorry for all the edits. I'm apparently a little buggy right now.

Let's see...each struct takes 8 bytes and you want 300 of them, so 8 * 300 = 2400 bytes - 2000 bytes of available SRAM, leaving -400 bytes to be taken from...?

econjack:
Let’s see…each struct takes 8 bytes and you want 300 of them, so 8 * 300 = 2400 bytes - 2000 bytes of available SRAM, leaving -400 bytes to be taken from…?

8? I thought they were 12. Aren’t the IP addresses 4 bytes each? Why would you store it both ways? It’s easy enough to convert on the fly if you need it.

Nick…help! I thought on an Uno that, since the address space is only 32K, they were 2 byte addresses. Who has the answer?

since the address space is only 32K, they were 2 byte addresses. Who has the answer?

Variable addresses ARE 2 bytes. IP addresses are 4 or 6 bytes each. The Arduino only support IP4, so IP addresses are 4 bytes.

It's one of these:

class IPAddress : public Printable {
private:
    union {
	uint8_t bytes[4];  // IPv4 address
	uint32_t dword;
    } _address;

PaulS:
Variable addresses ARE 2 bytes. IP addresses are 4 or 6 bytes each. The Arduino only support IP4, so IP addresses are 4 bytes.

When are they 6 byte values?

I am storing the IP address both ways because I sort the array by IP address to present it. I don't want to have to munge the LE address into the BE address every time I run through the sort. I just fix and store it both ways then sort on the BE format, print the LE format and it is easy. Not sure why the library creator decided to store it backwards, but he did. Took me a while to figure that out.

Memory is NOT a problem with ESP8266. Nor Heap. Got about 20KB of free heap most of the time. Varies slightly. No more Uno's for me! Mega is better but it is physically big, expensive and has no native WiFi or Ethernet. ESP8266 controllers (WeMos D1, especially but all of them, too) are tiny to small, cost about $5 and have WiFi already built into it.

This project is online, already. Static28.ILikeTheInternet.com and Static29.ILikeTheInternet.com. It is still a work in process. I don't have the outside thermometer connected yet. Maybe tomorrow. Then adding a second line to the .svg temperature graph. There are some other limitations on String sizes for WiFi output (2922 bytes per Client.Write) and it seems an 8K limit on a single array. Not sure what all that's about but it is workable so I don't complain.

Will test the data file writing tomorrow with the casting. Then I might have to repost the request on how to read this thing back into the array.

Thanks to Delta_G for the one answer that talked about the casting. That might get me half way there. Results soon.

Mike

When are they 6 byte values?

When using IP6 addressing.

An IP4 address has 4 octets. At some point, that is not going to be enough to handle every thing that wants to be connected to the internet. So, IP6 addressing, while still relatively rare, was developed to handle that. IP6 address have 6 octets, allowing a lot more devices to have unique IP addresses.

PaulS:
When using IP6 addressing.

An IP4 address has 4 octets. At some point, that is not going to be enough to handle every thing that wants to be connected to the internet. So, IP6 addressing, while still relatively rare, was developed to handle that. IP6 address have 6 octets, allowing a lot more devices to have unique IP addresses.

Two years older than dirt and still learning something new every day...thanks, Paul.

The only actual attempted answer out of all these replies does not work:

buf = (uint8_t)MyCallers; // Name of array is pointer to first element

gets:

cast from 'IPCallers*' to 'uint8_t {aka unsigned char}' loses precision [-fpermissive]

So back to the search of how to actually write binary data to an SD file. Is there nobody who knows how to do this?

Mike

Here is the code as simple as I can make it that fails. It also shows variations I have tried (mine and suggestions from this thread). Nothing will compile.

Can someone actually make some variation of this that will compile, please? Then I can see if it will actually work.

The desire is to write actual real binary data to an SD card. This, so far, seems impossible. The error message for each variation is in a comment above the statement I tried to use.

#include <ESP8266WebServer.h>

#define   MAX_CALLERS 300
typedef struct
{
  IPAddress iaddr;    // Little Endian
  IPAddress iaddrBE;  // Big Endian
  long      count;
} IPCallers;

IPCallers MyCallers[MAX_CALLERS];

const size_t BUF_SIZE = sizeof(IPCallers);
//IPCallers buf;  
uint8_t buf[BUF_SIZE];
  
void setup() {

//cannot convert 'IPCallers' to 'char*' for argument '1' to 'char* strcpy(char*, const char*)'
//  strcpy(buf, MyCallers[0]);  // Name of array is pointer to first element

//no match for 'operator=' (operand types are 'IPCallers' and 'IPCallers [300]')
//  buf = MyCallers; 

//One suggestion by Delta_G
//cast from 'IPCallers*' to 'uint8_t {aka unsigned char}' loses precision [-fpermissive]
//  buf = (uint8_t)MyCallers;  // Name of array is pointer to first element

//invalid cast from type 'IPCallers' to type 'uint8_t {aka unsigned char}'
  buf = (uint8_t)MyCallers[0];  // Name of array is pointer to first element
}

void loop() {

//The target of all this is this statement to write binary to a file.  
//I will eventually want to read it back in, too.  Don't post about missing SD/file setup code.  
//This was put here to show the needed action but not clutter the post with boring stuff.

//  dataFile.write(buf, sizeof(buf));

}

Can someone actually make some variation of this that will compile, please?

#include <ESP8266WebServer.h>

#define   MAX_CALLERS 300
typedef struct
{
  IPAddress iaddr;    // Little Endian
  IPAddress iaddrBE;  // Big Endian
  long      count;
} IPCallers;

IPCallers MyCallers[MAX_CALLERS];

const size_t BUF_SIZE = sizeof(IPCallers);
//IPCallers buf;  
uint8_t buf[BUF_SIZE];
  
void setup() {
memcpy(buf,MyCallers,sizeof(IPCallers));
//this also compiles for, I think, picking another member of the struct array
//memcpy(buf,&MyCallers[1],sizeof(IPCallers));

}

void loop() {

//The target of all this is this statement to write binary to a file.  
//I will eventually want to read it back in, too.  Don't post about missing SD/file setup code.  
//This was put here to show the needed action but not clutter the post with boring stuff.

//  dataFile.write(buf, sizeof(buf));

}

Sorry. Should be uint8_t*.

PaulS:
When using IP6 addressing.

An IP4 address has 4 octets. At some point, that is not going to be enough to handle every thing that wants to be connected to the internet. So, IP6 addressing, while still relatively rare, was developed to handle that. IP6 address have 6 octets, allowing a lot more devices to have unique IP addresses.

IPv6 addresses are 128 bits, or 16 bytes, long.

Did you ever figure this out?

I think the answer may be in this code, but I'm not sharp enough to figure out if that's what's happening for sure.

https://github.com/greiman/SdFat/blob/master/examples/LowLatencyLoggerADXL345/LowLatencyLogger.ino