read csv from sd line by line and print each line to can shield

Projects Programming
1

Hello,

im having a hard time combining code snippets.

i have a arduino uno and a seeed studio can shield (CS= pin 9) and a seeed studio sd shield (CS= pin 4)

ive searched for many many "read csv file from sd card" code examples but they usually just print to the serial line.

for the send to the can shield code, the example that seeed provides is pretty straight forward

// demo: CAN-BUS Shield, send data
#include <mcp_can.h>
#include <SPI.h>

// the cs pin of the version after v1.1 is default to D9
// v0.9b and v1.0 is default D10
const int SPI_CS_PIN = 9;

MCP_CAN CAN(SPI_CS_PIN);                                    // Set CS pin

void setup()
{
    Serial.begin(115200);

    while (CAN_OK != CAN.begin(CAN_500KBPS))              // init can bus : baudrate = 500k
    {
        Serial.println("CAN BUS Shield init fail");
        Serial.println(" Init CAN BUS Shield again");
        delay(100);
    }
    Serial.println("CAN BUS Shield init ok!");
}

unsigned char stmp[8] = {0, 1, 2, 3, 4, 5, 6, 7};
void loop()
{
    // send data:  id = 0x00, standrad frame, data len = 8, stmp: data buf
    CAN.sendMsgBuf(0x00, 0, 8, stmp);
    delay(100);                       // send data per 100ms
}

however, id like to replace the stuff in parentheses:

CAN.sendMsgBuf([b]0x00, 0, 8, stmp[/b])

with a single variable that is the line that is read from the csv file on the SD card.

i know that id have to rework that variable just above the CAN.sendMsgBuf()

i use the can shield to log my vehicles can data straight into a csv file already.

id like to send 7 lines of the saved can data in the csv file on the SD card to the can shield(the 7 variables that ive previously logged into a file and will send later) and then pause for 1 sec, and then send the next 7 lines.

the problem im having is finding a sketch that is close to what im looking to do for the sd text read that is not crazy complicated/under documented so that i dont get overwhelmed trying to work my way down through the code to figure out how to edit it to do what i want.

for clarification, it would flow something like this

read csv line 1 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 2 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 3 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 4 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 5 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 6 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
read csv line 7 till \n-> save to variable (line) ->CAN.sendMsgBuf(line)
pause(1000)

anyone have any pointers. im muddling through this and reading all the reference material i can read and im still having a hard time.

Thanks

Brian

2

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2016.10.31 15:11:00 =~=~=~=~=~=~=~=~=~=~=~=
CAN BUS Shield init ok!
7DF,2,1,D,0,0,0,0,0,
7E8,3,41,D,0,0,0,0,0,
7DF,2,1,C,0,0,0,0,0,
7E8,4,41,C,B,E6,0,0,0,
7E8,4,41,C,B,E6,0,0,0,
7DF,2,1,2F,0,0,0,0,0,
7DF,2,1,2F,0,0,0,0,0,
7E8,3,41,2F,61,0,0,0,0,
7DF,2,1,D,0,0,0,0,0,
7DF,2,1,D,0,0,0,0,0,
7E8,3,41,D,0,0,0,0,0,
7DF,2,1,2F,0,0,0,0,0,
7DF,2,1,2F,0,0,0,0,0,
7E8,3,41,2F,61,0,0,0,0,
7E0,3,22,1E,1C,0,0,0,0,
7E8,5,62,1E,1C,4,8C,0,0,

this is the type of data thats in my csv file.

3

You can apply the principles in Serial Input Basics - updated to read from file; there is basically no difference between reading from serial and reading from file.

The example with '\n' as the endmarker will probably do; you might need to adjust a little for the different options for end-of-line in the different operating systems

DOS/Windows: '\r' followed by '\n
Linux/Unix: '\n'
Mac: '\r'

Do your CAN messages need to be terminated with one of the above?

4

however, id like to replace the stuff in parentheses ... with a single variable that is the line that is read from the csv file on the SD card.

Well, pigs would like to fly, too. Gravity sucks, though. So do your unrealistic expectations.

Now, you could read a line from the file, parse the line to get the 4 tokens, and then convert some of the tokens to values, and pass the values to the method.

If what you REALLY meant was that you want to read the data that goes in the array from the file, that is a completely different story.

5

sterretje:
You can apply the principles in Serial Input Basics - updated to read from file; there is basically no difference between reading from serial and reading from file.

The example with '\n' as the endmarker will probably do; you might need to adjust a little for the different options for end-of-line in the different operating systems

DOS/Windows: '\r' followed by '\n
Linux/Unix: '\n'
Mac: '\r'

Do your CAN messages need to be terminated with one of the above?

When viewing the file in notepad++ and showing all characters it does have a /r/n at the end of each line so I do need to use the dos/windows endmarkers.

6

PaulS:
Well, pigs would like to fly, too. Gravity sucks, though. So do your unrealistic expectations.

Now, you could read a line from the file, parse the line to get the 4 tokens, and then convert some of the tokens to values, and pass the values to the method.

If what you REALLY meant was that you want to read the data that goes in the array from the file, that is a completely different story.

Ok so I can't do that then. My ignorance shows.

Now, you could read a line from the file, parse the line to get the 4 tokens, and then convert some of the tokens to values, and pass the values to the method.

I believe this is what I want to do.

7

reliant_turbo:
When viewing the file in notepad++ and showing all characters it does have a /r/n at the end of each line so I do need to use the dos/windows endmarkers.

You can't easily use the combination of both. But you can read up to '\n' and discard the '\r'.

8

I believe this is what I want to do.

Why don't you show us what the file on the SD card is going to contain. I can't imagine that it makes sense to put the name of an array in the file.

9

sterretje:
You can't easily use the combination of both. But you can read up to '\n' and discard the '\r'.

Ok I'll try to implement that. Or I can find/replace the \r character in notepad++ to delete them all.

10

PaulS:
Why don't you show us what the file on the SD card is going to contain. I can't imagine that it makes sense to put the name of an array in the file.

I'm sure I'm not communicating perfectly what I want to have happen, due to my lack of knowing the correct jargon to use. I logged the file via the can bus shield, and I just wanted to reply back the data the same way it came in after some manipulation of course. I was logging both send and receive for certain values, I'm going to sort the file and get rid of the send commands and just relay onto the can bus the ones I received.

The file has typically around 30,000 logged values or more and is a 700 kilobyte text file.

My second Post in this thread is an excerpt of the file. I can do one up and attach it to this thread so it has the actual data that's going to be in it.

11

My second Post in this thread is an excerpt of the file. I can do one up and attach it to this thread so it has the actual data that's going to be in it.

You said you want to read a record, and pass that as an argument to CAN.sendMsgBuf(), so what would

CAN.sendMsgBuf(7DF,2,1,D,0,0,0,0,0,);

be expected to do?

As I said, reading the data from the file, parsing it, putting the values created from the tokens in an array, passing the array (and it's length) to CAN.sendMsgBuf() does make sense.

12

PaulS:
You said you want to read a record, and pass that as an argument to CAN.sendMsgBuf(), so what would

CAN.sendMsgBuf(7DF,2,1,D,0,0,0,0,0,);

be expected to do?

As I said, reading the data from the file, parsing it, putting the values created from the tokens in an array, passing the array (and it's length) to CAN.sendMsgBuf() does make sense.

Yes that's exactly what I want to do..... Plus add a counter so I can select how many lines to read, parse and send and then pause(1000) so it sends a chunk of data once every second.

Thanks,

Brian

13

Yes that's exactly what I want to do.

So, what part are you having trouble with? Robin2 has a great tutorial on reading Serial data.
http://forum.arduino.cc/index.php?topic=396450.0

While the thread is specifically targeting the Serial class, it really applies to any class that derives from Stream, as Serial, EthernetClient, File, and a host of other classes do.

Collecting the data in an array until the end of packet marker arrives (the carriage return or line feed) is very simple.

Using strtok() to tokenize the array is pretty simple (Robin2 shows how). Converting a token to a value is not difficult (atoi(), atof(), strtoul(), etc. do this).

Storing the values in an array, and incrementing the index, is not difficult. When there are no more tokens, parsing is complete and the index tells you how many elements are in the array.

14

I think strtoul is rewuired in this case as the input is in hexadecimal representation.

15

PaulS:
So, what part are you having trouble with? Robin2 has a great tutorial on reading Serial data.
http://forum.arduino.cc/index.php?topic=396450.0

reading comprehension and following code and ADD lol. (true)

and im an off-the-charts extrovert that has to process everything verbally..... so my wife got an earful as i took some time and sat down and started working my way down through the code and looking up things i didnt know.... all processed verbally.

I now have a text document with all three code snippets that i need pasted into it.

SD card read stuff

Robin2 Example 5

send to CAN code

im not really good with code... took one C++ class years ago in college and barely passed. not my strong suit. but ill keep plugging away and try not to pester you guys to death.

Thanks

Brian

16

OK, this should get you started

#include <SD.h>

void setup()
{
  Serial.begin(115200);
  
  for you to do, setup the card
  ...
  ...
}

void loop()
{
  // buffer to store one line
  // adjust size to needs (I counted around 30 characters per line
  char buffer[64];
  // where to store a character that was read
  int index = 0;
  // file pointer variable
  File fp = SD.open("yourdatafile.txt");
  // if opening failed
  if (fp == NULL)
  {
    // inform user
    Serial.println("Error opening file");
    // wait forever
    for (;;);
  }

  // read as long as there are characters in the file
  while (fp.available() > 0)
  {
    // read character and store in buffer
    buffer[index] = fp.read();
    // if newline or carriagereturn encountered
    if (buffer[index] == '\n' || buffer[index] == '\r')
    {
      // replace newline or carriagereturn by string terminator character so we have a proper C-style string
      buffer[index] = '\0';
      // reset index so next line will fill buffer from beginning
      index = 0;

      // if there is actual text
      if (strlen(buffer) != 0)
      {
        // parse and send the data
        parseAndSend(buffer);
      }
    }
    else
    {
      index++;
    }
  } // end_of_while

  // close the file
  fp.close();
}

void parseAndSend(char *text)
{
  Serial.print("line: '");
  Serial.print(text);
  Serial.println("'");

  for you to implement further
  ...
  ...
}

Note that the code assumes an empty line at the end of the file! The code checks for both '\r' and '\n'; as a result a sequence at the end of the line "\r\n" will result in one line of text and one empty line (that will be ignored).

The next steps for you
1)
setup the SD card in setup() and test if the complete lines are displayed in serial monitor
2)
implement the parsing and check if it works correctly
3)
implement data sending

Code compiles, but not tested

17

and im an off-the-charts extrovert that has to process everything verbally..... so my wife got an earful

One of the suggestions I've seen for understanding code is to explain the code to a teddy bear. Maybe your wife needs a break. Get a teddy bear.

At least, when you throw the teddy bear across the room because it didn't spot your mistake, it won't complain.

18

thanks guys.... still working on it.

things are busy (2 small kids) but i got a few minutes and set up the card.

im guessing in the sections where the SD card is being used, chip select needs to be set to pin 4?

and in the sections where the CAN shield is being used, the chip select needs to be set to pin 9?

i dont think i can just declare them in the setup at the top without calling the chip select for each shield?

thanks

Brian

19

ok i think i have the card setup correctly.

gotta try to compile and then upload the code to my other arduino and test it.

#include <SD.h>

void setup()
{
  Serial.begin(115200);

const int cs = 4;  

  Serial.print("Initializing card...");
  // make sure that the default chip select pin is declared OUTPUT
  
  pinMode(10, OUTPUT);
  
  // see if the card is present
  if (!SD.begin(cs)) 
  {
    Serial.println("Card failed to initialize, or not present");
  
    return;
  }
  Serial.println("card initialized.");
}

void loop()
{
  // buffer to store one line
  // adjust size to needs (I counted around 30 characters per line
  char buffer[64];
  // where to store a character that was read
  int index = 0;
  // file pointer variable
  File fp = SD.open("yourdatafile.txt");
  // if opening failed
  if (fp == NULL)
  {
    // inform user
    Serial.println("Error opening file");
    // wait forever
    for (;;);
  }

  // read as long as there are characters in the file
  while (fp.available() > 0)
  {
    // read character and store in buffer
    buffer[index] = fp.read();
    // if newline or carriagereturn encountered
    if (buffer[index] == '\n' || buffer[index] == '\r')
    {
      // replace newline or carriagereturn by string terminator character so we have a proper C-style string
      buffer[index] = '\0';
      // reset index so next line will fill buffer from beginning
      index = 0;

      // if there is actual text
      if (strlen(buffer) != 0)
      {
        // parse and send the data
        parseAndSend(buffer);
      }
    }
    else
    {
      index++;
    }
  } // end_of_while

  // close the file
  fp.close();
}

void parseAndSend(char *text)
{
  Serial.print("line: '");
  Serial.print(text);
  Serial.println("'");

  for you to implement further
  ...
  ...
}
20 
  // make sure that the default chip select pin is declared OUTPUT
 
  pinMode(10, OUTPUT);

That is NOT what that code does.

  File fp = SD.open("yourdatafile.txt");

That is not a valid 8.3 file name.