I'm reading values from a binary file on an SD card. How do I convert what was written as a unsigned int to what I need it within my sketch? For example, the value written to the file is '300'. When I read that back in, I get:
44
1
0
0
I know that is the correct information, but what I don't know is how to change that back into '300'. The math for that is
44 * 256^0
1 * 256^1 +
0 * 256^2 +
0 * 256^3 +
= 300
Isn't there an easier way?
Just a small example:
union x {
struct {
byte bts[4];
} buf;
long nummer;
} field;
void setup()
{
char str[50];
// Add your initialization code here
field.buf.bts[0] = 44;
field.buf.bts[1] = 1;
field.buf.bts[2] = 0;
field.buf.bts[3] = 0;
//
// make a connection to the terminal
//
Serial.begin(9600);
sprintf(str, "Long number = %5ld", field.nummer);
Serial.write(str);
}
// The loop function is called in an endless loop
void loop()
{
//Add your repeated code here
}
bts is your buffer where you put the bytes in
nummer is a long. But if you know how the data is ordered you could make a second struct containing all the fields in the right order and then just read the buffer and all the fields are filled.
They're being read in one by one. The file header is as follows:
unsigned char version
unsigned int cols
unsigned int rows
char r
char g
char b
char r
char g
char b
ad infinitum (or till EOF is reached)
So the first thing I do is grab the version, which is a single byte. cols and rows however are not. Then r, g, and b are all one byte each. So when I read the file back in, line by line, I see this:
1 // version
44 // cols
1
0
0
48 // rows
0
0
0
148 // r
100 // g
127 // b
144 // r
106 // g
139 // b
As you can see, both cols and rows take up four lines, or four file.read() to the SD card where this file is on. At least till I can figure out how to pull in multiple lines with one read ...
You know the format of the first piece of the file so you can read that at once (9 bytes) then you know the columns and rows
The format of a row is apparently a number of rgb sets.
next question is how you want to process the data. Keep in mind dat you might be limited by the amount of data storage. The UNO has a max of 1000 bytes.
Is it row by row? of load the whole set at once. Than you might have a problem (300 x 300 bytes = 9000 bytes).
structs /unions could be:
struct header {
char version;
long columns;
long rows;
};
struct rgbset {
char r;
char g;
char b;
};
union {
char ReadBuffer[9];
header hdr;
} Header;
union {
char ReadBuffer[3];
rgbset rgb;
} RGB;
What a weird way of saving data!
Both solutions will work. I am not so fond of shifting bits around as it is unclear what the intention is. But that is personal. And as the format is a known thing, why not just describe it.
As for the saving of data....Once upon a time many years ago (70'ies) we always stored data in a binary format. ]
we still do... or you'd not be able to save anything.....
One of the things I like about C is the fact that if you have your data in memory somewhere in the format you want, you can typecast it and use it as is. Or, you can typecast it while you're getting it in the format you want, and then use it. The compiler makes it very efficient and you don't need extra code to convert it from format to format.
unsigned char version;
unsigned long cols;
unsigned long rows;
...
version = foo.read();
for (byte i = 0; i < 4; ++i)
((byte *)&cols)[i] = foo.read();
for (byte i = 0; i < 4; ++i)
((byte *)&rows)[i] = foo.read();
nicoverduin:
You know the format of the first piece of the file so you can read that at once (9 bytes) then you know the columns and rows
The format of a row is apparently a number of rgb sets.
Ok cool, I'll have to do some digging to get to that.
nicoverduin:
next question is how you want to process the data. Keep in mind dat you might be limited by the amount of data storage. The UNO has a max of 1000 bytes.Is it row by row? of load the whole set at once. Than you might have a problem (300 x 300 bytes = 9000 bytes).
Well, the r, g, b values are read in and passed directly to another library, CRGB(r, g, b) and it does the rest. However, I need to push 48 sets of rgb values to the library before it does an update. Think of a shift register that needs a certain number of bits before latching. Same principle.
I'm not limited to the UNO by the way. I can pick something else that has more memory to process this. I am actually looking at the AT90USB1286 AVR for the USB connectivity. It has 4K of SRAM. I don't need to read the whole image at ones, I just need to be able to hold a few rows in memory and do a constant running loop. Read in, say 10 sets of 48 rgb values, start pushing them out to the LED library one by one and continue reading the SD card to maintain a sort of "buffer". So push set #1 of 48, read set #11 from SD, push set #2 out, read set #12, etc., etc. At least, that's how it's working in my head ... not sure how that's going to work in reality.
Riva:
What arduino are you writing this on as unsigned int is normally 2 bytes. On the Duo it may be 4 bytes.
The data is coming from a computer that's processing the images. I've changed them to unsigned short ints which are 2 bytes (on the computer.) That's really all I need for space. I'm reading the data in on an AVR though.
cjdelphi:
What a weird way of saving data!
Okay, with a statement like that, how would you store the data knowing that you need individual values of r, g, and b, with absolute minimum processing on the reader (Arduino) part? I can't think of a quicker way to read the data and shove it right back out as-is. I don't really care about the header to be honest, except that I do need the cols and rows values.
That other library is of course also on the Arduino (or other board?)
Well that will be a challenge in such:
The real issue would be that you would like to use a multi tasking concept. But these boards do not, to my knowledge, support that. So it will be processing either one or the other.
loop
read 48 sets
send them to the library
process library
until an end of file is reached.
An alternative could be to use 2 Arduino's and create a form of parallel processing in which one arduino reads the SD card and puts it in a buffer. Once it has a minimum amount of rows in it, it starts pumping it (through SPI) to the other Arduino. Once that has received it's 48 sets, it can start processing. While that is going on, the first arduino starts reading the next data until the buffer is full (or earlier pending the time the library needs to process the data). When the library has finished its work it tells the first Arduino to send some more data.
Just set up a communication dialog between both boards.
This concept is basically the same as multi threading where the link between threads is a piece of common memory (queue). As this is not possible you could more or less simulate this using the SPI interface. So they are not fully independent but it would be a nice concept.
Somewhere in my ancient archives I recal having some documentation on my first OS/2 multi thread program written in C. But I am sure there is something to find on the web.
nicoverduin:
That other library is of course also on the Arduino (or other board?)
Same board. It's just an LED library, FastSPI to be specific. It passes values to an LED array prior to updating the string. So for the POV system, I'm counting columns and rows and sending each column to the string. On an image that has 200 columns and 48 rows (permanent), I do:
MAX_COLS = 200;
for (int col = 0; col < MAX_COLS; cols ++) {
for (int px = 0; px < NUM_LEDS; px++) { // NUM_LEDS is also 48, which is why all images are capped at 48 rows high
// get r,g,b values from wherever they're coming
leds[px] = CRGB(r, g, b);
}
LEDS.show(); // update string
_delay_us(STRETCH); // this varies for each image
}
nicoverduin:
Well that will be a challenge in such:
- Reading 48 sets should not be a problem. If the file by itself is just a long stream of rgb sets with no CR or LF after each row but just a big blob. Based on the row and column definition, you know how many there will be.
The real issue would be that you would like to use a multi tasking concept. But these boards do not, to my knowledge, support that. So it will be processing either one or the other.
Yeah, I know, it's a bottle neck. This may end up being a futile attempt but I would like to at least figure it out and see what can or can't be done. Right now I'm storing all the images in PROGMEM, and depending on the size of the image (as far as color information) I can upwards of 10-30 images (lower numbers on an UNO, higher numbers on a 2560.) However, I'm also only storing 64 levels for each color, packed down into 0b0rrggbb0 and stored. I can read PROGMEM on the fly as the POV is working:
unsigned char cur = (uint32_t)(pgm_read_byte_near(&image1[i + j * NUM_LEDS]));
unsigned char r = cur & Rmask;
unsigned char g = (cur & Gmask) << 2;
unsigned char b = (cur & Bmask) << 4;
leds1[i] = CRGB(r, g, b);
The problem is, being a performer (and also having a small group of performers), if I need to update the images during a particularly long day of performances, I would have to lug around my laptop and re-program the board every time. So that's why I would like to convert it to an SD system where the program remains the same and won't require redoing every time, but I can swap out SD cards with new images. But it means having to read the card and pushing data out. Oh, and I also want to go to full color.
nicoverduin:
An alternative could be to use 2 Arduino's and create a form of parallel processing in which one arduino reads the SD card and puts it in a buffer. Once it has a minimum amount of rows in it, it starts pumping it (through SPI) to the other Arduino. Once that has received it's 48 sets, it can start processing. While that is going on, the first arduino starts reading the next data until the buffer is full (or earlier pending the time the library needs to process the data). When the library has finished its work it tells the first Arduino to send some more data.
I thought about this as well, but I don't know if this will be any faster. I still need to send data from one to the other and when push comes to shove, I don't know if it's any faster than reading an open stream from an SD card.
This entire project intrigues me. What kind of hardware are you using for the POV? Do you have a link to a picture or video of one of your performances?
TanHadron:
What kind of hardware are you using for the POV?
At the moment, 328p AVR driving four custom strings of 48 RGB LEDs each, inside of a polycarbonate tube.
TanHadron:
Do you have a link to a picture or video of one of your performances?
Unfortunately no. These are kids who perform on a word of mouth basis. Not a professional group at all.
Haven't worked with SD card so I cannot help you on that. May as well just try it out.
Are you using poi or batons/staves or hoops?
How are you triggering transitions between your images? Music timing?
It looks like some effects work better than others. Which effects do your audiences seem to like the best?
KirAsh4:
nicoverduin:
That other library is of course also on the Arduino (or other board?)Same board. It's just an LED library, FastSPI to be specific. It passes values to an LED array prior to updating the string. So for the POV system, I'm counting columns and rows and sending each column to the string. On an image that has 200 columns and 48 rows (permanent), I do:
MAX_COLS = 200;
for (int col = 0; col < MAX_COLS; cols ++) {
for (int px = 0; px < NUM_LEDS; px++) { // NUM_LEDS is also 48, which is why all images are capped at 48 rows high
// get r,g,b values from wherever they're coming
leds[px] = CRGB(r, g, b);
}
LEDS.show(); // update string
_delay_us(STRETCH); // this varies for each image
}
> nicoverduin: > Well that will be a challenge in such: > - Reading 48 sets should not be a problem. If the file by itself is just a long stream of rgb sets with no CR or LF after each row but just a big blob. Based on the row and column definition, you know how many there will be. > > The real issue would be that you would like to use a multi tasking concept. But these boards do not, to my knowledge, support that. So it will be processing either one or the other. Yeah, I know, it's a bottle neck. This may end up being a futile attempt but I would like to at least figure it out and see what can or can't be done. Right now I'm storing all the images in PROGMEM, and depending on the size of the image (as far as color information) I can upwards of 10-30 images (lower numbers on an UNO, higher numbers on a 2560.) However, I'm also only storing 64 levels for each color, packed down into 0b0rrggbb0 and stored. I can read PROGMEM on the fly as the POV is working:unsigned char cur = (uint32_t)(pgm_read_byte_near(&image1[i + j * NUM_LEDS]));
unsigned char r = cur & Rmask;
unsigned char g = (cur & Gmask) << 2;
unsigned char b = (cur & Bmask) << 4;
leds1[i] = CRGB(r, g, b);The problem is, being a performer (and also having a small group of performers), if I need to update the images during a particularly long day of performances, I would have to lug around my laptop and re-program the board every time. So that's why I would like to convert it to an SD system where the program remains the same and won't require redoing every time, but I can swap out SD cards with new images. But it means having to read the card and pushing data out. Oh, and I also want to go to full color. > nicoverduin: > An alternative could be to use 2 Arduino's and create a form of parallel processing in which one arduino reads the SD card and puts it in a buffer. Once it has a minimum amount of rows in it, it starts pumping it (through SPI) to the other Arduino. Once that has received it's 48 sets, it can start processing. While that is going on, the first arduino starts reading the next data until the buffer is full (or earlier pending the time the library needs to process the data). When the library has finished its work it tells the first Arduino to send some more data. I thought about this as well, but I don't know if this will be any faster. I still need to send data from one to the other and when push comes to shove, I don't know if it's any faster than reading an open stream from an SD card.
Actually, I think the big bottle neck here is the SPI bus. I don't know what or when FastSPI is using the SPI bus, but I would imagine it basically takes over the hardware SPI bus at LED.show(). Pity, because the time when it's shifting out instead of delay()ing you could be loading the next 48 sets of rgb. The only SD shields I know of use the SPI bus (basically because the SD card itself is a SPI device...). It isn't as durable (bent pins to worry about), but I wonder if an I2C EEPROM in a ZIF socket would work. (Do they make 8-pin ZIF sockets?) That way you could be reading something via I2C while FastSPI is chugging out over the SPI bus. Instead of using the delay(), you set a timer in the BlinkWithoutDelay style. (Unless I'm totally misunderstanding what _delay_us(STRETCH); does...)
Back to your original question, I think the already suggested union method is probably the easiest and most self-documenting way to progress. But that is just IMHO, you have several options to choose from.
TanHadron:
Are you using poi or batons/staves or hoops?
45cm long baton swung like a poi.
TanHadron:
How are you triggering transitions between your images? Music timing?
Timing. Basically each image has a repeat value and it repeats till it has reached a specific time, 5 seconds, 15 seconds, 1 second, whatever I set it to.
TanHadron:
It looks like some effects work better than others. Which effects do your audiences seem to like the best?
It really depends on what we're doing. Little kids like to see images of their favorite cartoon, or video game images. Others just like the abstract stuff. When we do an actual music routine, it's purely abstract images, ribbons, celtic designs, stuff that's bright and changes colors constantly. But they're not triggered by music. That's on the long TODO list for another revision down the road.
This might be interesting : Maximum speed that the Arduino can read an SD card - Storage - Arduino Forum