ESP32 xQueue, how to copy struct (memory) efficiently ?

[Deva_Rishi]

@kenb4

But are you doing that anyway? Do you need to access the data again after you want it written to the SD? In fact, you can read it again -- access to memory is serialized and therefore safe

The problem is that i want to copy (LED) pixeldata to the SD. This pixeldata is available the moment that i start processing it for transmitting it to the addressable LEDs. The transmission itself happens using I2S and so the main pixelbuffer would be available straightaway to start receiving new data into, the moment the actual transmission starts. If i am to write this main pixelbuffer also onto an SD-card, i can not start writing to that buffer until the whole buffer has been written. The SD-card is significantly slower than doing memcpy(). The whole pixelbuffer may be as big as 2720 pixel = 8160 bytes. I could just copy the whole thing + the timecode which also needs to be included, but with the double buffer at the receiving end (and at the transmitting end due to the timecode) the waste of memory appears to be just a bit to much for the program i have at the moment. Memory is a fairly scarce resourse, partly due to the use of I2S for the LED signal. Hence the copying of 512 byte blocks.
I have been given to understand that the writing to the SD card happens also in that block size. (more or less)
So anyway the data needs to be copied so the buffer can be written to again (which is the first process that happens after it has been processed for transmission) The heap is already being used dynamically by the webserver and some other small things and is managed manually, so i don't really want to start declaring blocks of data onto it and freeing them up again. the xQueue will declare it's own dedicated area. I am hoping that i will have enough memory left over to declare a 16-block Queue, which would be a complete frame, but even if i end up having only half of that, it still means that after half of the writing to the SD-card is complete, the main processor is free to start receiving new data.

@kenb4

Having separate queues for the filename and file data seem problematic. The ideal case would have a single structure, for which you can then allocate some variable amount of space -- as long as you know how much in advance

Yes it does, but having a 30 byte section of a structure in a 16 block Queue ends up being 480 bytes.

Having just the actual data and it's size is intended for efficiency reasons at the receiving end. The file is created, it has a small header written to it, this is all not very time sensitive. Then the blocks start arriving and they should just be written as is. Once the 'recording' is complete, timing is no longer sensitive. No more block will be added to the Queue, the process of receiving from the Queue doesn't need to happen, and the filename can be cleared. Any blocks that get written to the Queue by mistake will be written and discarded (this really should not be possible, but i may as well have that part of the process still running, it doesn't take much)

@kenb4

The filename can be empty for "same file"; maybe have a magic value for "done with the current file". You can then push those pointers onto the queue, and the next task can access the data. If the SD writer is the last thing, it can free the block. If it needs to stick around and needs to be managed by the main task (or another cleanup task), then maybe there is a bool or bit in the struct to indicate that the write is done.

For a more universal solution i would go that way, since that would allow for writing to multiple files as well and actually would be a really nice library to have available on an ESP32. It could probably be even more efficient if it would be a part of the SD library.

@kenb4

The main setup+loop task runs on the "APP" core (core #1). The "PRO"-for-protocol core, core #0, handles the WiFi and Bluetooth (and more). Those high-priority tasks running on the "other" core are grabbing bytes out the air. If you're going to run the SD writer there as well, let us know how it works out if you run it at the the same high priority, where it will be time-sliced with the normal protocol stuff, to make those bytes available on the APP core; or at a lower priority.

Since i started out on the main project on a devBoard and had that selected as my board, i had actually missed the setting, and both the APP as the Events were running on core 1 until i changed that. It didn't actually cause to much issue until i started playback. Reading from the SD-card is also not that fast. That said, BT is not used at all, and WiFi is only used for the webserver and the main thing interfering is actually WiFi maintaing it's connections to devices, the UDP packages are coming in over an Ethernet connection through an SPI bus and are happening on the main core. It is just to much data for WiFi to do this reliably. I am intending to fiddle with the priority a bit, i guess it doesn't need to be all that high, it is an unknown as yet.

@qubits-us

fun stuff.. ~q

Yeah the thoughts about it also tickle me. It would be nice to create a universal solution, though i would avoid declaring on the heap to avoid memory fragmentation.
If one is to use the heap to transfer the date, then it should be done with a fixed size block at compile time, and never really be freed again. Otherwise end users may not safely use the heap anymore at all.