Sorry in advance, the answer shouldn’t be very complicated.
*For this project I currently have an arduino uno, but I am willing to buy an external adc or another microcontroller such as a Teensy or some other arduino models (I would like to stick with arduino ide if possible). I know there are multiple discussions of this topic that break down the problem in graphic detail, so I don’t need complicated answers.
My project has me reading an analog signal from a transducer and eventually sending it to my pc which will then take that data and turn it into a graph in matlab. Sending data to my pc can be done in whatever convenient way works (ie. my data can be sent whenever my arduino’s memory is mostly filled and my graph can update whenever that happens). I need a data point every 100us (10,000 per second) on the graph at the bare minimum, so I imagine I need a sample rate on my adc of greater than 20kHz. I know it’s a lot of data, but I would rather get a system that works before I worry about the memory of the pc. It may be important to note that I have multiple arduino uno R4s I can use, so no arduino needs to use too much memory or cpu. The following will outline what the process should do:
Send analog signal through adc with sampling rate enough to give me at least 10,000 data points per second to send to the computer.
Read that signal in a microcontroller (optional) and send that data to my pc through usb or some other method (required). The reason I want to use an arduino or other microcontroller is because of how easy it is to send and read data with it along with it having all of the required components (such as an adc) built in. If I can use an external adc in this process, that could also work. This step is generally the main issue because reading and sending data take a lot of time, more than rate I need, so I must optimize this. My solutions are further down this post.
Take that data on my pc and plot it (preferably in matlab but if I have to use python I can manage for now). I’m not so worried about this step.
How I was planning on implementing it:
Set the analogRead function on arduino or other microcontroller to read data faster much than its default by changing the clock settings (should be relatively easy)
Store that data in the arduino’s memory and let it fill mostly up before sending it serially with dma or some interrupt. If I’m not using dma, I plan to use spi and its interrupt in a similar way. (Seems reasonable but I’m only really familiar with dma and uart)
read the data (don’t really care how right now)
This seems like it should work, but after reading around, it seems like it may make more sense to buy a teensy for a more powerful clock and analog read for better results or use an external adc and send the data from that directly into the arduino serial monitor tx pin which I could maybe get the data from on my pc (?) or use digital read pin and store the data and use dma or some other method to send it after a while. Do any of these sound like a good solution, or am I trying to solve this in the wrong way?
I’m decently flexible with how I get from the analog signal to the graph but I would like to keep the general process the same unless there are way better solutions. Whatever I do shouldn’t be very expensive - ideally less than $60, but if buying something expensive is the best solution, it will still be considered.
How are you going to transmit/process 10k samples per second? Have you calculated data rates, based on the precision that you need? Is the data stream continuous or in bursts?
If you can accept 8 bit data resolution for the ADC, you cut your data streaming requirements in half. Other considerations:
do you need any form of synchronization, or will a continuous stream be acceptable?
is a monotonic sample rate essential? (transmitting serial means interrupts will, well, interrupt, and thereby delay your data acquisition)
a Timer generating interrupts may be a better way to trigger your acquisition, but IIRC, the ADC process is ponderous, and your timing requirements are...precise.
other data transmission methods should be considered, other than serial.
Others may differ but I consider your project to be beyond the reach of an Uno, unless(possibly) you can enlist someone to tight-code the minimum using assembly language, or perhaps code it in as efficient a method as is possible in C. Really, you're pushing the edge here.
Yeah I figured the graphing may have some issues but it’s not something I’m concerned with at the moment. The main problem I want to tackle is getting the data onto my computer. My group has a graphtec data logger but it doesn’t sample fast enough for what we want I believe. Sorry for my lack of info, this stuff is kinda new to me and I don’t know too much about the project.
Any way I can send the data would work. If sending it continuously works, I can go with that, but I assume sending it in bursts would be better so that I’m not wasting time sending data whenever I get a datapoint.
How are you going to transmit/process 10k samples per second?
Again, anyway that works is good enough for me right now, but my plan was to have a baud rate of 115200 and use uart and dma to send the data in bursts because I am more familiar with uart, but I can use I2C or SPI and similar methods. When I receive data onto my computer I would probably store it in a file like a csv if I can. By sending it in bursts, I would get roughly evenly spaced samples in time except during the point in time when I have to send the burst of data, so when I go back to use the data, I would need to account for the time shift. How I would actually do that on my computer is a problem that I don't need to solve right now
Have you calculated data rates, based on the precision that you need?
I'm quite new to this so sorry if my conceptual understanding is wrong. I imagine since I need 10k data points, my sampling rate must be at least greater than 20kHz due to the nyquist theorem but it will likely be greater than that for oversampling and because it is probably not possible to set the rate to that exact value on the microcontroller.
Thanks for sending this. Based on what I see on there, that would work for what I'm doing, however it is a bit more expensive than what I want. Ideally, the most money I would like spend is probably around $30-$60 to get a more powerful microcontroller. Ideas like this are great though, if I end up dropping this project, stuff like this is good to know . If my group was to buy something for around $200 , we would need more of a reason to buy this than to have a fast sample rate for 1 part of 1 project.
But if that part of the project causes the bottleneck then your whole project will not work.
It's your choice.
You can up the sample rate on the Arduino Uno to about 40KHz by changing the A/D prescaller register.
But you still have to get it out of your Arduino and into your PC. Serial transfer on the PC is not what it should be.
Others may differ but I consider your project to be beyond the reach of an Uno, unless(possibly) you can enlist someone to tight-code the minimum using assembly language, or perhaps code it in as efficient a method as is possible in C. Really, you're pushing the edge here.
Thank you for your opinion. At the very least it seems like I would end up using a different micro controller such as a Teeny for this project or some other system.
other data transmission methods should be considered, other than serial.
Looking around, that seems to be the general consensus. Outside of maybe using the wifi feature of arduino uno R4, I don't know of any way to send data without serial communication for the arduino uno so it seems like different solution is warranted.
do you need any form of synchronization, or will a continuous stream be acceptable?
Sorry if I misunderstand, but I don't care how the data eventually gets sent. I'm happy to use whatever method I can get to work. To my understanding, synchronous communication is looking like the best option as it is designed to work for high speeds with lots of data. I was planning on using SPI communication assuming it works for this. It doesn't matter to me how fast my data can be received and/or graphed on my computer as long as I end up with the amount of data I need.
is a monotonic sample rate essential? (transmitting serial means interrupts will, well, interrupt, and thereby delay your data acquisition)
At the end of the day, I need to be able to analyze this data in voltage vs time. I don't need a monotonic sample rate, but I need to account for that changing sample rate. Assuming that the time between each data point remains constant for most of the time, I should be able to calculate roughly how far apart each data point is in time and graph it. If that data gets shifted in time at some point, (ie. I take some time to send my data over to my pc), I should be able to account for it as long as I know how much time it took between the previous data point and my first data point after sending data. I imagine I could calculate or test how long that takes, so it doesn't seem like a major constraint. I would get a hole in my data, but this should be acceptable if I am getting enough data.
a Timer generating interrupts may be a better way to trigger your acquisition, but IIRC, the ADC process is ponderous, and your timing requirements are...precise.
Probably. I was also considering using an external ADC and either sending the data straight to my computer, or through the microcontroller via digitalRead() and/or the tx/rx pins.
But if that part of the project causes the bottleneck then your whole project will not work.
It's your choice.
Thanks for your concern but for the time being, this shouldn't hold up any part of the project. We can already collect the data and graph it using a graphtec data logger, just not to the specifications we eventually need. There are many other parts of this project and it isn't at the phase where finding a solution for this is urgent. At the end of the day, if we need to buy a $200+ usb daq, I'm sure we will end up doing so.
Thanks! My data points are a 2 byte int from analogRead(), so I can only send 5,760 data points per second which is obviously too slow if I need 10,000 points per second. Even if cpu time wasn't a factor it would take 1.7361s to send the 10,000 points. Since cpu is a factor, I can't send the data point every time I read it otherwise I will slow down the rate at which I can read my analog signal. If I store those data points in memory with an array and/or a dynamic array, I can retain most of the speed of my analogRead however my memory will fill up quickly, so I need to send that data out when my memory is filled. The issue is that if I store 10,000 points and I'm storing 10,000 points per second and then send them out - it would take about 1.7361 seconds to send the data in memory but only 1 second to store that memory meaning that I can't send data faster than the rate at which my memory fills up. This is all to say that this method will not work (Using uart at least).
If you do that you are in an even bigger mess that you are now. Plus the Uno only has 2000 bytes of memory for everything. If sending them out one sample at a time is too slow, then waiting until you have gathered lots of data points is only going to be slower.
It seems to me that you have not grasped the fundamentals of your problem.
You're not limited to 115200 baud; I regularly use 250KB between an Arduino and the computer, and it's always reliable. At that speed, your data rate could theoretically be as much as 25K bytes/second. You also don't need to use an interrupt to send every byte of data, though if you use Serial.begin() and Serial.print(), then there will be interrupts. It's pretty easy to control the serial port without those routines, and if time is of the essence(!!) I'd definitely suggest it.
I can imagine a program with a 10KHz timer interrupt, where if you're sending a single byte per sample, then what the timer does is read the A/D result and drop it into the serial buffer, then fire the A/D again. You could be certain that in 100usec, the result of the conversion will be available, so there's no need to check whether it's finished or have an interrupt at the end of the conversion. If you need to send 2 bytes per sample, it's more complicated. Maybe have a 20KHz interrupt, and send the high byte and the low byte alternately, but only fire the A/D on half the cycles. Or only interrupt at 10KHz, but wait in the interrupt for the first byte sent to complete, and then send the second one. But a problem: how does the receiving end of the system know whether it's getting the high byte or the low byte of a sample?
Whether the computer can absorb data at the rate you're talking about is another issue, but it's not an Arduino problem.
You can use the rp2040 to send data to the pc at rates of 1.4M bits a second by abusing the fact it can act as a usb sound card. Then open up audacity and you will have a real time plot of whatever data you chose to stuff the buffers with. The code to do this is provided with the official mbed core and is a wonderful feature
It seems to me that you have not grasped the fundamentals of your problem.
I explained in my post that it shouldn’t work. I know that doing it this way should not work. I also have to balance sampling/acquiring my data with sending it. My goal is to find potential work arounds to this and which one is the best.
My fault. Analog read has a resolution of 10bits. I assumed that data would be sent in 1 byte chunks so all data of type ‘int’ would have 2 bytes that need to be sent even if the resolution is only 10bits. I figured the data can’t be sent in 10 bit chunks but it would be good to know if it can.
Although, I guess I could try and cut my resolution/data to 8 bits and send only 1 byte.
Interesting. I guess I just assumed that I should use 115200 baud based on what I’ve read.
Seems like a problem that could theoretically be solved by syncing up the sender and receiver. If the data sent will always be a high byte followed by a low byte and we start receiving at some random point in time, we may be able to check if we started receiving on a high byte. If the resolution of our data is 10 bits, then the low byte will have 6 bits that should always be a 0. If we receive a byte and any of those 6 bits were a 1, then we know we are on a high byte and the next byte will be a low byte. We would then be synced. There are some pitfalls in this but it is an interesting puzzle. This was a very insightful post, thank you.