I have an Arduino Uno with 2 AS5600 hall effect angle sensors. I stream the data of the Arduino per serial to my Mac, with a frequency of ±100 Hz. This includes the millis() timestamp, rawangle1, rawangle2. With some tweaking the consistency of the frequency can still be improved.
Then I read the data with Python and pyserial, process the raw values to angles, radians and based on these, calculate the x,y,z coordinates, before saving it to a .csv file.
What I want to do:
have the Arduino nearly constantly streaming data
visualise this data stream (preferably the processed data, like angles in degrees and x,y,z coordinates)
read the data with python and perform some real time analysis on it. e.g. change of angle above a certain threshold. Then select this data sequence and save it.
My questions are:
How can I do this? Either code examples or preferably some concepts or ideas on how I could do it or what to search for, would be useful.
Is using the Arduino only for streaming the right approach? Or should it already do some of the data processing, which would then result in larger amounts of data to transfer via serial. I also have an ESP32 which I could use.
For the visualisation part, I tried matplotlib already, but this is definitely too slow. I also used threading to plot and save small sequences of data simultaneously, but this wasn't working at all, and e.g. displayed/saved only 4 seconds of data during a 10 second recording period.
I will probably try Dear Pygui for the visualisation part, but then I still don't know how to do the live analysis. Or should I switch to something web/browser based, which I would then use offline?!
not done this with Python but have done similar data acquisition and display using C++, C# and Java - points to consider
do youtransfer the information as text or in binary?
you can usually increase the volume of data transferred per second by using binary ???
rather then using serial you could increase the volume/sec using Ethernet or WiFi TCP/IP protocol???
you then have to do something with the received data
e.g. if you are receiving 10000 data points/second do you need to plot all 10000points?
can you in practice plot 10000 points/sec?
this depends on your processor and graphics display speed - possibly only plot one in every 10 data values received?
if you store the data received in arrays or on disk you can later zoom in on points of interest
I think you need to experiment
@J-M-L Thank you, this was an interesting read (I know french), I will definitely use some of your code snippets. However, I tried threading already before, together with queues and found it to be performance limiting. Probably because I did not set it to 'daemon'. Furthermore, I was not checking if data was available, but just simply using ser.readline() in a while loop.
@horace Currently I transfer the data as text. But its limiting and I actually want to switch to binary in future to increase the throughput, but also the reliability of the frequency. If using the ESP32 I could actually use wifi... or a higher baudrate.
As it are for now just 2x100 points/sec I think the live plotting should be possible. I do not want to store all of the streamed data, but rather live check if anything happens (e.g. movement above a certain threshold) and then pick this data sequence (length ~5s) out and save it.
So I will probably have to save a certain amount of data in a buffer?? And have another task analyse this data in the buffer??
@MrMark thanks, this is very interesting, I will try this one too!
if you use an ESP32 at 115200baud you can transfer approximatly 11000 characters/sec
this would allow 11000/200 = 55 characters/data point
with a modern PC you should be able to plot in real time 200data point/sec
does Python support multitreaded code?
@horace yes Python supports threading and asyncio. I have a small logger that uses a list as a buffer, actually it is a list of lists so that at the end of logging I write the list of lists to a csv file. I capture the data as ASCII characters in a separate thread and @funky-fresh yes you can analyze the data as it comes in. 6 bytes @ 115200 would be ~500 uS ? so there is possibly time to do a little additional processing, it depends on the packet size and the frequency the packets are transmitted. I think a list can hold 500 million elements, if that is right that is quite a reasonable buffer. I can post a partial script if your interested.