Store/process image on Nano BLE 33 Sense for classification

Projects Programming
1

Hi all, recently received a Nano 33 BLE Sense, a project I am working on is converting one of our Acute Lymphoblastic Leukemia classifiers to be compatible with this board. The images are images of peripheral blood samples and are 100px x 100px, around 21kb. The classifier expects the images to be this size.

I have thought of sending the images (one by one) from another bluetooth device to the board, saving/classifying and then deleting before the next image is sent.

I also though about uploading the image(s) rather than having the be received.

I realize storage capacity is most probably the biggest issue, probably the best way to do this is convert the image to bytes on the sender side, then there is no need to store the image.

What would be the best way to accomplish image classification based on points above, I am assuming the 3rd.

This is the first time I have worked with images on an Arduino so sorry for the noob question and thanks in advance to anyone can help with advice or provide links to documentation / similar projects so I can work it out.

Cheers all.

2

You want to do medical image classification on an Arduino board? I'm not an expert in such image classifications but I would expect the task to be far outside of what the resources of the Arduino provides. Maybe it's that ease as to just find a few purple pixel inside the picture, that may be done but more complex analyses most probably need more then the 256kB of RAM the board's processor provides.

3

If it can do object detection (which it can) and facial recognition (which it can) and voice recognition (which it can) then there will be no issues with classification of peripheral blood samples that I can see. I will find out once I implement the network, I already have the trained and converted TFLite model, I just need to work out best way of getting the test data to the device. I think I will just send the bytes (encoded) via bluetooth.

4

Welcome to the forum.

Here are a few points for thought and some questions.

BLE was not designed for bandwidth but for low energy. Transferring large amounts of data will require time.

  • Why did you choose BLE for the transfer?
  • What is the device that sends the images e.g., another Arduino, phone, PC?
  • Where are those images coming from and how?
  • How far are the devices apart?
  • How much time do you think is required to compute a result? ~ 1s, 1min, 1h ...

That part is unclear. There is no such thing as binary when it comes to images. There are file formats (e.g., jpg, png) and pixel data (e.g., RGB values with 8-bits per channel). It might make life easier if you send pixel data. I suspect that is what your model requires.

If BLE speed is not an issue, you can create a service with a couple of characteristics that will make this comfortable to use. You will likely need to send the pixel data as fragments (a couple of pixels at a time). You can use notifications to tell the central that you have received a fragment and the next one can be sent. The same is true for the processing. The central device can be notified when the results has arrived.

With some more information (see questions above), I may be able to give you some more specific advice.

1 Like
5

Hi Klaus.

We are not sending a lot of data, I mentioned one image at a time, the max sent will be 21kb this can easily be achieved by chunking.

  • Because Nano 33 BLE only supports BLE, I could easily hook up a ESP8266 or 32 but the purpose of this project is to explore the use of BLE and machine learning on an Arduino.
  • This project is part of our hospital automation server/network HIAS Core. We have (or will soon) have IoT agents that communicate with the Bluetooth/BLE protocols. These agents also can receive/send HTTPS requests. Devices on the network will send an image via HTTPS to the Bluetooth/BLE agent which will then forward the request to the devices that can only communicate in Bluetooth or BLE.
  • The agent will be located quite close but not close enough for serial via RX/TX
  • I don't know no the time yet, this is a research project.
  • I meant encoded as bytes, sorry hadnt slept for two days yesterday.

I think what you have suggested is pretty much aligned with what I was almost decided on, thank you for the input :slight_smile:

6

I now have an SD card module which I will use, the first version rather than sending the images over BLE, I will upload the images onto the SD card and have the classifier read the images from the SD card. Let's see how it does, if the results are good I will look at sending the images to the board via BLE and have the classifier process them.

7

Getting there.

arduino-33-ble-sd-card
arduino-33-ble-sd-card3112×2334 1.26 MB

acute-sd-card
acute-sd-card989×836 85.6 KB

(Images are not actually that size in the application they are reduced to 100px x 100px)

8

I think it is bricked lol, uploaded sketch including the model was 99%. After that the device cannot be recognized port switched from 26 to 25 and never flipped back, any suggestions ? It was worth a try, with a bit more resources I reckon it would have worked, however, dead board now :smiley: I have other ideas how it can be used on our network but it really is just a toy AI device, looking forward to future advancements where can load some proper models on it.

EDIT: Got the board back to life, double click rest and uploaded blink, all good, going to try get this working.

10

I know I am talking to myself here but hey ho :smiley: Maybe it would be useful for others in the future, so the issue was I could not find Zero Padding layer in the ops, seeing as now all Tensorflow Github documentation is broken and no longer exists, I decided to use AllOpsResolver (Yes I know I was lazy), this is where the increase was I now found out, if I use MicroMutableOpResolver it cuts the size down to 82%.

I realized AddPad works for zero padding and, AddReshape is flatten and AddFullyConnected is Dense. Now the problem lies in hitting memory limits due to the size of the tensor arena size. I can almost get it, but incrementing enough maxes out memory so far.

So there is the story so far, to be continued :wink:

My model is now matched:

static tflite::MicroMutableOpResolver<8> micro_op_resolver;
micro_op_resolver.AddPad();
micro_op_resolver.AddConv2D();
micro_op_resolver.AddPad();
micro_op_resolver.AddConv2D();
micro_op_resolver.AddMaxPool2D();
micro_op_resolver.AddReshape();
micro_op_resolver.AddFullyConnected();
micro_op_resolver.AddSoftmax();

11

You were right in a way about the ram, it's not the classification that is the first issue though the model itself and the arena size takes up 97% of the ram.

Arena size is too small for all buffers. Needed 2497920 but only 209248 was available

If increase this now it crashes the board. I saw steps in a notebook about additional quantization that helps reduce further but Tensorflow have decided to break all their documentation and the bookmarks I had saved for getting started are all dead.

12

By the way this forum is really anti privacy I am pretty sure it breaks GDPR, you can't edit or delete a post without leaving a history.

13

Found the documentation for the quantization code I saw Post-training quantization  |  TensorFlow Lite The major problem here is the size of the model, if this doesn't reduce the memory usage and model size I think it will be time to admit defeat on this project, the model itself is a relatively small model it can't be physically made smaller unless this optimization works. Literally about 10 lines of code away from having this done, retraining now and using the post training optimizations outlined in that document. Final attempt then will move onto another project.

Full integer quantization should make the model 4 x smaller and 3 times faster, this would make it smaller than the person detector model by almost half, so maybe...

WARNING: Do not use uint8 quantization it is not supported anymore.

14

This is just painful:

Arduino_TensorFlowLite\src\tensorflow\lite\micro\kernels\cmsis-nn\fully_connected.cpp Hybrid models are not supported on TFLite Micro.
Node FULLY_CONNECTED (number 6f) failed to prepare with status 1
AllocateTensors() failed

Hybrid model ? It is a simple keras model. Optimizing solved the issue with the upload though. Size is now 179936

Sketch uses 325120 bytes (33%) of program storage space. Maximum is 983040 bytes.
Global variables use 50576 bytes (19%) of dynamic memory, leaving 211568 bytes for local variables. Maximum is 262144 bytes.
15

The optimization I used for now was Post-training quantization  |  TensorFlow Lite because full quantization was failing, I now know that hybrid issue is because full quantization was not used, I have an issue with open with TF TFlite from keras model full integer quantization fails with zero padding · Issue #50725 · tensorflow/tensorflow · GitHub

16

I think you have a few minutes after you posted to delete a post or edit it without creating a history. After that you can ask a moderator to delete a post.

1 Like
17

Full quantization indeed solved the issue, however the process makes the model incompatible :smiley: :smiley: :smiley:

quantize.cpp:58 input->type == kTfLiteFloat32 || input->type == kTfLiteInt16 || inp was not true.
19:11:56.480 -> Node QUANTIZE (number 0f) failed to prepare with status 1
19:11:56.837 -> AllocateTensors() failed

18

OK so TF no longer support uint8 quantization, nothing in the docs about it :slight_smile: I will keep this task and later will ask to have it removed, for now I am keeping progress here and will write an article as it is pretty hard to make your own model and run it on Arduino lol.

19

Does anyone have any experience here with this? My model is almost half the size of the person detection one, yet it requires an arena of 627 * 1024, which is not possible because it maxes out the memory and cannot upload. Not only is the model smaller but the code is smaller than the example.

20

This is the problem code, without this (obviously it won't work :smiley: ) But without it the size is 16% dynamic memory, with this code introduced and an arena of 627 x 1024 it crashes.

static tflite::MicroInterpreter static_interpreter(

      model, micro_op_resolver, tensor_arena, kTensorArenaSize, error_reporter);

  interpreter = &static_interpreter;
21

Got the model uploading.... and ....

Didn't find op for builtin opcode 'SOFTMAX' version '2'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?

I am going to swear soon lol