BLE Log File Dump - MKRIoT to Android: can it be done?!

Please be gentle, I'm still fairly new at this and I'm sure my code is a mess.

As part of a crop pest research project I'm attempting to build an inexpensive environmental condition datalogger that can collect data from a site and dump the log file to my phone once a week (sites are 2.5 hrs from WiFi, but will visit them weekly). Ideally I'd like to avoid deconstructing the logger each week to remove the SD card, and was hoping to use its BLE function to gather each week's log file instead. However, I'm having a lot of trouble figuring out how exactly to send and receive said file, and am starting to wonder whether it's a lost cause.

From what I've gathered BLE is limited in comparison to classic Bluetooth and can only send information in relatively small packets, meaning that I may need to somehow compress my data into a smaller format before sending it, but I can't figure out how to adapt the protocols that I've seen involving strings to something like the .csv file that my setup would produce.

I've spent some time looking around at source documents, references, and similar projects, but much of the information I've found either died years ago, includes solutions that refer to broken links (@Nick_Pyner , do you happen to still have a link to your "Bluetooth graphic file retrieval" code?), describes limited functions that I can't figure out how to adapt to larger files, uses different systems that I don't know how to decipher, or even describes almost exactly what I hope to accomplish but in reverse! (and I can't figure out how you'd go about sending it the other way!)

I'm stuck!

My Setup:

  • Arduino Explore IoT kit, including MKR WiFi 1010 board and MKR IoT Carrier
  • Android phone using lightBlue as a BLE monitor

My Code:
I've trimmed out any of the carrier-centric display, sensor gathering, clock setting, calibration stuff and left in the SD file creation / validation stuff, as well as anything BLE related.

#include <SPI.h>
#include <SD.h>
#include <Arduino_MKRIoTCarrier.h>
MKRIoTCarrier carrier;
#include <ArduinoBLE.h>
BLEService newService("180A");
BLEStringCharacteristic logFileCharacteristic("2A56", BLERead | BLEWrite, 20);
String fileName = "";

File logFile; //the function used to store our sensor data is named logFile
char logFileName[13] = "a-000000.csv"; // data log file Name
char device_Name[] = "Logger-01";
long previousMillis = 0;

//------------------------ END OF INITIALIZATION ---------------------------

void setup() {
  CARRIER_CASE = false;
  carrier.begin();

  Serial.begin(9600);
  while (!Serial);   // wait for user to open the serial monitor
  Serial.print(device_Name);
  Serial.println(" Environmental Data Logger Debug");
  Serial.println();

  // check the SD card
  if (!SD.begin(SD_CS)) {
    Serial.println("SD Card initialization failed!");
    while (1);
    } else {
    Serial.println("SD Card found");
  }

  logFile = SD.open(logFileName, FILE_WRITE); // the name of the file is defined by the logFileName function
  if (logFileName) {
    logFile.println("Epoch, day, month, year, hour, minute, second, temperature, humidity, pressure, light, Gx, Gy, Gz, Ax, Ay, Az, soil moisture");
    logFile.close();
    Serial.print("File initialized! Name used: ");
    Serial.println(logFileName);
  }
  else {
    Serial.print("Failed to create / open / write to file:");
    Serial.println(logFileName);
  }

  if (SD.exists(logFileName)) { // verify logFileName file creation
    Serial.print(logFileName);
    Serial.println(" exists.");
  } else {
    Serial.print(logFileName);
    Serial.println(" doesn't exist.");
  }

  logFile = SD.open(logFileName); // open the logFileName file for a read test
  if (logFileName) { // verify contents of logFileName
    Serial.print("File opened! Contents of: ");
    Serial.println(logFileName);
    while (logFile.available()) {
      Serial.write(logFile.read());
    }
    logFile.close();
  }
  else {
    Serial.print("Failed to read file:");
    Serial.println(logFileName);
  }
  
  Serial.println("BLE Initializing...");

  if (!BLE.begin()) {   // Try to initialize BLE
    Serial.println("Terminal Error: Could not start BLE!");
    while (1);
  }
  
  BLE.setDeviceName(device_Name); // set advertised device / local name
  BLE.setLocalName(device_Name);
  BLE.setAdvertisedService(newService);
  
  newService.addCharacteristic(logFileCharacteristic); // add characteristics
  BLE.addService(newService);
  logFileCharacteristic.writeValue(fileName); // set initial value for characteristic
  BLE.advertise();
  
  Serial.print("Device name: ");
  Serial.println(device_Name);
  Serial.println("Bluetooth device active, waiting for connections...");
}

//----------------------- END OF VOID SETUP --------------------------

void loop() {
  BLEDevice central = BLE.central();
  
  if (central) 
  {
    Serial.print("Connected to central: ");
    Serial.println(central.address());
    while (central.connected()) 
    {
      logFile = SD.open(logFileName);
      if(logFileCharacteristic.written()) 
      {
        while(logFile.available())
        {
          logFile.read();
          fileName = logFileCharacteristic.value();
        }
      }
    }
    Serial.print("Disconnected from central: ");
    Serial.println(central.address());
  }
}

// ----------------------- End of Code ------------------------

The majority of the above code is an amalgamation from multiple sources, so I'm sure that some of it is backwards, out of order, or otherwise wrong (and I would greatly appreciate any advice on how to improve it!), but what I mainly need help with is:

a) How do I format (or process using a protocol) the file "a-000000.csv" so that it can be transmitted via BLE? Is a 20-length string the right idea? Would a different approach be better? Examples would be really helpful!

b) What do I do on the other side of the BLE monitor (when reading the file presented by the logger) to save it as a usable .csv file? Is lightBlue suitable for this purpose or will I likely need to use something else?

If you've read this far, thanks for taking the time to do so! I'd appreciate any helpful advice or comments about this issue - been banging my head against it for a while now, would really like to know whether it's possible and, if so, which method or protocol would best fit this application.

Welcome to the forum.

I think the easier option is likely to use WiFi. Here is why:

  • your phone has WiFi build in
  • your Arduino has WiFi as well and can act as an Access Point (no network needed)
  • WiFi is faster than BLE
  • with WiFi you can use a browser as an interface
  • browsers have file save functionality

Here is why I believe in your case BLE is not the right solution.

  • BLE needs an app on your phone which most likely you must write yourself, unless we find a clever way of using a generic BLE app
  • BLE is slow

There might still be a use case for BLE. Because WiFi needs a lot of power and you are only visiting every week, it would be useful to disable WiFi. You can do this by adding a button or use BLE. BLE and WiFi cannot be used at the same time, but you can switch between them. So, your app starts with BLE running continuously. You connect with your phone to BLE and enable WiFi using an app. Then you switch to WiFi, open your browser and download the files. At the end you switch back by clicking on a link or automatically after a time out.

There are a few concepts I always wanted to try. I will be happy to provide you some examples if you like this approach. There might be some hick ups along the way but I am fairly confident this can be done.

I'm not entirely sure what this is about, but it looks like I need to put my hand up for something.

  1. you are referring to Bluetooth Graphics Terminal by Cetin. This is not compatible with BLE but, combined with Arduino and HC-0x, it gives you the menage-a-trois made in heaven. That said, its graphic feature is not used when downloading data, nor is there anything exclusive about it in that role. Just about any terminal will do.

  2. The real code you need is the dumpfile facility which is one of the examples included in the SD library. This passes the data to the serial port, irrespective of the means of transmission from there.

  3. I'm not sure that a BLE must only transmit small packets, I believe that it's just that it loses efficiency when transmitting large ones. I submit that any attempt to compress your data is probably a bad idea.

  4. What you need is a plain-vanilla terminal. I don't think LightBlue is one and I suggest you ditch it for Morich's bluetooth terminal, which is. Then try sending your data to same as you aquire it and send to SD. If you can do that, you can then address the file dumping.

Note that the terminal will log your data as it receives it anyway.

  1. Arduino will copy a 250k .csv file from SD to phone via HC-0x in just a few seconds.

  2. I know nothing about your Explore IoT kit - apart from the eyewatering price - or the MKR 1010 therein, but it may be that, if you cannot do this in Bluetooth, you may be better off using WiFi, which I guess is what the MKR is really all about. The dumpfile routine would surely be essentially the same. I only mention this because I believe I have heard of people having grief with MKR Bluetooth before.

I have just realised that the broken links you mentioned are probably an old home page with files about Bluetooth. They are now available at:

BLE only gets a passing mention but there is a programme which includes a "dumpfile-on- demand" routine. This works using the date as a filename.

Hi Klaus, thanks for the advice! I didn't consider using WiFi for file transfer, but it seems that I should have - those are some convincing advantages. I'm not very familiar with connecting or interfacing arduino with WiFi (aside from connecting to the arduino IoT cloud and requesting epoch for RTC) but am willing to learn!

I'm intrigued by the idea of using BLE as a WiFi switch - the 'full' code that I'm using already incorporates a switch between BLE and WiFi so I don't think that would be too much of an issue for me to implement. What concepts were you thinking of?

(Full project code - includes WiFi for RTC, sensor reading + SD writing, analog calibration, incomplete battery monitoring, and lots of display commands for the OLED, as well as the currently-ineffective BLE code)
BlEnDaLo_build.ino (24.9 KB)

Hi Nick, thanks for the reply! I really appreciate your input, and the links to your projects! I've read through some of your previous posts and tried to get the dumpfile facility to work with BLE but just couldn't figure it out - there's still a lot I don't understand about the differences between BLE and classic bluetooth, or how to make best use of either.

I have been trying to use Kai-Morich's terminal as well as LightBlue, but considering I couldn't get either to work I figured the issue lay elsewhere (or there was something significant I was missing). It sounds like, from both yourself and Klaus, that WiFi might be a simpler and more efficient route to pursue for this application and setup - so pursue I will! Any suggestions you might have regarding said pursuit would be appreciated (Currently planning to start with the NiNa WiFi library resources and work from there).

In regards to the Explore IoT kit, I've been pleasantly surprised by the components and capabilities of the board, shield, and sensors. It contained almost everything I was looking for (minus a soil temp sensor), and very little that I didn't need or couldn't use. Considering that a setup with similar capabilities from a supplier like Hoskins or Campbell Scientific would cost upwards of $2500 (more precise measurements for sure, and no setup, but $2500?!?) I'm more than happy to devote some time to learning and refining a decent-quality arduino logger. The online tutorial projects did a great job of going through the board's functions and features, and there's a lot of room for flexibility and modification. Quite excited for this little board's potential.

I'm afraid I have very little experience with BLE, but this may be just a matter of improper wiring. Check you have Tx>Rx and Rx<Tx!! The dumpfile routine works exactly the same for BLE as it would for HC-05. The Morich Terminal does seem to be the goto app for this sort of thing with BLE.

There is nothing I said that suggests that, indeed I think it's nonsense, BUT I vaguely recall that all is not well with the MKRs bluetooth. I could be quite wrong about that. On the matter of simplicity, it cannot get simpler than Bluetooth. The expertise you require to do it is exactly the same as that required to send data to the serial monitor. As for efficiency, I reckon that talk is all theory and no practice, and it certainly does not apply to what I understand you are doing.

Forget what i said about the kit. If it suits you, it suits, and I was out of order. I'm the guy whose first Arduino was an EtherTen - and then found you can make up a comprehensive datalogger for about $25. You may find that you can do the next one one hell of a lot cheaper than you might think.....

There is nothing I said that suggests that, indeed I think it's nonsense

My bad - I misunderstood and assumed you meant that BLE was a bad idea when you said you wouldn't recommend data compression to increase BLE efficiency.

It does sound like classic bluetooth is incredibly simple, and would probably work well for my application, but I'm not sure that statement also applies to BLE - the tutorials I've come across seem to imply that BLE file transfer is possible, but I've yet to encounter a working example of it. There are multiple examples of data transfer, but I'm not sure what would be necessary to adapt those utilities to something like a .csv file.

Since the MKR wifi 1010 is only capable of BLE, I'm uncertain whether I should continue attempting to make that work. Do you know what I might have to change / add to the code in my first post that would enable it to send the small .csv file it creates over BLE?

LOL. I meant that I don't think it would solve anything and would merely add the problem of decompressing the data at the phone end.

I'm sure it is possible, and the answer is probably to be found in the IOS arena. The main issue here is to consider if it is worth the effort. I don't actually think it is. I have been sitting on HM-10 modules for years with no action - and no motivation either. I only recently found that the real best-kept secret in Bluetooth is that you need a BLE capable terminal before you can use it. I did find that I could do a G'day<>G'day just like an HC-05, but only then assumed it would transfer a file in the same manner. I never actually put the BLE into service, as there was no point in it. The main outcome of all this was that it was clear that HM-10 offers absolutely nothing over an HC-05 for this purpose. I don't know how HM-10 compares with your BLE. You might check what Martyn Currey has to say on this.

It is easy for me to flippantly say ditch it, I'm a Mega+HC-05 man and became that way before the MKR was invented. But, you need to keep back of mind that you can make a thoroughly capable datalogger
Pro-mini+HC-0x+RTC+SD
for about $10 or so, + PSU and peripherals. On a similar tack, you might consider simply adding a $3 HC-0x to what you already have. You may need to tread carefully in the power department, but you might find that this is common practice!

The filedump subroutine should be universal. There would be absolutely no changes required for Bluetooth of any sort. I have never used WiFi for this purpose.

Most WiFi server examples create a very simple web pages trough inline printing HTML code. When you already have a SD card it would be nice to just use some files on the SD card (HTML, CSS and your data files for instance) for the web page displayed. This will allow much nicer web pages without creating large sketches with lots of HTML and CSS code.

I build a test sketch already that loads the html file and sends it to the client. But it needs some work to allow other files and insert data in the right place.

I had a look at your sketch and there are several issues I believe.

  • your loop function has a blocking while, if you have a BLE client connected all other functions e.g., reading sensors will stop

  • your BLE UUID are 16-bit UUIDs, they are reserved for use of services and characteristics standardized by the Bluetooth SIG. You must use 128-bit random UUIDs for your own services and characteristics.

  • you have a lot of mixed code e.g., connect_WiFi has two lines of code for WiFi and more than 10 lines of code displaying stuff

  • you function naming scheme is inconsistent. I recommend using camelCase as used by Arduino examples.

  • you have a lot of code in setup. I prefer to only have code there that needs to run once. Any code that might need restarting like WiFi, BLE, SD I would handle in functions called by loop.

  • your code inside while(logFile.available())
    -- does not make sense
    -- I believe String characteristics cannot be read using value.

  • your code inside if(logFileCharacteristic.written())
    -- does not make sense
    -- is not about getting data from the characteristic being written by a client

  • you are using long data type for millis code, the reference says you should use unsigned long
    -- https://www.arduino.cc/reference/en/language/functions/time/millis/

  • you seem to be reading all your sensor, all at the same time. This does not make sense. Sensors do not just create numbers. They have a physical meaning. Reading acceleration and temperature at the same interval will not create useful data. Ether you are reading your temperature faster than it can possibly change or you are reading acceleration data slower than needed to do anything useful with it. I guess you were just reading everything because you could. What are you going to do with the values? I recommend you focus on the values that you want to use for something useful. e.g., logging weather data.

I have been trying to use Kai-Morich's terminal as well as LightBlue, but considering I couldn't get either to work

My understanding is that the BLE peripheral needs to be using the Nordic UART service for communication with Serial Bluetooh Terminal.

I'm afraid I don't know anything about that. It sounds rather like yet another good reason for not bothering with BLE for this purpose. All I did was prove the point with a simple two-way hello, using Morich's terminal and HM-10 as found. It didn't seem too reliable, but I thought that was just my crude setup. I have published notes on using Bluetooth. In the light of what you have said, I am now considering removing my rather sketchy reference to HM-10. As I said, I believe HM-10 offers nothing in this arena anyway. Nothing that is, other than being a sop for IOS users - for whom I have no sympathy!

Oh wow, thanks Klaus! Some great info hear, really appreciate that you were willing to go through my program and offer suggestions / critique.

Excellent! I'll give this a shot and let you know what I come up with. I think this could be a good alternative to fiddling with BLE - at least until I can figure out what I'm doing with that...

That was intentional, I'd like the logger to pause while I collect the data that it has logged during the week. Are there situations where you can foresee this being a problem? Considering that I'll be adjusting that section anyways in order to introduce a BLE-controlled wifi switch I'd love to know if there are pitfalls I can avoid.

Is that required? If so I can easily change it, but I was under the impression that if the services / characteristics I'm using are listed in the 16-bit UUID list (ex: GATT Service 0x181A Environmental Sensing) it was okay to use that UUID. Are the 128-bit random UUID's a regulation thing or simplicity thing?

Yup, I am still very new at this and don't really know what the common practice is for code structure. Since the loggers I'm using will not be plugged into a computer I won't always have access to a serial monitor for debugging. Instead, I wanted to have some way of visually confirming that the logger was able to complete each individual step during setup - so I have every section print out onto the display what it's doing, specific resources its using or creating, and whether it was successful or not.

I've included the serial notifications within the DEBUG switch, allowing me to turn off serial notifications when the logger is no longer connected to a computer, and will probably do the same for display notifications once the project is up and running. Is there a better way to structure debugging notifications like this?

Ah! Good catch, I'll go through and reformat for consistency.

I'm not entirely sure I understand - shouldn't the RTC, BLE, and SD initialization only need to run once? Control of the WiFi and BLE connections will be part of the loop, to enable data file transfer, but I'm not sure what purpose would be served by including their initialization code (i.e. status print, clock set, device naming) into the loop. I agree there is a LOT of code in setup, but the majority of it is simply debug notifications sent to serial or display. Would you suggest packing each section into its own function to clean up and simplify Setup?

That is fair and accurate! I don't know what's going on there, and was getting really confused trying to understand what BLE was asking for, what format my data was in, how I needed to change it so it could be read / sent / received, or how to construct the function correctly. I still don't know!

Good catch, thanks!

Correct, and while the sensors currently read every 5 seconds I hope to have them record data at 10 minute intervals while in the field. The majority of those are directly related to the research I'm conducting, and there's even two sensors I still haven't added to that list as I'm waiting on some connectors for them (soil temperature and a different brand of temp/humid sensor for accuracy comparison). However, the accel and gyro data are, as you mentioned, functionally useless for typical interpretation. I considered axing them from the code, but kept them in for two reasons:

a) These loggers will be planted upright in soil at a particular orientation in remote farm fields. It could be beneficial to have a record of any disturbance or adjustment to their placement, environmental or otherwise, to correlate with recorded data (ex: wind or 'some jerk' knocks the logger over, gyro data shows when the tip happened, lets us know which data can still be used and which can't).
b) As far as I can tell it's not very costly, resource-wise, for the logger to include these measurements in its routine. The file size is tiny, and the energy cost is negligible (unless I'm very wrong and reading the gyro/accel actually involves reasonably significant energy use?)

Having said the above, I'm not strongly tied to the idea of keeping this information in the final build, and would love to hear arguments against their inclusion.

Is that the same as the nRF UART app? I'm not familiar with either, but I'd love to know if they could be used for something like that! So far I still haven't even been able to figure out how to use BLE as a serial monitor... D:

I don't know about that specific app, but when I program a BLE peripheral to use Nordic UART Service I can connect and communicate with Kai Morich Serial Bluetooth Terminal

#define SERVICE_UUID           "6E400001-B5A3-F393-E0A9-E50E24DCCA9E" // UART service UUID
#define CHARACTERISTIC_UUID_RX "6E400002-B5A3-F393-E0A9-E50E24DCCA9E"
#define CHARACTERISTIC_UUID_TX "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"

Here is a sketch written for the ESP32 which talks back and forth with the Serial Bluetooth Teminal in UART mode.


#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

BLEServer *pServer = NULL;
BLECharacteristic * pTxCharacteristic;
bool deviceConnected = false;
bool oldDeviceConnected = false;
bool newMessage = false;
uint8_t txValue = 0;
bool mR = false;
std::string rxValue = "";

#define SERVICE_UUID           "6E400001-B5A3-F393-E0A9-E50E24DCCA9E" // UART service UUID
#define CHARACTERISTIC_UUID_RX "6E400002-B5A3-F393-E0A9-E50E24DCCA9E"
#define CHARACTERISTIC_UUID_TX "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"


class MyServerCallbacks: public BLEServerCallbacks {
    void onConnect(BLEServer* pServer) {
      deviceConnected = true;
    };

    void onDisconnect(BLEServer* pServer) {
      deviceConnected = false;
    }
};

class MyCallbacks: public BLECharacteristicCallbacks {
    void onWrite(BLECharacteristic *pCharacteristic) {
      rxValue = pCharacteristic->getValue();

      if (rxValue.length() > 0) {
        Serial.println("*********");
        mR = true;
        Serial.print("Received Value: ");
        for (int i = 0; i < rxValue.length(); i++)
          Serial.print(rxValue[i]);

        Serial.println();
        newMessage = true;
        Serial.println("*********");
      }
    }
};


void setup() {
  Serial.begin(115200);
  //setup LED
  pinMode(16, OUTPUT); //onboard led LOW to turn on
  digitalWrite(16, HIGH);

  // Create the BLE Device
  BLEDevice::init("UART Service");

  // Create the BLE Server
  pServer = BLEDevice::createServer();
  pServer->setCallbacks(new MyServerCallbacks());

  // Create the BLE Service
  BLEService *pService = pServer->createService(SERVICE_UUID);

  // Create a BLE Characteristic
  pTxCharacteristic = pService->createCharacteristic(
                        CHARACTERISTIC_UUID_TX,
                        BLECharacteristic::PROPERTY_NOTIFY
                      );

  pTxCharacteristic->addDescriptor(new BLE2902());

  BLECharacteristic * pRxCharacteristic = pService->createCharacteristic(
      CHARACTERISTIC_UUID_RX,
      BLECharacteristic::PROPERTY_WRITE);

  pRxCharacteristic->setCallbacks(new MyCallbacks());

  // Start the service
  pService->start();

  // Start advertising
  pServer->getAdvertising()->start();
  Serial.println("Waiting a client connection to notify...");
}
void respond(std::string  send_message) {
  pTxCharacteristic->setValue(send_message);
  pTxCharacteristic->notify();
}

void loop() {

  if (Serial.available() > 0)
  {
    char monitorInput[50] = "";
    byte len = Serial.readBytesUntil('\n', monitorInput, 49);
    monitorInput[len] = '\0';
    Serial.println("Sending message input from Serial Monitor");
    Serial.println(monitorInput); //echo back to monitor
    Serial.println("*********");
    respond("Received from monitor\n");//shows in phone
    respond(monitorInput);//send on BLE UART
  }

  if (deviceConnected && newMessage) {
    newMessage = false;
    if (rxValue == String("hello\n").c_str()) {
      digitalWrite(16, LOW);
      delay(1000);
      digitalWrite(16, HIGH);
      respond(String("world\n").c_str());
    }

    respond("Confirm received from phone\n");
    respond(rxValue);
  }

  // disconnecting
  if (!deviceConnected && oldDeviceConnected) {
    delay(500); // give the bluetooth stack the chance to get things ready
    pServer->startAdvertising(); // restart advertising
    Serial.println("start advertising");
    oldDeviceConnected = deviceConnected;
  }
  // connecting
  if (deviceConnected && !oldDeviceConnected) {
    // do stuff here on connecting
    oldDeviceConnected = deviceConnected;
  }
}

Oh cool! Not sure I understand the difference between Nordic UART and regular UUIDs, but also not sure whether or not that would help me overcome the difficulties that I've been having with BLE. I really appreciate the example sketch you've included, but whenever I attempt to run it I get:

Arduino: 1.8.15 (Windows Store 1.8.49.0) (Windows 10), Board: "Arduino MKR WiFi 1010"

Multiple libraries were found for "BLEDevice.h"

 Used: C:\Users\Mocki\Documents\Arduino\libraries\ArduinoBLE

In file included from C:\Users\Mocki\Documents\Arduino\BLE_UART_Serial_test\BLE_UART_Serial_test.ino:2:0:

C:\Users\Mocki\Documents\Arduino\libraries\BLE\src/BLEServer.h:10:10: fatal error: sdkconfig.h: No such file or directory

 #include "sdkconfig.h"

          ^~~~~~~~~~~~~

compilation terminated.

 Not used: C:\Users\Mocki\Documents\Arduino\libraries\BLE

exit status 1

Error compiling for board Arduino MKR WiFi 1010.



This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

...which I believe is a long way of saying "this code is built for ESP32, not MKR WiFi 1010". I have no idea how to rectify that, and am also not sure whether putting in the effort to do so would solve the issue I've been having with how to transfer .csv data files via BLE.

Yes. Uses ESP32 libraries. I'd steer away from BLE if at all possible.

Haha fair! Wifi transfer it is :grinning_face_with_smiling_eyes:

The 128-bit random UUID ensure, you can create your own services and characteristics without collisions with other devices and apps. When you use 16-bit UUIDs you must only use them as described by the Bluetooth SIG specifications. This ensures all apps and devices that use them can work together e.g. you can by a hear-rate monitor from company A and use it with a sports watch from company B. The watch reads the services and characteristics UUID and knows the format of the data.

Nope, your RTC will drift and you will need to sync the time on a regular basis. When you switch between BLE and WiFi, only one can be active at a time and you will need to restart the connection. With SD card what do you do, when something goes wrong with one file or the SD card is removed and then reinserted? I try to write code in a way that it can recover from errors. Especially when I want it to run for long times.

Regarding the orientation sensing. That makes sense.

That makes sense. Considering that the loggers will be pretty far from civilization I doubt I'll run into any conflicts, but it's always good to be careful.

That's fair! Hadn't considered the drift, and it's good to integrate that ability to recover. I'll try to shift stuff around and build functions for set-up, but I'm still pretty new at building this stuff. Will post once I've got something shambled together that works (or I run into another wall...)

In the meantime, this is the current working WiFi Server code I've come up with - a combination of the Arduino MKR IoT Greenhouse Web Server tutorial code and the file management system from this thread (kept the French comments for reference, haven't typed out the translations yet)

#include <SPI.h>
#include <SD.h>
#include <WiFiNINA.h>
#include <Arduino_MKRIoTCarrier.h>
MKRIoTCarrier carrier;

#define DEBUG  // uncomment this line for serial monitor output

File root; //the drive which stores log files
File logFile; //files used to store sensor data
char logFileName[13] = "a-000000.csv"; // custom data log file Name

char ssid[] = "TELUS0487-2.4G";
char pass[] = "pf8nt8ccwy";
int keyIndex = 0;
int status = WL_IDLE_STATUS;
WiFiServer server(80);
WiFiClient client = server.available();

void setup() {
  Serial.begin(9600);
  while (!Serial);
  CARRIER_CASE = false;
  carrier.begin();

  // check the SD card
  if (!SD.begin(SD_CS)) {

#ifdef DEBUG
    Serial.println("SD Card initialization failed!");
#endif

    while (1);
  }

  else {
    carrier.display.fillScreen(ST77XX_BLACK);
    carrier.display.setTextColor(ST77XX_GREEN);
    carrier.display.setCursor(20, 70);
    carrier.display.print("SD card found!");

#ifdef DEBUG
    Serial.println("SD Card found");
#endif
    delay(2000);    // to let user know
  }

  enable_WiFi();
  connect_WiFi();

  server.begin();
  printWiFiStatus();

  printDirectory(root, 0);
  root = SD.open("/");
  Serial.println("Directory contents:");
  printDirectory(root, 0);
  logFile = SD.open(logFileName, FILE_WRITE); // the name of the file is defined by the logFileName function
  if (logFileName) {
    carrier.display.setTextColor(ST77XX_GREEN);
    carrier.display.setCursor(20, 170);
    carrier.display.print("Data stored as:");
    carrier.display.setTextColor(ST77XX_WHITE);
    carrier.display.setCursor(10, 190);
    carrier.display.print(logFileName);
    logFile.println("Epoch, day, month, year, hour, minute, second, temperature, humidity, pressure, light, Gx, Gy, Gz, Ax, Ay, Az, soil moisture");
    logFile.close();
#ifdef DEBUG
    Serial.print("File initialized! Name used: ");
    Serial.println(logFileName);
#endif

  }
  else {
    carrier.display.setTextColor(ST77XX_RED);
    carrier.display.setCursor(20, 170);
    carrier.display.print("File write failed!");

#ifdef DEBUG
    Serial.print("Failed to create / open / write to file:");
    Serial.println(logFileName);
#endif
  }

  if (SD.exists(logFileName)) { // verify logFileName file creation
    Serial.print(logFileName);

    Serial.println(" exists.");
  } else {
    Serial.print(logFileName);
    Serial.println(" doesn't exist.");

  }

  logFile = SD.open(logFileName); // open the logFileName file for a read test
  if (logFileName) { // verify contents of logFileName
    carrier.display.setTextColor(ST77XX_GREEN);
    carrier.display.setCursor(20, 170);
    carrier.display.print("Contents of:");
    carrier.display.setTextColor(ST77XX_WHITE);
    carrier.display.setCursor(10, 190);
    carrier.display.print(logFileName);

#ifdef DEBUG
    Serial.print("File opened! Contents of: ");
    Serial.println(logFileName);
    while (logFile.available()) {
      Serial.write(logFile.read());
    }
#endif

    logFile.close();
  }
  else {
    carrier.display.setTextColor(ST77XX_RED);
    carrier.display.setCursor(20, 170);
    carrier.display.print("File read failed!");

#ifdef DEBUG
    Serial.print("Failed to read file:");
    Serial.println(logFileName);
#endif
  }
  delay (4000);
}

void loop() {
  client = server.available();
  if (client) {
    printFiles();
  }
}

void printFiles() {
  Serial.println("new client");
  int index = 0;
  int bufSize = 100;
  char clientLine[bufSize];
  char name[17];
  while (client.connected()) {
    if (client.available()) {
      char c = client.read();
      Serial.write(c);
      if (c != '\n' && c != '\r') {
        clientLine[index] = c;
        index++;
        // are we too big for the buffer? start tossing out data
        if (index >= bufSize)
          index = bufSize - 1;

        // continue to read more data!
        continue;
      }

      // got a \n or \r new line, which means the string is done
      clientLine[index] = 0;

      // Print it out for debugging
      Serial.println(clientLine);

      // Look for substring such as a request to get the file
      if (strstr(clientLine, "GET /") != 0) {
        // this time no space after the /, so a sub-file!
        char *filename;

        filename = clientLine + 5; // look after the "GET /" (5 chars)  *******
        // a little trick, look for the " HTTP/1.1" string and
        // turn the first character of the substring into a 0 to clear it out.
        (strstr(clientLine, " HTTP"))[0] = 0;

        if (filename[strlen(filename) - 1] == '/') { // Trim a directory filename
          filename[strlen(filename) - 1] = 0;      //  as Open throws error with trailing /
        }

        Serial.print(F("Web request for: ")); Serial.println(filename);  // print the file we want

        File file = SD.open(filename, O_READ);
        if ( file == 0 ) {  // Opening the file with return code of 0 is an error in SDFile.open
          client.println("HTTP/1.1 404 Not Found");
          client.println("Content-Type: text/html");
          client.println();
          client.println("<h2>File Not Found!</h2>");
          client.println("<h3>Couldn't open the File!</h3>");
          break;
        }

        Serial.println("File Opened!");

        client.println("HTTP/1.1 200 OK");
        if (file.isDirectory()) {
          Serial.println("is a directory");
          //file.close();
          client.println("Content-Type: text/html");
          client.println();
          client.print("<h2>Files in /");
          client.print(filename);
          client.println(":</h2>");
          ListFiles(client, LS_SIZE, file);
          file.close();
        } else { // Any non-directory clicked, server will send file to client for download
          client.println("Content-Type: application/octet-stream");
          client.println();

          char file_buffer[16];
          int avail;
          while (avail = file.available()) {
            int to_read = min(avail, 16);
            if (to_read != file.read(file_buffer, to_read)) {
              break;
            }
            // uncomment the serial to debug (slow!)
            //Serial.write((char)c);
            client.write(file_buffer, to_read);
          }
          file.close();
        }
      } else {
        // everything else is a 404
        client.println("HTTP/1.1 404 Not Found");
        client.println("Content-Type: text/html");
        client.println();
        client.println("<h2>File Not Found!</h2>");
      }
      break;
    }
  }
  // give the web browser time to receive the data
  delay(1);
  client.stop();
}

void printWiFiStatus() {
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
  long rssi = WiFi.RSSI();
  Serial.print("Signal Strength: ");
  Serial.print(rssi);
  Serial.println(" dBm");
  Serial.print("Access at http://");
  Serial.print(ip);
}

void enable_WiFi() {
  if (WiFi.status() == WL_NO_MODULE) {
    Serial.println("Communication with WiFi module failed!");
    while (true);
  }
  String fv = WiFi.firmwareVersion();
  if (fv < "1.0.0") {
    Serial.println("Please upgrade the firmware");
  }
}

void connect_WiFi() {
  while (status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    status = WiFi.begin(ssid, pass);
    delay(5000);
  }
}

void ListFiles(WiFiClient client, uint8_t flags, File dir) { //function to list files for HTML
  client.println("<ul>");

  while (true) { //on prefere utiliser une boucle infinie
    File entry = dir.openNextFile(); //parcours le repertoire en lisant successivement les fichiers

    // fin de chaine : plus de fichiers suivants
    if (! entry) {
      break;
    }

    // identation et listing html
    client.print("<li><a href=\"");
    client.print(entry.name()); //on prend le nom du fichier entry declare precedemment
    if (entry.isDirectory()) { //si c'est un repertoire, affichage special
      client.println("/");
    }
    client.print("\">");

    // si on clique sur un dossier, il affiche de nouveau le contenu (on change de repertoire en quelque sorte)
    //le nouveau repertoire est le repertoire fils et on re test la presence de contenu
    client.print(entry.name());
    if (entry.isDirectory()) {
      client.println("/");
    }
    client.print("</a>");
    client.println("</li>");
    entry.close(); //fermeture du fichier entry car traitement effectue
  }
  client.println("</ul>");
}

void printDirectory(File dir, int numTabs) {
  while (true) {
    File entry =  dir.openNextFile();
    if (! entry) {
      // no more files
      break;
    }
    for (uint8_t i = 0; i < numTabs; i++) {
      Serial.print('\t');
    }
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println("/");
      printDirectory(entry, numTabs + 1);
    } else {
      // files have sizes, directories do not
      Serial.print("\t\t");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

Will be taking some time to go through this setup, figure out how to implement the 'delete file' function described later on in that thread, and clean it up to make it purty.

Any suggestions?

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.