Advanced data processing Question

Dear Programmers and hobbyist,

My name is Jeff, and i started programming many years ago.
I used to be a programmer for small projects on my job.
I can say im clearly a intermediate programmer these days.

Okay well to the point, i have a question as you can see.

I've searched alot on forums and stuff but i could not find anything related to my question.
I want to make a week clock that i can program from things like the serial monitor.
It's neccesary that i can put different times on specific day's that a relay or led need's to be turned on.

The following question is, I need to know how i can handle data like this and make a algoritm for it.
How can i save it to the arduino, so if it's turned off and then on again it didnt lose it's data.

I use the following setup for it,

  • Arduino Nano 328P or a Nano Every.
  • DS3231 RTC Module.
  • microSD card Module.

So what i want to learn is as followed,
I want to send a string like:

Day,Hour_on,Minute_on,Hour_off,Minute_off

Monday,12,12,12,13

And save that to the EEPROM or sdCard, i clearly specify the sdCard for it.
After that the arduino needs to run the loop and read the file saved on the sdCard till it matches it's if or while statement.

Hope someone have's a solution for me and want to give me some hints or tips!

Best wishes, and hope you all doin great these days.

I'm not really sure what part of your problem you want help with.

If you are asking about sending data to the Arduino then have a look at the examples in Serial Input Basics - simple reliable non-blocking ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

...R

Thanks for your reply! well that's not the question indeed.
I know how to use the string format and how to process that into integers and Strings.

The thing is, i want to know how i can make a algorithm that processes this data into a .cfg or .txt or something like that. And after all that's done the arduino can read each line and if one of the line's meets the value's of the DS3231 it go's on. it supposed to go on between the on and the off time.

Again i really appreciate your answer! Hope you doin well these day's.

What algorithm? What advanced? A simple scan of a table for matches seems to be all you would need, where a "match" means the current time is greater than or equal to the stored time.

That's exact what i ment! haha but how can this been done?
Normally i make kind of smaller and more simplified projects.
Now i got an BIG order that i see as a challange, but really can't find any thing related to it.

What i hope is that some of you can give me some idea's and maybe a point to start from.

Best wishes, Jeffrey.

j_molenaar:
Hope someone have's a solution for me and want to give me some hints or tips!

I do have some hints.

Break your project down into individual parts.
Get each one working independently of the others.
Then integrate them.

For breaking down, I would suggest these pieces:

  • SD card write and read
  • RTC read
  • Serial input of new calendar data
  • Actual turning on and off of your other hardware

I assume that since you did not ask about LEDs and relays, you are comfortable with controlling these devices. Of course if this is not the case, ask.

For the SD card. Start with tutorials or examples that come with the library for the hardware you have.
Eventually, I suspect that you will want multiple records for the various on/off events.
In your example you list day, on time and off time.
I would recommend 2 small changes.
Include a device identifier. You want the LED to turn on and off Tuesday. And you want the Relay to turn on and off Thursday. Include the device identifier in the record.
Include the day for both the on and off time. That way you can turn on an LED Tuesday at noon and turn it off Wednesday at 2 AM.

For the RTC. Make sure you can read it and display it to the serial monitor.

For the Serial management of the calendar. I expect you will want to print to the serial monitor with the current calendar records along with some command list.
D-xx to delete record number xx
A-DDDHHMM-DDDHHMM-DEVxx to add a record for device number xx with the day, hour and minute on and off time defined by DDDHHMM
example
A-MON0137-TUE1520-DEV07 could be the command to add a calendar record to turn on device number 7 at Monday at 1:37am and turn it off at Tuesday at 3:20pm.
After processing any input from the serial monitor, reprint the new calendar.

Does that give you some framework to use? Does it make sense?

It would behoove you to become familiar with the Time library. It will greatly simplify a lot of your calculations.

aarg:
It would behoove you to become familiar with the Time library. It will greatly simplify a lot of your calculations.

And, when comparing times, using Epoch Time will make your life much simpler.

It make's sense indeed! And i'ts nice to get some input that i can think about!

I've been working on it yesterday, and what i already have working is that i can input setTime in serial and then proceed to process on setting time in the next format = h_on, m_on, h_off, m_off.
i can save that to the EEPROM and read the eeprom value's to turn the led on or off.
At this point i only use a LED, which later will be an relais.
so i can read the DS3231 and can catch serial inputs and convert it to integers and strings.
The only thing what i need the most is how can handle a serial input that the arduino can convert and save directly into the cfg. after all the programming through the serial is done. the arduino must start "scanning" the cfg file till it finds the matching indentifications.

Again i really really appreciate all the help i've got!

And for the librarie's ive made one my own, where i can get all the data out and calculate etc. haha :smiley:

I would not constantly scan the file.
I would scan it once to look for the next event that needs to be handled.
Then watch the time until it matches the time of that event.
Then handle it.
Then scan the file to find the next event that needs handled.

vinceherman:
I would not constantly scan the file.
I would scan it once to look for the next event that needs to be handled.
Then watch the time until it matches the time of that event.
Then handle it.
Then scan the file to find the next event that needs handled.

But what if their are multiple, overlapping events? What if an event gets added or removed during handling of a previous event?

I think I would scan every time new info is added.

Personally I think it's indeed better to scan the file when new data is been added or removed.
This means that I have to create a class that can handle this kind of data.
The following question I have for this epic community is.
How to scan a .cfg or .txt file?
How can I perform a class or void to scan the file.
Does anybody out here have a simple sketch that i can start with. To learn from and move on.

I really appreciate your help! And the fact this community is helping each other! My grand dad always says "if you want to become a pro, work with a pro, act like a pro and think like a pro!" This keeps me always busy, and want to move forward!

You will probably end up using a CSV (comma separated variable) format, there are lots of examples and maybe even some libraries for reading those.

For time storage, the easiest text format for conversion and handling is a decimal Epoch time. For example, it is now 1586470938 in Epoch time. You can convert that string into an unsigned int (or time_t) binary variable for processing.

Thanks for your reply! This was actually the thing that i needed!

And i actually find a beautifull and easy library that i could use for this kind of projects.
How i'm going to do it is as followed:
Firs't im going to search for "ID=(int)" as a string.
After that there's going to be the device format like an integer, this for selecting the led pin.
After that the Date in format YYYY;MM;DD, seperated by ;.
Then im going to read the on and off hours using the format h_On;m_On;h_Off;m_Off.
If all these things are ok it will fill in the old value's to the new value's.
If all these match these old value's and the state=off again it will search for the next ID and it will look if the date or day matches. if this is true it wil overwrite the old value's.

You're making more work for yourself than necessary. You have been advised multiple times that using Epoch Time will make your life much easier.

You should continue to have all interaction with the user via YY:MM:DD:HH:MM:SS format. But, internally all times should be stored and manipulated using Epoch Time. The conversion between the two is trivial if you use the PJRC Time Library.

i'm going to take a look at it, but i know from some of the programmer's that Epoch will end in 2032.

03:14:07 UTC on 19 January 2038

Do you care about this given that it's 17+ years in the future?

j_molenaar:
i'm going to take a look at it, but i know from some of the programmer's that Epoch will end in 2032.

The Time library implements time_t as an unsigned integer. That will work until year 2106. The only problem is that dates after 2038 may need different treatment in conversion to and from YMDHMS values.

I made a sketch to count time just before the Y2038 rollover, at 03:14:07 UTC on 19 January 2038.

/*
 * Test Y2038 rollover
 * by aarg on the Arduino forum
*/
#include <TimeLib.h>

void setup()  {
  Serial.begin(9600);
  setTime(3, 14, 5, 19, 1, 2038);
}

void loop() {
  while (millis() < 5000) {
    digitalClockDisplay();
    Serial.println((signed long)now());
    delay(1000);
  }
}

void digitalClockDisplay() {
  // digital clock display of the time
  Serial.print(hour());
  printDigits(minute());
  printDigits(second());
  Serial.print(" ");
  Serial.print(day());
  Serial.print(" ");
  Serial.print(month());
  Serial.print(" ");
  Serial.print(year());
  Serial.println();
}

void printDigits(int digits) {
  // utility function for digital clock display: prints preceding colon and leading 0
  Serial.print(":");
  if (digits < 10)
    Serial.print('0');
  Serial.print(digits);
}

You can see that the Time library handles times after the 2038 rollover quite well. I have printed the epoch time as a signed number, to demonstrate the point at which it becomes negative.

3:14:05 19 1 2038
2147483645
3:14:06 19 1 2038
2147483646
3:14:07 19 1 2038
2147483647
3:14:08 19 1 2038
-2147483648
3:14:09 19 1 2038
-2147483647

I did not make any changes to the library for this.
So let's get together in 2106 for a beer!

Haha okay well that work's indeed, i might dive in to that a little bit further.
Well i have a question i am currently using this library GitHub - fivitti/Arduino-CSV-File: Extend class for SdFat library for easy work with CSV files..
But the problem is when the arduino is powered off and back on the data got lost and i can't read it because the file is empty.
How can i do this? because the whole point of saving the "on off" data will be lost then.
i'm getting more and more frustrated about doing this.

Hope you guys might have an answer! Cheers.