AUTOMA: a smarter photovoltaic DHW boiler (feat. OWL Intuition)

[roberto_nosfe]

This project is my Hello World onboard the Arduino, and 10 years have passed since my last programming effort, so please be forgiving about the quality of my coding.

Here's the scenario:

• I own a house.
• I installed a grid connected photovoltaic system.
• I bought a modern domestic hot water ("DHW") heat pump boiler.

I could let the boiler do its job, and a hot shower but poor self-consumption would be the result.
I could connect the boiler to the PV system through a wattmeter driven relay. This would raise my self-consumption (i.e. free hot water), but sometimes the sun won't be enough so an occasional cold shower is to be expected.
I could use natural gas as a heat backup for such occasions, but my PV has plenty of electricity available through net-metering, and avoiding natural gas usage was and still is my main goal.

There are plenty of Arduino projects that provide electricity measurement and logging, and the vast majority of DHW boilers can read an external dry contact acting as "call for heat", so a relay shield could do the job.
I chose... neither of these approaches.

I began by logging the electricity usage of a stardard family for a complete year with a commercial datalogger, I'm talking a family with both a PV system of at least 3 kWp and a heat pump DHW boiler of at least 250 lt. .
I collected as many as 2.5 million records per family, and did this with 3 different real world families of 2, 3 and 4 people.
Then I managed to set up humongous nighttime simulation sessions looking for a way to optimize the warm up cycles of the boiler according to BOTH the self-consumption ratio AND the minimum level of service needed.

Here's what I found.

First of all, when there is enough excess energy, the heat pump should turn on. A good threshold should never exceed boiler consumption, so far so good.
Now, since we are heating for the evening and the following morning, the threshold should slowly decrease as we approach the evening but of course go up again when the evening begins.
It may sound too easy, but this "decreasing threshold" got the job done pretty well in all my simulated scenarios.
Of course, this is true provided we find a clever way to evaluate the threshold decrease rate in real time.

So, our energy balance will repeatedly be triggered against our threshold, decreased by an escalation factor.
This guy will be something like N watts/minute from XX:XX in the morning to YY:YY in the evening.
Escalation will be based on expected PV production, weather forecast for the next hour and water temperature gradient expected, i.e. desired temp minus actual temp. We can't predict any household appliance, so let's move on.
Expected PV production (i.e. season factor) deals with solstices and equinoxes, with a fixed 31 days of global thermal inertia (trust me on this). Southern emisphere inhabitants may want to fix this one.
Weather forecast (i.e. weather factor) is gathered via Multicast once every 40 minutes.
The reason for these two is quite easy to understand:

• Little excess energy in a sunny midday of July? Who cares! There's plenty of time for heating, let the washing machine finish its job!
• No excess energy in a snowy midday of January? Go go go! It won't get any better today!

The real time threshold on the other hand, is represented by an average consumption (see tech specs of your boiler) altered by these 2 factors:

• water temperature
• intake air temperature

We have to go by trial and errors for these. My own device is fine with 7 and 0 respectively.
This is to avoid turning the boiler on just to measure its energy consumption (heat pumps hate this behaviour).

[roberto_nosfe]

Let's have a look at the hardware now.
The datalogger I used to gather all the electrical data was an OWL Intuition, a fairly cheap device easily available throughout Europe.
Occasionally, I discovered that this thing can Multicast its readings and control an optional external boiler thermostat.
Gee! This toy can act as a UDP controlled relay for our boiler, while logging the hell out of it!
Speaking with OWL, I realized they had no product matching my specific needs, and were not interested in developing one, so here's where I step in with my AUTOMA: Another Unmanned Thingo OWL Missed to Accomodate.

Our shopping list is as follows:

• 1 OWL Intuition -lc
• 1 OWL Tank Sensor
• 1 Arduino Ethernet

Intuition -lc is basically a Network OWL (which acts as router for all Intuition products) plus a wireless electricity monitor called CMR with its 3 current sensors. In my project the sensors are not applied to the 3 phases but to consumption, production and boiler consumption, all of them being single phase. Refer to Intuition -pv manuals.
The tank sensor is a radio controlled thermostat with a water temperature sensor and an internal heating time clock, just like a room thermostat. Refer to Intuition -h manuals.
A single Network OWL is able to deal with radio signals from and to both devices.
Since this thing does both Multicast and Unicast, our Arduino will gather data from the former and command the heat pump through the latter.
Note that Multicast support is provided to the Arduino Ethernet library via the added beginMulti() method created by Alasdair Allan.

You may wonder why I didn't just connect a relay shield to the dry contact of the heat pump in the first place.
I didn't even use the tank sensor relay command (MOREHW via Unicast).
Instead, I chose to create, destroy and recreate on the fly the internal thermostat time clock settings, according to our previous "decreasing threshold" paradigm.
In so doing, we are no more switching on and off a compressor, we are creating the setting in which the thermostat can autonomously decide to do so.
We are no longer saying "Ok go", but rather "I need 55 degrees right now. Mr. Thermostat, cope with that!".
This represents a smarter approach, for Intuition logs everything and puts all data online for us to see. I'm talking last 24 hours of thermal graphics, last 7 days of compressor scheduling and last 6 months of electrical data logging.

I put 2 cryptic variables in the sketch, that are in need of an explanation: EPC Rating and Phantom Power.

EPC Rating was intended to provide some sort of pre-heat mechanism for OWL's room thermostats, and is available through Intuition web interface.
Since it showed to be totally useless, I hijacked it for my own purpose, which is guessing the intake air temperature from the ambient and outside temperature, both retrieved via Multicast.
"A" means pure inside air, so ambient temperature carries the right information.
"G" means outside air, so outside temperature is to be taken.
Other values average between the figures and are useful if you feed your heat pump intake with, let's say, air from your garage while the boiler itself is inside (which is the most common scenario).
This approach saves a temperature sensor on one duct end, potentially too far away from Arduino to be precise. Neiter is this, of course, but at least Arduino has no sensors in it, so it can be put literally wherever you want inside your LAN.

Phantom Power refers to the supposed inability to directly measure the boiler consumption.
In a perfect world you should measure your house consumption with the 1st CMR channel and the sole boiler with the 3rd one.
This means a couple of dedicated conductors must feed the boiler and nothing else.
If you can't deploy such wiring scheme, just set an odd figure as system voltage (i.e. 231 volts) in the Intuition web interface et voilà, we have a virtual boiler consumption.

EPC Rating and Phantom Power were NOT stored in the sketch itself, because I needed to remotely play with them.

[roberto_nosfe]

The final result is a pretty neat 80% annual boiler self-sufficiency, and if you own a PV system yourself you already know how hard going above 30% is.
The remaining 20% is due to bad weather and is provided simply drawing energy from the grid, and paid back through PV net-metering.

In the last 12 months I spent a grand total of 12 euros for my DHW (net-metering doesn't come for free, in Italy) without ever experiencing a single heat shortage.
And of course, I saved 500 euros of natural gas.
It seems Arduino paid for itself...

#include "SPI.h"
#include "Ethernet.h"
#define VERBOSE //comment this to avoid compiling Serial messaging and save some program space

byte mac[] = {0x01, 0x02, 0x03, 0x04, 0x05, 0x06}; //...of Arduino
const char macId[] = "123456789012";   //...of N-OWL, see www.intuition.com
const char udpKey[] = "12345678";      //...of N-OWL, see www.intuition.com
const char tankId[] = "12345678";      //...of N-OWL, see www.intuition.com
const unsigned int capacity = 3300;    //electrical grid capacity for the user (watts)
const unsigned int basePayload =  650; //average electrical consumption of the heat pump (watts)
const byte hysteresis = 150;           //to avoid ping-ponging (watts)
const long eBegin = 9.0 * 3600;        //the time the escalation will start kicking in (seconds)
const long eEnd = 19.0 * 3600;         //nighttime mode starts here (seconds)
const byte tripDelay = 15 * 60;        //minimum time between on/off state change (seconds)
const byte wTempEsc = 7;               //water temperature escalation factor
const byte aTempEsc = 0;               //air temperature escalation factor
const byte undersampling = 3;          //electricity samples to average onto
const byte seasonFactor  = 160;        //season change escalation factor magnifier
const byte weatherFactor = 160;        //weather change escalation factor magnifier

const unsigned int PACKET_SIZE = 320; //longest OWL XML v2.0 ever recorded is 850 chars, all important data lies in the first 317
const byte DEF_W_TEMP = 35;           //reference for the average heat pump consumption, i.e. payload as it is
const byte DEF_A_TEMP = 20;           //reference for the average heat pump consumption, i.e. payload as it is
const byte MAX_PERIODS = 10;          //max no. of hot water periods OWL can take per day
const char INT_PATTERN[] = "%u";
const char LONG_PATTERN[] = "%lu";
const char EPC_RATINGS[] = "ABCDEFG";
const unsigned int WAIT_CYCLE = 2000;
const byte MULTICAST_TIMEOUT = 120;
const byte UNICAST_TIMEOUT = 10;

const char API_SEPARATOR = ',';
const char DECIMAL_SEPARATOR = '.';
const char GET[] = "GET";
const char SET[] = "SET";
const char DEL[] = "DEL";
const char SAVE[] = "SAVE";
const char OK[] = "OK";
const char CLOCK[] = "CLOCK";
const char ELECTRICITY[] = "ELECTRICITY";
const char PROPERTY[] = "PROPERTY";
const char HOTWATER[] = "HOTWATER";
const char HWDAY[] = "HWDAY";
const char HWPERIOD[] = "HWPERIOD";
const char MAC_TAG[] = "id='";
const char ELECTRICITY_TAG[] = "<electricity";
const char SOLAR_TAG[] = "<solar";
const char WEATHER_TAG[] = "<weather";
const char HOTWATER_TAG[] = "<hot_water";
const char ELECTRICITY_STR[] = "<chan id='0'><curr units='w'>";
const char SOLAR_STR[] = "<generating units='w'>";
const char WEATHER_STR[] = "code='";
const char O_TEMP_STR[] = "<temperature>";
const char I_TEMP_STR[] = "<ambient>";
const char W_TEMP_STR[] = "<current>";

const IPAddress mIP(224, 192, 32, 19); //Intuition multicast group
const unsigned int mPort = 22600;      //Intuition multicast port
IPAddress uIP;                         //Network OWL local IP
const unsigned int uPort = 5100;       //Network OWL command interface listen port
const unsigned int lPort = 9000;       //local port for UDP unicast

EthernetUDP Multicast;                 //for listening only
EthernetUDP Unicast;                   //for commanding the ICU
char buffer[PACKET_SIZE];              //for both
char *idx = buffer;
unsigned long hwPeriods[MAX_PERIODS][3]; //HW period object container
byte periodIdx;
byte ratings = strlen(EPC_RATINGS);
char sprintfField[6];

unsigned int prod;    //solar
unsigned int consReads[10];
unsigned int readsIdx;
unsigned int cons;    //consumption without heat pump
int balance;          //without heat pump (this is to be triggered against threshold)
byte phantomPower;    //if payload has to be considered always true, or just for it's initial state (i.e. before the HP kicks in); this is for dealing with scenarios where the 3rd CMR channel does NOT measure the true HP consumption
int threshold;        //the real-time evaluated threshold, to be triggered against balance
float escalation;     //how much the threshold is to be decreased by (watts/sec)
int setpoint;         //target temp of water tank i.e. the MORE H/W temperature set in the OWL web interface
int oTemp = 15;       //outside temperature, defaults to 15 deg.
int iTemp = 20;       //ambient temperature, defaults to 20 deg.
int wTemp = 12;       //water temperature, defaults to 12 deg.
int aTemp = 17;       //intake air temperature, defaults to 17 deg.
byte rating;          //as an EPC_RATINGS offset; this is used to evaluate air temp based on inside and outside temp: rating 0 --> aTemp=iTemp; rating 6 --> aTemp=oTemp
unsigned int payload = basePayload;
unsigned int dayOfYear;
unsigned int weather = 122; //defaults to overcast
byte dayOfWeek;
boolean state;               //actual state of the heat pump
boolean request;             //desired state of the heat pump
byte pending = 1;            //the api command stream id that is pending (the command stream is a monolithic command sequence than can be safely re-triggered in case of datagram collision)
unsigned long now;           //seconds since midnight
unsigned long lastSwitch;    //seconds since midnight of last state change (to avoid ping-ponging)
unsigned long lastUnicast;   //seconds of last UDP command
unsigned long lastMulticast; //seconds of last Multicast datagram
unsigned long lastMillis;    //last cycle timestamp

void setup () {
#ifdef VERBOSE
  Serial.begin(57600);
  while (!Serial);
#endif
  Ethernet.begin(mac);
  Multicast.beginMulti(mIP, mPort);
  Unicast.begin(lPort);
}

void loop() {
  if (now - lastMulticast > MULTICAST_TIMEOUT) {
    Multicast.stop();
    Unicast.stop();
    Ethernet.begin(mac);
    Multicast.beginMulti(mIP, mPort);
    Unicast.begin(lPort);
    request = true;
    pending = 251; //emergency mode trigger
    lastUnicast = 0; //immediate trigger
    lastMulticast = now; //to avoid flooding
  }

  if (uIP && now - lastUnicast > UNICAST_TIMEOUT) {
    if (pending % 250 == 1) api(GET, CLOCK);
    if (pending % 250 == 2) api(GET, HOTWATER);
    if (pending % 250 == 3) api(DEL, HWDAY, dayOfWeek, tankId);
  }
  
  if (Multicast.parsePacket()) {
    memset(buffer, 0, PACKET_SIZE);
    Multicast.read(buffer, PACKET_SIZE);
    if (received(MAC_TAG, macId)) {
      uIP = Multicast.remoteIP();
      if (received("", ELECTRICITY_TAG) && pending % 254 == 0) {
          lastMulticast = now;
          pending = 0; //eventually quit emergency mode
      }
    }
    if (!received(MAC_TAG, macId) || pending) {
      //nothing to do
    } else if (received("", ELECTRICITY_TAG)) { //---------------------------------------------------------------------------------- <electricity/>
      consReads[readsIdx % undersampling] = getValue("", ELECTRICITY_STR);
      cons = 0;
      for (byte i = 0; i < undersampling; i++)
        cons += consReads[i];
      cons /= undersampling;
      readsIdx++; //overflows without causing any harm
      cons -= payload * state * phantomPower * (cons > payload);
    } else if (received("", SOLAR_TAG)) { //----------------------------------------------------------------------------------------- <solar/>
      prod = getValue("", SOLAR_STR);
    } else if (received("", WEATHER_TAG)) { //--------------------------------------------------------------------------------------- <weather/>
      weather = getValue("", WEATHER_STR);
      oTemp = getValue("", O_TEMP_STR);
    } else if (received("", HOTWATER_TAG)) { //-------------------------------------------------------------------------------------- <hot_water/>
      iTemp = getValue(tankId, I_TEMP_STR);
      wTemp = getValue(tankId, W_TEMP_STR);
    }
  }

[roberto_nosfe]

  if (Unicast.parsePacket()) {
    memset(buffer, 0, PACKET_SIZE);
    Unicast.read(buffer, PACKET_SIZE);
#ifdef VERBOSE
      Serial.println(buffer);
#endif

    if (pending % 250 == 1) {
      if (received(OK, CLOCK)) {
        now = getValue(1);
        dayOfYear = 1 + (now / 86400 - 11) % 365; //11 leap years already
        dayOfWeek = (now / 86400 - 3) % 7;
        now %= 86400;
        lastMulticast = now;
        api(GET, ELECTRICITY);
      } else if (received(OK, ELECTRICITY)) {
        phantomPower = ((byte)getValue(2)) & 1;
        api(GET, PROPERTY);
      } else if (received(OK, PROPERTY)) {
        rating = getValue(0, EPC_RATINGS);
        api(DEL, HWDAY, dayOfWeek, tankId);
      } else if (received(OK, HWDAY)) {
        api(SAVE);
      } else if (received(OK, SAVE)) {
        for (byte i = 0; i < MAX_PERIODS; i++)
          for (byte j = 0; j < 3; j++)
            hwPeriods[i][j] = 0;
        state = false;
        pending += pending >= 250;
        pending *= pending >= 250;
      }
    }

    if (pending % 250 == 2) {
      if (received(OK, HOTWATER)) {
        setpoint = (int)getValue(2);
        turn(request);
        api(DEL, HWDAY, dayOfWeek, tankId);
        pending++;
      }
    }
    
    if (pending % 250 == 3) {
      if (received(OK, HWDAY)) {
        periodIdx = 0;
        api(SET, HWPERIOD, dayOfWeek, hwPeriods[periodIdx][0], hwPeriods[periodIdx][1], hwPeriods[periodIdx][2], tankId);
      } else if (received(OK, HWPERIOD)) {
        periodIdx++;
        if (periodIdx < MAX_PERIODS && hwPeriods[periodIdx][0] != 0)
          api(SET, HWPERIOD, dayOfWeek, hwPeriods[periodIdx][0], hwPeriods[periodIdx][1], hwPeriods[periodIdx][2], tankId);
        else
          api(SAVE);
      } else if (received(OK, SAVE)) {
        state = request;
        lastSwitch = now;
        pending += pending >= 250;
        pending *= pending >= 250;
      }
    }
  }
 
  if (!pending) {
    balance = prod - cons;
    aTemp = (iTemp * rating + oTemp * (ratings - rating - 1)) / (ratings - 1);
 
    payload = basePayload;
    payload += (wTemp - DEF_W_TEMP) * wTempEsc;
    payload -= (aTemp - DEF_A_TEMP) * aTempEsc;

    escalation = sin(float(dayOfYear + 61) / 365 * PI * 2) + 1.0;
    escalation *= float(seasonFactor) / 64;
    escalation += float(113 - constrain(weather, 113, 122) * weatherFactor) / 512;
    escalation /= 60.0;
  
    threshold = payload;
    threshold -= int(escalation * (now - eBegin)) * (now > eBegin && now < eEnd);
    threshold = max(threshold, (int)(payload - capacity)); //capacity watchdog

    request = balance >= (threshold - state * hysteresis); //hysteresis is to be considered only if hp is on

    if (request != state && now - lastSwitch >= tripDelay) {
      pending = 2;
      lastUnicast = 0; //immediate trigger
    }
  }

#ifdef VERBOSE
  updateConsole();
#endif

  now += max(0, (millis() - lastMillis) / 1000); //millis() overflow workaround
  if (now >= 86400)
    pending = 1;
  now %= 86400;
  lastMulticast = min(now, lastMulticast); //date change workaround
  lastUnicast = min(now, lastUnicast);     //date change workaround
  lastMillis = millis();
  delay(WAIT_CYCLE);
}

void turn(boolean on) {
  if (state == on)
    return;
  for (byte i = 0; i < MAX_PERIODS; i++) {
    if (!on && (i == MAX_PERIODS - 1 || hwPeriods[i + 1][0] == 0)) {
      hwPeriods[i][1] = max(now, 60);
      hwPeriods[i][2] = wTemp;
      break;
    } else if (on && hwPeriods[i][0] == 0) {
      hwPeriods[i][0] = max(now, 60);
      hwPeriods[i][1] = pending >= 250 ? 86340 : eEnd;
      hwPeriods[i][2] = setpoint;
      break;
    }
  }
}

boolean received(const char* preamble, const char* str) {
  //returns wether or not the string after the preamble is exactly str
  idx = buffer;
  byte le = strlen(preamble);
  for (; idx[0] != NULL && strncmp(idx, preamble, le) != 0; idx++);
  for (byte i = 0; idx[0] != NULL && i < le; i++, idx++);
  if (idx[0] == API_SEPARATOR) idx++;
  le = strlen(str);
  return strncmp(idx, str, le) == 0;
}

int getValue(const char* preamble, const char* beginStr) {
  //returns the int value between beginStr and the first non numerical char
  //eventually jumping after the first occurence of preamble
  idx = buffer;
  byte le = strlen(preamble);
  for (; idx[0] != NULL && strncmp(idx, preamble, le) != 0; idx++);
  for (byte i = 0; idx[0] != NULL && i < le; i++, idx++);
  le = strlen(beginStr);
  int value = 0;
  for (; idx[0] != NULL && strncmp(idx, beginStr, le) != 0; idx++);
  for (byte i = 0; idx[0] != NULL && i < le; i++, idx++);
  boolean sign = (idx[0] != '-');
  if (!sign) idx++;
  for (; idx[0] != NULL && idx[0] >= '0' && idx[0] <= '9'; idx++)
    value = value * 10 + idx[0] - 48;
  return value * ((int)sign * 2 - 1);
}

long getValue(int offset) {
  //returns the long value after the "offset"th comma after the OK,COMMAND
  idx = buffer;
  long value = 0;
  for (int i = -2; i < offset; i++, idx++)
    idx = strchr(idx, API_SEPARATOR);
  boolean sign = (idx[0] != '-');
  if (!sign) idx++;
  for (; idx[0] != NULL && idx[0] >= '0' && idx[0] <= '9'; idx++)
    value = value * 10 + idx[0] - 48;
  return value * ((int)sign * 2 - 1);
}

byte getValue(int offset, const char* pattern) {
  //returns the position inside a given charArray of the first char found after the "offset"th comma after the OK,COMMAND
  idx = buffer;
  for (int i = -2; i < offset; i++, idx++)
    idx = strchr(idx, API_SEPARATOR);
  char theChar = idx[0];
  return strcspn(pattern, &theChar);
}

void apiWrite(const char* str) {
  Unicast.write(str);
  Unicast.write(API_SEPARATOR);
#ifdef VERBOSE
  Serial.print(str);
  Serial.print(API_SEPARATOR);
#endif
}

void apiWrite(byte num) {
  sprintf(sprintfField, INT_PATTERN, num);
  Unicast.write(sprintfField);
  Unicast.write(API_SEPARATOR);
#ifdef VERBOSE
  Serial.print(sprintfField);
  Serial.print(API_SEPARATOR);
#endif
}

void apiWrite(long num) {
  sprintf(sprintfField, LONG_PATTERN, num);
  Unicast.write(sprintfField);
  Unicast.write(API_SEPARATOR);
#ifdef VERBOSE
  Serial.print(sprintfField);
  Serial.print(API_SEPARATOR);
#endif
}

boolean apiOpen() {
  return Unicast.beginPacket(uIP, uPort);
}

void apiClose() {
  Unicast.write(udpKey);
  Unicast.endPacket();
#ifdef VERBOSE
  Serial.println(udpKey);
#endif
  lastUnicast = now;
}

void api(const char* command) {
  apiOpen();
  apiWrite(command);
  apiClose();
}

void api(const char* command, const char* field) {
  apiOpen();
  apiWrite(command);
  apiWrite(field);
  apiClose();
}

void api(const char* command, const char* field, const char* subfield) {
  apiOpen();
  apiWrite(command);
  apiWrite(field);
  apiWrite(subfield);
  apiClose();
}

void api(const char* command, const char* field, byte num, const char* subfield) {
  apiOpen();
  apiWrite(command);
  apiWrite(field);
  apiWrite(num);
  apiWrite(subfield);
  apiClose();
}

void api(const char* command, const char* field, byte num, long frameInit, long frameEnd, long temp, const char* subfield) {
  apiOpen();
  apiWrite(command);
  apiWrite(field);
  apiWrite(num);
  apiWrite(frameInit);
  apiWrite(frameEnd);
  apiWrite(temp);
  apiWrite(subfield);
  apiClose();
}

#ifdef VERBOSE
void updateConsole() {
  Serial.print("--- pen ");
  Serial.print(pending);
  Serial.print(" | con ");
  Serial.print(cons);
  Serial.print(" | pro ");
  Serial.print(prod);
  Serial.print(" | bal ");
  Serial.print(balance);
  Serial.print(" | thr ");
  Serial.print(threshold);
  Serial.print(" | del ");
  Serial.print(tripDelay);
  Serial.print(" | php ");
  Serial.print(phantomPower);
  Serial.print(" | sp ");
  Serial.print(setpoint);
  Serial.print(" | wT ");
  Serial.print(wTemp);
  Serial.print(" | iT ");
  Serial.print(iTemp);
  Serial.print(" | oT ");
  Serial.print(oTemp);
  Serial.print(" | EPC ");
  Serial.print(rating);
  Serial.print(" | aT ");
  Serial.print(aTemp);
  Serial.print(" | pay ");
  Serial.print(payload);
  Serial.print(" | ");
  Serial.print(request ? "ON":"OFF");
  Serial.print(" | ");
  Serial.println(state ? "ON":"OFF");
}
#endif

[mart256]

Dude, this is awesome. Thanks for sharing.