Best approach for sending frequent packets from a sensor

Good evening all!

I have made a home power sensor based on http://openenergymonitor.org/emon which needs to take precisely one reading per second and send it to a server for persistence and real-time analytics.

My current prototype uses an ATMEGA328P-AU based on the Arduino Pro Mini 5V to take the measurements - this takes about 348-350ms to read from the two 100A legs on our home power here in Canada.

Then I use an Xbee pro S2 to send each pair of readings every second to another Xbee controller on an Arduino Uno connected to a computer via a serial connection. The computer is set up to send the data encoded as JSON to my remote server (via a REST API endpoint) where I want to store the data. The Xbee’s transmit fast and reliably and I love working with them (but they are expensive).

This solution works great - but I would like to remove the computer from the solution and ideally broadcast the readings directly from my power sensor unit to my remote server via WiFi.

I am currently testing using an ESP8266 serial WiFi solution but since the TCP communication happens synchronously and I must wait for the response from the remote server and then for the ESP8266 to respond via serial. This typically takes from 1 - 2.5 seconds which prevents me from taking a reading every second. My remote REST API is not the bottleneck - it responds very quickly, generally within 20-30ms. I’m currently using a wrapper library I found highly recommended for the ESP8266 functionality (I realize this does not work asynchronously): GitHub - chrisstubbs93/ESP8266: Various Arduino ESP8266 projects including a basic library for POST

I’d like to somehow decouple the networking delay and be able to guarantee the Arduino can take the readings every single second. I thought having about 650ms free inbetween readings would be enough to accomplish this - but with this solution it doesn’t seem so.

I’m not an EE - just a hobbyist - but have a good technical background mostly on the software side. Here are things I’m considering but have no idea what to pursue or if there is a better way:

  • Getting a second MCU to decouple/handle the networking (could potentially use what exists on the ESP8266?)
  • Getting a better base processor with networking capabilities such as TI CC3200 (may be a huge headache for me to figure out and get working - but supports TLS/SSL which would be a nice bonus)
  • Using an EEPROM or similar local storage to buffer several readings and call the network stack less frequently (maybe every 5 or 10 seconds)
  • Switch to a lighter weight TCP protocol like MQTT

I’ve made a custom PCB for my current prototype which works great (thanks @OSHPark) - and I have several friends and family for whom I’m going to create/replicate these for - so I want to keep the circuits as simple as possible and with components I’ll be able to figure out (open source stuff like the Arduino sure helps!!).

Any help would be greatly appreciated just to point me in the right direction. Thanks and have a super evening!