I am attempting to use an ESP8266 to read E1.31 (SACN) data over WiFi and then send it out to a Mega 2560 over serial. The ultimate goal is to control RGB pixels. I am programming in IDE 1.6.4. I've been able to get it to work but there is latency in the serial transmission. The longer I transmit E1.31 data, the latency increases when reading the output on the Mega. The ESP8266 is connected to Serial1 on the Mega and I am using Serial0 to print to the serial monitor over USB. I tried adjusting the serial baud rate from 115200 to 250000 but that did not seem to have any affect. Any ideas on how to reduce latency?
// enter desired universe and subnet (sACN first universe is 1)
#define DMX_SUBNET 0
#define DMX_UNIVERSE 1
#define E131_BUFFER_MAX 638
#define E131_ADDRESS_OFFSET 126
// buffers for receiving and sending data
unsigned char E131Buffer[E131_BUFFER_MAX]; //buffer to hold incoming packet,
void setup()
{
Serial.begin(115200); //initialize serial for USB serial monitor
Serial1.begin(115200); //initialize serial input from ESP8266
Serial.println("Setup Complete");
} //end setup
void E131Received(unsigned char* pbuff, int count)
{
if ( pbuff[113] == DMX_SUBNET )
{
if ( pbuff[114] == DMX_UNIVERSE )
{
if ( pbuff[E131_ADDRESS_OFFSET-1] == 0 ) //start code must be 0
{
Serial.println(pbuff[126]);
}
}
}
}
//checks to see if packet is E1.31 data
int checkE131Headers(unsigned char* messagein, int messagelength)
{
if ( messagein[1] == 0x10 && messagein[4] == 0x41 && messagein[12] == 0x37)
{
int ChannelCount = messagein[123] * 256 + messagein[124]; // number of values plus start code
return ChannelCount -1; //Return how many values are in the packet.
}
return 0;
}
void loop()
{
if (Serial1.available())
{
int packetSize = Serial1.readBytes(E131Buffer,E131_BUFFER_MAX); //read data into buffer and output size. Size should be 638 bytes.
if(packetSize)
{
int TotalChannels = checkE131Headers(E131Buffer, packetSize);
if (TotalChannels)
{
E131Received(E131Buffer, TotalChannels); //process data function
}
}
}
} //end loop
int packetSize = Serial1.readBytes(E131Buffer,E131_BUFFER_MAX);
This statement in the Mega code is going to wait until it has read 638 bytes OR the default serial timeout of 1 second has expired. Would this explain the latency you are seeing?
I believe that may be the issue. I've tried writing code that uses Serial.read() but I can't get it to work. The E1.31 packets have a start code but no end code.
Do the packets have any data in them that specifies the length? If so, could you use read() and process the length data as its received?
I've not tested it but it sounds like the timeout is restarted with each byte received. That would explain longer packets causing higher latency - for packets less than the max length, 1s is added on.
There is a method to set the timeout. If you know that the sending device sends the packets at full serial rate without gaps, you could reduce the timeout significantly.
The E1.31 packets have a start code but no end code.
That seems to be a somewhat common issue. If you can put a small delay between the transmissions from the ESP8266, then you might be able to use receiving code similar to the below.
//zoomkat 6-29-14 Simple serial echo test
//type or paste text in serial monitor and send
String readString;
void setup() {
Serial.begin(9600);
Serial.println("Simple serial echo test"); // so I can keep track of what is loaded
}
void loop() {
while (Serial.available()) {
char c = Serial.read(); //gets one byte from serial buffer
readString += c; //makes the String readString
delay(2); //slow looping to allow buffer to fill with next character
}
if (readString.length() >0) {
Serial.println(readString); //so you can see the captured String
readString="";
}
}
I've looked at the ANSI E1.31 protocol specification. The first 16 bytes of each packet are a fixed sequence of bytes. The next two bytes contain the length of the root layer packet. So it would be possible to parse the incoming data to get the length and read the next N bytes, with a short timeout in case of loss of data.
I wrote a very simple sketch that uses Serial.read() only to read the packets but I'm still having the latency issue.
unsigned char Buffer[638];
int i = 0;
void setup()
{
Serial1.begin(250000);
Serial.begin(250000);
}
void loop()
{
while (Serial1.available())
{
char c = Serial1.read(); //gets one byte from serial buffer
Buffer[i] = c; //read byte into buffer
i++; //increment buffer by 1 for next byte
if (i == 638) //received whole packet
{
Serial.println(Buffer[126]); //byte 126 = DMX channel 1 value
i = 0; //reset to get new packet
}
}
}
Can you explain a bit more about your system? Where do the packets originate? How do they get to the ESP8266? How long is the latency, and what are you using as the "start" and "finish" events? What firmware are you running on the ESP8266?
Maybe sketch out a diagram of how it all fits together - photo of hand drawn is fine.
On your latest test program, are you sure that each packet is the maximum length of 638 bytes? could you be receiving multiple packets, so that the 638 count is only reached part way through a second or third packet?
The packets originate on my computer using a program called SACN View. The packets are sent over my WiFi network unicast to the ESP8266 IP address. I am using the plugin for the Arduino IDE to program the ESP8266 using a USB FTDI chip. The plugin is here
The latency is lower when you first begin sending packets. Around a second. The longer you let the stream from the computer run, the longer the latency becomes. If I change a channel value from the computer, it takes up to about five seconds to print to the serial monitor. I am using Putty.
The RX and TX from the ESP are connected to the RX1 and TX1 on the Mega. Pins 18 and 19. I have 3.3v regulator powering the ESP and the grounds are all connected together.
The code I am running on the ESP is posted at the top of this thread. On the ESP if I print directly to the serial monitor without going through the Mega there is no latency. Also, I know the packet is 638 bytes because of udp.parsepacket(). It always returns 638.
The default serial buffer size on the ESP is 256 and I changed the Mega to match, but that does not seem to have any effect.
Here is the test code on the ESP that works great:
#include <ESP8266WiFi.h>
#include <WiFiUdp.h>
char ssid[] = "SSID";
char pass[] = "PASSWORD";
#define E131_BUFFER_SIZE 638
unsigned int localPort = 5568;
unsigned char E131Buffer[E131_BUFFER_SIZE];
WiFiUDP udp;
void setup()
{
Serial.begin(250000);
WiFi.begin(ssid, pass);
while (WiFi.status() != WL_CONNECTED)
{
delay(500);
}
udp.begin(localPort);
}
void loop()
{
int packetSize = udp.parsePacket();
if(packetSize)
{
Serial.print("Packet Size:");
Serial.println(packetSize); //print the size of the UDP packet parsed
udp.read(E131Buffer,E131_BUFFER_SIZE);
//Serial.write(E131Buffer,E131_BUFFER_SIZE);
Serial.print("Channel 1 Value:");
Serial.println(E131Buffer[126]); //print the value of DMX channel 1
}
} //end loop
So, using ESP8266 code in your original post and viewing its serial output on IDE serial monitor, result is no latency.
Using same ESP8266 code with test Mega code in reply # 8 does result in latency. Presumably the latency is in how long it takes to see the output of this statement?
Serial.println(Buffer[126]); //byte 126 = DMX channel 1 value
Maybe try adding changing that test program like this:
while (Serial1.available())
{
uint8_t c = Serial1.read(); // CHANGE THIS STATEMENT
Serial.print(c, HEX); Serial.print(" "); // ADD THESE STATEMENTS
Buffer[i] = c;
...
Is there latency in the start of the data stream being displayed?
ADDED Using the ESP8266 test code in reply # 10, you don't get latency when viewing the ESP8266 on serial monitor. I noticed that this code does not print out the packet contents:
if(packetSize)
{
Serial.print("Packet Size:");
Serial.println(packetSize); //print the size of the UDP packet parsed
udp.read(E131Buffer,E131_BUFFER_SIZE);
//Serial.write(E131Buffer,E131_BUFFER_SIZE);
Serial.print("Channel 1 Value:");
Serial.println(E131Buffer[126]); //print the value of DMX channel 1
}
Any change in latency if you restore the commented out statement?
The latency is lower when you first begin sending packets. Around a second. The longer you let the stream from the computer run, the longer the latency becomes.
Does SACN View let you send just one packet and then stop? If so, do you see the one second latency on that single packet?
Hackscribble:
Any change in latency if you restore the commented out statement?
So it looks like you are right. If I uncomment the Serial.write(E131Buffer,E131_BUFFER_SIZE); in the ESP code and print the value of one byte at the same time, there is latency. The latency is seen when writing to the serial monitor from the ESP and not even using the Mega. It seems to be slow at writing the whole buffer.
I did a quick test by just sending one byte value to the Mega and there is no latency when reading from the Mega serial monitor.
Part of ESP code:
char PacketBegin = 0x10;
static int StartCode = 0;
void loop()
{
int packetSize = udp.parsePacket(); //parse packet
if(packetSize)
{
udp.read(E131Buffer,E131_BUFFER_SIZE); //read packet into buffer
if(E131Buffer[1] == PacketBegin && E131Buffer[125] == StartCode) //quick check to see if packet is E1.31
{
Serial.write(E131Buffer[126]);
}
}
} //end loop
I was thinking I could just write the 512 channel values from the ESP to reduce the amount of data being transmitted. Bytes 126-637 in the packet.
I was thinking I could just write the 512 channel values from the ESP to reduce the amount of data being transmitted. Bytes 126-637 in the packet.
That will reduce the data by about 20%, which should help.
A rough calculation. Assuming that the serial port can actually deliver 250,000 bits/second continuously, that is 25,000 bytes/second. So at about 500 bytes per packet, that would allow 50 packets/second.
Do you know if the packets are sent out by SACN View continuously or only when a setting changes?
You could try this modification of your ESP8266 code to print out the number of packets received every 100ms. It counts the UDP packets received but discards the data. Note: compiles OK but not tested.
Hackscribble:
Do you know if the packets are sent out by SACN View continuously or only when a setting changes?
You could try this modification of your ESP8266 code to print out the number of packets received every 100ms. It counts the UDP packets received but discards the data. Note: compiles OK but not tested.
SACN View does send the packets continuously even if no data has changed. The timing code does work and I get an average of five packets every 100ms.
I tried using the code below to just send the channel data bytes but it seems to crash to the ESP after a few seconds.
void loop()
{
int packetSize = udp.parsePacket(); //parse packet
if(packetSize)
{
udp.read(E131Buffer,E131_BUFFER_SIZE); //read packet into buffer
if(E131Buffer[1] == PacketBegin && E131Buffer[125] == StartCode) //quick check to see if packet is E1.31
{
for(int i = 126; i <= E131_BUFFER_SIZE; i++)
{
Serial.write(E131Buffer[i]); //write channel values 1-512 bytes 126-637 in packet
}
}
}
} //end loop
The issue still appears to be with the ESP. I reduced the baud rate to 115200 and 57600 and there is still latency when reading the output directly from the ESP and not even using the Mega. I tried reducing the amount of bytes being written in the FOR loop and that removes the latency. I think the issue is with the serial buffer on the ESP.
I've run a test on serial performance on a Uno and an ESP8266 with the code below. No UDP involved, so not a direct comparison with your program. The code sends 125,000 bytes of data and prints the elapsed time.
On both the Uno and the ESP8266, there was no latency that I could see. Both of them output 125,000 bytes in just over 5s at 250,000 bits/second. So, well above the output rate that you need.
However, on the ESP8266, the program hung on about one attempt in four or five. So, the serial (or my USB-serial converter) is not bullet proof.
But on your problem, where the amount of serial output is much less than this test, I wonder if the interaction with handling network traffic might be causing problems? I saw this on the Github page for the ESP8266 core:
Remember that there is a lot of code that needs to run on the chip besides the sketch when WiFi is connected. WiFi and TCP/IP libraries get a chance to handle any pending events each time the loop() function completes, OR when delay(...) is called. If you have a loop somewhere in your sketch that takes a lot of time (>50ms) without calling delay(), you might consider adding a call to delay function to keep the WiFi stack running smoothly. There is also a yield() function which is equivalent to delay(0).
Might be worth testing with a call to yield() added into the for loop in your program.
Test program:
#define BLOCK_SIZE 500UL
#define NUM_BLOCKS 250UL
char buffer[BLOCK_SIZE] = {0};
void setup()
{
Serial.begin(250000);
for (uint16_t i = 0; i < BLOCK_SIZE; i++)
{
buffer[i] = (char)('A' + (i % 16));
}
pinMode(13, OUTPUT);
digitalWrite(13, HIGH);
uint32_t startTime = millis();
for (uint8_t i = 0; i < NUM_BLOCKS; i++)
{
for (uint16_t j = 0; j < BLOCK_SIZE; j++)
{
Serial.write(buffer[j]);
}
Serial.println();
}
uint32_t endTime = millis();
digitalWrite(13, LOW);
Serial.print(endTime - startTime);
Serial.print("ms for ");
Serial.print(BLOCK_SIZE * NUM_BLOCKS);
Serial.println(" bytes");
}
void loop()
{
}
That's strange that it works for you but not me. I was able to get it working by increasing the serial baud rate to 500,000. There is now no latency when reading on the Mega and I am sending almost the entire packet. I changed the code around slightly to do some verification that the data is E1.31.
Here is the code on the ESP:
#include <ESP8266WiFi.h>
#include <WiFiUdp.h>
char ssid[] = "SSID";
char pass[] = "PASSWORD";
WiFiUDP udp;
unsigned int localPort = 5568;
#define E131_BUFFER_SIZE 638
unsigned char E131Buffer[E131_BUFFER_SIZE];
void setup()
{
Serial.begin(500000); //500,000 bits per second to keep up with UDP data coming in
WiFi.begin(ssid, pass);
while (WiFi.status() != WL_CONNECTED)
{
delay(500);
}
udp.begin(localPort);
}
//checks to see if packet is E1.31 data
int checkE131Headers(unsigned char* messagein, int messagelength)
{
if ( messagein[1] == 0x10 && messagein[4] == 0x41 && messagein[12] == 0x37)
{
int ChannelCount = messagein[123] * 256 + messagein[124]; // number of values plus start code
return ChannelCount -1; //Return how many values are in the packet.
}
return 0;
}
void loop()
{
int packetSize = udp.parsePacket(); //parse packet
if(packetSize)
{
udp.read(E131Buffer,E131_BUFFER_SIZE); //read packet into buffer
int count = checkE131Headers(E131Buffer,E131_BUFFER_SIZE); //pass data to see if it is E1.31
if ( count ) //if it comes back genuine
{
for(int i = 4; i < E131_BUFFER_SIZE; i++)
{
Serial.write(E131Buffer[i]); //write bytes 4-638 in packet to serial port.
}
Serial.flush(); //flush transmit buffer at end of for loop
}
}
} //end loop
My ultimate goal is to drive RGB LED pixels using the FastLED library so that is my next step in the project. Thanks for your help!