Using online data with shift registers

Good day all,

I am a self taught PHP developer and I have now started playing with arduino.

I am trying to download data from a text file off the internet and use this information to light up a sequence of LEDs

So far I have found 2 sketches that work admirably on their own, but for my project to work, I need to "marry" them somehow.

The two sketches I am using are:

  1. Using a 74HC595 output shift register as a port-expander by Nick Gammon http://www.gammon.com.au/forum/?id=11518 (I've got this working on its own and I can submit a sequence of commands via the serial monitor to light up specific LED's)
  2. How to Download and Use Online Data with Arduino by Nathan Yau How to Download and Use Online Data with Arduino | FlowingData (I also have this working and can extract data from a text file on the internet)

In PHP I have code that creates a text file with a sequence of commands and saves this to an accessible web location.

For example, I create a text file with the line C5S9S15S (In Nick's sketch this equates to Clear All, Light up LED 5, Light up LED 9, Light up LED 15). I then read this file from the internet using Nathan's sketch. This then generates a string of characters C5S9S15S.

My problem is how do I get Nick's sketch to read my string of C5S9S15S from Nathan's sketch instead of reading the sequence of bytes C5S9S15S from the Serial monitor.

Any help would be greatly appreciated.

Is your C5S9S15S in a variable and if so, which type of variable ?
Is the data output by Nick's program in a variable and, if so, which type of variable ?

Hi UKHeliBob,

Thanks for your reply.

In Nick's sketch the sequence "C5S9S15S" is input via the serial monitor and is read as a sequence of bytes using byte c = Serial.read();
If I add the line Serial.println(c); after byte c = Serial.read();
The following is output to the serial monitor

99
53
115
57
115
49
53
115

In Nathan's sketch the sequence "C5S9S15S" is the result of reading from a text file off the internet and is stored in a String variable. If I add the code

for (int i = 0; i <= justLeds.length() - 1; i++)
{
    Serial.print((byte) justLeds[i]);
}

The following is output to the serial monitor 9953115571154953115

The same as before just in one line

The only reason that the two sets of data look different is that one is printed using Serial.print() the other with Serial.println().

If you read the characters from the text file and put each of them in a zero terminated array of chars you will have a C style string (note lowercase s not S). It looks from the snippet of code from Nathan's program is using a String to save its input. If you change it to store its data in a zero terminated string then you will be able to use Nick's program to parse and use the string. You may find Serial input basics useful in understanding how to store input in zero terminated array of chars.

Hi UKHeliBob,

Thanks for your help so far.

The only reason that the two sets of data look different is that one is printed using Serial.print() the other with Serial.println().

That was my bad - an obvious mistake which I have corrected.

I have modified Nathan's code to convert the String to a null terminated char array

String justLeds = currLeds.substring(0, currLeds.length()-5);
Serial.println("LED list from file: " + justLeds);

byte numChars = justLeds.length() + 1;
char receivedChars[numChars];
int count = 0;
for (int i = 0; i <= justLeds.length(); i++)
{
    receivedChars[count] = (byte) justLeds[i];
    count++;
}
receivedChars[count] = (byte) '\0';

A bit convoluted I know.

But when I try immediately following still nothing happens >:(

Serial.println(receivedChars);
if (Serial.available())
{
    processInput();
}

From Nick's code

void processInput ()
{
    static long receivedNumber = 0;
  
    byte c = Serial.read ();
  
    switch (c)
    {
      
        // accumulate digits
        case '0' ... '9': 
            receivedNumber *= 10;
            receivedNumber += c - '0';
            break;

        default:
            Serial.print ("Unexpected input: ");    
            Serial.println (c);
      
        // fall through to start new number
      
        // carriage-return or line-feed starts new number
        case '\r':
        case '\n':
            receivedNumber = 0;
            break;
      
        case 'A' ... 'Z':
        case 'a' ... 'z':
            processCommand (toupper (c), receivedNumber);
            receivedNumber = 0;
            break;
      
        // ignore spaces
        case ' ':
            break;
    } // end of switch on received character
}  // end of processInput

I have the impression that Nick's code interprets the bytes as they are received.

I think it would be better to receive all the bytes/chars first and then figure out what to do with them. That way it won't matter where the string of chars has come from.

Have a look at the 2nd and 3rd examples in Serial Input Basics

You could then modify Nick's processInput() function to take the data from the string of chars. Replace
byte c = Serial.read ();
with
byte c = get next char from string of received chars

...R

I have modified Nathan's code to convert the String to a null terminated char array

Why have a String in the first place ? Look at Robin's excellent examples in the thread linked to.

UKHeliBob:
Why have a String in the first place ? Look at Robin's excellent examples in the thread linked to.

I use a String because the length of the sequence of received characters is dynamic depending on which LED's I want to toggle. As far as I can determine when you create an array in C the length of the array must be defined, therefore with a String I can find the length before I convert to array of characters

String justLeds = currLeds.substring(0, currLeds.length()-5);
Serial.println("LED list from file: " + justLeds);

byte numChars = justLeds.length() + 1;
char receivedChars[numChars];

int count = 0;
for (int i = 0; i <= justLeds.length(); i++)
{
    receivedChars[count] = (byte) justLeds[i];
    count++;
}
receivedChars[count] = (byte) '\0';

I use a String because the length of the sequence of received characters is dynamic depending on which LED's I want to toggle.

OK, I understand, but by doing it that way you will fragment memory by constantly changing the size of the String which may lead to problems.

What is the maximum length of the expected user input ?

kcyster:
I use a String because the length of the sequence of received characters is dynamic depending on which LED's I want to toggle.

In the small memory of the Arduino it is better to allocate space for the largest message and use strings (small s).

...R

UKHeliBob:
What is the maximum length of the expected user input ?

Robin2:
In the small memory of the Arduino it is better to allocate space for the largest message and use strings (small s)

The maximum number of characters I will receive is 339
The maximum number of LEDs will be 112

I have reworked my sketch from the ground up to use strings, and to delimit the individual LEDs with colons so that I can read the entire string and then toggle the LEDs.

My sketch is now as follows (I have used snippets of code from Nathan, Nick, and Robin)

#include <SPI.h>

#include <Dhcp.h>
#include <Dns.h>
#include <Ethernet.h>
#include <EthernetClient.h>
#include <EthernetServer.h>
#include <EthernetUdp.h>


/** === Starting Ethernet and parsing declarations === **/
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "my.server.com";
String dataLocation = "/my/file/location/arduino.txt HTTP/1.1"; //test location

EthernetClient client;
//const int requestInterval = 900000; // ms delay between requests (15 min) live time
const int requestInterval = 60000; // ms delay between requests (15 min) test time
boolean requested;         // whether you've made a request since connecting
long lastAttemptTime = 0;  // last time you connected to the server, in ms
const byte numChars = 339;
char receivedChars[numChars];
char *parsed[112];
int ledCount = 0;
boolean newData = false;

/** === Starting LED declarations === **/
const byte LATCH = 10;
const byte numberOfChips = 2;
const byte maxLEDs = numberOfChips * 8;
byte LEDdata [numberOfChips] = { 0 };  // initial pattern
unsigned long delayAmount = 100;

void setup() 
{
    SPI.begin ();    
    Serial.begin (9600);
    Serial.println ("Starting ...");
    
    // start the Ethernet connection:
    if (Ethernet.begin(mac) == 0) 
    {
        Serial.println("Failed to configure Ethernet using DHCP");
        // no point in carrying on, so do nothing forevermore
    }
    Serial.println(Ethernet.localIP());
    // Connect to server 
    connectToServer();
}

void refreshLEDs ()
{
    digitalWrite (LATCH, LOW);
    for (int i = numberOfChips - 1; i >= 0; i--)
    {
        SPI.transfer (LEDdata [i]); 
        digitalWrite (LATCH, HIGH);
    }
} // end of refreshLEDs

void connectToServer() 
{
    // attempt to connect, and wait a millisecond:
    Serial.println("connecting to server...");
    if (client.connect(server, 80)) 
    {
        Serial.println("making HTTP request...");
       
        // make HTTP GET request to dataLocation:
        client.println("GET " + dataLocation);
    //    client.println("Host: bmi-ctn.no-ip.biz");
        client.println("Host: 192.168.165.173");
        client.println();
    }
    // note the time of this connect attempt:
    lastAttemptTime = millis();
}

// turn an LED number into the position in the array, and a bit mask
boolean getChipAndBit (unsigned int led, int & chip, byte & mask)
{
    if (led > maxLEDs)
    {
        Serial.print ("LED ");
        Serial.print (led);
        Serial.println (" too high.");
        return true;  // error
    } // end of too high
    
    led--;  // make zero relative
   
    // divide by 8 to work out which chip
    chip = led / 8;  // which chip
   
    // remainder is bit number
    mask = 1 << (led % 8);
   
    return false;  // no error
}  // end of getChipAndBit

// clear LED n (or all if zero)
void clearLED (const long n)
{
    // zero means all
    if (n == 0)
    {
        for (int i = 0; i < numberOfChips; i++) 
        {
            LEDdata [i] = 0;
        }
        return;
    }  // end of if zero
    
    int chip;
    byte mask;
    if (getChipAndBit (n, chip, mask))
    {
        return;  // bad number
    }
    
    LEDdata [chip] &= ~ mask;
}  // end of clearLED

// set LED n (or all if zero)
void setLED (const long n)
{
    // zero means all
    if (n == 0)
    {
        for (int i = 0; i < numberOfChips; i++) 
        {
            LEDdata [i] = 0xFF;
        }
        return;
    }  // end of if zero
    
    int chip;
    byte mask;
    if (getChipAndBit (n, chip, mask))
    {
        return;  // bad number
    }
    
    LEDdata [chip] |= mask;
}  // end of setLED

void loop() 
{
    recvWithStartEndMarkers();
    parseData();
}

void recvWithStartEndMarkers() 
{
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    if (client.connected())
    {
        while (client.available() > 0 && newData == false) 
        {
            rc = client.read();
    
            if (recvInProgress == true) 
            {
                if (rc != endMarker) 
                {
                    receivedChars[ndx] = rc;
                    ndx++;
                    if (ndx >= numChars) 
                    {
                        ndx = numChars - 1;
                    }
                }
                else 
                {
                    receivedChars[ndx] = '\0'; // terminate the string
                    recvInProgress = false;
                    ndx = 0;
                    newData = true;
                }
            }
            else if (rc == startMarker) 
            {
                recvInProgress = true;
            }
        }
    }
    else if (millis() - lastAttemptTime > requestInterval) 
    {
        // if you're not connected, and two minutes have passed since
        // your last connection, then attempt to connect again:
        connectToServer();
    }
}

void parseData()
{
    if (newData == true)
    {
        Serial.println("This just in ... ");
        char *str;
        char *p = receivedChars;
        while ((str = strtok_r(p, ":", &p)) != NULL)
        {
            parsed[ledCount] = str;            
            Serial.println(parsed[ledCount]);
            ledCount++;
        }
        ledCount = ledCount - 1;
        newData = false;
        lightLEDs();
    }
}

void lightLEDs()
{
    Serial.println("About to light them up ... ");
    Serial.println("Refreshing LEDs ... ");
    refreshLEDs();    
    Serial.println("Clearing ALL LEDs ... ");
    clearLED(0);

    for (int i = 0; i <= ledCount; i++)
    {
        long LED = atol(parsed[i]);
        Serial.print("Setting LED: ");
        Serial.println(LED);
        setLED(LED);
    }
}

Points to note:
On my test board I only have 2 shift registers and the code reflects this
My input sequence (from PHP) has now changed from "" to "<5:9:15>"
The output displayed on the serial monitor is what I expect

Starting ...
127.0.0.1
connecting to server...
making HTTP request...
This just in ... 
5
9
15
About to light them up ... 
Refreshing LEDs ... 
Clearing ALL LEDs ... 
Setting LED: 5
Setting LED: 9
Setting LED: 15

BUT instead of LED 5, 9 and 15 lighting up LED 9, 10, 11 and 14 are lit up

Thanks to everyone for their help.

I finally have my system working as expected.

The sketch is as follows:

#include <SPI.h>

#include <Dhcp.h>
#include <Dns.h>
#include <Ethernet.h>
#include <EthernetClient.h>
#include <EthernetServer.h>
#include <EthernetUdp.h>

/** === Starting Ethernet and parsing declarations === **/
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "my.server.com"; //internal server address
String dataLocation = "/my/file/location/arduino.txt HTTP/1.1"; //test location

EthernetClient client;
//const int requestInterval = 900000; // ms delay between requests (15 min) live time
const int requestInterval = 60000; // ms delay between requests (15 min) test time
long lastAttemptTime = 0;  // last time you connected to the server, in ms
char receivedChars[532];
char *ledsArray[160];
int ledCount = 0;
boolean newData = false;

/** === LED stuff === **/
int SER_Pin = 2;   //pin 14 on the 75HC595
int RCLK_Pin = 3;  //pin 12 on the 75HC595
int SRCLK_Pin = 4; //pin 11 on the 75HC595

//How many of the shift registers - change this
#define number_of_74hc595s 2

//do not touch
#define numOfRegisterPins number_of_74hc595s * 8

boolean registers[numOfRegisterPins];

void setup() 
{
    Serial.begin (9600);
    Serial.println("Starting ...");
    pinMode(SER_Pin, OUTPUT);
    pinMode(RCLK_Pin, OUTPUT);
    pinMode(SRCLK_Pin, OUTPUT);
   
    // start the Ethernet connection:
    if (Ethernet.begin(mac) == 0) 
    {
        Serial.println("Failed to configure Ethernet using DHCP");
        // no point in carrying on, so do nothing forevermore
    }
    // Connect to server 
    connectToServer();
}

void connectToServer() 
{
    Serial.println("Connecting to server ...");
    // attempt to connect, and wait a millisecond:
    if (client.connect(server, 80)) 
    {
        client.println("GET " + dataLocation);
        client.println("Host: my.server.com");
        client.println();
    }
    // note the time of this connect attempt:
    lastAttemptTime = millis();
}

void loop() 
{
    recvWithStartEndMarkers();
    parseData();
}

void recvWithStartEndMarkers() 
{
    static boolean recvInProgress = false;
    static int ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    if (client.connected())
    {
        if (client.available() && newData == false) 
        {
            rc = client.read();
    
            if (recvInProgress == true) 
            {
                if (rc != endMarker) 
                {
                    receivedChars[ndx] = rc;
                    ndx++;
                    if (ndx >= 532) 
                    {
                        ndx = 531;
                    }
                }
                else 
                {
                    receivedChars[ndx] = '\0'; // terminate the string
                    recvInProgress = false;
                    ndx = 0;
                    newData = true;
                    client.stop();
                }
            }
            else if (rc == startMarker) 
            {
                // reset all variables
                recvInProgress = true;
                receivedChars[0] = '\0';
                ledCount = 0;
                ledsArray[0] = '\0';
            }
        }
    }
    else if (millis() - lastAttemptTime > requestInterval) 
    {
        // if you're not connected, and two minutes have passed since
        // your last connection, then attempt to connect again:
        connectToServer();
    }
}

void parseData()
{
    if (newData == true)
    {
        Serial.println("Incoming data ...");
        Serial.println(receivedChars);
        char *str;
        char *p = receivedChars;
        while ((str = strtok_r(p, ":", &p)) != NULL)
        {
            ledsArray[ledCount] = str;            
            ledCount++;
        }
        ledCount = ledCount - 1;
        newData = false;
        lightLEDs();
    }
}

void lightLEDs()
{
    clearRegisters();
    writeRegisters();
    for (int i = 0; i <= ledCount; i++)
    {
        long LED = atol(ledsArray[i]);
        setRegisterPin((LED - 1), HIGH);
        Serial.print("Setting LED: ");
        Serial.println(LED);
    }
    writeRegisters();
}

//set all register pins to LOW
void clearRegisters()
{
    for(int i = numOfRegisterPins - 1; i >=  0; i--)
    {
        registers[i] = LOW;
    }
    writeRegisters();
} 


//Set and display registers
//Only call AFTER all values are set how you would like (slow otherwise)
void writeRegisters()
{
    digitalWrite(RCLK_Pin, LOW);
    for(int i = numOfRegisterPins - 1; i >=  0; i--)
    {
        digitalWrite(SRCLK_Pin, LOW);

        int val = registers[i];

        digitalWrite(SER_Pin, val);
        digitalWrite(SRCLK_Pin, HIGH);

    }
    digitalWrite(RCLK_Pin, HIGH);
}

//set an individual pin HIGH or LOW
void setRegisterPin(int index, int value)
{
    registers[index] = value;
}

The only glitch now is that when I set LED 8 (last pin on first register) to high it takes 17 seconds for the LEDs following to light up. However if LED 8 is not set to high the LEDs following are lit up immediately.