Reading from Ethernet and storing byte array

I am trying to read a byte array from my PHP server using the Ethernet client. I want to then pass this data into the Adafruit Thermal Printer “printBitmap” function.

Firstly, how can I download a known size of data (4608 bytes) and store this as a uint8_t?

I know I need to add the bytes into my array with imagedata[index] but client.read() only adds a single character at a time, and not the whole ‘0x00’ byte

Shall I read the data from the server as a string, and then convert it into a byte array?

I’m totally lost and would really appreciate some help just to walk me through what to do next.

Thank you

A string is just a char array will a null terminating character at the end. A byte is just an unsigned char. Unless you're talking about Strings (notice the capitalization difference), then it would be a waste of resources if you need it in the form of a byte array. Since the ethernet client reads one byte at a time, what's stopping you from putting it in the array as it comes in?

Perhaps posting what you have so far can shed some light on what you are trying to do.

How are you sending the data from php. If you send it as actual raw bytes it makes your job easier on Arduino side.

This is how I send an ip address as binary:

$formatted_ip = chr( $ipa ).chr( $ipb ).chr( $ipc ).chr( $ipd )."";

the ."" converts the result into a string.

The difference between pasting the characters straight in and sending as raw characters is shown in the links below ( from my own site ).
http://arduino.genx.biz/ip/: This is a string.
http://arduino.genx.biz/ip/?f=b: This is binary.

Once you have the data formatted correctly you can send it directly into the buffer. This is just a possible approach:

void loop(){

  static uint16_t counter = 0;
  static bool buffer_ready = false;

  if( client.available() ){
  
    byte_array[ counter++ ] = client.read();
  
    //Buffer full
    if( counter == 4608 ){
      counter = 0;
      buffer_ready = true;
    }
  }
  //Do something if buffer ready
}

If you sent the data as a string you may have to offset each value:

char c = client.read();

if( c >= '0' || c <= '9' ) c -= '0';

Then after you have 3 values say for ‘100’ you can multiply them together. Too much work in my opinion, if you use numbers/binary data just let PHP do the hard work.

I know I need to add the bytes into my array with imagedata[index] but client.read() only adds a single character at a time, and not the whole '0x00' byte

This makes no sense. A char is a byte-sized variable. The presence of the quotes around the 0x00 makes me question how you are sending the data (ASCII vs. binary).

Show your whole Arduino sketch AND the PHP code.

I am sending the data as ASCII which is probably part of the problem. This is my PHP script:

<?php

$i = 0;
$n = 0;

$response = array ("content"=>"");

if(!isset($_POST['imageData']) || empty($_POST['imageData'])) {
	$response['error'] = "Invalid Request";
	die(json_encode($response));
}

$drawing = base64_decode(substr($_POST['imageData'],22));
$im = imagecreatefromstring($drawing);

if($im) {
	imagefilter($im, IMG_FILTER_GRAYSCALE);
	imagefilter($im, IMG_FILTER_CONTRAST, -100);
	
	$height = imagesy($im);
	$width = imagesx($im);
	
	$rowBytes   = floor(($width + 7)/8);
	$totalBytes = $rowBytes * $height;
	
	for($y=0;$y<$height;$y++) {
		$response['content'] .= "\n";
		for($x=0; $x<$rowBytes; $x++) {
			$lastBit = ($x < $rowBytes - 1) ? 1 : (1 << ($rowBytes * 8 - $width));
			
			$sum = 0; // Clear accumulated 8 bits
			for($b=128; $b>=$lastBit; $b >>= 1) { // Each pixel within block...
				$row = floor($i/$width);
				$col = $i%$width;
				$i++;
				
				$rgb = imagecolorat($im,$col,$row);
				$rVal = ($rgb >> 16) & 0xFF;
				$gVal = ($rgb >> 8) & 0xFF;
				$bVal = $rgb & 0xFF;
				
				if($rVal==0) {
					$sum |= $b;
				}
			}
			$response['content'] .= "0x".sprintf("%02X",$sum);
			
			if(++$n < $totalBytes) {
				$response['content'] .= ",";
			}
		}
	}
}

$response['content'] = ltrim($response['content'],"\n");

// Update the convert code
$fh = fopen("convert.txt","w");
if($fh) {
	fwrite($fh,$response['content']);
	fclose($fh);
}

// Also create an image
imagepng($im,mktime().".png");

die(json_encode($response));

I don’t have the Arduino sketch to hand, but I have been using the Ethernet client to connect to my server and read from that convert.txt file (again, probably totally wrong)

Part of my problem is know exactly what to search for to get help because I’m not sure of how to do what I want to do. That said, I have found this SO post that may be more in the right direction:

I had also posted another threat about the wider project (including how to convert the processing sketch into PHP) and that can be found here: http://forum.arduino.cc/index.php?topic=161141

I’ll post up my Arduino sketch this evening, but at the moment it’s pretty empty apart from the Ethernet client.connect() and the loop with client.read()

Thanks for the replies so far and apologies for struggling with this.

I am sending the data as ASCII which is probably part of the problem.

No probably about it. You either need to send binary data or expect and parse ASCII data. To do that you WILL need some separator between characters.

There is NO guarantee that serial data will be delivered, so you will need to deal with the possibility that data gets lost.

Where do you send the data?

You pipe it to a file, but if you want to send it to the output, you need:

ImagePng( $im );

Make sure you cleanup:

ImageDestroy( $im );

pYro - the output also writes out to a text file:

0xFF,0xFF,0xFF,0xF0,0xCF,0xE0,0x00,0x00,x00,0x00,0x00,0x00,0x00......etc

EDIT: My sketch then tried to read this text file which I now know is the wrong way to approach this.

PaulS - it sounds like I should let PHP do as much work on the server and pass the data as binary rather than ASCII.

If this is the case, I will need to point my Arduino at a PHP page (rather than the text file) and have that send the data directly. I’ll have a go at the PHP side of things first and see if I can show you the correctly presented data. What would I need to do on the Arduino end? Should I read into a variable or just send the input directly to the printer?

Printer library is here: https://github.com/adafruit/Adafruit-Thermal-Printer-Library/blob/master/Adafruit_Thermal.cpp

specifically:

void Adafruit_Thermal::printBitmap(int w, int h, Stream *stream) {
  int rowBytes, rowBytesClipped, rowStart, chunkHeight, x, y, i, c;

  rowBytes        = (w + 7) / 8; // Round up to next byte boundary
  rowBytesClipped = (rowBytes >= 48) ? 48 : rowBytes; // 384 pixels max width

  for(rowStart=0; rowStart < h; rowStart += 255) {
    // Issue up to 255 rows at a time:
    chunkHeight = h - rowStart;
    if(chunkHeight > 255) chunkHeight = 255;

    writeBytes(18, 42, chunkHeight, rowBytesClipped);

    for(y=0; y < chunkHeight; y++) {
      for(x=0; x < rowBytesClipped; x++) {
        while((c = stream->read()) < 0);
        PRINTER_PRINT((uint8_t)c);
      }
      for(i = rowBytes - rowBytesClipped; i>0; i--) {
        while((c = stream->read()) < 0);
      }
    }
    timeoutSet(chunkHeight * dotPrintTime);
  }
  prevByte = '\n';
}

void Adafruit_Thermal::printBitmap(Stream *stream) {
  uint8_t  tmp;
  uint16_t width, height;

  tmp    =  stream->read();
  width  = (stream->read() << 8) + tmp;

  tmp    =  stream->read();
  height = (stream->read() << 8) + tmp;

  printBitmap(width, height, stream);
}

Thank you again for your responses.

This is my Adruino sketch as it stands. I realise it’s wrong based on the earlier posts, but have left it for now so you can see what I was trying to do!

#include <SPI.h>
#include <Ethernet.h>
#include <SoftwareSerial.h>
#include "Adafruit_Thermal.h"
#include <avr/pgmspace.h>   // http://arduino.cc/en/Reference/PROGMEM


byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
IPAddress ip(192,168,0,12);

EthernetClient client;
char serverName[] = "myservername.com";
char imageData[4608];


unsigned long lastConnectionTime = 0;          // last time you connected to the server, in milliseconds
const unsigned long requestInterval = 60*1000;  // delay between updates, in milliseconds
boolean lastConnected = false;                 // state of the connection last time through the main loop
boolean readingData = false;


//String imageData = "";

int printer_RX_Pin = 2;  // This is the green wire
int printer_TX_Pin = 3;  // This is the yellow wire
unsigned long index = 0;

Adafruit_Thermal printer(printer_RX_Pin, printer_TX_Pin);


void setup() {
   
   Serial.begin(9600);
   while (!Serial) {
      ; // wait for serial port to connect. Needed for Leonardo only
   }
   
   Serial.println("Attempting to get an IP address using DHCP:");
   if (!Ethernet.begin(mac)) {
      Serial.println("failed to get an IP address using DHCP, trying manually");
      Ethernet.begin(mac, ip);
   }
   Serial.print("My address:");
   Serial.println(Ethernet.localIP());

   printer.begin();

   connectToServer();
}


void loop() {
   if (client.available()) {
      char c = client.read();
      //Serial.print(c);
      
      if(c == '#' && readingData==false) {
         Serial.println("reading data");
         readingData = true;
         //index = 0;
      } else if(c == '#' && readingData) {
         Serial.print("finished reading ");
         Serial.print(index);
         Serial.println("bytes");
         readingData = false;  
      }
      
      if(readingData && c != '#' && c != '\n') {
         // build image data here
         //imageData[index] = c;
         index++;
      }
   }

   if (!client.connected() && lastConnected) {
      Serial.println();
      Serial.println("disconnecting.");
      client.stop();
      //printer.printBitmap(150, 124, imageData);
   }
   

   if(!client.connected() && (millis() - lastConnectionTime > requestInterval)) {
      connectToServer();
   }
   lastConnected = client.connected();
}



void connectToServer() {
   // attempt to connect, and wait a millisecond:
   Serial.println("connecting to server...");
   if (client.connect(serverName, 80)) {
      Serial.println("making HTTP request...");
      client.print("GET /arduino/convert.txt HTTP/1.1");
      client.print(" HTTP/1.1\r\nHost: ");
      client.println(serverName);
      client.println();
   }
   // note the time of this connect attempt:
   lastConnectionTime = millis();
}

Bumping for attention - don't leave me hanging!

I changed my PHP sketch to use the chr() function so that it looks like this:

$_POST['imageData'] = "data:image/png;base64,iVBORw0K.....(shortened)..";

$i = 0;
$n = 0;

$drawing = base64_decode(substr($_POST['imageData'],22));
$im = imagecreatefromstring($drawing);

ob_start();

if($im) {
	imagefilter($im, IMG_FILTER_GRAYSCALE);
	imagefilter($im, IMG_FILTER_CONTRAST, -100);
	
	$height = imagesy($im);
	$width = imagesx($im);
	
	$rowBytes   = floor(($width + 7)/8);
	$totalBytes = $rowBytes * $height;
	
	for($y=0;$y<$height;$y++) {
		for($x=0; $x<$rowBytes; $x++) {
			$lastBit = ($x < $rowBytes - 1) ? 1 : (1 << ($rowBytes * 8 - $width));
			
			$sum = 0; // Clear accumulated 8 bits
			for($b=128; $b>=$lastBit; $b >>= 1) { // Each pixel within block...
				$row = floor($i/$width);
				$col = $i%$width;
				$i++;
				
				$rgb = imagecolorat($im,$col,$row);
				$rVal = ($rgb >> 16) & 0xFF;
				$gVal = ($rgb >> 8) & 0xFF;
				$bVal = $rgb & 0xFF;
				
				if($rVal==0) {
					$sum |= $b;
				}
			}
			echo chr($sum);
		}
	}
}

$content = ob_get_clean();

$length = strlen($content);
header("Content-Length: $length");

echo "#".$content."#";

In my sketch, I am trying to define an array to contain the data passed from the server, but I don’t get any output from the serial monitor:

I define my array: uint8_t imageData[4608]; and fill it with imageData[index++] = c

I can manually send the commands to the printer to trigger it to receive bitmap data, but using printer.write(c) is slow (and clearly the wrong way!)

I know I am close to getting this working, but I am getting in a muddle with variable definitions and casting my data.

Did this code below return anything at all in the serial monitor ( when the line was uncommented ):

   if (client.available()) {
      char c = client.read();
      //Serial.print(c);

If you aren't getting the data correctly, maybe send the data in a different form:

header("Content-Type: application/octet-stream");

Also, open the source of the returned file in your browser ( do up a mock page to replicate the arduino side ). Then you can see the pre amble before the actual content is sent, And you can also verify the actual content.

Hi pYro, this is the output from the serial monitor:

Attempting to get an IP address using DHCP:
My address:192.168.3.2
connecting to server...
making HTTP request...
HTTP/1.1 200 OK
Date: Sat, 01 Jun 2013 11:24:36 GMT
Server: Apache
X-Powered-By: PHP/5.2.17
Content-Length: 4610
X-Powered-By: PleskLin
Connection: close
Content-Type: text/html

#aÿÿðÿÿþø??À`0ÿÀàÿÿà?að?àø`?ÀaÀ0??ð`À?>À`??`ð`|```à``ð`` ``@``?````?``@`` ``0aÿÿ``?ÿÿ`À0``?````?``l``8`?ÿà>`?ÿàg?ÀÀÀ?a`0 ÀÀ?ø`a<Àaü?ÿÿà? ÿÿÿÿÀÀÿÿ?a??À8`p À??|aàÀðÿ?ÿÿ?ÿÿ6üfgþfxÿÿøFpÿÿüÀÿÆ``À
ÿ??``À0
?aÀ?`8`Àÿð
?`?```Àÿ
?8`?`À`À@À0`?a??`ÀÀÀ0`A?À`àÀ`0`ÃÀ`àÀ ``?ÎÀ`àÀ0``ØÀ``À``ÎÀ``À``?ÇÀ`0À>`?ÃàÀ`0À@?ÁðÀ`0ÿ??À?`0?À`? 0?Àa`?ÿøà`ÀÀÀÿÿü`?ÿøà`ÀÀ@ÿÿøppÀ`À?à??#
disconnecting.

Does that look correct? It’s not printing the null bytes, but I can see that my “index” counter is returning the correct size and also if I do Serial.println(c) then I see lots of empty lines which I think is correct.

What I don’t understand are a couple of things. Firstly, how do I define the array that will hold this data? should I use

uint8_t imageData[4608];

Secondly, how do I assign the byte data into this. I have tried imageData[index++] = c; but this seems to stop my sketch from running. I don’t get anything at all from the serial monitor - not even the initial Ethernet connection responses.

Which arduino do you have? Some, such as the Uno, don't have enough RAM for such an array - in which case, you'll have to start sending to the thermal printer before all the data has arrived from the php page.

It's the Uno, but there should be enough space. The array is 4608 bytes and my sketch is 18k so there should be enough left over.

Eventually the images I pass will be larger - but I want to get a simple test working first, and then build on that.

I'm most stuck with how to populate the array - do you have any pointers?

Thanks

The array is 4608 bytes and my sketch is 18k

Apples and oranges.

The apple barrel is 2048 bytes. The orange barrel is 32K.

I’m most stuck with how to populate the array - do you have any pointers?

The way you were doing it was fine, except that you don’t have the RAM for it.

I don't understand. If I manually copy the image data into my sketch, it will happily upload onto the Arduino (and print out) - so I know that there is enough memory.
Are you saying that I am trying to exceed the size of my array? I know the total number of bytes passed via my server is 4608 (excluding the headers and the two # that I used to mark the block of image data with) - I have measured this with PHP and also in my sketch with a counter.

If the data can fit on the sketch manually - why can I not assign an array with enough room to accomodate the information being sent from my server?

If I manually copy the image data into my sketch, it will happily upload onto the Arduino (and print out) - so I know that there is enough memory.

Show us that code! If that code is storing data in PROGMEM, you will NOT be able to replicate the functionality dynamically fetching data, since PROGMEM is read-only.

An uno only has 2048 bytes of ram, maybe the manual version was cleverly converted into ldi instructions by the compiler.
However a dynamic solution will not work unless you print while the data is incoming. This will allow you to keep a buffer within the limits of the available RAM.

When sending the data, you can accommodate this feature by looping through chunks of data to send, delaying in-between. This will allow the arduino time to print off some data, making room for more.