Hi Speed serial data transfer PC to Arduino via USB2

By "tutorial", are you referring to this:

or this:

If you meant the latter, that is not a tutorial. It is a blog post that explains the auto-reset circuit on the UNO R3 board.

When I wrote this in my previous reply:

I was referring to this tutorial:

Specifically this part:

The Mega 2560 board has the exact same auto-reset circuit as the UNO R3 board.

Yes, I was referring to the Rheingold Heavy blog, which you provided a link to, when I mentioned the 'Tutorial' in my reply,

(Actually, that site has other stuff I found useful).

Yeah, it is really well done.

It's the name of a company that makes USB bridge ICs

It’s upon first opening the serial port, then it’s fine as long as the PC maintains the connection open.

Just for curiosity, I cobbled together a quick test. I was able to send data without the Arduino resetting, and with a baudrate of 500k was able to send 32 bytes every 1ms. There is a bit of jitter sending data, there are probably ways to improve the performance with better knowledge of Windows internals.

To get the Arduino to clock out data quickly enough I wrote a direct register access version of shiftOut.

This is rough code, so use as a guide.

Arduino sketch

const byte PIN_DATA = 8;
const byte PIN_CLK = 9;
const byte PIN_SYNC = 10;

const int BUF_LEN = 32;
char buffer[BUF_LEN];

void x_shiftOut (uint8_t value)
{
  for (uint8_t bit = 0; bit < 8; bit++)
  {
    // set DATA pin
    if (value & 1)
      PORTB |= (1 << 0);
    else
      PORTB &= ~(1 << 0);

    // clock high
    PORTB |= (1 << 1);
    // delayMicroseconds (bitTimeUs);

    value = value >> 1;

    // clock low
    PORTB &= ~(1 << 1);

    // delayMicroseconds (bitTimeUs);
  }
}


void setup()
{
  Serial.begin(500000);

  pinMode(PIN_DATA, OUTPUT);
  pinMode(PIN_CLK, OUTPUT);
  pinMode(PIN_SYNC, OUTPUT);

  digitalWrite (PIN_SYNC, 1);
  delay (100);
  digitalWrite (PIN_SYNC, 0);
}

void loop()
{
  if (Serial.available())
  {
    Serial.readBytes(buffer, BUF_LEN);
    PORTB ^= (1 << 2);

    for (int j = 0; j < BUF_LEN; j++)
      x_shiftOut(buffer[j]);
  }
}

Visual Studio Community 2022 (64-bit), C++ console app

// test_serial.cpp : This file contains the 'main' function. Program execution begins and ends there.
//

#include <iostream>
#include <windows.h>

#define UART_MAX_READ_TIMEOUT          20
#define UART_DEFAULT_READ_TIMEOUT      10
#define UART_MIN_READ_TIMEOUT          5

#define MAX_FILE_LEN	512

const int BUF_LEN = 32;

typedef enum bsl_error_codes_t {
	BSL_NO_ERROR = 0x00,

	ERROR_BSL_COMINIT = 0xF3,
} bsl_error_codes_t;


typedef struct
{
	HANDLE hComm;

	unsigned int ser_interface;		// only ASC_INTERFACE supported
	unsigned int device;			// -d -dev 1..6
	unsigned int dwBaudrate;		// -b -baud 19200
	char comPort[MAX_FILE_LEN];	// -p -port COM1


} parameters_t;

bool interactive_mode = false;

parameters_t g_parameters;

double ticks_per_us;

/*------------------------------------------------------------------------------
Function Name   : sleep_us()
Description     : High Performance Counter, to put the system to sleep less than 1 ms.
Function Called : None
Input Parameter : timeUs  => Time to sleep in us.
Output Parameter: None
Return Value    : None
---------------------------------------------------------------------------------*/
void sleep_us (double timeUs)
{
	LARGE_INTEGER tick;
	double ticks_per_us;
	double start, end;

	QueryPerformanceFrequency(&tick);
	ticks_per_us = (double)tick.QuadPart / 1e6;

	QueryPerformanceCounter(&tick);
	start = (double)tick.QuadPart / ticks_per_us;
	end = start;
	while (end < (start + timeUs)) 
	{
		QueryPerformanceCounter(&tick);
		end = (double)tick.QuadPart / ticks_per_us;
	}
}

double micros(void)
{
	LARGE_INTEGER tick;

	QueryPerformanceCounter(&tick);
	return (double)tick.QuadPart / ticks_per_us;
}

/*------------------------------------------------------------------------------
Function Name   : init_uart()
Description     : Responsible to initialize the UART Port (COM PORT)
				  Following actions are done:
					-) Initialize the chosen COM PORT with the selected baudrate
					-) Return the hComm handle.
Function Called : None
Input Parameter : *cPortName     => Port Name (e.g: "COM1", "COM2", etc)
				  dwBaudrate     => Baudrate
Output Parameter: *hComm         => Valid Communication Handle.
Return Value    : Error Code
---------------------------------------------------------------------------------*/
unsigned int init_uart(HANDLE* hComm, char* cPortName, DWORD dwBaudrate)
{
	DCB				BSLdcb;
	COMMTIMEOUTS	uartCommTimeouts = { 0 };
	char			portString [200];

	uartCommTimeouts.ReadIntervalTimeout = 0;
	uartCommTimeouts.ReadTotalTimeoutConstant = 0;
	uartCommTimeouts.ReadTotalTimeoutMultiplier = UART_DEFAULT_READ_TIMEOUT;

	snprintf(portString, 200, "\\\\.\\%s", cPortName);

	*hComm = CreateFileA((const char *)portString,
		GENERIC_READ | GENERIC_WRITE,
		0,
		0,
		OPEN_EXISTING,
		FILE_FLAG_WRITE_THROUGH,
		0);

	if (*hComm == INVALID_HANDLE_VALUE)
	{
		return ERROR_BSL_COMINIT;
	}

	GetCommState(*hComm, &BSLdcb);

	// Setting for the COM1 PORT
	BSLdcb.BaudRate = dwBaudrate;
	BSLdcb.StopBits = ONESTOPBIT;
	BSLdcb.Parity = NOPARITY;
	BSLdcb.fRtsControl = RTS_CONTROL_DISABLE;
	BSLdcb.fDtrControl = DTR_CONTROL_DISABLE;
	BSLdcb.ByteSize = 8;
	BSLdcb.fOutxCtsFlow = FALSE;
	BSLdcb.fOutxDsrFlow = FALSE;
	BSLdcb.fOutX = FALSE;
	BSLdcb.fTXContinueOnXoff = TRUE;
	BSLdcb.fInX = FALSE;

	//Sleep(1);

	if (!SetCommState(*hComm, &BSLdcb))
	{
		CloseHandle(*hComm);
		return ERROR_BSL_COMINIT;
	}

	if (!SetCommMask(*hComm, EV_RXCHAR))
	{
		CloseHandle(*hComm);
		return ERROR_BSL_COMINIT;
	}

	if (!SetCommTimeouts(*hComm, &uartCommTimeouts))
	{
		CloseHandle(*hComm);
		return ERROR_BSL_COMINIT;
	}
	return BSL_NO_ERROR;
}


/*------------------------------------------------------------------------------
Function Name   : close_interface()
Description     : Responsible to close all of the communication channel (UART or JTAG)
Function Called : None
Input Parameter : *hComm        => Communication Handle.
Output Parameter: None
Return Value    : Error Code
---------------------------------------------------------------------------------*/
unsigned int close_interface(HANDLE* hComm)
{
	CloseHandle(*hComm);

	*hComm = INVALID_HANDLE_VALUE;

	return BSL_NO_ERROR;
}


void test_send(void)
{
	// attempt to connect
	if (init_uart(&g_parameters.hComm, &g_parameters.comPort[0], g_parameters.dwBaudrate) == BSL_NO_ERROR)
	{
		unsigned char chWrite[BUF_LEN];
		DWORD dwNumOfBytes = 0;

		for (int j = 0; j < BUF_LEN; j++)
			chWrite[j] = 64 + j;

		double start = micros();

		while (true)
		for (int j = 0; j < 5; j++)
		{
			WriteFile(g_parameters.hComm, &chWrite[0], BUF_LEN, &dwNumOfBytes, NULL);

			while (micros() - start < 1000)
			{
			}
			start += 1000;

			//Sleep (100);
		}
	}
	else
	{
		std::cout << "Error opening port";
	}
}

int main()
{
	std::cout << "Serial test" << std::endl;

	strcpy_s (g_parameters.comPort, "COM13");

	g_parameters.dwBaudrate = 500000;

	LARGE_INTEGER tick;
	QueryPerformanceFrequency(&tick);
	ticks_per_us = (double)tick.QuadPart / 1e6;

	test_send();
}


Scope trace :

1 Like

:+1:

:+1:

Wow. That's just about all I need to get going.
Actually, this project is not a high priority as currently I use the DIO board from Accessio. However, price is becoming an issue.
Overnight, I realised I was spending too much time on this and it was taking up time I need to be spending on other things. So I had decided to flag it for a while until I have more time to devote to this problem.

However, now with Bob's generous contribution above, I'm all keen again to follow up on what he has posted.

My requirement is to send 32 bytes in less than, or approximately, 1mSec, and to do this every 100mSec. (ie. 10Hz)

Thanks everybody for your time and suggestions.

I've now have had a chance to experiment with Bob's code.
I have some questions (surprise, surprise).

I can get the C++ code and the Arduino code to compile.
However, if I have the Arduino IDE window open, then the console app reports an error
obtaining the file handle. The call to CreateFileA fails.

If I do not open the Arduino IDE then CreateFileA succeeds.
I then can see the TX led on the Mega board flashing.
I had to change "COM13" to "COM7" to match my Mega board.
I have a Chinese clone that has a C Type connector for USB and I think
The USB chip is a CH340 (according to what I see in Windows Device Manager).

I also changed the line start += 1000; to start += 1000000; to slow down the frequency of the calls to WriteFile(g_parameters.hComm, &chWrite[0], BUF_LEN, &dwNumOfBytes, NULL);
Then I changed the baud rate to 9600 so that the TX led flash is brighter (takes more time to send the bytes).
At this point the TX led flashes brightly at about 1Hz.

My questions..

  1. I thought it should be the RX led that flashes upon receiving bytes from the PC.
  2. There seems to be some conflict with the ports ? If I open the Arduino IDE then the Console app fails to obtain a HANDLE to the device.
    If I run the Console app with the Arduino IDE closed, then the Console app succeeds in obtaining the HANDLE, and I can see the TX led flashing. Now, with the Console app running, I can open the Arduino IDE and the TX led continues flashing. However, in this situation, the Arduino app will not upload. (NOTE: My Sketch for Arduino is an empty Sketch with no code at this point
  3. In trying to understand the console app., There are some references that I cannot find any information on, even after searching all Microsoft documentation.
    For example DCB BSLdcb; I'm assuming this is one of those Windows Structures.
    also GetCommState(*hComm, &BSLdcb); Can't find information on this ?
    Simarly with SetCommState(*hComm, &BSLdcb) ???

Forget question 3 above. I have since found the information in winbase.h (more study todo)

I have been reading this thread and I hope you get your problem solved. I would just suggest that you give Ethernet of WiFi a look. It definitely solves any speed issues and it is much more robust. Just a suggestion.

It's more than likely that you have the serial monitor open in the IDE. If your communicating over serial, only one application can have the serial port open at a time.

I'm not about to get into Python. My app is a Windows Desktop app, consisting of about 20 modules, written entirely in C++. I have all the tools I need for C++.

However, Python is familiar enough to allow me to benefit from studying the examples generously recommended to me.

Thanks for your assistance.

You're absolutely right sterretje. The Serial Monitor was open. I didn't realise it was. I closed the Serial Monitor and now, with the Arduino IDE open, I can run the PC Consolde app and I don't get the error message anymore regarding the INVALID_FILE_HANDLE.
One problem solved. Moving on......

This is an interesting suggestion Frank. The Intel NUC PC's that I use do indeed have an Ethernet port as standard. However, my external hardware does not.
So, unless there is an Arduino board with Ethernet, I would have to make my own interface board. Not a huge problem. I have all the tools.
Regarding WiFi. Again, the NUC's have WiFi capability as standard. Right now I use the WiFi
to stream back to me the desktop on the remote system. I have these in several countries and am able to do upgrades and troubleshoot from my location in New Zealand.

(EDIT) I see there are Ethernet Shields available)

Yes I think C++ is good although I am not very familiar with using it in a Windows environment.
I like Python too it is especially good for rapid cross platform development, GUI's and communication apps.
I can see you are involved and whichever way you go it's going to be fun.

For those of you who may be interested, this is a video of the machine that uses our current system, sorting white asparagus. The machine is situated in Techero, North East Greece, next to the border with Turkey.
https://youtu.be/ujKW9GFNfqg?si=0GSbVFrCSIbHt5ci
This is quite an old machine which uses two cameras to image the tip and butt ends of each spear at approximately 11 spears per second. Each camera is attached to a separate computer. I now use one NUC PC with two hi-resolution USB3 cameras attached.
Each spear is graded for length, diameter, color and maturity, and then allocated to one of the chutes. When a chute has the correct number of spears in it, the door opens allowing the bunch to drop into the lower section.
There is electronic hardware controlling the conveyor cups that eject the spears and also controlling the chute doors.
The PC sends out the necessary information to the external hardware via USB2.
This project is to use an Arduino to interface between the PC and the electronic hardware. This function is currently provided by a Hi-Speed DIO card that I purchase from USA.

When I started this many years ago, I used a frame grabber that cost $12k and a camera that cost $7k. Now I use USB cameras that are only one cubic inch in size. They cost less than $1k and no frame grabber is required, such is the speed of USB3. Plus, they have a lot more capability.

Such is progress.

1 Like

I think C++ is the only choice for hardware intensive Windows Applications. There is so much C++ support from the various hardware suppliers. Often their examples are console apps, however, these are easily portable to the Windows environment.

A bit of progress..
I attached an LCD, 2 x 16 display shield to the Arduino so that I could output the result of reading the serial buffer. However, I spent a bit of time getting this to work until I realized that the LCD pins were conflicting with the ones that Bob had assigned for DATA, CLK and SYNC. So, I changed these pins to 20, 21, 22.
All good now.
I'm receiving bytes from the PC and displaying them on the LCD.

This works well with slower baud rates.
Up to 112,000 works okay, however, 128,000 and above, and I get incorrect characters.

Next move is to 'scope the pins and check latency/jitter etc.

Thinking about Frank's suggestion of using Ethernet.
That would free up the USB connection.
I see there is an Arduino Shield for ethernet.
I'll get one today.
The PC's I use have ethernet as standard.
My Laptop does not (very thin).
I have ordered an adaptor for my lapto, Type C to Ethernet.
Now end -of-financial year so must crack on with that.