Go Down

Topic: Ethernet w5200/w5500 library uses DMA + SD card DMA (success) (Read 689 times) previous topic - next topic

pav2000

Good afternoon!

Has slightly reworked the standard library wiznet. The basis for the development of DUE Zoo https://github.com/manitou48/DUEZoo

Now the library uses DMA. Supported chips w5200 w5000 (w5100 not tested)
I got the speed of the return on the internal network about 1.4 megabytes per second.

Also, I managed to get two devices to work on the same SPI bus (SD card and wiznet chip) at the same time. The map uses the standard Sdfat library. The speed of reading files is about 900 kilobytes per second. (Test.dat - the file is generated DUE manual.pdf - read from the SD card)


Using FREE RTOS for DUE, a multithreaded web server (up to 4 threads) was implemented. The demo uses 3 threads (this can be seen in debugging chrome). More threads are limited to RAM.
Everything was tested on the IDE 1.8.2+ SDK DUE 1.6.4 (on newer there are problems with stability)

The real work of the server can be found at 77.50.254.24:25401
It's a demo. While the server is running.

I hope my work will be useful.

PS I'm sorry for my bad English.

rockwallaby

I'm impressed with what you have done.
I spent some time looking at your demo system and like what I see.

It has some similarities to my own renewable energy project which runs on a Taijuino DUE, where the DUE communicates to a small ARM based Linux board (Cubieboard II) via Modbus RTU. For the Cubieboard, I developed a basic program to act as Modbus master using libmodbus as well as act as HTTP server using libmircohttpd libraries.

Now I am working on another project where I will use the DUE as the sensor and controller board and be the HTTP web server.
I have ordered a number of very small w5500 boards and will be interested to look closely at your library.
I too, also want the DUE to be able to handle the Ethernet TCP connections and also access to SD card smoothly and better than what could be achieved with w5100 with AVR Mega.

I have not looked at FREE RTOS for DUE, not yet anyhow, but I wonder what might be the benefits over traditional methods of coding. Maybe support for various DUE libraries ?

I notice each page is separate and takes time to load, and wonder if you have considered the idea of a single page application, where the whole web app gets loaded only once and then data is the only thing loaded from then on using AJAX. I see you are using AJAX for data updates.

Well done :)
_____________
Paul - VK7KPA

pav2000

#2
May 05, 2017, 08:51 am Last Edit: May 05, 2017, 09:16 am by pav2000
You guessed it -)) is a geothermal heat pump controller. The first version supports the start-stop compressor, now I run the second version with frequency control of the BLDC compressor.
The frequency converter is used, Omron MX2, link on modbus.

Regarding the operating system for the free rtos.
Any operating system solves three main tasks:
1. The division of "computer time" between tasks
2. Memory allocation between tasks
3. Division of hardware  between tasks.
Free rtos first two problems and provides tools for solving the third task (mutexes and semaphores)

How to solve the problem of "dividing" spi between two devices (wiznet and SD card) is an example in the code below:
Two switching functions are implemented
Code: [Select]
// SPI switching functions between two devices
inline void SPI_switchW5200 ()
{
  DigitalWriteDirect (4, HIGH);
  DigitalWriteDirect (10, LOW);
}

inline void SPI_switchSD ()
{
  DigitalWriteDirect (10, HIGH);
  DigitalWriteDirect (4, LOW);
}


Example of file transfer over the network:
       
Code: [Select]
  // Чтение с карты  файлов
          SPI_switchSD();
          if (!card.exists(filename))  // проверка на сущестование файла
              {
               SPI_switchW5200();
               sendConstRTOS(thread,HEADER_FILE_NOT_FOUND);
               journal.jprintf((char*)"$WARNING - Can't find %s file!\n",filename);
               return;
              } // файл не найден

          for(i=0;i<SD_REPEAT;i++)   // Делаем SD_REPEAT попыток открытия файла
          {
              if (!webFile.open(filename, O_READ))    // Карта не читатаеся
              {
                if (i>=SD_REPEAT-1)                   // Исчерпано число попыток
                 {
                  SPI_switchW5200();
                  sendConstRTOS(thread,HEADER_FILE_NOT_FOUND);
                  journal.jprintf("$ERROR - opening %s for read failed!\n",filename);
                  HP.message.setMessage(pMESSAGE_SD,(char*)"Ошибка открытия файла с SD карты",0);    // сформировать уведомление об ошибке чтения
                  HP.set_fSD(false);                                                                 // Отказ карты, работаем без нее
                  return;
                  }//if
              }
              else  break;  // Прочиталось
              vTaskDelay(50/portTICK_PERIOD_MS);
              journal.jprintf("Error opening file %s repeat open . . .\n",filename);

          }  // for

          SPI_switchW5200();         // переключение на сеть

                // Файл открыт читаем данные и кидаем в сеть
                 #ifdef LOG
                   journal.jprintf("$Thread: %d socket: %d read file: %s\n",thread,Socket[thread].sock,filename);
                 #endif
                 sendConstRTOS(thread,HEADER_FILE_WEB);
                 SPI_switchSD();
               while ((n=webFile.read(Socket[thread].outBuf,sizeof(Socket[thread].outBuf))) > 0)
                 {
                  SPI_switchW5200();
                  if (sendBufferRTOS(thread,(byte*)(Socket[thread].outBuf),n)==0) break;
                  SPI_switchSD();
                } // while
               SPI_switchSD();
               webFile.close();
               SPI_switchW5200();


From the code it is clear how the SPI is switched
But this does not solve the problem completely!
We do not forget that we have a multitasking web server, and two tasks (a web server stream) can access the SPI simultaneously.
There is still a transmission level using binary semaphores:
Example code below:
Code: [Select]
uint16_t sendPacketRTOS(uint8_t thread, const uint8_t * buf, uint16_t len,uint16_t pause)
{
  uint8_t  status=0;
  uint16_t ret=0;
  uint16_t freesize=0;
  uint8_t  s=Socket[thread].sock;

 SPI_switchW5200();

  if (len > W5100.SSIZE)  ret = W5100.SSIZE;  else ret = len;

  if (pause==0) // Честно ждем ack
   {
      do    // Ожидание освобождения буфера
      {
        xSemaphoreGive(HP.xThreadSemaphore);                                                             // Мютекс потока отдать
        taskYIELD();
        if (xSemaphoreTake(HP.xThreadSemaphore,(W5200_TIME_WAIT/portTICK_PERIOD_MS))==pdFALSE) {journal.jprintf("Socket: %d mutex is buzy\n",s); return 0;} // Захват мютекса потока или ОЖИДАНИНЕ W5200_TIME_WAIT
        taskENTER_CRITICAL();
           freesize = W5100.getTXFreeSize(s);
           if (freesize>=ret) {taskEXIT_CRITICAL(); break;}
           status = W5100.readSnSR(s);
        taskEXIT_CRITICAL();
        if ((status != SnSR::ESTABLISHED) && (status != SnSR::CLOSE_WAIT))  { ret = 0;  break;  }
       }
      while (freesize < ret);
   }
   else  // Не ждем ACK а просто делаем задержку
    {
    xSemaphoreGive(HP.xThreadSemaphore);                                                             // Мютекс потока отдать
    vTaskDelay(pause/portTICK_PERIOD_MS);                                                            // Ждем pause мсек
    if (xSemaphoreTake(HP.xThreadSemaphore,(W5200_TIME_WAIT/portTICK_PERIOD_MS))==pdFALSE) {journal.jprintf("Socket: %d mutex is buzy\n",s); return 0;} // Захват мютекса потока или ОЖИДАНИНЕ W5200_TIME_WAIT
    }

  if(GETBIT(Socket[thread].flags,fABORT_SOCK)) {SETBIT0(Socket[thread].flags,fABORT_SOCK);return 0;}              // Произошел сброс прерываем передачу

  // послать данные
  taskENTER_CRITICAL();
  W5100.send_data_processing_offset(s, 0, (uint8_t *)buf,ret);
  W5100.execCmdSn(s,Sock_SEND);
  taskEXIT_CRITICAL();


  // +2008.01 bj : reduce code
  while ( (W5100.readSnIR(s) & SnIR::SEND_OK) != SnIR::SEND_OK )
  {
    if ( W5100.readSnSR(s) == SnSR::CLOSED )
    {
      close(s);
      return 0;
    }
  }
  W5100.writeSnIR(s, SnIR::SEND_OK);

  return ret;
}


A very important point these three lines
Code: [Select]
xSemaphoreGive(HP.xThreadSemaphore);                                                             // Мютекс потока отдать
taskYIELD();
if (xSemaphoreTake(HP.xThreadSemaphore,(W5200_TIME_WAIT/portTICK_PERIOD_MS))==pdFALSE) {journal.jprintf("Socket: %d mutex is buzy\n",s); return 0;} // Захват мютекса потока или ОЖИДАНИНЕ W5200_TIME_WAIT


About the use of one web page.
Heat pumps are installed in the countryside. Internet only through the cellular network (2G 3G 4G). Speed ​​of 1-8 megabits per second, so to accelerate the loading it is interesting to "split" the files into small pieces.

Also I want to note that wiznet chips do not have a pipeline. Sending the packet the chip expects to receive an ACK before sending the next packet, and the transmission speed is highly dependent on the ping network.

In the cellular network (ping 40-50 msec) my server's transmission rate drops to 30-40 kbps. To speed up, I entered a special mode without waiting for the ACK, running on delays. The speed in this mode reaches 300 kbps. What is an acceptable result.
I did some work and put the server on a "bad" network stably.
Additional options can be found at http://77.50.254.24:25401/lan.html


Libraries and free RTOS.
Here it is necessary to look that for libraries and where are used.
Library problems when using free RTOS
1. Monopoly capture of the entire processor time. (Using delay). You need to adjust the code. But for example OneWire is friendly with free RTOS
2. "hardware" divisions - semaphores to help you
3. We always remember that another task can change the variable.


I have consistently worked 14 tasks with 5 priorities.
My record is uptime more than 23 days, then I updated the software -)






ard_newbie


You might be interested by a preemptive multitask kernel for Arduino Due:

http://perso.ensta-paristech.fr/~pessaux/alius/arduino.html#Babix

and TurboSpi for sam3x (with AHB DMA):

https://github.com/anydream/TurboSPI



rockwallaby

Yes, thanks newbie_ard  :)  (not so newbie now I guess), I saw your post on Turbo SPI the other day and have it open in another tab for checking out in the coming days.

All this is good stuff and sort of breathes some sort of new life into the DUE for me.
It would be great to see the DUE carry on and see further development as the AVR side has.

ard_newbie


Further developments on the DUE side would be welcomed. I am thinking of the USB 2.0 OTG high speed with its dedicated DMA, not a piece of cake.

Perspectives are impressive ( 480 Mbps ) but there is a hugh amount of complexity with the upper layer of USB 2.0 ( isochronous, bulk, OTG, usb hub,...) and the AHB DMA. Maybe a full time job for a team of 3 for half a year. This sort of development would be much welcomed for data logging in parallel of data acquisition thanks to the DMA.

pav2000

Newbie_ard:
I think that FREE rtos is more "stable and stable" operating system than your development.
The SPI DMA (TurboSPI) library is very useful, where you were before -)) ???
On good it should be combined with the network library.


The DUE will never reach a speed of 480 mbps. With a CPU frequency of 84 MHz, I think the speed will be 2-3 MByte. And this will be a part of the speed.
The theoretical transmission rate of SPI is 42 mbps, but there are still overheads and calculations.

For most "home" applications, the speed of 1 MB per second is sufficient.

ard_newbie

Hi pav2000,

I am not sure to understand your last post... Nevertheless the native USB port is already (without DMA) super fast compared to the programming port (at least 849 K bytes per second).

I;e. a while ago some experiments where done, see this thread reply #12:

http://forum.arduino.cc/index.php?topic=132811.0

From Sam3x datasheet:

39.2 Embedded Characteristics (page 1053)

Compatible with the USB 2.0 specification
Supports High (480Mbps), Full (12Mbps) and Low (1.5Mbps) speed communication and On-The-Go
10 pipes/endpoints
4096 bytes of Embedded Dual-Port RAM (DPRAM) for Pipes/Endpoints
Up to 3 memory banks per Pipe/Endpoint (Not for Control Pipe/Endpoint)
Flexible Pipe/Endpoint configuration and management with dedicated DMA channels
On-Chip UTMI transceiver including Pull-Ups/Pull-downs
On-Chip OTG pad including VBUS analog comparator


It is 480 M bauds per second.

This is obtained via UPLL to achieve 480 MHz (see datasheet chap. 39.4.2 page 1057).

pav2000

I will continue the topic.
I developed a special board for connecting the periphery of the heat pump. The board installs the module w5500 and arduino DUE. This allowed us to optimize the length of the conductors SPI.
Now I have reached the frequency SPI of 42 MHz, which allows you to get the speed of delivery of packets over the network to 2 MB per second. Reading a card allows you to achieve a speed of 1.6 MB per second.

I hope that this will be useful information. :)

rockwallaby

Hello pav2000, I like the looks of that board and can see some of the various parts you have on it for the different functions.
I have on my desk, a development setup using some of these parts, the Taijuino DUE with DS3231 real time clock and MCP3234 ADC on I2C, with the Wiznet5500 Ethernet module and ILI9341 320 x 240 LCD on SPI. There will be more electronics or modules added for the specific projects I will use this for.

At present, my code is not based on FreeRTOS, just plain loop based code with functions called on timer interrupt at specific time periods.

The PCB looks like it could be useful for other projects, yes ?

Your web site is looking good still, I guess you are now not in demo mode, is that correct ?
(You may see my IP address logged in your LAN file, starts with 115  :)  )

Keep up the good work and I would be keen to hear more about your progress as you develop it further.
_____________
Paul - VK7KPA

vincent84

Hello Pav2000,

Can you post your software and hardware. I was very interesting about this. I wan't to use arduino for m swimming pool and garden (communicate with ethernet, sensor, ...)

thanks

Go Up