Go Down

Topic: Benchmark STM32 vs ATMega328 (nano) vs SAM3X8E (due) vs MK20DX256 (teensy 3.2)  (Read 18501 times) previous topic - next topic

trycage

Dear All
In my research lab we are developing a custom application that required a bit of cost per performance evaluation in number crunching capabilities. The software we used to benchmark comprises simple loops, structured in a way to avoid compiler simplifications. Not very sophisticated, but it resembles many operations we are presently doing (including the direct storing of a result in one of the operators). Sure, there are more sophisticated bench around, but I like the idea to share the results.
These are the results we got, with the environment Arduino 1.6.9.

--------------------------------------------------------------------
Update - 1.01
-I introduced a modification suggested by Riva
-Westfw pointed out the different default compiler optimizations on different platforms
Now the compiler optimization is fixed to -O1, and as expected the Due is closer to the Teensy 3.2 in terms of performances
- I added the bench on the Teensy LC

05/04/2018 Update on the results
-Arduino Zero added
-Arduino Pro 1284 (24MHz) added (Thanks Budvar10)

01/05/2018
-Adafruit Metro M4 Express (samd51 @120MHz) cache on added(Thanks gdsports)


Generic STM32F103C8T6 72MHz (Cortex-M3)
INT_LOOP(30000) bench...= 2924 microseconds 10.26MIPS
LONG_LOOP(30000) bench...= 2926 microseconds 10.25MIPS
FLOAT_DIV(30000) bench...= 27979 microseconds 1.20MFLOPS
DOUBLE_DIV(30000) bench...= 38000 microseconds 0.86MFLOPS
FLOAT_MUL(30000) bench...= 20463 microseconds 1.71MFLOPS
DOUBLE_MUL(30000) bench...= 25891 microseconds 1.31MFLOPS

Arduino Nano (ATMega328 16MHz AVR)
INT_LOOP(30000) bench...= 7544 microseconds 3.98MIPS
LONG_LOOP(30000) bench...= 13408 microseconds 2.24MIPS
FLOAT_DIV(30000) bench...= 154792 microseconds 0.21MFLOPS
DOUBLE_DIV(30000) bench...= 154800 microseconds 0.21MFLOPS
FLOAT_MUL(30000) bench...= 156744 microseconds 0.21MFLOPS
DOUBLE_MUL(30000) bench...= 156736 microseconds 0.21MFLOPS

Arduino Zero (Atmel ATSAMD21G18 48MHz Cortex-M0+)
INT_LOOP(30000) bench...= 116898 microseconds 11.92MIPS
LONG_LOOP(30000) bench...= 116898 microseconds 11.93MIPS
FLOAT_DIV(30000) bench...= 116898 microseconds 0.38MFLOPS
DOUBLE_DIV(30000) bench...= 113126 microseconds 0.27MFLOPS
FLOAT_MUL(30000) bench...= 92387 microseconds 0.33MFLOPS
DOUBLE_MUL(30000) bench...= 116898 microseconds 0.26MFLOPS


Arduino Due (Atmel SAM3X8E 84 MHz Cortex-M3)
INT_LOOP(30000) bench...= 1074 microseconds 27.93MIPS
LONG_LOOP(30000) bench...= 1107 microseconds 27.10MIPS
FLOAT_DIV(30000) bench...= 25859 microseconds 1.21MFLOPS
DOUBLE_DIV(30000) bench...= 37966 microseconds 0.81MFLOPS
FLOAT_MUL(30000) bench...= 18659 microseconds 1.71MFLOPS
DOUBLE_MUL(30000) bench...= 25450 microseconds 1.23MFLOPS


Teensy LC (MKL26Z64 Cortex-M0 48MHz)
INT_LOOP(30000) bench...= 2508 microseconds 11.96MIPS
LONG_LOOP(30000) bench...= 2512 microseconds 11.94MIPS
FLOAT_DIV(30000) bench...= 76705 microseconds 0.40MFLOPS
DOUBLE_DIV(30000) bench...= 101840 microseconds 0.30MFLOPS
FLOAT_MUL(30000) bench...= 80471 microseconds 0.38MFLOPS
DOUBLE_MUL(30000) bench...= 106242 microseconds 0.29MFLOPS


Teensy 3.2 (MK20DX256 Cortex-M4 96 MHz)
INT_LOOP(30000) bench...= 940 microseconds 31.91MIPS
LONG_LOOP(30000) bench...= 944 microseconds 31.78MIPS
FLOAT_DIV(30000) bench...= 10977 microseconds 2.99MFLOPS
DOUBLE_DIV(30000) bench...= 21317 microseconds 1.47MFLOPS
FLOAT_MUL(30000) bench...= 8463 microseconds 3.99MFLOPS
DOUBLE_MUL(30000) bench...= 13162 microseconds 2.46MFLOPS


Teensy 3.2 (MK20DX256 Cortex-M4 72MHz)
INT_LOOP(30000) bench...= 1253 microseconds 23.94MIPS
LONG_LOOP(30000) bench...= 1256 microseconds 23.89MIPS
FLOAT_DIV(30000) bench...= 14635 microseconds 2.24MFLOPS
DOUBLE_DIV(30000) bench...= 25083 microseconds 1.26MFLOPS
FLOAT_MUL(30000) bench...= 11288 microseconds 2.99MFLOPS
DOUBLE_MUL(30000) bench...= 17551 microseconds 1.84MFLOPS

ESP8266 esp-12e 160MHz
INT_LOOP(30000) bench...= 752 microseconds 39.89MIPS
LONG_LOOP(30000) bench...= 751 microseconds 39.95MIPS
FLOAT_DIV(30000) bench...= 7500 microseconds 4.45MFLOPS
DOUBLE_DIV(30000) bench...= 8063 microseconds 4.10MFLOPS
FLOAT_MUL(30000) bench...= 9938 microseconds 3.27MFLOPS
DOUBLE_MUL(30000) bench...= 10688 microseconds 3.02MFLOPS


ESP8266 esp-12e 80MHz
INT_LOOP(30000) bench...= 1504 microseconds 19.95MIPS
LONG_LOOP(30000) bench...= 1501 microseconds 19.99MIPS
FLOAT_DIV(30000) bench...= 15001 microseconds 2.22MFLOPS
DOUBLE_DIV(30000) bench...= 16126 microseconds 2.05MFLOPS
FLOAT_MUL(30000) bench...= 19876 microseconds 1.63MFLOPS
DOUBLE_MUL(30000) bench...= 21377 microseconds 1.51MFLOPS



#From mantoui

teensy3.6 @180mhz
      INT_LOOP(30000) bench...= 500 microseconds 60.00MIPS
      LONG_LOOP(30000) bench...= 502 microseconds 59.76MIPS
      FLOAT_DIV(30000) bench...= 2503 microseconds 14.99MFLOPS
      DOUBLE_DIV(30000) bench...= 9343 microseconds 3.39MFLOPS
      FLOAT_MUL(30000) bench...= 667 microseconds 181.82MFLOPS
      DOUBLE_MUL(30000) bench...= 7008 microseconds 4.61MFLOPS

teensy3.6 @120mhz
     INT_LOOP(30000) bench...= 752 microseconds 39.89MIPS
     LONG_LOOP(30000) bench...= 753 microseconds 39.84MIPS
     FLOAT_DIV(30000) bench...= 3756 microseconds 9.99MFLOPS
     DOUBLE_DIV(30000) bench...= 14019 microseconds 2.26MFLOPS
     FLOAT_MUL(30000) bench...= 1001 microseconds 120.97MFLOPS
     DOUBLE_MUL(30000) bench...= 10514 microseconds 3.07MFLOPS

teensy3.5@120mhz
     INT_LOOP(30000) bench...= 752 microseconds 39.89MIPS
     LONG_LOOP(30000) bench...= 755 microseconds 39.74MIPS
     FLOAT_DIV(30000) bench...= 3758 microseconds 9.99MFLOPS
     DOUBLE_DIV(30000) bench...= 18797 microseconds 1.66MFLOPS
     FLOAT_MUL(30000) bench...= 1003 microseconds 120.97MFLOPS
     DOUBLE_MUL(30000) bench...= 10529 microseconds 3.07MFLOPS

teensy3.2@120mhz
     INT_LOOP(30000) bench...= 751 microseconds 39.95MIPS
     LONG_LOOP(30000) bench...= 755 microseconds 39.74MIPS
     FLOAT_DIV(30000) bench...= 8784 microseconds 3.74MFLOPS
     DOUBLE_DIV(30000) bench...= 17559 microseconds 1.79MFLOPS
     FLOAT_MUL(30000) bench...= 6771 microseconds 4.99MFLOPS
     DOUBLE_MUL(30000) bench...= 10533 microseconds 3.07MFLOPS

dragonfly@80MHz      
    INT_LOOP(30000) bench...= 1129 microseconds 26.57MIPS
    LONG_LOOP(30000) bench...= 1129 microseconds 26.57MIPS
    FLOAT_DIV(30000) bench...= 5641 microseconds 6.65MFLOPS
    DOUBLE_DIV(30000) bench...= 21813 microseconds 1.45MFLOPS
    FLOAT_MUL(30000) bench...= 1883 microseconds 39.79MFLOPS
    DOUBLE_MUL(30000) bench...= 16173 microseconds 1.99MFLOPS

#From Budvar10
  Arduino-PRO 1284 (ATmega1284P 24MHz)
  INT_LOOP(30000) bench...= 5024 microseconds 5.97MIPS
  LONG_LOOP(30000) bench...= 8992 microseconds 3.34MIPS
  FLOAT_DIV(30000) bench...= 96789 microseconds 0.34MFLOPS
  DOUBLE_DIV(30000) bench...= 96800 microseconds 0.34MFLOPS
  FLOAT_MUL(30000) bench...= 98058 microseconds 0.34MFLOPS
  DOUBLE_MUL(30000) bench...= 98059 microseconds 0.34MFLOPS
 
#From gdsports
  Adafruit Metro M4 Express (samd51 @120MHz) cache on
  INT_LOOP(30000) bench...= 752 microseconds 39.89MIPS
  LONG_LOOP(30000) bench...= 753 microseconds 39.84MIPS
  FLOAT_DIV(30000) bench...= 3756 microseconds 9.99MFLOPS
  DOUBLE_DIV(30000) bench...= 14022 microseconds 2.26MFLOPS
  FLOAT_MUL(30000) bench...= 1002 microseconds 120.48MFLOPS
  DOUBLE_MUL(30000) bench...= 10516 microseconds 3.07MFLOPS




the code is in attachment.

Very soon I will have a comparison of the relative typical noise in the A/D of the different platform. Indeed the Teensy platform seems to have more muscles, and  also the performance per MHz in integer operations is very solid. However, in terms of cost/performance the STM32 board is a generic clone acquired for around 2$, difficult to beat.

Cheers!



Trycage






ron_sutherland

I've not had a chance to try the Teensy 3.5 I got from Paul, but your bench test looks like an easy thing to try (thanks for posting it).

Teensy 3.5 (MK64FX512 Cortex-M4 120 MHz)
 INT_LOOP(30000) bench...= 765 microseconds 39.22MIPS
 LONG_LOOP(30000) bench...= 757 microseconds 39.63MIPS
 FLOAT_DIV(30000) bench...= 3762 microseconds 9.98MFLOPS
 DOUBLE_DIV(30000) bench...= 26316 microseconds 1.17MFLOPS
 FLOAT_MUL(30000) bench...= 1257 microseconds 60.00MFLOPS
 DOUBLE_MUL(30000) bench...= 10534 microseconds 3.07MFLOPS

NOTE: used Arduino IDE 1.6.12 with Teensyduino 1.31 beta1
I use an R-Pi Zero on an RPUpi shield to have a tool-chain at the network edge.

hansibull

#2
Oct 23, 2016, 11:02 pm Last Edit: Oct 24, 2016, 05:28 pm by hansibull Reason: Updated results from the v1.01 version
EDIT: benchmark with v1.01:
ESP8266 (Wemos D1 mini PRO) @ 80 MHz

 INT_LOOP(30000) bench...= 1504 microseconds 19.95MIPS
 LONG_LOOP(30000) bench...= 1500 microseconds 20.00MIPS
 FLOAT_DIV(30000) bench...= 15001 microseconds 2.22MFLOPS
 DOUBLE_DIV(30000) bench...= 16126 microseconds 2.05MFLOPS
 FLOAT_MUL(30000) bench...= 19876 microseconds 1.63MFLOPS
 DOUBLE_MUL(30000) bench...= 21376 microseconds 1.51MFLOPS



ESP8266 (Wemos D1 mini PRO) @ 160 MHz

 INT_LOOP(30000) bench...= 752 microseconds 39.89MIPS
 LONG_LOOP(30000) bench...= 750 microseconds 40.00MIPS
 FLOAT_DIV(30000) bench...= 7500 microseconds 4.44MFLOPS
 DOUBLE_DIV(30000) bench...= 8063 microseconds 4.10MFLOPS
 FLOAT_MUL(30000) bench...= 9938 microseconds 3.27MFLOPS
 DOUBLE_MUL(30000) bench...= 10688 microseconds 3.02MFLOPS



If anyone got an ESP32 please post your benchmarks! :)
MightyCore -  ATmega1284, mega644, mega324, mega164, mega32, mega16, mega8535
Github.com/MCUdude/MightyCore

MiniCore - ATmega8, mega48, mega88, mega168, mega328
Github.com/MCUdude/MiniCore

westfw

Quote
Arduino Due (Atmel SAM3X8E 84 MHz Cortex-M3)
 INT_LOOP(30000) bench...= 5268 microseconds 5.69MIPS
 LONG_LOOP(30000) bench...= 6712 microseconds 4.47MIPS

Teensy 3.2 (MK20DX256 Cortex-M4 96 MHz)
 INT_LOOP(30000) bench...= 956 microseconds 31.38MIPS
 LONG_LOOP(30000) bench...= 947 microseconds 31.68MIPS
Those are very strange results for the Due...

Budvar10

@rduino-PRO 1284 (ATmega1284P 24MHz)
INT_LOOP(30000) bench...= 9354 microseconds 3.21MIPS
LONG_LOOP(30000) bench...= 14240 microseconds 2.11MIPS
FLOAT_DIV(30000) bench...= 103296 microseconds 0.34MFLOPS
DOUBLE_DIV(30000) bench...= 103307 microseconds 0.34MFLOPS
FLOAT_MUL(30000) bench...= 104565 microseconds 0.33MFLOPS
DOUBLE_MUL(30000) bench...= 104576 microseconds 0.33MFLOPS
Arduino clone with ATmega1284P   http://forum.arduino.cc/index.php?topic=277260.0

Riva

Nice to have some more benchmarking tests but the test will give wrong values for the loop times as your printing several lines of text before calculating the loop duration.
I think the elapsed=micros()-elapsed; should come directly after the for loop.
Don't PM me for help as I will ignore it.

trycage

Those are very strange results for the Due...
I agree it seens quite slower. The specific loop for integers we are using is

"
 for (ic=ie; ic<(ie+30000); ic++)
 {
     
 }
"
The specific syntax avoids compiler simplifications (at least with the default compiler) flag as "ie" cannot be evaluated prior execution. It should be implemented as an integer increment and a comparison, however the Cortex-M4 has a more extended instruction set, and the M4 support saturation arithmetic, so probably the different speed is simply due to a different code generated. Hence, as usual the due is maybe not fast in doing this specific task. However, the specific iteration used is quite common in coding.



ron_sutherland

I use an R-Pi Zero on an RPUpi shield to have a tool-chain at the network edge.

westfw

Due, slightly modified; seems more reasonable.

Time (ms)...= 12083 ms
INT_LOOP(30000) bench...= 1151 microseconds 26.06MIPS
LONG_LOOP(30000) bench...= 1131 microseconds 26.53MIPS
FLOAT_DIV(30000) bench...= 28098 microseconds 1.11MFLOPS
DOUBLE_DIV(30000) bench...= 36951 microseconds 0.84MFLOPS
FLOAT_MUL(30000) bench...= 19788 microseconds 1.61MFLOPS
DOUBLE_MUL(30000) bench...= 24436 microseconds 1.29MFLOPS
-------------------------------------------

It turns out that Due compiles with optimization flag "-Os", while Teensy3 compiles with just "-O"
On AVR, -Os seems to incorporate nearly all of the useful optimizations from -O, but that doesn't seem to be the case for ARM.   With -Os, Due produces code like this for the integer loop:


Code: [Select]
 for (ic=ie; ic<(ie+30000); ic++) //this syntax avoid compiler semplifications
   8018a:    460b          mov    r3, r1
   8018c:    f501 42ea     add.w    r2, r1, #29952    ; 0x7500
   80190:    322f          adds    r2, #47    ; 0x2f
   80192:    429a          cmp    r2, r3
   80194:    db01          blt.n    8019a <loop+0x52>
   80196:    3301          adds    r3, #1
   80198:    e7f8          b.n    8018c <loop+0x44>


Notice that the branch at the end goes back to 8018c (the "add.w" instruction), so there are 5 instructions in the loop.
With -O, it does:

Code: [Select]
 for (ic=ie; ic<(ie+30000); ic++) //this syntax avoid compiler semplifications
   80186:       6823            ldr     r3, [r4, #0]
   80188:       4aa0            ldr     r2, [pc, #640]  ; (8040c <loop+0x2c4>)
   8018a:       6013            str     r3, [r2, #0]
   8018c:       f503 42ea       add.w   r2, r3, #29952  ; 0x7500
   80190:       3230            adds    r2, #48 ; 0x30
   80192:       4293            cmp     r3, r2
   80194:       da05            bge.n   801a2 <loop+0x5a>
   80196:       f247 5330       movw    r3, #30000      ; 0x7530
   8019a:       3b01            subs    r3, #1
   8019c:       d1fd            bne.n   8019a <loop+0x52>


This reduces the loop to a single instruction that it does an "equivalent" number of times, but it has to check the initial condition separately, so it's a bit bigger.  (Why they can't use the same 30000 for the add and the loop counter, I'm not sure...)  Sketch uses 28,792 bytes, vs Sketch uses 28,044 with -Os - about 3% larger...

Just for kicks, using -O3 makes for 29,528 bytes, and succeeds in completely optimizing the loops away, giving a speed of up to 769 MIPS  :-)







trycage

Nice to have some more benchmarking tests but the test will give wrong values for the loop times as your printing several lines of text before calculating the loop duration.
I think the elapsed=micros()-elapsed; should come directly after the for loop.

Thanks Riva,

I will implement your modification.

trycage

trycage

@ron_sutherland, @hansibull and @Budvar10, if it is not too much trouble
if you could re-run the latest version of bench (whihc now fixes the compiler options on every platform) I will publish your results at the top of the Post.

Thanks

Trycage

westfw

I suggest modifying a volatile variable inside the loop.


Code: [Select]
volatile byte dosomething;
  :
 for (lc=le; lc<(le+30000); lc++) //this syntax avoid compiler semplifications
  {
      dosomething = 0;
  }

Because null loops are pretty boring.  Then you won't need to be so tricky with your loops, either...

Various "long" variables used to hold timestamps should be "unsigned long"

Budvar10

...or put NOP instruction there
Code: [Select]
{
  __asm__ __volatile__("nop"); // AVR
}


EDIT: Forgot this. Several different processors - I totally missed, it was stupid idea. :)
Arduino clone with ATmega1284P   http://forum.arduino.cc/index.php?topic=277260.0

westfw

"nop" isn't guaranteed to be the right assembly on all chips.  (mind you, you'd have to be out of your mind as a chip designer not to have a "nop" instructions, but it could happen...)

Budvar10

@westfw
Yes, yes, while I realize a mistake, you've posted...
 :-*
Arduino clone with ATmega1284P   http://forum.arduino.cc/index.php?topic=277260.0

Go Up