AppLab 0.4.0 Fails Miserably Loading/Running Local Apps

Development Tools App Lab
1

This is going to be long post as to docuement the issue and what I tried.

2026-02-04 16:15:20.520 INFO - [MainThread] App:  App started
======== App is starting ============================
--- Running RAW Latency Test ---
Performing 10 test calls...
Error during ping execution 1: Request 'ping' timed out after 10s
Error during ping execution 2: Request 'ping' timed out after 10s

So tried power cycling the board and restarting applab which seemed to work:

Activating python virtual environment  <---- power off and on and then reran app
======== App is starting ============================
2026-02-04 16:17:44.680 INFO - [MainThread] App:  App started
--- Running RAW Latency Test ---
Performing 10 test calls...
787
787
787
787
787
787
787
787
787
787
--- Raw Latency Results ---
Min Latency: 89.20 ms
Max Latency: 94.52 ms
Avg Latency: 91.16 ms
----------------------------
Loop Time for 10 iterations:
16015.382942000002
----------------------------
--- Running RAW Latency Test ---
Performing 10 test calls...
  • Deleted the extra print statement and then it failed
Activating python virtual environment
======== App is starting ============================
--- Running RAW Latency Test ---
2026-02-04 16:19:29.999 INFO - [MainThread] App:  App started
Performing 10 test calls...
Error during ping execution 1: Request 'ping' timed out after 10s
...
  • Tried the cycling power and restarting applab and in debug got a major crash
[00:00:26.784,000] <err> os: ***** HARD FAULT *****
[00:00:26.790,000] <err> os:   Debug event
[00:00:26.794,000] <err> os: r0/a1:  0x00000000  r1/a2:  0x00000000  r2/a3:  0x40014800
[00:00:26.803,000] <err> os: r3/a4:  0x00002341 r12/ip:  0x00003e80 r14/lr:  0x08005507
[00:00:26.812,000] <err> os:  xpsr:  0x49000000
[00:00:26.817,000] <err> os: s[ 0]:  0x0f2f10f6  s[ 1]:  0x0801877c  s[ 2]:  0x08019a78  s[ 3]:  0x26e1cbef
[00:00:26.827,000] <err> os: s[ 4]:  0x08017fb0  s[ 5]:  0x0000790d  s[ 6]:  0x16b879a0  s[ 7]:  0x00000000
[00:00:26.838,000] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00:00:26.848,000] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00:00:26.859,000] <err> os: fpscr:  0x00000000
[00:00:26.864,000] <err> os: Faulting instruction address (r15/pc): 0x0800550c
[00:00:26.872,000] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:26.879,000] <err> os: Current thread: 0x20000558 (main)
[00:00:26.886,000] <err> os: Halting system
  • went ahead and reran anyway and got this in the debug window
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
Invalid sketch header
Monitor begin: 5016 millis
Bridge begin: 3 micros
Brige Provide: 6 millis

but~~~~ App failed to run

  • removed cache and docker container and power cycled:
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
Invalid sketch header
[00:00:26.784,000] <err> os: ***** HARD FAULT *****
[00:00:26.790,000] <err> os:   Debug event
[00:00:26.794,000] <err> os: r0/a1:  0x00000000  r1/a2:  0x00000000  r2/a3:  0x40014800
[00:00:26.803,000] <err> os: r3/a4:  0x00002341 r12/ip:  0x00003e80 r14/lr:  0x08005507
[00:00:26.812,000] <err> os:  xpsr:  0x49000000
[00:00:26.817,000] <err> os: s[ 0]:  0x0f2f10be  s[ 1]:  0x0801877c  s[ 2]:  0x08019a78  s[ 3]:  0x26e5cbef
[00:00:26.827,000] <err> os: s[ 4]:  0x08017fb0  s[ 5]:  0x0000790d  s[ 6]:  0x14b879a0  s[ 7]:  0x00000000
[00:00:26.838,000] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00:00:26.848,000] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00:00:26.859,000] <err> os: fpscr:  0x00000000
[00:00:26.864,000] <err> os: Faulting instruction address (r15/pc): 0x0800550c
[00:00:26.872,000] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:26.879,000] <err> os: Current thread: 0x20000558 (main)
[00:00:26.886,000] <err> os: Halting system
  • loaded Blink sketch from the ide which worked without the invalid header message
    Then Restarted applab and reloaded app:
DEBUG 
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
Invalid sketch header
*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
Invalid sketch header
Monitor begin: 5018 millis
Bridge begin: 3 micros
Brige Provide: 5 millis

Not sure why seeing all the reboots....
and ran the app and it worked again:

Using CPython 3.13.9 interpreter at: /usr/local/bin/python
Creating virtual environment at: .cache/.venv
Activating python virtual environment
======== App is starting ============================
2026-02-04 16:34:22.500 INFO - [MainThread] App:  App started
--- Running RAW Latency Test ---
Performing 10 test calls...
--- Raw Latency Results ---
Min Latency: 90.52 ms
Max Latency: 91.57 ms
Avg Latency: 90.97 ms
----------------------------
Loop Time for 10 iterations:
16013.567756999975
----------------------------
--- Running RAW Latency Test ---
Performing 10 test calls...
--- Raw Latency Results ---
Min Latency: 89.98 ms
Max Latency: 92.14 ms
Avg Latency: 91.17 ms
----------------------------
Loop Time for 10 iterations:
1014.6716309999988
  • and finally if I just try to re run the app with absolutely no changes to the app we back to the initial problem
Activating python virtual environment
======== App is starting ============================
--- Running RAW Latency Test ---
2026-02-04 16:35:55.390 INFO - [MainThread] App:  App started
Performing 10 test calls...
Error during ping execution 1: Request 'ping' timed out after 10s
Error during ping execution 2: Request 'ping' timed out after 10s
Error during ping execution 3: Request 'ping' timed out after 10s

Its like the sketch is not being loaded and run. Just an impression.

NOTE: This is on the 2gb but seeing the same issues on the 4gb.

Not sure what the heck is going on.

UPDATE:
Just tried to run the Blink LED example after getting the issue with latency app and got the following errors:

======== App is starting ============================
2026-02-04 17:23:24.905 INFO - [MainThread] App:  App started
Traceback (most recent call last):
File "/app/python/main.py", line 16, in <module>
App.run(user_loop=loop)
~~~~~~~^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/arduino/app_utils/app.py", line 101, in run
self.loop(user_loop)
~~~~~~~~~^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/arduino/app_utils/app.py", line 118, in loop
user_loop()
~~~~~~~~~^^
File "/app/python/main.py", line 14, in loop
Bridge.call("set_led_state", led_state)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/arduino/app_utils/bridge.py", line 64, in call
return ClientServer().call(method_name, *params, timeout=timeout)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/arduino/app_utils/bridge.py", line 342, in call
raise ValueError(f"Request '{method_name}' failed: {err_msg} ({err_code})")
ValueError: Request 'set_led_state' failed: method set_led_state not available (2)
exited with code 1

So I ran the BLINK LED sketch from ide 2.3.7 again and reran the blink led app and still getting the error. So just tried power cycling/restarted applab and got it working again.

2

Some of this feels familiar.

So far I can only throw stones and see if something works.

Wondering if you try going back to previous Zephyr release on it does it change?

Trying to remember if the new release of Zephyr is where maybe they are not including c++ at the Zephyr build? Could there be issues with static objects and order their constructors are called?

Could it be the did new of some objects in the cases that I was looking at and maybe the new operator is not working? Corrupted memory? Not dword aligned?
...

3

Can not remember the timeline here. Think it started with the release of applab 4.0, think 0,53 core -zephyr was before that and it was working???? Can not remember at this point but would be interesting to revert but not sure how to that.

Yeah doesn;t it :slight_smile:

You got me on that one

4

Well downgraded to ArduinoCore-zephyr 0.52.0 and still seeing exactly the same issues as described above!!!!

5

Ok. On the Q4 I reflashed the firmware (debian) and updated applab and going to 0.53.

The sketches ran a couple of times and then the same issue appeared. Also tried to run the test app in routerbridge library which didn't run at all.

This is getting nuts.

6

Final entry to this issue - I reflashed firmware but this time did not do any update to Applab so still running 0.3.2 also did not flash core 0.53. This time I am not having any issues with the 2 posted sketches.

UPDATE: Keeping the old applab and just updating to 0.53.0 and retesting guess what - back to the same issues with the sketch failing and having weird issues!

7

Maybe it is caused by this bug:

8

Could be - will wait to see what happens in the next update which hopefully will be soon. Thanks for the response.

9

I had the same, you should downgrade with 3 lines, otherwise some shit still there

10

@ptillisch

Ok I went ahead and incorporated that PR into the 0.53 core and rebuilt the core. Copied the files over into the core on the Q into the .Arduino15 directory.

Results. Seem to be better in terms that it I am not seeing the bus hard faults anymore but issue still persists if I run the sketch a couple of times. However, power cycling the board seems to make it recover for another 1 or 2 times.

Will have to wait until next release to really test.

Not sure what the issue really is with Bridge communication but running 0.52 seems to fix it.

11

@ptillisch

A bit of an updated. Just as an experiment decided to load the sketch part of the app via the IDE and delete the sketch.ino from applab which will then just run the python script from AppLab.

Ran the python script several times without seeing the issue rear its ugly head.

Talking out loud here but could be a weird timing issue between sketch start and python start? Think on the q forum someone was mentioning that?

Followup: Now that I not testing v3 (very large string) within AppLab the BridgeLatency App is running with issues as well. No clue

12

@pennam - it may resolve some issues, or maybe just mask them?

That is one (probably most) of my sketches which crash using the current release is not > 64K in size.

Example my Weather2...

Starting app "Weather2"
Sketch profile configured: Name="default", Port=""
The library ArduinoGraphics has been automatically added from sketch project.
The library ArxContainer has been automatically added from sketch project.
The library ArxTypeTraits has been automatically added from sketch project.
The library DebugLog has been automatically added from sketch project.
The library MsgPack has been automatically added from sketch project.
WARNING: library ArduinoGraphics claims to run on samd, renesas_uno architecture(s) and may be incompatible with your current board which runs on zephyr architecture(s).
Sketch uses 32116 bytes (1%) of program storage space. Maximum is 1966080 bytes.
Global variables use 7056 bytes (1%) of dynamic memory, leaving 516568 bytes for local variables. Maximum is 523624 bytes.
Open On-Chip Debugger 0.12.0+dev-ge6a2c12f4 (2025-05-22-15:51)
Licensed under GNU GPL v2

Which crashes:

*** Booting Zephyr OS build v4.2.0-38-g994b835f5936 ***
Invalid sketch header
[00:00:00.220,000] <err> os: ***** BUS FAULT *****
[00:00:00.225,000] <err> os:   Precise data bus error
[00:00:00.231,000] <err> os:   BFAR Address: 0x4b6e21fc
[00:00:00.237,000] <err> os: r0/a1:  0x00011b24  r1/a2:  0x4b6e21f0  r2/a3:  0x00000003
[00:00:00.245,000] <err> os: r3/a4:  0x646f6874 r12/ip:  0x00000007 r14/lr:  0x080130dd
[00:00:00.254,000] <err> os:  xpsr:  0x29002c00
[00:00:00.259,000] <err> os: s[ 0]:  0x0000200c  s[ 1]:  0x200326d8  s[ 2]:  0x00000001  s[ 3]:  0x20029420
[00:00:00.270,000] <err> os: s[ 4]:  0x00000002  s[ 5]:  0x08013155  s[ 6]:  0x00000002  s[ 7]:  0x200326d8
[00:00:00.280,000] <err> os: s[ 8]:  0x20008bd8  s[ 9]:  0x20029420  s[10]:  0x00000001  s[11]:  0x20029418
[00:00:00.291,000] <err> os: s[12]:  0xffffffff  s[13]:  0x08013477  s[14]:  0x00000000  s[15]:  0x00000001
[00:00:00.301,000] <err> os: fpscr:  0x20008bd8
[00:00:00.306,000] <err> os: Faulting instruction address (r15/pc): 0x08013012
[00:00:00.314,000] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:00:00.322,000] <err> os: Current thread: 0x20000558 (main)
[00:00:00.328,000] <err> os: Halting system

Maybe it might resolve it. Note this sketch worked on the previous release of applab and it works on my 2GB now that I downgraded zephyr on it to 0.52.

Note: I have localized it down to faulting in Bridge.begin() and it appears to be somewhere the first new of an object within it... Gets to the line before it (printk above it), does not get to printk after the new... Also I don't believe the constructor of that object is getting called, Wild Guess could easily be heap corruption... Maybe uninitialized variable? ...

13

I'm thinking it's the size of the .elf file..
the bridgetest sketch produced a 67.1KB elf file..
located inside .cache/sketch
i could be wrong and would also like to know..
~q

14

Been looking at this for a while now..
don't think this has anything to do with us..
that memory only gets allocated when there is no valid sketch read from flash..
then it's zero'd, matrix animation plays..
then it hard loops trying to read a header from it which of course is not there..
i'm thinking this is what it does when there is no sketch..
else the buffer is allocated to the len reported in the header..
idk why they bothered to increase it??
looks like Serial.flush() needs the mutex added..
changed mine too..

void arduino::ZephyrSerial::flush() {
    //added tx.sem lock..
    k_sem_take(&tx.sem, K_FOREVER);
	while (ring_buf_size_get(&tx.ringbuf) > 0) {
		k_yield();
	}
    k_sem_give(&tx.sem);//give it up..
	while (uart_irq_tx_complete(uart) == 0) {
		k_yield();
	}
}

maybe i'm wrong..
sorry.. ~q

15

I looked at it also, unclear if it uses this or not:
if ((!sketch_valid) || !(sketch_hdr->flags & SKETCH_FLAG_IMMEDIATE)) {
That is the code will be run if either there is not a valid sketch OR if the sketch that it found
does not have the SKETCH_FLAG_IMMEDIATE...

Not sure what options App lab uses in their builds:
IDE builds by default wait for the board to boot...

image
image540×593 25.6 KB

There is another option shown in the image, for Flash Mode, which on IDE defaults
to Flash, the other option is "RAM". My gut tells me that App lab uses the RAM option.
Why I believe this, is earlier I had setup the weather2 app I mentioned to run at boot, I later than programmed it with IDE 2, with a graphic test program which runs... I then rebooted a few times, and it starts up first with graphic test program and after a bit, it started up the weather app... This repeated each time I powered up the board...

16

then i don't see where they load anything into that allocated buffer..
i even checked into the matrix code..
~q

17

ok, i see it only allocated the buffer if the sketch is invalid..

if (!sketch_valid) {
			ram_firmware = (uint8_t *)malloc(SKETCH_RAM_BUFFER_LEN);
			if (!ram_firmware) {
				printk("Failed to allocate RAM for firmware\n");
				return -ENOMEM;
			}
			memset(ram_firmware, 0, SKETCH_RAM_BUFFER_LEN);
			*ram_start = (uint32_t)&ram_firmware[0];
		}

then plays ani..
then only if not a valid sketch..

while (!sketch_valid) {
			__asm__("bkpt");
			// poll the first bytes, if filled try to use them for booting
			sketch_hdr = (struct sketch_header_v1 *)(ram_firmware + 7);
			if (sketch_hdr->ver == 0x1 && sketch_hdr->magic == 0x2341) {
				// Found valid data, use it for booting
				base_addr = (uintptr_t)ram_firmware;
				*ram_start = 0;
				sketch_valid = true;
			}
		}

ram_firmware has all 0's

~q

18

I believe they use debugger to load the sketch in to the memory probably the
bkpt

19

ok, i saw the break point..
if there's is a limiting size to sketches other than flash size it needs to be documented..
128k is not enough..
the length should be known before allocation in a perfect world..

wow.. ~q

20

why is it not blowing up now on core 52 the buffer is still at 64k??
need some error checking..

while (!sketch_valid) {
  __asm__("bkpt");
  // poll the first bytes, if filled try to use them for booting
  sketch_hdr = (struct sketch_header_v1 *)(ram_firmware + 7);
  if (sketch_hdr->ver == 0x1 && sketch_hdr->magic == 0x2341) {
    // Found valid data, check size
    if (skecth_hdr->len < SKETCH_RAM_BUFFER_LEN)
    { //use it for booting
      base_addr = (uintptr_t)ram_firmware;
      *ram_start = 0;
      sketch_valid = true;
    } else
    {
      printk("Sketch too big\n");
      memset(ram_firmware, 0, SKETCH_RAM_BUFFER_LEN);
    }
  }
}

more looking. the real magic is in these 2 lines..

uint32_t *ram_start = (uint32_t *)0x20000000;

*ram_start = (uint32_t)&ram_firmware[0];

the first line create a pointer at a specific memory location..
then the location of the first byte in the newly created buffer is stored there..

then look in /tmp/remoteocd/
there's the bin that gets uploaded and mine is 67.2KB and a cfg file..

the config file reads in a 32 bit value stored at the fixed address ram_start is sitting at, which contains the first byte of the newly allocated buffer..

see that number printed in between, shutdown command invoked and python provisioning, i do believe that is the output of puts $base..
the address the bin is being dropped into..

quite genius really..

maybe allocate half the ram, load sketch, allocate exact size and move mem and free the first, otherwise quite limiting..
sorry, ranting..

~q