NODEMCU+ESPAsyncWebServer Fails To Load Javascript Files

I'm having a problem that has me completely stumped. I have an application running on a NODEMCU, using ESPAsyncWebServer. It has been working perfectly for months. Now, rather suddenly, it has become really squirrelly, and the problem seems to be, consistently, with loading javascript files. What''s really odd is I have not touched ANY of the content files in months. The HTML, CSS, javascript and all other resources have not been changed in any way since December or January.

When I run using a Chrome client on a PC, using the Developer tools, I consistently see problems in loading the javascript files. About 99% of the time, I am getting ERR_INVALID_HTTP_RESPONSE, flagging one of the lines that loads a javascript file, like this:

 <script type="text/javascript" src="gauge.js"></script>
 <script type="text/javascript" src="dashbrd.js"></script>

There are four pages of content, each consists of a single HTML file, three CSS files, and either one or two javascript files. ONLY the javascript files are having a problem. Sometimes one of them loads, and the other does not. Sometimes, rarely, both load. Sometimes, neither loads. Always the error is ERR_INVALID_HTTP_RESPONSE.

When I run the pages locally, on the PC, I get no unexpected errors, so it is not a syntax error or similar on the content itself.

When I run the pages on the NODEMCU, and repeatedly refresh, the .js files will, perhaps one time out of every 5 or 10 refreshes, load correctly. The other times, ALWAYS, I get the ERR_INVALID_HTTP_RESPONSE on one of the .js files. The two pages that load two .js files are consistently less reliable than the other two which load only one .js file each.

If I comment out the lines in the HTML that load the javascript file(s), then the pages load correctly every time, without fail. Of course, then they don't work properly, since, much of the functionality is implemented in those scripts.

I have absolutely no clue where to even look for this, or how to debug it. I see nothing unusual happening on the NODEMCU - loop(), which is empty, continues to run, there are no crashes or exceptions. I currently have essentially all of the application code bypassed, so there is nothing running BUT the web server, yet the problem persists. What can possibly cause ONLY loading of javascript files to be affected, and only sometimes?

Is there a way to SEE the responses coming from the server on the script file loads, since the browser seems to not like the response it's getting?

Being FAR less than an HTML expert, I'm completely at a loss here....

Regards,
Ray L.

Where are these .js files stored on the NodeMCU? Hardcoded or some SPIFFS file?

You should be able to see what is being returned on Chrome via the developer's tools -> network traffic or in the console.

All files are stored in the SPIFFS filesystem.

I'm not real clear on how to read the network traffic in Chrome. I loaded the page, and it failed to load "dashbrd.js". This appears to be the request and response for that operation:

Raw Request:

GET /dashbrd.js HTTP/1.1
Host: 10.0.1.1:1234
Connection: keep-alive
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Mobile Safari/537.36
Accept: /
Referer: http://10.0.1.1:1234/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9

Parsed Request:

Accept: /
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Host: 10.0.1.1:1234
Referer: http://10.0.1.1:1234/
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Mobile Safari/537.36

Parsed Response:

Accept-Ranges: none
Content-Disposition: inline; filename="dashbrd.js"
Content-Length: 8592
Content-Type: application/javascript

Raw Response:

HTTP/1.1 200 OK
Content-Length: 8592
Content-Type: application/javascript
Content-Disposition: inline; filename="dashbrd.js"
Accept-Ranges: none

The Preview and Response windows both show only "Failed to load response data"

Does that give any clues?

Regards,
Ray L.

I do see something odd. When I look at the request and response for a successful .js file load, the file length shown in the response is exactly the same as the file length I see with File Explorer. But I notice in the above failed load, the response indicates a file length of 8592 bytes, while File Explorer shows the correct file length is actually 8566 bytes.

Could it be the SPIFFS file uploader is somehow mangling the files, so the length is not correct? I seem to recall about a month ago, I can into some bug in the 8266 uploader, and had to "update" it to a newer version. Perhaps the newer version is whacked?

Regards,
Ray L.

Oops! The length discrepancy was my bad! The SPIFFS and my PC were out of sync, from a previous debugging attempt. A few lines of the SPIFFS .js file were commented out, accounting for the incorrect length. The 8592 length is correct for the file that in the SPIFFS. After syncing the SPIFFS, the length is now correct, but the .js file still does not load most of the time.

After refreshing the page several times, it eventually loaded without error, though this time I did get something I have not seen before:

[Violation] 'load' handler took 166ms

This refers to the window.onload handler, which creates several graphical elements (graphical representations of analog gauges) on-the-fly. I have the impression this is simply a warning, not an actual problem?

I also just found I can type the URL into the browser as:

10.0.1.1:1234/dashbrd.js

And it will correctly load the .js file every time, displayed as plain text. So why is it blowing up when the file is loaded by the link block HTML? There is clearly nothing at all wrong with the file itself, and it is perfectly capable of sending the file to the browser with no problem.

I'm more confused than ever now....

Regards,
Ray L.

This seems significant, though I don't know what it means....

It seems that if I re-start the NODEMCU, or sometimes just disconnect, then re-connect to its network, the FIRST access to a given page will succeed. Subsequent accesses fail. Looking at the two requests, they are different.

Here are the request and response for the first, successful request:

GET /dashbrd.js HTTP/1.1
Referer: http://10.0.1.1:1234/dashbrd.html
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Mobile Safari/537.36

HTTP/1.1 200 OK
Content-Length: 8566
Content-Type: application/javascript
Content-Disposition: inline; filename="dashbrd.js"
Accept-Ranges: none

And here are the request and response that I see on all the failed requests:

GET /dashbrd.js HTTP/1.1
Host: 10.0.1.1:1234
Connection: keep-alive
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Mobile Safari/537.36
Accept: */*
Referer: http://10.0.1.1:1234/dashbrd.html
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9

HTTP/1.1 200 OK
Content-Length: 8566
Content-Type: application/javascript
Content-Disposition: inline; filename="dashbrd.js"
Accept-Ranges: none

Why are the requests different?

One of the pages also, at least once, generated an "Internal Server Error":

GET /median.js HTTP/1.1
Host: 10.0.1.1:1234
Connection: keep-alive
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Mobile Safari/537.36
Accept: */*
Referer: http://10.0.1.1:1234/fuelcal.html
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9

HTTP/1.1 500 Internal Server Error
Content-Length: 0
Connection: close
Accept-Ranges: none

I wish I understood more of what is going on here....

Regards,
Ray L.

Though it makes no sense, it seemed to me that loading two .js files in a single page was causing something to go sideways. So, I merged the two files into one, for each of the two pages that loaded two .js files. That seems to have things working A LOT better! I'm not sure if it's really 100%, but so far it looks ok.

So why on earth does loading two files NOT work, after it HAS worked for at least the last 4-5 months? It makes no sense at all...

Regards,
Ray L.

I have confirmed, several times, that using TWO .js files causes problems, MOST of the time. NOT all of the time. Merge the two into a single file, and the whole system works perfectly.

This makes no sense at all to me, and I struggle to come up with a logical explanation, given that I CAN reliably access the file 100% of the time, just transferring it on its own. I can come up with only a few logical explanations, but I don't know enough about how the ASyncWebServer, or the browser, actually work to have the first clue how to get to the bottom of the problem.

  1. A memory problem? Seems very unlikely to me, since I am using <50% of FLASH, and only 51% of RAM. Besides, I don't see why the NODEMCU would care about the file sizes, as it transfers them in chunks, and never needs to put them all in memory any one time. Besides, the single file is larger than either separate file. The browser has tons of memory (~7Gb free RAM on the PC).

  2. I have, once or twice, caught an Exception 29 on the NODEMCU, which is a watchdog timeout. This NEVER occurs when the single files, so it appears something is going wrong when using two files that is causing something, no doubt deep in the bowels of the server code, to hang. There are NO loops in my code that can cause that, and virtually none of my code was running at the time anyway.

Any ideas how to get to the bottom of this, without spending days, or weeks, understanding all the server code in detail?

Regards,
Ray L.

Just wondering.....

Maybe is has to do with the browser issuing multiple requests rather than sequential. I can't image the NODEMCU having so much horsepower that it can deal with multiple requests. Not at all an expert but you could imagine Chrome sending the initial request and receiving the page. Inside the page are 2 .js files so it does some sort of threading to simultaneously issue a get for each of them. the NODEMCU gets these two requests and times out or something on one of them???

With only one .js file, you never just multiple requests at the same time.

Actually, chrome does exactly that, all the time. ALL browsers do. ALL of the requests hit the server at nearly the same time, then chrome waits until they've all been satisfied or timed out. The data is returned asynchronously, in more or less random order. I often see the second of the two .js files completes loading while the first fails, though it is sometimes just the opposite. The NODEMCU is (cooperatively) multi-tasking, so this should not present a problem for it, and, indeed, it does not seem to be in any other cases. In addition to the HTML, CSS ans .js files, there are also a bunch of content files, mostly images, that need to be loaded as well, so it is always loading a dozen or so files each time any page loads. There is never a problem except with the .js files. And, as I indicated, it WAS working just fine for months with multiple .js files, so I am really mystified why it now does NOT work. I have no clue what changed, as I have not changed the server code, or any of the content in months. It is bizarre.

Regards,
Ray L.