Pages: [1] 2   Go Down
Author Topic: Gobetwino is running very slow after a few hours - anyone else had this?  (Read 2031 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi there,

I'm using Gobetwino for 2-way communication between my Arduino based kiln controller and the PC. This is a very lightly loaded connection. Temperature data is logged every few seconds, and the PC is polled for "instructions" also every few seconds. I run the connection at 9600.

I use the LOGFIL command type to add log entries to text files, and I used RFLIN commands to read a text file to pick up any instructions.

Normally things work very well. Response is virtually instantaneous, and when Gobetwino needs to print a multi-line status message, it displays at lightening speed, as you would expect.

However twice now I have left the system running overnight, and in the morning, the Gobetwino status log has slowed to a crawl, and my PCs fan is blowing a gale. In Windows Task Scheduler, Gobetwino is consuming 50% of my CPU! Yet nothing has changed at the Arduino end. Also GBT has a Command Output window, and this appears to be showing duplicates of some of the data it is returning to the Arduino after a RFLIN request. All very strange.

When I unplug the USB cable, the fan goes off, which in other circumstances might suggest serious "traffic" on the link, but as i have said the intended traffic is extremely light, and there is no handshaking that goes on in my application that might get in a loop. The Gobetwino is still successfully receiving correctly formatted polling requests from the Arduino, so the Arduino is still functionng, and not sending garbage. So I'm tentatively concluding that the GBT is getting itself in a twist, and it's not me doing it.

Anyone had similar experience? Has anyone had successful experience of running Gobetwino continuously over extended periods? Using RFLIN and LGFIL type commands?

Are you there Mikko? - I'm looking forward to the next release - any time soon?
Logged

Offline Offline
Full Member
***
Karma: 2
Posts: 212
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

it sounds like a gobetwino (which i dont know), has problems. Perhaps a memory leak.
such leaks reserve small amounts of memory inside their functions but never release it back to the system, after a short or long while systems become slow or even unstable afer a while. I once had it on mail servers there the leak was so small it took 3 months to crass a mail server.
those are hard bugs to get a grip on, because finding what leaks is not easy.

a work around might be to close and restart the program yourself (or use windows scheduler for that)

you better inform the author / programmer of gobetwino about this, he's able to change the code.
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm here, and then again....

I'm on vacation and sitting at the end of what is possibly the slowest internet connection in the world.

I will look at this problem when i'm home after the weekend.

If it happens again, could you please check the task manager on your PC before resitting things and see if it also shows GoBetwino using a lot of memory.

It would also be nice to see your Arduino code, so i can try to replicate the problem.
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

The fan going off after unplugging the USB cable could also be because your Arduino is drawing a lot of power from the USB port.

I'm not saying the GoBetwino is flawless (as a matter of fact i know it isn't :-) ) but do the electronics part of your project draw a lot of power from the USB port ?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi Mikmo,
Thanks for getting back
I've been working on other aspects so haven't experienced the problem again. I'll try and re-create it and test the memory usage as you ask.

There is no load at all on the Arduino, apart from the Ocean Controls Thermocouple Shield - i would doubt this as the cause, but I'll see if I can rule it out.

Can I safely run Gobetwino at baud rates higher than 9600?

Here are the sections of code relating to GBT

Code:
// GLOBALS

// globals for tracking processes and reading serial using PCcomm
int GBT_serInLen = SBS;
char GBT_serInString[SBS];
int GBT_pId =0;
char* GBTCommandBase[4]; // 1 + 3 log file types


// logging
  GBTCommandBase[INTERVAL_LOG]="IC";
  GBTCommandBase[TEMPERATURE_LOG]="TC";
  GBTCommandBase[CONTROL_LOG]="CTRLLOG";
 
void listenForInstruction(char*buf){
// buf must already be null - can safely do a jump return without assigning to itt
  char sbuffer[10];
  static int instSerial=0;
  int returnCode;
  buf[0]=NULL;  // anticipate failure
  returnCode=sendGobetwinoCommand("KILNINST",itoa(instSerial+1,sbuffer,10),INSTRUCTION_HEADER, INSTRUCTION_HEADER_LENGTH, "",0,1000); 
  // "" means no prescribed "Good" code(ends wait for more reads), 0 is its length, finally milliseconds to respond
  // deal with transport level return codes
  // TIMEOUTS ARE EXPECTED
  if (GBT_serInString[0]==NULL) return;
  if (returnCode&B1) alert("Bad before Instr");     // 1st bit shows garbage - warning only

  // now deal with GOBETWINO return codes
  if ((strcmp(GBT_serInString,"-2")==0) || (strcmp(GBT_serInString,"-3")==0)) return;  // gives -3 at end of file, -2 beyond end
  if (strcmp(GBT_serInString,"-1")==0) {alert("No Inst file");  return;}    // no instruction file

  instSerial+=1;                 // got a valid instruction
  strcpy(buf,GBT_serInString);   // transfer to applcation level buffer

  if (echo)  sendKDmsg("EC", GBT_serInString,CONTROL_LOG,NA);  // NB original instruction geets overwitten here
  return;
}


//Mikko - following is a little complicated, and tailored to my applcation
// a lot of the logic concerned with checking for incoming GOOD strings, but essentially
// it is a sequence of reads with 50ms delays

////////SERIAL COMMUNICATIONS  -- READING
int readSerialString(char *strArray, int siz, char*required, int rlen, char*good, int glen, long timeOut) {
   //read a string from the serial and store it in an array  - with timeout
   long startTime=millis();
   boolean timedOut=true;
   boolean garbage=false;
   char ch;
   int i=0;
   while (millis()-startTime < timeOut){
     timedOut=false;  // since we're still looking!
     if (glen && i && (glen==i) && strncmp(good,strArray,glen)==0) break;
     // if a Good string has been specified, if i>0, if we've the right number of characters to make the comparison and if the comparison succeeds ...
     if (Serial.available()>0) {            // if there's something to look at
       strArray[i] = Serial.read();       // look at it
       ch=strArray[i];                     // so as not to continually look up array, yet have strArray ready for return
       if (ch==MESSAGE_END) { Serial.print("EOMessage"); break;}
       if (ch == '\n') break;    // new line
       if (ch == '\r') break;    // carriage return
       delay(50);                         // ensure time for channel to react before next read
       if (i>=rlen) {                         // aleady got the required markers
         i++;                             // advance the index beyond the current accepted characters
         if (i>=siz-1) stop("RS1"); // for size=10, max index is 9, but 9th position must be null so limit to 8     
       }
       else if  ((rlen==2)&&(i==1)) {  // if a two character key, and we're at the second one (indx 1)     
         // can only get here if we have one of two instruction marker
         if (strArray[i]==required[1]) i++;       // instruction confirmed
         else {
           i=0 ;                          // it was just coincidence - back to try again
           garbage=true;
         }
       }
       else if (i==0) {                   // must be true or big probs   
         if (strArray[i]==required[0]) i++;       // got our first marker, now look for second (if required), otherwise ignore
         else {
           garbage=true;
         }
       }
       else  stop("RS2");
     }  // if not available, just loop
     timedOut=true;  // set in case the timeout test fails
   }
   strArray[i]= '\0';  // null on end to terminate string
   return (timedOut<<1)+garbage;   
}

////////SERIAL COMMUNICATIONS  -- WRITING

  // Gobetwino layer
 
int sendGobetwinoCommand(char* commandType, char* content,char* required, int rlen, char* good, int glen, long timeout) {
     Serial.print("#S|");
     Serial.print(commandType);
     Serial.print("|[");
     Serial.print(content);
     Serial.println("]#");
 
     // normally wait up to 1000 ms for answer from Gobetwino, answer will be in GBT_serInString , answer is 0 if all is OK
     // but using only RDFILN and LGFIL wait only up to 100 ms
     // could tailor timeout to command type
     return readSerialString(GBT_serInString ,GBT_serInLen, required, rlen, good, glen, timeout);
}



Thanks for Gobetwino.

Is your code available for customisation?

Kenny
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

There should not be any problems with running at higher speeds, i don't think it will makes things faster though, as i assume the major slice of time is taken by GoBetwino parsing command and doing the logging / file reading.

I will look at your code as time permits.

The source code it not available yet, for the simple reason that i need to clean it up and translate a very large amount of comments to english. It is my plan to make the code available at some time. Probably after the next update.
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

How large does your log files get during an "all night session" ?

How large is the file you use with the RFLIN command ?

i'm beginning to suspect that you might experience severe memory fragmentation caused by repeadetly opening large files.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi MikMo

There are two type of log files, one of which can grow to a considerable size. A single thermocouple can generate a reading every three seconds, each requiring a string of about 10 characters to be transmitted.  Then GBT adds in the time stamp (about 20 character).

That makes 30 * 24hrs a day * 60mins per hour * 60 secs per min / 3secs  = .9 MB per day

The kiln can be continuously operational for over a week in some circumstances, although typically a firing is less than 24 hours. So a maximum of say 7 MB.

There are up to eight thermocouples, but the data from each thermocouple goes to a separate log.

The file involved with RDFIL will rarely get above 100 records long.

severe memory fragmentation caused by repeadetly opening large files.

Which part of the environment do you suspect of doing the fragmenting? - GBT, Windows, something else?

Hope this helps. I will try and run GBT tonight.

Kenny
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I had a quick look at your code and i can't see anything in there that looks wrong. And it works ok for some times before the problems start.

So right now i have two "suspects" memory fragmentation caused by GoBetwino repeatedly opening large files, (even more suspicious now when you mention multiple log files), and the other suspect is the status and command output textboxes that will grow very large with your usage pattern. Maybe so large that it becomes a problem for Windows (there is a limit to the amount of text a textbox can contain)

The problem with memory fragmentation is that it is difficult to avoid

I will try to make an Arduino sketch that mimics your usage pattern and see how it runs here.

What version of Windows are you running ?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm on Windows XP.

Do you recall that I mentioned some repetition of GBT responses in the Output text box - would that symptom tend to favour any of your theories?

You mention the Windows limit. Is there possibly a delay involved from Windows when the text box gets very full,  a delay that affects GBT's ability to service the next incoming request from Arduino, so that it gets behind?

There were two log files were in use when the problem occurred, one as I described previously, and the other a reflection of all traffic between GBT and Arduino, in essence a duplicate (in "user" language) of your own log. It also would get quite large.

However I'm having difficulty in understanding why the size of the log files should make any difference - surely you are simply appending to the end of them. Would size affect the time taken to do that?

I can understand some size issues with respect to the RFLIN command, since presumably you are obliged to read through the file sequentially to access the record I requested. However as I said this file is short, so unlikely to be the culprit?

Thanks for giving it your attention
Kenny
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Appending text to the end of a file requires you to open the file, i'm currently investigating if that actually involves reading the entire file.

The filehandeling involved with the RFLIN command do read the entire file every time the command is isued. I read all the lines of the file into a string array to be able to get a specific line from the file.

The large amount of text in the two textboxes might cause problems. I hope my test's will tell my if that is what is going on.

I will change things in the upcoming version so the textboxes only hold the last 100 or so lines to eliminate this as a potential problem.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi Mikmo,

Some further data for you.

I left a session running two hours ago; When i returned, the problem had also returned with similar symptoms: input window running slowly, fan on - no duplication this time though.

I called up Task Manager, and found that GBT was using more than 50% of cpu. Memory usage was 51 Megabytes!!!! Page faults were up to 60,000,000!!!

I unplugged the USB to the Arduino. The outputs stopped immediately of course, but the input screen is still processing the inputs that must have been stacked up in the input buffer - it's now 30 minutes since I unplugged  and its still going!!!

This confirms your memory theory I think.

What is your current estimate of when the next version will be released? I would like to continue using GBT if possible, but I cant wait more than a month.

I have a few suggestions for you for the next version:
  • have a command type, like RFLIN, but which reads fixed length records - this could be opened for random access, and would thereby avoid the need to read from the start each time. Open it for shared access if possible.
  • have a new command, similar to LGFIL, which anticipates the need to constantly update the same file. Keep it open for appending until the user issues another command signalling that he needs access to it, at which point you can close it.
  • allow for the deletion of command types
  • allow for some customization of your own log - this would in many cases eliminate the need for a separate user log file - they could simply extract their data from the system log

Regards
kenny
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

A quick suggestion:

Try to disable "Show status messages" on the first tab in the settings dialog"

The (huge) memory usage could be the textboxes or the file access. If it is the file access then it's not easy to fix. It' really more a Windows problem than a GoBetwino problem.

Next version probably will not be ready in a month. It's vacation time and i'm away for a couple of weeks from this weekend.

Thanks for the suggestions, i will consider them.

Mikael
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 38
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi Mikael

Started another test before I had your latest reply - ran it without the GBT status window and without the internal GBT logging - everything running fine with my files after 8 hours.

Now I'll add back the status window and see what happens.
Kenny
Logged

Copenhagen / Denmark
Offline Offline
Edison Member
*
Karma: 5
Posts: 2360
Do it !
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

That is good news. The textbox issue will be so much simpler to fix than the memory issue.

When i'm back home i will look at a solution.

Thanks for testing.

Mikael
Logged

Pages: [1] 2   Go Up
Jump to: