Road to solve the delay on the Arduino IDE

Have been experienced some delays/slowdowns with the arduino ide using Windows?

Specially when you left your bluetooth serial ports enabled?, when starting or clicking on "TOOLS" menu like you can see here?:

http://screencast.com/t/aBsiowgaA3

After a while playing with the Java code of the IDE, I noticed (http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1236880178) that the problem is a bit more hard to solve than fixing the Arduino IDE, even when I solved another small problem road to my final solution ::):

http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1236999449

You can download the "fixed" Arduino 0014 file here (replace this in the Arduino/lib/ path):
http://servicios.ried.cl/arduino/temp/pde_arduino-0015_fixed_2009-03-15.rar

Well, the problem resides in the rxtx serial library, library that Arduino IDE uses to enumerate the com ports and for communication with the board. This library is open source and multi platform: http://www.rxtx.org/

After hours of trying to find the problem, I isolated the problem: the initialization routines are guilty, because even if I don't call the enumeration (clicking on the TOOLS menu) from the Arduino IDE, the delays appear with any sketch upload or turning on the serial monitor.

The problem begin from this code, in the SerialImp.c file:

#ifndef WIN32
      pid = getpid();
#else
      char full_windows_name[80]; 
#endif /* WIN32 */

      ENTER( "RXTXPort:testRead" );
#ifdef TRENT_IS_HERE_DEBUGGING_ENUMERATION
      /* vmware lies about which ports are there causing irq conflicts */
      /* this is for testing only */
      if( !strcmp( name, "COM1" ) || !strcmp( name, "COM2") )
      {
            printf("%s is good\n",name);
            sprintf( message, "testRead: %s is good!\n", name );
            report( message );
            (*env)->ReleaseStringUTFChars( env, tty_name, name );
            return( JNI_TRUE );
      }
      (*env)->ReleaseStringUTFChars( env, tty_name, name );
      return( JNI_FALSE );
#endif /* TRENT_IS_HERE_DEBUGGING_ENUMERATION */
#ifdef WIN32
      strcpy( full_windows_name, DEVICEDIR );
      strcat( full_windows_name, name );
      ret = serial_test((char *) full_windows_name );
      
      ret = serial_test((char *) name );
      (*env)->ReleaseStringUTFChars( env, tty_name, name );
      return(ret);
#endif /* WIN32 *

For each com port, we call "serial_test((char *) full_windows_name );" to check if we can open that port. In Windows each com port, device, etc is a file too (like in Unix, but Windows does not allow us to see this directly using a "class-device" approach, not a file-based one), ports as any other device are part of the "//./" namespace, for example "//./COM12" is the path for my LOCALHOST "." COM12 port. If the system can open this port/file, I will treat this port as valid, in the termios.c file:

          unsigned long *hcomm;
          hcomm = CreateFile( filename, GENERIC_READ |GENERIC_WRITE, 0, 0, OPEN_EXISTING, 0, 0 );
          if ( hcomm == INVALID_HANDLE_VALUE )
          {
                if (GetLastError() == ERROR_ACCESS_DENIED)
                {
                      ret = 1;
                }
                else
                {
                      ret = 0;
                }
          }
          else
          {
                ret = 1;
          }
          CloseHandle( hcomm );

Opening a file... yeah right but there is a small issue with this method. We want to check a lot of port quicky (each time that the Arduino/Tool menu is shown, uploaded a sketch, etc) so we need a non blocking operation, something like Synchronous vs Asynchronous (Synchronous and Asynchronous I/O - Win32 apps | Microsoft Learn), but for this task CreateFile is very buggy. It delays if the "file" does not respond until a timeout, so if my bluetooth dongle is connected and there are some virtual bt com ports, the process is very laggy and time-consuming.

Thanks to EnumSerialPorts v1.02 - Enumerating Serial Ports - CodeProject I discovered another way to check ports. It is only for Windows NT, so we need to keep the compatibility backwards. Even when the new function is a lot faster, its very dirty (not as the old solution) because to keep the compatibility I just overwrite the "serial_test" function, and this function only works for one port at once. But in fact, using the QueryDosDevice API (for NT) I have the full list of devices at once, not one by one (so in the future the next release of this rxtx library will be a lot quicker).

Here is the brand-new ultra fast serial_test ;):

/*----------------------------------------------------------
serial_test

   accept: filename to test
   perform:
   return:      1 on success 0 on failure
   exceptions:
   win32api:    CreateFile CloseHandle
   comments:    if the file opens it should be ok.
----------------------------------------------------------*/
int serial_test( char * filename )
{
    int ret = 0;
    
    // Getting the Windows Version
    OSVERSIONINFO osvi;
    osvi.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);
    BOOL bGetVer = GetVersionEx(&osvi);
    
    // Using the QueryDosDevice API (on NT)
    if (bGetVer && (osvi.dwPlatformId == VER_PLATFORM_WIN32_NT))
    {
        // This solution is based on http://www.codeproject.com/KB/system/enumports.aspx
        TCHAR szDevices[65535];
        DWORD dwChars = QueryDosDevice(NULL, szDevices, 65535);
        
        if (dwChars)
        {
          int i=0;
               
          for (;;)
          {
            //Get the current device name
            char* pszCurrentDevice = &szDevices[i];

            if (strlen(pszCurrentDevice) > 3 && strcmp(pszCurrentDevice,filename)==0)
            {
                ret = 1;
                break;
            }
            
            // Go to next NULL character
            while(szDevices[i] != '\0')
              i++;
            
            // Bump pointer to the next string
            i++;
            
            // The list is double-NULL terminated, so if the character is
            // now NULL, we're at the end
            if (szDevices[i] == '\0')
              break;
          }
        }
    }
    else
    {
        // Buggy way to test if we can open the comport (on Win9x)
          unsigned long *hcomm;
          hcomm = CreateFile( filename, GENERIC_READ |GENERIC_WRITE, 0, 0, OPEN_EXISTING, 0, 0 );
          if ( hcomm == INVALID_HANDLE_VALUE )
          {
                if (GetLastError() == ERROR_ACCESS_DENIED)
                {
                      ret = 1;
                }
                else
                {
                      ret = 0;
                }
          }
          else
          {
                ret = 1;
          }
          CloseHandle( hcomm );
    }
      return(ret);
}

And the new and fixed compiled library is here (for all those have problems with delays and slowdowns in their arduino IDE, startup, uploading or displaying the TOOLS menu, any build, just replace the file "rxtxSerial.dll")
http://servicios.ried.cl/arduino/temp/rxtxSerial-2.2_fixed_2009-03-17.rar

Hope it helps for your problem/ or as experience to solve other problems. The key is a whole night, Pepsi and reading the README files!

That sucks, nobody wants to have to maintain windows specific code I'm sure. How long is the delay anyway? (video didn't work under firefox/ubuntu).

So even knowing which port you want won't help?

This is the problem with this psedo-multiplatform solutions.

The delay on my machine was about 30 seconds at startup, then 20-30 seconds everytime I open the TOOLS menu to choose a port.

Now, with the fixed library, its barely noticeable (0.1 sec or less).

I was using the windows console to upload the sketches because this delay (I often use Bluetooth so disabling it every time I use arduino was a worse solution than using the console), even I am thinking on code a C# new arduino IDE, but until this moment I haven't found a free richtextbox component with visual studio-kinda capabilities, collapsible sourcecode, intellisense, etc.

I'm very happy with the results anyway, even when the creator of that library was aware of this error. (now a lot of projects using rxtx will be updated with the version 2.2 of this library thanks to this small research)

Knowing the port would help (this way you did not need to scan those bt-related laggy ones), but the way arduino IDE is, its necessary to scan all ports to show the TOOLS/SERIAL PORT submenu

"This is the problem with this psedo-multiplatform solutions."

You can safely blame windows for that, as evidenced here you have to do it their way or it sucks. Sometimes I think they pay people to break stuff.

I tried your fixed library this morning and it worked perfectly. Later during the day, I was programming a Sanguino with my tablet & the new library and it seemed to still have some delays. The delays were not as long as before, but also not as short as when I tested it in the morning. Just now I tested the new library again and, once again it was working quite well.

Much appreciated!

I will try to release a even better fixed library to reduce those small lags

New update:

http://servicios.ried.cl/arduino/temp/rxtxSerial-2.2_fixed_2009-03-17.rar

(only for WinNT, 2000, XP, Vista or newer only)

it works 100% thanks 4 the Master GuRu from Chile! Viva Eried!

I have that same problem with the arduino-0017 IDE on Windows. Incredible delays every time I open a serial monitor, upload something, or when I startup the application.

Has this fixed library been included in the 0017 IDE ? Will it work with 0017?

Thanks

Just replying to say THANKS!

Your fix worked for me using 0017 and Windows 7 (RC).

Devs: please include this update in Arduino 0018. The enhanced rxtx solves one of the most annoying problems I've run into in the Arduino IDE. If you have any sleeping Bluetooth SPP devices, the default Arduino IDE will take a long time to load, and then a long time to open the Tools menu every time.

In addition, your sleeping Bluetooth SPP connections are trashed. The old rxtx somehow sits on them and prevents them from being used by other applications. Just by opening the Arduino IDE once, you now have to reset multiple Bluetooth devices, or unpair/repair them.

The new rxtx dll above fixes that issue completely.

is this a all windows thing or exclusive to vista + (wondering cause i have it on both xp 32 bit and xp 64 bit and its been working fine)

If you don't have bluetooth serial devices then you probably aren't going to notice anything.

Yes, yes, yes, please add this to 0018! The difference is dramatic! My startup goes from a little below 1 minute to only several seconds. Opening the tools menu even took me two minutes (no joke) and now it shows immediately.

(Thanks to the OP) [smiley=thumbsup.gif] [smiley=thumbsup.gif] [smiley=thumbsup.gif]

Hello togheter,

I have also problems with my arduino17 on a vista pc. I would try this fix, but cannot open the rar file.
Can anybody give me a hint, how to open this file,
Thank's

andi

I would try this fix, but cannot open the rar file.
Can anybody give me a hint, how to open this file

Also, from RAR - Wikipedia

Third party programs that can only read (unpack) RAR files include: WinZip (Windows), 7-Zip (multiplatform), IZArc (Windows), PeaZip (Windows, Linux), Zipeg (Windows, Mac OS X), ALZip (Windows), along with the free version of unrar (which is available for Linux and FreeBSD). Mac OS X readers include Stuffit Expander, The Unarchiver and Zipeg. Stuffit Expander is also available for Mac OS Classic with RAR support for this platform.

If you go to the second link above, all those third party programs have their own links. I'm too lazy on a Saturday to copy the links into this reply. :wink:

My fixed dll is still working with the version 0018 :slight_smile: and of course for those who asked, with the 0017 too. ;D

Another big thanks for fixing this problem. I have an older laptop running XP, and the arduino application was completely unusable due to the delays in the 0018 distribution DLL.

Your modified DLL fixed the entire problem, instantly. :sunglasses:

Jim

Dear eried,

Please could I have the source code to your modified version of the rxtx library?

Thanks,

Chris.

I will try to look for it. But you can build your own patched version. First look the point 7 of this document, ignore the text, just copy the "serial_test" function code.

Then get the full source from http://users.frii.com/jarvi/rxtx/

And finally, replace the code and Compile.