SD file search takes much time

Hi my project is an arduino based hand held point of sale my problem is it searches 8500 items in a text file to find the specific item and displays its data if the item is near the end of the file it may take 30 seconds searching , that is boring how can i make it quick thanks for your help

fidamon: Hi my project is an arduino based hand held point of sale my problem is it searches 8500 items in a text file to find the specific item and displays its data if the item is near the end of the file it may take 30 seconds searching , that is boring how can i make it quick thanks for your help

Describe one entry in the text file. Are all entries the same length? What is your program looking for when it "searches"?

How often is the 8500 item file created? By what system?

Do you have access to both the file creation, as well as the hand held code?

Paul

here is a line of my file "6223002141760,SINOPRIL 10,TAB,3,30.6,2,18/03,45, ," it is a drug (pharmacy) when i get the BarCode identification i read from the file (each item has 9 "," ) and then compare my barcode number with that i read from the file (6223002141760) if it is identical that is my item , if not read next item i depend on number of "," not record length because not all entries the same length the file is created by a vb6 software (i made ) so i create it whenever i like thanks for your help

The first thing to do is change your vb6 code so that it writes the entries in numerical order. If those drug numbers are fairly evenly distributed (roughly a tenth of them begin with 0, 1, etc) A first simple crack at this would be to also create an index of where in the file the first record beginning with 0, 1, 2 etc are. Then when you need to find 6223002141760, you first look up the index for 6, seek to that point in the main file and then do a linear search from there. You will only have to search about one tenth of the file because either you find 6223002141760 or you reach a number which begins with 7 and then you know there can't be any more beginning with 6. If most of the drug numbers begin with the same digit, choose a different digit that is more evenly distributed and sort on that. You could also use two digits as the index which gives one hundred entries in the index and would cut the search time to roughly one percent.

Pete

The first thing to do is change your vb6 code so that it writes the entries in numerical order.

The second thing to do is make the VB app write records that are all the same length. Reading fixed length records is much faster than reading variable length records. Once you know that the record is not the interesting one, you can skip to the next record much faster than having to read every character, looking for the end of the record.

Thank you for your help
yet i need more help about some points
1- how to make index and use it in such file
2 - more than 95% begin with 622
3 - it is easy to make vb6 write fixed length records , how to make arduino read fixed length record

void drugData(char barCode[14] )
{
 // Serial.println(String(barCode));
  char drgid[5] = "" ;
  drugID = 0;
  digitalWrite(10, LOW);
  int checked = 0 ;
   pinMode(dataPin, OUTPUT);
    tft.setTextSize (3);
    //tft.setTextColor(0xFFFF,0x001F);
 
  if ((barCode[0] == 13) or (barCode[0] == 10)) {
    for (int h = 0 ; h < 13  ; h += 1) {
      barCode[h] = barCode[h + 1];
      // barCode[h]=0;
    }

  }//if ((barCode[0] == 13)
 tft.setCursor(10,10);
  tft.println(barCode);

  for ( conta = 0; conta < 150; conta += 1) {
    linea[conta] = 0;
  }
  conta = 0;
  icount = 0;
  // re-open the file for reading:
 if (swType==1){

    myFile = SD.open("check.txt" );
 }
 else {   myFile = SD.open("drug.txt");}
  filePos  = 0 ;
     
  if (myFile) {

    // read from the file until there's nothing else in it:
    while (myFile.available()) {
      linea[conta] = myFile.read();
   conta += 1 ;
      if ((linea[conta - 1] == 13) or (linea[conta - 1] == 10)) {
        conta = conta - 1;
               
      }//(linea[conta - 1] == 13)

      if (linea[conta - 1] == ',') {
        indices[icount] = conta - 1;
        icount += 1;  // counting ","
      } //(linea[conta - 1] == ',')

      if (icount  == 9)  {

        checked += 1;
         bien = 0 ;

        for ( int j = 0 ; j < indices[0]; j += 1)  {// comparing barcodes
          if (barCode[j] == linea[j]) {
            bien += 1;
          }
  
          if (bien == indices[0] - 1) {
//that is my drug do whatever you want
    
           myFile.close();
            return;
          }//if (bien==12)

        } //  for(int i= 0 ;i<13;i +=1)

        for ( int k = 0 ; k < 150; k += 1)  {
          linea[k] = 0;
        }
        conta = 0 ;
        icount = 0;
             } //if (conta  == 6)
      if (conta > 149) {
        for ( conta = 0; conta < 150; conta += 1) {
          linea[conta] = 0;

        }
        conta = 0;
        icount = 0;
      }

    } // while (myFile.available())
    
    // close the file:
    myFile.close();
  } //  if (myFile)
  else {
  
    tft.println(F("error opening file"));
  } //   else

}//drugData

1- how to make index and use it in such file

Write the index to a separate file at the same time as the other file. Read the index file in at the start of your sketch. If your index is of just the last digit, you'll have ten entries in the index file, each one being the 32-bit address of that entry in the main file.

2 - more than 95% begin with 622

Sort on the last 1 or 2 digits instead.

3 - it is easy to make vb6 write fixed length records , how to make arduino read fixed length record

It is also easy to read fixed length records with the SD library. See "seek" and "read" in https://www.arduino.cc/en/Reference/SD. It would make things more efficient if you chose a record length that was a power of 2.

Pete

thank u el_supremo you gave great ideas

This is all called "systems analysis" or "data analysis", which is what you will need to learn how to do.

Paul