Go Down

Topic: How to delete first line from a text file (Read 7597 times) previous topic - next topic

Hamzakhani3

Aug 29, 2015, 06:29 pm Last Edit: Aug 29, 2015, 07:04 pm by Hamzakhani3
Iam making a datalogger which will send data to a web server on periodic intervals. The data is stored in a text file , line by line and each line is of a fixed length.I am sending data each line at a time and once the sever acknowledges receiving the data i want to delete that line from the text file(which would be the first line) , but iam unable to figure out how?  Can anyone help ?

robtillaart

overwrite it with spaces?
special char on first position marking it as deleted?
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Hamzakhani3


robtillaart

can you post your code, Then we can look what is the easiest way.
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

chucktodd

Iam making a datalogger which will send data to a web server on periodic intervals. The data is stored in a text file , line by line and each line is of a fixed length.I am sending data each line at a time and once the sever acknowledges receiving the data i want to delete that line from the text file(which would be the first line) , but iam unable to figure out how?  Can anyone help ?
you can't actually 'delete' a line from a file, to remove the first line of a file, you have two choices:

move line2 into line1, move line3 into line2, ....
and when all of the lines have been moved up, reduce the total file size by one line (which truncates the last line of the file).

Or as Robtillaart recommended:
overwrite it with spaces?
special char on first position marking it as deleted?

you decide on the 'special char' and all of your routines that access this file obey the 'delete' status.

The easier method is the 'space overwrite'.  Just position the file pointer to the beginning of the line you you want to delete, then print() the EXACT number of spaces to overwrite the 'deleted' data. Be aware of the cr/lf at the end of the line.
Code example to 'overwrite line'.
Code: [Select]

File f;  // actually use SD to open the Card and assign the file to f
#define LINELEN 34  // length of line including EOL characters, I assume crlf  0x0D 0x0A
  // so the line is actually 32 text characters plus 2 end of line characters for 34 total.

void delLine( File f, uint32_t line ){ // line is 1..n, no Zero Line!
uint32_t S= (line-1)*LINELEN;

f.seek(S);  //position the 'file pointer' to where I want to start over-writing.

char ch[35]; // got to remember c++ strings need a \0 to mark the end.

// build the 'delete line'
for(uint8_t i=0;i<32;i++) ch[i]=' ';
ch[32]='\r'; //  (cr)
ch[33]='\n'; // newline (lf)
ch[34]='\0'; // end of string marker;

// over write (delete) the existing data
file.print(ch); // all marked as deleted! yea!
}


Chuck.



Chuck.
Currently built mega http server, Now converting it to ESP32.

PaulS

The usual way to delete a record from a file is to write the data that you want to keep to a temporary file, delete the original file, and rename the temporary file to have the original file's name.
The art of getting good answers lies in asking good questions.

Hamzakhani3

Thank you everyone .. Chuck u are a life saver!! thanks man.... i will try your method and share my code here :)

Southpark

#7
Jan 04, 2017, 02:26 am Last Edit: Jan 04, 2017, 03:33 am by Southpark
Code: [Select]

File f;  // actually use SD to open the Card and assign the file to f
#define LINELEN 34  // length of line including EOL characters, I assume crlf  0x0D 0x0A
  // so the line is actually 32 text characters plus 2 end of line characters for 34 total.

void delLine( File f, uint32_t line ){ // line is 1..n, no Zero Line!
uint32_t S= (line-1)*LINELEN;

f.seek(S);  //position the 'file pointer' to where I want to start over-writing.

char ch[35]; // got to remember c++ strings need a \0 to mark the end.

// build the 'delete line'
for(uint8_t i=0;i<32;i++) ch[i]=' ';
ch[32]='\r'; //  (cr)
ch[33]='\n'; // newline (lf)
ch[34]='\0'; // end of string marker;

// over write (delete) the existing data
file.print(ch); // all marked as deleted! yea!
}


Chuck.
Hi Chuck! ..... a year or so late, but the variable named 'line'. It doesn't seem to have been given any value, right?

In the following two lines of code..... what's the value for 'line - 1' ?

void delLine( File f, uint32_t line ){ // line is 1..n, no Zero Line!
uint32_t S= (line-1)*LINELEN;


Just asking as I'm interested in this topic, as I'm trying to periodically (every 5 mins or so) overwrite the very first line of a file in an SD-card. And the previous poster (the OP) never got back to this thread.

Will it be correct to say that putting the function you provided into a sketch, and calling delLine with input arguments being filename and the line number we want to delete....... the program will then proceed to delete that particular line only?

I'm just seeing if your code will allow me to just write to the very first line of a file (every single time), so that the file will just be a one-line file all the time. The purpose is really for saving initial starting values, so that whenever the arduino powers up again, it can just collect the most recent operating values from the first line. Eg....  "15000 22000 30000"

Thanks!!


[UPDATE] - worked out what I needed to do, in order to just keep writing to the first line of the file, thanks to you with the f.seek function. In my case, my file was arbitrary given the handle of myFile, so I used myFile.seek(0) to make the file pointer go to location 'zero'. Next, I just used the usual printing function, like myfile.println("123 etc") to print the line. Works nicely. Thanks!

chucktodd

Hi Chuck! ..... a year or so late, but the variable named 'line'. It doesn't seem to have been given any value, right?
the variable line is passed into the function delLine() it is the 'line' you wish to delete.  It's range is 1..maxLineInFile, so if your 'file' has 10 lines, you could code delLine(myFile,1); Which would delete the 'first' line of the file.

In the following two lines of code..... what's the value for 'line - 1' ?

void delLine( File f, uint32_t line ){ // line is 1..n, no Zero Line!
uint32_t S= (line-1)*LINELEN;


Just asking as I'm interested in this topic, as I'm trying to periodically (every 5 mins or so) overwrite the very first line of a file in an SD-card. And the previous poster (the OP) never got back to this thread.
this text file has a defined format, every line has 32 characters of data followed by /r/n (window txt format).

So, (line-1)*LINELEN, calculates the byte offset into the file where the line to be deleted starts.

Lets' say you want to delete line 5:
line one occupies byte offsets 0:33,
line two occupies byte offsets 34:67,
line three occupies byte offsets 68:101,
line four occupies byte offsets 102:135
so, line five starts at (5-1)*34 = 136

so S now contains 136, and myFile.seek(S) causes the SD library to move the filepointer to position 136.  Then I overwrite the next 34 bytes with 32 blanks and cr lf.


Will it be correct to say that putting the function you provided into a sketch, and calling delLine with input arguments being filename and the line number we want to delete....... the program will then proceed to delete that particular line only?

almost,
  file is not the name of a file, file is an OPENED File object.

I'm just seeing if your code will allow me to just write to the very first line of a file (every single time), so that the file will just be a one-line file all the time. The purpose is really for saving initial starting values, so that whenever the arduino powers up again, it can just collect the most recent operating values from the first line. Eg....  "15000 22000 30000"

Thanks!!
as long as your file is a fixed line length txt file, that consists of 32 bytes of text followed by a carriage return and a line feed.

Yes.

Chuck.
Currently built mega http server, Now converting it to ESP32.

Southpark

Thanks very much Chuck! I'll store all the information you provided, as it is really invaluable information. I'll certainly be applying what you mentioned a lot, from now on. Totally appreciative of your help and time - going out of your way to help people like this. We become students of you! Thanks again!

witkov63210

Chuck,
almost 2 years later but this is Southpark,s stated mission:

"I'm just seeing if your code will allow me to just write to the very first line of a file (every single time), so that the file will just be a one-line file all the time. "

Wouldn't simply doing a seek(0) before writing every record satisfy that requirement?

With all the discussion on this topic no one has mentioned the solution I am about to try.

I have created a functional telephone call blocker to intercept telemarketers calls. I'm  using the HT9032 module to monitor the Caller ID. When the human determines the caller is to be blocked he presses a button and the Caller ID info is recorded on the SD chip. Each incoming call checks the file for its number and, if found, blocks the call by playing the "The number you have reached has been disconnected or is no longer in service." message.

Problem: Telemarketers have started using legitimate local numbers to increase pickups. If they use the number for your Pharmacy or your Doctors office then that number is blocked. I want to have the numbers to be purged after they reach a certain age.

My solution: Create a separate file containing nothing but the record number for outdated records. That file would be consulted whenever a call comes in and those records would be skipped. The next call to be blocked would be written into an available space and the number removed from the 'available' file. It should be much easier to remove one number from a small file than to copy and delete and rename the entire file.

What do you think?

sterretje

#11
Nov 26, 2018, 04:30 am Last Edit: Nov 26, 2018, 04:30 am by sterretje
Quote
I have created a functional telephone call blocker to intercept telemarketers calls. I'm  using the HT9032 module to monitor the Caller ID. When the human determines the caller is to be blocked he presses a button and the Caller ID info is recorded on the SD chip. Each incoming call checks the file for its number and, if found, blocks the call by playing the "The number you have reached has been disconnected or is no longer in service." message.
That's nice, I like it.

Quote
Problem: Telemarketers have started using legitimate local numbers to increase pickups. If they use the number for your Pharmacy or your Doctors office then that number is blocked.
That's basically identity theft in my view. I would inform the doctor/pharmacy/... and let them open a case of identity theft.

Quote
My solution: Create a separate file containing nothing but the record number for outdated records. That file would be consulted whenever a call comes in and those records would be skipped. The next call to be blocked would be written into an available space and the number removed from the 'available' file. It should be much easier to remove one number from a small file than to copy and delete and rename the entire file.
I think that directly overwriting is a lot simpler. If you add a date to the records, you can check while reading from file.

format:
Code: [Select]
number, yyyyMMdd

It's the easiest to use a fixed width record so you can seek().

Loop through file to find number. While looping, record first outdated record (e.g. older that 6 months) if found.
If number not found and user decides to block
1)
if outdated record found, go to the outdated record that you found and overwrite record with number and new date.
2)
if no outdated record found, append new record with number and date.
If you understand an example, use it.
If you don't understand an example, don't use it.

Electronics engineer by trade, software engineer by profession. Trying to get back into electronics after 15 years absence.

witkov63210

Thank you, sterret. I like your advice because it is almost exactly what I have done.

The first exception is that I have called back many numbers including Bank Of America and ATT. the person I reach is an individual sitting at a desk that is not in the least inconvenience by the fact that his/her number is being used. They are annoyed by telemarketers but not enough to do anything about them.

I am using fixed length records. The Caller ID info is 38 characters including the number, the name, the month, day, and time. I add 4 more bytes at the end for flags to indicate status. at the moment I am using a byte to indicate Blocked or Pass(through). I only need a bit but using a byte was a little simpler to code. I am using one flag to indicate a deleted record.

By storing the record number in a separate file I don't have to search through many records for the record to overwrite. What I am working on right now is eliminating the available record number from the available record number file. What I plan to do is read the file into an array and write everything back out omitting the first byte. I wish I could use a variable array size to accommodate an unknown file size but I haven't researched that.

I like your background. I have done much industrial electric and electronic work.

jimLee

#13
Dec 02, 2018, 08:15 am Last Edit: Dec 02, 2018, 08:16 am by jimLee
Wait wait wait..

You guys want an object that runs an SD card file like RAM? Give it a file path to set it up. Then you store buffers in it and it hands back bufferIDs. You can read or write to those ID all you want. Change sizes, whatever. And, you can have as many IDs as you like. It manages everything for you, like dynamically sized mailboxes.

Business end of the interface :

Code: [Select]


class blockFile {

  public:
          blockFile(char* inFilePath);
  virtual ~blockFile(void);

  unsigned long   readInitialBlockID(void);                     // HINT : You store the information you need to decode your file in here.
  unsigned long   getNewBlockID(void);                          // The first blockID issued to you, will be saved as your initial block.
  unsigned long   addBlock(char* buffPtr,unsigned long bytes);  // That can come from  getNewBlockID() or addBlock().
  bool            deleteBlock(unsigned long blockID);
 
  bool            writeBlock(unsigned long blockID,char* buffPtr,unsigned long bytes);  // I have my ID, put this buffer in there.
  unsigned long   getBlockSize(unsigned long blockID);                                  // How much did I store in there? I forget..
  bool            getBlock(unsigned long blockID,char* buffPtr,unsigned long bytes);    // Here's my ID and a buffer, give me my data.
 
  void            cleanup(unsigned long allowedMs);                                     // Not written yet.
  void            deleteBlockfile(void);                                                // Mark file to be erased when object is deleted.
  int             checkErr(bool clearErr=false);                                        // Lets see what the last error was?
  bool            isEmpty(void);                                                        // No data buffers.



-jim lee
PNW Ardiuno & Maker club
1012 9Th Street, Anacortes, WA 98221 (Around the back of building)

NavyVet1959

Telemarketers and other scammers call so much that my landline is useless and I have to leave it unplugged.  I have DSL, so not having a landline is not an option.  I think the best solution would be to have a system that answered the phone and then presented the caller with a voice menu of options / extensions that they could reach.  When you gave your number to someone that was legitimate, you would give them a valid extension that they could use to contact you.  Or maybe do something like this:

"Hello, you have reached my phone interface system.

Press 1 if you are a telemarketer wanting to sell me something.
Press 2 if you are advocating a particular political candidate or party and are stupid enough to think that you might change my current preference.
Press 3 if you are a scammer claiming to be from Microsoft or Apple and want to try to convince me that one of my machines has a virus.
Press 4 if you are some other type of scammer.
Press 0 if you have been given an extension for someone at this number."

Go Up