Buffering Methods / Project Help

Hello, I am new to this forum and android, so apologies in advance for the mistakes I will most likely make.

I feel like I should explain the overview of what I am doing so you can tell me if I am approaching it wrong. I am embedding an arduino into a machine and having it create an event log. The arduino has either a wifi shield or an ethernet shield to connect to the internet. Both of these shields have a SD card. So pretty much the buffer comes in play in two ways. I need the buffer to manage data when uploading the log via internet connection, but also if the internet is down I want to use the SD card until the internet is restored. From there I want to be able to obviously upload the SD card data as well as any new data. I am thinking that I might just keep adding to the SD card until it is fully uploaded then go back to the ethernet. I also have never used buffer methods, but have done some research on circular buffers and FIFO. I don't fully understand them to the point where I can understand which would be the most effective for me.

If you guys could give me a couple ideas and/or point me in the right way to study up some more on buffers.

Just some notes, these events shouldn't be extremely frequent, but some might happen within the same second (up to around 5-10) and not have another error until possibly minutes or more later so I shouldn't be too worried about data layover due to interrupts or delays between ethernet/SD/arduino. The code in the arduino is going to be fairly complicated working with uploading information to the internet, so using less memory would be nice. I am working with a Uno, but have a Mega if need be.

Thanks in advance.

P.S. Also I was wondering if you guys new the read/writes of the arduino, SD, ethernet, and Wifi. I just want to know where I can possibly predict any bottlenecking. All the shields are also main brand arduino, not a third-party.

How does the data from the log get to the internet? Are you pushing it out from the Arduino using some sort of store-and-forward algorithm, of will some device on the internet be contacting the Arduino to request data from it? In either case, what protocol will you be using?

I am actually just pushing data using basic HTML protocols. The way I contact the internet is dependent on a internet website on which I have no control, but I am pretty sure it will be a high level PHP/HTML or likewise protocol. As of right now the information is all push, the client/browser is doing no pulling other than the initial connect. The algorithm and data being passed is very complicated in the way it will be sent to the HTML because some it will have to reformatted such as the event log. I hope this answers your question and thanks for the reply.

That's left me slightly confused about how that transfer over the network is going to work. You plan to have a browser connecting directly to the Arduino (client-side mashup)? Is the Arduino providing a web service to accept the incoming request for data? How's that web service protocol going to work?

It sounds as if you will need to completely decouple the incoming event stream from the outgoing event stream, since the outgoing stream doesn't seem to be under your control. In that case you will need to use some sort of store/forward system. Do you need the storing to be persistent i.e. the buffered events will be retained across a power outage? Can you give some indication of how much data a single log entry will contain, and how many of these you expect to need to buffer during a worst case outage of the network connection?

I suggest you look into syslog protocol it's hugely simpler than http

Regarding buffering: if you expect to be off line much of the time, as PeterH said You need to decouple the input and the logging. This can be done by always writing the data to the SD card and always sending available data from the SD card to syslog host, the SD card is your buffer.

Since it seems that timely updating is not a concern (since you may be offline some of the time). Consider opening an SD file and writing a well defined amount of data to it. Then close it and begin another file.

The sending part can send as many files it can when the system is online. When it completes one either delete it or renaming it to indicate that it has been sent. Retaining a fixed number of sent files permits the ability to retransmit the data in the event of data loss on the server.

Okay sorry for the confusion. I have an Ethernet shield working connecting to a 4G SIM Modem (or in some cases a wifi shield connects it to a local network). From there it grabs its public IP address where the broadcast is (as well as the port). This is where arduino is posting/getting the HTML. The arduino uses a basic internet client (Ethernetclient or such on Wifi). After that point I am not positive because I am still waiting to meet up with the website I am interacting with. I believe the website might just grab the information from the IP and port I am using. This seems the most likely case and because I am dealing with simple .csv or basic pin reads of dry contact switches this information is light and can be done easily over HTTP which is just the most likely to be used as of right now. Could you explain or forward me more information on understanding exactly what you mean by your store/forward algorithm and by decoupling the streams. I think I understand, but for clarity it would be much appreciated. A background on the log (event based log) is that the machine the arduino is grabbing from it can store a max of 130. The internet down time of this time should be relatively short compared to events. It should only be out for a day or two in which less than 50 logs, probably less than 10, will occur. The down time is not expected to be frequent by any means, but in the case of it going down, the information in the log is very important and cannot be lost, even if new events occur. I am looking into more of this syslog protocol now, but it doesn't seem too much more simple than HTTP and PHP, for basic event logging. I could be wrong though.

Thanks again.