Forum Failures

Hi Everyone,

As you have seen, we had 2 major issues tonight.

Basically our DB instance was no more reachable even if we have a failover instance due to a DNS issue.

So during my tries I needed to double the size of out DB instance, overwrite the DNS config of the host. We are now waiting for a DNS fix from AWS.

This is very bad but there are some good news.

  • Apparently we did not really need the cache system so 1 less point of failure
  • There is a way to duplicate the forum on 2 server and we are going to provide a roadmap for it early next year
  • Now the DB is twice big and fast as before, we will see how large will be the cost but the performances look very good

In addition to all of this some of our team members (me included) were on holiday so we are planning an availability rotation for the near future.

Thanks,

Luca

Thanks for fixing and this feedback,

Rob

Thank you for fixing things. I had feared it would not be attended to before Monday.

...R

OK, so I note the problem with a loop between forum.arduino.cc and id.arduino.cc.

Was that the cache?

The other problem is that something generates these "token" redirects. Whatever that "something" is, is clearly the stupidest example of coding that we have seen in a long time. If you know, or can determine what it is, then that part of the system needs not to be fixed, but permanently and conclusively removed from the system as it has been the principal source of headaches for a considerable time.

Was that the cache?

Please do tell, because it may well help others to know of a software package that should never, ever be deployed!

Thanks a lot, you did a good job there.

It appears that removing the cache front end has corrected some problems I thought were due to my browser cache. After posting a response and logging out, the forum wouldn't show my post for a couple minutes. Now all looks as it should.

No hard measurement but the forum feels faster.

What is the relation between this issue and the e-mail notifications? It seems that today I receive this kind of notifications!

Rob wrote:

No hard measurement but the forum feels faster.

I'm trying to work out how you can 'feel' such a thing as a forum let alone whether it is slow or fast. :o

Maybe you meant to say 'appears faster' or that you 'think it is faster' ?
Sorry, I get caught up when I notice people saying they 'feel' when, in the context of what they say, there is nothing to feel.

No hard feelings :grin:


Paul

I don't think it "feels faster". It is faster. :slight_smile:

I still think it is due to disabling the cache front end on the forum server. Before last night, the Online users updated only every couple minutes. Now it updates every time I download the forum home page.

I think it still downloads at the same speed. Just what it is displaying is current.

rockwallaby:
Rob wrote: I'm trying to work out how you can 'feel' such a thing as a forum let alone whether it is slow or fast. :o

Maybe you meant to say 'appears faster' or that you 'think it is faster' ?
Sorry, I get caught up when I notice people saying they 'feel' when, in the context of what they say, there is nothing to feel.

No hard feelings :grin:


Paul

LOL
appears might be the better word, for me there is definitely a "feeling". As in look&feel of an application.

When opening 4 or 5 tabs in parallel and they are all loaded before I switch to them, the forum feels fast (enough). I do not know if it is fast in an absolute sense. I experience that (the total of all tech needed to give me the illusion of a forum) is faster than I am. And that feels good.

in a similar sense a car feels faster for me if I am closer to the road e.g. in a Mini Cooper vs a truck :wink:

Now I can see the forum using IE once again. Wonder how long this will last before the forum admins break the forum again. Seems a lot of the recent forum issues with IE have been consciously introduced.

I noticed the forum was down last night for a few hours and emailed Mastrolinux at about O4:30 am GMT, if my time zone conversions are correct.

I thought possibly the administrators might be on holiday and did not notice that the form was down.

I don't know whether my report was the first one he received however congratulations on resolving the problem very promptly after that.

This might be a good time to suggest some sort of system for alerting the administrators if the forum is down such as is used by online games.

For example a job running every 30 minutes on a totally independent computer that checks if the forum is up and if it is not sends an email to selected people.

Hi Nick. The forum was not down. The problem was nobody could log in.

He Nick,

I think that should fit in an UNO with Ethernet shield ...

SurferTim:
Hi Nick. The forum was not down. The problem was nobody could log in.

I used an independent site verification site which gave me the following response:

ARDUINO.CC WEBSITE IS NOT WORKING

For me, receiving a blank page attempting to connect is "down".

Serving up blank pages is not my definition of the forum being up.

That may have been true after the administrators took it down to fix it, but a few hours prior to that it was up and returning all pages I requested, but the "Online users" showed nobody logged in, and I could not log in. I could log in on the playground, so the login server was working, just not for the forum.

I'm pretty good at network troubleshooting. :slight_smile:

edit: Since there was a cache server front end until last night, it could have been the cache server responding to my requests, not the actual forum server.

I very rarely login myself, like once every few weeks, I just ask it to access the page.

Just as another point, the forum could be viewed if not logged in, but if using a browser that had been used with a login, that browser was redirected to the blank login page. Some times with IE when the typical blank page is received, viewing the page source received shows stuff is returned, but there appears to be a display issue.

robtillaart:
He Nick,

I think that should fit in an UNO with Ethernet shield ...

That's not such a silly idea.

I have got a Uno with an ethernet shield set up to monitor my son's Minecraft server so the concept certainly could work.

Each of the forum administrators could have one set up in the office or next to their bed to wake them up or alert them when forum goes down.