Pages: [1] 2   Go Down
Author Topic: Website very slow  (Read 4301 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The web site (and specially the forum) is very slow, with most requests taking minutes. I observed this on most occasions I access the site since I joined the community a few weeks ago.

Is this generalized or just for my area? (I'm from Spain)
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 217
Posts: 13707
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Here in the Netherlands it is also muich slower than a few weeks ago. I think the increased # visitors reading all that fine knowledge is the root cause smiley

Again we might need a faster server...

Might be an idea to freeze threads older than a year into a frozen archive (like the ...- 2010 threads).
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Grand Blanc, MI, USA
Offline Offline
Faraday Member
**
Karma: 95
Posts: 4084
CODE is a mass noun and should not be used in the plural or with an indefinite article.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

What usually passes for normal here is semi-OK response time, but today is significantly slower, by several times or more.
Logged

MCP79411/12 RTC ... "One Million Ohms" ATtiny kit ... available at http://www.tindie.com/stores/JChristensen/

Wales
Offline Offline
Jr. Member
**
Karma: 0
Posts: 57
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

No offence intended, but this site urgently needs someone who knows how to run a webserver to get involved.

I've never seen a site fail so badly on so many points - http://www.webpagetest.org/result/121126_PQ_HFC/

There are LOADS of suggestions there that could easily be implemented and would massively speed the site up.
In addition you could switch from Apache to something fast like Nginx, and the version of php is hopelessly out of date.
The very first and easiest free thing to do would be to use Cloudflare; that would take at least 50% of the load of the server and do a lot of optimisation.
Then you could add in Varnish cache - end result: 200ms page response times rather than 20seconds, net cost: half a day of the time of someone that knows what they're doing. No need to spend money on new servers.

But in short, this site isn't suffering because of visitor count, it's suffering because it's hopelessly unoptimised and urgently needs some TLC.
Logged

Forum Administrator
Offline Offline
Jr. Member
*****
Karma: 9
Posts: 95
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Today's problems have been mostly caused by the huge traffic drived by the latest post on our blog
http://arduino.cc/blog/2012/11/26/kickstarter-trademarks-and-lies/
For all the day the load average on the server has been above 200, and our 8 processors never stop to run.

That permitted to put in evidence large optimization margins still existing on this server: discrete performance tuning has been constantly in place in last two months, targetting above all the forum, but other major portions of the whole Arduino web platform still needs some revision.

In addition you could switch from Apache to something fast like Nginx, and the version of php is hopelessly out of date.

nginx has been considered but actually is not an option: migration would require a large porting of current Rewrite rules, now massively used to glue together all the pieces. Not impossible, but requires some time.

Quote
The very first and easiest free thing to do would be to use Cloudflare

I've observed bad performance of Cloudflare on at least one website far smaller than the Arduino's one, it impressed me quite negatively.
Did you had some positive experience?

Quote
Then you could add in Varnish cache

Yes, this is already on my todolist ;-)


Sorry for the bad service seen today, your sysadmin is working for you.
Logged

BOFH

Grand Blanc, MI, USA
Offline Offline
Faraday Member
**
Karma: 95
Posts: 4084
CODE is a mass noun and should not be used in the plural or with an indefinite article.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Response is much better now, FWIW.
Logged

MCP79411/12 RTC ... "One Million Ohms" ATtiny kit ... available at http://www.tindie.com/stores/JChristensen/

Global Moderator
Boston area, metrowest
Offline Offline
Brattain Member
*****
Karma: 538
Posts: 27089
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I couldn't even log in at work today to respond to anything.  My entries were accepted - and then nothing, was treated just as guest.
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

Wales
Offline Offline
Jr. Member
**
Karma: 0
Posts: 57
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

My apologies for my rant yesterday, I was out of order having a go like that. I know you are working hard, but it was incredibly frustrating as I was just trying to learn and read docs and I could see that so many things could easily be done to fix it. So instead of me just complaining, perhaps I can help a bit.

I've re-run that test and I can see that a lot of improvements have been made since then. However, there are still some simple things that would take minutes at most to help things.

The version of jquery used is 1.4.4 from 2010, over 2 years old, isn't very optimized and doesn't play well with modern browsers.
The version of mootools is from 2006(!!). Why would two overlapping frameworks be needed? jquery should do everything.

Because you're hosting jquery yourself AND browser caching isn't in use, every page load and every visit is loading 70k of (obsolete) jquery.

Let's say you have 10,000 page loads per day. Instead of hosting jquery yourself, let Google take the strain and take advantage of browser caching.

Code:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>

Remember you can always link to the current version by omitting the .version. eg:

 
Code:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.8/jquery.min.js"></script>

will always give you the latest 1.8 version and

 
Code:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>

will roll you over to 1.9 when it's released if you feel brave!

Either way, that's 684Mb and 10,000 file-serves saved every single day just for changing 1 single line.

And if you HAVE to load mootools as well:
Code:
<script src="//ajax.googleapis.com/ajax/libs/mootools/1.4.5/mootools-yui-compressed.js"></script>

In addition you could switch from Apache to something fast like Nginx, and the version of php is hopelessly out of date.

nginx has been considered but actually is not an option: migration would require a large porting of current Rewrite rules, now massively used to glue together all the pieces. Not impossible, but requires some time.

Yes, it's a learning curve but it's SO worth learning. The difference between Apache+PHP mod and nginx+php5-fpm is unbelievable.
Take a look at this: http://www.garron.me/linux/apache-vs-nginx-php-fpm-varnish-apc-wordpress-performance.html

At 40 concurrent users on his 512Mb VPS, his Wordpress starts to hit 1 second response times.
With Apache + PHP with APC and Varnish he's service 150 concurrent users with a 40ms response time.

That's nearly 4 times as many users getting served 25x quicker, all from the same box.

Definitely worth looking into! And the difference between mod_rewrite and conf files isn't actually that big once you get the syntax.

Here are my 3 conf files for Wordpress, phpbb3 and ZenCart.

There's nothing you can't do with Apache .htaccess that you can't do far more efficiently with Nginx conf directives. And the nginx forum and Stackoverflow will always help you if you get stuck.

The ideal is that the only hit you see in the log on the server which you are running php + mysql is 1 single hit to serve the dynamic page content.
That's not always possible, but Cloudflare will massively help toward that too.

Quote
The very first and easiest free thing to do would be to use Cloudflare

I've observed bad performance of Cloudflare on at least one website far smaller than the Arduino's one, it impressed me quite negatively.
Did you had some positive experience?

I've be interested to know about this bad performance and what the conditions were.

Here's why I trust and use Cloudflare (and no, I don't get paid for saying this!):

Quote
"Last year, when a Web site launched by infamous hacker group LulzSec was knocked offline by a massive DDoS attack it turned to CloudFlare, a 30-person San Franicisco start-up for protection.

For the next 23 days, LulzSecurity.com stayed online, and CloudFlare, which had launched in private beta only 12 months before, withstood withering attacks from hackers around the globe.

The quality of CloudFlare’s service and its rock-bottom prices have been attracting new customers at the rate of 1,500 a day, Prince said, including Fortune 500 companies, major government agencies in the United States and abroad, and even sites owned by the White House. In two years, CloudFlare has grown so big that on any given day 25 percent of the Internet’s visitors pass through its network, Prince said. CloudFlare currently has servers running in 14 data centers around the world, including 7 in the United States, 3 in Asia and 4 in Europe."

And those figures are from Feb, so bound to be much higher. If it's good enough for 25 percent of the internet and the White House, it's good enough for me!

So, if you were to use Cloudflare, that takes care of the 20 static assets which should be being combined and minified.

http://www.webpagetest.org/result/121127_9W_N3Q/1/performance_optimization/#cache_static_content

Here's an example: I run 5 sites: 3 Wordpress, 1 busy phpbb forum and 1 horribly inefficient but busy old e-commerce system called Zen Cart - (80 db queries for the front page alone, nothing I can do about it sadly).

I run nginx, php5-fpm, apc, and varnish cache on the Wordpress sites, and Cloudflare sits in front of all of that. I run ALL of these sites from one single-core 1Gb RAM VPS costing me £10 a month.
Load rarely tops 25% according to "top", according to Google analytics my average total pageload time (all assets loaded) is 1.25 seconds, making all my sites faster than around 80% of comparable sites.

Here are my cloudflare stats for all 5 sites for the last 30 days:

Code:
51.1 GB bandwidth saved by CloudFlare
115.0 GB total bandwidth

3,844,627 requests saved by CloudFlare
5,232,005 total requests

777,088 Page views
581,657 regular traffic
95,391 crawlers/bots
100,040 threats

That's 100,000 pageviews saved from hackers even getting to your site.

And the advantage of Cloudflare is, if you don't like it, you login to the control panel and switch it off instantly.
Total time: 3 minutes to make a new Cloudflare account. 2 minutes to login to your registrar and change the name servers. (Plus propogation time of course!).
Logged

Forum Administrator
Offline Offline
Jr. Member
*****
Karma: 9
Posts: 95
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

So instead of me just complaining, perhaps I can help a bit.

Pleased to see your collaboration :-)

Quote
The version of mootools is from 2006(!!). Why would two overlapping frameworks be needed? jquery should do everything.

Good question, the response is "I don't know". Mootools seems required by the current SMF's theme but it is unclear to me the usage, I will try to disable it on our development instance and see what it happens.
About using Google's CDN for jQuery: you are absolutely in reason, the test for migration to 1.8 is in my schedule (since far too long time, really).

Quote
I've be interested to know about this bad performance and what the conditions were.

I've observed the experience of a friend of mine, on a small media coverage website with low traffic (peaks of 50 concurrent visitors).
Too often the HTTP request failed in an error by Cloudflare complaining it was unable to find the target content, and very little hints was provided to correct the issue.
Not a so good impression...


This morning I began to put a cache plugin on the Wordpress instance driving the Arduino blog, just to provide a quick and dirty solution to the still moderately high traffic driven by the latest post published. And the Varnish documentation is opened on one of my browser tabs, ready to be implemented on the development server in next days.
Logged

BOFH

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 614
Posts: 49343
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I can stand the forum being slow (for a while) as long as it is obvious that someone is at least making an effort to do something about the situation. Thank you for that. Keep us posted on your efforts, please.
Logged

Left Coast, CA (USA)
Offline Offline
Brattain Member
*****
Karma: 361
Posts: 17294
Measurement changes behavior
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This same situation (slow slow response time) has happened at least three other times in my 3+ years on this site. Each of those times it did take some time for someone to actually do and/or change something fundamental (new server, new hosting software, etc) to improve the site's basic performance. So whether it's the continued growth of the site or something else, the site does seem to need some careful care and feeding again as it has needed in the past.

Thanks for any help that can improve things here.

Lefty 
Logged

Forum Administrator
Offline Offline
Jr. Member
*****
Karma: 9
Posts: 95
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Keep us posted on your efforts, please.

Improvement of the day: PmWiki's FastCache has been enabled on Arduino Playground also (on the main Arduino website it was already in place).

Current load average of the server is between 7 and 10, quite good compared to the common 20 and the 240 peak of two days ago.
Logged

BOFH

Forum Administrator
Offline Offline
Jr. Member
*****
Karma: 9
Posts: 95
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Today we hit a new low in server responsiveness, but cause is unknow: traffic was the same as always, no major content was published and visited, no anormal processes were running... I'm still reading logs to find a reason...

Also, I spent just a little time in reading Varnish documentation and testing elsewhere: still some issue about cache management, but I think next week I can run it on main server. All of you will be alerted and asked to report me about undesired behaviours, I expect your collaboration ;-)
Logged

BOFH

Belgium
Online Online
Edison Member
*
Karma: 68
Posts: 1919
Arduino rocks; but with my plugin it can fly rocking the world ;-)
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm happy to notice the performance is actually actively monitored
Best regards
Jantje
Logged

Do not PM me a question unless you are prepared to pay for consultancy.
Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 614
Posts: 49343
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
All of you will be alerted and asked to report me about undesired behaviours, I expect your collaboration
If it doesn't time out trying to post...  smiley-cool
Logged

Pages: [1] 2   Go Up
Jump to: