Arduino Forum

Forum 2005-2010 (read only) => General => News => Topic started by: DojoDave on Jan 18, 2011, 12:41 pm

Title: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 18, 2011, 12:41 pm
PLEASE CHECK THIS BLOG POST: http://arduino.cc/blog/2011/01/18/forum_migration_jan_24/

Twitt it, Facebook it, shout out loud we're making it
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 18, 2011, 01:02 pm
Anyone else think a lack of migration is a major concern?

What forum type are we going to?
Is migration not possible or have you just chosen not to do it for some reason?

Mowcius
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Selby96 on Jan 18, 2011, 01:31 pm
Quote
Anyone else think a lack of migration is a major concern?



Yes.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: jmrobert on Jan 18, 2011, 02:11 pm
Yeah its cheap.  How is it people can make a data repository, and NOT know HOW to move the data?

As for the "new" features, who cares?

1. rearranged?  so?
2. pictures video?  don't care, can't view videos thru the corp firewall anyway.
3. email, new profiles?  meh
4. help page for newcomers?  this made me laugh, newbies never read anything.
5. pop-ups???  are you serious, as if ANYONE wants MORE popups.
6. look and feel?  branding?  so?

So the ONLY thing that would be considered important, years of searchable forum messages, will be dumped?  

Can we just continue to use the old forum, and ignore the new one? Now that sounds like a plan.

Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: pwillard on Jan 18, 2011, 02:24 pm
I can't help feeling this is going to be sort of messy.

Good luck.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: robtillaart on Jan 18, 2011, 03:35 pm
Does the new forum handle SPAM in a proper way?
Or will there be a separate section for commercial messages?
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 18, 2011, 03:36 pm
orbitalair, mowcius:

as for moving the data, it is actually pretty simple, thing is ... if you are willing to rearrange 350664 Posts within 49236 Topics from 37469 registered members into the categories of the new forum, I will be more than happy to pass the task over to you guys.

;-)

That was obviously a joke. Now seriously, thing is that over the last 5 years this forum has generated a lot of data which is valuable to many, however there is also a lot of the same question asked many many times. We want to have a clean start also bringing the community in acting as moderators to curate the best forum possible.

The information will still be available and you will keep your user name and profile. If there are interesting posts or opened topics you want to keep on working with, you are more than welcome to bring them in. It is our goal to offer a more inclusive system that will also allow for non-english speaking people to have more and better subcategories, etc.

orbitalair:

I know you personally don't care about rebranding the forum to look as the Arduino forum, but there have been others that have asked for the opposite. It is a matter of opinion, as you know.

Sure we could make this on Yabb, but it is a fact, and this has been a complain from people for some time, that Yabb is running on Perl and that makes it very slow for the pace of growth Arduino is experiencing. Just to give you some numbers, our online system registers over 30.000.000 hits per month, with over 600.000 unique IPs, that eats the 8CPUs of our machine pretty fast, so optimizing and moving to a DB based system is crucial.

mowcius:

Regarding the forum system we are moving into, it is a strongly tweaked version of SMF. We made a survey over a group of users and there were no clear preferences for any of the forums under evaluation.

We have implemented our own user profiling system and a menu bar "Google style", that will allow users know where in the site they are, whether they are logged in or not, and whether they have editing rights for a certain page. We have in the pipeline implementing other services, but without making this move, we weren't able of making it the proper way.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: cr0sh on Jan 18, 2011, 04:22 pm
Quote
Regarding the forum system we are moving into, it is a strongly tweaked version of SMF.


David - please say that whatever you're doing, you're keeping the old smiley emoticons; I mean, I really love the ones here on YaBB - other forums I'm on (I guess my guess of vBulletin was wrong, btw) the emoticons just don't look good. The ones on YaBB are clean, concise, and convey what they need to convey nearly perfectly.

-sigh-

So - how is "number of posts" going to be handled? Will everybody go back to "newbie" status? How will real newbies (ie - those who just bought their first Arduino Uno) know the "old timer god members" (those who have had an Arduino for a while, or those who have an original hand-soldered homebuilt ATMega8 board from way-back-when)?

That would have to be my main concern (waaay above the emoticon issue!)...

:)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 18, 2011, 05:37 pm
A lot of EXPECTATIONS as well as DOUBTS ! the only thing i can do is to [size=12]WAIT AND SEE [/size]
and yeah I guess every one out here loves these emoticons  :)
Quote
So - how is "number of posts" going to be handled? Will everybody go back to "newbie" status?

Though that wouldn't mean much to me at least the REAL GOD'S out there should have their status to themselves ( lol i would be even happier if even i have a title God member ;) )

P.S i never had any idea this was really going to happen
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Big Oil on Jan 18, 2011, 05:51 pm
Make an open source community program to transfer all the data.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: FEDERICO on Jan 18, 2011, 06:05 pm
Perhaps it could be possible to integrate the actual forum in the search engine of the new one? Hope this is "migration" is decided not to have less duplicates, because I think that the same questions are going to be asked forever...
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 18, 2011, 06:50 pm
Quote
as for moving the data, it is actually pretty simple, thing is ... if you are willing to rearrange 350664 Posts within 49236 Topics from 37469 registered members into the categories of the new forum, I will be more than happy to pass the task over to you guys.


I will do it, that is one of the things a computer can handle, now which forum system are you moving to so I can start to prepare, and when can I expect a copy of the data

not joking (I may not be successful but I will at least try my best, and I have moved a forum nearly as large over 4 systems to get from A to B with a rotten database)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: cr0sh on Jan 18, 2011, 07:56 pm
Quote
I may not be successful but I will at least try my best, and I have moved a forum nearly as large over 4 systems to get from A to B with a rotten database


Be careful what you ask for, Osgeld... ;)

Based on the blog posting (and I may be completely wrong about this), there seemed to be the implication that this current forum -isn't- using a DB back-end under Perl (flat text files, anyone?)...

If true, such a system could be a real nightmare to migrate without some serious script-fu to stuff the data from those files into a DB properly. It wouldn't be impossible, but it wouldn't be quick nor easy, either.

Good luck if you manage to pull the task!

:)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 18, 2011, 08:17 pm
it is a flat file system, but step 1 is to get it into SMF,which there are scripts available already, from there it can be converted to just about anything
;)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 18, 2011, 09:36 pm
Quote
as for moving the data, it is actually pretty simple, thing is ... if you are willing to rearrange 350664 Posts within 49236 Topics from 37469 registered members into the categories of the new forum, I will be more than happy to pass the task over to you guys.

Yeah I'm willing to help with this.

Quote
So - how is "number of posts" going to be handled? Will everybody go back to "newbie" status? How will real newbies (ie - those who just bought their first Arduino Uno) know the "old timer god members" (those who have had an Arduino for a while, or those who have an original hand-soldered homebuilt ATMega8 board from way-back-when)?

Also a concern.

Presumably post counts could be manually edited in the system. Give everyone the post count they currently have.
Also what about 'member since' - is that going to be reverted? - Please no...

And my private messages?
Quote
Good Evening mowcius, you have 274 messages, and no new messages.

What's to happen with them if the system gets changed - can I still access them at least?

Mowcius
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 18, 2011, 09:45 pm
I would not really worry about putting things into new catagories, convert, stuff in a read only archive subsection and call it a day, but with conversion you have everything searchable in 1 system and usernames / passwords / private messages are preserved

1.3 cents
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 18, 2011, 09:48 pm
Quote
I would not really worry about putting things into new catagories, convert, stuff in a read only archive subsection and call it a day

Perhaps it would be possible for a user to transfer a thread when required from the archive - then they choose which subsection to put it under and it then starts a new thread (with the old posts included).

Not sure if that would be possible/useful.

Anyway - conversion is required...

Mowcius
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 18, 2011, 09:52 pm
depends on the system, most plain forum system it would involve a moderator action, maybe a request form, on more real things its quite easy but they are much more than just a forum
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 18, 2011, 10:36 pm
I think most of the comments you are making are pretty meaningful and I must admit some of them we had probably not given enough importance to.

Let me elaborate this, I will do it as a list, since it will allow for further commenting by those of you interested. I think the discussion at this level is very healthy, and even if I might not be able of sleeping in the incoming weeks, I will do my best to make your concerns come to a good end.

1) the icons: dude, if you like them, I will make sure they are there, but please first give a try to the new ones :-)

2) the current status and amount of messages posted: this is doable, we didn't have this into our list, but I can understand the importance of the status for you guys and I will talk to the team in order to try to make this happen, if not the day planned for the migration, then later. I must say our idea was to promote those of you really active to moderators, but that was of course a given

3) transferring the previous posts to the new system (this is a longer issue, let me explain it)

Yabb is a file based system, it counts with 4 files per user: userinfo, user logs, user posts, and user messages. 37.000 users means 4 times more files ... IN A SINGLE FOLDER

The converters existing for moving from Yabb to SMF are NOT working, sorry for the SMF developers, but it took me a week to hack it and make it work again.

Right now we have stream-lined the process of migrating the users but not so much the posts. Mostly because we consider the new forum has to be structured in a completely different way. After 5 years with the current set of categories, we have gained quite some experience in the topics that matter the most to people and have identified the categories that seem to make the most sense.

The time it takes to migrate the 37.000 users to SMF is irrelevant, in my 2.3GHZ laptop I need about 3 hours. Then the database has to be filtered, because as explained, this is not just about the forum, this is about having a so-called Single Sign On system that will work all over the Arduino system: A single usernam and password to work on the forum, the wiki, the blog (the day we open it for external editors), other wikis, the store, and future services we have in our roadmap.

This means, our SMF is not really SMF, it is SMF made into our own identification system.

However, even if I think it is a perfectly valid point the one of having a single location where to search for information from the forum, we didn't conceive it like that in the first place. One possible solution, that cannot be scheduled for next Monday, but could be in a week more, is to make a category called "old" where to push in all the old messages into a single table, having the current categories as subcategories of that old forum. We could make it so that that table would not be editable but by moderators. Then users could request to move some of those topics to the new boards.

This is a lot of work, but if there are 15 volunteers, we could consider it. To me it will me I will sleep one week less this year, but it will be worth the effort.

Summarizing, I think the following things have to be considered:

- moving the current user status to the new system
- adding the current messages as a subcategory just to have them searchable
- checking the emoticons and maybe use the current ones (just to please you above all the other 36.999 users :-) )

Something to think about:

- letting a bunch of moderators access to a read-only archive to -upon request- move topics of importance to the new boards

Any other thoughts?
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 18, 2011, 10:52 pm
Quote
- moving the current user status to the new system

Sounds good.
Quote
- adding the current messages as a subcategory just to have them searchable

That sounds like a much better way to have it.
Quote
- checking the emoticons and maybe use the current ones (just to please you above all the other 36.999 users Smiley )

Hehe - I see what he means - can't remember what the SMF ones are like atm but perhaps that is a discussion to have on a thread when the new forum turns up.

Quote
Quote
We could make it so that that table would not be editable but by moderators. Then users could request to move some of those topics to the new boards.

Quote
- letting a bunch of moderators access to a read-only archive to -upon request- move topics of importance to the new boards

Yes - something like that sounds like it could be an acceptable way to go - not sure how 'requests' would be best submitted. Perhaps we can have a 'request' button on the 'archive' forum section.

Quote
Any other thoughts?

Private messages? Can they be stored at least or some way of backing them up - I won't have the time to try and back them up individually before a forum switch (and that would be one hell of a copy/paste job).

Maximum dimensions on images (preferably with auto-resize) so we don't get people posting a few thousand px images slowing it all down and making the threads look a mess. Thumbnails for images over ~600px wide is possibly the best way to go here.

Quote
This is a lot of work, but if there are 15 volunteers, we could consider it.

Count me in for helping.

Mowcius
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 18, 2011, 11:05 pm
not to be blunt but if your thinking of a single sign on system why not use a single system (ie joomla) to handle it all instead of patchworking all these little systems together through bridges and hacks which makes even a simple future upgrade (ie security) a nightmare as you fight with rehacking stuff and opps bridge 1.2.1 doesnt work with front end 1.1.2 and etc

count me in on helping too (if that were not evident)

(and for the record I have had the smf converter working for yabb 2.2.3 for a few months now, whenever it was you put up that poll / test forum  ;) though its not perfect, its never perfect)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: cmiyc on Jan 19, 2011, 12:12 am
What about a preview of what the new forum is going to look like?

It is pretty clear everyone has hopes and wishes for whatever is coming next.  Not everyone is going to be happy.  But it would take a lot of guesswork (and apparently worrying) if the interested parties could see what the new forum looks like.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: liuzengqiang on Jan 19, 2011, 12:25 am
May I suggest more local boards, like USA/MN so we can organize local gatherings to share experience face to face. Helps to know who does what in your area, don't you think so?
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: macegr on Jan 19, 2011, 06:59 am
I've seen "migration" attempts where one forum was left frozen as an archive. It's horrible.

First of all, the huge number of posts in an existing forum should be taken as MORE of an incentive to move that data into the new forum. The more posts you have, the greater the value of that data set. If you had only a few hundred posts then it wouldn't be a big deal when you delete the old forum.

Second, you'll eventually have to delete the old forum and that will cause major havoc. Since the old forum with its wealth of knowledge will be there, people in the new forum will be crosslinking to it like crazy. "Oh yeah your problem is just like this thread, check it out." In the meantime, the old forum become more and more dated, possibly exposing security issues. You might be forced to shut down the forum entirely...causing big problems in the new forum because all these links point nowhere.

The existence of two forums will be very confusing to new users. Anyone stumbling into the old forum from a search result will be struck by the tackiness of having a giant banner at the top directing them to the new forum. It the kind of stunt people don't do anymore, really.

I really hope the old forum data can be migrated into the new forum somehow...even an Archive section is much better than the alternative. Of course, when you choose another forum system in a couple years, you start having archives within archives...refactoring the old topics into the new structure is a better overall solution. And yeah, it will take more work than just putting an empty forum up.

I have zero time to contribute to migration beyond my above warnings from past experience, but I hope you guys can figure it out somehow.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Senso on Jan 19, 2011, 07:37 am
I like SMF emoticons/smiley's, but just deleting everything.... ::)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Yot on Jan 19, 2011, 08:51 am
Nice to see some constructive discussion on this. Timing could be a bit better but hey, better late than never.

Jeroen
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 19, 2011, 10:38 am
Quote
I really hope the old forum data can be migrated into the new forum somehow...even an Archive section is much better than the alternative.

Get it migrated into an archive and then let some people move threads over to the new topics. I'm sure that it would not take too long to get the threads moved manually. I'd say I could easily do 100 a day or so without too much trouble.

Mowcius
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 19, 2011, 01:04 pm
Wish i could help you people !
:( i don't know anything related to converting things but
I could help you probably in the dumbest possible ways ;)
i can just copy paste the members status and post count into a spread sheet  ::)  :-[
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 19, 2011, 01:43 pm
Quote
i can just copy paste the members status and post count into a spread sheet

I think that would be rather a waste of time :D
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 19, 2011, 02:02 pm
Quote

I think that would be rather a waste of time Cheesy

I know that   !  :) but just in case is they would find it  helpful i would do it i guess at max a ten hours work ! i could do it for all the wonderful people out here ( including you ;)  
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Selby96 on Jan 19, 2011, 02:36 pm
But.....but.....but.....,
SMF already provide a YABB converter!?!
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 19, 2011, 03:24 pm
Quote
But.....but.....but.....,
SMF already provide a YABB converter!?!

Quote
Quote
David Cuartielles
The converters existing for moving from Yabb to SMF are NOT working, sorry for the SMF developers, but it took me a week to hack it and make it work again.

REad through the posts ;) :)


Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Selby96 on Jan 19, 2011, 04:01 pm
From the SMF website -
Quote
In case there is no converter for the forum software you are using, you can request a converter.

If it hasn't already been done, it's got to be worth a try.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: roypardi on Jan 19, 2011, 05:22 pm
Good luck with the migration. I hope the new system isn't the same as the dreadful one Processing migrated too (slow, buggy on OSX, over-designed).

Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 19, 2011, 05:25 pm
the converter for yabb 2.0 works fine (for a converter) it does require some tinkering to get it working with 2.2.x which is what is in use right now, and major tinkering to get it working with anything newer, but there is a mod that mostly works  

you can request, but its not like the smf team is going to drop everything and get right on it, if there is someone else who knows how to do it and has a need to do it then they just [glow]might[/glow] volunteer to do it, it may be tomorrow, it may be well over a year, it may be never as in the case of the yabb 2.4 converter where someone forked out money to an outside developer to make it
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: hypermiler on Jan 20, 2011, 08:05 pm
Just an FYI: vBulletin's free ImpEx tool (free if you buy a copy of vbulletin) can manage an import from YaBB2. If the cost of buying a VB license is the issue, I bet we could drum up the dough.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 20, 2011, 10:14 pm
guys,

we are working on this, we might have to delay the launch a couple of days to get it to work, but believe me when I say we are working around the clock.

/d
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 20, 2011, 10:43 pm
http://bambuser.com/v/1343038

and there goes the proof I am working with this right now

/d
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 21, 2011, 01:05 am
man we have waited for a while now, it's not going to kill anyone to wait more, so don't go killing yourself over it
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 21, 2011, 02:03 am
haha,

no worries ... we want to pull this out asap. Cristian and I are trying to get it done as scheduled mostly because our agenda gets really messy later on. However if I don't see a clear way out by tomorrow night, I might postpone the launching of this. But I feel its pretty close.

Also, once it is ported we will need all the help from everyone to generate other things like the help panels, moving posts, etc. There will be fun for everyone.

Thanks for your concerns though, or as we say in my hometown "OSPA" :-)

/d
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 21, 2011, 02:35 am
Just to keep you all updated, I just managed to import the whole forum to the new system. I have to do some more CSS work, but we will continue the migration according to the plan.

/d
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: macegr on Jan 21, 2011, 03:17 am
Sweeeeeeeeeeeeeet.  [smiley=thumbsup.gif]
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 21, 2011, 06:16 am
good work!
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 21, 2011, 12:37 pm
Great!  :) so everything will remain same :)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: mowcius on Jan 21, 2011, 12:48 pm
Quote
Great!   so everything will remain same

Umm, no.

Good news though. I look forward to trying to break it on Tuesday
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 21, 2011, 01:07 pm
hehe well yeah everything will not remain same Yes !  :) but i mean about the post's and topics being viewable and not frozen
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: AltairLabs on Jan 23, 2011, 07:39 pm
Count me in as a volunteer to sift out "significant" archive threads and move them to relevant new categories.  I'm still new enough to be discovering "jewels" in the trove.  Although my agenda sometimes gets messy, I might contribute significantly in hardware forums by whittling away over the long term.
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: newman on Jan 24, 2011, 03:45 am
Is it Today ?  :) Waiting for It  8-)
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: Osgeld on Jan 24, 2011, 05:30 am
it might be pushed back a little as per previous post
Title: Re: FORUM MIGRATION SCHEDULED JAN 24th
Post by: DojoDave on Jan 24, 2011, 09:19 am
guys,

it is starting now

/d