Ammolytics has some pretty in depth stuff on their blog about doing this. The short answer is it's really simple, but you sort of have to know where to look for the information you need. There's also quite a bit of stuff over on the Accurateshooter forums from the early stages of the Autotrickler development. If you want to re-invent the wheel, though, I'm all for it.
First of all, I'd pore over your manual for your scale and see what exactly you can do with it. The RS232 stuff is probably the only daunting part of all of this and that's all a matter of the quality of the documentation. You may be able to set target weights and whatnot directly on the scale. (Can be done with some A&D scales for sure.) You may only be able to pull data, in which case you simply need to be able to read it and format it in a way that you can compare it to a value.
If you want to get into bluetooth or wi-fi and apps and all that you'll have to research those aswell. That territory is very well covered, and since this project is really very simple at heart (Set a value A, read a value B, compare values A and B, output a simple on or off signal) almost any type of user interface can be made to work. Ammolytics might have a plug and play solution for this depending on your desired set-up and existing equipment.
As for the mechanical design part, I'd look at all the current electronic tricklers and figure out what route you want to go. They can be as simple as a 3 volt vibrating motor or as "complex" as a stepper or servo. (Neither are complex, but they are slightly more complex than a basic, two wire, DC motor.) You'll want a motor driver for any of them, but that's mostly to protect the board and to prevent noise. A cheap motor driver for a basic motor of sufficient size costs about a dollar or two.
I've actually been about as far down this particular rabbit hole as a person could get here in the last few months. I'm building a system around a 50 year old opto-mechanical analytical balance, though, so my solutions are going to be pretty different.