I have started a couple of "Special Interest" groups over the past 25 years with a degree of success. From my perspective, the hardest thing to do is make the community aware that the group exists, what it aims to do and what members of the community stand to gain from involvement.
Advertising is the key but obviously, there will be no money for any kind of advertising. There are however 3 things that you can try to start spreading the word that cost nothing more than your time:
1... Local newspapers will often print editorial as a community service if you present them with a pre-prepared, well written and concise offering.
2... If you have a community radio station, they are always looking for local content and can be really valuable with regard to spreading the word.
3... Shopping Centre (Mall) displays. Appealing to shopping centre managers for a small area free of charge in which to set up a meet and greet table can often work quite well in kick starting such ventures. The more people that you talk to, the more interest will be developed.
You should be aware though, that you might run into the issue of Public Liability Insurance. I have in the past, overcome this by having a local business sponser the meet and greet and cover this requirement with their PLI.
Good luck with this. It is certainly a worth while venture and in my experience can be a whole lot of fun.
Ian B ;)