A couple of points you make stick out to me. I used to get a ton of spammy comments and one or two valid ones. Now, I get mostly legitimate comments and a couple of spammy ones, so that is a plus for me as well.
I also think it will be great when the people start to move in and we have more eyes on the site. If we can create a culture of curation it will be awesome. We will be able to teach the spammers right off the bat that they need to suit up and play well, or go away. I am glad you brought that up, because it is another thing to prep for.
I think you will do well even in a crowd, and you do have a network of people who will continue to support you. Those of us who actively use the site, know who the other people are who actively use the site. :)