My proposal for this is that the development team should be responsible for providing the community with a test plan. This test plan should be a documented series of tests that they performed prior to the release of code in order to verify the proper functioning of every change. The test plan should be executable by community testers on the testnet.
I think one way to go about this is to look at unittests, like those found here:
https://github.com/steemit/steem/tree/master/tests
Then, do a code review of a unittest, similar to this:
https://steemit.com/steem/@inertia/code-review-clear-null-account
Once there's a clear description of what the c++ tests do, there's a path to community testing. It's not always clear-cut because the unittests can do things like summon funds into existence or simulate the passage of time. That's where more tools will be required.
IMO there is a role for both automated tests, and 'human' tests. The amount of people that are qualified to understand and properly audit the automated tests is very limited. If we rely on people being able to do this in order to properly test, we will be significantly underutilizing the resources we have (i.e. the community members who may not understand C++, but are willing to spend time playing around in the testnet).
Agreed, automated tests can only account for a certain number of cases. Unpredictable humans will always find ways to use a system beyond what the developers intended.
Your proposal above is good.