You are viewing a single comment's thread from:

RE: Do Users Leave Hive Because of Downvotes?

in Hive Statisticslast year (edited)

The problem with this test design is that it's very hard for the samples to be genuinely independent. Without performing an actual controlled experiment, ie. setting the population and introducing the variable yourself, attempts to sample users who received the treatment (downvotes in this case) basically always involve sampling bias - there is some reason this user was downvoted and one other was not, and this is difficult to control for.

The advantage of the method above with granger analysis is that it works reasonably well retrospectively, by not making such attempts to sample - looking at the whole population.

Sort:  

That makes sense.

There would still be some level of a correlation you could detect though right?

The sampling bias would mainly cause a problem for being sure if the correlation indicated causation right?

But it could at least be seen - does downvoting correlate to users not participating anymore. There could still be a question as to whether that's a good thing, a bad thing, or if the downvote was likely the cause or not.

Keep in mind... this is all with only the vaguest of understandings of proper controls and testing on what you're describing - fully aware that my thoughts can be full of blindspots :)

You could see such things as those users who receive downvotes being more likely to leave, but there are many forms of sampling bias that could end up giving you that same conclusion mistakenly. It would take a greater degree of rigor to control for such things. Possible to do but takes more work and expertise.

It might be worth doing these kinds of tests anyway, because it can be interesting to examine even while being aware of the limitations.