Unfortunately, it’s often impossible to tell what will have the biggest impact if we always give everyone what they think they want. Science is a bitch sometimes.-Jodi Beggs my favorite keyboard playing economist on her blog Economists Do It with Models.
She has a great post up about a program in New York that is randomly denying funds to people who apply to get help with their rent to avoid becoming homeless. The program has found that 90% of people who get help stay in their home, but the question is what would it be if there was no help. Perhaps it would 0% or 89% would still stay in their homes, a random experiment is the most accurate way of measuring the effect. However, it might not be the nicest.
One thing that happens in the developing world when a new program comes the group who doesn't receive the benefit(the control group) gets some other benefit even if it is not part of the design. For example I heard of one country where in half of villages children were payed to go to school and in the other half no benefits should have been received. But the government added in free school meal programs in many of the control communities increasing school enrollment in control communities and biasing the effect size downward.
I also heard of a randomize worker training program where those in the control group were given extra information about where else they could look for jobs.
So yes science can be a pain!