The Burdens and Benefits of Evaluation

William Schambra, Director of the Bradley Center for Philanthropy and Civic Renewal at the Hudson Institute, makes a fair point in his article, “Measurement is a futile way to approach grant making,” when he laments that “ever more elaborate schemes for ensuring measurable outcomes” have imposed “a substantial and growing burden of measurement for the nonprofit world.”  And he’s right, too, when he observes that “even when measurements have been duly gathered, research shows that they have little impact on actual grant making, not affecting the amount of money spent on a program.”  But he overshoots the mark when he acknowledges, with refreshing candor, that “I happen to believe that measurement is finally a futile way to approach grant making.”  Useful evaluations of nonprofit performance are not a one-size-fits-all proposition, and the costs and value of assessing effectiveness can and should be calibrated as befits the purpose of evaluation:  to get more funding with less effort to more effective organizations.

Mr. Schambra (who wrote a rather nice review of my book for the Wall Street Journal and hosted a panel discussion with the provocative title, “Do We Need a Nonprofit Capital Market,” to discuss it) leads a distinguished policy center that “aims to encourage foundations and charitable donors to direct more resources toward support of small, local, often faith-based grassroots associations that are the heart of a vital civil society.”  So he speaks from experience when he says that “small, grass-roots nonprofits -- which are so often a key source of innovation -- are automatically frozen out of money by the burden of measurement.”


Alas, measurement and evaluation are expensive and difficult, and the small charities of which Mr. Schambra speaks will almost always have more vital work to do with their limited resources.  Nearly all of the work going on to advance nonprofit evaluation (including such efforts as the Social Innovation Fund and the Social Impact Bonds that my organization, Social Finance, Inc., is working on) are aimed at larger, more mature nonprofits.  (As I’ve written previously, one notable exception is Social Velocity, ably led by Nell Edgington, which does impressive work bringing growth capital to small nonprofits.)

[While I’m on the subject, could we please stop referring to randomized-control trials as the “gold standard” of evaluation?  That’s like saying Mt. Everest is the gold standard of mountain climbing, when 99.99% of climbers will never see, much less climb, Mt. Everest.]

But just because it’s hard to justify imposing the burdens of robust evaluation on small charities doesn’t compel the same conclusion for larger organizations.  In fact, Mr. Schambra identifies reasonable criteria for legitimate evaluation that aptly describe where I believe the social sector is heading.  To his credit, he notes that “the burden of measurement might be endurable were we confident that all those numbers we were collecting were somehow adding up to a coherent science of grant making.”  He welcomes an arrangement in which funders and nonprofits “decide jointly on a simple, coherent, user-friendly system to which we can both pay attention, which will prevail over bureaucratic inertia and political connections, and which will feed into a serious body of knowledge.”

I think Mr. Schambra has fairly described some encouraging developments that share a common purpose of trying to make measurement and evaluation more useful and more consequential.  That is just as it should be:  for all the reason he gives, the burden of proof must rest entirely on funders and evaluators, not on beleaguered nonprofits.

Examples abound.  FSG Social Impact Advisors has taken the lead in two areas relating to Mr. Schambra’s concerns:  “shared measurement systems” designed “to evaluate ... impact across multiple grants and stakeholders” and action-oriented approaches that offer “a fundamental transition in the way foundations use evaluation.”  The  Independent Sector, BBB Wise Giving Alliance, and GuideStar USA have launched a “Charting Impact” initiative grounded in “five simple but powerful questions” that give nonprofits “a shared vehicle for concisely conveying your plans and progress to key stakeholders.”  Root Cause’s “Social Impact Research” project (on which I consulted some time ago) “aggregates, analyzes, and disseminates information to help social impact investors identify and support the most effective, efficient, and sustainable organizations working to solve social problems.”  Charity Navigator has launched “Charity Navigator 2.0” (another consulting project) to transform its widely-used but much-maligned star-rating system into a more informative tool for intelligent giving.

These are not the kinds of “ephemeral metric fads” that Mr. Schambra derides.  Rather, they are serious and sustained efforts to shift some, but by no means all or even most, philanthropic giving in a more performance-driven direction, one that actually would significantly affect the amount of funding that the most effective nonprofits attract.

In his seminal article, “Money to Grown On,” William Foster of the Bridgespan Group identified “a small but growing number of organizations ... involved in providing growth capital” and argued that “donors can learn how to scout out and grow the best nonprofits ... [and] certain nonprofits can ... learn how to attract cash for expansion.”  These organizations (which my book calls “growth-ready mid-caps”) meet all of the following seven characteristics:  they address a critical need; they have strong leadership; they have strategic clarity; their programs are demonstrated successes; their programs are cost-effective; they have grown successfully; and they have a sustainable funding model.

Most assuredly, these are not Mr. Schambra’s “small, grass-roots nonprofits,” whose contributions to civic life he rightly celebrates.  But they are the kinds of organizations that have important potential for “scaling what works,” which has attracted the Social Innovation Fund and a coalition of more than 20 funders collaborating under the flag of Grantmakers for Effective Organizations.  All of these efforts are aiming at developing evaluation systems that are not more trouble than they’re worth and that could actually influence the flow of money to nonprofits that are producing the greatest social impacts.

Evaluation critics aren’t far off the mark when they complain about the long-standing imbalance between the large costs and modest benefits of evaluation.  But the fulcrum is shifting because we have an opportunity to leverage the tremendous social innovation the sector is generating with new kinds of financing that are specifically designed to amplify impact and support growth.    While Mr. Schambra has a point when he says “devotion to measurable outcomes is hardly new,” the opportunity to combine more pragmatic and actionable evaluation tools with an emerging interest by “social impact investors” in growth-capital funding is new.  Without evaluation, we can’t hope to convince impact-minded funders to make the kinds of larger, longer and less restricted grants that are needed to overcome the capital fragmentation that holds back social progress.

Mr. Schambra fears that “the burden of futile and pointless measurement will only continue to grow.”  In the broadest sense, I’m not prepared to disagree with him.  I do believe, however, that a strong and promising counterpoint must emerge if the sector hopes to move the needle of social change, and I’m quite optimistic that such a movement is well underway.