Saturday, January 5, 2008

Hullaballoo about NPO evaluations

The debate continues on how to evaluate NPO's. Here's a comment I posted on the blog Tactical Philosophy.

As a former public school teacher and administrator and the current Director of an NPO, I have been involved with the challenge of evaluation for a very long time. And I couldn’t agree more with Kevin Jones when he wrote: “It’s (evaluation of NPO’s by fixed metrics) an epistemologically impoverished frame to impose a manufacturing metaphor” and “the first thing metrics have to prove is that their imposition does no harm. stop and think before you stick some crazy metric dashboard into an enterprise”.

Most people are aware of the numerous ways in which schools class children and the role that this sorting plays in the reproduction of society (Bowles and Gintis, Illich, etc). Many smart and dedicated people who are concerned with the problem have spent a lot of time and energy designing and experimenting with alternatives. Although there have been some noteworthy reforms (i.e. the introduction of state-wide standards such as the Vermont Curriculum Standards, the creation of new tools such as rubrics and the student portfolio and the National School Reform’s retraining of teachers as “critical friends”), the general consensus among educators is the following: there is no silver bullet for school reform.

There is no single metric which can be applied to a student, because there is no model person and no given path to any particular end. That is not to say that standardized tests are of no use, but rather that there is no single tool or collection of tools that can tell us everything that is important about an individual. Likewise, there is no standardized collection which tells us all we need to know about an NGO. Because NGO’s are as diverse as people and there a innumerable ways in which they can be effective.

Evaluation is best considered as a process, not a score, a process of give and take between the evaluator and the evaluated. The tools that comprise the process need to be multiple, varied, frequent and responsive in order to accommodate an organization’s difference and growth organizations and the ever-changing values of society.

Furthermore — and Kevin makes this point beautifully — it is critically important that the evaluating process not harm the evaluated. On the contrary, in my mind, a useful test is one that causes the test taker to learn something — a prompt towards self-evaluation and self-directed growth.

NGO’s, like schools, are in the people business, which requires a combination of clarity of mission, rigorous self-evaluation and reporting and constant revision and retelling of the story to garner support. The evaluation process needs to respect the organic nature of the work and support, rather than hinder, its development.

We shouldn’t let pressure (public or market) force us to provide a product that the sector doesn’t call for. Instead, we should try to teach the public how to conduct due diligence and encourage NPO’s to turn the public’s need to know into an opportunity for improved governance.

2 comments:

AJ said...

Posted by Maureen |

Here’s a comment which I wrote and posted at metafilter. I’m reposting all but the blog which is the one above:

“Below, you will find the blog Miko referred to — “mob” reference and all.

Of course, not everyone “from MeFi” who commented on GiftHub acted as part of a mob. And some people “from Gift Hub” acted with mob-like mentality. Many participants, however — including Miko — listened carefully and expressed themselves with intelligence and consideration.

I expect, having read and signed the same user agreement you have signed at MeFi, that I will not be ripped to shreds over my blog. Or at least before having done so, you will consider the point of view expressed below.

F.Y.I. I’m not Gen Y and I’m not a Boomer either. I’m 44. I got involved in GiftHub for the first time in late December when I wrote a comment that Phil reposted on the Main Page. I’ve continued to participate in the givewell discussion, because as the Director of a small NPO, I care about charity fund-raising, and as a disgruntled U.S. citizen, I’m concerned with the way most Americans are unable to talk with each other about differences more complex than preferred pizza toppings.

I am particularly interested in the taboo in our society concerning identity, including the question of alternative discursive styles or communication techniques, which is, as you all know, at the heart of the givewell crisis and the ensuing debate on GiftHub.”

Jeff also posted on this thread:

“There has been a lot said here about how the same rules of honesty and transparency that apply in the real world should apply on the web. (I couldn’t agree more.) The one thing I don’t quite get is this. Isn’t it also true that the same rules of civility that apply in the real world should apply on the web? Look at how you all are treating Phil who has come over in the spirit of peace and reconciliation. Is this the way you would treat someone in the ‘real’ world?

As Phil remarked in a comment on GiftHub, “This is one of those web sagas that may draw wider scrutiny.” For those of you haven’t quite cottoned to what is going on here, Phil is setting the stage for you to demonstrate that you are a mob. Or prove that you are not.

I am tempted to say, satirically, “Go ahead, crucify him. Remove all doubt.” But actually, I am hoping that you will do the opposite. (Though at the moment, the odds don’t look good.)

Just in case things don’t work out, in which case I won’t be reposting, I will add a few words directed at the more apparently reasonable among you, such as Miko, Jessamyn, Cortex, etc. This is a rhetorical question. No public answers expected or desired, or for that matter, structurally possible: Speaking of rules that ought to apply in both worlds, why should you not be judged by the company that you keep?”

Some of the MeFi commentators were proposing to “gravedance” in response to the announcement of Holden’s demotion from the position of CEO at Givewell, but a number of MeFi regulars, including the paid moderators Jessamyn and Cortex, spoke up against that response. Jessamyn pointed out that MeFi faces the very big challenge of optimizing freedom of expression in a community of 65,000 (of whom 5000- 6000 make a post per week). This fact points to a very pressing question for MeFi and other virtual communities: Can MeFi expand its userbase indefinitely? Is there a limit to the number of people who can participate in a free, frank, egalitarian and civil conversation? I am interested to follow the developments in this site (which I find fascinating for the range of topics and conversational gambit), as well as, the metadiscussions on GiftHub, Tactical Philanthropy and elsewhere.

Jan 7, 3:53 PM

AJ said...

Posted by Maureen

The tension rose as the discussion continued, leading ultimately to a kind of cyber war between GiftHub and MetaTalk. Both sites contributed to fanning the fires. I left MetaTalk after a few days of unsatisfying exchanges, because it struck me as an untenable situation. As I explain in the reposted comment (from Gift Hub below), MetaTalk struck me as a totalizing system:

Thank you JJ for the link to Michelle’s comment. I follow her argument but reject her framing of the situation. I signed off at MetaTalk, because it is a totalizing system. I posed my question to Michelle here, rather than on MetaTalk, because I wanted to broaden the conversation beyond a rationalizing defense of MetaTalk and explore the difficulties inherent to speaking in public (whether the conversation is on- or off-line).
What do I mean by MetaTalk being a “totalizing system”? It’s precisely the logic you find in the linked discussion, which I recap in the following narrative:
A newcomer (a”welcomed guest”) to MetaTalk takes a look around, notices a few things about the conversational dynamics and decides to point them out, because this is the only forum for meta-discussion on MeFi. In particular, she explains why she finds the meanness gratuitous and how she feels the habitual meanness impedes open discussion.
Many people who are successful in this space (the “insiders”), including some who lament that users don’t exercise more self-restraint, argue that permitting meanness is necessary for open discussion (ironically, all the while simultaneously condemning certain modes of discourse, such as allegory and satire).
The “newcomer” (as labeled by the “insiders” ) is advised to check out other MeFi spaces that are less “rough and tumble” in order to get to know the larger community before coming back to MetaTalk and expressing an opinion. The “Newcomer” is informed that through exposure over time the culture will grow on her. She is also told that there’s a steep learning curve and given instructions on how to post effectively (i.e. in a way that minimizes exposure to “snarky” attacks).
She takes away from this interaction that the bottom line requirement for participating in MetTalk — the only space for a meta-discussion of this “open” forum — is adjustment.
The “newcomer” (who now fully assumes the label and role of “outsider”) refuses to accept the bottom line prerequisite of adjustment and tries to further explain her point of view. The reaction of the group is unchanged, except for becoming more forceful and “snarky”: “Adjust, stick around and critique our mode of discussion (but we won’t listen) or walk”.
“Insiders” now rechristen “newcomer” as “newbie”. “Newbie” who has already decided against adjustment (she’s opposed on moral grounds) is now confronted with a difficult choice between limited and unappealing options: 1) play the game 2) play a different game 2) not play at all.
If she decides to stick around and plays, she has some further choices to make:
1) She can unleash the id and join ‘em.
2) She can keep the id on a short leash and not talk about the elephant in the room (the fact that it’s not okay to have a meta-discussion about the way people talk in this “open” forum). 
3) She can keep the id on a short leash and continue to try to explain herself.
The “newbie” decides against all 3 options because:
1) She finds it dehumanizing and unconstructive, not to mention imprudent, to “go native”. 
2) She isn’t willing to ignore the elephant, because she doesn’t accept the argument that id-unleashing is a pre-requiste to earnest expression and doesn’t want to implicitly condone that argument by participating.
3) She has tried playing a different game and failed. In a system that has no play (no opportunity for reframing the situation or altering the nature of interactions) there is no point to trying to play at all. The only logical — and sane — alternative, is to go away.
So she decides to leave, even though she knows that the “insiders” will label her departure as another “newbie’s flame out”, disregard her critique, circle the wagons, invent a story to rationalize the outcomes, declare victory and resume business as usual at the self-coronated quintessential open forum.
The message is loud and clear on MetaTalk : No make-ie the rules, no play-ie the game.
I for one have no interest in participating in a totalizing system and recognize that participation in such a system merely serves to keep it alive.

Lkewise, when the discussion degraded on GiftHub, I also signed off there, but only from the discussion thread not the site. Although I am disgusted by the reactions of some of the GiftHub “regulars”, I think that because the site has interesting information and is not brutal nor “totalizing”, it’s worth spending some time there:

Seems to me there are some lessons to be learned here from South Africa’s process of Truth and Reconciliation.

In this discussion, the historical record has been more or less set “straight”. Unless there’s a good reason (i.e. criminal charges) for continuing to probe the givewell crisis and subsequent cyber wars, I’d say it’s time to call it “truth” and move on to reconciliation.

Since no amount of conversation will ever overcome some of the irreconcilable differences between the various agents, it seems to me that it is time to decide to disengage.

Maybe if everyone counted to three and yelled “uncle”, we’d all be content to call it quittin’ time!

I have the good fortune to be going away on a business trip where I won’t have a regular Internet connection (in fact South Africa to work with museums) and therefore will not be able to reengage with this discussion till early February (by which time I hope it’s over)!

Jan 15, 2:11 PM