A quick visit to www.greatschools.org can be quite instructive into the world of evaluation – but not for the reasons one might expect. The website essentially aggregates the publicly available data on almost any school in the United States and then assigns the institution a score between one and ten. Standardized testing provides much of the information used to derive the ranking, so the results naturally mirror the flaws of the process. But if the data-generation processes are least equally flawed within a state or region, the rankings can still be indicative.
However, what is most interesting about the site is the opportunity it affords to juxtapose its positivist approach with the comments and rankings of the parents of students. A decidedly unscientific and unrepresentative sampling indicates there’s no firm correlation, but I was struck by the overwhelmingly positive commentary and 5-star public rating on a school currently scored by the site at an abysmal 2 out of 10. Various possibilities crossed my mind.
1) The data-based ranking is entirely wrong. (Possible but improbable)
2) The school is manipulating the site mechanisms to improve its image (possible, but not immediately verifiable)
3) The parents are responding from the context of a multiplicity of biases (quite probable)
4) A combination of all of the above (also quite probable)
Beyond the fact that humans are just inherently bad at examining data objectively, if a parent is compelled by civic statute and financial circumstance to send his or her offspring to a public institution, would that parent acknowledge in any open forum the fact that the children are receiving an awful education, thus condemning the children as much as the school?
As proud father of the two most beautiful and intelligent girls in the world, I’m quite convinced that parents are least able to respond objectively about their children (they tend to disagree with my first point) so I grow concerned when I read evaluation after evaluation of Sistema programs which rely heavily on parental input. The extremely heavily positive weighting of the majority of responses strikes me as suspect. Are parents trying to safeguard a free-babysitting service, regardless of quality; attempting to please the assessor; or avoid characterizing their own progeny as developmentally slow? Consider the context: would anyone ever say “No, my child has no more confidence at all!” in response to the typical survey question? The obvious point to be made is that it’s not the answer that is at fault, or even the respondent, it’s the question.
But then, what’s the right question? Or should we ask at all? It’s easy to say evaluation is the major challenge facing Sistema programs right now (I do it all the time), but extraordinarily difficult to make sense of the nature and scope of that issue – which is why the paper presented by four members of the outgoing class of Sistema Fellows ranks in my mind as a document I would actually describe as one of the very few must-reads for anyone genuinely serious about having impact in this sphere of endeavour (my own output is not included on that shortlist, for the record.) Aside from being impeccably written and devoid of the usual insightless rhetoric, it is one of very few papers to connect its proposals and frameworks to a broad base of scientific research while clarifying the major points under consideration. It is intelligent, readable and most of all, useful. Congratulations to Andrea, Carlos, Sara and Elaine for having made a substantive contribution to the literature even beyond the narrow Sistema community. I’d hire them all, if I could.
As we’re looking to refine and focus the new models or frameworks, there are others attempting to do the same within the existing ones. I was made aware of one such program recently, the Opportunity Music Project in New York City. With its focus on private instruction it’s not technically Sistema, but as I frequently am compelled to remind people, much great work in music education is done beyond Sistema programs. I was particularly stuck by the founder’s unique articulation of the idea of parents and teachers as “co-collaborators in the success of the student.” They’re running an Indiegogo campaign right now to support their work, and I would encourage you to give. What persuaded me to pull out my credit card was this video.
The content, the messaging was largely consistent with our industry as a whole, but the tipping point came at the three minute mark. The founder, Jessica Garand, unvaryingly articulate and sincere up to that point, sounds like another person entirely as she outlines her vision for the program. The excitement, dedication and clarity of purpose in her voice and in her message are all unmistakable. If you can skip one Starbucks this week and send $5 their way, please do. Not a positivist response on my part – but since when were our decisions really data-driven?