Research vs. Evaluation

Our office recently administered a survey designed to examine the relationship between congregational vitality and ministerial competence / excellence. In particular, the goal was to see whether–on a large scale–particular indicators of vitality correlated with characteristics and qualities of ministerial competence. Last fall, we asked UCC Conference Ministers (leaders of the 38 regional judicatory bodies of the denomination) to share with us issues and questions of importance that would be helpful to research and would help us to plan our research agenda for the next two years. This particular question was posed by one such minister, and our research intern was eager to explore this area of study.

Specifically, this survey was meant for congregants / worship attenders and asked individuals to think about items of vitality and pastoral skill / excellence pertaining to their own church. However, what has been most interesting in the dissemination of this survey has been some ministers’ reactions to it. Some saw this survey as an actual evaluation of their ministry in the congregation, and a few took this a step farther and feared that the national setting would be using this “evaluation” of ministers to spy on individual churches and pastors. Because we did ask respondents to state the name and location of their congregation, this was interpreted by some to mean that we would then be sending feedback to Conferences, Associations, and churches (there are actually research-related reasons for the collection of this information, such as the validation of survey responses and the comparison with overall church demographics).

Because of the strong feedback we received from some ministers, I have realized (among many things) that the purposes and tasks of research are not familiar to our denomination as a whole. I believe there are three main reasons for this:

a) The UCC’s national setting has not conducted a research project in over a decade, so this process has not been “a part of what we do” for awhile and, therefore, is considered new, unknown, and not completely trustworthy. Other research offices in denominations like the PC(USA), United Methodist Church, and ELCA have had much more sustained, robust research programs where multiple surveys of congregations and ministers are conducted every year. For them, it’s just part of “what they do.”

b) Our polity is one in which autonomy is a central feature of our life together. Specifically, the congregation is the locus of its own authority. When a national setting begins asking questions (no matter how important they might be or how they might contribute to shaping the future of the church), there exists unease among churches and ministers. As I shared several weeks ago, question asking is meant to get at the heart of the matter, to tell us something about what’s happening in our churches–not for the purposes of evaluation (judgment, that then informs decision making on specific programs), but for the purposes of research (knowledge, that then informs policy and procedure). It also stands to reason that denominations which are more hierarchical in nature would have an easier time gathering data.

c) We don’t know what we don’t know. No one wants to admit that at times, but it makes all the difference to understand this crucial bit of knowledge: There are real differences between the tasks of research and the tasks of evaluation. They seek inherently different outcomes, though the methods for arriving at those outcomes may be similar. A very succinct, yet valuable summarization of these differences from the National Institutes of Health says it all:

Research: 
– Produces generalizable knowledge
– Scientific inquiry based on intellectual curiosity
– Advances broad knowledge and theory
– Controlled setting

Evaluation:
– Judges merit or worth
– Policy and program interests of stakeholders paramount
– Provides information for decision-making on specific programs (or individuals or entities)
– Conducted within setting of changing actors, priorities, resources, and timelines

I also really appreciate Michael Scriven’s take on why the differences between social science research and evaluation are not widely understood or accepted. He articulates that evaluation originated as a profession, whereas social science research is rooted in a discipline. Before I entered my Ph.D. program, I wouldn’t have been able to tell the difference between these two equal, yet divergent tasks because the differences are truly visible only in the intent and the outcome. The fact that the methods are exactly the same is what caused several ministers to see this as an evaluation. As a UCC minister myself, I understand completely. Who else besides other social science researchers would know the difference? Because of this, in future survey introductions the importance of clarity and transparency will be paramount in our work.

Ultimately, in writing about this specific experience and my subsequent learning / realizations, I’m no longer the only one who knows the differences. As the UCC continues to live into being a body that conducts regular, robust research for the goals of increased knowledge, insight, and transformation, it is exciting to see that the research process itself is having the same effects. I have appreciated the conversations that have taken place as a result of our first research attempt (whether they are positive, negative, and somewhere in between) because they have created some “buzz” about the role and task of research, the how / when / why of information sharing, and the importance of asking good questions. That is the best possible outcome this minister-researcher could ever hope for.

researchevaluation

 

Graphic from Unite for Sight

Advertisements

One thought on “Research vs. Evaluation

  1. Sarah Drummond in Holy Clarity reminds us that, “Good leaders and good evaluators know better than to think that there are no surprises or mysteries hidden within their ministries. To accept this reality with humility is a first step toward finding holy clarity.”

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s