Home Capacity Building Research & Evaluation
Research & Evaluation PDF Print E-mail

This is one in a series of "think pieces" on which we invite comment and discussion. Each of these short pieces raises an issue that any nonprofit might want to consider when the Board begins to develop policies that should undergird the institution's move toward "self-sufficiency." This piece was written by E. Maynard Moore, Ph.D., one of the partners of CommunityNexus Consulting LLC.

Building the Capacity for Research & Evaluation in a Non-Profit Organization

Practitioners in the nonprofit sector who are continually faced with real problems that demand workable solutions often make decisions "flying by the seat of their pants." That need not be the case.

Institutional research can be an effective tool for gaining knowledge about improving the efficiency and the effectiveness of the organization for purposes of management and decision-making. Let's begin with a rather straightforward definition of institutional research: "a process by which practitioners attempt to study data systematically in order to guide, direct and evaluate their decisions and actions in an organizational setting."

For the busy nonprofit executive (here you can read "president," "director," or minister of a church), a systematic approach to decision-making is both wise and practical. It is in this sense that research and evaluation within the organization involves a set of procedures for generating data that can be analyzed and applied in such a way that delivery of programs and services might be improved.

The knowledge gained in this fashion has the greatest utility for the decision-maker, since it represents a systematic study of the situation that otherwise one tends to take for granted.

A teacher, for example, who compares two distinct methods of instruction and finds that one is more effective in the setting she faces, will likely apply this knowledge in future classes because she now knows what works best for her. The president of any nonprofit should do as well.

Institutional research is thus different from basic (or fundamental) research in the laboratory. It differs in several ways:

  1. Institutional Research is conducted for the purpose of improving a specific set of practices, while basic research is conducted to formulate broad hypotheses or establish generalizations.
  2. Institutional Research is done by people who want to analyze and evaluate their own work, their environment and their policies, while fundamental research is usually conducted by specialists who operate with pre-funded mandates or broader goals in mind.
  3. Institutional Research has its power derive from the involvement of program people (teachers, program directors, service coordinators, department heads, nurses, etc.) who have a commitment to improve their practice by virtue of their own learning, while fundamental research is based on the belief that practices generally will improve as a result of disseminating results throughout the profession, and only incidentally will focus attention on a given organization.
Information for making decisions is only as useful as the extent to which individuals are predisposed to accept it. When it has been obtained through one's own efforts and then verified by objective means, information is more likely to be referenced seriously, since it has passed the test of personal experience. It is in this way that nonprofit practitioners should learn to conduct research without having to rely on experts utilizing academic analytical canons. Even when a nonprofit engages an outside firm to set up the parameters for research and evaluation, the goal should be the development of the internal capacity for ongoing research within the organization. That is what we mean by capacity building.

When conducting institutional research, the practitioner functions as a decision-maker. The program director becomes self-conscious about particular methodologies that may well have been taken for granted before. In the role of research, she evaluates certain practices, determining which ones are effective, which are to be modified, which are to be discarded. When information is gained from such efforts demonstrating usefulness for approaching a similar situation in the future, what has been learned in the concrete situation is important as input into the decision-making process. Thus, from the perspective of institutional research, studying a problem is done for the purpose of deciding which course of action a practitioner should take.

When a research project is established within the context of any nonprofit, whether a school, a church, a community food-bank, a hospital, or an art museum, specific objectives should be defined prior to the application of specific methodologies. These we want to call "categories" for research:

1. Problem Identification:

The task in this category of research is to identify the particular dimensions of a problem whenever such a problem surfaces in the collective intuition of the organization. People might "sense" that there is a problem within the organization, but it is extremely important to identify the problem, not just symptoms. A common example is the matter of "student attrition" in a school. People may get a "feeling" that attrition from one semester to another is abnormally high. Can that feeling be substantiated with data analysis, and if attrition rates are established: (a) can causative factors be identified? (b) are the rates here excessive compared to schools elsewhere? (c) are the causative factors susceptible to internal control?, and etc. A project of institutional research can provide the answers to these and other questions, and as a result, institutional decision-makers will be better equipped to forge some solutions to the perceived problem. There are a number of methods that might be utilized in the research process in this category of Problem Identification:
  • statistical tabulation
  • discrepancy analysis
  • audit techniques
  • comparative data
  • review of the literature
  • analysis of financial reports
  • analysis of demographic data
  • analysis of controlling regulations.

2. Applied Research:

This category of research consists of activities which provide general information about the organization and its environment, e.g. a study of the scheduling patterns from department to department as the basis on which to determine whether distribution requirements can be more easily met. Another common issue for nonprofits is the challenge of establishing comparable salaries within a given field of endeavor, e.g. the salaries of field directors for a wildlife protection society when compared to field directors in other environmental and conservation organizations. Another concern, and this is becoming important across the nonprofit spectrum, is the challenge of determining the attitudes and impressions of the organization's key constituencies, e.g. those who regularly use a hospital's facilities for dialysis, as over against persons who only occasionally come for service or become a patient on a one-time basis. These are situations in which an institution would do well to formulate an Applied Research project, using such methodologies as:

  • survey research
  • institutional climate analysis
  • attitudinal instrumentation
  • analysis of previous data reports
  • participant documentation
  • telephone follow-up techniques
  • peer interview instruments
  • cost/effectiveness studies
  • defining arenas for contextual analysis
  • input/resource assessments.

3. Action Research:

Here we are concerned with research activities associated with specific planning, teaching, program delivery, or administrative policies about which long-range decisions have to be made. Here we are hoping to answer such questions as: (a) how can we modify the patient intake process so as to lower waiting time as well as more efficiently utilize personnel? (b) how can we more evenly distribute the burden of week-end hours for personnel in the field services division? (c) how many sections of a given course should be scheduled in the downtown adult learning center?, and etc. When reports are prepared in the Action Research process, the reader learns the dimensions of the problem under investigation, the methods utilized in analyzing the problem, the conclusions inferred from the results, and what recommendations might change the situation under
circumstances that remain constant. Some of the methods used in Action Research are:

  • experimental design
  • ethnographic methodologies
  • the case study approach
  • survey research
  • hypothesis validation
  • random sampling procedures
  • cognitive and affective measurement
  • direct activity observation
  • identifying transportable strategies
  • field testing methodologies.

4. Program Evaluation:

In this task we will need to assemble the quantitative and qualitative data on which basis previous ad hoc decisions have been made by administrators in various program or administrative units. Here we are concerned with how well particular programs are meeting the purposes for which they have been established: (a) how effectively a program is treating patients who come for service? (b) how efficiently a given department is filing and storing its records? (c) what must be done for an agency to meet the specifications required for state licensing? (d) how effectively a program is producing graduates who meet the standards expected in the industry?, and etc. In order to conduct research that is worthy of evaluative decision-making, one must first establish baseline data against which outcomes can be measured, and establish criteria for decision-making when one knows from the start that no program will ever reach 100% of the benchmarks one might wish for. Some of the methods available for Program Evaluation are:

  • monitoring of progress on a regular basis
  • reviewing interim products
  • updating dynamic action plans
  • correlation analysis: results with expectations
  • interviews & direct interface techniques
  • surveys and random sampling
  • soliciting constituency feedback
  • structured focus groups
  • modified Delphi analysis
  • formative oral interactions
  • impact analysis.

Building the capacity for Institutional Research and Evaluation can be an important and cost-effective project for an organization concerned with maximizing its resources to better accomplish its mission. The goal should be one of building a coordinated system of data collection, analysis and assessment, which contributes significantly to the economy of the organization. If the goal is to build a formalized Office of Research and Evaluation, it can have a solid impact on both the efficient achievement of short-term objectives and the effective attainment of long-term goals. Through the deliberate use of external consulting assistance, the organization's internal capacity should be developed, enhanced, and implemented, with results including:

 

  • Centralization of data and information needed for transactional decision-making and long-term planning.
  • Coordination of market research, needs assessment, constituency follow-up, customer satisfaction, and registration patterns, all of which can lead to more effective program planning.
  • Better utilization of the computer and internal information systems for data management and program planning at the department levels.
  • Coordination of data gathering across all units of the organization and across geographic and community boundaries, allowing for wise decision-making each year in the allocation of resources.

We will be pleased to work with any organization desiring to enhance its process of Research and Evaluation.