Practice Evaluation and Social Group Work in Elementary Schools

Abstract The year-end reports of the group work practice of 15 elementary school social workers were analyzed to determine clarity of goals, intervention methods, and evaluation strategies. It was found that goals were articulated but not quantified. Purposes and intervention methods were clearly documented. Goals for individual members were identified more frequently than group goals. Evidence of group processes was commonly used as markers of group success. Social workers used a variety of activities and discussion to accomplish these goals. Pragmatic outcome indicators, such as member, parent, and teacher statements about goal accomplishment, improved quality of life, and increased ability to cope, were used more often to assess outcomes than empirical indicators. Identification of group processes and member enjoyment of the group were indicators of group success that emerged from the reports. These pragmatic indicators appeared to be somewhat useful in evaluating group work processes in these elementary settings. Outcome effectiveness was not well documented in these groups, however, because empirical results were rarely defined. Because of the ethical mandate to evaluate practice effectiveness, and because clients prefer the use of objective outcome measures, school social workers would be well advised to document outcomes more effectively in their group work practice.


PRACTICE EVALUATION AND SOCIAL GROUP WORK IN ELEMENTARY SCHOOLS
Evaluation of the effectiveness of social work practice is not only an ethical mandate (Meyers & Thyer, 1997;NASW 1996;2002), but a political one as well (Depoy & Gilson, 2003). School social workers must be willing and able to show that their interventions are successful in meeting student needs (Allen-Meares, Washington, & Welsh, 2000). Funding, and sometimes even continued employment, depends on the demonstration of practice effectiveness (Franklin, 1999).
School social workers frequently work with students in small groups to help foster educational and socio-emotional success. Evaluation of group work outcomes may be even more complicated than evaluating individual successes because of the complexity of group work. Not only are there several clients to be monitored simultaneously, the group itself becomes an additional client (Shulman, 1992) in need of evaluation. Rose (1984) suggests that group processes should be monitored systematically by gathering data on group cohesion, relevance, and satisfaction. It is not totally clear how well school social workers have combined their group work efforts with practice evaluation in order to articulate the effectiveness of their group work practice. This research is an attempt to obtain information about the nature of school social workers' evaluation of their group work practice.
This paper describes a qualitative document analysis (Monette, Sullivan, & DeJong, 2002) of the group work descriptions in three years of annual reports of social workers who serve 18 elementary schools in one Midwest school district. A content analysis of these reports was conducted to determine the extent to which these social workers were articulating the outcomes of their group work practices. There were three primary research questions in this study: 1) Are school social workers articulating their group work interventions in their practice evaluations? 2) Are school social workers evaluating group outcomes as well as the outcomes of individual members? 3) What evaluation methods are school social workers using to monitor their group work practice?

Literature Review
The tenets of research-based practice mandate that social work practitioners use empirically-based interventions whenever possible and to proceed with caution when their interventions have not been validated (Monette et al., 2002). While there are many excellent empirically validated curricula available for school social workers to use with their student clients, it appears that school social workers are modifying them freely to meet the needs of their individual clients (Garrett, 2002). Thus, school social workers, need to articulate clearly their intervention strategies (Drisko, 2000;Klein & Bloom, 1995) and should use a wide variety of research methodologies tailored to the context of each practice situation to evaluate their practice efforts (Cheetham, 1997) and communicate the results to others (Klein & Bloom, 1995).

Barriers to Practice Evaluation
Single-system designs have been praised as an excellent way to evaluate social work practice (Thyer, 1996). But "practitioners remain stubbornly resistant to the enticements of single-system designs" (Shaw & Shaw, 1997, p.78).
Instead, social workers are more likely to use such subjective indicators as colleague feedback, observations of client appearance or moods, client statements regarding their improvements in coping or quality of life, worker intuition (Gerdes, Edmonds, & McCartney, 1996;Ventimiglia, Marschke, Carmichael, & Loew, 2000), evidence of incremental changes, feelings about the way treatment is going, and an social worker self-assessments that they are maintaining an ethical balance (Shaw & Shaw, 1997) to evaluate the success of their practice. Clients, on the other hand, prefer the use of objective outcome measures to practitioner feelings to gauge the success of their interventions (Staudt, 1997).
In fact, some social workers have been hesitant to embrace practice evaluation of any kind. Some fear that practice evaluation is too narrow and unfeeling, focusing on numbers at the expense of human contact. Social workers also may be reluctant to engage in practice evaluation for fear that clinical judgment will be undervalued, as compared to empirical knowledge.
Others are intimidated by research design and statistical methodology (Bilsker & Goldner, 2001). Drisko (2000) believes social workers fail to evaluate practice because of a concern that outcome evaluation oversimplifies practice, missing the complexities and meaning of client behavior, affect, cognition, attitude, motivation, process, and environmental influence. Some avoid practice evaluation because it takes too much time, delays treatment while a baseline is gathered, or has built-in ethnic and gender biases. Those who view social work as an art may believe that it cannot be measured with scientific methods (Staudt, 1997) Perhaps one of the greatest barriers to practice evaluation is viewing it as research rather than an internal component of practice. But evaluation "is not research-rather. . . it is a part of practice, akin to . . . assessment and intervention" (Staudt, 1997, p. 105). As such, many of the above concerns fall away. As a necessary component of practice, evaluation takes no more time than any other aspect of practice, and a baseline can be determined as part of the assessment process. Failing that, intervention can begin without a baseline, monitoring outcomes to see if they are improving over the measurements taken in the early stages of treatment. And the awareness of the inherent biases in evaluation sensitizes the worker to individualize intervention according to the needs of each client; the challenge is to find the best evaluation method for any particular client situation. Monitoring specific changes makes the worker more aware of ongoing changes (or their lack). It can actually help address the complexities of practice by identifying and monitoring them. And while social work may be considered an art, so can the creative thinking needed to develop sound evaluation (Staudt, 1997).

Multiple Evaluation Methods
While single-system designs are often considered desirable (Thyer, 1996), the many activities and complexities of social work practice mandate an eclectic approach (Cheetham, 1997). "Practitioners and agencies need to be knowledgeable about a variety of methodologies in which to [measure outcomes], and relying on single-system research as the primary method of evaluating practice outcomes limits us in this endeavor" (Knox, 1996, p. 102). In addition to single-system designs, social workers can use qualitative and quantitative field research, individual and comparative case studies, historical analyses, surveys, program evaluation, natural experiments, available data, experimental and quasi-experimental designs, and description of individual practice experiences (Austin, 1992). Workers can also use client surveys, especially if they request detail beyond simply asking clients if they were satisfied or not. And outcomes can be assessed with independent variables of client priorities, processes used, and agency or contextual issues (Cheetham, 1997).

Methodology
In an effort to identify the kinds of practice evaluation school social workers are doing to evaluate their practice with groups, the group work sections of year-end reports from one school social work program were analyzed to determine what form these evaluations were taking. This program was funded through a federal grant, the Elementary School Counseling Demonstration Project, through Safe and Drug Free Schools. It is likely that these social workers were more aware than the general population of school social workers as to the need for detailed documentation of outcomes and of the political necessity to show positive outcomes. It cannot, therefore, be assumed that this sample is representative. Instead it is more likely to represent a good example of outcome reporting

Sampling
Over the course of the three years of the grant, 18 elementary schools were served by a total of 16 different social workers. After being informed about the research procedures, voluntary participation, and confidentiality, 15 of the social workers agreed to participate in the study by allowing their reports to be analyzed. The social workers served one or two buildings, depending on the need of the school; three workers served buildings full time. Social workers submitted individual reports for each school at the end of the academic year. One report was randomly chosen from each social worker for analysis, so 15 reports were analyzed.
The instructions for the group work section of the report asked the social workers to list each group, the goals of the group, the number of meetings, the number of members, measurements, and outcomes. The reports were analyzed to determine the nature of the group-work practice being conducted. The purposes and goals of the groups were recorded, as were the interventions listed. An attempt was made to determine if the social workers were individualizing group goals according to member needs or if standard goals were set for members of the entire group.
The reports were then coded following an outline adapted from Gerdes et al.'s (1996)  were broadly conceived to include statements by group members, parents, teachers, and school staff. In addition to the predetermined codes from Gerdes et al.'s (1996) study, open coding was conducted to see if there were any ways the workers were evaluating their groups that did not fit into any of these predetermined categories.

Findings
Of the 15 randomly chosen reports, 5 were from the first year of the grant, 2 were from year two, and 8 were from the third year of the grant. Three of the social workers were reporting on group work in schools being served full time, and the remainder were serving the school on a part-time basis. The range in numbers of groups in the full-time buildings was from 12 groups to 39 groups conducted over the course of the year. In the part-time schools, social workers reported conducted between 8 and 22 groups during the school year. These groups ranged in size from 2 to 12 students. Six workers did not report group size. There were a total of 212 groups specifically reported, but two of the workers did not report individual groups, so the actual total number of groups was higher.

Purposes and Goals
Group purpose was clearly stated in all reports. The group purposes listed were to help students build friendships/social skills, cope with family change/divorce, adjust to a new school, manage anger, make positive choices, reduce bullying, cope with an attention deficit disorder, develop "awesome attitudes," increase self-esteem, adapt to living in the United States, make positive choices, cope with parent illness, accept the death of a family member, increase empowerment, and become effective leaders. Friendship/social skills and family change/divorce groups were the most common group purpose, and every worker reported conducting these two types of groups.
Goals are somewhat more specific than group purposes. Identification of goals makes it possible to assess the effectiveness of the groups. Of the 15 reports analyzed, 10 of the workers identified member goals in some way. Often these goals were identified as outcomes. In other cases, these goals had to be inferred from the lists of tasks the groups accomplished. For example, a worker listed descriptions of session topics such as "learn about behaviors that hurt and help their friendships" and "gain a better understanding of divorce." Others clearly identified group goals. Goals listed were sorted into 5 categories: social skills, coping skills, leadership skills, behavior change, and knowledge/understanding. Leadership skills included developing positive leadership skills and doing a group service project. None of these goals were stated in quantifiable terms. In other words, specific target goals were not stated.

Interventions
For most of the workers (13 of 15) it was possible to identify interventions used in the groups, although most did not provide specific activities or descriptions of individual group sessions. One of the most specific descriptions of interventions was this description of friendship groups: In each half-hour session, students are offered time to share feelings or current life events. Students are then presented with friendship skills, such as teamwork, compromise or honesty. The session is concluded with each participant identifying one thing they learned or did well in that group session.
A briefer summary of intervention was more typical: "Drawings, games, role-playing, books and discussions were used to achieve desired goals." There was no mention of a theoretical framework in any of the reports. All of the workers reported using activities. They described using games, books, stories, discussion, worksheets, role-playing, drawings or other art, homework, music, puppets, skits, portfolios of group products (i.e., art and writings), and guest speakers. Generally the session topic was chosen by the worker, but a there were a few groups in which members chose topics to be discussed. Some of the social workers mentioned specific curriculum they use for their groups: Complete Group Counseling for Children of Divorce, Teaching Friendship Skills, and Creative Coaching Curriculum for Children with ADHD.
There were two general styles for determining intervention methods.
Some social workers appeared to have a general pattern for groups for various age groups. There was often a set of topics to be covered for groups. For example a social worker who conducted several family change groups for first grade students used a very similar set of topics for all of the groups. These topics were slightly different for third or fourth grade family change groups. The second pattern that was noted was to individualize the set of group topics to member needs. This was more common in friendship groups in which specific social skills were addressed to meet the needs of the members of a particular group.

Group and Individual Outcomes
While goals were generally stated for members of groups in general, the measurement of those goals was rarely articulated. When goals and outcomes were articulated, they tended to address individual issues such as individual behaviors: The purpose of this group was to allow students to give, receive, and practice ideas to help them cope with various friendship issues. The group is for students who may be experiencing difficulty forming new friendships or maintaining current friendships. . . Students improved on their conversation skills within the group they were able to ask for things appropriately and solve problems that arose in group.
It was the exception to have different goals for each member. Instead all members were working on a set of skills that all students needed to improve.
Only two workers reported a group in which members were working on individual goals that were not common for all members. Another indicated that she asked members to write down what they wanted to learn in group and incorporated their interests into the group sessions.
Three social workers conducted leadership groups that had goals of group outputs rather than individual change. These leadership groups were very different from most of the groups conducted. They included some goals for the individual members ("take pride in their efforts and learn positive leadership skills, as well as promoting social skills.") and included both students who were already leaders in their classes and those who had "leadership potential." There was also a group task goal, to "choose a project to work on that benefited the school, the city, or the world." These groups gathered school supplies for a needy school in Panama; raised money for a local humane society; organized a school fun night, donating proceeds to a local shelter; and ran a booth at the year-end family picnic to buy equipment for the school.

Gerdes et al. (1996) separated client statements and evaluations from
reports by others. These social workers interchanged statements on group success by members, parents/guardians, and teachers. For this reason, reports from all sources were analyzed together. The reports indicated that parents/guardians, teachers, and members all participated in evaluation methods of the groups.

Empirical Indicators
The reports were searched for evidence of empirical practice evaluation (Gerdes et al, 1996). Although virtually all of the workers were monitoring client change, few other empirical-practice indicators were identified. While target goals were generally known, there was little evidence that they were operationalized or systematically measured. Single-system designs were rare.
One worker reported that classroom bullying reports dropped following a violence-prevention group. Four other social workers used pre-and posttests to measure changes as a result of their groups. Of these four, two workers used the measures consistently, reporting changes in attitude or behavior for members in all or most of their groups. The other two used pre-and posttests for one or two groups. These designs used surveys in which members, teachers, and/or parent/guardians rated the effectiveness of the group in meeting goals. One of these social workers purposely chose not to do a pre-and posttest evaluation in one instance, a grief group, reporting that the use of measures of grief seemed insensitive. Two more workers used posttest-only measures of group success, asking for information from teachers, parents/guardians, or the members about their perception of change resulting from the group as the group ended. There was no evidence of the use of standardized scales or behavioral indicators such as frequency counts or duration of behaviors.

Pragmatic Indicators
The reports indicated the social workers were using a large number of Statements about problems. All but two of the workers reported statements of improvement in problems or symptoms as evidence of the success of their groups. In doing so, they indicated that members, teachers, and parents/guardians had a firm grasp of the goals and purposes of the groups and were aware when the problems for which the members were referred improved (or not). One said " [Members] indicated satisfaction about their increased ability to develop friendships, to enjoy their friendships, and to constructively address problems and differences." A report quoted a member who wrote, "I learned to act friendly." Teacher feedback was also identified: "improvement was noted by the classroom teacher on how these 4 students related to others and how their peers related to them." And "parents [of students in a family change group] commented that their children seemed to be more accepting of their new family situations and more comfortable knowing that they (the children) were not 'the only one with split families.'" Group process observation. The second most commonly reported way to note success emerged from the data rather than from Gerdes et al.'s (1996) framework. It was observations by the social worker of successful group processes attained by the group. All but three of the workers reported such things as group cooperation, universality, reluctance to end the group, contributing to a group task goal, statements about the group as a safe place, and mutual aid in reporting on their group work. One worker wrote, "this group appears to value their time together and often have exchanges with each other that are supportive and encouraging to the other group members." Another wrote, "parents commented that they. . . feel less isolated knowing that others have similar issues." Another noted that "[members] did not want the group time to end as they felt connected with me [the worker]." And one wrote, " the boys often shared their coping strategies with one another." Quality of life indicators. Nine workers reported improvements in areas that might be called "quality of life." These statement included vague statements by members, parents/guardians, or teachers that things were getting better for a member or that the group had been helpful. For example, one wrote that "parents and teachers. . . indicated that children appeared to benefit from their participation." Another teacher wrote to the social worker that "my student always came back [from group] ready to work." Also included in this category were comments by teachers and parents/guardians that students had made improvements in areas other than the group goals. Another wrote in a description of a divorce group that she had received feedback from a teacher that a student "seemed more relaxed and had better peer interaction since participating in the Enjoyment of the group. Nine workers referred to students enjoying the group in commenting on the outcomes of their group work. This was another theme that emerged from the reports and had no parallel in Gerdes et al.'s (1996) framework. One reported "teachers report that the students are anxious to come to group and enjoy the opportunity to visit with one another." One student wrote "I love group time. I don't ever want it to stop." And "parents of the girls commented that they look forward to group." It appears that enjoyment is a fair marker of success in children's groups for these social workers.

Improvement in coping.
Six of the workers reported group success in terms of members' learning ways to cope with problems. Coping included using new techniques to deal with problems, particularly anger management and relaxation techniques. A worker quoted a student statement in a feedback questionnaire that "I liked practicing relaxing." Coping also included statements by students that they viewed group as a place where they could solve problems.
For example, a student wrote, "In group I can talk about sad stuff without having to cry." A worker reported that "students [in a new student group] liked talking about making new friends and said that they were able to use some of the ideas discussed in group to form new friendships."

Discussion
There were three primary research questions in this study: 1) Are school social workers articulating their group work interventions in their practice evaluations? 2) Are school social workers evaluating group outcomes as well as the outcomes of individual members? 3) What evaluation methods are school social workers using to monitor their group work practice?

Articulation of Intervention Methodology
Because communication of practice methodologies is an important component of evaluating practice (Klein & Bloom, 1995), social workers need to report what methodologies they use in their practice. While the reports were brief, these social workers indicated that they were using a combination of activities and discussion in their groups. These identified methods were linked to the needs of students and purposes of the groups.

Evaluation of group and individual outcomes
It is clear from the reports that the social workers are setting goals that can be measured for their student members and that the goals are understood by members, parent/guardians, and teachers. It is also clear that social workers are consulting with the members, parents/guardians, and teachers to see if these goals are being met. While the goals are usually clear, they rarely stated them in a quantified way. For example, student members know that the social skills groups are to help them learn the skills to be better friends. They (and parents and teachers) are able to report that members are making improvement (and when they do not seem to be improving). So while goals are not quantified, they are clearly understood in ways that appear to demonstrate progress or the lack thereof.
If the group is to be considered an additional client (Shulman, 1992), then the group should also be evaluated for success. Group processes (Rose, 1984) were apparently never measured in these groups, but the workers were clearly aware of process, and often reported positive group processes as evidence of successful group outcomes. In fact, group processes such as universality, mutual aid, and cooperation were reported by 12 of the 15 workers, indicating that group processes may not be identified overtly as group goals, but these processes are being well used as measures of success. In the more taskoriented leadership groups group products also served as goals, and the workers reported what the students accomplished a service project to "benefit the school, the city, or the world."

Evaluation Methods
Similar to Gerdes et al.'s (1996) findings, social workers in this study are using pragmatic more than empirical methods to evaluate their practice. There was very little evidence that these social workers are using empirical indicators to evaluate their group work practice. While all of the workers are monitoring their practice, only four are using single-subject design procedures. None list specific outcome data, but several articulated improvement in goals based on the comparison of client perceptions related to goals. This finding is in clear contrast to clients' stated preferences for objective outcome measures (Staudt, 1997).
These social workers are using pragmatic indicators regularly. The workers frequently used statements about symptoms and problems to assess their practice. They also commonly used statements about quality of life and coping. Two additional categories emerged from these reports that were not a part of Gerdes' et al.'s (1996)  Several of Gerdes et al.'s (1996) categories did not appear with any frequency in these social workers' reports. There was no mention of using colleague feedback as a measure of success. Increase in communication was mentioned twice, but because it was a goal in the social skills groups, it was coded separately as a statement about problems. Increased insight and decreased resistance did not appear to be commonly used, perhaps because the members were children rather than adults. Collateral contacts with parents/guardians and teachers were analyzed along with member statements rather than separately because they were used so extensively and interchangeably with member comments as evidence of success. This may also be a function of the fact that all the group members were children.
From this research, it seems that as long as the goals and purposes are clear to members, parents, guardians, and teachers, these social workers do not feel that it is essential to use a strict single-system design to evaluate practice outcomes. The pragmatic indicators appear to show that the workers perceive the groups to be working for the members in making improvements in the problems for which they were referred. The qualitative nature of the reports also added depth and understanding of what theses social workers were trying to accomplish. The anecdotes used in the reports demonstrated a human side to the group goals and were often charming. Direct quotes from the young members ("I learned to act friendly.") state success in a way an empirical report can not. Such an informal system of evaluation reinforces Staudt's (1997) contention that practice evaluation is an important part of practice. These reports also appear to capture some of the richness of practice, thus addressing Drisko"s (2000) concern that social workers may avoid practice evaluation out of fear that outcome evaluation oversimplifies practice and misses the complexities and meaning of client behavior and attitudes. These social workers seem to be meeting Cheetham's (1997) recommendation that social workers use an eclectic approach to practice evaluation. It is unfortunate, however, that at least some of the measurements of success in these groups were not stated in measurable terms. It would be helpful to triangulate outcomes, supplementing these pragmatic indicators with empirical measures to corroborate the feelings of the workers that students are, in fact, making good progress towards goals in these groups.
There are clear limits to this research, the main one was that these social workers may not be representative of all group workers. They were grant sponsored and knew their reports might be used to obtain continued funding.
Another limit is that these workers were serving only elementary-school aged members. Generalization of these results should be done with caution.
Nevertheless, the reports provide a rich description of the group work practice going on in one district's elementary schools. It would be interesting to replicate the study looking at other practice reports on group work generated by social workers in other settings. One interesting question to come from this research is whether social workers use different pragmatic indicators of practice success with group work interventions than they do for individual practice. Another question is whether group work practice evaluation is different in children's groups than in adult groups. It is possible that the emphasis on member enjoyment and the development of group process indicators might be unique to evaluation of group work practice with children.

Conclusion
These reports analyzed for this research indicated that the social workers who wrote them were clearly identifying goals, purposes, and methodologies in their elementary school-based groups that were clear to members, teachers, and parents/guardians. Unfortunately, these goals were rarely quantified. The workers used some pre-and posttest designs, but were much more likely to rely examples. It appears that the identification of purposes and goals that are clearly understood by members and collateral sources is extremely important in assessing practice outcomes, even when those goals are not identified in measurable terms. The use of flexible, informal pragmatic methods of evaluating group work practice appears to be one way to keep track of group processes for groups of elementary school students.
Yet social workers must find ways to describe the results of their interventions in clear, objective terms. The social workers whose groups were described in these reports began the process of evaluating their practice by developing goals and purposes and communicating them to student members, parents, and teachers. Yet they did not follow through as thoroughly as they might have. They used few of the empirical indicators of practice success. Their outcome evaluations would have been far more effective had they also monitored frequency counts and duration measures of behavioral goals and/or to used scales or other measures, such as standardized instruments, worker-developed instruments, or self-anchored scales completed by either student members or collateral sources (parents/guardians, teachers, and school staff) (Gerdes et al., 1996). Such empirical indicators would have been far more effective in monitoring the effectiveness of these group interventions.
Social workers have an ethical mandate to monitor and evaluate their practice outcomes. While pragmatic indicators are charming and give some indication of success, they are not sufficient. Social workers would be wise to remember that clients and those who provide funding also appreciate the use of objective outcome measures.