Assessment and Evaluation Toolkit
Toolkit prepared by Kim Cochran Kiesewetter, MS, Data Analyst and Consultant, WiLS, 2022
Table of Contents
Toolkit Background and Summary
In March 2022, the eleven participating City Library Collective (CLC) libraries were awarded an ARPA Grant from the Wisconsin’s Department of Public Instruction (DPI) to commence their project, “Resiliency Preparedness: City Library Collective (CLC) Supporting Communities in Crisis through Concepts of Whole Person Librarianship.” Part of the project design involved the development of a shared assessment and evaluation strategy for the collective. While the timeline for the ARPA grant is limited to three months, the nature of the project involves a staff training component in hopes of facilitating long-term, environmental change. This toolkit is designed to be a repository of the assessment and evaluation strategies and tools designed as a part of the project.
The development of this toolkit is in support of multiple project outcomes. These include the opportunity:
- To outline the resources of the immediate-term assessment and evaluation plan for CLC libraries.
- To outline suggestions for assessing and evaluating longer-term impacts that will likely occur beyond the scope of this project’s three-month timeline.
- To provide a resource for the broader library community on resources and strategies for assessing the impact of similar initiatives and projects at different libraries beyond the CLC.
To begin the process of developing an assessment and evaluation plan, a project logic model was developed in conjunction with CLC members.
Based on the project logic model, a Data Collection Plan was developed. This process ensured that all data collected in support of assessing and evaluating the project was clearly tied to project outcomes identified as part of the logic model.
|
Intended Outcome/Impact |
Operationalized Metric |
Data Collection Method |
|
| Short-Term (Learning) | Library staff gain knowledge of holistic, person-centered concepts. | Staff display an increase in knowledge after the WPL training about holistic, person-centered concepts including in the training opportunity. | Staff Pre/Post-Test based around WPL Librarianship concepts. |
| The development of a network of contact points and resources for WPL. | Development of a document of contact points and resources for WPL to be shared out. | Artifact – Was item created or not? | |
| The development of shared impact measurements for WPL services. | Development of shared impact measurements. | Artifact – Was item created or not? | |
| Medium-Term (Action/Behavior Change) | Library staff implement WPL as a change in method of service delivery (e.g., reduction in police calls). | Reduction in calls to police after the WPL training. | Library incident reports. |
| The implementation of shared impact measurements for WPL services. | Library staff implement the shared impact measures for WPL services. | Self-reported from libraries at end of data collection period – did they implement any/all of the shared impact measurements? | |
| Long-Term (Environmental Conditions Change) | Library staff gain confidence in their ability to assist patrons through a WPL lens. | In addition to a gain in knowledge and the implementation of shared impact measures, staff will indicate an increase in confidence in their ability to assist patrons through a WPL lens. | Staff Pre/Post-Test. (Can be same one as WPL Librarianship concepts if possible.) |
| Patrons feel the library is a safe connection and that their lives are impacted positively by library services and interactions. | Over time, patrons will indicate a self-reported increased sense that the library is a safe place where they have positive interactions as measured by an annual community survey. | Annual Community Survey. (Will collect baseline for this project period.) | |
| Increased community awareness of how libraries serve their communities. | Over time, community members and leaders will indicate an increased awareness of how libraries serve their community as measured by an annual community survey. | Annual Community Survey. (Will collect baseline for this project period.) | |
As mentioned above, due to the short timeline of the grant-funded project, it is unlikely that there will be significant movements towards achieving the medium-term and long-term outcomes. The goal of documenting a clear evaluative process, though, is to provide CLC libraries (and any other libraries embarking on similar initiatives) a resource for assessing the longer-term impacts of projects based around concepts of Whole Person Librarianship (WPL). At the root of this effort is the desire to begin the process of helping libraries develop the needed infrastructure to assist patrons who enter the library looking for implicit or explicit support in meeting their physiological and/or psychosocial needs. For purposes of this toolkit, “physiological needs” refers to basic human needs to survive and live with dignity in society, such as access to sufficient food, water, shelter, and personal hygiene resources, while “psychosocial needs” refers to the intersection between a patron’s psychological needs which include “the internal, emotional and thought processes, feelings and reactions” and their social needs which include “relationships, family and community network, social values and cultural practices.”[1]
This project was funded with federal funds distributed by the Institute for Museums and Library Services (IMLS). As such, all developed items are open and free for use, but it is requested that all items only be used with appropriate attribution following the IMLS Acknowledgement Requirements, usually with a credit line and/or logo, depending on the usage context.
Data Collection Resources
Staff Pre-/Post-Test
The first data collection method developed for this project was a library staff pre-test designed to be completed prior to engagement with the grant-funded WPL training, conducted by Sara Zettervall. Due to the nature of the project, the staff pre-test focused on gathering baseline data on staff’s: 1) perception of the frequency and severity of patrons’ needs for physiological and/or psychosocial support, 2) confidence in their personal ability to assist patrons with these needs, 3) belief that providing such assistance is a part of their job duties, and 4) perception of receiving adequate training and resources from their library to provide this type of support to patrons. In addition, general questions (to preserve staff anonymity) were included to gauge if time in a library career and/or participation in previous social work-oriented library training(s) had influence on the above items.
While this data collection tool was designed to focus on the specific training opportunity with Sara Zettervall, the questions are generally adaptable to any context in which there is professional development or continuing education around the intersection of patron’s physiological and/or psychosocial needs and library work.
Staff Pre-Test:
|
Data Collection Strategy
The Staff Pre-Test should be completed prior to beginning any training opportunities as a means for collecting baseline data for the project on the items outlined above. For purposes of this specific project, that meant staff completed the pre-test one week prior to the WPL training kick-off in early April 2022. The following list provides and outline of steps for utilizing this survey instrument:
- The Staff Pre-Test can be built out in any survey platform for electronic dissemination. Google Forms is a free tool, however, it could easily be used in any available survey dissemination platform such as SurveyMonkey, Qualtrics, etc. If paper surveys are preferable, they can be disseminated anytime prior to the training event for completion by staff, however, it should be noted that even free tools like Google Forms will provide basic data visualizations that can make analysis of findings much simpler.
- The Pre-Test should be completed prior to initiating any type of social services-oriented training opportunity with staff. It is recommended to have staff complete it 1 to 2 weeks prior to the training event. If not all staff are participating in a training opportunity, consider including everyone to compare findings between staff who participate in the training and staff who do not as it can help highlight any differences and/or shifts for staff based on participation.
- Once the survey is closed, baseline data can be analyzed and reported out.
The Staff Post-Test is identical to the Staff Pre-Test except for re-wording the second question to: “Did you participate in….” The Staff Post-Test should be completed after your training opportunity has ended. Because measuring something that involves a longer-term goal of changing culture can take time to assess, ideally the post-test will be administered annually over a period of time to track changes over a multi-year period implementing new service delivery approaches. Due to the time-limited nature of this grant cycle, the Staff Post-Test was administered after the training was conducted at the end of the grant period in October 2022. Survey dissemination can be done in an identical manner as the Staff Pre-Test.
Library Incident Tracking
An outcome of this collective project was that staff would gain knowledge on holistic, person-centered concepts that would lead to a change in service delivery over time. To identify any potential shifts in this realm, the volume and nature of library incident reports were tracked for participating CLC libraries. Since many libraries already had different processes and mechanisms in place to track certain types of incidents at their library, library directors were asked to report the date of an incident and select the “type” from an exhaustive list generated by the CLC that included standardized theming for the patron behavior/need that led to the incident, as well standardized theming for the library’s response to the patron’s behavior/need.
Type of Incidents:
| Patron Behavior/Need | ► | Library Response |
| Security Incident | ► | Security Contacted |
| Medical Incident | ► | Medical Assistance Initiated |
| Disruptive Behavior | ► | Referral |
| Refusal to Leave | ► | Temporary Removal |
| Violation of Existing Restrictions | ► | Permanent Ban |
| Direct Request for Assistance | ► | Handled Internally at Library |
| Other | ► | Other |
Library staff were given the option of entering incidents into a spreadsheet or a survey format based on preference (both options gathered the same information). Incidents were then looked at over a specific timeline and by type to see if there were any trends identified during the grant activity period with increases and/or decreases overall, as well as by patron behavior/need and library response.
Data Collection Strategy
Incident tracking should begin prior to implementation of new social services-oriented training and/or other related changes at the library, such as hiring a social worker, and continue throughout the change(s) being implemented.
- Baseline data about incidents needs to be gathered to establish a starting point for the library – ideally over a 2 to 3 month period. This means tracking incidents prior to beginning any changes to see the most common type of incidents, the most common types of library responses, and the frequency of incidents.
- Once the social services-oriented change(s) are being implemented, continue tracking per usual throughout the process and its conclusion/full inclusion in daily library activities.
- Incidents can then be analyzed to see if there were any changes in 1) frequency of incidents, 2) the types of incidents occurring in the library, and/or 3) the way library staff respond to incidents.
Note: It is likely that incidents could also be related to things like the time of year – people may struggle with housing needs more/less depending on the season, weather and time of day can also potentially influence incidents, etc. Longer term incident tracking on an annual basis can create a picture for the library of if there are any trends that might be related to some of these outside factors.
Community Survey
The final data collection tool developed as a part of this project was a community survey. As a community-facing project, this data collection tool was designed to include their insights in the process based around two project goals: 1) patrons’ perception of the library being a safe connection for positive experiences, and 2) an increased awareness by the broader community about the types of services that libraries provide.
Community Survey:
|
Survey Introduction: In addition to being a place to find and check out books (e-books included!), libraries can and do provide many other additional services to their communities. Some of these services include helping community members who visit the library meet and/or find resources to meet their social, emotional, and physical needs like finding safe housing, getting help with job searches, and other social-oriented services. This brief survey asks questions related to community members’ experiences with and awareness of these types of services. Your participation in this survey is voluntary and all answers are confidential and will be used to help libraries by providing them with important information from the communities they serve. Thank you for your time and insights! Section I: Introductory Questions
Section II: Library Usage and Perception (Only for Library Users)
Section III: Library Service Awareness and Future PlanningIn addition to being a place to go for books, libraries can and do provide many additional services to their communities. They assist community members with certain social, emotional, and physical needs such as referring people to community organizations, assisting individuals with job searches or employment skills, etc. The following list is exhaustive and covers services that may be offered at your library or might be offered in the future.
|
Data Collection Strategy
The Community Survey is, ideally, conducted on an annual basis. The first time it is conducted should be early on in a social services-oriented project to gather baseline community data about how people currently view the library in addition to awareness questions about social work-oriented services the library may already provide. By conducting on an annual basis, trends over time can be gathered as projects and initiatives are implemented to see if there are any changes to either/both library perception and service awareness.
- The Community Survey can be disseminated through printed copies and/or electronic dissemination. For analysis purposes, electronic dissemination is much easier, however, pairing with printed surveys (if there is staff available to enter hard copies into the electronic survey at the end of the survey period for inclusion in analysis) can reach additional population members who may be less likely to be “online” or to take an electronic survey.
- Consider leveraging any options you have for reaching community members – listservs, email lists, social media posts, partner organizations, etc. This is a survey of convenience so the goal is to get the survey in front of as many different people as possible with the least amount of stress.
- Some survey platforms such as SurveyMonkey allow for the easy creation of QR codes that link to a survey. This can be an option for capturing more participants. Consider posting the QR code around your library and/or community. Since the Covid pandemic, more people are familiar with QR codes (a person opens their camera on their phone and aims it at the code to bring up the survey link on their device) and it can be a way to capture people who you don’t have a contact for or who might not be following your social media channels. You can simply copy/paste the survey QR code and resize as desired onto a Word document with language promoting your survey.
- The survey should be promoted for at least a 2 week period, though longer can be beneficial if there are regular reminders. The greatest volume of responses will typically occur in the first 1-2 days when the survey is “pushed” or promoted through a new channel (e.g. a social media post or an email to a listserv). Promoting the survey through various channels over a 2-3 week window will likely increase the number of responses gathered.
- Once the survey is closed, baseline data can be analyzed and reported out to stakeholders.