The evaluation utilized both quantitative and qualitative data collection methods to holistically assess Auraria’s PB process. Data sources included survey questionnaires, field observations, and participant interviews. The evaluation team also incorporated data from campus reports, City of Denver reports, news articles, and other sources to contextualize the process and its results.
The TMHAS Evaluation Committee was comprised of seven individuals, including a graduate student from the University of Colorado Denver, staff members from the other Denver PB process, a graduate student from Portland State University, a graduate student from the University of Colorado Boulder, and an undergraduate student from the University of Colorado Boulder. The committee included researchers, activists, and practitioners, and they worked collaboratively to create data collection instruments (e.g., survey questionnaires, interview protocols, and observation protocols) in accordance with national standards of PB research established by Public Agenda.
The research team collected data about Auraria’s budget, the campus population, and PB to understand how Auraria’s process might impact student knowledge, skills, attitudes, and practices related to public engagement on campus. Data sources included the American Community Survey, Community College of Denver Spring 2017 Census, University of Colorado Denver Fall 2017 Census, Metropolitan State University Fact Sheet 2018–2019, Auraria Higher Education Center Financial and Compliance Audit, and University of Colorado Denver Student Fee Plan, to name a few.
Quantitative Data Collection
PB participants completed surveys across three phases of Auraria’s process: idea collection, project development, and voting. Survey respondents included steering committee members, budget delegates, project idea submitters, and PB voters. All surveys were administered both online and on paper to generate as high of a completion rate as possible.
The survey questionnaires were designed to collect demographic information for PB participants across all phases. Steering committee and budget questionnaires were designed to assess levels of prior civic and campus engagement. Idea submitter and voter questionnaires included questions about attitudes related to PB and campus administrators, prior civic engagement, and how they learned about the PB event where they participated.
Qualitative Data Collection
Qualitative data was collected from in-depth interviews and participant observations. To ensure multiple viewpoints were collected, a variety of stakeholders were interviewed for the purpose of the evaluation, including steering committee members, idea collection participants, budget delegates, PB organizing staff, and campus staff members. All interviews were conducted individually face-to-face, over the phone, or through email. Each interview lasted approximately 30–60 minutes.
Interview questions were tailored for each stakeholder group and were developed in collaboration with members of the TMHAS Evaluation Committee. In general, interviews were designed to understand how people participated in PB, how PB affected relationships between students and campus staff, and how PB contributed to attitudes and practices of community engagement outside of the process.
Fieldnotes were recorded across all the stages of Auraria’s PB process. Observation protocol for fieldnotes was designed to document and assess the quality of public deliberation at events. Researchers were attentive to points of negotiation, conflict, forms of inclusion, and how authority operated in meetings and public outreach events.
These participant observations allowed for the generation of direct, experiential knowledge that generated of how Auraria PB unfolded through real-time interactions with participants. Fieldnotes included quotes from participants as well as thoughts, ideas, and experiences of researchers as they occurred at PB events. Field observations provided a more nuanced, detailed view of the PB process that could not captured solely through surveys.