By: Joanne Minh Triet Lieu
Developing tools to collect insights for program improvement is not an easy task. Learning how to conduct surveys and run focus groups that inform your work is a skill that is shaped and re-shaped by practice in the field. Here are five key insights I learned as a Student Affairs professional conducting assessment to inform professional development service delivery during a pandemic.
I began as the Coordinator for Professional Development Initiatives at the Canadian Association for Colleges and Universities Student Services (CACUSS) at the beginning of March 2020. One of the projects that fell under my portfolio was revisiting the 2016 Member Professional Development Survey conducted by Jennie Massey and Kyle D. Massey. As four years had gone by since the last assessment, CACUSS looked forward to revisiting the needs assessment to ensure that the association’s current and future programming reflects the needs of its members. Eager to shed light on the voices of Student Affairs professionals across Canada, we anticipated launching the survey in mid-May with focus groups running into mid-June.
Then the pandemic hit.
With many of our members finding their world of work and life shifting rapidly, CACUSS decided to momentarily pause its delivery of the member survey. CACUSS learned and grew as the entire world adjusted. The anticipated annual CACUSS Conference: “Learn, Unite, Act” led by colleges was cancelled, alongside upcoming in-person fall professional development opportunities. In place of the conference was the growth of CACUSS ON-Line. CACUSS extended its membership renewal deadline and whereas possibly make individual accommodations.
We recognized that acknowledging and documenting this change was more important than ever if we were to continue supporting the future of Student Affairs in Canada. From early June to the end of July, a national survey was released and nine focus groups were conducted with members on professional development. The focus groups supported the survey and supplemented the results with rich, descriptive dialogue that was aimed to:
- Identify improvements on how we currently deliver professional development programming to CACUSS members
- Determine the membership’s understanding of the CACUSS competencies in programming
- Identify new programming for members
Here are five key insights that we learned along the way:
- Adapt survey instruments to the changing times
While we had a strong foundation from the 2016 Member Professional Development Survey, we needed to adjust questions to capture the nuances of professional development in the here and now. The changes included additional questions such as members’ experiences using our new platform, community.cacuss.ca and questions related to the impact of COVID-19. Another large update to the survey was the new professional development competencies that CACUSS did not have in 2016. We were interested in knowing: Which professional development competencies did members recognize as a priority? Were there regional differences in what members identified as a priority?
2. Revisions, revisions, and revisions
Jennifer Hamilton, Executive Director of CACUSS, and I exchanged numerous calls and emails as we revisited and re-envisioned the survey and subsequently developed the focus group questions. The goal was to ensure the questions measured what we intended and that the survey presented all possible response options. The questions were grouped into three themes,
I) CACUSS’s professional development opportunities
II) Professional development outside of CACUSS
III) Existing and future gaps.
3. Survey and focus group instruments are growing and living documents
As one of my professors shared, there is no such thing as a perfect survey. Throughout the process, I learned how accurate that was; surveys grow with feedback. One memorable moment of learning was when Jeff Burrow, Manager of Assessment and Analysis at the University of Toronto, reached out to let me know that respondents were required to answer all questions in order to proceed through the survey. He suggested that making the questions optional would elicit more responses, although some would be incomplete. After sifting through the copious amount of data, this early advice was on point! A similar truth could be said of focus group questions. The purposeful placement of prompts resulted in participants sharing more in-depth experiences.
4. Creating community in a virtual focus group
Virtual focus groups offer a new set of opportunities and challenges that in-person focus groups do not.
First, facial cues and gestures made by focus group members are not as easily perceived in virtual calls, regardless of whether the camera is on or off. In a more massive virtual call, it becomes harder to know when someone will speak and yes, the awkward unmute at the same time will happen. Two strategies to work around this is to set ground rules early on as a friendly reminder to allow a person to finish their thought before contributing or use existing features such as the raise hand function in Zoom.
Second, virtual engagement calls on participants to provide more space and energy than in-person focus groups demand. Participants are calling in at their homes where they may be a caregiver, parent, or attending to other needs that require their attention. It is essential to remind yourself that participants are inviting you to their personal space, and it is important to understand and acknowledge that interruptions may happen. Your role is to facilitate a space that is warm and welcoming to members who may be meeting you for the very first time after exchanging several emails.
In the focus groups, I make the main question available in the chatbox after I have read it aloud, so that if a person needs to switch for a moment or recall the question asked, it is available in written format. Using prompts also helped members to provide and me to gather detailed and descriptive responses. I refer to Merriam and Tisdell’s guide (2015) in framing questions such as:
Tell me about a time when…
Give me an example of…
Tell me more about that…
What was it like for you when…
5. Connecting nationally offers great perspective and appreciation
Lastly, virtual focus groups offer an exceptional way of connecting with participants across geography. We had members calling in from coast-to-coast, morning and afternoon. To connect with so many people across geography and time zones is genuinely remarkable. Just be sure to include your time zone in emails!
Throughout this process, I enjoyed listening and learning from CACUSS members. It is a reward and privilege to be in a virtual room of thoughtful student affairs leaders. Transcribing and reviewing the transcript for themes on professional development served to reinforce the importance of connecting, especially during this time. Thank you to all the CACUSS members who were part of the focus group and survey process! While conducting surveys and focus groups using telecommunications is not a novel concept, it does bring new challenges during a pandemic. At the same time, now more than ever we need to employ thoughtful assessment instruments to inform our future work.
Most profound appreciation to Jennifer Hamilton and Jeff Burrow for your insights and feedback throughout this process, and to everyone for sharing your incredible experiences, learning, and vision for CACUSS’s future professional development programming.
Joanne Minh Triet Lieu is the Coordinator of Professional Development Initiatives (Coordonnatrice des initiatives de perfectionnement professionnel) for the Canadian Association of College and University Student Services (l’Association des services aux étudiants des universités et collèges du Canada).
Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation (4th ed.). San Francisco, California: John Wiley and Sons.