San Francisco needed a way to ensure they are designing online services that are easy to use and accessible to all. We began in 2017 with a demonstration program to test the viability of a citywide user testing program that invited diverse sample of community members to provide real-time feedback on City services. Since that time, we helped the Digital Services team develop the infrastructure for ongoing public research, including participant outreach and recruitment, participant pool management, and research scheduling and coordination. Through annual evaluations, we help them iterate regularly based on community and staff feedback.

We have learned that while community-based organizations (CBOs) can play a key role in helping to bring diverse voices into research, these partnerships must be mutually beneficial. To that end, we developed a paid partnership model with CBOs to ensure that the City’s digital services are meeting the needs of the most vulnerable groups, such as English language learners, older adults, underrepresented neighborhoods, persons living with disabilities, persons with limited digital literacy or access, as well as low-income and historically marginalized populations.

Since 2017 we have helped administer, evaluate and evolve the user research program. In addition to overseeing partnerships and maintaining relationships with CBOs, we have developed a brand and messaging guide to coordinate consistent communications about the program and designed multilingual promotional materials for citywide distribution.

Despite the COVID-19 pandemic, during 2020 we were able to grow the community research community by over 700 people, many of which are considered to be in marginalized or sensitive groups. That year also saw the City conduct their first user tests in Spanish and Chinese.

Building successful digital services requires rethinking entire real-life processes and including end-users in their design and development. The user research program ensures that those most in need of City services have a voice in this process.