Skip to main content

Case study: San Francisco 'SF Shines' Program Evaluation and Implementation

Building opportunities for continuous improvement of a small business grant program

The San Francisco Office of Economic and Workforce Development administers the SF Shines program, which provides grants, design assistance and project management for small businesses to improve their storefront façades and interiors.

After 10 years, City staff were eager to evaluate the program’s strengths and understand additional opportunities to continue supporting local businesses.

Community Engagement

Tags

Project Team

Lawrence Grodeska
Cristelle Blackford
Leah Tremblay-Adams

Listen

We engaged stakeholders from across the SF Shines system through a series of one-on-one interviews, surveys and ideation sessions. We began by developing a draft service blueprint with San Francisco Office of Economic and Workforce Development (SFOEWD) staff. To ensure we had a well-rounded view of the program and user experience, we connected with local business owners (grant recipients) as well as the architects, graphics designers, contractors, and program staff responsible for implementing the grants.

To ensure program improvements were defined and led by those closely connected to SF Shines, we conducted each stage of the process in collaboration with our partners, from mapping the current state and building personas to identifying and prioritizing challenges.

Learn

In our conversations with grant recipients, we learned that the care and support from the SF Shines program team helped reduce friction between businesses and the City, especially in navigating legal requirements and compliance. This was the case across a wide range of businesses, whether they were business savvy entrepreneurs; advocates of their business corridor; or multi-generational business owners.

After defining a more nuanced picture of ‘small business owners,’ we convened people from across the SF Shines community to discuss needs and prioritize opportunities. We learned that top priorities included: (1) making the program easier to navigate for business owners with limited english proficiency (who were facing a higher risk of closure/displacement); and (2) supporting businesses to stretch their grant funding more creatively.

Make

Over the course of the evaluation, we developed several artifacts to help visualize the program and its impacts. This included a system map, small business personas, user journey maps, and a revised theory of change. These visualizations, and ongoing conversations with stakeholders, helped us compile a list of opportunities which were validated and prioritized.

After completing the evaluation project, we were invited to further develop prototypes and test the improvements that we helped define with the SF Shines community.

We designed four prototypes – a program fact sheet; program participant guidebook; exit interview; and follow-up survey – and tested them with former and new SF Shines businesses, at their places of work. The team is currently evaluating how they can integrate these prototypes into the program.

“The CivicMakers team went above my expectations to conduct a meaningful program evaluation and to help set up new systems to ensure we can continue improving the program. The team took care to truly understand our program, the program’s clients and their needs, ensuring the inclusion of diverse stakeholders to brainstorm program changes and new directions. They are a thoughtful, passionate, fun, dedicated, and capable team.”

Darcy Bender SF Shines Project Manager, San Francisco Office of Economic & Workforce Development

View Case Study
Cristelle is presenting insights from SF Shines research to collaborating stakeholders at a co-design session.

Phase 1

Evaluation

The SF Shines Program is an economic development service of the Invest In Neighborhoods initiative that provides focused, customized assistance to help small businesses thrive, increase quality of life, improve physical conditions, and build community capacity.

SF Shines worked with CivicMakers to conduct a program evaluation, its first formal program evaluation since it started 10 years prior. The goal of the evaluation was to (1) gain a holistic view of the entire program, (2) assess program effectiveness, efficiency and equity, and (3) make recommendations for solutions and ongoing evaluation.

We engaged stakeholders throughout the program, including staff, program participants, partner nonprofits, small business consultants, contractors and designers. The process included a preliminary system mapping stage where we outlined the program and its theory of change. These were then tested against user experiences of the program, which helped us to develop personas and more detailed user journey maps.

While conducting user interviews and surveys, we compiled a list of pain points that occurred across the system. We presented these opportunity areas to stakeholders from across the program and invited them to co-design solutions and prototypes. This culminated in a list of recommendations for program improvements, informed by end users, many of which are currently under development today.

Cristelle is conducting an SF Shines interview with a small business owner at their place of work.

Phase 2

Implementation

After completing the evaluation project, we were invited to continue working with SF Shines to develop and test improvements that we helped identify. This included the development of participant resources, externally-facing collateral and messaging, as well as evaluation tools targeted at measuring the impact of the program and participant satisfaction.

The Implementation Phase included three rounds of prototyping and testing to incorporate feedback from Invest in Neighborhoods staff, past program participants (closing the loop with participants from Phase 1) and current program participants at various stages in the process.

Outcomes

  1. Designed externally-facing messaging and resources
  2. Conducted usability testing with past and current small businesses
  3. Developed and tested prototypes of process improvements prioritized during the Phase 1 ideation session and multi-stakeholder survey
  4. Implemented a set of metrics and evaluation tools to support ongoing evaluation of the program and participant satisfaction

Related blog posts, events, resources, and case studies