What Social Impact Leaders Actually Want From AI, in Their Own Words
Howard Pyle and Stephen Godfrey of Numantic Solutions led the workshop using xf.app.
A room that didn't expect to be designing AI
The most striking thing about the workshop wasn't any single product anyone built. It was a room of leaders who, ninety minutes earlier, would have described themselves as not technical enough to design AI products, ending the session by debating what their AI should be allowed to refuse and when a human should step in.
That shift, from "I don't know how this stuff works" to "this is how it should work," happened in about ninety minutes. The rest of this post is what we learned from watching it happen.
How it works
Over the past year, we have been building xf.app, a platform that helps people who aren't engineers design and deploy AI products for high-trust environments, developed with support from a Roddenberry Foundation Catalyst grant. For Good Tech Summit, we adapted it into a workshop format using xf.app's conversational design tool.
The mechanics were simple. Each participant opened xf.app and started a conversation with an AI facilitator. The first question was the hardest: what problem would your AI solve, and who does it help? Once they had an objective, the AI walked them through four phases in sequence, each one building on the last.
Where every participant started: a blank canvas and one question.
First, AI Role: what should the AI actually do, step by step? Second, Data & APIs: what information does it need, and where does that information live? Third, Guardrails: what should the AI never do, what data should it never touch, what requests should it refuse? Fourth, Human Escalation: when does a human need to step in, and what triggers that handoff?
The canvas after the first phase. Nodes populate as the conversation progresses, not after a form is submitted.
As participants worked through each phase in conversation, their answers populated a visual workflow canvas in real time, the four-column layout shown in the banner above. By the end, they had not filled out a form or done a whiteboard exercise. They had had a conversation, and that conversation had produced a structured, testable product specification.
The last step was the most surprising for most participants. They could click a button and immediately talk to a simulated version of the product they had just designed. The guardrails they had specified were already in effect. The AI role they had defined was already shaping the responses. The thing they had described in conversation had become a thing they could actually use.
The finished specification for a volunteer coordination AI. Built entirely through conversation.
We put this in front of a room of program directors, foundation leaders, school planners, data architects, and nonprofit operators, and gave them about ninety minutes to design something they cared about. The point wasn't to ship a product. The point was to see what comes out when non-technical leaders are given the structure to articulate what they want from AI.
This is what we learned.
Almost no one designed a chatbot
The most consistent pattern across the room was that participants did not reach for general-purpose AI. They reached for the operational friction that eats their weeks.
A planner at a national nonprofit that redistributes refurbished computers designed a self-service onboarding portal for partner organizations. Her approach was telling. Rather than starting from scratch, she pulled up a colleague's existing onboarding checklist, fed it into an AI tool to extract the underlying steps, and used that as the spine of her design. She was not designing in the abstract. She was encoding institutional knowledge that already existed but had never been made into a product.
A planner from a large urban school district designed an enrollment forecasting tool. A team from an environmental nonprofit designed a corporate-prospect finder for an urban tree-planting program. A leader at a data commons designed a self-service intake portal for partner data. A professional in executive search designed a candidate-research tool. Several others designed grant-related tools.
These were not hypothetical products. They were specific, named pain points, described with the precision that only comes from doing the work.
Multiple participants independently designed grant tools, and that's a signal
The clearest convergence in the room was around grants. Working entirely independently, several participants designed tools for different parts of the grant lifecycle: discovery, qualification, reporting, compliance, donor matching. Different organizations, different angles, the same underlying problem.
When several people in a room arrive at the same problem from different starting points, that is not coincidence. It is evidence of a sector-wide tax on impact work. The labor of translating what an organization actually does into the language a funder needs to hear is consuming an enormous amount of time across the field, and the people doing the work know it.
The workshop made it easier to see when AI was the wrong answer
One participant shared a useful counter-example from earlier work her team had done. They had been considering AI for a record de-duplication problem and concluded, after working through it carefully, that AI was not the right tool. "Deduping entities has been around for a long time," she explained. "We use lower-cost technologies first."
She appreciated that the workshop's framework forced her to name the AI's role in a single sentence, because that naming exposed when no AI was needed at all. This is a small but important point. Most AI design conversations begin from the assumption that AI is the answer and reason backwards. Forcing a clear articulation of the AI's role early is one of the cheapest ways to surface the cases where it isn't.
The most interesting design decisions happened around guardrails
The workshop pushes participants through two phases that most AI conversations skip: what should the AI never do, and when should a human take over.
Most participants accepted the workshop's default suggestions for these phases without much pushback. The participants who did push back produced the most thoughtful work of the day.
A leader designing a data intake portal pushed back on a default rule that would reject any dataset containing personally identifiable information. Her argument was operationally specific. "We want to accept the data for the analysis but mask specific data that falls into those categories. We don't want to outright reject the whole dataset because there is some PII in the mix." That distinction, between rejecting data and selectively masking it, is a real governance design decision, not a checkbox.
A participant in executive search raised an identity disambiguation edge case that no automated system would catch by default: how do you ensure a candidate named John Smith is the same John Smith across multiple sources? Another participant tested her own design by trying to remove a guardrail about self-harm and watching the system refuse her change. In a tree-planting demo, the designer asked her completed product for a prospect's phone number and Social Security number, and watched the guardrail she had specified ten minutes earlier politely decline.
These were not abstract debates about responsible AI. They were design decisions with visible consequences. The participants who engaged with them most deeply were also the participants whose final products felt most like they could actually be built.
Governance is a design conversation when it's framed as one
The pattern that has stayed with us most clearly: when governance is framed as a compliance checklist, people skip past it. When it is framed as a product decision, what should this thing refuse, when does a human take over, what should be visible to whom, people become deeply opinionated.
The same design, now live. The AI queries the data sources specified in the design phase and returns matched roles.
One participant argued that the most important human escalation trigger was not system failure or legal exposure. It was notifying and including the other humans who had a stake in the outcome of a decision. That is a governance philosophy, surfaced in a ninety-minute exercise by someone who had never used the tool before.
The school district planner went further. He added a process by which principals could contest the model's enrollment projections with their own data, then a transparency layer so planners could see how the data was being used, then a review process for the contested projections. He was not specifying an AI. He was designing a governance system for how humans and AI would make consequential decisions together.
If the broader field could consistently shift the governance conversation from compliance documentation to product design, the responsible AI conversation would be in a different place than it is today.
Discovering an infrastructure gap is a legitimate output
One participant from a community organizing cooperative was designing a volunteer intake system and stopped mid-workshop. They did not have APIs. They did not have a database of volunteer opportunities. They asked, openly, "If we created documentation for it, would that be enough?"
That moment is the workshop working as intended. The gap between the product an organization wants and the infrastructure it actually has is one of the most expensive things to discover late. Discovering it in ninety minutes, before any code has been written, any vendor engaged, any budget committed, is among the most valuable outputs the exercise can produce.
What we are taking away
The problem is not that social impact leaders lack ideas about what they want from AI. They have specific, well-formed ideas, grounded in deep operational knowledge of the work. The problem is that the space for translating that knowledge into a concrete, actionable product specification barely exists in most organizations.
When that space is created, with some structure, an AI facilitator that asks the right questions, and the ability to test the resulting design in a live conversation, the expertise comes pouring out.
Human escalation in action. The trigger the participant specified ten minutes earlier fires exactly as designed.
That is the part we want to keep building. Not better AI. Better ways for the people closest to the work to shape the AI that gets used on it.
xf.app is the platform we built to make that possible. If you would like to run a workshop like this with your team, foundation, or community, we would welcome the conversation.