Dev Insights: Giving 3,000+ developers a say in what we built
Developers can be captive users, too. In a large enterprise, they don't choose their internal tools, platforms, or processes. They use what they're given, every day, whether it works or not. I founded Dev Insights so their experience could shape where the platform organization invested.
- ROLE
- Founder and Lead, Dev Insights
- TEAM
- Enterprise Developer Experience, a shared-service team building internal tools for developers
- POPULATION
- 3,000+ developers across 6 technology orgs
- FRAMEWORK
- DX Core 4, DX Snapshot self-assessment
The problem
Leadership knew developers were losing time. No one could say where.
Directors and VPs saw 3,000+ developers as one group: "full-stack." One label, one set of assumptions. Developers had no channel to say what was broken, and no choice but to keep using it.
The real problem wasn't a missing survey. Leadership had no evidence to decide where to invest in developer experience. Dev Insights gave them that evidence.
My role
I founded the program and led every part of it.
I was the only designer on the Enterprise Developer Experience team, a shared-service team building internal tools for 3,000+ developers.
I started Dev Insights after usage of Chassis CodeGen, one of our own products, dipped. We wanted to know why and had no way to ask developers. Dev Insights became that way.
I ran it alongside my product design work. I led every engagement, from planning to the readout, with support from teammates who observed and took notes. I presented findings to the VP's leadership team and built the ChatGPT project that let leadership query the research directly.
- I owned: program design, recruiting, research plans, facilitation, synthesis, journey maps, and readouts to leadership.
- My team's Senior Director and managers: sponsorship and team priorities.
- The VP's leadership team (Principals, Directors, and Senior Directors of the Developer Platform portfolio): the audience for findings, and the owners of the OKRs those findings set.
- Participant team managers: triaged their own team's Snapshot results.
- Developers: 450 participants at every level, from Associate to Advisor.
Research and discovery
I built a standing panel instead of running one-off studies, so the method could match the question.
I started with referrals. I asked the people who worked closest to developers who they liked working with. Then I asked those developers the same question, and kept pulling the thread. I characterized each panelist: front-end, back-end, or full-stack; IDE; tenure; years of experience; assets supported; and reporting chain.
That let me pick the method per question. If a product owner needed to know whether a workflow was broken or just unfamiliar, I could pull five back-end developers on JetBrains who supported that asset. If we needed the whole division's picture, we ran the Snapshot.
Division of 3,000+ developers
Developers from all across CIO
Dev Insights Team
Methods: 1:1 interviews, focus groups and empathy sessions, design reviews and cognitive walkthroughs, usability testing and UAT, priority card sorting, the quarterly DX Snapshot, and DX Studies deep-dive surveys.
Activities
Planning
Scheduling the activity, determining which developers to include, emailing their managers, determining the kind of activity to run, and other preparation tasks.
The Activity
This is the core of the research initiative. An activity could be a design review, a series of developer interviews, a survey, usability test, card sorting exercise, etc.
Analysis
After the activity, the team must take the time to review the feedback data. The analysis should include recommendations for product optimization.
Optimization
Using the analysis, the team works to optimize the product based on the feedback analysis - updating the design, revising the workflow, updating the code, etc.
Each activity looped back to Planning once it was done, so the next one started from what had changed. Activities with developers in the room had a facilitator and an observer: the facilitator ran the session and talked with developers, and the observer took notes and watched how they responded.
Findings
- "Full-stack" hid real differences in role, tooling, and experience that changed what each group needed.
- Deep work and build-and-test scored below the industry P75 benchmark. Developers weren't failing at their jobs. Meetings, tool switching, and slow feedback loops kept them from doing them.
- Work around the code cost as much as work in it. Registration processes ran through manual spreadsheets and repeated validation cycles (see section 4).
Journey maps and workflow artifacts
I mapped the work developers had to do around their code, not just in it.
To better understand the developer experience, I built customer journey maps for two registration processes every data-producing team had to complete: interface registration (ICR) and dataset registration (DSET). Each map followed a full-stack developer through seven stages, from awareness to access provisioning and feedback. Each stage had six layers: steps, pain points, positive points, frontstage, backstage, and support.
View the journey maps in FigJam (opens in a new tab)
Findings
- Developers entered every field of the source-to-target mapping by hand.
- Preparation and submission ran as a loop. Developers went through it four to five times. Errors came back in about an hour, and most were formatting errors.
- Developers resubmitted a spreadsheet they had already submitted during dataset registration. Earlier submissions were hard to find.
- Some integration patterns needed three separate requests: interface registration, producer offerings, and consumer contracts.
- Developers asked for one UI to select data mappings and generate the request, instead of a chain of spreadsheets.
View the Customer Research roadmap and service offerings in FigJam (opens in a new tab) - a 12-week plan of monthly check-ins, weekly office hours, 1:1s and focus groups, the Snapshot release, deep-dive surveys, and a test-plan → usability test → analysis → optimization cycle.
Working with users and partners
Leaders and developers were two audiences, so I gave each a different deal.
- Leaders (advocates): shared their business goals for developer experience, gave us access to their developers, passed updates to their teams, and attended monthly updates.
- Developers: about five hours a month. Attend activities and monthly meetings. Speak up in group sessions. Cameras on. Bring back friction points from their teams.
We ran it all through a dedicated Teams channel, monthly check-ins, and weekly office hours. Each Snapshot participant's manager could triage their team's results, so findings reached someone who could act on them at team level, not only at the VP level.
Five hours a month from the panel gave us a standing line into how thousands of developers actually worked. Teams building public products pay a lot for that kind of access.
Iterations and decisions
The first panel worked, but it didn't represent enough of the organization. So I rebuilt it.
- Version 1: 45 developers, recruited by referral. It got us passionate, engaged developers fast. But referral sampling favors the visible and well-connected. Too many teams and portfolios across the six orgs had no voice.
- Version 2: 450 developers, recruited across all six orgs. When the DX Intelligence Platform was approved, I used the rollout to expand the panel to 450, roughly one in seven developers in the division. I used ChatGPT to sort the org data and find the strongest candidates from teams across every org.
- What happened to the original 45: they became champions. They rallied their teams to take the quarterly Snapshot and stayed our first call for interviews, focus groups, and design reviews.
- Pilot before scale: we ran a pilot survey to validate the platform before the division's first full Snapshot.
The panel after expansion
- 45 : ChampionsThe original 45, now champions
- 450 : Snapshot participantsRecruited across all six orgs, roughly one in seven developers
- 3,000 : All the developers in CIOEvery developer in the division
Two tiers, two jobs. The champions gave us depth and outreach. They would rally the dev teams they worked in, listened to their pain points, jotted down their own pain points, and encouraged developers on their team to take the quarterly Snapshot survey.
The 450 gave us a survey population that represented 1 in 7 developers. Imagine having a product where you could speak to one-seventh of your customer base.
That kind of representation made this a true Dev Insights team. We stood up the DX Intelligence Platform on the DX Core 4 framework and ran the division's first developer self-assessment — the Snapshot — alongside driver deep-dive surveys and product assessments.
The April Snapshot pointed hard at two things. Deep work scored low — developers weren't failing at their jobs, they were being prevented from doing them, fragmented across meetings, tools, and context switches. Build and test scored low alongside it: the loop between writing code and knowing whether it worked was slow enough to break concentration on its own.
That reframed the problem for leadership from “developer productivity” to “developer focus,” and it set the priorities for the year.
Information hierarchy
The April Snapshot flagged several problem areas. I put two in front of leadership.
I chose deep work and build-and-test using two criteria: how far each fell below the industry P75 benchmark, and how much developers said it mattered. The other areas stayed in the data and in the research repository, but out of the headline.
That choice reframed the conversation from "developer productivity" to "developer focus," and it set the year's priorities.
The ChatGPT project (section 8) handled the depth. The readout carried the headline. Leaders who wanted detail queried the research directly instead of digging through slides.
What we built
A research practice, not a research project.
- Two-tier panel: 45 champions and 450 Snapshot participants, across all six technology orgs.
- A quarterly rhythm: DX Snapshot, deep-dive studies, and a standing rotation of qualitative activities.
- Journey maps for the processes developers depended on.
- An AI research assistant: I built a ChatGPT project loaded with Snapshot results, interview findings, and journey maps. I wrote its instructions and skills so it produced consistent reports and spreadsheets. My team and senior leadership, up through the portfolio VP, queried it directly.
The results
From the April to the July Snapshot, the same 450 developers reported a 12% improvement in deep work and a 12.5% improvement in build and test.
+12%
Deep work
April → July Snapshot, same 450 developers
Driver: No Recurring Meeting Thursdays
+12.5%
Build and test
April → July Snapshot, same 450 developers
Driver: GitHub Copilot and Codex, chosen through the AI tool pilot assessments
- Findings set CIO-level OKRs, roadmap priorities, and DevEx North Star metrics.
- Deep work intervention: No Recurring Meeting Thursdays, a division-wide protected day.
- Build and test intervention: GitHub Copilot in VS Code and Codex, chosen through the AI tool pilot assessments.
- An AI research assistant: I built a ChatGPT project loaded with Snapshot results, interview findings, and journey maps. I wrote its instructions and skills so it produced consistent reports and spreadsheets. My team and senior leadership, up through the portfolio VP, queried it directly.
The division attributes the build-and-test gain to the assistants and the assessment process that picked them, together. Deep work's gain maps to one program cleanly. But one cycle isn't a trend, and no survey wave separates a protected Thursday — or a well-targeted pilot — from everything else that happened in the same quarter. We'd want several more waves before calling either durable.