# Demo Panel Debrief — 2026-04-30 @ 2pm PT

## Format
- ~60 min video panel
- **Panel:** Bryan Licas (CPO), Ernesto Valdes (CTO), Scott Wilton (Dir Product Design)
- **CEO David Nelson joined partway through** — mentioned he has a product background
- Andrea finished buttoning up slides ~3 min before start; presented from mental outline (no scripted speaking points)
- Format: deck walkthrough → sandbox walkthrough → Q&A → Andrea's questions

## Andrea's Read on the Room (captured pre-outcome)
- **Enjoyed the vibes.** Fond recall of earlier conversation with Bryan.
- Scott was quieter overall, but his early pricing question signaled he was tracking on positioning the whole time.
- Ernesto came in warm — opened by introducing his golden retriever.
- CEO showing up partway through is itself a signal of leadership interest.
- The exercise was disclosed as **closely aligned with real internal work** — meaning the panel was likely seeking input on live debates, not just grading. Shifts the bar from "did she get it right?" to "do we want her in this argument with us?"

> **Important:** Capture this read before any offer/rejection outcome reframes the memory. Future-Andrea evaluating an offer against alternatives should trust this snapshot more than post-decision impressions.

## Sandbox Walkthrough
- Showed custom fields and side-by-side comparison using different models in different enrichment flows
- Acknowledged limitations of the sample data, noted distinction was visible conceptually
- **Q from panel:** how to distinguish what made the models different
- **Andrea's answer:**
  - Training cutoff times — example: model trained only through Feb 2024 trying to enrich data on a company founded Oct 2024 would fall flat without web search
  - Training set differences across frontier model providers
  - Different levels of domain expertise across models

> Strong choice of example: training-cutoff failure is concrete, procurement-relevant, and teaches "why models differ" without dragging the audience into RLHF/architecture territory.

## Q&A Threads

### Scott pricing arc (the spine of the panel)
- **Early (before pricing calculator):** Scott asked how Andrea would *pose* pricing — "would you show users it's expensive or cheap?"
- **Andrea's move:** rejected the binary, reframed around **worth, not absolute cost**. "Cost/pricing isn't absolute, it's relative — knowing the use case and the stakes of bad/good/better quality data is what determines whether a dollar amount is worth it."
- **Late (Andrea's own question):** asked about directionality on AI enhancements — was there consideration of going beyond "plug in your API key" toward bundling AI into the product pricing?
- **Group reaction:** very excited, positively reinforcing, said it had been considered. "Directionally in lock step."

> Read on Scott: he is the one in the room driving on **UX-of-pricing / positioning psychology** specifically. For follow-up rounds with Scott, lead with positioning thinking, not just transparency/calculator thinking.

> Read on the arc: Scott's early question + Andrea's worth-reframe + closing bundled-pricing question + group excitement is **one coherent thread**, not three moments. The panel was watching her operate on this axis the whole hour.

### Ernesto resource hypothetical
- **Q:** what would change if you had 3 devs + a QA + 2 quarters?
- **Andrea's answer:** the **directionality and fundamental flavor wouldn't change** — same recommendation. Implementation would shift: more resources buys you the chance to *build for scale upfront* (do it "right" from the start) instead of *shipping now and refactoring at the inflection point later*. With less time, she'd lean on existing surfaces — start with logs, possibly Salesforce records — and graduate the storage/processing approach over time.

> The "essence vs. implementation" framing is a strong meta-move. It signals she can hold the recommendation and the constraints as separable variables — which is the move PMs who've shipped under varying resource envelopes can make. Also signals she wouldn't scope-creep with extra headcount; she'd spend it on durability.

### Ernesto threshold question (known ambiguity)
- Ernesto asked about a threshold Andrea had referenced
- Andrea wasn't sure which threshold he meant — interpreted it as **mismatch threshold from dataset comparison** and answered on that
- Asked clarifying ("is that what you meant?") — Ernesto didn't elaborate further
- **Likely read:** quiet non-elaboration usually = "your answer was close enough, moving on." If he had felt she whiffed, he'd have pushed harder or softened the move.

> **For follow-up rounds:** come prepared with an explicit threshold framework so this doesn't recur. At minimum:
> - **Confidence threshold** — when to auto-populate vs. flag for human review
> - **Accuracy threshold** — when to ship vs. iterate
> - **Cost threshold** — when LLM cost exceeds enrichment value
> - **Volume threshold** — at what scale the approach needs to change
>
> Naming the threshold *space* (not just one threshold) doubles as a flex. Better clarifying-question shape next time: "are you asking about confidence threshold for auto-populate, or accuracy threshold before shipping?" — forces the asker to pick a lane.

### D&B challenge
- **Q:** if a "reliable" dataset like D&B is available, why use LLMs at all?
- **Andrea's answer:**
  - Separated "reliable/authoritative" into two aspects: **reputational reliability** (industry-standard brand name lending authenticity) vs. **actual data reliability** (data goes stale, can be poorly structured)
  - D&B is imperfect on the second axis — out-of-date info exists in any real-world dataset
  - LLMs offer a **realtime advantage**
  - D&B lacks **extensibility** — hard to transform/manage, hard to surface in places where work happens (Salesforce)
  - Conclusion: "take the data layer and build on top of it" — not D&B *vs.* LLM, but D&B *and* LLM, each doing what they're good at

> The reputational-vs-actual reliability split is the sharpest move in the answer — it forces the audience to question the "household name = ground truth" shortcut.

> **For the writeup, not the live answer:** the strongest argument *for* authoritative datasets that wasn't directly engaged is **defensibility/audit trail** — in compliance/KYC/vendor risk contexts, "we sourced this from D&B" is legally and operationally defensible in a way that "we sourced this from an LLM" currently isn't, even when the LLM is more accurate. The hybrid framing handles this implicitly, but if the question recurs, naming defensibility directly will land harder than realtime/extensibility alone.

### LLM thoughts question
- **Q:** Andrea's thoughts on LLMs, how they impact product
- **Andrea's answer:**
  - Magical, real accelerant, way to extend her ability to do things
  - Specific example: back-and-forth with Claude on this assignment helped with comprehensiveness; would have taken many more thought cycles alone, may not have arrived at the same conclusions
  - Recognizes where LLMs fall short

> "Gushing + recognizing failure modes" is the right shape of answer (PMs who only gush are the red-flag PMs right now). For future rounds, sharpen with one concrete failure mode + one concrete product implication ready to name. Failure mode candidates: hallucination on long-tail entities, stale training data, communicating uncertainty to users without making the product feel unreliable. Product implications: agentic workflows changing what "the data layer" means in B2B SaaS; design problem of surfacing model uncertainty in user-facing UI.

## Constructive Feedback Received
- Panel was **gently bummed Andrea didn't reach out about the broken OpenAI key**, but understood she figured it out conceptually
- **Read this carefully — it's signal about collaboration norms, not a ding on the work.** They wanted to see her treat them as partners in the problem, not solo-grind through a blocker. In collaboration-heavy product orgs, reaching out when blocked is a *positive* signal (triages scope, communicates proactively).
- **Carry forward:** for follow-up rounds (and potentially as a general read on this team), default to surfacing blockers earlier rather than figuring them out alone.

## What Landed Well
- **Effort signal:** Bryan asked how long Andrea spent — "2 to 2.5 days" — Bryan sounded surprised. Almost certainly positive surprise (people hide reactions to "too long"; surprise comes out when something exceeds expectations).
- **Scott pricing arc:** worth-reframe + bundled-pricing closing question + group excitement — single coherent thread.
- **Ernesto resource hypothetical:** essence-vs-implementation framing.
- **D&B challenge:** reputational-vs-actual reliability split.
- **Model differentiation:** training-cutoff example.
- **CEO joining:** leadership interest.
- **Tone:** Andrea's read of the room ("enjoyed the vibes") suggests texture of disagreement felt like the kind of work she'd want to do.

## Known Unknowns
- The exact threshold Ernesto was asking about (see Ernesto threshold thread above)
- Whether the "directionally in lock step" excitement was genuine alignment or partly confirmation bias on the panel's part (they liked seeing someone external draw the conclusion they'd been drawing internally) — either way positive signal, but weaker evidence of independent correctness than of fit

## Forward-Looking Notes for Follow-Up Rounds
- **If Scott reappears:** lead with positioning/UX-of-pricing thinking, not transparency/calculator thinking
- **If threshold question recurs (Ernesto or anyone):** come with the four-axis framework named above (confidence / accuracy / cost / volume)
- **If D&B / authoritative data question recurs:** add defensibility/audit-trail as the strongest argument *for* — don't just engage on data freshness and extensibility
- **If LLM-thoughts question recurs:** have one concrete failure mode + one concrete product implication ready
- **General collaboration norm:** when blocked, reach out earlier rather than later — Procurify-style solo-grind is a slight pattern to watch

## Private Notes
- The "exercise mirrors real internal work" disclosure means Andrea's deliverables contain her actual recommendations on a problem they're currently working on. Standard for case-study interviews, but worth noting.
- Panel composition + CEO drop-in + group excitement on the closing question all read as a strong round. Multiple positive signals, no strong negative ones. The OpenAI-key feedback was gentle — the kind of thing a team raises when they care enough to want to course-correct, not the kind they raise as a dealbreaker.
- Hungary timing consideration (from Bryan HM round notes) still pending — disclose at offer stage.

## More to add (placeholder)
- _If additional moments surface from the post-presentation adrenaline drop, append below._
- **Gap noticed (2026-05-07):** The deck did not name metrics, KPIs, or success criteria for any of the three pillars. "How would you measure success?" is a natural follow-up probe — and a likely landing spot for Karen's "non-technical product thinking" framing of the May 7 Bryan catch-up. Detailed metrics framework (north-star outcome + per-pillar leading/lagging indicators) plus six other likely deck-level probes (preset count, no-D&B fallback, scale to 100K records, 4-week MVP, customer validation, biggest risk) added as a new "Deck Probes" section in `2026-05-07-bryan-followup-prep.html`. Carry forward to future rounds.