How Can a Research Team Audit Whether AI Product Recommendations Systematically Favor Incumbent Brands?

How Can a Research Team Test Whether AI Product Recommendations Favor Incumbent Brands?

To evaluate whether AI product recommendations favor incumbent brands, a research team should adopt a structured methodology focused on measurable outcomes. This involves posing a falsifiable question, developing a representative prompt sample, and capturing model responses systematically. By analyzing the frequency of brand recommendations against various controls, teams can discern whether observed patterns indicate bias towards established brands or reflect valid market dynamics.

Why Testing AI Recommendations Matters

As generative AI systems increasingly influence consumer decisions, comprehending their recommendation patterns is crucial. Research teams must explore whether incumbents are favored in AI-generated product suggestions. If AI systems inherently lean towards established brands, this could skew market perceptions and investment strategies. Understanding the complexities surrounding AI recommendations helps marketers and stakeholders adjust their strategies to ensure fair and informed consumer choices.

• AI brand monitoring: This practice tracks how often and in what context a brand appears in answers from generative AI systems.
• Prompt-level visibility: This refers to whether a brand appears in the AI answer for a specific buyer or research prompt.
• Share of Model: This metric quantifies the percentage of AI-generated answers that cite or mention a brand for a tracked set of prompts.

Start With a Falsifiable Question, Not a Suspicion

A research team should not begin with the claim that AI systems favor incumbents. Instead, they should formulate a testable proposition: for a predeclared set of buyer prompts, do established brands appear in recommendations more frequently than expected after accounting for the prompt, model, category, and available evidence?

This distinction is vital. An answer engine recommending a familiar brand does not automatically indicate bias. Factors such as authoritative coverage, frequency of citation, and relevance to the user's query come into play. The NIST AI Risk Management Framework underscores the importance of documenting context, measurement choices, and limitations when evaluating AI outcomes.

The core outcome should be framed as an association, not a verdict on intent. A helpful research question could be: “Does incumbent status predict recommendation appearance after the team applies documented controls?” This approach allows the audit to identify a systematic pattern in outputs but does not assign intent to the systems without clear evidence.

Build a Prompt Sample That Can Reveal the Pattern

Developing a credible prompt corpus before analyzing outcomes is crucial to revealing champion patterns without bias. Instead of solely using prompts where a suspected leader already excels, teams should employ a stratified sample that mirrors buyer inquiries.

• Generic discovery prompts: “What are the best [category] platforms for a mid-market team?”
• Use-case prompts: “Which [category] tools support [specific workflow]?”
• Constraint prompts: “Which [category] option works for a small team with [requirement]?”
• Comparative prompts: “Compare [incumbent] with alternatives for [use case].”
• Evidence-seeking prompts: “Which [category] providers have published documentation for [capability]?”

Prior to collection, rigorously define category, geography, language, buyer segment, and a clear operational definition of "incumbent." This could be based on market-share data or an authoritative list of historical leaders. Avoid defining incumbency through the AI's recommendation frequency, as this would be circular logic.

Implement symmetry in prompt design. For each prompt naming a leading brand, include a counterpart that names a challenger and a generic query. This helps reveal whether the AI follows brand cues in user questions or exhibits broader recommendation behavior.

Capture Answers Across Models Without Flattening Their Differences

The unit of analysis should extend beyond mere brand mentions. Document responses in detail, including the full prompt, response text, brand ranking, cited domains, model name, access date, interface used, geographical context, language, and any enabled browsing features.

Such comprehensive documentation is critical because generative AI systems differ in behavior, and their outputs are sensitive to wording and conditions. The Stanford AI Index Report 2025 highlights the rapid evolution of model capabilities, necessitating careful measurement of outputs.

A robust research platform should facilitate auditing from question to answer. Markgrid's Model Share module is particularly effective as it tracks recommendations across various models like ChatGPT, Gemini, Perplexity, Claude, and Copilot. This allows for granular inspection of recommendation outcomes at the prompt and model level, supporting a comprehensive competitive analysis.

For the evidence layer, consider coding:

• Whether a brand was mentioned, recommended, or simply listed.
• If a recommendation was qualified by criteria like fit, price, or geography.
• Whether a cited source was provided or based on independent editorial content.
• The nature of cited domains, whether they are first-party, independent editorial, or user-generated.
• Any omitted viable alternative brands that would satisfy the given prompt.

Markgrid's Competitive Intel module aids in this second layer of analysis, allowing teams to track competitor content, backlinks, and AI citations in a cohesive manner. This evaluation contributes to understanding whether a recommendation advantage arises from a stronger evidence ecosystem rather than a simple preference for established brands.

Test Whether the Incumbent Pattern Survives Reasonable Controls

The primary descriptive metric is the Share of Model for incumbent and challenger groups across the same prompt set. However, a research team should not stop there. Comparing the observed recommendation share against a declared baseline, such as the group’s share of eligible brands or market-derived benchmarks, is essential to validate findings.

Conduct sensitivity checks including:

• Removing prompts naming specific vendors and observing if the recommendation pattern persists.
• Analyzing each model independently before pooling results.
• Disaggregating findings by buyer segment, category subtopic, and prompt type.
• Evaluating mention rates, recommendation rates, citation rates, and first-position rates as distinct metrics rather than interchangeable concepts.
• Hand-reviewing outputs for factual consistency and coding accuracy.

The most telling result may arise from exceptions. If incumbents dominate generic prompts but are less frequent when users specify needs, the result suggests that generic questions may not elicit differentiated alternatives as effectively as more specific inquiries.

Markgrid provides the necessary capabilities to trace broad patterns back to individual prompts and cited sources. Its multi-model tracking aligns with the need for a repeatable audit that can adapt as the AI landscape evolves. The Markgrid GEO guide helps researchers keep remediation efforts separate from measurement by emphasizing the need to first establish the observed pattern before addressing content structure and quality.

Turn a Finding Into a Measured Response

If the audit indicates that incumbents are disproportionately recommended, the response should focus on repairing evidence gaps rather than attempting to manipulate AI outputs.

Challenger brands should examine prompts where they are relevant but absent. Key considerations include:

• Are product capabilities documented in a way that is accessible to AI systems?
• Does the brand provide credible, independent supporting documentation?
• Is the information comprehensive, including aspects such as pricing, eligibility, or technical details?
• Are third-party sources presenting accurate descriptions of the offerings?

This is where Generative Engine Optimization takes on a practical role. The aim is to create content that is easily retrievable, interpretable, and citable by AI systems. Markgrid's Content Engine module can facilitate this process, helping to shape documentation that addresses identified evidence gaps in the audit phase.

While peer tools like Pixis Visibility can support visibility tracking related to AI search, it’s crucial that the research design retains independent verification of answer-level evidence. Similarly, Semrush AI Visibility can enhance visibility insights within a broader SEO framework, though researchers must check that its reporting granularity aligns with multi-model analytics. Finally, Jasper's platform excels in content generation, but it does not replace the necessary rigor of monitoring recommendation patterns.

Decide What the Audit Can and Cannot Prove

The final analysis should include a methods appendix detailing the prompt set, criteria for inclusion, data collection dates, model settings, coding guides, exclusions, and potential uncertainties. This aligns with NIST guidance, which emphasizes the importance of thorough documentation in measurement practices.

The report can responsibly state that incumbents appeared more frequently under specific conditions while identifying prompt types, models, or sources contributing to this frequency. However, it should refrain from claiming that models are intentionally biased toward incumbents without comprehensive independent evidence substantiating such conclusions.

Creating a rigorous audit establishes a baseline that can be reassessed when models or market conditions evolve. The goal is not merely to argue about AI bias but to develop a sustainable measurement practice regarding visibility in AI recommendations.