The SA AI Visibility Index

Methodology.

The Index only works if the numbers can be trusted and reproduced. This page is the full method — engines, prompts, sampling, scoring, crawler testing, rounding, fairness rules and stated limitations — versioned alongside the data it governs.

Methodology version 1.0 · applies to Edition 1 — July 2026

What the Index measures

The Index measures which South African brands AI assistants name — and actively recommend — when asked realistic buyer questions about an industry. The unit of measurement is the grounded answer: a response produced by an engine with live web search enabled, the same surface a real user sees, not the model’s training memory.

Engines

Four engines are queried per edition: ChatGPT (gpt-4o-mini), Claude (claude-haiku-4-5), Gemini (gemini-2.5-flash), Perplexity (sonar). All four run with grounding (live web search) enabled. Model identifiers are recorded per answer and published with each vertical, because models drift — a figure is only meaningful alongside the model and date that produced it.

Prompt panels

• Each vertical has a fixed panel of 30–40 prompts, published in full on the Index page. Fixed panels keep editions comparable; changes to a panel are versioned and noted in the changelog.
• Panels mix five prompt classes the way real buyers ask: shortlist (“who are the best…”, national + metro), category (use-case specific), process (“how do I choose…”), comparison (brand vs brand, platform vs professional) and trust (regulatory checks, avoiding overcharging).
• Prompts are phrased in English. This is a stated limitation — South African buyers also ask in other languages, and multilingual panels are on the roadmap.
• Any client of Auto Alpha Advisory is excluded from the tracked set of its vertical, and noted as such. (Their brands can still surface in open capture like anyone else’s — we just don’t rank brands we’re paid to improve.)

Sampling

Every prompt runs 3 times per engine per edition. AI answers are non-deterministic; repeated runs smooth single-run noise and let us report how consistently a brand appears. A typical vertical is therefore 36 prompts × 4 engines × 3 runs = 432 grounded answers. Failed calls are retried; only successful answers are scored, and per-engine answer counts are published in the data files.

Scoring

• Open capture (authoritative). An LLM pass (gpt-4o-mini, temperature 0) reads every answer and extracts each brand named, judging whether it is recommended (presented as a top/best choice) or merely mentioned. It resolves name variants, distinguishes a platform named as a brand from a generic phrase, and — critically — captures brands outside the tracked set, so the Index can’t miss a winner it didn’t know to look for.
• Deterministic cross-check. A word-boundary alias match runs alongside as a fast, fully-reproducible check on the LLM pass. Where they diverge (franchise sub-brands are the usual case), the open capture is used and the divergence is reviewed by hand.
• Share of mention = answers naming the brand ÷ all scored answers in the vertical. Shortlist share = the same measure over shortlist-class prompts only. Conviction = of the answers naming the brand, the share presenting it as a recommended choice.
• Merging. Name variants (“PwC” / “PwC South Africa”) merge to one brand. A franchise’s local offices (“RE/MAX Living”) do not merge into the national brand — the engines treat them as distinct answers, so the Index reports them as the engines see them.
• Rounding. Percentages are rounded half-up to whole numbers on the page; the downloadable CSVs carry one decimal; raw counts are published alongside so any figure can be recomputed.

The crawler test

AI engines can only read what their crawlers are served. Each tracked brand’s website is fetched as a normal browser and as GPTBot, ClaudeBot and PerplexityBot — twice per user-agent — recording HTTP status, server, word count and structured data, plus the site’s robots.txt directives.

• A 403 served to a named crawler, reproduced on a second fetch, is reported as a hard block, scoped to exactly the bots it affects. Nothing else is called a block.
• A 429 (rate-limit) is reported as throttling, not blocking. A site whose content only renders client-side is reported as “content not server-rendered”.
• Gemini / Google-Extended. Google-Extended is a robots.txt permission token rather than a live crawler, so it is assessed from robots.txt alone — a robots-only signal beyond the three serve-time bots above. The Index’s serve-time crawlability verdict is measured on GPTBot, ClaudeBot and PerplexityBot; a Google-Extended disallow is reported separately, as a permission the site has withdrawn from Gemini.
• The crawler verdict is presented beside share-of-mention, never as its explanation. Absence from AI answers is multi-causal; the Index claims a cause only where it is directly measured.

How tracked sets are built

Each vertical tracks 8–14 brands — the recognised national names a buyer would expect to see ranked — plus the platforms and marketplaces buyers weigh against hiring a professional (property portals, legal-expenses insurers, tax software). Platforms are reported separately: an engine pointing you at a place to search is not the same as an engine recommending who to call. The open-capture pass surfaces everyone else; a brand repeatedly surfacing untracked is a promotion candidate for the next edition, noted in the changelog.

Fairness and framing

• The Index measures AI visibility, not business quality. A low share of mention is not a judgement of any brand’s service, people or results, and the Index never presents it as one.
• Every figure is date-stamped and attributed to specific model versions. No figure should be quoted without its measurement window.
• Only confirmed, reproduced crawler blocks are attributed to a brand’s infrastructure, scoped to the exact bots tested.
• Brands named in the Index can query any figure or flag an error: hello@autoalphaadvisory.co.za. Corrections are published in the changelog on this page.

Known limitations

• Point-in-time. Models change weekly; each edition holds for its measurement window. Drift between editions is a feature of the Index, not a flaw in it.
• Sample sizes thin out at the margins. Whole-vertical figures rest on ~400+ answers; single-region or single-prompt slices can rest on a dozen. Single-digit shares should be read as “barely registers”, not as precise rank.
• English-only prompts, and a fixed panel. Real buyers phrase questions in ways no panel fully covers.
• Consumer surfaces only. The Index queries the assistants’ standard chat surfaces with grounding; API behaviour, paid tiers and other model sizes may differ.
• The extraction pass is itself an LLM and can err; the deterministic cross-check and published raw counts exist so its work can be audited.

Changelog

• 2026-07-05 — Edition 1 — July 2026published: residential property (measured 2026-06-14) and attorneys & law firms (measured 2026-07-04). Methodology v1.0.
• 2026-07-05— Data update: accounting, audit & tax firms added (measured 2026-07-04; the run completed 2026-07-05 after an interrupted sweep — failed calls were re-run, only successful answers scored, per the sampling rules above).