The SA AI Visibility Index

Methodology.

The Index only works if the numbers can be trusted and reproduced. This page is the full method — engines, prompts, sampling, scoring, crawler testing, rounding, fairness rules and stated limitations — versioned alongside the data it governs.

Methodology version 1.1 · applies to Edition 1 — July 2026 (restated 3 September 2026)

What the Index measures

The Index measures which South African brands AI assistants name — and actively recommend — when asked realistic buyer questions about an industry. The unit of measurement is the grounded answer: a response produced by an engine with live web search enabled, the same surface a real user sees, not the model’s training memory.

Engines

Four engines are queried per edition: ChatGPT (gpt-4o-mini), Claude (claude-haiku-4-5), Gemini (gemini-2.5-flash), Perplexity (sonar). All four run with grounding (live web search) enabled. Model identifiers are recorded per answer and published with each vertical, because models drift — a figure is only meaningful alongside the model and date that produced it.

Prompt panels

• Each vertical has a fixed panel of 30–40 prompts, published in full on the Index page. Fixed panels keep editions comparable; changes to a panel are versioned and noted in the changelog.
• Panels mix five prompt classes the way real buyers ask: shortlist (“who are the best…”, national + metro), category (use-case specific), process (“how do I choose…”), comparison (brand vs brand, platform vs professional) and trust (regulatory checks, avoiding overcharging).
• Prompts are phrased in English. This is a stated limitation — South African buyers also ask in other languages, and multilingual panels are on the roadmap.
• Any client of Auto Alpha Advisory is excluded from the tracked set of its vertical, and noted as such. (Their brands can still surface in open capture like anyone else’s — we just don’t rank brands we’re paid to improve.)

Sampling

Every prompt runs 3 times per engine per edition. AI answers are non-deterministic; repeated runs smooth single-run noise and let us report how consistently a brand appears. A typical vertical is therefore 36 prompts × 4 engines × 3 runs = 432 grounded answers. Failed calls are retried; only successful answers are scored, and per-engine answer counts are published in the data files.

Scoring

• Open capture (authoritative). An LLM pass (gpt-4o-mini, temperature 0) reads every answer and extracts each brand named, judging whether it is recommended (presented as a top/best choice) or merely mentioned. It resolves name variants, distinguishes a platform named as a brand from a generic phrase, and — critically — captures brands outside the tracked set, so the Index can’t miss a winner it didn’t know to look for.
• Deterministic cross-check. A word-boundary alias match runs alongside as a fast, fully-reproducible check on the LLM pass. Where they diverge (franchise sub-brands are the usual case), the open capture is used and the divergence is reviewed by hand.
• Prose filter (v1.1, applied to every edition). An extracted brand is published only if its name or an alias appears in the answer prose with cited URLs removed, or in the prompt itself. A brand that appears only inside a source URL is a citation, not a mention, and citations are counted separately. Added 2026-09-03 after the open-capture pass was found to credit domain-named brands with mentions from URL lists; Edition 1 was restated on the same rule so the edition-to-edition delta is measured with one instrument. Restatement note.
• Share of mention = answers naming the brand ÷ all scored answers in the vertical. Shortlist share = the same measure over shortlist-class prompts only. Conviction = of the answers naming the brand, the share presenting it as a recommended choice.
• Merging. Name variants (“PwC” / “PwC South Africa”) merge to one brand. A franchise’s local offices (“RE/MAX Living”) do not merge into the national brand — the engines treat them as distinct answers, so the Index reports them as the engines see them.
• Rounding. Percentages are rounded half-up to whole numbers on the page; the downloadable CSVs carry one decimal; raw counts are published alongside so any figure can be recomputed.

The crawler test

Each tracked brand’s website is fetched as a normal browser and as each AI crawler — twice per user-agent — recording HTTP status, server, word count and structured data, plus the site’s robots.txt directives. From Edition 2 the Index reports two crawler verdicts.

• Search-crawler verdict (new in Edition 2). Measured on OAI-SearchBot, Claude-SearchBot and PerplexityBot, the crawlers OpenAI, Anthropic and Perplexity document as the ones that surface sites in their search answers. A 403 reproduced on a second fetch, or a robots.txt rule opting the crawler out, is reported as a block of that engine’s search, scoped to exactly the crawlers it affects. The browser fetch is the control: if the site refuses the browser too, a crawler’s 403 is not the crawler’s, and it is reported as not tested. Edition 1 did not use this control. Applied afterwards to Edition 1’s stored fetches on 2 October 2026, it footnotes two rows, Rawson and Bowmans, whose browser fetch was refused too; their labels are kept as first published.
• Edition 1 verdict (kept for like-for-like comparison). Measured on GPTBot, ClaudeBot and PerplexityBot, serve-time only, exactly as in Edition 1. By their operators’ own descriptions GPTBot and ClaudeBot collect training data, and OpenAI states that “Each setting is independent of the others.” A GPTBot block is a training opt-out, not evidence that ChatGPT search can’t find the site.
• A 403 that is not reproduced is reported as uneven, never as a block. A 429 (rate-limit) is reported as throttling. A crawler that gets no response on either fetch is reported as not tested, never as visible. A site whose content only renders client-side is reported as “content not server-rendered”.
• Gemini / Google-Extended. Google-Extended is a robots.txt permission token rather than a live crawler, so it is assessed from robots.txt alone and sits outside both verdicts above; a Google-Extended disallow is reported separately, as a permission the site has withdrawn from Gemini.
• The crawler verdict is presented beside share-of-mention, never as its explanation. Absence from AI answers is multi-causal; the Index claims a cause only where it is directly measured. An engine can name a brand whose site it cannot crawl.
• Method note, 1 October 2026. Edition 2 adds the search-crawler verdict. GPTBot’s and PerplexityBot’s user-agent strings were updated to the versions their operators’ documentation shows on that date, and OAI-SearchBot uses OpenAI’s published string. Anthropic publishes crawler tokens but not full user-agent strings, so Claude-SearchBot is fetched with the string third-party bot directories observe; servers match on the token. robots.txt is read per RFC 9309, where consecutive User-agent lines share one group. None of this moved an Edition 1 verdict: re-deriving all 42 published Edition 1 crawler labels from the stored fetches gives the same result. Fetches come from our servers, not from the crawler operators’ published IP ranges, so an IP-level rule at a site’s host or CDN is not tested.

How tracked sets are built

Each vertical tracks 8–14 brands — the recognised national names a buyer would expect to see ranked — plus the platforms and marketplaces buyers weigh against hiring a professional (property portals, legal-expenses insurers, tax software). Platforms are reported separately: an engine pointing you at a place to search is not the same as an engine recommending who to call. The open-capture pass surfaces everyone else; a brand repeatedly surfacing untracked is a promotion candidate for the next edition, noted in the changelog.

Fairness and framing

• The Index measures AI visibility, not business quality. A low share of mention is not a judgement of any brand’s service, people or results, and the Index never presents it as one.
• Every figure is date-stamped and attributed to specific model versions. No figure should be quoted without its measurement window.
• Only confirmed, reproduced crawler blocks are attributed to a brand’s infrastructure, scoped to the exact bots tested.
• Brands named in the Index can query any figure or flag an error: hello@autoalphaadvisory.co.za. Corrections are published in the changelog on this page.

Known limitations

• Point-in-time. Models change weekly; each edition holds for its measurement window. Drift between editions is a feature of the Index, not a flaw in it.
• Sample sizes thin out at the margins. Whole-vertical figures rest on ~400+ answers; single-region or single-prompt slices can rest on a dozen. Single-digit shares should be read as “barely registers”, not as precise rank.
• English-only prompts, and a fixed panel. Real buyers phrase questions in ways no panel fully covers.
• Consumer surfaces only. The Index queries the assistants’ standard chat surfaces with grounding; API behaviour, paid tiers and other model sizes may differ.
• The extraction pass is itself an LLM and can err; the deterministic cross-check and published raw counts exist so its work can be audited.

Changelog

• 2026-07-05 — Edition 1 — July 2026 published: residential property (measured 2026-06-14) and attorneys & law firms (measured 2026-07-04). Methodology v1.0.
• 2026-07-05 — Data update: accounting, audit & tax firms added (measured 2026-07-04; the run completed 2026-07-05 after an interrupted sweep — failed calls were re-run, only successful answers scored, per the sampling rules above).
• 2026-09-03 — Methodology v1.1: export-time prose filter (above). Edition 1 restated on the same rule — 45 property, 10 legal and 4 accounting tracked-brand hits removed, all on non-shortlist prompts; shortlist shares unchanged; two rank changes (Chas Everitt above Rawson in property; STBB above Barnard in legal). Files as first published are archived unchanged; the restated CSVs carry a restated column.