Search engines no longer operate alone. Your content is now consumed by
Google, Bing, Perplexity, ChatGPT, Claude, Gemini, and dozens of other
AI driven systems that crawl the web and extract answers.
Classic SEO focuses on ranking. Modern discovery also requires AEO (Answer Engine Optimization) which focuses on being understood and extracted by AI systems. A marketing page must therefore satisfy four technical conditions:
- It must be crawlable
- It must be indexable
- It must be structured so machines understand it
- It must contain content that AI systems can extract and summarize
Many sites fail before content quality even matters. Robots rules block
crawlers, canonical tags are missing, structured data is absent, or the
page simply contains too little readable content.
The easiest way to diagnose this is to run a single script that inspects
the page like a crawler would.
The following Bash script performs a quick diagnostic to check whether
your page is friendly for both search engines and AI answer systems.
The script focuses only on technical discoverability, not marketing copy
quality.
2. What the Script Checks
The script inspects the following signals.
Crawlability
- robots.txt presence
- sitemap.xml presence
- HTTP response status
Indexability
- canonical tag
- robots meta directives
- noindex detection
Search Metadata
- title tag
- meta description
- OpenGraph tags
Structured Data
- JSON LD schema detection
Content Structure
- heading structure
- word count
- lists and FAQ signals
AI Extraction Signals
- presence of lists
- FAQ style content
- paragraph density
This combination gives a quick technical indication of whether a page is
discoverable and understandable by both crawlers and AI systems.
3. Installation Script
Run the following command once on your Mac.\
It will create the diagnostic script and make it executable.
cat << 'EOF' > ~/seo-aeo-check.sh
#!/usr/bin/env bash
set -euo pipefail
URL="${1:-}"
if [[ -z "$URL" ]]; then
echo "Usage: seo-aeo-check.sh https://example.com/page"
exit 1
fi
UA="Mozilla/5.0 (compatible; SEO-AEO-Inspector/1.0)"
TMP=$(mktemp -d)
BODY="$TMP/body.html"
HEAD="$TMP/headers.txt"
cleanup() { rm -rf "$TMP"; }
trap cleanup EXIT
pass=0
warn=0
fail=0
p(){ echo "PASS $1"; pass=$((pass+1)); }
w(){ echo "WARN $1"; warn=$((warn+1)); }
f(){ echo "FAIL $1"; fail=$((fail+1)); }
echo
echo "========================================"
echo "SEO / AEO PAGE ANALYSIS"
echo "========================================"
echo
curl -sSL -A "$UA" -D "$HEAD" "$URL" -o "$BODY"
status=$(grep HTTP "$HEAD" | tail -1 | awk '{print $2}')
ctype=$(grep -i content-type "$HEAD" | awk '{print $2}')
echo "URL: $URL"
echo "Status: $status"
echo "Content type: $ctype"
echo
if [[ "$status" =~ ^2 ]]; then
p "Page returns successful HTTP status"
else
f "Page does not return HTTP 200"
fi
title=$(grep -i "<title>" "$BODY" | sed -e 's/<[^>]*>//g' | head -1)
if [[ -n "$title" ]]; then
p "Title tag present"
echo "Title: $title"
else
f "Missing title tag"
fi
desc=$(grep -i 'meta name="description"' "$BODY" || true)
if [[ -n "$desc" ]]; then
p "Meta description present"
else
w "Meta description missing"
fi
canon=$(grep -i 'rel="canonical"' "$BODY" || true)
if [[ -n "$canon" ]]; then
p "Canonical tag found"
else
f "Canonical tag missing"
fi
robots=$(grep -i 'meta name="robots"' "$BODY" || true)
if [[ "$robots" == *noindex* ]]; then
f "Page contains noindex directive"
else
p "No index blocking meta tag"
fi
og=$(grep -i 'property="og:title"' "$BODY" || true)
if [[ -n "$og" ]]; then
p "OpenGraph tags present"
else
w "OpenGraph tags missing"
fi
schema=$(grep -i 'application/ld+json' "$BODY" || true)
if [[ -n "$schema" ]]; then
p "JSON-LD structured data detected"
else
w "No structured data detected"
fi
h1=$(grep -i "<h1" "$BODY" | wc -l | tr -d ' ')
if [[ "$h1" == "1" ]]; then
p "Single H1 detected"
elif [[ "$h1" == "0" ]]; then
f "No H1 found"
else
w "Multiple H1 tags"
fi
words=$(sed 's/<[^>]*>/ /g' "$BODY" | wc -w | tr -d ' ')
echo "Word count: $words"
if [[ "$words" -gt 300 ]]; then
p "Page contains enough textual content"
else
w "Thin content detected"
fi
domain=$(echo "$URL" | awk -F/ '{print $1"//"$3}')
robots_url="$domain/robots.txt"
if curl -s -A "$UA" "$robots_url" | grep -q "User-agent"; then
p "robots.txt detected"
else
w "robots.txt missing"
fi
sitemap="$domain/sitemap.xml"
if curl -s -I "$sitemap" | grep -q "200"; then
p "Sitemap detected"
else
w "No sitemap.xml found"
fi
faq=$(grep -i "FAQ" "$BODY" || true)
if [[ -n "$faq" ]]; then
p "FAQ style content detected"
else
w "No FAQ style content"
fi
lists=$(grep -i "<ul" "$BODY" || true)
if [[ -n "$lists" ]]; then
p "Lists present which helps answer extraction"
else
w "No lists found"
fi
echo
echo "========================================"
echo "RESULT"
echo "========================================"
echo "Pass: $pass"
echo "Warn: $warn"
echo "Fail: $fail"
total=$((pass+warn+fail))
score=$((pass*100/total))
echo "SEO/AEO Score: $score/100"
echo
echo "Done."
EOF
chmod +x ~/seo-aeo-check.sh 4. Running the Diagnostic
You can now check any page with a single command.
~/seo-aeo-check.sh https://yourdomain.com/page Example:
~/seo-aeo-check.sh https://andrewbaker.ninja The script will print a simple report showing pass signals, warnings,
failures, and an overall score.
5. How to Interpret the Results
Failures normally indicate hard blockers such as:
- missing canonical tags
- no H1 heading
- noindex directives
- HTTP errors
Warnings normally indicate optimization opportunities such as:
- missing structured data
- thin content
- lack of lists or FAQ style sections
- missing OpenGraph tags
For AI answer systems, the most important structural signals are:
- clear headings
- structured lists
- question based sections
- FAQ schema
- sufficient readable text
Without these signals many AI systems struggle to extract meaningful
answers.
6. Why This Matters More in the AI Era
Search engines index pages. AI systems extract answers.
That difference means structure now matters as much as keywords. Pages that perform well for AI discovery tend to include:
- clear headings
- structured content blocks
- lists and steps
- explicit questions and answers
- schema markup
When these signals exist, your content becomes much easier for AI
systems to interpret and reference. In other words, good AEO makes your content easier for machines to read, summarize, and cite. And in an AI driven discovery ecosystem, that visibility increasingly
determines whether your content is seen at all.