Wagtail Search & Site Navigation: Search Backends, Indexing, and Navigation Menus

Wagtail's pluggable search backend system lets you switch from the default database search to PostgreSQL full-text search or Elasticsearch with a single config change. Combined with Wagtail's tree-based navigation methods, you can build a site where users find content instantly through ranked, filterable search results and menus that update automatically when the page tree changes.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Configure Wagtail's search backends — Elasticsearch, PostgreSQL full-text search, and the default database search
• Define search_fields on Page models and snippets to control what gets indexed
• Build custom search views with faceted filtering by category, date, and tags
• Create dynamic site navigation using Wagtail’s built-in get_children(), get_siblings(), and custom menu patterns
• Implement autocomplete and live search with Wagtail’s search API
• Optimize search performance with index updates, query boosting, and field weighting

2. Why This Matters (Real-World Scenario)

Imagine you manage a documentation site with over 500 pages and 2,000 blog posts. Your users rely on search to find answers fast, but the default Django ‘icontains’ lookup is slow, returns irrelevant results, and cannot handle typos. Meanwhile, your navigation menu is a static list that requires manual updates every time you add a new section. Junior editors regularly forget to add new pages to the menu, leaving content hidden from visitors.

Wagtail Search solves the first problem: PostgreSQL full-text search handles fuzzy matching, ranking, and weighting out of the box, while Elasticsearch adds faceted filtering and blazing-fast performance at scale. For navigation, Wagtail’s page tree provides get_children(), get_siblings(), and get_ancestors() for dynamic menu generation, letting you build self-updating navigation that reflects the page tree automatically. This lesson walks you through both halves — search and navigation — from setup to production-grade implementation.

3. Core Concepts

3.1 Wagtail Search Backends

Wagtail abstracts search behind a pluggable backend system. Three backends are available:

• Database backend (default) — Uses Django’s icontains lookups. Zero configuration, works immediately, but scales poorly beyond a few hundred pages and offers no ranking or fuzzy matching.
• PostgreSQL backend (recommended for small-to-medium sites) — Uses PostgreSQL’s full-text search (tsvector/tsquery). Provides ranking, stemming, and language-aware search without external dependencies. Configure it in 5 minutes.
• Elasticsearch backend (recommended for large-scale sites) — Full-featured search engine with faceted filtering, autocomplete, proximity queries, and horizontal scaling. Requires running an Elasticsearch cluster.

You configure the active backend in Django settings:

WAGTAILSEARCH_BACKENDS = {
    'default': {
        'BACKEND': 'wagtail.search.backends.database',
    },
    'elasticsearch': {
        'BACKEND': 'wagtail.search.backends.elasticsearch8',
        'URLS': ['http://localhost:9200'],
        'INDEX': 'wagtail',
        'TIMEOUT': 5,
        'OPTIONS': {},
        'INDEX_SETTINGS': {},
    },
}

3.2 Search Indexing with search_fields

Wagtail determines what content gets indexed by reading the search_fields attribute on Page models and registered snippets. Each field gets a SearchField, AutocompleteField, or FilterField:

• SearchField — Main content field, included in the full-text search query. Supports boost for weighting.
• AutocompleteField — Indexed for prefix-matching autocomplete suggestions. Ideal for titles and short labels.
• FilterField — Not searched directly, but stored as a filterable facet (category, date, author).

from wagtail.search import index

class BlogPage(Page):
    introduction = RichTextField(blank=True)
    body = StreamField(BlogStreamBlock, use_json_field=True)
    category = models.ForeignKey('blog.Category', on_delete=models.SET_NULL, null=True)
    author = models.ForeignKey('auth.User', on_delete=models.SET_NULL, null=True)
    publish_date = models.DateField()

    search_fields = Page.search_fields + [  # Inherit title, etc. from Page
        index.SearchField('introduction', boost=2),
        index.SearchField('body', boost=0.5),
        index.AutocompleteField('title', boost=10),
        index.FilterField('category'),
        index.FilterField('author'),
        index.FilterField('publish_date'),
    ]

python manage.py update_index

3.3 Dynamic Navigation with the Page Tree

Wagtail pages are arranged in a tree structure (django-treebeard). Every page has access to tree methods that make navigation dynamic:

• page.get_children() — Direct child pages (for sub-navigation menus)
• page.get_siblings() — Sibling pages (for horizontal nav or breadcrumbs)
• page.get_ancestors() — All ancestor pages up to the root (for breadcrumb trails)
• Page.objects.live().in_menu() — Only pages with “Show in menus” checked (for main navigation)

By leveraging these methods in template tags, you create navigation that updates automatically when the page tree changes. No manual menu editing required.

4. Hands-On Practice: Building Search & Navigation

4.1 Enable PostgreSQL Full-Text Search

Switch from the default database backend to PostgreSQL full-text search for proper ranking and stemming:

# settings/base.py
WAGTAILSEARCH_BACKENDS = {
    'default': {
        'BACKEND': 'wagtail.search.backends.database_postgres',
        'SEARCH_CONFIG': 'english',  # Language-aware stemming
    },
}

Run the migration that enables the PostgreSQL full-text search functions, then rebuild the index:

python manage.py migrate wagtailsearch
python manage.py update_index

4.2 Adding search_fields to Your Models

Open your BlogPage model and add search configuration that boosts titles and headings above body content:

from wagtail.search import index

class BlogPage(Page):
    introduction = RichTextField(blank=True)
    body = StreamField(BlogStreamBlock, use_json_field=True)
    category = models.ForeignKey('blog.Category', on_delete=models.SET_NULL, null=True)
    tags = ClusterTaggableManager(through=BlogPageTag, blank=True)
    publish_date = models.DateField()

    search_fields = Page.search_fields + [
        # High boost for key fields
        index.AutocompleteField('introduction', boost=5),
        index.AutocompleteField('title', boost=10),
        index.SearchField('introduction', boost=5),
        index.SearchField('body', boost=2),
        # Filter fields for faceted search
        index.FilterField('category'),
        index.FilterField('publish_date'),
        index.FilterField('tags'),
        index.RelatedFields('category', [
            index.SearchField('name'),
        ]),
    ]
    # Also index snippet content via RelatedFields

4.3 Building a Custom Search View with Faceted Filtering

Create a dedicated search page that supports filtering by category and date range:

# blog/views.py
from wagtail.search.models import Query
from wagtail.search.backends import get_search_backend

def search_view(request):
    query = request.GET.get('q', '')
    category = request.GET.get('category', '')
    date_from = request.GET.get('date_from', '')
    date_to = request.GET.get('date_to', '')
    page = request.GET.get('page', 1)

    results = BlogPage.objects.live().specific()

    # Filter first (faster)
    if category:
        results = results.filter(category__slug=category)
    if date_from:
        results = results.filter(publish_date__gte=date_from)
    if date_to:
        results = results.filter(publish_date__lte=date_to)

    # Then search
    if query:
        s = get_search_backend()
        results = s.search(query, results, operator='and')
        Query.get(query).add_hit()  # Track popular searches

    # Paginate
    paginator = Paginator(results, 12)
    page_obj = paginator.get_page(page)

    context = {
        'query': query,
        'results': page_obj,
        'categories': Category.objects.all(),
        'selected_category': category,
        'date_from': date_from,
        'date_to': date_to,
    }
    return render(request, 'blog/search.html', context)

{% extends 'base.html' %}
{% load wagtailcore_tags %}

{% block content %}
<form method="get" action="{% url 'search' %}">
  <input type="text" name="q" value="{{ query }}" placeholder="Search...">
  <select name="category">
    <option value="">All Categories</option>
    {% for cat in categories %}
      <option value="{{ cat.slug }}" {% if selected_category == cat.slug %}selected{% endif %}>
        {{ cat.name }}
      </option>
    {% endfor %}
  </select>
  <input type="date" name="date_from" value="{{ date_from }}">
  <input type="date" name="date_to" value="{{ date_to }}">
  <button type="submit">Search</button>
</form>

{% if query %}
  <p>{{ results.paginator.count }} result(s) for "{{ query }}"</p>
{% endif %}

<ul>
{% for result in results %}
  <li>
    <a href="{% pageurl result %}">{{ result.title }}</a>
    <p>{{ result.introduction|truncatewords:30 }}</p>
    <small>{{ result.publish_date }}</small>
  </li>
{% endfor %}
</ul>
{% endblock %}

4.4 Adding Autocomplete with Wagtail's Search API

Wagtail provides an autocomplete method on the search backend for real-time suggestions. Hook it up to a REST endpoint for AJAX autocomplete:

# blog/api.py
from wagtail.search.backends import get_search_backend

def autocomplete_api(request):
    query = request.GET.get('q', '')
    if len(query) < 2:
        return JsonResponse([], safe=False)

    s = get_search_backend()
    # Use autocomplete method for prefix matching
    results = s.autocomplete(query, BlogPage.objects.live().specific())[:10]

    suggestions = [
        {
            'title': page.title,
            'url': page.url,
            'category': page.category.name if page.category else None,
        }
        for page in results
    ]
    return JsonResponse(suggestions, safe=False)

4.5 Building a Dynamic Main Navigation

Create a custom template tag that renders navigation from the page tree, respecting the ‘Show in menus’ flag:

# blog/templatetags/navigation_tags.py
from django import template
from wagtail.models import Page

register = template.Library()

@register.simple_tag(takes_context=True)
def main_navigation(context):
    """Return top-level pages marked for menu display."""
    root = Page.objects.get(slug='root')
    # Only pages with show_in_menus=True
    menu_pages = Page.objects.live().in_menu().filter(depth=3)  # Depth 3 = immediate children of home
    return menu_pages

@register.simple_tag(takes_context=True)
def breadcrumbs(context):
    """Return ancestor trail for the current page."""
    page = context.get('self') or context.get('page')
    if not page:
        return []
    ancestors = page.get_ancestors().filter(depth__gte=2)  # Skip root
    return ancestors

@register.simple_tag(takes_context=True)
def child_navigation(context):
    """Return child pages of the current section."""
    page = context.get('self') or context.get('page')
    if not page or not page.get_children_count():
        return []
    return page.get_children().live().in_menu()

{% load wagtailcore_tags navigation_tags %}

<nav class="main-nav">
  <ul>
    {% main_navigation as nav_items %}
    {% for item in nav_items %}
      <li class="nav-item">
        <a href="{% pageurl item %}">{{ item.title }}</a>
        {% if item.get_children_count %}
          {% child_navigation item as children %}
          <ul class="sub-nav">
            {% for child in children %}
              <li><a href="{% pageurl child %}">{{ child.title }}</a></li>
            {% endfor %}
          </ul>
        {% endif %}
      </li>
    {% endfor %}
  </ul>
</nav>

<nav class="breadcrumbs">
  {% breadcrumbs as trail %}
  {% for page in trail %}
    <a href="{% pageurl page %}">{{ page.title }}</a>
    {% if not forloop.last %}&rsaquo;{% endif %}
  {% endfor %}
</nav>

4.6 Configuring Elasticsearch for Production

When your site grows beyond a few thousand pages, switch to Elasticsearch for better performance, faceted aggregations, and advanced query features:

# Install Wagtail’s Elasticsearch backend
docker run -d -p 9200:9200 -e 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:8.12.0
pip install wagtail[search]
# or: pip install elasticsearch

# Python: configure settings (already shown in 3.1)
# Then rebuild the index:
python manage.py update_index

# Advanced: custom index settings for better relevance
WAGTAILSEARCH_BACKENDS = {
    'default': {
        'BACKEND': 'wagtail.search.backends.elasticsearch8',
        'URLS': ['http://localhost:9200'],
        'INDEX': 'wagtail_prod',
        'TIMEOUT': 10,
        'INDEX_SETTINGS': {
            'settings': {
                'index': {
                    'number_of_shards': 1,
                    'number_of_replicas': 1,
                },
                'analysis': {
                    'analyzer': {
                        'default': {
                            'type': 'standard',
                            'stopwords': '_english_',
                        }
                    }
                }
            }
        }
    }
}

4.7 Scheduled Index Updates with Cron

For production sites, run update_index periodically to keep search results fresh. Use a cron job or Celery beat for this:

# Run every 10 minutes to capture new/modified pages
*/10 * * * * cd /path/to/project && python manage.py update_index --age 1440

# Full rebuild daily (handles deletions and reindexes everything)
0 3 * * * cd /path/to/project && python manage.py update_index

5. Common Errors & Solutions

5.1 update_index Fails with ‘relation does not exist’

django.db.utils.ProgrammingError: relation "wagtailsearch_indexentry" does not exist

Cause: The wagtailsearch migration has not been applied. The search index table is created by Wagtail’s own migration, not by your app’s migrations.

Solution: Run python manage.py migrate wagtailsearch to create the required tables, then rerun update_index.

5.2 PostgreSQL Backend Returns No Results

wagtail.search.backends.database_postgres.SearchException: Full-text search requires PostgreSQL 10+ with the pg_trgm extension

Cause: The PostgreSQL backend requires the pg_trgm extension for trigram-based fuzzy matching.

Solution: Run CREATE EXTENSION IF NOT EXISTS pg_trgm; in your database, then run python manage.py migrate wagtailsearch and python manage.py update_index.

5.3 Autocomplete Returns Nothing

Cause: The autocomplete method only searches fields that have AutocompleteField in search_fields. If you only defined SearchField, autocomplete will return zero results.

Solution: Add index.AutocompleteField('title', boost=10) and index.AutocompleteField('introduction', boost=5) to your model’s search_fields definition. Rebuild the index with python manage.py update_index and test again.

5.4 Navigation Template Shows No Pages

Cause: The in_menu() query method only returns pages with “Show in menus” checked in the Wagtail admin. If no pages have this flag set, main_navigation returns an empty queryset.

Solution: Edit each page in the Wagtail admin, go to the “Promote” tab, and check “Show in menus”. Alternatively, use Page.objects.live().filter(depth=3) without in_menu() to bypass the flag during development.

5.5 Elasticsearch Connection Refused

elastic_transport.ConnectionError: Connection error caused by: ConnectionRefusedError(111, 'Connection refused')

Cause: The Elasticsearch server is not running or the URL in WAGTAILSEARCH_BACKENDS is incorrect.

Solution: Verify Elasticsearch is running: curl http://localhost:9200. Check the URL in settings — if using Docker, the host may be http://elasticsearch:9200 instead of localhost. Confirm the port mapping matches your Docker run command.

6. Summary Checklist

• Choose a search backend: database (default), PostgreSQL (recommended), or Elasticsearch (production scale)
• Add WAGTAILSEARCH_BACKENDS to Django settings with the correct backend path
• Define search_fields on every model that should appear in search results
• Use SearchField for full-text content, AutocompleteField for suggestions, FilterField for faceted filtering
• Apply boost values to prioritize title and introduction over body content
• Run python manage.py update_index after any search_fields change
• Build faceted search views that filter before searching for performance
• Create dynamic navigation using get_children(), get_ancestors(), and in_menu()
• Set up a cron job for periodic index updates in production
• Test autocomplete by adding AutocompleteField to your models

7. Practice Exercise

Challenge: Build a Tag-Based Search with Navigation Highlighting

Extend the blog from this lesson:

1. Add a tags FilterField to your BlogPage search_fields definition
2. Create a search view that accepts a ?tag= query parameter and filters results by tag slug
3. Add a “Popular Tags” sidebar that lists all tags with their article counts (use BlogPageTag.objects.values('tag__name').annotate(count=Count('tag')))
4. When the user clicks a tag, highlight the active tag in the sidebar using a CSS class
5. Update the navigation so the current section’s menu item gets a CSS class of active

Test your implementation by searching for an article by tag and verifying the sidebar highlights the active tag. Check that the navigation item for the current page shows an “active” indicator.

8. Next Steps

Congratulations! You have built a production-ready search and navigation system for your Wagtail site. Your users can now find content quickly with ranked search results, faceted filtering, and autocomplete suggestions. Your site’s navigation updates automatically as the page tree changes, eliminating manual menu maintenance.

The next topic in the Wagtail CMS series is Wagtail Deployment & Production Best Practices: Hosting, Static Files, Media Storage, and Performance Tuning. You will learn how to deploy Wagtail to production on platforms like Railway.app or Fly.io, configure S3-compatible media storage, serve static files through a CDN, set up Gunicorn + Nginx, and tune performance with caching headers and database connection pooling.

Further resources:

• Wagtail Search Documentation — official reference for backends, indexing, and custom search views
• Wagtail’s Built-in Search Views — how to use Wagtail’s default search template tags
• Wagtail Page Tree Methods — reference for get_children(), get_siblings(), and tree traversal
• Elasticsearch Python Client Docs — advanced query building and aggregation
• Wagtail Performance Tuning Guide — caching, database optimization, and CDN setup

Common pitfalls to avoid:

• Don’t forget to add AutocompleteField separately from SearchField — they serve different purposes and are not interchangeable
• Avoid running update_index during peak traffic without a lock mechanism — schedule it during off-peak hours or use a transactionally safe approach
• Never expose the raw search API without rate limiting or authentication to prevent abuse
• Don’t use the database backend with more than 1,000 pages — switch to PostgreSQL or Elasticsearch for acceptable performance