Content Prefetch

速盾网络 Team

Content prefetching proactively loads content into edge caches before users request it. This ensures cache hits from the first request, eliminating cold-start latency and improving user experience.

How Prefetching Works

code

Traditional Flow:
User Request → Cache MISS → Origin Fetch → Cache Store → Response

Prefetch Flow:
Prefetch Job → Origin Fetch → Cache Store (ready)
User Request → Cache HIT → Immediate Response

Benefits of Prefetching

Benefit	Description
Eliminate cold starts	First visitors get cached content
Faster page loads	No origin round-trip delay
Predictable performance	Consistent response times
Origin protection	Spread load over time

Prefetch Methods

URL List Prefetch

Prefetch specific URLs:

json

{
  "urls": [
    "https://example.com/",
    "https://example.com/products",
    "https://example.com/about",
    "https://example.com/css/styles.css",
    "https://example.com/js/app.js"
  ]
}

Best for: Known critical pages, landing pages, marketing campaigns

Sitemap Prefetch

Automatically prefetch URLs from your sitemap:

code

Sitemap URL: https://example.com/sitemap.xml

Sudun parses your sitemap and prefetches all listed URLs:

xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page1</loc>
    <lastmod>2024-01-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/page2</loc>
  </url>
</urlset>

Best for: Full site warming, SEO-critical pages

Crawl Prefetch

Automatically discover and prefetch linked content:

json

{
  "start_url": "https://example.com/",
  "depth": 2,
  "max_urls": 100
}

The crawler follows links to discover content:

code

Depth 0: Homepage
Depth 1: Pages linked from homepage
Depth 2: Pages linked from depth 1 pages

Best for: Dynamic sites, discovering new content

Configuring Prefetch

Dashboard Setup

Go to Domains → Select your domain
Navigate to Caching → Prefetch
Click Create Prefetch Job
Configure:

- Source: URL List, Sitemap, or Crawl

- Schedule: One-time or recurring

- Options: Headers, regions, concurrency

Prefetch Options

Option	Description	Default
Regions	PoPs to prefetch into	All regions
Concurrency	Parallel requests to origin	5
Headers	Custom headers for prefetch requests	None
Ignore Robots	Skip robots.txt restrictions	No

Region-Specific Prefetch

Prefetch to specific geographic regions:

json

{
  "urls": ["https://example.com/"],
  "regions": ["us-east", "eu-west", "asia-pacific"]
}

Available Regions:

Region ID	Location
us-east	North America East
us-west	North America West
eu-west	Europe West
eu-central	Europe Central
asia-pacific	Asia Pacific
asia-south	South Asia

Scheduling Prefetch

One-Time Prefetch

Run prefetch immediately or at a scheduled time:

json

{
  "schedule": {
    "type": "once",
    "run_at": "2024-01-15T06:00:00Z"
  }
}

Recurring Prefetch

Set up automatic recurring prefetch:

json

{
  "schedule": {
    "type": "recurring",
    "interval": "daily",
    "time": "04:00",
    "timezone": "UTC"
  }
}

Schedule Options:

Interval	Description	Use Case
hourly	Every hour	Frequently updated content
daily	Once per day	Standard websites
weekly	Once per week	Static content
custom	Cron expression	Complex schedules

Cron Expression

For advanced scheduling:

json

{
  "schedule": {
    "type": "cron",
    "expression": "0 */6 * * *"
  }
}

Common patterns:

0 4 * * * - Daily at 4 AM
0 */2 * * * - Every 2 hours
0 0 * * 0 - Weekly on Sunday

Custom Request Headers

Send specific headers during prefetch:

json

{
  "headers": {
    "Accept-Language": "en-US",
    "Accept-Encoding": "gzip, br",
    "X-Prefetch": "true"
  }
}

User-Agent

Configure the prefetch user agent:

json

{
  "user_agent": "Sudun-Prefetch/1.0"
}

Tip: Your origin can detect prefetch requests via the User-Agent or custom header and log them separately.

Prefetch Triggers

Deployment Trigger

Automatically prefetch after deployment:

yaml

# CI/CD integration
- name: Deploy
  run: deploy-script.sh

- name: Warm Cache
  run: |
    curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
      -H "Authorization: Bearer $API_KEY" \
      -d '{"sitemap": "https://example.com/sitemap.xml"}'

Webhook Trigger

Trigger prefetch via webhook:

bash

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch/trigger \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"job_id": "prefetch-job-123"}'

Content Update Trigger

Prefetch when content changes:

json

{
  "trigger": {
    "type": "purge",
    "action": "prefetch_after_purge"
  }
}

Monitoring Prefetch

Job Status

Track prefetch job progress:

Status	Description
Pending	Job queued
Running	Prefetch in progress
Completed	All URLs prefetched
Failed	Job failed (see errors)
Partial	Some URLs failed

Prefetch Metrics

View metrics in the dashboard:

URLs Processed: Total URLs prefetched
Success Rate: Percentage of successful prefetches
Average Latency: Origin response time during prefetch
Cache Fill: Amount of content cached

Prefetch Logs

json

{
  "job_id": "pf-abc123",
  "status": "completed",
  "started_at": "2024-01-15T04:00:00Z",
  "completed_at": "2024-01-15T04:05:32Z",
  "stats": {
    "total_urls": 150,
    "successful": 148,
    "failed": 2,
    "skipped": 0
  }
}

Best Practices

Prioritize Critical Content

Prefetch most important pages first:

json

{
  "urls": [
    {"url": "https://example.com/", "priority": "high"},
    {"url": "https://example.com/products", "priority": "high"},
    {"url": "https://example.com/blog", "priority": "medium"},
    {"url": "https://example.com/about", "priority": "low"}
  ]
}

Respect Origin Capacity

Configure concurrency to avoid overloading origin:

json

{
  "concurrency": 3,
  "rate_limit": "10/second"
}

Schedule During Off-Peak

Run prefetch when traffic is lowest:

code

Peak Hours: 9 AM - 6 PM
Prefetch Window: 2 AM - 5 AM

Exclude Dynamic Content

Skip URLs that shouldn't be prefetched:

json

{
  "exclude_patterns": [
    "/api/*",
    "/admin/*",
    "*.json",
    "*?session=*"
  ]
}

Troubleshooting

Prefetch Not Working

Check origin accessibility: Ensure origin responds to prefetch requests
Verify URL format: URLs must be fully qualified (https://...)
Check robots.txt: Enable "Ignore Robots" if blocked
Review error logs: Check for specific URL failures

High Origin Load

If prefetch overloads your origin:

Reduce concurrency setting
Add rate limiting
Schedule during off-peak hours
Use incremental prefetch

Cache Not Warming

If content isn't cached after prefetch:

Verify cache rules allow caching
Check origin response headers
Ensure TTL isn't too short
Confirm prefetch completed successfully

API Reference

Create Prefetch Job

bash

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/",
      "https://example.com/products"
    ],
    "regions": ["us-east", "eu-west"],
    "concurrency": 5
  }'

Prefetch from Sitemap

bash

curl -X POST https://api.Sudun.com/v1/domains/example.com/prefetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "sitemap": "https://example.com/sitemap.xml",
    "schedule": {
      "type": "daily",
      "time": "04:00"
    }
  }'

Get Prefetch Status

bash

curl -X GET https://api.Sudun.com/v1/domains/example.com/prefetch/job-123 \
  -H "Authorization: Bearer YOUR_API_KEY"

Need help with content prefetching? Contact support@Sudun.com

Content Prefetch

How Prefetching Works

Benefits of Prefetching

Prefetch Methods

URL List Prefetch

Sitemap Prefetch

Crawl Prefetch

Configuring Prefetch

Dashboard Setup

Prefetch Options

Region-Specific Prefetch

Scheduling Prefetch

One-Time Prefetch

Recurring Prefetch

Cron Expression

Custom Request Headers

User-Agent

Prefetch Triggers

Deployment Trigger

Webhook Trigger

Content Update Trigger

Monitoring Prefetch

Job Status

Prefetch Metrics

Prefetch Logs

Best Practices

Prioritize Critical Content

Respect Origin Capacity

Schedule During Off-Peak

Exclude Dynamic Content

Troubleshooting

Prefetch Not Working

High Origin Load

Cache Not Warming

API Reference

Create Prefetch Job

Prefetch from Sitemap

Get Prefetch Status

Tags

How Prefetching Works

Benefits of Prefetching

Prefetch Methods

URL List Prefetch

Sitemap Prefetch

Crawl Prefetch

Configuring Prefetch

Dashboard Setup

Prefetch Options

Region-Specific Prefetch

Scheduling Prefetch

One-Time Prefetch

Recurring Prefetch

Cron Expression

Custom Request Headers

User-Agent

Prefetch Triggers

Deployment Trigger

Webhook Trigger

Content Update Trigger

Monitoring Prefetch

Job Status

Prefetch Metrics

Prefetch Logs

Best Practices

Prioritize Critical Content

Respect Origin Capacity

Schedule During Off-Peak

Exclude Dynamic Content

Troubleshooting

Prefetch Not Working

High Origin Load

Cache Not Warming

API Reference

Create Prefetch Job

Prefetch from Sitemap

Get Prefetch Status