Following POC provides a way to implement a caching strategy that serves custom content to Bots (e.g., Googlebot) while serving default content to Humans, without sacrificing performance. This kind of approach can be desirable, especially to news sites or sites with paywalled content, where they want bots to have access to the content for SEO reasons, but the paywall to still in place for logged-out users
Goal:
Bots: Serve a specific “Bot Version” of the page (cached).
Humans: Serve the standard “Default Version” of the page (cached).
Performance: Ensure the Cache HIT for both segments.
2. The Challenge: Why PHP Alone Wasn’t Enough
WP Engine uses Varnish as a caching layer that sits in front of the PHP application.
The “Chicken and Egg” Problem: By the time a request reaches PHP to check User-Agent, Varnish has already processed the request.
Cache Collision: If PHP detects a bot and serves “Bot Content,” Varnish might unknowingly cache that content under the “Default” bucket. This leads to Cache Poisoning, where human users accidentally see the Bot version.
No Varnish Access: Since WP Engine is Managed Host you cannot easily modify Varnish VCL, and making changes in Nginx is insufficient because Varnish sits on top of it.
3. The Solution: Cloudflare “Edge Injection”
We utilise Cloudflare Workers to intercept the request at the edge (before it reaches WP Engine).
Architecture Flow
Request: User/Bot visits the site.
Cloudflare Worker: Checks the User-Agent.
If Bot: Injects the wpe-us=bot cookie into the request headers transparently.
If Human: Passes the request through untouched.
WP Engine Varnish: Receives the request.
If Bot: Sees the injected cookie, translates it to X-WPENGINE-SEGMENT: bot, and serves from the Bot Cache Bucket.
If Human: Sees no cookie, serves from the Default Cache Bucket.
PHP Application: Receives the request (if uncached) with the correct X-WPENGINE-SEGMENT header already set.
Result: Zero PHP overhead for detection, cache compatibility, and zero risk of cache poisoning.
4. Implementation Details
A. Cloudflare Worker Script
Create a new Worker in Cloudflare and map it to example.com/*.
Worker Code:
export default {
async fetch(request, env, ctx) {
const ua = request.headers.get('User-Agent') || '';
// Detect Googlebot or our Custom Test Agent
// Case-insensitive regex match
// Customer able to modify the list to their desire list of bot User Agents
if (/Googlebot|Caching-Bot/i.test(ua)) {
// Clone the request to modify headers
const newRequest = new Request(request);
const currentCookies = newRequest.headers.get('Cookie') || '';
// Inject the 'wpe-us=bot' cookie
// If cookies exist, append it. If not, create it.
const newCookies = currentCookies ? `${currentCookies}; wpe-us=bot` : 'wpe-us=bot';
newRequest.headers.set('Cookie', newCookies);
// Forward modified request to Origin (WP Engine)
return fetch(newRequest);
}
// Pass human traffic through untouched
return fetch(request);
}
};
B. WordPress Shortcode (Testing Logic)
Add this to functions.php to create the [segmentation_test] shortcode. This PHP handles the Content Logic (what to show) and ensures the Vary header is set.
Open Chrome DevTools → cmd+shift+P → Network Conditions.
Uncheck “Select automatically” next to User agent.
Test Bot: Select “Custom” and paste Caching-Bot.
Result: Pink Banner (“BOT VERSION”).
Cache:HIT.
Test Human: Uncheck “Custom” (Default UA).
Result: Green Banner (“HUMAN VERSION”).
Cache:HIT.
How to Verify (Curl)
Run these commands in a terminal to verify without browser cache interference.
Bot Test (Expect “BOT VERSION”):
curl -A "Caching-Bot" https://your-site.com/test-page/ | grep "BOT VERSION"
Human Test (Expect “DEFAULT VERSION”):
curl -A "Mozilla/5.0" https://your-site.com/test-page/ | grep "DEFAULT VERSION"
FAQ: Why is the Cookie “None” in PHP?
You may notice that when testing the Bot, PHP reports the wpe-us cookie as None, even though the Bot logic is working.
Reason: The cookie is injected by Cloudflare at the edge. When it hits WP Engine, Varnish consumes (removes) the cookie to generate the X-WPENGINE-SEGMENT header.
Conclusion: This is expected behaviour. The Header is the source of truth for the application.
Found this useful?
There's more where that came from — explore the rest of the documentation, experiments, and production work.
