Cache Repeat Workers AI Requests with AI Gateway

Introduction

An AI model normally generates a new response each time an application calls it. That work takes time and consumes model usage even when the request is exactly the same as one the model just answered. A cache keeps a reusable response for a limited time so an identical request can be answered without another model call.

Caching is useful only when reuse is safe. A public, fixed FAQ question is a good candidate because every caller may receive the same answer. A personalized support prompt is not: two customers must never be grouped under a shared cache key merely to improve speed. AI Gateway's default cache key protects this lab by including the provider, endpoint, model, provider credential and full request body. Any body change creates a different entry.

You will create one disposable authenticated gateway with a five-minute cache lifetime. You will send a small public Workers AI question and observe a cache MISS, repeat the exact request and prove a HIT, then change the question and see another MISS. Finally, you will bypass the existing cached answer when freshness matters and confirm through gateway logs that the request reached the model.

If you entered this course directly, first complete Connect LabEx to Your Cloudflare Account. It teaches the LabEx VM terminal, Wrangler device authorization, learning-account confirmation and explicit account IDs. Complete Route Inference Through a Gateway first as well, because this lab reuses its separate gateway and upstream authorization boundaries.

The lab uses the Cloudflare-hosted @cf/meta/llama-3.3-70b-instruct-fp8-fast model with Standard Workers AI billing. Workers Paid, Unified Billing and an external provider account are not required. Only three requests should reach the model; the exact repeat should come from cache. Stop rather than repeatedly retrying if the shared daily Workers AI allocation is unavailable.

Setup installs Node.js 22.22.0 and project-local Wrangler 4.132.0 in /home/labex/project/ai-gateway-cache. It prepares independent read-only assessments, but it does not authorize Wrangler, create cloud resources or send model traffic. LabEx destroys the temporary VM when the lab ends; you will still delete the gateway and token before logout because VM destruction alone cannot remove cloud resources.

Authorize the VM and Name the Cache Experiment

In this step, you will connect the fresh VM to your learning account and generate names for one disposable gateway and token.

A cache is shared infrastructure, so its scope must be deliberate. This lab uses one uniquely named gateway and only synthetic public questions. The random suffix prevents your experiment from colliding with another gateway in the same learning account.

Enter the prepared project, confirm the pinned CLI and authorize this VM:

cd /home/labex/project/ai-gateway-cache
npx wrangler --version
npx wrangler login --device --browser=false --scopes account:read user:read ai:write

Open the displayed link, enter the code and authorize the intended learning account. Confirm structured identity:

npx wrangler whoami --json

Expect Wrangler 4.132.0 and loggedIn: true. Replace YOUR_ACCOUNT_ID below with the actual 32-character ID shown for the intended account:

GATEWAY_ID="labex-c09-g03-$(openssl rand -hex 6)"
TOKEN_NAME="$GATEWAY_ID-token"
cat > .labex/state.json <<JSON
{
  "accountId": "YOUR_ACCOUNT_ID",
  "gatewayId": "$GATEWAY_ID",
  "tokenName": "$TOKEN_NAME"
}
JSON
cat .labex/state.json

These nonsecret identifiers stay in a local inventory so every later read, verification and cleanup targets only this lab's resources.

Create an Authenticated Gateway with a Short Cache

In this step, you will create the gateway and give cached answers a five-minute time to live, or TTL. A TTL is the maximum time an entry may be reused before it becomes stale and must be refreshed from the model.

Open the Cloudflare Dashboard and choose AI → AI Gateway → Create gateway → Custom gateway. Use the saved gatewayId as the gateway name. Keep request logging and gateway authentication on, enable Cache responses, and set its TTL to exactly 300 seconds. Keep rate limits, spend limits and retries off, and keep Workers AI billing on Standard.

The gateway settings show authentication, logging and a 300-second response cache

After creation, confirm the unique gateway ID in the breadcrumb. The short TTL is long enough to repeat this experiment but prevents the example response from lingering unnecessarily.

Choose Create an AI Gateway authentication token. Use the saved tokenName, include only the intended learning account and set exactly these permissions:

AI Gateway — Run for entering the authenticated gateway;
AI Gateway — Edit for reading cache evidence and deleting this disposable gateway.

Do not add Workers AI permission. Wrangler provides the separate short-lived upstream credential. Create the token after reviewing the account and permissions, then store its one-time value without echoing it:

bash -c '
while :; do
  read -ersp "Paste the AI Gateway token: " GATEWAY_TOKEN
  printf "\n"
  [ -n "$GATEWAY_TOKEN" ] && break
  printf "Token cannot be empty; paste it again.\n" >&2
done
umask 077
printf "%s" "$GATEWAY_TOKEN" > .labex/gateway-token
unset GATEWAY_TOKEN
chmod 600 .labex/gateway-token
'

Verify the exact cache configuration through the authenticated management API:

ACCOUNT_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).accountId')
GATEWAY_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).gatewayId')
GATEWAY_TOKEN=$(cat .labex/gateway-token)
curl --http1.1 -fsS \
  -H "Authorization: Bearer $GATEWAY_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID" \
  | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{const b=JSON.parse(s),g=b.result||{};console.log(JSON.stringify({success:b.success,id:g.id,collect_logs:g.collect_logs,authentication:g.authentication,cache_ttl:g.cache_ttl},null,2))})'
unset GATEWAY_TOKEN

Expect the saved ID, collect_logs: true, authentication: true and cache_ttl: 300.

Send the First Public FAQ Request

In this step, you will send a small public question that is safe for every learner to reuse. The first eligible request cannot already have an entry under this new gateway, so it should be a cache MISS. A miss means AI Gateway forwards the request to Workers AI and then stores the successful response.

The default cache key includes the upstream provider credential. Save this VM's current Wrangler credential privately so all four requests use one controlled key. This is a short-lived lab file, not a production secret pattern:

umask 077
npx wrangler auth token --json \
  | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>process.stdout.write(JSON.parse(s).token))' \
  > .labex/upstream-token
chmod 600 .labex/upstream-token

Send the first request and save the response headers, body and HTTP status without printing either credential:

ACCOUNT_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).accountId')
GATEWAY_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).gatewayId')
MODEL='@cf/meta/llama-3.3-70b-instruct-fp8-fast'
METADATA='{"lab":"g03-cache","case":"public-faq","synthetic":true}'
GATEWAY_TOKEN=$(cat .labex/gateway-token)
UPSTREAM_TOKEN=$(cat .labex/upstream-token)
STATUS=$(curl --http1.1 -sS -D .labex/first-headers.txt \
  -o .labex/first-response.json -w '%{http_code}' \
  -H "cf-aig-authorization: Bearer $GATEWAY_TOKEN" \
  -H "Authorization: Bearer $UPSTREAM_TOKEN" \
  -H "cf-aig-metadata: $METADATA" \
  -H 'Content-Type: application/json' \
  --data '{"prompt":"In one short sentence, what does an AI gateway do?","max_tokens":32}' \
  "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID/workers-ai/$MODEL")
unset GATEWAY_TOKEN UPSTREAM_TOKEN METADATA
printf '%s\n' "$STATUS" | tee .labex/first-status.txt
awk 'BEGIN{IGNORECASE=1} /^cf-aig-cache-status:/ {gsub("\r","",$2); print toupper($2)}' .labex/first-headers.txt \
  | tail -1 | tee .labex/first-cache-status.txt
node -e 'const b=require("./.labex/first-response.json"); console.log(b.result?.response ?? b.result)'

Expect HTTP 200, cache status MISS and a short generated answer. The request body contains no customer data, so temporarily reusing this answer is safe.

Repeat the Exact Request and Prove a Cache Hit

In this step, you will send exactly the same provider, endpoint, model, credential and request body. AI Gateway can therefore reuse the entry created in the prior step. A cache HIT means the answer came from the gateway cache without a new model generation.

Cache storage is asynchronous, so give the successful first response a few seconds to settle before repeating it:

sleep 5
GATEWAY_TOKEN=$(cat .labex/gateway-token)
UPSTREAM_TOKEN=$(cat .labex/upstream-token)
METADATA='{"lab":"g03-cache","case":"public-faq","synthetic":true}'
STATUS=$(curl --http1.1 -sS -D .labex/repeat-headers.txt \
  -o .labex/repeat-response.json -w '%{http_code}' \
  -H "cf-aig-authorization: Bearer $GATEWAY_TOKEN" \
  -H "Authorization: Bearer $UPSTREAM_TOKEN" \
  -H "cf-aig-metadata: $METADATA" \
  -H 'Content-Type: application/json' \
  --data '{"prompt":"In one short sentence, what does an AI gateway do?","max_tokens":32}' \
  "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID/workers-ai/$MODEL")
unset GATEWAY_TOKEN UPSTREAM_TOKEN METADATA
printf '%s\n' "$STATUS" | tee .labex/repeat-status.txt
awk 'BEGIN{IGNORECASE=1} /^cf-aig-cache-status:/ {gsub("\r","",$2); print toupper($2)}' .labex/repeat-headers.txt \
  | tail -1 | tee .labex/repeat-cache-status.txt
cmp -s .labex/first-response.json .labex/repeat-response.json \
  && echo 'response bytes match the cached source'

Expect HTTP 200 and HIT; matching bytes are a helpful extra observation, while the HIT response header and cached Dashboard log are the authoritative evidence. AI Gateway cache storage is asynchronous and volatile, so do not send the two requests simultaneously. If the sequential repeat is still a miss, wait a few seconds and rerun this exact block once.

Open the gateway's Logs view in the Dashboard. Find the two public-faq requests and compare their cache indicators, durations and token usage. One row should show the model-backed miss and the other the cached hit.

Gateway logs show the first public FAQ miss beside the exact-repeat hit

Change the Question and Observe a New Miss

In this step, you will change only the prompt. The full request body participates in the default cache key, so this new question must not receive the previous answer.

GATEWAY_TOKEN=$(cat .labex/gateway-token)
UPSTREAM_TOKEN=$(cat .labex/upstream-token)
METADATA='{"lab":"g03-cache","case":"changed-question","synthetic":true}'
STATUS=$(curl --http1.1 -sS -D .labex/changed-headers.txt \
  -o .labex/changed-response.json -w '%{http_code}' \
  -H "cf-aig-authorization: Bearer $GATEWAY_TOKEN" \
  -H "Authorization: Bearer $UPSTREAM_TOKEN" \
  -H "cf-aig-metadata: $METADATA" \
  -H 'Content-Type: application/json' \
  --data '{"prompt":"In one short sentence, name one benefit of an AI gateway.","max_tokens":32}' \
  "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID/workers-ai/$MODEL")
unset GATEWAY_TOKEN UPSTREAM_TOKEN METADATA
printf '%s\n' "$STATUS" | tee .labex/changed-status.txt
awk 'BEGIN{IGNORECASE=1} /^cf-aig-cache-status:/ {gsub("\r","",$2); print toupper($2)}' .labex/changed-headers.txt \
  | tail -1 | tee .labex/changed-cache-status.txt
node -e 'const b=require("./.labex/changed-response.json"); console.log(b.result?.response ?? b.result)'

Expect HTTP 200 and MISS. This exact-match behavior is deliberately narrower than semantic similarity: two questions that sound related still have different bodies and different cache entries.

Do not replace the default key with one shared key such as support-answer for personalized prompts. A custom key is safe only when every request grouped under that key is authorized to receive the same response.

Bypass the Cache When Freshness Matters

In this step, you will return to the original question but explicitly skip its cached answer. Bypass means “ask the provider now,” even if a valid cache entry exists. This is useful when an application needs fresh output for a particular request.

The cf-aig-skip-cache: true header controls only this request. It does not disable the gateway's cache for other callers:

GATEWAY_TOKEN=$(cat .labex/gateway-token)
UPSTREAM_TOKEN=$(cat .labex/upstream-token)
METADATA='{"lab":"g03-cache","case":"fresh-bypass","synthetic":true}'
STATUS=$(curl --http1.1 -sS -D .labex/bypass-headers.txt \
  -o .labex/bypass-response.json -w '%{http_code}' \
  -H "cf-aig-authorization: Bearer $GATEWAY_TOKEN" \
  -H "Authorization: Bearer $UPSTREAM_TOKEN" \
  -H "cf-aig-metadata: $METADATA" \
  -H 'cf-aig-skip-cache: true' \
  -H 'Content-Type: application/json' \
  --data '{"prompt":"In one short sentence, what does an AI gateway do?","max_tokens":32}' \
  "https://gateway.ai.cloudflare.com/v1/$ACCOUNT_ID/$GATEWAY_ID/workers-ai/$MODEL")
unset GATEWAY_TOKEN UPSTREAM_TOKEN METADATA
printf '%s\n' "$STATUS" | tee .labex/bypass-status.txt
awk 'BEGIN{IGNORECASE=1} /^cf-aig-cache-status:/ {gsub("\r","",$2); print toupper($2)}' .labex/bypass-headers.txt \
  | tail -1 | tee .labex/bypass-cache-status.txt
node -e 'const b=require("./.labex/bypass-response.json"); console.log(b.result?.response ?? b.result)'

Expect HTTP 200 and no HIT. Depending on the current gateway response, the header may describe a bypass or simply remain non-hit; the authoritative gateway log must show cached: false for fresh-bypass.

Return to Logs in the Dashboard and open the fresh-bypass request. Compare it with the cached public-faq row. The same question reached Workers AI because the per-request bypass overrode the gateway default.

The fresh-bypass detail shows model duration and token usage for the original question

Delete the Disposable Gateway

In this step, you will remove the gateway while the management credential can still prove its absence. Deleting this owned gateway also removes its short-lived cache namespace and logs.

ACCOUNT_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).accountId')
GATEWAY_ID=$(node -p 'JSON.parse(require("fs").readFileSync(".labex/state.json")).gatewayId')
GATEWAY_TOKEN=$(cat .labex/gateway-token)
curl --http1.1 -fsS -X DELETE \
  -H "Authorization: Bearer $GATEWAY_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID" \
  | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{const b=JSON.parse(s);if(!b.success)process.exit(1);console.log("gateway deletion accepted")})'
unset GATEWAY_TOKEN

GATEWAY_TOKEN=$(cat .labex/gateway-token)
curl --http1.1 -fsS \
  -H "Authorization: Bearer $GATEWAY_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways" \
  > .labex/gateways-after-delete.json
unset GATEWAY_TOKEN
node -e 'const b=require("./.labex/gateways-after-delete.json"),id=process.argv[1],found=(b.result||[]).some(g=>g.id===id);console.log("gateway absent:",!found);if(found)process.exit(1)' "$GATEWAY_ID"

Expect gateway absent: true. This authenticated inventory distinguishes real deletion from a missing page caused by logout or a network failure.

Delete the Token and Log Out

In this step, you will revoke the remaining cloud credential, erase both temporary token copies and disconnect the VM.

In the Cloudflare Dashboard, open My Profile → API Tokens. Find the exact saved tokenName, open Actions, choose Delete, inspect the confirmation and delete only that token. It is safe to revoke now because the gateway deletion is already proved.

Erase the gateway and upstream token files, then end Wrangler's separate authorization:

shred -u .labex/gateway-token .labex/upstream-token
npx wrangler logout
npx wrangler whoami --json || true
test ! -e .labex/gateway-token -a ! -e .labex/upstream-token \
  && echo "local token files removed"

Expect loggedIn: false and local token files removed. The Dashboard session is separate and remains signed in. When the lab ends, LabEx destroys this temporary VM instead of saving it.

Summary

You configured a short AI Gateway response cache for one safe public question. The first request produced a MISS, the exact repeat became a HIT, and changed input created a separate entry. You then used a per-request bypass when freshness mattered and confirmed through logs that the model—not the cached copy—handled it.

The next lab adds traffic controls. You will learn the difference between limiting how often requests arrive and limiting how much model usage a gateway may spend, while keeping the test volume and cost deliberately small.

Cache Repeat Inference Requests