A 50-query benchmark of what actually maps to managed services — and what you’re giving up when it does.

This post is a benchmark of an AWS Bedrock migration — 50 queries, four swaps, five gaps.

TLDR

• I expected AWS to win on retrieval. It went 0 for 5 on the content type that matters most to this series
• Four swaps across the stack. One clean win, two with caveats, one that caught me off guard
• AWS is 2.7× cheaper — at the token layer. The real bill is 5–10× higher than what the audit log shows
• The most marketed feature (Bedrock Knowledge Bases) caused the most friction — RAG chunking is an architecture decision AWS makes for you silently at setup time
• HIERARCHICAL chunking is the Pareto winner: 48% HIGH confidence vs 38% with the SEMANTIC 500 default, at 2× the token cost
• Bigger chunks trigger more output blocks from the grounding guardrail. This shows up as task completion rate, not tokens
• Five gaps AWS still doesn’t close, even after you pay for it
• All numbers from a 50-query stress test: same model (Claude Haiku 4.5), same four docs, same prompt. Code linked at the bottom

Why I built this

Post 1, Post 2, and Post 3 built the full Policy Pal pipeline locally — Anthropic SDK, regex PII scrubber, keyword retrieval, CSV audit log — for roughly $0 in infrastructure.

After Post 3, I had one question: what does it cost to replace each handbuilt layer with the AWS managed equivalent? Not “should I use AWS” — that’s a product decision. But “what do I actually get, what do I lose, and where does the abstraction break down?”

This is a mapping exercise. Four local patterns, four AWS replacements, 50 queries of empirical data. What transfers cleanly, what transfers with caveats, what doesn’t transfer at all.

Dev Note

The stress test harness originally lived in /tmp — not committed, not reproducible. That was sloppy for a post that asks you to trust the numbers. It’s committed now in policy-pal-aws/scripts/stress_test.py. Reproducibility matters more than looking like you had it together from the start.

The four swaps

Swap 1: CSV audit → DynamoDB + CloudWatch ✅

The local version uses a CSV file with a threading.Lock. It works for 200 queries. It breaks under concurrency, can’t be queried by user or document, and has no alerting — three real gaps for a multi-user enterprise tool.

DynamoDB gives you queryable audit rows with GSIs by user, by source document, by confidence level, by date range. CloudWatch gives you escalation rate alarms and a cost dashboard. In the 50-query run, every audit row persisted correctly to both backends — 50/50 each. Reliability is a wash. The difference is queryability and observability at scale.

The migration is mechanical. The same PolicyResponse dataclass field names carry over exactly — same ScrubResult, same PolicyChunk. The pipeline structure is identical; only the persistence layer changed.

# local: policy_engine.write_audit_log() — appends CSV row with threading.Lock
# aws:   audit.py — one DynamoDB put_item + four CloudWatch metric puts
# same PolicyResponse fields, same interface, different backend

Of the four swaps, this is the one I’d do first. No architectural surprises, and you gain something the local version genuinely can’t match at scale.

Swap 2: Regex PII scrubber → Amazon Comprehend ⚠️

The local version is 47 lines of regex — six hard-coded patterns, English only, boolean output. Comprehend gives you a maintained entity catalogue, per-entity confidence scores, and Spanish support.

# local pii_scrubber.py  — 47 lines of regex patterns
# aws pii_scrubber.py    — ~12 effective lines calling detect_pii_entities()
# same ScrubResult dataclass, different engine

In the 50-query run, Comprehend caught the one Spanish PII query that local missed — 1/1 vs 0/1. The Comprehend es model caught an entity the regex had no pattern for.

The trap: detect_pii_entities ships English and Spanish models only. Comprehend’s broader multilingual support covers sentiment, entity detection, and language classification — but PII is a hard [en, es] enum. Pass it French input and it silently falls back to English, which catches universal formats (email, SSN, credit card) but misses locale-specific names and addresses.

In the run, a French query had its email caught — universal format, the en fallback handles it fine. A French name or street address would have passed through unredacted with no error raised.

I added detect_dominant_language auto-routing: detect language first, route to detect_pii_entities with the right code when supported, fall back to en otherwise. The fallback gap is documented, not closed. I found this in a stress test. A production deployment without multilingual PII requirements in scope would have shipped without catching it.

Dev Note

“Comprehend handles multilingual PII” is technically true for two languages. The gap isn’t documented prominently. Your stress test will find it. Your production deployment shouldn’t be the first time.

Swap 3: Local guardrails → Bedrock Guardrails ⚠️

The local version is ~330 lines: regex injection detection, denied topics list, grounding score, relevance check. Bedrock Guardrails gives you NLI-based prompt attack detection, formal topic policies, server-side PII masking on output, and enterprise compliance certification.

In the 50-query run, adversarial protection was symmetric — both backends blocked all 5 prompt injection attempts. Bedrock Guardrails offered no measurable advantage on my test set, partly because regex catches the obvious attacks and my set didn’t probe subtle multi-turn jailbreaks where NLI detection would likely win.

The real asymmetry is one specific control: AWS takes observability away from you.

In Post 2, I built a grounding score — a word-overlap heuristic that tells the caller how well the model’s answer is supported by the retrieved source. I built it specifically because I knew enterprise audit trails need that signal.

Bedrock Guardrails enforces a 0.75 grounding threshold internally. It does not return the score to the caller. You know it passed or blocked. You don’t know by how much.

# local guardrails.py — grounding_score exposed on PolicyResponse
return PolicyResponse(grounding_score=self._score_grounding(answer, context), ...)
# aws guardrails.py — still computed locally because Bedrock won't give it back
return PolicyResponse(grounding_score=self._score_grounding(answer, context), ...)

The local grounding computation stayed in the AWS version. Word-overlap is crude. Observable is better than blind trust in a threshold you can’t see.

The silent audit misreport: Bedrock has two independent guardrail evaluation paths — apply_guardrail() standalone, and the inline guardrail bound to invoke_model. They’re evaluated separately. On a French query in the run, apply_guardrail returned PASS while the inline path returned INTERVENED. The audit row recorded PASS while the user saw a blocked response.

policy_engine.ask_llm() now parses amazon-bedrock-guardrailAction from the response payload and surfaces a guardrail_intervened flag. ask_policy() synthesises a BLOCK result from that flag. The fix is in guardrails.py — see TestInlineGuardrailIntervention.

The audit row recorded PASS. The user saw a blocked response. I found it on a French query in the 50-query run — not in code review, not in testing.

Swap 4: Keyword retrieval → Bedrock Knowledge Bases 🔴

This is the treacherous one. Also the most marketed feature. That combination is worth paying attention to.

The local version is a keyword overlap function — count matching non-stop-word tokens, return the highest-scoring document. Crude by design; I built it in Post 2 to illustrate the pattern before touching managed services.

Bedrock Knowledge Bases gives you semantic vector search backed by Titan v2 embeddings, stored in S3 Vectors — GA’d at re:Invent 2025, roughly 90% cheaper than OpenSearch Serverless for this scale.

Dev Note

I was about to build the CDK stack around OpenSearch Serverless. The minimum for a collection is ~$700/month just to exist, which would have swamped the entire cost story at demo scale. S3 Vectors shipped the same week I was building. This is what it feels like to build on a platform that’s still finding its shape.

The semantic retrieval improvement on factual queries is real and measurable. On HR factual questions, AWS went from 6/10 HIGH confidence to 10/10 HIGH across 50 queries. Keyword retrieval gets distracted by word overlap — “time off” matched the 4-day workweek council document because those words appear more often in that file. Bedrock’s embedding understood the intent and routed to the HR policy’s vacation section every time.

But I was wrong about where semantic search wins.

So where did AWS actually lose?

I built the benchmark expecting AWS to win on retrieval across the board. The data didn’t cooperate.

Council / deliberative queries: Local 4/5 HIGH, AWS 0/5 HIGH.

Both backends retrieved the same council documents — semantic search routed correctly. The failure was downstream: local fed the entire 7–15 KB policy document to the model. AWS fed a 500-token semantic chunk.

Council-generated policies don’t answer questions in paragraphs. The answer to “what counts as a reasonable expense for remote engineers” lives across three sections — the definition, the exceptions, the approval threshold. A 500-token chunk carries one of those. The model got a focused slice of the right document and returned LOW confidence because it couldn’t synthesise across sections.

This wasn’t a Bedrock limitation. It was a chunking decision I didn’t know I was making. Bedrock Knowledge Bases sets a default chunk size. I accepted the default. The benchmark exposed it.

So why does the default chunk size matter this much?

After the Council reversal, I ran a follow-up — 50 queries across four chunking strategies:

HIERARCHICAL won on HIGH rate. NONE — the obvious fix of feeding full documents — didn’t win cleanly, and produced 10 output blocks vs SEMANTIC’s 2.

Wait — why do bigger chunks produce more failures?

Bedrock’s contextual grounding guardrail evaluates model output against the retrieved source. Default threshold is 0.75 — responses scoring below are blocked before reaching the user.

Bigger chunk → LLM generates a longer, richer answer → more sentences → more chances for one sentence to score below 0.75 → guardrail blocks the whole response.

In production that’s not a cost number or a latency number. It’s task completion rate. It surfaces as support tickets. Most chunking comparisons measure tokens and latency and stop there. I measured block count and watched it scale with chunk size — 2 blocks with SEMANTIC 500, 10 blocks with NONE.

Dev Note

The grounding threshold is configurable via contextualGroundingPolicyConfig — any value from 0 to 0.99, BLOCK or NONE action. But tuning it requires understanding your content type and your risk tolerance. It’s not a hot-plug. It’s a product decision AWS can’t make for you.

Why HIERARCHICAL wins

Retrieval and synthesis want opposite things from chunk size.

Retrieval wants small chunks. Titan v2 produces 1,024-dimension embeddings regardless of whether it’s encoding 300 tokens or 2,000. The more text compressed into that fixed vector, the lossier the representation:

This is why SEMANTIC 2000 underperformed SEMANTIC 500 — more context, worse retrieval routing, because the embedding had to represent four times as much text in the same vector space.

Synthesis wants large chunks. The model needs surrounding context to reason across sections. One paragraph isn’t enough for a multi-section policy question.

HIERARCHICAL breaks the tradeoff with two storage layers:

• Children (~300 tokens) are the only things embedded. Retrieval matches against these — small, precise, high signal-to-noise
• Parents (~1,500 tokens) are never embedded. Stored as plain text, keyed to their children

Query time: embed query → match child vectors → fetch the parent containing that child → send parent to the LLM. Retrieval precision from children. Synthesis context from parents. Not a compromise — a different pipeline architecture.

The cost: 2× tokens vs SEMANTIC 500 ($0.001443 vs $0.000714). At 2,000 queries/month that’s ~$0.73 more in model tokens plus proportional Guardrails text-unit costs — roughly $5–10/month total uplift. Worth it when queries span more than one paragraph of context, which is most real corpora.

The limit: Even with HIERARCHICAL, Council deliberative content only reached 1/5 HIGH. I expected chunking was the problem. Feeding the full document (NONE) only got to 2/5. The bottleneck is Bedrock’s grounding guardrail blocking long inferential answers — not retrieval. That’s a content problem and a calibration problem, not a chunking problem.

Five things AWS still doesn’t close

These gaps remain after the full upgrade. They’re the natural result of AWS generalising across thousands of customer workloads — no default fits everyone.

1. Retrieval transparency. Bedrock returns one cosine score per chunk; chunking decisions are opaque. No “show me which sentence in hr_policy.txt answered this query.” Local _score_grounding is a crude mitigation, not a fix.
2. Hybrid search. Bedrock KB on S3 Vectors is pure semantic. Acronyms (PIP, MFA), product names, and ticket IDs under-rank against semantically-similar prose. Production deployments with internal jargon need a parallel keyword index — OpenSearch Managed Cluster, not S3 Vectors.
3. Per-team cost governance. CloudWatch metrics aren’t dimensioned by team. AWS Budgets is monthly and tag-based — too coarse for chargeback. A real multi-team deployment would need DynamoDB atomic counters and per-team rate limiting. I don’t have that built yet — it’s the gap I’d hit first at actual org scale.
4. PII coverage beyond English and Spanish. No native AWS solution for French, German, or Japanese PII. Machine-translate first, fall back to en (catches universal formats only), or bring your own model. This is a model boundary, not a config problem.
5. Inline guardrail observability. apply_guardrail() and the inline guardrail on invoke_model are evaluated independently. Without parsing amazon-bedrock-guardrailAction, audit rows silently misreport. AWS treats these as two separate features rather than one observability surface.

Managed services close the easy gaps and surface the hard ones.

The cost section nobody publishes

“AWS is 2.7× cheaper” is true at one specific layer. Here’s what that layer actually is.

Every number in the 50-query run — every per-query figure in the Streamlit audit log, every CloudWatch cost_usd metric — counts only Bedrock Claude Haiku 4.5 token cost. calculate_cost() multiplies token counts by published per-token rates. Nothing else.

At 2,000 queries/month (200 employees × 10 queries), the realistic breakdown:

If your finance team extrapolates from the per-query audit log cost, they’ll undercount the real AWS bill by 5–10×.

Dev Note

My original estimate was $34–44/month. The measured number is $24–29/month — lower because semantic chunks cut input tokens 4.4× on factual queries, more than I projected. But I was measuring token cost and calling it “cost.” Bedrock Guardrails is probably the largest single line item on the actual bill. It doesn’t appear in the invoke_model response payload, so it never made it into calculate_cost().

The real argument for AWS isn’t the token math. It’s maintenance cost — which doesn’t appear in any benchmark. The 47-line regex PII scrubber needs updating when PII formats change. The local denied-topic list needs maintenance. The CSV audit breaks under concurrency. AWS takes that off your plate. At a company where the AI team is also the platform team, that compounds quietly.

When does AWS actually make sense?

Not universally. Workload-conditionally.

AWS makes sense when the controls you’re giving up — observability, chunking transparency, hybrid search — matter less than the controls you’re gaining: compliance certification, maintained PII catalogues, managed infrastructure. And when you don’t have the engineering capacity to maintain the custom layers as the workload grows.

AWS makes less sense when your corpus is primarily deliberative or narrative content where chunking defaults hurt you and calibration requires ongoing tuning. Or when your employees operate in languages beyond English and Spanish with PII requirements. Or when you need grounding score visibility for audit trails and aren’t willing to compute it locally alongside Bedrock.

That’s the honest answer I keep landing on — it depends on the workload. I built the benchmark to find out for mine.

The open question

I know why AWS struggled on Council deliberative content — the grounding guardrail blocks long, inferential answers at the 0.75 threshold. HIERARCHICAL chunking helped everywhere except there.

What I haven’t tested: lower the grounding threshold to 0.5 specifically for the deliberative data source. Bedrock supports per-data-source guardrail configuration. That’s a CDK stack change, not a code change. The question is how much of the Council gap is the content genuinely being hard to answer with high confidence, and how much is a guardrail calibrated for a different workload type.

That’s not a Bedrock question. It’s a question about what “confidence” means for deliberative content — a product decision, not a configuration choice.

AWS is actively working this out. The chunking defaults, the grounding threshold, the hybrid search gap — these will change as they work with customers on real workloads. The platform is still finding its shape. That’s not a criticism. It’s the honest state of where enterprise AI infrastructure is right now.

The question is whether you want to be the customer helping them figure it out, or whether you wait until the abstractions stabilize.

I don’t have a clean answer. I have 50 queries of data, a working CDK stack, and a cdk destroy --all scheduled for next Friday.

Now so do you.

Appendix: The 50 questions

Every claim in this post is based on running scripts/stress_test.py against both backends — same 50 questions, same order, same four policy documents. Here’s exactly what was asked.

Rows 39–43 are the ones that explain the chunking section. Each question spans multiple sections of a 7–15 KB deliberative policy document. A 500-token chunk carries one of those sections. That’s why local won 4/5 and AWS won 0/5 on those five — not a retrieval failure, a context window failure.

Test files:
- policy-pal-aws/scripts/stress_test.py — runs the 50-question suite against the AWS backend
- policy-pal/scripts/stress_test.py — same script, local backend
- policy-pal-aws/scripts/stress_compare.py — diffs two JSONL outputs side by side
- policy-pal-aws/compare_models.py — Haiku vs Sonnet head-to-head (8 queries)
cd policy-pal-aws && uv run python scripts/stress_test.py --output /tmp/aws_stress.jsonl
cd ../policy-pal   && uv run python scripts/stress_test.py --output /tmp/local_stress.jsonl
cd ../policy-pal-aws && uv run python scripts/stress_compare.py --local /tmp/local_stress.jsonl --aws /tmp/aws_stress.jsonl

Next: Post 5 — the same four swap patterns applied to a code review bot. Different domain, same architecture. That’s the point.

Code: policy-pal · policy-pal-aws · policy-pal-infra