Stop Guessing Which Model Is Better: Amazon Bedrock Model Evaluation Hands-On

"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships. In this hands-on, we'll answer the question with data: Amazon Bedrock Model Evaluation , run in two modes — automatic metrics scored against reference answers, and LLM-as-a-Judge , where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with jq and awk . Prefer video? This entire hands-on is also on YouTube: The two evaluation modes Automatic evaluation runs your dataset through the target model and scores each response against your referenceResponse with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers. LLM-as-a-Judge has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model. Run both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security. Note on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names. Step 1: Environment and bucket All of this runs in CloudShell: export REGION = us-east-1 export ACCOUNT_ID = $( aws sts get-caller-identity --query Account --output text ) export EVAL_BUCKET = "bedrock-eval-$ACCOUNT_ID " aws s3 mb "s3:// $EVAL_BUCKET " --region$REGION Step 2: The evaluation dataset The dataset is JSONL — one JSON object per line, each with a prompt and a referenceResponse (the answer you consider correct). Ten AWS-basics Q&A pairs: cat > /tmp/eval-dataset.jsonl << ' EOF ' {"prompt":"Explain the main use of AWS Lambda in one sentence","referenceResponse":"A compute service that runs code serverlessly in response to events"} {"prompt":"What kind of service is Amazon S3?","referenceResponse":"A highly available and highly durable object storage service"} {"prompt":"What kind of database is Amazon DynamoDB?","referenceResponse":"A fully managed NoSQL key-value database"} {"prompt":"What is a CloudWatch alarm?","referenceResponse":"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold"} {"prompt":"What is an IAM role?","referenceResponse":"A mechanism for granting temporary permissions to AWS resources and users"} {"prompt":"What is Amazon VPC?","referenceResponse":"A service for building a logically isolated virtual network on AWS"} {"prompt":"What are the characteristics of Amazon RDS?","referenceResponse":"A service that runs relational databases in a fully managed way"} {"prompt":"What is the main function of CloudFront?","referenceResponse":"A content delivery network (CDN) that uses edge locations"} {"prompt":"What kind of service is Amazon SQS?","referenceResponse":"A managed message queuing service"} {"prompt":"What is Amazon Bedrock?","referenceResponse":"A service that provides foundation models through a serverless, unified API"} EOF aws s3 cp /tmp/eval-dataset.jsonl "s3:// $EVAL_BUCKET /input/dataset.jsonl" Ten pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file is the asset worth investing in: it becomes your regression test for every future model release. Step 3: IAM role for the evaluation jobs Bedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models: cat > /tmp/eval-trust.json << ' EOF ' {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"bedrock.amazonaws.com"},"Action":"sts:AssumeRole"}]} EOF EVAL_ROLE_ARN =$( aws iam create-role \ --role-name BedrockEvaluationRole \ --assume-role-policy-document file:///tmp/eval-trust.json \ --query 'Role.Arn' --output text ) cat > /tmp/eval-policy.json << EOF { "Version": "2012-10-17", "Statement": [ {"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket","s3:PutObject"], "Resource":["arn:aws:s3::: $EVAL_BUCKET ","arn:aws:s3:::$EVAL_BUCKET /"]}, {"Effect":"Allow","Action":["bedrock:InvokeModel"], "Resource":["arn:aws:bedrock:::foundation-model/","arn:aws:bedrock:::inference-profile/"]} ] } EOF aws iam put-role-policy --role-name BedrockEvaluationRole \ --policy-name inline --policy-document file:///tmp/eval-policy.json Note the trust policy: the principal is bedrock.amazonaws.com — this role is for the service, not for you or a Lambda. Step 4: Create the two evaluation jobs (console) In the Bedrock console (Inference and Assessment → Evaluations), create two jobs against the same dataset — the video above walks through every screen: Automatic evaluation — pick the model to evaluate, task type Q&A, your dataset at s3://$EVAL_BUCKET/input/dataset.jsonl , the BedrockEvaluationRole , and an output path under s3://$EVAL_BUCKET/output/auto/ LLM-as-a-Judge — same dataset and role, pick the model to evaluate and a judge model (use a stronger model than the one being judged), output under s3://$EVAL_BUCKET/output/judge/ Both jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size. Step 5: Pull the results and make them readable When the jobs complete, sync everything down and find the result files: aws s3 sync "s3://$EVAL_BUCKET /output/" /tmp/eval-out/ find /tmp/eval-out/ -type f # List the main output files find /tmp/eval-out/ -name "_output.jsonl" -not -name ".out" The output is deeply nested JSONL — one record per prompt, each carrying a scores array. Aggregate the automatic metrics into per-metric count / mean / max: for f in /tmp/eval-out/auto/ * / * /models/ * /taskTypes/ * /datasets/ * / * _output.jsonl ; do cat " f " | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"' done | awk -F '\t' ' {sum[1]+=$2; cnt[$1]++; if($2>max[$1]) max[$1]=$2} END {for (m in sum) printf "%s\tcount %d\tmean %.4f\tmax %.4f\n", m, cnt[m], sum[m]/cnt[m], max[m]} ' And the judge's scores, averaged per metric: JUDGE = $( find /tmp/eval-out/judge -name "*_output.jsonl" -not -name "*.out" | head -1 ) cat "$JUDGE " | jq -r '.automatedEvaluationResult.scores[] | "(.metricName)\t(.result)"' | \ awk '{sum[$1]+=$2; cnt[1]++} END {for (m in sum) printf "%s\tmean %.2f\n", m, sum[m]/cnt[m]}' Two numbers-reading tips: Don't compare across metric families. An automatic similarity score and a judge's correctness grade live on different scales; compare models within the same metric, not metrics with each other. The mean hides the failures. The per-record JSONL is right there — before trusting an average, look at the worst-scoring records and read what the model actually said. Ten minutes of reading beats any single number. Cleanup # S3 aws s3 rm "s3:// EVAL_BUCKET " --recursive aws s3 rb "s3:// $EVAL_BUCKET " # IAM aws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline aws iam delete-role --role-name BedrockEvaluationRole Wrapping up Model selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade. About the author Maruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain (AWS / Google Cloud / Azure) | Udemy instructor (100K+ students) 🎥 Video version of this hands-on: https://youtu.be/jbrFKA34hWc 📚 Full course — AWS Certified Generative AI Developer Professional (AIP-C01) Exam Prep: https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/ 👨‍🏫 All my courses: (Eng) https://www.udemy.com/user/maruchin-tech-2/ (Jpn) https://www.udemy.com/user/shan-wang-wan-jun-2/ 🎫 Monthly discount coupons: https://www.youtube.com/@MaruchinTech-cloud/posts