Amazon says AI agents wrote and reviewed its new AWS Emulator code, not engineers
In brief
- Amazon released AWS Emulator, a local tool that imitates AWS services, on Nov. 11.
- Engineer Matthew Goldberg said his team neither wrote nor reviewed the emulator code.
- Human-built checks for compilation, coding standards, security and regressions gate code merges.
- AWS says the emulator doesn't check KMS key or S3 bucket policies.
- AWS tells developers to test apps and security settings on real AWS before production.
Agents wrote it, people built the rails
In an Amazon engineering account, engineer Matthew Goldberg said his team neither wrote nor reviewed the emulator code. That's Amazon's claim (as reported by CryptoSlate), and it hasn't been independently verified.
Humans weren't out of the loop, though. Goldberg described substantial engineering work to turn the experiment into a product, including the guardrails and checks that determine which changes can ship. According to Amazon, every change faced checks for compilation, coding standards, security and regressions before merging.
Amazon described three teams of agents: one managing fidelity tests, one finding differences between the emulator and real AWS, and one fixing individual issues. The test agents worked from public AWS API models and verified their scenarios against the real cloud.
What "fidelity" means here
AWS uses the term "fidelity" for the degree of agreement between the emulator and its cloud services. Per AWS's fidelity documentation, a test passes only when the emulator matches AWS for every call in the workflow (status codes, resources, side effects and errors included). Returning success where AWS returns an error fails the test.
The public evaluation suite says every fidelity test was checked against at least two sandboxed AWS accounts. The evaluation repository also draws a hard line: behavior without a fidelity test is unverified.
No test, no verification.
Not a security tool
AWS's security guidance warns against using the emulator to determine whether an application is secure. The emulator doesn't check AWS KMS key policies or Amazon S3 bucket policies, so a request that AWS would deny may succeed locally.
It's meant for building and testing, not production workloads. AWS instructs developers to test their whole application and security settings on real AWS before production, which means the local tool doesn't replace the cloud for anything that's going live.
Frequently asked questions
Did humans have any role in building AWS Emulator?
Amazon says AI agents wrote and reviewed the emulator's implementation. Engineer Matthew Goldberg also described substantial human engineering work to turn the experiment into a product, including the guardrails and checks that determine which changes can ship. According to Amazon, those checks covered compilation, coding standards, security and regressions.
What does fidelity mean for AWS Emulator?
AWS uses fidelity to describe how closely the emulator agrees with its cloud services. Per AWS documentation, a test passes only when the emulator matches AWS for every call in a workflow, including status codes, resources, side effects and errors. The evaluation repository says behavior without a fidelity test is unverified.
Can AWS Emulator be used to check whether an application is secure?
No. AWS's security guidance warns against using the emulator for that purpose. It doesn't check AWS KMS key policies or Amazon S3 bucket policies, so a request AWS would deny may succeed locally. AWS tells developers to test their whole application and security settings on real AWS before production.
