Table of Contents
Tired of your API being hit by web-crawlers? These robots and other bad User Agents can ruin your analytics, place unnecessary load on your infrastructure and increase the cost of your AWS bill.
An API Gateway Resource Policy can be used to selectively block certain User Agents, alleviating these issues.
Using a Resource Policy
Resource Policies are one way of controlling access to your API. They are JSON policy documents that are applied to API resources either through the Console or the CLI.
AWS supplies official examples if you’d like to:
You can combine the official examples with our below policy documents to increase the security of your API and block bad User Agents or web-crawlers
Block Specific User Agents
In the below example we block the most popular web crawlers reported by KeyCDN:
Block User Agents with a wildcard
You can also use wildcards if you’d like a more flexible “catch-all” approach: