Permanently Blocking IP Addresses with AWS WAF Rate-Based Rule Limit

What is WAF?

AWS WAF is a web application firewall that lets you control access to your content based on the criteria that you specify. In simpler words, it protects your web applications from threats and provides you the ability to control the access of traffic that is visiting your site.

What is this blog about?

AWS WAF provides multiple ways to manage access to the traffic coming to your web application. One of the most important things we can do with this is to stop DDoS (Distributed Denial-of-Service).

Rate-based rule that counts incoming requests and rate limit requests when they are coming too fast. We can tweak the rules according to our requirements and prevent DDoS attacks on our server. 

But there are some limitations to it. In this blog, we are going to see what the limitations are and how we can surpass these limitations.

WAF and Other Resource Pricing

WAF is not a free service, and there is no free tier for that. But if you are learning, then testing is very cheap. During my testing period, it cost me around $0.18. For permanent setups, refer to the official documents from AWS.

A few things need to be noted down first:

AWS WAF can only be used with CloudFront Distributions and Regional Resources.

Regional Resources:

• Amazon API Gateway REST API
• Application Load Balancer
• AWS AppSync GraphQL API
• Amazon Cognito user pool
• AWS App Runner service
• AWS Verified Access instance

• PS: Here I am implementing this whole thing for the application load balancer.

Creating Web ACL and Rules

In this section, we will create a Web ACL with a rule that blocks IPs that exceed the defined rate limit.

1. Go to AWS WAF → Web ACLs → Select the region in which your load balancer or regional resource → Create Web ACL.

2. Select resource type: (for me) 

Regional resources, Region: Asia Pacific (Mumbai), 

Name: xxxx

3. Add application load balancer: 

Add Resources → Application Load Balancer → Select alb.

4. Add Rate-base Rules: 

Add rules → Add my own rules and rule groups.

5. Rule Set-Up that blocks IPs based on specified rate limit:

• Rule type: Rule builder
• Type: Rate-based rule

• Rate Limit: 100 (You can specify according to your traffic)
• Evaluation window: 1 minute (This means if someone makes more than 100 requests in 1 minute, then their IP will be blocked)
• Request aggregation: IP address in header (Why this option? Let’s discuss it later in this blog)

• Header field name: X-Forwarded-For

6. Create Rule → Set rule Priority → Enable Sampled request.

Here we have created Web ACL with a rule that blocks IP addresses if they exceed the rate limit of 100 requests per minute. 

Are we done??? Nope!

Problem With This Rate-base Rule

We don't know exactly why, but this rate-base rule blocks IP addresses very well, but for a few minutes only. After a few minutes, it releases blocked IPs, and the attacker is again able to attack our server.

Solution?

Get help from Lambda Functions. And there is another service from AWS WAF called IP sets. IP sets are a set of IP addresses stored in the IP set. We can utilize these IP sets to create new Web ACL rules that allow or block IPs from these IP sets.

Here’s what we are going to do:

• Create an IP set that contains IP addresses of attackers.
• Create a rule in Web ACL that blocks IP addresses from attackers IP set.
• Create Lambda Function that reads the list of Blocked IPs by Rate-Base Rule, and add those IPs to the attacker's IP set.
• Create one Eventbridge Rule that triggers lambda functions every minute. So, we can keep the attackers IP set list updated with newly blocked IPs.

Too many hectic tasks? Don’t worry, CloudFormation has your back. I have created a CloudFormation template that will create an IP set, Lambda Function, IAM roles and permissions and Eventbridge Rule.

Creating Lambda, IP set, IAM role and Eventbridge rule

Using the CloudFormation template we will create an IP set, Lambda Function, IAM roles and permissions and Eventbridge Rule.

1. You can find the CloudFormation template here.

→ CloudFormation Template

1. Go to CloudFormation → Stacks → Create Stack → With New Resources (Standard).
2. Prepare template: Choose an existing template. Template source: Upload a template file. (I've stored the template in S3, so you can use the S3 URL but you can directly upload a .YAML file.

4. Give name to the stack and fill the following Parameters.

• RateBasedRuleName: Name of our Rate-base- rule created in Web ACL.
• Scope: REGIONAL ( I am using ALB so it is a regional resource.)
• WebACLId: It can be found at Web ALC dashboard (AWS WAF → Web ACLs)
• WebACLName: Name of Web ACL.

5. Keep everything else default, and at the end Select the checkbox of “I acknowledge that AWS CloudFormation might create IAM resources” and Submit.

Now IP set, Lambda function, IAM roles, and Eventbridge Rule will be created by CloudFormation. We have to create Rule that blocks IPs from creating IP sets.

Create IP Set Blocking Rule

We are going to create a new Web ACL rule that blocks IP addresses from the IP set created by CloudFormation.

1. Go to AWS WAF → Web ACLs → your Web ACL → Rules → Add my own rules.

2. Block request from the IPs of attackers IP set.

• Rule type: IP set
• Name: xxxxx
• IP set: Select IP set created by CloudFormation.

• IP address to use as the originating address: IP address in header
• Header field name: X-Forwarded-For
• Position inside header: Any IP addresses

• Action: Block

3. Set priority of rule: select this rule and move this rule to the top.

We are done with our implementation. Now, it's time for testing.

Testing

I am using ApacheBench to send 200 requests in a minute to my server. I have created one EC2 instance running on Ubuntu and am going to use it for testing, you can install this on your local machine as well.

• Install ApacheBench on Ubuntu.

sudo apt-get update

sudo apt-get install apache2-utils

• Creating a shell script that sends 200 requests per minute to the domain (in other words, to the server)

#!/bin/bash

for i in {1..20}

do

 ab -n 10 -c 1 -H "X-Forwarded-For: 1.2.3.4" http://your-server-endpoint/

 sleep 3

done

• To run a script, run the following commands:

chmod +x test.sh

./test.sh

Now check the IP set list (AWS WAF → IP sets → …) There you can find this 1.2.3.4/32 IP, which means everything is working properly. (Maybe the list won’t get updated immediately as lambda triggers every 1 minute.).

Why Are We Using ‘X-Forwarded-For’?

For a moment, assume that you are a DDoS attacker. What are the things that need to be taken care of when you are DDoSing? Your IP address. As an attacker, your IP address is a very important thing that you don’t want to reveal to the victim or anyone else. 

So, to hide their IP attacker, use HTTP proxies (Zombies) to attack the web server. HTTP proxies will send requests to the webserver on behalf of attackers. 

HTTP proxies are hard to find as they may be situated at any random geolocation and their IPs keep changing. The diagram below is a very simple representation of DDoS attacks.

DDoS Attacks Using HTTP Proxies

In this case, blocking source IP addresses is not helpful at all, as source IP addresses are constantly changing and attackers are still able to attack your server easily from another HTTP proxy.

Is it impossible to stop DDoS attacks?

Nope, as per the research, no clear pattern could be found as IPs were keep changing and from entirely different locations across the planet, but one thing was clearly visible after inspecting HTTP requests: they all had headers that indicated that they were being forwarded from HTTP proxies: ‘X-Forwarded-For’.

Identifying Client IP from X-Forwarded-For Header

Any requests that are coming from HTTP proxies or load balancers have this common header ‘X-Forwarded-For’, which contains the originating client IP. 

The meaning of this header is that the request is being sent on behalf of the originating IP. This header is only present if the request is coming from HTTP proxies or a load balancer.

This header is non-standard, there is no guarantee you’ll get it, and the way it is handled can differ depending on the proxy implementation. You also have no guarantee that it will contain a proper IP.

Advantages of using ‘X-Forwarded-For’:

• If you use the Source IP address to block requests, then there is a high chance that your regular user might get blocked if they exceed the limit, while the attacker will still be able to increase the load of your server by using proxies.
• The ‘X-Forwarded-For’ header is only present in requests that are coming from HTTP proxies or load balancers, making it easier to differentiate between regular users and users with bad intentions.