Serverless Event Filtering Patterns for Cost and Performance

Introduction

Serverless architectures often suffer from excessive function invocations caused by processing events that could have been filtered upstream. Before event source filtering was available, Lambda functions had to receive every event, evaluate whether it was relevant, and discard the majority. This pattern wasted compute, increased costs, and added latency to legitimate event processing.

AWS now provides event filtering at the source mapping level for SQS, DynamoDB Streams, Kinesis, and Kafka. In production environments processing 10M+ events daily, proper filtering reduces Lambda invocations by 60-80% and costs by $500-2,000 per month without any business logic changes.

Event Filtering Architecture

The filter is applied before Lambda receives events, meaning filtered events never trigger an invocation:

Where Filtering Can Be Applied

Event Source	Filter Location	Filter Syntax	Max Patterns
SQS	Event Source Mapping	EventBridge-style	5 per source
DynamoDB Streams	Event Source Mapping	EventBridge-style	5 per source
Kinesis	Event Source Mapping	EventBridge-style	5 per source
MSK/Kafka	Event Source Mapping	EventBridge-style	5 per source
EventBridge	Rule pattern	EventBridge-style	Unlimited
SNS	Subscription filter	Attribute-based	5 per subscription
API Gateway	Request validation	JSON Schema	Per method

Event Source Mapping Filters

DynamoDB Streams Filter

Only process events for specific record types or state changes:

import boto3
import json

lambda_client = boto3.client('lambda')

# Create event source mapping with filter
response = lambda_client.create_event_source_mapping(
    EventSourceArn='arn:aws:dynamodb:us-east-1:123456789012:table/Orders/stream/2026-01-01T00:00:00.000',
    FunctionName='process-order-updates',
    StartingPosition='LATEST',
    BatchSize=100,
    MaximumBatchingWindowSeconds=5,
    FilterCriteria={
        'Filters': [
            {
                'Pattern': json.dumps({
                    "eventName": ["MODIFY"],
                    "dynamodb": {
                        "NewImage": {
                            "status": {
                                "S": ["shipped", "delivered", "cancelled"]
                            }
                        },
                        "OldImage": {
                            "status": {
                                "S": ["processing", "confirmed"]
                            }
                        }
                    }
                })
            }
        ]
    }
)

SQS Filter

Process only messages matching specific criteria:

# SAM template
OrderProcessor:
  Type: AWS::Serverless::Function
  Properties:
    Handler: handler.process_order
    Runtime: python3.12
    Events:
      OrderQueue:
        Type: SQS
        Properties:
          Queue: !GetAtt OrderQueue.Arn
          BatchSize: 10
          FilterCriteria:
            Filters:
              - Pattern: |
                  {
                    "body": {
                      "order_total": [{"numeric": [">=", 100]}],
                      "priority": ["high", "urgent"],
                      "region": [{"prefix": "us-"}]
                    }
                  }

Kinesis Stream Filter

# CDK / CloudFormation
KinesisProcessor:
  Type: AWS::Lambda::EventSourceMapping
  Properties:
    EventSourceArn: !GetAtt DataStream.Arn
    FunctionName: !Ref ProcessorFunction
    StartingPosition: LATEST
    BatchSize: 500
    MaximumBatchingWindowSeconds: 10
    FilterCriteria:
      Filters:
        - Pattern: |
            {
              "data": {
                "event_type": ["purchase", "refund"],
                "amount": [{"numeric": [">", 0]}],
                "currency": ["USD", "EUR", "GBP"]
              }
            }

Filter Pattern Syntax

The filter pattern syntax supports multiple comparison operators:

Operator	Syntax	Example	Matches
Exact match	["value"]	{"status": ["active"]}	status = "active"
Multiple values	["a", "b"]	{"type": ["A", "B"]}	type = "A" or "B"
Prefix	[{"prefix": "x"}]	{"id": [{"prefix": "ord-"}]}	id starts with "ord-"
Numeric	[{"numeric": [">", 100]}]	{"amount": [{"numeric": [">", 100]}]}	amount > 100
Numeric range	[{"numeric": [">=", 0, "<=", 100]}]	Between 0 and 100	0 <= value <= 100
Exists	[{"exists": true}]	{"email": [{"exists": true}]}	Field is present
Not exists	[{"exists": false}]	{"deleted": [{"exists": false}]}	Field is absent
Null	[null]	{"error": [null]}	Field is null
Anything-but	[{"anything-but": ["x"]}]	{"env": [{"anything-but": ["test"]}]}	env != "test"

EventBridge Rule Patterns

EventBridge supports the most expressive filtering with content-based routing:

{
  "source": ["com.myapp.orders"],
  "detail-type": ["OrderStateChange"],
  "detail": {
    "status": ["completed", "failed"],
    "total_amount": [{"numeric": [">=", 1000]}],
    "customer": {
      "tier": ["enterprise", "premium"],
      "region": [{"prefix": "us-"}]
    },
    "items": {
      "category": [{"anything-but": ["digital"]}]
    }
  }
}

Multi-Rule Routing

# High-value orders -> dedicated processor
aws events put-rule \
  --name "high-value-orders" \
  --event-pattern '{
    "source": ["com.myapp.orders"],
    "detail": {
      "total_amount": [{"numeric": [">=", 10000]}],
      "status": ["completed"]
    }
  }'

# Failed orders -> alert system
aws events put-rule \
  --name "failed-orders" \
  --event-pattern '{
    "source": ["com.myapp.orders"],
    "detail": {
      "status": ["failed"],
      "retry_count": [{"numeric": [">=", 3]}]
    }
  }'

Cost Impact Analysis

Before and After Filtering

Metric	Without Filtering	With Filtering	Improvement
Daily events received	10,000,000	10,000,000	-
Lambda invocations	10,000,000	2,500,000	-75%
Avg execution time	50ms	80ms*	+60%
Daily compute (GB-s)	65,104	26,042	-60%
Monthly Lambda cost	$1,085	$434	-60%
Monthly SQS cost	$4.00	$4.00	0%
Monthly total	$1,089	$438	-60%

*Execution time increases slightly because filtered events are the ones requiring actual processing.

Testing Filter Patterns

Use the EventBridge sandbox to validate patterns before deployment:

# Test event pattern match
aws events test-event-pattern \
  --event-pattern '{
    "detail": {
      "status": ["shipped"],
      "amount": [{"numeric": [">=", 100]}]
    }
  }' \
  --event '{
    "detail": {
      "status": "shipped",
      "amount": 250,
      "customer_id": "cust-123"
    }
  }'

# Response: {"Result": true}

Unit Testing Patterns

import pytest
import json

def matches_filter(pattern: dict, event: dict) -> bool:
    """Simplified filter matching for unit tests."""
    for key, criteria in pattern.items():
        if key not in event:
            return False
        if isinstance(criteria, list):
            if event[key] not in criteria:
                # Check for numeric operators
                for c in criteria:
                    if isinstance(c, dict) and 'numeric' in c:
                        ops = c['numeric']
                        value = event[key]
                        if not evaluate_numeric(ops, value):
                            return False
                    elif event[key] not in criteria:
                        return False
        elif isinstance(criteria, dict):
            if not matches_filter(criteria, event[key]):
                return False
    return True

class TestOrderFilter:
    PATTERN = {
        "status": ["shipped", "delivered"],
        "amount": [{"numeric": [">=", 100]}]
    }

    def test_matches_shipped_high_value(self):
        event = {"status": "shipped", "amount": 250}
        assert matches_filter(self.PATTERN, event) is True

    def test_rejects_processing_status(self):
        event = {"status": "processing", "amount": 250}
        assert matches_filter(self.PATTERN, event) is False

    def test_rejects_low_value(self):
        event = {"status": "shipped", "amount": 50}
        assert matches_filter(self.PATTERN, event) is False

Best Practices

Pattern Design Guidelines

1. Filter early, filter aggressively — The fewer events reaching Lambda, the lower the cost and higher the signal-to-noise ratio.

2. Use multiple patterns for OR logic — Each filter in the Filters array acts as OR; conditions within a pattern act as AND:

{
  "Filters": [
    {"Pattern": "{\"body\": {\"type\": [\"purchase\"]}}"},
    {"Pattern": "{\"body\": {\"type\": [\"refund\"], \"amount\": [{\"numeric\": [\">\", 100]}]}}"}
  ]
}

3. Monitor filtered vs. processed ratio — Track the percentage of events filtered to validate your patterns are working correctly.

Key Takeaways

• Event source mapping filters reduce Lambda invocations by 60-80% by discarding irrelevant events before function execution begins.
• Filters are free — there is no additional charge for event source mapping filtering; you only pay for events that pass the filter.
• Use numeric operators for threshold-based routing to direct high-value transactions, anomalous readings, or critical alerts to dedicated processors.
• Combine prefix matching with exact values for flexible routing that handles hierarchical data like regions, categories, or resource identifiers.
• Test patterns with EventBridge sandbox before deploying to production to verify correct matching behavior.
• Monitor filter ratios to detect upstream schema changes that might cause patterns to stop matching legitimate events.
• Start with broad filters and narrow over time based on observed event patterns rather than attempting to define perfect filters initially.