AWS CloudWatch Custom Metrics: Building Microservices Observability That Actually Works

Our monitoring was a dashboard graveyard. Fifty-three CloudWatch dashboards, most showing default EC2 and RDS metrics that nobody looked at. When an incident occurred, engineers spent 20 minutes navigating between dashboards trying to correlate symptoms. Our mean time to resolution (MTTR) was 34 minutes, and most of that was diagnostic — finding the broken component, not fixing it.

We redesigned our CloudWatch metrics strategy around business-meaningful custom metrics with carefully chosen dimensions. MTTR dropped from 34 minutes to 11 minutes. The key was not more metrics, but the right metrics with the right dimensional structure.

The Problem: Default Metrics Tell You What, Not Why

AWS default metrics answer infrastructure questions: CPU is high, disk is full, network is saturated. They do not answer business questions:

• Which API endpoint is slow?
• Which tenant is experiencing errors?
• Which downstream dependency is failing?
• Is this affecting all users or just a subset?

Without custom metrics, every incident starts with the same question: "Where do I look?" Custom dimensions turn that 20-minute search into a 30-second filter operation.

Metric Design Philosophy: The Four Golden Signals + Business Context

We standardized on four golden signals (latency, traffic, errors, saturation) plus two business dimensions (tenant, feature). Every microservice emits the same metric shape:

Metric Name	Dimensions	Unit
api/latency	service, endpoint, method, status_class	Milliseconds
api/requests	service, endpoint, method, status_code	Count
api/errors	service, endpoint, error_type, tenant_tier	Count
dependency/latency	service, dependency, operation	Milliseconds
dependency/errors	service, dependency, error_type	Count
business/events	service, event_type, tenant_tier	Count

The dimension choices are intentional. We include tenant_tier (free, pro, enterprise) but not individual tenant_id — the cardinality would explode costs. We include status_class (2xx, 4xx, 5xx) for latency but status_code for request counts — different granularity for different use cases.

Implementation: Embedded Metric Format for Lambda

For Lambda-based services, CloudWatch Embedded Metric Format (EMF) is the most efficient approach. Metrics are emitted as structured log lines that CloudWatch automatically extracts:

import { MetricsLogger, createMetricsLogger } from 'aws-embedded-metrics';

interface RequestContext {
  service: string;
  endpoint: string;
  method: string;
  tenantTier: 'free' | 'pro' | 'enterprise';
}

class ObservabilityMiddleware {
  private metrics: MetricsLogger;

  constructor() {
    this.metrics = createMetricsLogger();
    this.metrics.setNamespace('ProductionServices');
  }

  async recordRequest(
    context: RequestContext,
    handler: () => Promise<{ statusCode: number; body: string }>
  ): Promise<{ statusCode: number; body: string }> {
    const startTime = Date.now();
    let statusCode = 500;
    let errorType: string | null = null;

    try {
      const response = await handler();
      statusCode = response.statusCode;
      return response;
    } catch (error) {
      errorType = (error as Error).constructor.name;
      throw error;
    } finally {
      const duration = Date.now() - startTime;
      const statusClass = `${Math.floor(statusCode / 100)}xx`;

      // Latency metric with service-level dimensions
      this.metrics.setDimensions({
        Service: context.service,
        Endpoint: context.endpoint,
        Method: context.method,
        StatusClass: statusClass,
      });
      this.metrics.putMetric('ApiLatency', duration, 'Milliseconds');

      // Request count with full status code
      this.metrics.setDimensions({
        Service: context.service,
        Endpoint: context.endpoint,
        StatusCode: String(statusCode),
      });
      this.metrics.putMetric('ApiRequests', 1, 'Count');

      // Error tracking with tenant context
      if (statusCode >= 500 || errorType) {
        this.metrics.setDimensions({
          Service: context.service,
          Endpoint: context.endpoint,
          ErrorType: errorType || `HTTP${statusCode}`,
          TenantTier: context.tenantTier,
        });
        this.metrics.putMetric('ApiErrors', 1, 'Count');
      }

      await this.metrics.flush();
    }
  }
}

EMF avoids the cost of PutMetricData API calls. Metrics are embedded in CloudWatch Logs (which you are already paying for) and extracted at no additional charge. The only cost is the custom metric itself.

Dependency Tracking: The Missing Layer

Most outages are caused by downstream dependency failures, not your own code. We instrument every external call:

import { NodeHttpHandler } from '@smithy/node-http-handler';

class InstrumentedHttpHandler extends NodeHttpHandler {
  private readonly metrics: MetricsLogger;
  private readonly serviceName: string;

  constructor(serviceName: string) {
    super({ connectionTimeout: 3000, requestTimeout: 10000 });
    this.metrics = createMetricsLogger();
    this.metrics.setNamespace('ProductionServices');
    this.serviceName = serviceName;
  }

  async handle(request: any, options?: any): Promise<any> {
    const dependency = this.extractDependencyName(request);
    const operation = this.extractOperation(request);
    const startTime = Date.now();

    try {
      const response = await super.handle(request, options);
      const duration = Date.now() - startTime;

      this.metrics.setDimensions({
        Service: this.serviceName,
        Dependency: dependency,
        Operation: operation,
      });
      this.metrics.putMetric('DependencyLatency', duration, 'Milliseconds');
      this.metrics.putMetric('DependencyRequests', 1, 'Count');

      return response;
    } catch (error) {
      const duration = Date.now() - startTime;

      this.metrics.setDimensions({
        Service: this.serviceName,
        Dependency: dependency,
        ErrorType: (error as Error).name,
      });
      this.metrics.putMetric('DependencyErrors', 1, 'Count');
      this.metrics.putMetric('DependencyLatency', duration, 'Milliseconds');

      throw error;
    } finally {
      await this.metrics.flush();
    }
  }

  private extractDependencyName(request: any): string {
    const host = request.hostname || 'unknown';
    if (host.includes('dynamodb')) return 'DynamoDB';
    if (host.includes('sqs')) return 'SQS';
    if (host.includes('s3')) return 'S3';
    return host.split('.')[0];
  }

  private extractOperation(request: any): string {
    return request.headers?.['x-amz-target']?.split('.').pop() || request.method;
  }
}

With dependency metrics, the incident workflow changes from "something is slow, let me check each dependency" to "DynamoDB GetItem latency spiked from 5ms to 800ms in the order-service — that is the root cause."

Alerting Strategy: Burn-Rate SLO Alerts

Traditional threshold alerts create noise. "Latency > 500ms" fires during every traffic spike even when the overall error budget is healthy. We switched to burn-rate alerts based on SLO error budgets:

SLO	Target	Monthly Budget	Alert (Fast Burn)	Alert (Slow Burn)
Availability	99.9%	43.8 min downtime	14.4x burn for 5 min	3x burn for 60 min
Latency P99	< 500ms	0.1% requests > 500ms	14.4x burn for 5 min	6x burn for 30 min
Error rate	< 0.1%	4,320 errors/month	14.4x burn for 2 min	3x burn for 60 min

A "14.4x fast burn" alert means: at this rate, you will exhaust your entire monthly error budget in 2 days. Act now. A "3x slow burn" means: at this rate, budget exhausts in 10 days. Investigate soon.

Cost Management: Keeping CloudWatch Affordable

Custom metrics cost $0.30 per metric per month. With 22 services, 6 metric names, and multiple dimension combinations, costs can explode. Our strategies:

1. Limit dimension cardinality — Never use unbounded values (user IDs, request IDs) as dimensions
2. Use metric math instead of new metrics — Derive error rates from existing request counts
3. EMF over PutMetricData — Zero API call cost for metric ingestion
4. Consolidate dimensions — Use status_class (5 values) instead of status_code (dozens) where detail is unnecessary

Monthly cost breakdown:

Component	Cost
Custom metrics (1,120 unique metrics)	$336
Dashboard (4 dashboards)	$12
Alarms (67 composite alarms)	$6.70
Metric Insights queries	$0 (within free tier)
Total	$354.70

Results: MTTR Transformation

After 60 days with the new observability stack:

Metric	Before	After	Improvement
Mean time to detect (MTTD)	8.2 min	2.1 min	74% faster
Mean time to diagnose	18.4 min	4.8 min	74% faster
Mean time to resolve (MTTR)	34 min	11 min	68% faster
False positive alerts/week	23	3	87% reduction
Dashboards (active use)	8 of 53	4 of 4	100% utilization
P1 incidents/month	4.2	4.0	-5% (detection, not prevention)

The 68% MTTR improvement came almost entirely from faster diagnosis. We did not have fewer incidents — we found the root cause faster because metrics told us exactly which service, endpoint, and dependency was misbehaving.

Lessons Learned

Four dashboards beat fifty-three. We deleted 49 dashboards. The four survivors: Service Overview (golden signals), Dependency Health (all external calls), Business Metrics (conversion, revenue), and On-Call (SLO burn rates). If a dashboard is not opened during incidents, it should not exist.

Dimension selection is a product decision, not a technical one. Which dimensions you choose determines what questions you can answer. Include tenant_tier because "is this affecting enterprise customers?" is the first question leadership asks. Exclude individual tenant_id because the cost would be $12,000/month.

Composite alarms eliminate noise. A single metric breaching a threshold is often transient. Composite alarms requiring multiple conditions (high error rate AND high latency AND dependency failure) only fire for real incidents. Our false positive rate dropped 87%.

Start with 5 metrics per service, not 50. Teams that instrument everything drown in data. Start with the four golden signals plus one business metric. Add dimensions only when an incident reveals a diagnostic gap.

Conclusion

Effective microservices observability on CloudWatch requires intentional metric design, not exhaustive instrumentation. The combination of structured custom metrics with carefully chosen dimensions, EMF for cost-efficient emission, and burn-rate SLO alerts gives you fast diagnosis without alert fatigue. Our $355/month observability stack across 22 services reduced MTTR by 68% — paying for itself within the first resolved incident each month.