Gateway Certificates Retrieval & Distribution Automation System

Overview

The system automates the retrieval, packaging, and distribution of gateway certificates and shared configuration files (servertrust and endpoints) for each warehouse once they are made available by the GREF provisioning workflow. The system performs periodic checks at configurable intervals to detect new or updated files uploaded by GREF, ensuring each warehouse automatically receives the latest certificates and shared files without any manual intervention.

The system is highly scalable, robust, and efficient, capable of handling multiple warehouses and large volumes of gateway certificates simultaneously, while providing automated error handling, concurrent processing, and transparent logging.

Goals

• Retrieval: Automatically retrieve gateway certificates and shared files once they are provided by GREF.

• Packaging: Efficiently package certificates and automatically handle large file batches by splitting ZIPs when size limits are exceeded.

• Distribution: Maintain strict warehouse-level segregation and ensure emails include only files relevant to the respective warehouse.

• Monitoring & Logging: Provide transparent monitoring and detailed logging for audit and troubleshooting.

Architecture

Architecture Diagram

Open

Automated Gateway Certificates Retrieval System

GREF AWS Accounts (for sharing certificates with Milvian)

• Test Hub Account: 606635532841

• Production Hub Account: 172670235729 (All infrastructure described below is deployed in this account, us-east-1 region)

1. S3 Bucket

• Bucket Name: metering-project-files-us-east-1-172670235729

Purpose:

• Acts as the single source of truth for all gateway certificates and shared files.

• The GREF provisioning workflow uploads:

  • Gateway certificates (.pem and .key)

  • Shared files (servertrust_*.pem and endpoints_*.json)

Behavior:

• Any new object uploaded triggers an S3 notification → invokes the gateway-cert-bucket-processor Lambda.

• There are four warehouses (Warehouse_US, Warehouse_UK, Warehouse_EU, Warehouse_India).

• Each warehouse has its own gateway certificates, while shared files remain constant per warehouse and region.

• Shared files include region and account information in their filenames, whereas gateway certificates are named using the last 5 digits of the GatewayEUI

Folder Structure Example:

• Shared files always reside at the warehouse root, not inside individual gateway folders.

2. DynamoDB Table

• Table name: EmailGatewayCerts

Purpose:

• Tracks email delivery status per warehouse and per gateway (or shared files).

Example item:

3. Lambda Functions

(a) gateway-cert-bucket-processor

• Triggered by: s3:ObjectCreated:* in the bucket

• Purpose: Detects new or updated gateway/shared files.

Key Actions:

• Identifies file type (gateway vs shared).

• Updates the corresponding DynamoDB record:

  • Sets status → PENDING_EMAIL

  • Updates LastModified

• Logs consolidated information per gateway or shared group in CloudWatch.

(b) gateway-cert-email-processor

• Triggered by: EventBridge rule (every 15 minutes)

• Purpose: Sends certificate and shared file updates via email.

Key Actions:

• Queries DynamoDB for PENDING_EMAIL entries.

• Downloads all relevant gateway and shared files concurrently.

• Creates ZIP bundles (splitting automatically if size > 8 MB to stay below SES limits).

• Sends emails via SES to warehouse distribution addresses.

  • Default recipient: gatewaycerts@milvian.group (to be replaced with warehouse-specific shared emails in future)

  • Sender: no-reply@prod.gtsb.gref.amazon.dev

• Updates the corresponding DynamoDB records:

  • EMAILED → on success

  • FAILED_EMAIL → on failure

  • Updates LastEmailed

• Logs a summary for each warehouse in CloudWatch.

4. Email Handling & Subjects

Rules & Behavior:

• Emails are sent only if at least one gateway certificate is available per warehouse.

• Shared files are always included by default whenever gateways are updated.

• Emails may also be sent when only shared files are updated by GREF (rare scenario).

Email Subjects:

• Warehouse_<X>: Gateway Certificates and Server/Endpoints Files

  • Sent when any gateway certificate is added/updated

  • Shared files included by default

• Warehouse_<X>: Server and Endpoints Files

  • Rare case, sent when GREF updates shared files alone after initial upload

Notes:

• The system ensures warehouse-level segregation; only files relevant to the warehouse are included.

• Emails are bundled into ZIPs, splitting automatically if size > 8 MB.

5. EventBridge Rule

• Name: gateway-cert-email-processor-lambda-event-bridge-rule

• Frequency: Every 15 minutes

• Target: gateway-cert-email-processor

• Purpose: Triggers periodic email processing for newly updated files.

6. Amazon SES

Handles outbound emails containing:

• Warehouse name

• Number of gateways included

• List of updated gateways

• ZIP files

Scenarios:

• Gateway updates detected: Includes gateway certs + shared files

• Only shared files updated: Includes only shared files (rare scenario)

Workflow

Step 1 — File Upload

• New or updated file uploaded to S3 by GREF triggers gateway-cert-bucket-processor.

• Determines file type (gateway/shared)

• Marks status → PENDING_EMAIL in DynamoDB

• Updates LastEmailed timestamp in DynamoDB

• Shared files trigger emails only if at least one gateway certificate is included

Step 2 — Scheduled Email Processing

• Every 15 minutes, gateway-cert-email-processor:

  • Scans DynamoDB for PENDING_EMAIL entries

  • Downloads all relevant gateway folders + shared files

  • Bundles and splits ZIPs (>8 MB)

  • Sends consolidated SES emails

  • Updates DynamoDB with status and LastEmailed timestamp

Step 3 — File & Email Logging

• Example logs:

  Downloaded shared files for Warehouse_US: servertrust.pem, endpoints.json Downloaded files for gateway 00800000d000bc03: 0bc03.pem, 0bc03.pem Created zip: /tmp/Warehouse_US_20251030_084628.zip with 4 files ✅ Email sent to gatewaycerts@milvian.group for file Warehouse_US_20251030_084628.zip

Error Handling & Monitoring

Special Features

• Concurrent file downloads for efficiency, using up to 10 parallel workers.

• Automatic ZIP splitting (>8 MB) to respect SES 10 MB limit

• Automatic retries for FAILED_EMAIL (3–5 attempts via Lambda)

• Scalable, fault-tolerant, and fully automated - designed for large-scale provisioning and warehouse distribution

Conclusion

A robust and scalable automation system that ensures secure, reliable, and efficient delivery of gateway certificates and shared configuration files - eliminating manual steps and ensuring consistent synchronization across all warehouses.