local-aws/PROJECT_STRUCTURE.md

Local AWS - Project Structure Documentation

Overview

Local AWS is a NestJS-based emulator for AWS services. It implements AWS APIs locally for development and testing purposes, using SQLite for persistence and following AWS API specifications.

Key Architecture Points:

• Handler Pattern: Each AWS action is implemented as a separate handler class
• Modular Structure: One module per AWS service (IAM, KMS, S3, SNS, SQS, etc.)
• Multi-Server Architecture: Main app + separate S3 microservice + Consul KV microservice
  • S3 uses REST-style routing
  • Consul KV implements HashiCorp Consul API
• Database: SQLite with Prisma ORM
• Test Structure: Each module has its own test suite using AWS SDK or service-specific clients

---

Application Entry Points

Main Entry (src/main.ts)

Bootstraps two NestJS applications:

1. Main Application (default port 8081)

   • Handles IAM, KMS, SecretsManager, SNS, SQS, STS services
   • Uses query-based routing (Action parameter in body/query)
   • Body parsers: JSON (for SNS/SQS), XML, raw binary

2. S3 Microservice (default port 4567)

   • Separate app due to S3's REST/path-style routing
   • Routes based on HTTP method + path + query parameters
   • Body parsers: raw binary, URL-encoded

3. Consul KV Microservice (default port 8500)

   • Implements HashiCorp Consul Key-Value store API
   • REST-style routing at /v1/kv/* endpoints
   • Supports recursive queries, CAS operations, locks, flags
   • Body parsers: JSON, raw binary, text/plain

Key Global Features:

Date.prototype.getAwsTime(); // Extension for AWS timestamp format

---

Module Architecture

Module Structure Pattern

Each AWS service follows this structure:

<service>/
├── __tests__/
│   └── <service>.spec.ts              # Integration tests using AWS SDK
├── <action>-handler.ts                 # One handler per AWS action
├── <service>.constants.ts              # Action enum list & injection tokens
├── <service>.module.ts                 # NestJS module definition
├── <service>.service.ts                # Business logic layer
├── <service>-<entity>.entity.ts        # Prisma model wrapper classes
└── (optional) <service>.controller.ts  # Custom controller if needed

Current Modules

Main Application Modules

1. IAM (src/iam/)

   • Handlers: CreateRole, GetRole, DeleteRole, CreatePolicy, GetPolicy, AttachRolePolicy, PutRolePolicy, GetRolePolicy, etc.
   • Entities: IamRole, IamPolicy, IamRoleInlinePolicy
   • Database: Roles, policies, role-policy attachments, inline policies

2. KMS (src/kms/)

   • Handlers: CreateKey, DescribeKey, CreateAlias, Sign, GetPublicKey, EnableKeyRotation, etc.
   • Entities: KmsKey, KmsAlias
   • Database: KMS keys, aliases

3. Secrets Manager (src/secrets-manager/)

   • Handlers: CreateSecret, GetSecretValue, PutSecretValue, DeleteSecret, etc.
   • Database: Secrets with versioning

4. SNS (src/sns/)

   • Handlers: CreateTopic, Subscribe, Publish, etc.
   • Database: Topics, subscriptions

5. SQS (src/sqs/)

   • Handlers: CreateQueue, SendMessage, ReceiveMessage, etc.
   • Database: Queues, messages

6. STS (src/sts/)

   • Handlers: AssumeRole, GetCallerIdentity, etc.
   • Temporary credential generation

S3 Microservice Module

S3 (src/s3/)

• Special module with its own app (S3AppModule) and controller
• Handlers: CreateBucket, ListBuckets, PutObject, GetObject, DeleteObject, PutBucketAcl, GetBucketAcl, etc.
• Entities: S3Bucket, S3Object
• Custom routing logic in S3Controller based on HTTP method + path + query params
• Database: Buckets with tags/ACL/policy, objects with metadata

Consul KV Microservice Module

Consul KV (src/consul-kv/)

• Special module with its own app (ConsulKVAppModule) and controller
• Implements HashiCorp Consul KV API spec: https://developer.hashicorp.com/consul/api-docs/kv
• Service: ConsulKVService with getKey, putKey, deleteKey, listKeys, recursive operations
• Custom routing in ConsulKVController for /v1/kv/* paths
• Features:
  • Key-value storage with base64 encoding
  • Recursive queries with prefix matching
  • Keys-only listing with separator support
  • Check-And-Set (CAS) operations
  • Distributed lock acquisition/release
  • Flags and index tracking (createIndex, modifyIndex, lockIndex)
  • Multi-tenancy support (datacenter, namespace)
• Database: ConsulKVEntry model
• Tests: 19 integration tests using consul npm package

---

Handler Pattern Deep Dive

AbstractActionHandler (src/abstract-action.handler.ts)

Base class for all action handlers. All handlers must extend this class.

Required Properties:

abstract class AbstractActionHandler<T> {
  format: Format; // Format.Xml or Format.Json
  action: Action | Action[]; // AWS action enum(s)
  validator: Joi.ObjectSchema; // Request validation schema

  protected abstract handle(
    queryParams: T, // Validated query parameters
    context: RequestContext, // AWS properties + request metadata
  ): Record<string, any> | void;
}

Response Handling:

• XML format: Wraps result in {Action}Result node with RequestId metadata
• JSON format: Returns result directly
• Empty response: Returns ResponseMetadata only (for XML) or nothing (for JSON)

Handler Implementation Example

@Injectable()
export class CreateRoleHandler extends AbstractActionHandler<QueryParams> {
  constructor(private readonly iamService: IamService) {
    super();
  }

  format = Format.Xml;
  action = Action.IamCreateRole;

  validator = Joi.object<QueryParams>({
    RoleName: Joi.string().required(),
    Path: Joi.string().required(),
    AssumeRolePolicyDocument: Joi.string().required(),
    // ... other parameters
  });

  protected async handle(params: QueryParams, { awsProperties }: RequestContext) {
    const role = await this.iamService.createRole({
      accountId: awsProperties.accountId,
      name: params.RoleName,
      // ...
    });
    return { Role: role.metadata };
  }
}

Handler Registration Flow

1. Define handler class extending AbstractActionHandler
2. Add to module's handlers array in <service>.module.ts
3. ExistingActionHandlersProvider collects all handlers into a map keyed by Action enum
4. DefaultActionHandlerProvider fills gaps with stub handlers for unimplemented actions
5. Module provides injection token (e.g., IAMHandlers, S3Handlers) containing the handler map
6. Controller receives handler map and routes requests to appropriate handler

// In module
const handlers = [CreateRoleHandler, GetRoleHandler, /* ... */];

@Module({
  providers: [
    ...handlers,
    ExistingActionHandlersProvider(handlers),
    DefaultActionHandlerProvider(IAMHandlers, Format.Xml, allIamActions),
  ]
})

---

Controllers

Main Controller (src/app.controller.ts)

Routes all non-S3 AWS service requests.

Request Flow:

1. Extracts action from x-amz-target header or Action body parameter
2. Validates action exists in Action enum
3. Looks up handler from injected ActionHandlers map
4. Validates request params using handler's Joi validator
5. Calls handler.getResponse(params, context)
6. Returns XML or JSON based on handler's format

S3 Controller (src/s3/s3.controller.ts)

Custom controller for S3 REST-style API (separate microservice).

Request Flow:

1. Parses path into bucket/key components
2. Determines action from HTTP method + path + query parameters
   • PUT /{bucket} → CreateBucket
   • GET /{bucket} → ListObjects (or other sub-resource if query param present)
   • PUT /{bucket}?acl → PutBucketAcl
   • GET /{bucket}/{key} → GetObject
3. Normalizes query parameters (e.g., max-keys → MaxKeys)
4. Extracts metadata from x-amz-meta-* headers
5. Converts binary body to base64 for handlers
6. Routes to handler and returns response with appropriate headers

Special S3 Response Handling:

• Creates Location header for bucket creation
• Returns ETag, Last-Modified, Content-Type headers
• Returns raw binary data for GetObject (not XML/JSON)
• Returns empty 200 for PutBucketAcl, PutBucketTagging
• Returns 204 for DeleteBucket, DeleteObject

Consul KV Controller (src/consul-kv/consul-kv.controller.ts)

Custom controller for Consul KV REST API (separate microservice).

Request Flow:

1. Catches all /v1/kv/* paths
2. Extracts and URL-decodes the key from the path
3. Routes based on HTTP method (GET/PUT/DELETE)
4. Parses query parameters:
   • dc: datacenter (default: dc1)
   • ns: namespace (default: default)
   • recurse: recursive key retrieval
   • keys: return only key names
   • raw: return raw value (not JSON)
   • separator: group keys by separator
   • flags: numeric flags for application use
   • cas: Check-And-Set with modify index
   • acquire: session ID for lock acquisition
   • release: session ID for lock release
5. Processes request through ConsulKVService
6. Returns appropriate response format

Special Consul Response Handling:

• PUT/DELETE return plain text "true" or "false"
• GET returns JSON array of key metadata or raw value
• Keys-only returns JSON array of key strings
• 404 status for non-existent keys
• Base64 encoding for values in JSON responses
• Supports keys with slashes (proper URL decoding)

---

Service Layer

Each module has a service class that:

• Encapsulates business logic
• Interacts with Prisma for database operations
• Validates entity constraints
• Throws AWS-compatible exceptions

Example: IAM Service (src/iam/iam.service.ts)

@Injectable()
export class IamService {
  constructor(private readonly prisma: PrismaService) {}

  async createRole(data: CreateRoleInput): Promise<IamRole> {
    // Check for duplicates
    // Create in database
    // Return entity
  }

  async putRoleInlinePolicy(...): Promise<void> {
    // Verify role exists
    // Upsert policy
  }
}

---

Entity Layer

Entity classes wrap Prisma models and provide:

• Type safety
• Computed properties (e.g., ARNs, metadata formatters)
• Serialization logic

Example: IAM Role Entity (src/iam/iam-role.entity.ts)

export class IamRole implements PrismaIamRole {
  id: string;
  name: string;
  path: string;
  accountId: string;
  assumeRolePolicy: string;
  // ...

  get arn(): string {
    return `arn:aws:iam::${this.accountId}:role${this.path}${this.name}`;
  }

  get metadata() {
    // Returns AWS API response format
  }
}

---

Database Layer

Prisma Setup (src/_prisma/)

• prisma.service.ts: PrismaClient wrapper with connection lifecycle
• prisma.module.ts: Exports PrismaService globally

Schema Location: prisma/schema.prisma

Key Models:

• IamRole, IamPolicy, IamRoleIamPolicyAttachment, IamRoleInlinePolicy
• KmsKey, KmsAlias
• S3Bucket, S3Object
• SecretsManagerSecret
• SnsTopic, SnsSubscription
• SqsQueue
• ConsulKVEntry

Migrations: prisma/migrations/

---

Shared Infrastructure

AWS Shared Entities (src/aws-shared-entities/)

• aws-exceptions.ts: All AWS exception classes
  • Base AwsException with toXml() and toJson() methods
  • Specific exceptions: NoSuchEntity, ValidationError, AccessDeniedException, etc.
• attributes.service.ts: Common attribute parsing/validation
• tags.service.ts: Resource tagging utilities

Request Context (src/_context/)

• request.context.ts: TypeScript interfaces for request context

  interface RequestContext {
    action?: Action;
    format?: Format;
    awsProperties: AwsProperties; // accountId, region, host
    requestId: string;
  }
  

• exception.filter.ts: Global exception filter for AWS exceptions

Audit System (src/audit/)

• audit.interceptor.ts: Logs requests/responses (main app)
• s3-audit.interceptor.ts: Logs S3 requests/responses
• audit.service.ts: Audit persistence service (currently minimal)

Configuration (src/config/)

• local.config.ts: Loads environment variables
• config.validator.ts: Validates config on startup
• common-config.interface.ts: TypeScript interface for config

Environment Variables:

• AWS_ACCOUNT_ID (default: 000000000000)
• AWS_REGION (default: us-east-1)
• PORT (default: 8081) - Main app port
• S3_PORT (default: 4567) - S3 microservice port
• CONSUL_PORT (default: 8500) - Consul KV microservice port
• DATABASE_URL (default: :memory:) - SQLite connection string
• HOST, PROTO - Used for URL generation

Default Action Handler (src/default-action-handler/)

Provides stub implementations for unimplemented actions.

• default-action-handler.provider.ts: Creates default handlers that throw UnsupportedOperationException
• existing-action-handlers.provider.ts: Collects implemented handlers into map
• default-action-handler.constants.ts: Shared constants

Purpose: Allows modules to declare all AWS actions in their enum list, then provides automatic "not implemented" responses for actions without handlers.

---

Action Enum (src/action.enum.ts)

Central enum containing all AWS actions across all services:

export enum Action {
  // IAM
  IamCreateRole = 'CreateRole',
  IamGetRole = 'GetRole',
  IamPutRolePolicy = 'PutRolePolicy',

  // KMS
  KmsCreateKey = 'CreateKey',
  KmsSign = 'Sign',

  // S3
  S3CreateBucket = 'CreateBucket',
  S3GetObject = 'GetObject',
  S3PutBucketAcl = 'PutBucketAcl',

  // ... all other actions
}

---

Testing Strategy

Test Structure

Each module has a dedicated test suite in src/<module>/__tests__/<module>.spec.ts

Test Pattern:

1. Create isolated test app with in-memory SQLite database
2. Use actual AWS SDK clients pointing to local endpoints
3. Test against real AWS API contracts
4. Verify both API responses and database state

Example Test Setup:

describe('IAM Integration Tests', () => {
  let app: INestApplication;
  let iamClient: IAMClient;
  let prismaService: PrismaService;

  beforeAll(async () => {
    // Unique in-memory database per test suite
    process.env.DATABASE_URL = `file::memory:?cache=shared&unique=${Date.now()}`;

    const moduleFixture = await Test.createTestingModule({
      imports: [AppModule],
    }).compile();

    app = moduleFixture.createNestApplication();
    await app.init();

    // Real AWS SDK client pointing to local endpoint
    iamClient = new IAMClient({
      endpoint: `http://localhost:${testPort}`,
      credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
    });
  });

  it('should create a role', async () => {
    const response = await iamClient.send(new CreateRoleCommand({ ... }));
    expect(response.Role.RoleName).toBe('TestRole');
  });
});

Test Organization:

• src/iam/__tests__/iam.spec.ts - IAM tests
• src/s3/__tests__/s3.spec.ts - S3 tests
• src/kms/__tests__/kms.spec.ts - KMS tests
• etc.

Current Test Count: 222 tests across 8 test suites

---

Key Patterns & Conventions

1. Handler Naming

• Handlers named after AWS action: <Action>Handler (e.g., CreateRoleHandler)
• One handler per file: <action>-handler.ts (kebab-case)
• Handler must be Injectable and registered in module

2. Service Layer

• One service per module: <service>.service.ts
• Service injected into handlers via constructor
• Services interact with Prisma, not handlers directly

3. Entity Layer

• Entities wrap Prisma models: <service>-<entity>.entity.ts
• Implement Prisma type: implements Prisma<Entity>
• Provide computed properties and formatting methods

4. Module Constants

• <service>.constants.ts contains:
  • Injection token: export const ServiceHandlers = Symbol('ServiceHandlers');
  • Action list: export const serviceActions: Action[] = [...]

5. Error Handling

• Throw AWS exception classes from aws-shared-entities/aws-exceptions.ts
• Exception filter converts to XML or JSON automatically
• Include descriptive messages matching AWS API

6. Database Conventions

• Use Prisma for all database operations
• Models use camelCase, match entity class names
• Migrations created via npx prisma migrate dev --name <description>
• Always include accountId for multi-tenancy support

7. Request Validation

• Use Joi schemas in handler's validator property
• Validate early, fail fast with ValidationError
• Allow unknown properties for forward compatibility

---

Development Workflow

Adding a New Handler

1. Create handler file src/<service>/<action>-handler.ts

   @Injectable()
   export class MyActionHandler extends AbstractActionHandler<QueryParams> {
     format = Format.Xml;
     action = Action.ServiceMyAction;
     validator = Joi.object({
       /* ... */
     });
     protected async handle(params, context) {
       /* ... */
     }
   }
   

2. Add to module in src/<service>/<service>.module.ts

   const handlers = [
     // ... existing handlers
     MyActionHandler,
   ];
   

3. Implement service method if needed in src/<service>/<service>.service.ts

4. Add tests in src/<service>/__tests__/<service>.spec.ts

   it('should perform my action', async () => {
     const response = await client.send(new MyActionCommand({ ... }));
     expect(response).toBeDefined();
   });
   

5. Run tests npm test

Adding a New AWS Service Module

1. Create directory src/<service>/

2. Create files:

   • <service>.module.ts - NestJS module
   • <service>.service.ts - Business logic
   • <service>.constants.ts - Injection tokens and action list
   • <handler>-handler.ts - Handler implementations
   • <service>-<entity>.entity.ts - Entity classes
   • __tests__/<service>.spec.ts - Tests

3. Update src/action.enum.ts with service actions

4. Update src/app.module.ts to import new module

5. Create Prisma models in prisma/schema.prisma

6. Run migration npx prisma migrate dev --name add_<service>

7. Implement handlers following handler pattern

8. Write tests using AWS SDK client

---

Common Troubleshooting

Handler not found

• Check handler is in module's handlers array
• Verify action enum matches exactly
• Ensure handler is Injectable

Database errors

• Run npx prisma generate after schema changes
• Check migration applied: npx prisma migrate dev
• Verify Prisma model matches entity class

Validation errors

• Check Joi schema matches AWS API requirements
• Verify parameter casing (AWS uses PascalCase)
• Test with actual AWS SDK client

S3 routing issues

• S3 uses different routing logic (HTTP method + path)
• Check determineS3Action() in S3Controller
• Verify query parameter handling (e.g., ?acl)

Test failures

• Ensure unique database: file::memory:?cache=shared&unique=...
• Check port conflicts (8086-8090 commonly used)
• Clean up resources in afterEach hooks

---

Architecture Decisions

Why Separate S3 and Consul KV Microservices?

S3 Microservice: S3 uses REST-style routing (path-based) rather than action-based routing. The main app routes based on Action parameter, while S3 needs to parse URLs like /{bucket}/{key} and route based on HTTP method + query parameters.

Consul KV Microservice: Consul KV implements the HashiCorp Consul API specification, which uses REST-style routing at /v1/kv/* endpoints. It requires different routing logic, query parameter handling, and response formats than AWS services. Running it as a separate microservice allows:

• Clean separation of concerns between AWS and Consul APIs
• Different port (8500) matching standard Consul deployment
• Independent body parsing and response formatting
• Compatibility with existing Consul client libraries

Why Handler Pattern?

• Modularity: Each AWS action is isolated
• Testability: Easy to test individual handlers
• Scalability: Easy to add new actions
• Type Safety: Each handler has its own request type
• Validation: Built-in Joi validation per action

Why Prisma + SQLite?

• Simplicity: No external database required
• Type Safety: Generated TypeScript types
• Migrations: Schema versioning out of the box
• Performance: Fast for local development/testing
• Portability: Single file database or in-memory

Why Dual Format Support (XML/JSON)?

Different AWS services use different protocols:

• IAM, KMS: XML (AWS Query protocol)
• SNS, SQS: JSON (AWS JSON protocol)
• S3: XML with custom headers

---

File Organization Summary

src/
├── main.ts                          # Application bootstrap
├── app.module.ts                    # Main app module
├── app.controller.ts                # Main API controller
├── abstract-action.handler.ts       # Base handler class
├── action.enum.ts                   # All AWS actions
├── app.constants.ts                 # App-level constants
│
├── _context/                        # Request context types
├── _prisma/                         # Prisma client wrapper
├── audit/                           # Request/response logging
├── aws-shared-entities/             # AWS exceptions & utilities
├── config/                          # Environment configuration
├── default-action-handler/          # Stub handler generation
│
├── iam/                             # IAM service module
│   ├── __tests__/
│   ├── *-handler.ts                 # Action handlers
│   ├── iam.service.ts               # Business logic
│   ├── iam.module.ts                # Module definition
│   ├── iam.constants.ts             # Constants
│   └── *.entity.ts                  # Entity classes
│
├── s3/                              # S3 service module (microservice)
│   ├── __tests__/
│   ├── s3-app.module.ts             # Separate app module
│   ├── s3.controller.ts             # Custom controller
│   ├── *-handler.ts                 # Action handlers
│   └── ...
│
├── consul-kv/                       # Consul KV module (microservice)
│   ├── __tests__/
│   ├── consul-kv-app.module.ts      # Separate app module
│   ├── consul-kv.controller.ts      # Custom controller
│   ├── consul-kv.service.ts         # Business logic
│   └── ...
│
├── kms/                             # KMS service module
├── secrets-manager/                 # Secrets Manager module
├── sns/                             # SNS service module
├── sqs/                             # SQS service module
└── sts/                             # STS service module

prisma/
├── schema.prisma                    # Database schema
└── migrations/                      # Schema migrations

---

Quick Reference

Start Development:

PORT=8081 S3_PORT=4567 CONSUL_PORT=8500 yarn start:dev

Run Tests:

npm test                           # All tests
npm test -- iam.spec.ts           # Specific suite

Database Commands:

npx prisma migrate dev --name <description>
npx prisma generate
npx prisma studio                 # GUI database browser

Test AWS CLI:

# IAM
aws iam create-role --role-name TestRole --assume-role-policy-document '{}' \
  --endpoint-url http://localhost:8081

# S3
aws s3 mb s3://my-bucket --endpoint-url http://localhost:4567
aws s3 ls s3://my-bucket --endpoint-url http://localhost:4567

Test Consul API:

# Using curl
curl -X PUT http://localhost:8500/v1/kv/my-key -d 'my-value'
curl http://localhost:8500/v1/kv/my-key
curl -X DELETE http://localhost:8500/v1/kv/my-key

# Using consul CLI (if installed)
export CONSUL_HTTP_ADDR=http://localhost:8500
consul kv put my-key my-value
consul kv get my-key
consul kv delete my-key

---

This documentation should be your starting point for understanding the project. Read this first before making changes to understand the patterns and conventions in use.