local-aws/PROJECT_STRUCTURE.md

# Local AWS - Project Structure Documentation

## Overview

Local AWS is a NestJS-based emulator for AWS services. It implements AWS APIs locally for development and testing purposes, using SQLite for persistence and following AWS API specifications.

**Key Architecture Points:**

- **Handler Pattern**: Each AWS action is implemented as a separate handler class
- **Modular Structure**: One module per AWS service (IAM, KMS, S3, SNS, SQS, etc.)
- **Multi-Server Architecture**: Main app + separate S3 microservice + Consul KV microservice
  - S3 uses REST-style routing
  - Consul KV implements HashiCorp Consul API
- **Database**: SQLite with Prisma ORM
- **Test Structure**: Each module has its own test suite using AWS SDK or service-specific clients

---

## Application Entry Points

### Main Entry (`src/main.ts`)

Bootstraps two NestJS applications:

1. **Main Application** (default port 8081)

   - Handles IAM, KMS, SecretsManager, SNS, SQS, STS services
   - Uses query-based routing (Action parameter in body/query)
   - Body parsers: JSON (for SNS/SQS), XML, raw binary

2. **S3 Microservice** (default port 4567)

   - Separate app due to S3's REST/path-style routing
   - Routes based on HTTP method + path + query parameters
   - Body parsers: raw binary, URL-encoded

3. **Consul KV Microservice** (default port 8500)
   - Implements HashiCorp Consul Key-Value store API
   - REST-style routing at `/v1/kv/*` endpoints
   - Supports recursive queries, CAS operations, locks, flags
   - Body parsers: JSON, raw binary, text/plain

**Key Global Features:**

```typescript
Date.prototype.getAwsTime(); // Extension for AWS timestamp format
```

---

## Module Architecture

### Module Structure Pattern

Each AWS service follows this structure:

```
<service>/
├── __tests__/
│   └── <service>.spec.ts              # Integration tests using AWS SDK
├── <action>-handler.ts                 # One handler per AWS action
├── <service>.constants.ts              # Action enum list & injection tokens
├── <service>.module.ts                 # NestJS module definition
├── <service>.service.ts                # Business logic layer
├── <service>-<entity>.entity.ts        # Prisma model wrapper classes
└── (optional) <service>.controller.ts  # Custom controller if needed
```

### Current Modules

#### Main Application Modules

1. **IAM** (`src/iam/`)

   - Handlers: CreateRole, GetRole, DeleteRole, CreatePolicy, GetPolicy, AttachRolePolicy, PutRolePolicy, GetRolePolicy, etc.
   - Entities: IamRole, IamPolicy, IamRoleInlinePolicy
   - Database: Roles, policies, role-policy attachments, inline policies

2. **KMS** (`src/kms/`)

   - Handlers: CreateKey, DescribeKey, CreateAlias, Sign, GetPublicKey, EnableKeyRotation, etc.
   - Entities: KmsKey, KmsAlias
   - Database: KMS keys, aliases

3. **Secrets Manager** (`src/secrets-manager/`)

   - Handlers: CreateSecret, GetSecretValue, PutSecretValue, DeleteSecret, etc.
   - Database: Secrets with versioning

4. **SNS** (`src/sns/`)

   - Handlers: CreateTopic, Subscribe, Publish, etc.
   - Database: Topics, subscriptions

5. **SQS** (`src/sqs/`)

   - Handlers: CreateQueue, SendMessage, ReceiveMessage, etc.
   - Database: Queues, messages

6. **STS** (`src/sts/`)
   - Handlers: AssumeRole, GetCallerIdentity, etc.
   - Temporary credential generation

#### S3 Microservice Module

**S3** (`src/s3/`)

- Special module with its own app (`S3AppModule`) and controller
- Handlers: CreateBucket, ListBuckets, PutObject, GetObject, DeleteObject, PutBucketAcl, GetBucketAcl, etc.
- Entities: S3Bucket, S3Object
- Custom routing logic in `S3Controller` based on HTTP method + path + query params
- Database: Buckets with tags/ACL/policy, objects with metadata

#### Consul KV Microservice Module

**Consul KV** (`src/consul-kv/`)

- Special module with its own app (`ConsulKVAppModule`) and controller
- Implements HashiCorp Consul KV API spec: https://developer.hashicorp.com/consul/api-docs/kv
- Service: ConsulKVService with getKey, putKey, deleteKey, listKeys, recursive operations
- Custom routing in `ConsulKVController` for `/v1/kv/*` paths
- Features:
  - Key-value storage with base64 encoding
  - Recursive queries with prefix matching
  - Keys-only listing with separator support
  - Check-And-Set (CAS) operations
  - Distributed lock acquisition/release
  - Flags and index tracking (createIndex, modifyIndex, lockIndex)
  - Multi-tenancy support (datacenter, namespace)
- Database: ConsulKVEntry model
- Tests: 19 integration tests using `consul` npm package

---

## Handler Pattern Deep Dive

### AbstractActionHandler (`src/abstract-action.handler.ts`)

Base class for all action handlers. All handlers must extend this class.

**Required Properties:**

```typescript
abstract class AbstractActionHandler<T> {
  format: Format; // Format.Xml or Format.Json
  action: Action | Action[]; // AWS action enum(s)
  validator: Joi.ObjectSchema; // Request validation schema

  protected abstract handle(
    queryParams: T, // Validated query parameters
    context: RequestContext, // AWS properties + request metadata
  ): Record<string, any> | void;
}
```

**Response Handling:**

- XML format: Wraps result in `{Action}Result` node with RequestId metadata
- JSON format: Returns result directly
- Empty response: Returns ResponseMetadata only (for XML) or nothing (for JSON)

### Handler Implementation Example

```typescript
@Injectable()
export class CreateRoleHandler extends AbstractActionHandler<QueryParams> {
  constructor(private readonly iamService: IamService) {
    super();
  }

  format = Format.Xml;
  action = Action.IamCreateRole;

  validator = Joi.object<QueryParams>({
    RoleName: Joi.string().required(),
    Path: Joi.string().required(),
    AssumeRolePolicyDocument: Joi.string().required(),
    // ... other parameters
  });

  protected async handle(params: QueryParams, { awsProperties }: RequestContext) {
    const role = await this.iamService.createRole({
      accountId: awsProperties.accountId,
      name: params.RoleName,
      // ...
    });
    return { Role: role.metadata };
  }
}
```

### Handler Registration Flow

1. **Define handler class** extending `AbstractActionHandler`
2. **Add to module's handlers array** in `<service>.module.ts`
3. **ExistingActionHandlersProvider** collects all handlers into a map keyed by Action enum
4. **DefaultActionHandlerProvider** fills gaps with stub handlers for unimplemented actions
5. **Module provides injection token** (e.g., `IAMHandlers`, `S3Handlers`) containing the handler map
6. **Controller receives handler map** and routes requests to appropriate handler

```typescript
// In module
const handlers = [CreateRoleHandler, GetRoleHandler, /* ... */];

@Module({
  providers: [
    ...handlers,
    ExistingActionHandlersProvider(handlers),
    DefaultActionHandlerProvider(IAMHandlers, Format.Xml, allIamActions),
  ]
})
```

---

## Controllers

### Main Controller (`src/app.controller.ts`)

Routes all non-S3 AWS service requests.

**Request Flow:**

1. Extracts action from `x-amz-target` header or `Action` body parameter
2. Validates action exists in Action enum
3. Looks up handler from injected ActionHandlers map
4. Validates request params using handler's Joi validator
5. Calls `handler.getResponse(params, context)`
6. Returns XML or JSON based on handler's format

### S3 Controller (`src/s3/s3.controller.ts`)

Custom controller for S3 REST-style API (separate microservice).

**Request Flow:**

1. Parses path into bucket/key components
2. Determines action from HTTP method + path + query parameters
   - `PUT /{bucket}` → CreateBucket
   - `GET /{bucket}` → ListObjects (or other sub-resource if query param present)
   - `PUT /{bucket}?acl` → PutBucketAcl
   - `GET /{bucket}/{key}` → GetObject
3. Normalizes query parameters (e.g., `max-keys` → `MaxKeys`)
4. Extracts metadata from `x-amz-meta-*` headers
5. Converts binary body to base64 for handlers
6. Routes to handler and returns response with appropriate headers

**Special S3 Response Handling:**

- Creates Location header for bucket creation
- Returns ETag, Last-Modified, Content-Type headers
- Returns raw binary data for GetObject (not XML/JSON)
- Returns empty 200 for PutBucketAcl, PutBucketTagging
- Returns 204 for DeleteBucket, DeleteObject

### Consul KV Controller (`src/consul-kv/consul-kv.controller.ts`)

Custom controller for Consul KV REST API (separate microservice).

**Request Flow:**

1. Catches all `/v1/kv/*` paths
2. Extracts and URL-decodes the key from the path
3. Routes based on HTTP method (GET/PUT/DELETE)
4. Parses query parameters:
   - `dc`: datacenter (default: dc1)
   - `ns`: namespace (default: default)
   - `recurse`: recursive key retrieval
   - `keys`: return only key names
   - `raw`: return raw value (not JSON)
   - `separator`: group keys by separator
   - `flags`: numeric flags for application use
   - `cas`: Check-And-Set with modify index
   - `acquire`: session ID for lock acquisition
   - `release`: session ID for lock release
5. Processes request through ConsulKVService
6. Returns appropriate response format

**Special Consul Response Handling:**

- PUT/DELETE return plain text "true" or "false"
- GET returns JSON array of key metadata or raw value
- Keys-only returns JSON array of key strings
- 404 status for non-existent keys
- Base64 encoding for values in JSON responses
- Supports keys with slashes (proper URL decoding)

---

## Service Layer

Each module has a service class that:

- Encapsulates business logic
- Interacts with Prisma for database operations
- Validates entity constraints
- Throws AWS-compatible exceptions

**Example: IAM Service** (`src/iam/iam.service.ts`)

```typescript
@Injectable()
export class IamService {
  constructor(private readonly prisma: PrismaService) {}

  async createRole(data: CreateRoleInput): Promise<IamRole> {
    // Check for duplicates
    // Create in database
    // Return entity
  }

  async putRoleInlinePolicy(...): Promise<void> {
    // Verify role exists
    // Upsert policy
  }
}
```

---

## Entity Layer

Entity classes wrap Prisma models and provide:

- Type safety
- Computed properties (e.g., ARNs, metadata formatters)
- Serialization logic

**Example: IAM Role Entity** (`src/iam/iam-role.entity.ts`)

```typescript
export class IamRole implements PrismaIamRole {
  id: string;
  name: string;
  path: string;
  accountId: string;
  assumeRolePolicy: string;
  // ...

  get arn(): string {
    return `arn:aws:iam::${this.accountId}:role${this.path}${this.name}`;
  }

  get metadata() {
    // Returns AWS API response format
  }
}
```

---

## Database Layer

### Prisma Setup (`src/_prisma/`)

- **prisma.service.ts**: PrismaClient wrapper with connection lifecycle
- **prisma.module.ts**: Exports PrismaService globally

### Schema Location: `prisma/schema.prisma`

**Key Models:**

- `IamRole`, `IamPolicy`, `IamRoleIamPolicyAttachment`, `IamRoleInlinePolicy`
- `KmsKey`, `KmsAlias`
- `S3Bucket`, `S3Object`
- `SecretsManagerSecret`
- `SnsTopic`, `SnsSubscription`
- `SqsQueue`
- `ConsulKVEntry`

**Migrations:** `prisma/migrations/`

---

## Shared Infrastructure

### AWS Shared Entities (`src/aws-shared-entities/`)

- **aws-exceptions.ts**: All AWS exception classes
  - Base `AwsException` with `toXml()` and `toJson()` methods
  - Specific exceptions: `NoSuchEntity`, `ValidationError`, `AccessDeniedException`, etc.
- **attributes.service.ts**: Common attribute parsing/validation
- **tags.service.ts**: Resource tagging utilities

### Request Context (`src/_context/`)

- **request.context.ts**: TypeScript interfaces for request context
  ```typescript
  interface RequestContext {
    action?: Action;
    format?: Format;
    awsProperties: AwsProperties; // accountId, region, host
    requestId: string;
  }
  ```
- **exception.filter.ts**: Global exception filter for AWS exceptions

### Audit System (`src/audit/`)

- **audit.interceptor.ts**: Logs requests/responses (main app)
- **s3-audit.interceptor.ts**: Logs S3 requests/responses
- **audit.service.ts**: Audit persistence service (currently minimal)

### Configuration (`src/config/`)

- **local.config.ts**: Loads environment variables
- **config.validator.ts**: Validates config on startup
- **common-config.interface.ts**: TypeScript interface for config

**Environment Variables:**

- `AWS_ACCOUNT_ID` (default: 000000000000)
- `AWS_REGION` (default: us-east-1)
- `PORT` (default: 8081) - Main app port
- `S3_PORT` (default: 4567) - S3 microservice port
- `CONSUL_PORT` (default: 8500) - Consul KV microservice port
- `DATABASE_URL` (default: :memory:) - SQLite connection string
- `HOST`, `PROTO` - Used for URL generation

### Default Action Handler (`src/default-action-handler/`)

Provides stub implementations for unimplemented actions.

- **default-action-handler.provider.ts**: Creates default handlers that throw `UnsupportedOperationException`
- **existing-action-handlers.provider.ts**: Collects implemented handlers into map
- **default-action-handler.constants.ts**: Shared constants

**Purpose:** Allows modules to declare all AWS actions in their enum list, then provides automatic "not implemented" responses for actions without handlers.

---

## Action Enum (`src/action.enum.ts`)

Central enum containing all AWS actions across all services:

```typescript
export enum Action {
  // IAM
  IamCreateRole = 'CreateRole',
  IamGetRole = 'GetRole',
  IamPutRolePolicy = 'PutRolePolicy',

  // KMS
  KmsCreateKey = 'CreateKey',
  KmsSign = 'Sign',

  // S3
  S3CreateBucket = 'CreateBucket',
  S3GetObject = 'GetObject',
  S3PutBucketAcl = 'PutBucketAcl',

  // ... all other actions
}
```

---

## Testing Strategy

### Test Structure

Each module has a dedicated test suite in `src/<module>/__tests__/<module>.spec.ts`

**Test Pattern:**

1. Create isolated test app with in-memory SQLite database
2. Use actual AWS SDK clients pointing to local endpoints
3. Test against real AWS API contracts
4. Verify both API responses and database state

**Example Test Setup:**

```typescript
describe('IAM Integration Tests', () => {
  let app: INestApplication;
  let iamClient: IAMClient;
  let prismaService: PrismaService;

  beforeAll(async () => {
    // Unique in-memory database per test suite
    process.env.DATABASE_URL = `file::memory:?cache=shared&unique=${Date.now()}`;

    const moduleFixture = await Test.createTestingModule({
      imports: [AppModule],
    }).compile();

    app = moduleFixture.createNestApplication();
    await app.init();

    // Real AWS SDK client pointing to local endpoint
    iamClient = new IAMClient({
      endpoint: `http://localhost:${testPort}`,
      credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
    });
  });

  it('should create a role', async () => {
    const response = await iamClient.send(new CreateRoleCommand({ ... }));
    expect(response.Role.RoleName).toBe('TestRole');
  });
});
```

**Test Organization:**

- `src/iam/__tests__/iam.spec.ts` - IAM tests
- `src/s3/__tests__/s3.spec.ts` - S3 tests
- `src/kms/__tests__/kms.spec.ts` - KMS tests
- etc.

**Current Test Count:** 222 tests across 8 test suites

---

## Key Patterns & Conventions

### 1. Handler Naming

- Handlers named after AWS action: `<Action>Handler` (e.g., `CreateRoleHandler`)
- One handler per file: `<action>-handler.ts` (kebab-case)
- Handler must be Injectable and registered in module

### 2. Service Layer

- One service per module: `<service>.service.ts`
- Service injected into handlers via constructor
- Services interact with Prisma, not handlers directly

### 3. Entity Layer

- Entities wrap Prisma models: `<service>-<entity>.entity.ts`
- Implement Prisma type: `implements Prisma<Entity>`
- Provide computed properties and formatting methods

### 4. Module Constants

- `<service>.constants.ts` contains:
  - Injection token: `export const ServiceHandlers = Symbol('ServiceHandlers');`
  - Action list: `export const serviceActions: Action[] = [...]`

### 5. Error Handling

- Throw AWS exception classes from `aws-shared-entities/aws-exceptions.ts`
- Exception filter converts to XML or JSON automatically
- Include descriptive messages matching AWS API

### 6. Database Conventions

- Use Prisma for all database operations
- Models use camelCase, match entity class names
- Migrations created via `npx prisma migrate dev --name <description>`
- Always include `accountId` for multi-tenancy support

### 7. Request Validation

- Use Joi schemas in handler's `validator` property
- Validate early, fail fast with `ValidationError`
- Allow unknown properties for forward compatibility

---

## Development Workflow

### Adding a New Handler

1. **Create handler file** `src/<service>/<action>-handler.ts`

   ```typescript
   @Injectable()
   export class MyActionHandler extends AbstractActionHandler<QueryParams> {
     format = Format.Xml;
     action = Action.ServiceMyAction;
     validator = Joi.object({
       /* ... */
     });
     protected async handle(params, context) {
       /* ... */
     }
   }
   ```

2. **Add to module** in `src/<service>/<service>.module.ts`

   ```typescript
   const handlers = [
     // ... existing handlers
     MyActionHandler,
   ];
   ```

3. **Implement service method** if needed in `src/<service>/<service>.service.ts`

4. **Add tests** in `src/<service>/__tests__/<service>.spec.ts`

   ```typescript
   it('should perform my action', async () => {
     const response = await client.send(new MyActionCommand({ ... }));
     expect(response).toBeDefined();
   });
   ```

5. **Run tests** `npm test`

### Adding a New AWS Service Module

1. **Create directory** `src/<service>/`

2. **Create files:**

   - `<service>.module.ts` - NestJS module
   - `<service>.service.ts` - Business logic
   - `<service>.constants.ts` - Injection tokens and action list
   - `<handler>-handler.ts` - Handler implementations
   - `<service>-<entity>.entity.ts` - Entity classes
   - `__tests__/<service>.spec.ts` - Tests

3. **Update** `src/action.enum.ts` with service actions

4. **Update** `src/app.module.ts` to import new module

5. **Create Prisma models** in `prisma/schema.prisma`

6. **Run migration** `npx prisma migrate dev --name add_<service>`

7. **Implement handlers** following handler pattern

8. **Write tests** using AWS SDK client

---

## Common Troubleshooting

### Handler not found

- Check handler is in module's `handlers` array
- Verify action enum matches exactly
- Ensure handler is Injectable

### Database errors

- Run `npx prisma generate` after schema changes
- Check migration applied: `npx prisma migrate dev`
- Verify Prisma model matches entity class

### Validation errors

- Check Joi schema matches AWS API requirements
- Verify parameter casing (AWS uses PascalCase)
- Test with actual AWS SDK client

### S3 routing issues

- S3 uses different routing logic (HTTP method + path)
- Check `determineS3Action()` in S3Controller
- Verify query parameter handling (e.g., `?acl`)

### Test failures

- Ensure unique database: `file::memory:?cache=shared&unique=...`
- Check port conflicts (8086-8090 commonly used)
- Clean up resources in afterEach hooks

---

## Architecture Decisions

### Why Separate S3 and Consul KV Microservices?

**S3 Microservice:**
S3 uses REST-style routing (path-based) rather than action-based routing. The main app routes based on `Action` parameter, while S3 needs to parse URLs like `/{bucket}/{key}` and route based on HTTP method + query parameters.

**Consul KV Microservice:**
Consul KV implements the HashiCorp Consul API specification, which uses REST-style routing at `/v1/kv/*` endpoints. It requires different routing logic, query parameter handling, and response formats than AWS services. Running it as a separate microservice allows:

- Clean separation of concerns between AWS and Consul APIs
- Different port (8500) matching standard Consul deployment
- Independent body parsing and response formatting
- Compatibility with existing Consul client libraries

### Why Handler Pattern?

- **Modularity**: Each AWS action is isolated
- **Testability**: Easy to test individual handlers
- **Scalability**: Easy to add new actions
- **Type Safety**: Each handler has its own request type
- **Validation**: Built-in Joi validation per action

### Why Prisma + SQLite?

- **Simplicity**: No external database required
- **Type Safety**: Generated TypeScript types
- **Migrations**: Schema versioning out of the box
- **Performance**: Fast for local development/testing
- **Portability**: Single file database or in-memory

### Why Dual Format Support (XML/JSON)?

Different AWS services use different protocols:

- IAM, KMS: XML (AWS Query protocol)
- SNS, SQS: JSON (AWS JSON protocol)
- S3: XML with custom headers

---

## File Organization Summary

```
src/
├── main.ts                          # Application bootstrap
├── app.module.ts                    # Main app module
├── app.controller.ts                # Main API controller
├── abstract-action.handler.ts       # Base handler class
├── action.enum.ts                   # All AWS actions
├── app.constants.ts                 # App-level constants
│
├── _context/                        # Request context types
├── _prisma/                         # Prisma client wrapper
├── audit/                           # Request/response logging
├── aws-shared-entities/             # AWS exceptions & utilities
├── config/                          # Environment configuration
├── default-action-handler/          # Stub handler generation
│
├── iam/                             # IAM service module
│   ├── __tests__/
│   ├── *-handler.ts                 # Action handlers
│   ├── iam.service.ts               # Business logic
│   ├── iam.module.ts                # Module definition
│   ├── iam.constants.ts             # Constants
│   └── *.entity.ts                  # Entity classes
│
├── s3/                              # S3 service module (microservice)
│   ├── __tests__/
│   ├── s3-app.module.ts             # Separate app module
│   ├── s3.controller.ts             # Custom controller
│   ├── *-handler.ts                 # Action handlers
│   └── ...
│
├── consul-kv/                       # Consul KV module (microservice)
│   ├── __tests__/
│   ├── consul-kv-app.module.ts      # Separate app module
│   ├── consul-kv.controller.ts      # Custom controller
│   ├── consul-kv.service.ts         # Business logic
│   └── ...
│
├── kms/                             # KMS service module
├── secrets-manager/                 # Secrets Manager module
├── sns/                             # SNS service module
├── sqs/                             # SQS service module
└── sts/                             # STS service module

prisma/
├── schema.prisma                    # Database schema
└── migrations/                      # Schema migrations

```

---

## Quick Reference

**Start Development:**

```bash
PORT=8081 S3_PORT=4567 CONSUL_PORT=8500 yarn start:dev
```

**Run Tests:**

```bash
npm test                           # All tests
npm test -- iam.spec.ts           # Specific suite
```

**Database Commands:**

```bash
npx prisma migrate dev --name <description>
npx prisma generate
npx prisma studio                 # GUI database browser
```

**Test AWS CLI:**

```bash
# IAM
aws iam create-role --role-name TestRole --assume-role-policy-document '{}' \
  --endpoint-url http://localhost:8081

# S3
aws s3 mb s3://my-bucket --endpoint-url http://localhost:4567
aws s3 ls s3://my-bucket --endpoint-url http://localhost:4567
```

**Test Consul API:**

```bash
# Using curl
curl -X PUT http://localhost:8500/v1/kv/my-key -d 'my-value'
curl http://localhost:8500/v1/kv/my-key
curl -X DELETE http://localhost:8500/v1/kv/my-key

# Using consul CLI (if installed)
export CONSUL_HTTP_ADDR=http://localhost:8500
consul kv put my-key my-value
consul kv get my-key
consul kv delete my-key
```

---

This documentation should be your starting point for understanding the project. Read this first before making changes to understand the patterns and conventions in use.