s3 impl
This commit is contained in:
812
PROJECT_STRUCTURE.md
Normal file
812
PROJECT_STRUCTURE.md
Normal file
@@ -0,0 +1,812 @@
|
||||
# Local AWS - Project Structure Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
Local AWS is a NestJS-based emulator for AWS services. It implements AWS APIs locally for development and testing purposes, using SQLite for persistence and following AWS API specifications.
|
||||
|
||||
**Key Architecture Points:**
|
||||
|
||||
- **Handler Pattern**: Each AWS action is implemented as a separate handler class
|
||||
- **Modular Structure**: One module per AWS service (IAM, KMS, S3, SNS, SQS, etc.)
|
||||
- **Multi-Server Architecture**: Main app + separate S3 microservice + Consul KV microservice
|
||||
- S3 uses REST-style routing
|
||||
- Consul KV implements HashiCorp Consul API
|
||||
- **Database**: SQLite with Prisma ORM
|
||||
- **Test Structure**: Each module has its own test suite using AWS SDK or service-specific clients
|
||||
|
||||
---
|
||||
|
||||
## Application Entry Points
|
||||
|
||||
### Main Entry (`src/main.ts`)
|
||||
|
||||
Bootstraps two NestJS applications:
|
||||
|
||||
1. **Main Application** (default port 8081)
|
||||
|
||||
- Handles IAM, KMS, SecretsManager, SNS, SQS, STS services
|
||||
- Uses query-based routing (Action parameter in body/query)
|
||||
- Body parsers: JSON (for SNS/SQS), XML, raw binary
|
||||
|
||||
2. **S3 Microservice** (default port 4567)
|
||||
|
||||
- Separate app due to S3's REST/path-style routing
|
||||
- Routes based on HTTP method + path + query parameters
|
||||
- Body parsers: raw binary, URL-encoded
|
||||
|
||||
3. **Consul KV Microservice** (default port 8500)
|
||||
- Implements HashiCorp Consul Key-Value store API
|
||||
- REST-style routing at `/v1/kv/*` endpoints
|
||||
- Supports recursive queries, CAS operations, locks, flags
|
||||
- Body parsers: JSON, raw binary, text/plain
|
||||
|
||||
**Key Global Features:**
|
||||
|
||||
```typescript
|
||||
Date.prototype.getAwsTime(); // Extension for AWS timestamp format
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Architecture
|
||||
|
||||
### Module Structure Pattern
|
||||
|
||||
Each AWS service follows this structure:
|
||||
|
||||
```
|
||||
<service>/
|
||||
├── __tests__/
|
||||
│ └── <service>.spec.ts # Integration tests using AWS SDK
|
||||
├── <action>-handler.ts # One handler per AWS action
|
||||
├── <service>.constants.ts # Action enum list & injection tokens
|
||||
├── <service>.module.ts # NestJS module definition
|
||||
├── <service>.service.ts # Business logic layer
|
||||
├── <service>-<entity>.entity.ts # Prisma model wrapper classes
|
||||
└── (optional) <service>.controller.ts # Custom controller if needed
|
||||
```
|
||||
|
||||
### Current Modules
|
||||
|
||||
#### Main Application Modules
|
||||
|
||||
1. **IAM** (`src/iam/`)
|
||||
|
||||
- Handlers: CreateRole, GetRole, DeleteRole, CreatePolicy, GetPolicy, AttachRolePolicy, PutRolePolicy, GetRolePolicy, etc.
|
||||
- Entities: IamRole, IamPolicy, IamRoleInlinePolicy
|
||||
- Database: Roles, policies, role-policy attachments, inline policies
|
||||
|
||||
2. **KMS** (`src/kms/`)
|
||||
|
||||
- Handlers: CreateKey, DescribeKey, CreateAlias, Sign, GetPublicKey, EnableKeyRotation, etc.
|
||||
- Entities: KmsKey, KmsAlias
|
||||
- Database: KMS keys, aliases
|
||||
|
||||
3. **Secrets Manager** (`src/secrets-manager/`)
|
||||
|
||||
- Handlers: CreateSecret, GetSecretValue, PutSecretValue, DeleteSecret, etc.
|
||||
- Database: Secrets with versioning
|
||||
|
||||
4. **SNS** (`src/sns/`)
|
||||
|
||||
- Handlers: CreateTopic, Subscribe, Publish, etc.
|
||||
- Database: Topics, subscriptions
|
||||
|
||||
5. **SQS** (`src/sqs/`)
|
||||
|
||||
- Handlers: CreateQueue, SendMessage, ReceiveMessage, etc.
|
||||
- Database: Queues, messages
|
||||
|
||||
6. **STS** (`src/sts/`)
|
||||
- Handlers: AssumeRole, GetCallerIdentity, etc.
|
||||
- Temporary credential generation
|
||||
|
||||
#### S3 Microservice Module
|
||||
|
||||
**S3** (`src/s3/`)
|
||||
|
||||
- Special module with its own app (`S3AppModule`) and controller
|
||||
- Handlers: CreateBucket, ListBuckets, PutObject, GetObject, DeleteObject, PutBucketAcl, GetBucketAcl, etc.
|
||||
- Entities: S3Bucket, S3Object
|
||||
- Custom routing logic in `S3Controller` based on HTTP method + path + query params
|
||||
- Database: Buckets with tags/ACL/policy, objects with metadata
|
||||
|
||||
#### Consul KV Microservice Module
|
||||
|
||||
**Consul KV** (`src/consul-kv/`)
|
||||
|
||||
- Special module with its own app (`ConsulKVAppModule`) and controller
|
||||
- Implements HashiCorp Consul KV API spec: https://developer.hashicorp.com/consul/api-docs/kv
|
||||
- Service: ConsulKVService with getKey, putKey, deleteKey, listKeys, recursive operations
|
||||
- Custom routing in `ConsulKVController` for `/v1/kv/*` paths
|
||||
- Features:
|
||||
- Key-value storage with base64 encoding
|
||||
- Recursive queries with prefix matching
|
||||
- Keys-only listing with separator support
|
||||
- Check-And-Set (CAS) operations
|
||||
- Distributed lock acquisition/release
|
||||
- Flags and index tracking (createIndex, modifyIndex, lockIndex)
|
||||
- Multi-tenancy support (datacenter, namespace)
|
||||
- Database: ConsulKVEntry model
|
||||
- Tests: 19 integration tests using `consul` npm package
|
||||
|
||||
---
|
||||
|
||||
## Handler Pattern Deep Dive
|
||||
|
||||
### AbstractActionHandler (`src/abstract-action.handler.ts`)
|
||||
|
||||
Base class for all action handlers. All handlers must extend this class.
|
||||
|
||||
**Required Properties:**
|
||||
|
||||
```typescript
|
||||
abstract class AbstractActionHandler<T> {
|
||||
format: Format; // Format.Xml or Format.Json
|
||||
action: Action | Action[]; // AWS action enum(s)
|
||||
validator: Joi.ObjectSchema; // Request validation schema
|
||||
|
||||
protected abstract handle(
|
||||
queryParams: T, // Validated query parameters
|
||||
context: RequestContext, // AWS properties + request metadata
|
||||
): Record<string, any> | void;
|
||||
}
|
||||
```
|
||||
|
||||
**Response Handling:**
|
||||
|
||||
- XML format: Wraps result in `{Action}Result` node with RequestId metadata
|
||||
- JSON format: Returns result directly
|
||||
- Empty response: Returns ResponseMetadata only (for XML) or nothing (for JSON)
|
||||
|
||||
### Handler Implementation Example
|
||||
|
||||
```typescript
|
||||
@Injectable()
|
||||
export class CreateRoleHandler extends AbstractActionHandler<QueryParams> {
|
||||
constructor(private readonly iamService: IamService) {
|
||||
super();
|
||||
}
|
||||
|
||||
format = Format.Xml;
|
||||
action = Action.IamCreateRole;
|
||||
|
||||
validator = Joi.object<QueryParams>({
|
||||
RoleName: Joi.string().required(),
|
||||
Path: Joi.string().required(),
|
||||
AssumeRolePolicyDocument: Joi.string().required(),
|
||||
// ... other parameters
|
||||
});
|
||||
|
||||
protected async handle(params: QueryParams, { awsProperties }: RequestContext) {
|
||||
const role = await this.iamService.createRole({
|
||||
accountId: awsProperties.accountId,
|
||||
name: params.RoleName,
|
||||
// ...
|
||||
});
|
||||
return { Role: role.metadata };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Handler Registration Flow
|
||||
|
||||
1. **Define handler class** extending `AbstractActionHandler`
|
||||
2. **Add to module's handlers array** in `<service>.module.ts`
|
||||
3. **ExistingActionHandlersProvider** collects all handlers into a map keyed by Action enum
|
||||
4. **DefaultActionHandlerProvider** fills gaps with stub handlers for unimplemented actions
|
||||
5. **Module provides injection token** (e.g., `IAMHandlers`, `S3Handlers`) containing the handler map
|
||||
6. **Controller receives handler map** and routes requests to appropriate handler
|
||||
|
||||
```typescript
|
||||
// In module
|
||||
const handlers = [CreateRoleHandler, GetRoleHandler, /* ... */];
|
||||
|
||||
@Module({
|
||||
providers: [
|
||||
...handlers,
|
||||
ExistingActionHandlersProvider(handlers),
|
||||
DefaultActionHandlerProvider(IAMHandlers, Format.Xml, allIamActions),
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Controllers
|
||||
|
||||
### Main Controller (`src/app.controller.ts`)
|
||||
|
||||
Routes all non-S3 AWS service requests.
|
||||
|
||||
**Request Flow:**
|
||||
|
||||
1. Extracts action from `x-amz-target` header or `Action` body parameter
|
||||
2. Validates action exists in Action enum
|
||||
3. Looks up handler from injected ActionHandlers map
|
||||
4. Validates request params using handler's Joi validator
|
||||
5. Calls `handler.getResponse(params, context)`
|
||||
6. Returns XML or JSON based on handler's format
|
||||
|
||||
### S3 Controller (`src/s3/s3.controller.ts`)
|
||||
|
||||
Custom controller for S3 REST-style API (separate microservice).
|
||||
|
||||
**Request Flow:**
|
||||
|
||||
1. Parses path into bucket/key components
|
||||
2. Determines action from HTTP method + path + query parameters
|
||||
- `PUT /{bucket}` → CreateBucket
|
||||
- `GET /{bucket}` → ListObjects (or other sub-resource if query param present)
|
||||
- `PUT /{bucket}?acl` → PutBucketAcl
|
||||
- `GET /{bucket}/{key}` → GetObject
|
||||
3. Normalizes query parameters (e.g., `max-keys` → `MaxKeys`)
|
||||
4. Extracts metadata from `x-amz-meta-*` headers
|
||||
5. Converts binary body to base64 for handlers
|
||||
6. Routes to handler and returns response with appropriate headers
|
||||
|
||||
**Special S3 Response Handling:**
|
||||
|
||||
- Creates Location header for bucket creation
|
||||
- Returns ETag, Last-Modified, Content-Type headers
|
||||
- Returns raw binary data for GetObject (not XML/JSON)
|
||||
- Returns empty 200 for PutBucketAcl, PutBucketTagging
|
||||
- Returns 204 for DeleteBucket, DeleteObject
|
||||
|
||||
### Consul KV Controller (`src/consul-kv/consul-kv.controller.ts`)
|
||||
|
||||
Custom controller for Consul KV REST API (separate microservice).
|
||||
|
||||
**Request Flow:**
|
||||
|
||||
1. Catches all `/v1/kv/*` paths
|
||||
2. Extracts and URL-decodes the key from the path
|
||||
3. Routes based on HTTP method (GET/PUT/DELETE)
|
||||
4. Parses query parameters:
|
||||
- `dc`: datacenter (default: dc1)
|
||||
- `ns`: namespace (default: default)
|
||||
- `recurse`: recursive key retrieval
|
||||
- `keys`: return only key names
|
||||
- `raw`: return raw value (not JSON)
|
||||
- `separator`: group keys by separator
|
||||
- `flags`: numeric flags for application use
|
||||
- `cas`: Check-And-Set with modify index
|
||||
- `acquire`: session ID for lock acquisition
|
||||
- `release`: session ID for lock release
|
||||
5. Processes request through ConsulKVService
|
||||
6. Returns appropriate response format
|
||||
|
||||
**Special Consul Response Handling:**
|
||||
|
||||
- PUT/DELETE return plain text "true" or "false"
|
||||
- GET returns JSON array of key metadata or raw value
|
||||
- Keys-only returns JSON array of key strings
|
||||
- 404 status for non-existent keys
|
||||
- Base64 encoding for values in JSON responses
|
||||
- Supports keys with slashes (proper URL decoding)
|
||||
|
||||
---
|
||||
|
||||
## Service Layer
|
||||
|
||||
Each module has a service class that:
|
||||
|
||||
- Encapsulates business logic
|
||||
- Interacts with Prisma for database operations
|
||||
- Validates entity constraints
|
||||
- Throws AWS-compatible exceptions
|
||||
|
||||
**Example: IAM Service** (`src/iam/iam.service.ts`)
|
||||
|
||||
```typescript
|
||||
@Injectable()
|
||||
export class IamService {
|
||||
constructor(private readonly prisma: PrismaService) {}
|
||||
|
||||
async createRole(data: CreateRoleInput): Promise<IamRole> {
|
||||
// Check for duplicates
|
||||
// Create in database
|
||||
// Return entity
|
||||
}
|
||||
|
||||
async putRoleInlinePolicy(...): Promise<void> {
|
||||
// Verify role exists
|
||||
// Upsert policy
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entity Layer
|
||||
|
||||
Entity classes wrap Prisma models and provide:
|
||||
|
||||
- Type safety
|
||||
- Computed properties (e.g., ARNs, metadata formatters)
|
||||
- Serialization logic
|
||||
|
||||
**Example: IAM Role Entity** (`src/iam/iam-role.entity.ts`)
|
||||
|
||||
```typescript
|
||||
export class IamRole implements PrismaIamRole {
|
||||
id: string;
|
||||
name: string;
|
||||
path: string;
|
||||
accountId: string;
|
||||
assumeRolePolicy: string;
|
||||
// ...
|
||||
|
||||
get arn(): string {
|
||||
return `arn:aws:iam::${this.accountId}:role${this.path}${this.name}`;
|
||||
}
|
||||
|
||||
get metadata() {
|
||||
// Returns AWS API response format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Layer
|
||||
|
||||
### Prisma Setup (`src/_prisma/`)
|
||||
|
||||
- **prisma.service.ts**: PrismaClient wrapper with connection lifecycle
|
||||
- **prisma.module.ts**: Exports PrismaService globally
|
||||
|
||||
### Schema Location: `prisma/schema.prisma`
|
||||
|
||||
**Key Models:**
|
||||
|
||||
- `IamRole`, `IamPolicy`, `IamRoleIamPolicyAttachment`, `IamRoleInlinePolicy`
|
||||
- `KmsKey`, `KmsAlias`
|
||||
- `S3Bucket`, `S3Object`
|
||||
- `SecretsManagerSecret`
|
||||
- `SnsTopic`, `SnsSubscription`
|
||||
- `SqsQueue`
|
||||
- `ConsulKVEntry`
|
||||
|
||||
**Migrations:** `prisma/migrations/`
|
||||
|
||||
---
|
||||
|
||||
## Shared Infrastructure
|
||||
|
||||
### AWS Shared Entities (`src/aws-shared-entities/`)
|
||||
|
||||
- **aws-exceptions.ts**: All AWS exception classes
|
||||
- Base `AwsException` with `toXml()` and `toJson()` methods
|
||||
- Specific exceptions: `NoSuchEntity`, `ValidationError`, `AccessDeniedException`, etc.
|
||||
- **attributes.service.ts**: Common attribute parsing/validation
|
||||
- **tags.service.ts**: Resource tagging utilities
|
||||
|
||||
### Request Context (`src/_context/`)
|
||||
|
||||
- **request.context.ts**: TypeScript interfaces for request context
|
||||
```typescript
|
||||
interface RequestContext {
|
||||
action?: Action;
|
||||
format?: Format;
|
||||
awsProperties: AwsProperties; // accountId, region, host
|
||||
requestId: string;
|
||||
}
|
||||
```
|
||||
- **exception.filter.ts**: Global exception filter for AWS exceptions
|
||||
|
||||
### Audit System (`src/audit/`)
|
||||
|
||||
- **audit.interceptor.ts**: Logs requests/responses (main app)
|
||||
- **s3-audit.interceptor.ts**: Logs S3 requests/responses
|
||||
- **audit.service.ts**: Audit persistence service (currently minimal)
|
||||
|
||||
### Configuration (`src/config/`)
|
||||
|
||||
- **local.config.ts**: Loads environment variables
|
||||
- **config.validator.ts**: Validates config on startup
|
||||
- **common-config.interface.ts**: TypeScript interface for config
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
- `AWS_ACCOUNT_ID` (default: 000000000000)
|
||||
- `AWS_REGION` (default: us-east-1)
|
||||
- `PORT` (default: 8081) - Main app port
|
||||
- `S3_PORT` (default: 4567) - S3 microservice port
|
||||
- `CONSUL_PORT` (default: 8500) - Consul KV microservice port
|
||||
- `DATABASE_URL` (default: :memory:) - SQLite connection string
|
||||
- `HOST`, `PROTO` - Used for URL generation
|
||||
|
||||
### Default Action Handler (`src/default-action-handler/`)
|
||||
|
||||
Provides stub implementations for unimplemented actions.
|
||||
|
||||
- **default-action-handler.provider.ts**: Creates default handlers that throw `UnsupportedOperationException`
|
||||
- **existing-action-handlers.provider.ts**: Collects implemented handlers into map
|
||||
- **default-action-handler.constants.ts**: Shared constants
|
||||
|
||||
**Purpose:** Allows modules to declare all AWS actions in their enum list, then provides automatic "not implemented" responses for actions without handlers.
|
||||
|
||||
---
|
||||
|
||||
## Action Enum (`src/action.enum.ts`)
|
||||
|
||||
Central enum containing all AWS actions across all services:
|
||||
|
||||
```typescript
|
||||
export enum Action {
|
||||
// IAM
|
||||
IamCreateRole = 'CreateRole',
|
||||
IamGetRole = 'GetRole',
|
||||
IamPutRolePolicy = 'PutRolePolicy',
|
||||
|
||||
// KMS
|
||||
KmsCreateKey = 'CreateKey',
|
||||
KmsSign = 'Sign',
|
||||
|
||||
// S3
|
||||
S3CreateBucket = 'CreateBucket',
|
||||
S3GetObject = 'GetObject',
|
||||
S3PutBucketAcl = 'PutBucketAcl',
|
||||
|
||||
// ... all other actions
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Test Structure
|
||||
|
||||
Each module has a dedicated test suite in `src/<module>/__tests__/<module>.spec.ts`
|
||||
|
||||
**Test Pattern:**
|
||||
|
||||
1. Create isolated test app with in-memory SQLite database
|
||||
2. Use actual AWS SDK clients pointing to local endpoints
|
||||
3. Test against real AWS API contracts
|
||||
4. Verify both API responses and database state
|
||||
|
||||
**Example Test Setup:**
|
||||
|
||||
```typescript
|
||||
describe('IAM Integration Tests', () => {
|
||||
let app: INestApplication;
|
||||
let iamClient: IAMClient;
|
||||
let prismaService: PrismaService;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Unique in-memory database per test suite
|
||||
process.env.DATABASE_URL = `file::memory:?cache=shared&unique=${Date.now()}`;
|
||||
|
||||
const moduleFixture = await Test.createTestingModule({
|
||||
imports: [AppModule],
|
||||
}).compile();
|
||||
|
||||
app = moduleFixture.createNestApplication();
|
||||
await app.init();
|
||||
|
||||
// Real AWS SDK client pointing to local endpoint
|
||||
iamClient = new IAMClient({
|
||||
endpoint: `http://localhost:${testPort}`,
|
||||
credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
|
||||
});
|
||||
});
|
||||
|
||||
it('should create a role', async () => {
|
||||
const response = await iamClient.send(new CreateRoleCommand({ ... }));
|
||||
expect(response.Role.RoleName).toBe('TestRole');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Test Organization:**
|
||||
|
||||
- `src/iam/__tests__/iam.spec.ts` - IAM tests
|
||||
- `src/s3/__tests__/s3.spec.ts` - S3 tests
|
||||
- `src/kms/__tests__/kms.spec.ts` - KMS tests
|
||||
- etc.
|
||||
|
||||
**Current Test Count:** 222 tests across 8 test suites
|
||||
|
||||
---
|
||||
|
||||
## Key Patterns & Conventions
|
||||
|
||||
### 1. Handler Naming
|
||||
|
||||
- Handlers named after AWS action: `<Action>Handler` (e.g., `CreateRoleHandler`)
|
||||
- One handler per file: `<action>-handler.ts` (kebab-case)
|
||||
- Handler must be Injectable and registered in module
|
||||
|
||||
### 2. Service Layer
|
||||
|
||||
- One service per module: `<service>.service.ts`
|
||||
- Service injected into handlers via constructor
|
||||
- Services interact with Prisma, not handlers directly
|
||||
|
||||
### 3. Entity Layer
|
||||
|
||||
- Entities wrap Prisma models: `<service>-<entity>.entity.ts`
|
||||
- Implement Prisma type: `implements Prisma<Entity>`
|
||||
- Provide computed properties and formatting methods
|
||||
|
||||
### 4. Module Constants
|
||||
|
||||
- `<service>.constants.ts` contains:
|
||||
- Injection token: `export const ServiceHandlers = Symbol('ServiceHandlers');`
|
||||
- Action list: `export const serviceActions: Action[] = [...]`
|
||||
|
||||
### 5. Error Handling
|
||||
|
||||
- Throw AWS exception classes from `aws-shared-entities/aws-exceptions.ts`
|
||||
- Exception filter converts to XML or JSON automatically
|
||||
- Include descriptive messages matching AWS API
|
||||
|
||||
### 6. Database Conventions
|
||||
|
||||
- Use Prisma for all database operations
|
||||
- Models use camelCase, match entity class names
|
||||
- Migrations created via `npx prisma migrate dev --name <description>`
|
||||
- Always include `accountId` for multi-tenancy support
|
||||
|
||||
### 7. Request Validation
|
||||
|
||||
- Use Joi schemas in handler's `validator` property
|
||||
- Validate early, fail fast with `ValidationError`
|
||||
- Allow unknown properties for forward compatibility
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Adding a New Handler
|
||||
|
||||
1. **Create handler file** `src/<service>/<action>-handler.ts`
|
||||
|
||||
```typescript
|
||||
@Injectable()
|
||||
export class MyActionHandler extends AbstractActionHandler<QueryParams> {
|
||||
format = Format.Xml;
|
||||
action = Action.ServiceMyAction;
|
||||
validator = Joi.object({
|
||||
/* ... */
|
||||
});
|
||||
protected async handle(params, context) {
|
||||
/* ... */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Add to module** in `src/<service>/<service>.module.ts`
|
||||
|
||||
```typescript
|
||||
const handlers = [
|
||||
// ... existing handlers
|
||||
MyActionHandler,
|
||||
];
|
||||
```
|
||||
|
||||
3. **Implement service method** if needed in `src/<service>/<service>.service.ts`
|
||||
|
||||
4. **Add tests** in `src/<service>/__tests__/<service>.spec.ts`
|
||||
|
||||
```typescript
|
||||
it('should perform my action', async () => {
|
||||
const response = await client.send(new MyActionCommand({ ... }));
|
||||
expect(response).toBeDefined();
|
||||
});
|
||||
```
|
||||
|
||||
5. **Run tests** `npm test`
|
||||
|
||||
### Adding a New AWS Service Module
|
||||
|
||||
1. **Create directory** `src/<service>/`
|
||||
|
||||
2. **Create files:**
|
||||
|
||||
- `<service>.module.ts` - NestJS module
|
||||
- `<service>.service.ts` - Business logic
|
||||
- `<service>.constants.ts` - Injection tokens and action list
|
||||
- `<handler>-handler.ts` - Handler implementations
|
||||
- `<service>-<entity>.entity.ts` - Entity classes
|
||||
- `__tests__/<service>.spec.ts` - Tests
|
||||
|
||||
3. **Update** `src/action.enum.ts` with service actions
|
||||
|
||||
4. **Update** `src/app.module.ts` to import new module
|
||||
|
||||
5. **Create Prisma models** in `prisma/schema.prisma`
|
||||
|
||||
6. **Run migration** `npx prisma migrate dev --name add_<service>`
|
||||
|
||||
7. **Implement handlers** following handler pattern
|
||||
|
||||
8. **Write tests** using AWS SDK client
|
||||
|
||||
---
|
||||
|
||||
## Common Troubleshooting
|
||||
|
||||
### Handler not found
|
||||
|
||||
- Check handler is in module's `handlers` array
|
||||
- Verify action enum matches exactly
|
||||
- Ensure handler is Injectable
|
||||
|
||||
### Database errors
|
||||
|
||||
- Run `npx prisma generate` after schema changes
|
||||
- Check migration applied: `npx prisma migrate dev`
|
||||
- Verify Prisma model matches entity class
|
||||
|
||||
### Validation errors
|
||||
|
||||
- Check Joi schema matches AWS API requirements
|
||||
- Verify parameter casing (AWS uses PascalCase)
|
||||
- Test with actual AWS SDK client
|
||||
|
||||
### S3 routing issues
|
||||
|
||||
- S3 uses different routing logic (HTTP method + path)
|
||||
- Check `determineS3Action()` in S3Controller
|
||||
- Verify query parameter handling (e.g., `?acl`)
|
||||
|
||||
### Test failures
|
||||
|
||||
- Ensure unique database: `file::memory:?cache=shared&unique=...`
|
||||
- Check port conflicts (8086-8090 commonly used)
|
||||
- Clean up resources in afterEach hooks
|
||||
|
||||
---
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### Why Separate S3 and Consul KV Microservices?
|
||||
|
||||
**S3 Microservice:**
|
||||
S3 uses REST-style routing (path-based) rather than action-based routing. The main app routes based on `Action` parameter, while S3 needs to parse URLs like `/{bucket}/{key}` and route based on HTTP method + query parameters.
|
||||
|
||||
**Consul KV Microservice:**
|
||||
Consul KV implements the HashiCorp Consul API specification, which uses REST-style routing at `/v1/kv/*` endpoints. It requires different routing logic, query parameter handling, and response formats than AWS services. Running it as a separate microservice allows:
|
||||
|
||||
- Clean separation of concerns between AWS and Consul APIs
|
||||
- Different port (8500) matching standard Consul deployment
|
||||
- Independent body parsing and response formatting
|
||||
- Compatibility with existing Consul client libraries
|
||||
|
||||
### Why Handler Pattern?
|
||||
|
||||
- **Modularity**: Each AWS action is isolated
|
||||
- **Testability**: Easy to test individual handlers
|
||||
- **Scalability**: Easy to add new actions
|
||||
- **Type Safety**: Each handler has its own request type
|
||||
- **Validation**: Built-in Joi validation per action
|
||||
|
||||
### Why Prisma + SQLite?
|
||||
|
||||
- **Simplicity**: No external database required
|
||||
- **Type Safety**: Generated TypeScript types
|
||||
- **Migrations**: Schema versioning out of the box
|
||||
- **Performance**: Fast for local development/testing
|
||||
- **Portability**: Single file database or in-memory
|
||||
|
||||
### Why Dual Format Support (XML/JSON)?
|
||||
|
||||
Different AWS services use different protocols:
|
||||
|
||||
- IAM, KMS: XML (AWS Query protocol)
|
||||
- SNS, SQS: JSON (AWS JSON protocol)
|
||||
- S3: XML with custom headers
|
||||
|
||||
---
|
||||
|
||||
## File Organization Summary
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.ts # Application bootstrap
|
||||
├── app.module.ts # Main app module
|
||||
├── app.controller.ts # Main API controller
|
||||
├── abstract-action.handler.ts # Base handler class
|
||||
├── action.enum.ts # All AWS actions
|
||||
├── app.constants.ts # App-level constants
|
||||
│
|
||||
├── _context/ # Request context types
|
||||
├── _prisma/ # Prisma client wrapper
|
||||
├── audit/ # Request/response logging
|
||||
├── aws-shared-entities/ # AWS exceptions & utilities
|
||||
├── config/ # Environment configuration
|
||||
├── default-action-handler/ # Stub handler generation
|
||||
│
|
||||
├── iam/ # IAM service module
|
||||
│ ├── __tests__/
|
||||
│ ├── *-handler.ts # Action handlers
|
||||
│ ├── iam.service.ts # Business logic
|
||||
│ ├── iam.module.ts # Module definition
|
||||
│ ├── iam.constants.ts # Constants
|
||||
│ └── *.entity.ts # Entity classes
|
||||
│
|
||||
├── s3/ # S3 service module (microservice)
|
||||
│ ├── __tests__/
|
||||
│ ├── s3-app.module.ts # Separate app module
|
||||
│ ├── s3.controller.ts # Custom controller
|
||||
│ ├── *-handler.ts # Action handlers
|
||||
│ └── ...
|
||||
│
|
||||
├── consul-kv/ # Consul KV module (microservice)
|
||||
│ ├── __tests__/
|
||||
│ ├── consul-kv-app.module.ts # Separate app module
|
||||
│ ├── consul-kv.controller.ts # Custom controller
|
||||
│ ├── consul-kv.service.ts # Business logic
|
||||
│ └── ...
|
||||
│
|
||||
├── kms/ # KMS service module
|
||||
├── secrets-manager/ # Secrets Manager module
|
||||
├── sns/ # SNS service module
|
||||
├── sqs/ # SQS service module
|
||||
└── sts/ # STS service module
|
||||
|
||||
prisma/
|
||||
├── schema.prisma # Database schema
|
||||
└── migrations/ # Schema migrations
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
**Start Development:**
|
||||
|
||||
```bash
|
||||
PORT=8081 S3_PORT=4567 CONSUL_PORT=8500 yarn start:dev
|
||||
```
|
||||
|
||||
**Run Tests:**
|
||||
|
||||
```bash
|
||||
npm test # All tests
|
||||
npm test -- iam.spec.ts # Specific suite
|
||||
```
|
||||
|
||||
**Database Commands:**
|
||||
|
||||
```bash
|
||||
npx prisma migrate dev --name <description>
|
||||
npx prisma generate
|
||||
npx prisma studio # GUI database browser
|
||||
```
|
||||
|
||||
**Test AWS CLI:**
|
||||
|
||||
```bash
|
||||
# IAM
|
||||
aws iam create-role --role-name TestRole --assume-role-policy-document '{}' \
|
||||
--endpoint-url http://localhost:8081
|
||||
|
||||
# S3
|
||||
aws s3 mb s3://my-bucket --endpoint-url http://localhost:4567
|
||||
aws s3 ls s3://my-bucket --endpoint-url http://localhost:4567
|
||||
```
|
||||
|
||||
**Test Consul API:**
|
||||
|
||||
```bash
|
||||
# Using curl
|
||||
curl -X PUT http://localhost:8500/v1/kv/my-key -d 'my-value'
|
||||
curl http://localhost:8500/v1/kv/my-key
|
||||
curl -X DELETE http://localhost:8500/v1/kv/my-key
|
||||
|
||||
# Using consul CLI (if installed)
|
||||
export CONSUL_HTTP_ADDR=http://localhost:8500
|
||||
consul kv put my-key my-value
|
||||
consul kv get my-key
|
||||
consul kv delete my-key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
This documentation should be your starting point for understanding the project. Read this first before making changes to understand the patterns and conventions in use.
|
||||
Reference in New Issue
Block a user