diff --git a/aws-transform/POWER.md b/aws-transform/POWER.md index ca22a82..619ca95 100644 --- a/aws-transform/POWER.md +++ b/aws-transform/POWER.md @@ -4,7 +4,7 @@ displayName: "AWS Transform" description: "Migrate, modernize, and upgrade codebases: .NET Framework to .NET 8/10, mainframe COBOL to Java, VMware VMs to EC2, SQL Server/Oracle/MySQL to Aurora, and Java/Python/Node.js version upgrades or AWS SDK migrations. Assess, plan, and execute code transformations from your IDE." keywords: ["migrate", "modernize", "mainframe", "cobol", "vmware", "dotnet", ".net framework", "windows", "sql server", "oracle", "mysql", "aurora", "ec2 migration", "rehost", "lift-and-shift", "replatform", "legacy", "code upgrade", "sdk migration", "boto3", "java upgrade", "atx", "continuous modernization", "AWS Transform - continuous modernization"] author: "AWS" -version: "2.3.0" +version: "2.4.0" --- # AWS Transform Power @@ -98,24 +98,53 @@ Once routed to continuous modernization: 1. **First-response telemetry notice (once per session).** The first time a request routes here in this session, ALWAYS prepend this line exactly to your reply to the user before doing anything else: - > Note: this power collects telemetry. To opt out, see [here](https://docs.aws.amazon.com/transform/latest/userguide/transform-usage-telemetry.html). + > Note: this power and the continuous modernization CLI, (`atx ct`), collect usage telemetry by default during transformation execution. The telemetry consists of different data points, such as, the IDE name (for example, VS Code or Kiro), the AI agent name (for example, Claude Code or OpenAI Codex), and the execution mode (local or remote). This data is used by AWS Transform to prioritize compatibility testing, as well as latency and reliability. To opt out, see [here](https://docs.aws.amazon.com/transform/latest/userguide/transform-usage-telemetry.html). + + Show it exactly once per session. Do NOT repeat it on subsequent continuous modernization requests in the same session. + + **Telemetry opt-out.** If the user explicitly asks to disable telemetry during the chat session: + 1. Omit `--telemetry` for the rest of the session (see each `workload-continuous-modernization-*.md` file's Telemetry section for the flag format). + 2. Prepend `ATX_DISABLE_TELEMETRY=true` inline on **every** `atx ct` command for the rest of the session — not only `analysis`/`remediation`, but also setup and diagnostic commands like `atx ct status`, `atx ct source ...`, and `atx ct setup ...`. The prefix must be on the same command line as the `atx ct` invocation (including inside compound commands, e.g. `which atx && ATX_DISABLE_TELEMETRY=true atx ct ...`), because the shell does not persist env vars between invocations: `ATX_DISABLE_TELEMETRY=true atx ct ...` - Show it exactly once per session. Do NOT repeat it on subsequent continuous modernization requests in the same session. If the user explicitly asks to disable telemetry, omit `--telemetry` for the rest of the session (see each `workload-continuous-modernization-*.md` file's Telemetry section for the flag format). 2. When invoking AWS Transform - continuous modernization (continuous modernization) commands, use `atx ct` (with a space). `atxct` (no space) is being deprecated; it remains functionally equivalent and hits the same backend, so an `atxct` invocation in the user's environment is not itself a problem. Do not warn the user about `atxct` and do not treat its presence as a failure cause. -3. Check if `atx ct` is installed AND up to date — run as a single command: + +3. **Verify local CLI dispatch before checking versions or AWS configuration.** Run this without redirecting stderr: ``` - INSTALLED=$(atx ct --version 2>/dev/null | head -1); LATEST=$(curl -fsSL "https://transform-cli.awsstatic.com/index.json" 2>/dev/null | grep -o '"latest"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"latest"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/'); echo "Installed: ${INSTALLED:-not found}, Latest: ${LATEST:-unknown}" + atx ct --version ``` -4. If `INSTALLED` is empty or `LATEST` is newer: `curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash && source ~/.bashrc` -5. If `atx ct` fails after install, the binary is rarely the cause — `atx ct` and `atxct` share the same backend and fail identically for env/auth/server reasons. Check those first: - - `ATXCT_FES_ENDPOINT` is set on the server process (not just the CLI shell) + + Classify failures before continuing: + - If the shell reports `atx: command not found`, install the AWS Transform CLI: `curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash`, then restart the shell or source its profile. + - If an `atx` process runs but reports `unknown command 'ct'`, do NOT reinstall blindly or investigate AWS credentials/region. Follow the [command-resolution troubleshooting](steering/workload-continuous-modernization-troubleshooting.md#atx-ct-reports-unknown-command-ct) first. + - If the command succeeds, continue with the version comparison. + +4. Check whether the working CLI is up to date: + + ``` + INSTALLED=$(atx ct --version | head -1); LATEST=$(curl -fsSL "https://transform-cli.awsstatic.com/index.json" 2>/dev/null | grep -o '"latest"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"latest"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/'); echo "Installed: ${INSTALLED:-not found}, Latest: ${LATEST:-unknown}" + ``` + + If `LATEST` is known and newer than `INSTALLED`, update with `curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash`, then restart the shell or source its profile. + +5. **Credential preflight.** Validate AWS credentials before starting any analysis or remediation — at minimum on the first continuous modernization request of the session (new or returning users), and again before any later run in a long session, since credentials can lapse mid-session: + + ``` + aws sts get-caller-identity + ``` + + If it fails or the credentials are expired, refresh them before continuing. Do NOT start any long-running work on expired or soon-to-expire credentials — an analysis started on credentials about to expire can strand the run mid-flight. Run the preflight silently; surface it to the user only if the credentials need refreshing. + +6. If local `atx ct` dispatch succeeded but a later command fails, then check runtime configuration: - `AWS_PROFILE` points at a valid account with refreshed credentials - - The server is running (`atx ct status --health`) + - `AWS_REGION` is set to a supported region + - `ATX_CUSTOM_ENDPOINT` is set in the environment (only if you use a custom endpoint) + + An `unknown command 'ct'` failure is a local command-resolution problem, not an AWS configuration problem; return to Step 3 instead. + +7. Ensure a supported region has been selected (see [workload-continuous-modernization-setup.md](steering/workload-continuous-modernization-setup.md) "Choose your region") and prefixed inline (`AWS_REGION=$ATX_REGION`) on every `atx ct` command. - Only after those check out, verify `atx --help` shows the `ct` subcommand and that `atxct-plugin.mjs` is co-located with the `atx` binary. -6. Start the server using the [continuous-modernization-server.md](steering/workload-continuous-modernization-server.md) skill — it will ask the user for their region, validate it against the supported list, and start with the correct `AWS_REGION`. Wait 5s, then verify with `atx ct status --health`. -7. Then use the appropriate continuous modernization steering file — see `steering/workload-continuous-modernization-routing.md` and the `workload-continuous-modernization-*.md` files referenced from it. +8. Then use the appropriate continuous modernization steering file — see `steering/workload-continuous-modernization-routing.md` and the `workload-continuous-modernization-*.md` files referenced from it. **When in doubt for a workload-unspecified request → continuous modernization.** This default applies ONLY after Step B has cleared — VMware, SQL, and mainframe never fall through to continuous modernization regardless of how the question is phrased; .NET only routes to continuous modernization after the user picks "analyze for tech debt / security / CVEs" in Step B's intent question (both "Modernize" and "Assessment for modernization" stay in the .NET workload). Once routed, do NOT manually read source files to find issues — that's what `atx ct analysis run` does. diff --git a/aws-transform/steering/auth.md b/aws-transform/steering/auth.md index 5d7022a..94299c0 100644 --- a/aws-transform/steering/auth.md +++ b/aws-transform/steering/auth.md @@ -36,6 +36,12 @@ Common CLI-side conditions: - `AccessDeniedException` → AWS credentials expired. Re-run `aws sso login` or refresh env vars. - `command not found: atx` → CLI not installed. Use MCP-based transforms instead, or install the CLI. +### WSL: separate AWS config on the Linux and Windows sides + +On Windows, running the CLI inside WSL introduces a common footgun. WSL's Linux home directory and the Windows user profile each hold their **own** `~/.aws/config` (WSL: `~/.aws/config`; Windows: `C:\Users\\.aws\config`). `aws sso login` refreshes the cached SSO token only for the environment it was run in — a login in Windows PowerShell does not produce a token the WSL side can read, and vice versa. + +When a user on Windows/WSL reports `AccessDenied`, expired-token, or "no credentials" errors from `atx` or `aws` **despite having signed in**, suspect this split before generic re-auth advice. Confirm which environment they run `atx` in (the WSL shell) versus where they ran `aws sso login`. The fix: run `aws sso login --profile ` **in the same WSL shell** where `atx` runs, so it reads the WSL-side `~/.aws/config` and caches the refreshed token under that side's `~/.aws/sso/cache/`. Verify from that same shell with `aws sts get-caller-identity`. (If a user deliberately shares one config across both sides, they must set `AWS_CONFIG_FILE`/`AWS_SHARED_CREDENTIALS_FILE` to the shared path in every shell — but same-environment login is the simpler default.) + ## Environment variables (MCP client config) Pre-set in `mcp.json` to skip an interactive `configure` call: diff --git a/aws-transform/steering/workload-continuous-modernization-analysis.md b/aws-transform/steering/workload-continuous-modernization-analysis.md index 3931499..7a3a6d9 100644 --- a/aws-transform/steering/workload-continuous-modernization-analysis.md +++ b/aws-transform/steering/workload-continuous-modernization-analysis.md @@ -33,11 +33,11 @@ If the user explicitly asks to disable telemetry, omit `--telemetry` for the res - **EC2** -- follow [continuous-modernization-ec2-execution](workload-continuous-modernization-ec2-execution.md) - **Batch** -- follow [continuous-modernization-batch-execution](workload-continuous-modernization-batch-execution.md) -## Repository limit per request (max 250) +## Repository limit per request (max 100) -A single `atx ct analysis run` can be associated with at most **250 repositories**. Before starting an analysis that targets many repos (for example a whole source), check how many repositories are in scope — `atx ct source list` or `atx ct repository list --source ` report the per-source count. (Bare `atx ct status` shows workspace-wide totals across all sources, so prefer a scoped form when the analysis targets a single source.) +A single `atx ct analysis run` can be associated with at most **100 repositories**. Before starting an analysis that targets many repos (for example a whole source), check how many repositories are in scope — `atx ct source list` or `atx ct repository list --source ` report the per-source count. (Bare `atx ct status` shows workspace-wide totals across all sources, so prefer a scoped form when the analysis targets a single source.) -If the scope exceeds 250 repositories, split it into multiple runs, each targeting at most 250 repos (pass the repos in batches via `--repo ::`), and tell the user you are breaking the work up because of the 250-repo-per-request limit. Never issue a single run associated with more than 250 repositories — it will be rejected. Example: 300 repos → two runs (250 + 50); 600 repos → three runs (250 + 250 + 100). +If the scope exceeds 100 repositories, split it into multiple runs, each targeting at most 100 repos (pass the repos in batches via `--repo ::`), and tell the user you are breaking the work up because of the 100-repo-per-request limit. Never issue a single run associated with more than 100 repositories — it will be rejected. Example: 300 repos → three runs (100 + 100 + 100); 600 repos → six runs (100 each). ## Commands @@ -86,6 +86,8 @@ atx ct analysis run --type tech-debt-comprehensive --source --wait --tele tail -f /tmp/atx-analysis.log ``` +The redirect captures the command's diagnostics — the in-process CLI writes logs to STDERR, which `2>&1` folds into the log file. Only warnings and errors are logged by default; when troubleshooting, prefix the command with `ATXCT_LOG_LEVEL=debug` for verbose output (e.g. `ATXCT_LOG_LEVEL=debug atx ct analysis run ... > /tmp/atx-analysis.log 2>&1 &`). + This applies to comprehensive scans, large multi-repo runs, and any analysis the user expects to take a while. Tell the user where the log is and how to check progress. ## Custom Analysis @@ -148,26 +150,6 @@ When polling with `atx ct analysis get --id --json`, the `status` field is **Note:** It's `complete`, NOT `COMPLETED` or `completed`. -## Artifacts - -After an analysis completes, its report artifacts can be listed and retrieved: - -```bash -# List all artifacts for an analysis -atx ct analysis list-artifacts --id --json - -# Get content of a specific artifact -atx ct analysis get-artifact --id --repo :: --name -``` - -### Artifact names by analysis type - -| Analysis Type | Artifact Names | -| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| tech-debt-comprehensive | `report`, `technical-debt-report/summary`, `technical-debt-report/outdated-components`, `technical-debt-report/maintenance-burden`, `technical-debt-report/remediation-plan` | -| agentic-readiness | `ara` (per repo); `_portfolio_ara` (portfolio-level) | -| modernization-readiness | `mod` (per repo); `_portfolio_mod` (portfolio-level) | - ## After Analysis Completes Once an analysis finishes, retrieve its findings by analysis ID and summarize for the user: @@ -175,12 +157,6 @@ Once an analysis finishes, retrieve its findings by analysis ID and summarize fo ```bash # Get findings produced by a specific analysis atx ct findings list --analysis-id --json - -# List artifacts to see available reports -atx ct analysis list-artifacts --id --json - -# Read a specific report -atx ct analysis get-artifact --id --repo :: --name report ``` ## When an analysis returns 0 findings @@ -223,3 +199,21 @@ If an analysis returns 0 findings on a repo that's obviously stale (Java 8, Node ### Pagination (nextToken) Depending on the CLI version, `atx ct analysis list` may return only a bounded page rather than every result — don't assume a fixed response shape. After each call, check whether the response carries a `nextToken`; if it's present and non-empty, call the command again with `--next-token ` and repeat until no `nextToken` remains. Never treat the first page as the complete set when a `nextToken` is present, or you'll silently miss analyses. + +## Tags (resource tagging) + +The `--tags` flag attaches IAM resource tags to an analysis at creation time, and propagates those tags through to any findings the analysis produces. + +```bash +# Run analysis with tags (comma-separated key=value pairs) +atx ct analysis run --type tech-debt-comprehensive --source --tags team=alpha,env=prod --wait --telemetry "agent=,executionMode=local" +``` + +**Behavior:** + +- `--tags key=value,key2=value2` accepts comma-separated pairs in a single flag (e.g. `--tags team=alpha,env=prod`). +- Tags are optional. If omitted, the analysis and its findings are untagged. +- Tags are injected into the `--atxct-configuration` payload passed to the transformation agent. When the agent calls `report_finding`, the tags are forwarded to `BatchCreateFindings` — so findings inherit the analysis's tags automatically. +- If `~/.aws/atx/settings.json` defines `applyTags` (an array of tag maps), those defaults are applied automatically even without explicit `--tags`. An explicit `--tags` override merges **per key** over the settings defaults. + +See the [source](workload-continuous-modernization-source.md) skill's Tags section for the full schema, merge semantics, and error behavior. diff --git a/aws-transform/steering/workload-continuous-modernization-batch-execution.md b/aws-transform/steering/workload-continuous-modernization-batch-execution.md index 5934ee7..4f0682b 100644 --- a/aws-transform/steering/workload-continuous-modernization-batch-execution.md +++ b/aws-transform/steering/workload-continuous-modernization-batch-execution.md @@ -1,1084 +1,274 @@ --- -name: batch-execution -description: Run continuous modernization analysis on AWS Batch (Fargate) using one container per submission. Each container runs `atx ct analysis run` (or `remediation create`) on the customer's logical source, then uploads artifacts via the upload script baked into the image at `/app/upload-ct-artifacts.sh`. +name: remote-batch +description: Run analysis or remediation at scale on AWS Batch (Fargate) using `atx ct remote` CLI commands. One container per (type x repo). Covers provisioning, job submission, status, cancel, and teardown. --- -# continuous modernization Batch/Fargate Execution +# Remote Batch Execution -## ⚠️ MANDATORY: Permission Consent (MUST be first interaction with customer) - -**CRITICAL: The VERY FIRST thing the agent says after the customer chooses Batch/Fargate is the consent message below. Do NOT ask ANY questions (source, analysis type, region, etc.) before showing this message and getting confirmation. No exceptions.** - -"To run the analysis on Batch/Fargate, these resources are created in your account: AWS Transform API access, CloudFormation stacks, Batch compute environments and job queues, a Batch job definition, Lambda functions for job management, S3 buckets for source code and results, a KMS key for encryption, IAM roles for task execution, CloudWatch logs and dashboard, secrets for source credentials, and security agent resources for vulnerability scanning. You will need admin permissions to deploy. After that, executor-only permissions are needed (see `$HOME/.aws/atx/custom/remote-infra/AWSTransformInfrastructureExecutorAccessBatch.json`). Note: AWS Transform does NOT create VPCs, subnets, or NAT gateways — you provide those." - -If the customer says **yes** → proceed with the rest of the workflow. -If the customer says **no** → respond with: "You may encounter permission errors during the setup process. We'll continue, but some steps may fail if permissions are missing." Then proceed with the workflow. +Run analysis or remediation at scale on AWS Batch (Fargate). Each job runs in its own container — fan-out is `types x repos = N containers`. All orchestration is handled by the CLI (`atx ct remote ...`); no raw AWS commands needed. ## Telemetry -When running `atx ct analysis run` or `atx ct remediation create`, always include `--telemetry`. +Include `--telemetry` on every `atx ct remote analysis` and `atx ct remote remediation`: -Format: `--telemetry "agent=,executionMode="` +``` +--telemetry "agent=,executionMode=fargate" +``` -- `agent` — the AI assistant driving this session (lowercase, no spaces). Use the real assistant name — e.g. kiro, claude, amazonq, copilot. +- `agent` — the AI assistant name (lowercase, no spaces): kiro, claude, amazonq, copilot - `executionMode` — `fargate` -If the user explicitly asks to disable telemetry, omit `--telemetry` for the rest of the session. - -Run continuous modernization analysis or remediation on AWS Batch (Fargate) with **one container per submission** (default). Each container starts an `atx ct server`, sets up the source's local config + credentials, and runs analysis or remediation across all repos in the source. For analysis on any provider, and for remediation on `local`-provider sources, artifacts are uploaded to S3. For remediation on `github` / `gitlab` sources, the backend pushes to a result branch — no S3 upload needed. Multiple parallel containers per batch are supported via the submission patterns (B/C) when the customer wants multiple analysis types on one source or multiple sources in one batch. +If the user explicitly opts out of telemetry, omit `--telemetry` for the rest of the session. ## When to Use -- Analyzing or remediating one or more sources at scale on AWS-managed compute (no EC2 to provision) -- Running multiple analysis types in parallel on the same source -- Running analysis/remediation across multiple sources in parallel (e.g., team-A's github + team-B's gitlab) -- Running per-repo parallel analysis (one container per repo via `--repo`) when the customer wants maximum parallelism or per-repo failure isolation -- Customer already has the Custom CDK stack deployed (reuses infrastructure) -- Source contains many repos (atx ct parallelizes up to 8 inside a single container; use per-repo Pattern D for more parallelism) +- Analyzing or remediating many repos in parallel (one container per repo) +- Running multiple analysis types across sources (fan-out: types x repos) +- One-shot batch jobs with no persistent infrastructure between runs +- Customer wants AWS-managed compute (no EC2 instance to manage) -## Architecture +For persistent compute with warm containers, use [remote EC2 execution](workload-continuous-modernization-ec2-execution.md) instead. -``` -Customer's local machine - ↓ atx ct source add (registers source with the backend) - ↓ "Run analysis on Batch" - ↓ aws lambda invoke atx-trigger-batch-jobs (one job per submission) -AWS Batch (Fargate) -- single container per submission - └── Container (public.ecr.aws/d9h8z6l7/aws-transform:latest) - ├── JOB_COMMAND (analysis): - │ - Install / upgrade atx ct CLI - │ - Start atx ct server - │ - github / gitlab: place token in ~/.atxct/sources//_token - │ - local: pull repo bundle from S3, discovery scan with --path override - │ - atx ct analysis run --type --source --wait - │ └─ CT server clones each repo as needed (github / gitlab) and performs analysis - │ - curl + /app/upload-ct-artifacts.sh -- zips each repo's working dir to S3 - │ └─ skipped for tech-debt-quick (no analysis artifacts to capture) - └── Container exits -``` +## Prerequisites -This pattern keeps `atx ct analysis run` as the unit of work. Source attribution is preserved end-to-end — findings carry the customer's source name. - -## Provider Compatibility - -| Provider | Container setup | Analysis output | Remediation flag | Remediation output | -| ---------- | ------------------------------------------------------------------------------------------------ | ------------------------- | -------------------- | ---------------------------------------------------- | -| **github** | Place `github_token` (no `source add` in container) | `code.zip` per repo in S3 | NO `--local` | Result branch pushed to source repo and PR is opened | -| **gitlab** | Place `gitlab_token` (no `source add` in container) | `code.zip` per repo in S3 | NO `--local` | Result branch pushed to source repo and MR is opened | -| **local** | Pull bundle from S3, `discovery scan --path` (idempotent — creates/updates local config + scans) | `code.zip` per repo in S3 | `--local` (required) | `code.zip` per repo in S3 | - -For github / gitlab, the customer must register the source on their own machine first via `atx ct source add --provider github|gitlab --org --token [--url ]`. The container only injects the token at runtime; everything else (provider type, base URL, identifier) comes from the backend's source record at clone time. - -## Step 0: Detect Infrastructure, Then Branch (Provision vs Operate) - -This is the entry gate for every Batch run. It is read-only and safe under any -credentials, including ReadOnly. - -**Network contract:** AWS Transform creates Batch, Lambdas, S3, KMS, IAM, and -security groups via CloudFormation. It does NOT create VPCs, subnets, NAT gateways, -or internet gateways — you provide those. If you don't specify a VPC, deployment -will fail rather than auto-provision. - -**Constraints:** - -- You MUST run this detection BEFORE asking the user anything about jobs, because - the answer decides whether the user needs to provision first or can submit work now. -- You MUST run: - - ```bash - aws cloudformation describe-stacks --stack-name AtxInfrastructureStack \ - --query 'Stacks[0].StackStatus' --output text 2>/dev/null || echo "NOT_DEPLOYED" - ``` - -- If the status is `CREATE_COMPLETE` or `UPDATE_COMPLETE`, You MUST treat the stack - as live and proceed to the OPERATE lifecycle (Step 1 onward — job submission). -- If the status is `NOT_DEPLOYED` or any non-complete state, You MUST enter the - PROVISION lifecycle (Step 0a–0c below) and You MUST NOT attempt to submit jobs, - because the Batch infrastructure does not yet exist. -- You MUST NOT run any provisioning command yourself in this step; detection is - read-only. -- You MUST NEVER run `aws ec2 create-default-vpc` or create any VPC, subnet, NAT - gateway, internet gateway, or route on the user's behalf. If the account has no - suitable VPC, surface the situation to the user and refuse to proceed. - -### Step 0a: Clone repo and discover VPCs - -**Constraints:** - -- You MUST clone (or update) the infra repo FIRST, before any reference to files - inside it: - - ```bash - ATX_INFRA_DIR="$HOME/.aws/atx/custom/remote-infra" - [ -d "$ATX_INFRA_DIR" ] || git clone -b atx-remote-infra --single-branch \ - https://github.com/aws-samples/aws-transform-custom-samples.git "$ATX_INFRA_DIR" - ``` - -- You MUST then discover the user's available VPCs by running: - - ```bash - aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,IsDefault,Tags[?Key==`Name`].Value|[0]]' --output table - ``` - -- Show the results to the user. If no VPCs exist (or none have private subnets with - NAT for Fargate), direct the user to the helper script: - - ``` - A utility script is available to create a Fargate-ready VPC with private subnets, - NAT gateway, and security group: - - cd "$HOME/.aws/atx/custom/remote-infra" && ./create-vpc.sh - - Run this from another terminal with admin credentials. It will print the VPC, - subnet, and security group IDs to use in cdk.json. - ``` - - You MUST NOT run `create-vpc.sh` yourself — present it for the user to run from - another terminal. After they run it, ask them for the output values to continue. -- Ask the user which VPC to use (MANDATORY). You MUST NOT recommend, suggest, or - guide the user toward any VPC (including the default VPC). Present all VPCs - neutrally without commentary on which is "simplest", "easiest", or "looks - pre-configured". The user must make their own informed choice. -- After the user picks a VPC, show its **private** subnets and security groups: - - ```bash - aws ec2 describe-subnets --filters "Name=vpc-id,Values=" "Name=map-public-ip-on-launch,Values=false" \ - --query 'Subnets[*].[SubnetId,AvailabilityZone,Tags[?Key==`Name`].Value|[0]]' --output table - aws ec2 describe-security-groups --filters "Name=vpc-id,Values=" \ - --query 'SecurityGroups[*].[GroupId,GroupName,Description]' --output table - ``` - -- You MUST only present **private subnets** (`MapPublicIpOnLaunch=false`) to the user. - The query above filters them via `map-public-ip-on-launch=false`. This is a hard - requirement: the CDK stack sets `assignPublicIp: DISABLED` on Fargate tasks, so tasks - in public subnets get a private IP that cannot route through an internet gateway. - If the result set is empty (all subnets in the VPC are public), tell the user: - "No private subnets found in this VPC. This stack does not support public subnets — - Fargate tasks require private subnets with NAT gateway for outbound connectivity." - Then direct them to `create-vpc.sh`. - If the VPC contained public subnets that were filtered out, include this note when - presenting results: "Some subnets in this VPC were not shown because they are public - (auto-assign public IP enabled). This stack does not support public subnets — Fargate - tasks require private subnets with NAT gateway for outbound connectivity." -- Then ask the user to select: - 1. `existing_subnet_ids`: which subnets (MANDATORY). - 2. `existing_security_group_id`: which security group (MANDATORY). - 3. Source provider: `github`, `gitlab`, `bitbucket`, or `local`. -- You MUST refuse to proceed without explicit VPC, subnet, and security group - selection — there is no default or fallback path. -- You MUST NOT choose the VPC, subnets, or security group on the user's behalf, even - if only one option exists or one appears obvious. Always ask and wait for the user - to explicitly state their selection before writing to cdk.json. -- You MUST NOT write VPC/subnet/SG values to cdk.json until the user has explicitly - confirmed their choice. Show them what you will write and get a "yes" before - proceeding. - -### Step 0b: Validate network inputs and rewrite cdk.json - -**Constraints:** - -- You MUST verify, and report results to the user, the following BEFORE rewriting - config, because each is a silent deploy-time or runtime failure if wrong: - - The selected subnets do NOT route to an internet gateway. Run: - - ```bash - aws ec2 describe-route-tables \ - --filters "Name=association.subnet-id,Values=," \ - --query 'RouteTables[*].Routes[?DestinationCidrBlock==`0.0.0.0/0`].[GatewayId]' --output text - ``` - - If no explicit association exists for a subnet, also check the VPC's main route - table: - - ```bash - aws ec2 describe-route-tables \ - --filters "Name=vpc-id,Values=" "Name=association.main,Values=true" \ - --query 'RouteTables[*].Routes[?DestinationCidrBlock==`0.0.0.0/0`].[GatewayId]' --output text - ``` - - If any result starts with `igw-`, REJECT and tell the user: "Subnet has a - default route to an internet gateway (igw-...). This stack does not support public - subnets — Fargate tasks deployed here would have no outbound network path. Please - select subnets that route through a NAT gateway or VPC endpoints." - You MUST NOT proceed to cdk.json rewrite if any selected subnet has an IGW route. - - The supplied subnets are in availability zones `${REGION}a` and `${REGION}b`, - because the stack hardcodes those AZs (`lib/infrastructure-stack.ts`). - - The subnets have egress (NAT gateway or VPC endpoints), because the stack does - NOT provision NAT. Tasks need outbound reach to the `atx ct` backend, ECR, S3, - and Secrets Manager. - - If the source is internal/self-hosted: a route exists from those subnets to the - internal git host (VPN / Direct Connect / peering). - - The security group's egress rules permit all of the above. -- You MUST NOT create NAT gateways, routes, VPNs, Direct Connect, or any other - network infrastructure on the user's behalf, because these are production network - changes with cost and blast-radius implications that require explicit human action. - If a precondition is missing, You MUST surface it and hand it to the user or their - network team. -- You MUST rewrite `$ATX_INFRA_DIR/cdk.json` `context` keys from the user's answers - and You MUST show the diff before deploy: - - ```json - "existingVpcId": "", - "existingSubnetIds": [""], - "existingSecurityGroupId": "" - ``` - -- You SHOULD leave `prebuiltImageUri` unchanged unless the user needs a runtime not - in the pre-built image, because blanking it switches to the Docker-required custom - image path. - -### Step 0c: Hand off the deploy (admin-gated) — DO NOT run it yourself - -**Constraints:** - -- You MUST present the deploy command for the user to run, and You MUST NOT run it - yourself, because the stack creates IAM roles and therefore requires admin / - role-creation permissions the agent should not assume. -- You MUST include this caveat verbatim in intent: "This stack creates IAM roles, so - deploying requires admin / role-creation permissions (`iam:CreateRole`, - `iam:PutRolePolicy`, `iam:PassRole`, instance profiles). Run it with an admin - identity. ReadOnly or runtime credentials are sufficient for everything afterward." -- You MUST present the command in in-session form so its output returns to the - conversation: - - ``` - ! cd "$HOME/.aws/atx/custom/remote-infra" && ./setup.sh - ``` - -- After deploy, You MUST direct the user to attach the executor policy - (`$HOME/.aws/atx/custom/remote-infra/AWSTransformInfrastructureExecutorAccessBatch.json`) - to their IAM role/user, so day-to-day job submission needs only least-privilege. -- You MUST stop the PROVISION lifecycle here and wait for the user to confirm the - deploy succeeded before entering the OPERATE lifecycle. -- On success You SHOULD persist `{executionModel:"batch", stackName, region, source, - byoNetwork}` to `.atx/context.json`, so a later session detects and reuses the stack. - -## Step 1: Verify Source and Enumerate Repos - -Before submitting jobs, confirm a source is registered locally (see [continuous-modernization-source.md](workload-continuous-modernization-source.md)) and run discovery (see [continuous-modernization-discovery.md](workload-continuous-modernization-discovery.md)) to get the list of repos. This determines what the container will analyze (Step 5). - -First, list existing registered sources to show the customer what's available: +### Environment ```bash -atx ct source list +export AWS_REGION=us-east-1 # required ``` -Show the list to the customer and ask: +### Permission Model -1. **Which source to analyze?** (pick from the list above) -2. **Source type:** GitHub, GitLab, or local -3. **Repos to analyze:** all repos in source, or a specific subset -4. **Analysis type:** `tech-debt-comprehensive`, `tech-debt-quick`, `security`, `agentic-readiness`, `modernization-readiness` +Two roles are needed: -If the list is empty, the customer wants to register a new source, or needs to update the token on an existing source, use the [continuous-modernization-source](workload-continuous-modernization-source.md) skill (`source add` for new, `source update` for existing), then return here. +- **Admin** — for `provision`, `update`, `teardown`, `credentials`, `network create` +- **Executor** — for `analysis`, `remediation`, `status`, `cancel`, `detect`, `network discover` -Once the source is selected, run discovery and enumerate: +Executor policy: [AWSTransformInfrastructureExecutorAccessBatch](https://code.amazon.com/packages/ATXControlTowerPolicies/blobs/mainline/--/managed-policies/draft/AWSTransformInfrastructureExecutorAccessBatch.json) + `AWSTransformCustomFullAccess` managed policy. -```bash -LOGICAL_SOURCE_NAME="" - -# Run discovery -atx ct discovery scan --source "$LOGICAL_SOURCE_NAME" +### Source Registration -# Enumerate the discovered repos -mapfile -t REPOS < <(atx ct repository list --source "$LOGICAL_SOURCE_NAME" --json | jq -r '.items[].full_name') -REPO_COUNT=${#REPOS[@]} -``` +Before submitting remote jobs, the customer must have a registered source with credentials stored for remote execution: -Works for all source providers (github, gitlab, local) — no provider-specific API calls. See [continuous-modernization-discovery](workload-continuous-modernization-discovery.md) for scan details. +```bash +# GitHub / GitLab +atx ct source add --name --provider github|gitlab --org --token -If the customer wants only a subset of repos, filter `${REPOS[@]}` before submitting. +# Bitbucket Cloud (--username and --email optional but needed for clone/push and API auth) +atx ct source add --name --provider bitbucket --org --token \ + --username --email -## Step 2: Prep Credentials +# Self-hosted GitLab or Bitbucket Data Center (add --url) +atx ct source add --name --provider gitlab|bitbucket --org --token \ + --url https://gitlab.mycompany.com -Give the user the relevant command below to run in their own terminal — do not ask them to paste the token into this chat. +# Local filesystem +atx ct source add --name --provider local --path /path/to/repos -**GitHub HTTPS — store the PAT** (the container fetches it from Secrets Manager at job start): +# Store credential for remote containers (required for github/gitlab/bitbucket) +atx ct remote credentials --source --token --ack -```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/github-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/github-token" \ - --secret-string "$TOKEN"; }; unset TOKEN +# Discover repos +atx ct discovery scan --source ``` -**GitHub SSH — store the private key:** +## Workflow -```bash -aws secretsmanager create-secret --name "atx/ssh-key" \ - --secret-string "$(cat )" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/ssh-key" \ - --secret-string "$(cat )" -``` +### 1. Detect Infrastructure -**GitLab HTTPS — store the PAT** (separate secret, the container fetches it from Secrets Manager at job start): +Check if a Batch stack is already deployed: ```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/gitlab-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/gitlab-token" \ - --secret-string "$TOKEN"; }; unset TOKEN -``` +# List all Batch stacks in the account (mode prefix) +atx ct remote detect --mode batch -**Bitbucket — store the API token** (the container fetches it from Secrets Manager at job start). Email and username are injected into the container command directly (not secrets — they're non-sensitive identifiers): +# Look up one stack directly +atx ct remote detect --mode batch --stack-name -```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/bitbucket-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/bitbucket-token" \ - --secret-string "$TOKEN"; }; unset TOKEN +# Discover the stack by its resource tags +atx ct remote detect --mode batch --tags env=prod,team=platform ``` -**Private package registries** (if the analysis builds the project): see [custom-remote-execution#private-package-registries](custom-remote-execution.md#private-package-registries) for the `atx/credentials` JSON pattern. - -## Step 2b: Validate Credentials (MANDATORY) +`--stack-name` and `--tags` are mutually exclusive. Tag discovery requires the +`tag:GetResources` permission (covered by the Executor policy). Omit both to +list every Batch stack with the `AtxInfrastructureStack` prefix. -**MANDATORY**: The agent MUST verify that the required secret exists in Secrets Manager BEFORE proceeding to Step 4 (Confirm and Submit). Do NOT submit jobs without confirming the credential is present. If the secret is missing, give the user the command to run in their own terminal to create it (do not ask them to paste the token into this chat). +If deployed → skip to step 3 (Submit Analysis). +If not deployed → proceed to step 2 (Provision). -The required secret depends on the source provider: +### 2. Provision -| Provider | Required Secret | -| ------------- | ------------------------ | -| **github** | `atx/github-token` | -| **gitlab** | `atx/gitlab-token` | -| **bitbucket** | `atx/bitbucket-token` | -| **local** | (none — no token needed) | +Requires Admin credentials. The CLI generates the CFN template, creates S3 buckets, bundles Lambda functions, and deploys the stack. -**Step A — Check secret exists:** +**Discover network resources first:** ```bash -# Replace with the provider-specific secret from the table above -aws secretsmanager describe-secret --secret-id --region 2>&1 +atx ct remote network discover --region us-east-1 ``` -- If `ResourceNotFoundException` → inform the user that the secret is missing. Give them the command below to run in their own terminal (do not ask them to paste the token into this chat): - -```bash -read -s TOKEN && { aws secretsmanager create-secret --name "" \ - --secret-string "$TOKEN" --region 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "" \ - --secret-string "$TOKEN" --region ; }; unset TOKEN -``` +Lists VPCs, private subnets, and security groups. Public subnets are excluded. -- If the secret exists → ask the customer: "Your `` token was last updated on ``. Would you like to rotate it, or is the current token still valid?" If they want to rotate: +**If no suitable VPC exists:** ```bash -read -s TOKEN && aws secretsmanager put-secret-value --secret-id "" \ - --secret-string "$TOKEN" --region ; unset TOKEN +atx ct remote network create --region us-east-1 --ack ``` -**Step B — Confirm SCM configuration with user (MANDATORY for non-local providers):** - -Before submitting any batch job (analysis or remediation), ask the user to confirm the SCM provider config that will be injected into the container. An incorrect identifier, email, or username causes clone failures inside the container after the setup phase. - -Present the config to the user via AskUserQuestion: - -| Provider | Fields to confirm | -| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **github** | `identifier` (GitHub org or username that owns the repos) | -| **gitlab** | `identifier` (GitLab group or username that owns the repos) | -| **bitbucket cloud** | `identifier` (Bitbucket workspace), `email` (Atlassian account email for API auth), `username` (Bitbucket username for git clone — visible in clone URLs at bitbucket.org) | -| **bitbucket self-hosted (Data Center)** | `identifier` (Bitbucket project key), `base_url` (instance URL, e.g. `https://bitbucket.corp.example.com`) | - -Example confirmation prompt: - -> "I'll use this config for the Batch job: -> -> - Provider: github -> - Identifier: `github-username` -> -> Is this correct?" - -## Step 3: Prep Local Sources (local source only) - -The single container needs all repos on disk. The skill zips ALL repos in the source as one bundle and uploads it to the managed source bucket: +**Provision the Batch stack:** ```bash -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) -ZIP_NAME="-bundle" # name the bundle (e.g., my-bundle) - -# Zip all repos as siblings in one archive (preserves .git in each) -cd /path/to/repos -zip -qr "/tmp/${ZIP_NAME}.zip" */ -x '*/node_modules/*' -aws s3 cp "/tmp/${ZIP_NAME}.zip" "s3://atx-source-code-${ACCOUNT_ID}/repos/${ZIP_NAME}.zip" -rm -f "/tmp/${ZIP_NAME}.zip" +atx ct remote provision --mode batch \ + --vpc \ + --subnets , \ + --securityGroup \ + --execute --ack ``` -> **Important:** The zip MUST include each repo's `.git/` directory. atx ct's local-provider discovery scanner identifies repos by the presence of `.git`. If the customer pre-emptively excludes `.git`, the scan finds zero repos and analysis fails with `Available: (none)`. - -The skill runs this on the customer's machine. GitHub and GitLab sources don't need this step — the container clones directly from the source provider via PAT. - -## Step 3b: Security Analysis Prerequisites (security type only) - -If `ANALYSIS_TYPE` is `security` (or `agentic-readiness` / `modernization-readiness` which depend on it), the security agent must be configured before submitting. Skip this step for all other analysis types. +Required flags: +- `--mode batch` — required +- `--vpc ` — required +- `--subnets ` — required (comma-separated, private subnets only) +- `--securityGroup ` — required for Batch -**Constraints:** +Optional flags: +- `--suffix ` — custom stack name suffix (default: no suffix) +- `--image-uri ` — container image override (default: public AWS Transform — continuous modernization image) +- `--job-timeout ` — per-attempt timeout, 60..604800 (default: 43200 = 12h) +- `--tags ` — resource tags applied to the stack. These same tags can later target the stack via `--tags` on `detect`/`analysis`/`remediation` (instead of `--stack-name`) -- You MUST verify `~/.atxct/shared/security_agent_config.json` exists locally. If - missing, the customer must run security agent setup first: +Without `--execute`, the command prints the CFN template (dry-run preview). - ```bash - atx ct analysis configure-security - ``` +### 3. Submit Analysis - See [continuous-modernization-setup](workload-continuous-modernization-setup.md) for details. -- You MUST confirm the config contains valid fields before proceeding: - - ```bash - jq -r '.s3Bucket, .role_arn // .roleArn' ~/.atxct/shared/security_agent_config.json - ``` - - If either field is empty or null, the security agent setup is incomplete — ask the - customer to re-run `atx ct analysis configure-security`. -- The CDK stack's `ATXBatchJobRole` already includes `securityagent:*` permissions, - S3 access to the security agent bucket, and `iam:PassRole` on the security agent - role. No additional IAM modifications are needed for the Batch path (unlike the EC2 - path which requires manual role augmentation). -- The security agent config is injected into the container at runtime via base64 - encoding in the job command (handled by `sec_config_inject()` in Step 5). No - manual sync to S3 or instance is needed. - -## Step 4: Confirm and Submit - -**Gate checks** (only the checks relevant to the source provider must pass): - -| Check | github | gitlab | bitbucket cloud | bitbucket self-hosted | local | -| ------------------------------------------------------------- | ------------------ | ------------------ | ----------------------------- | --------------------- | ----- | -| **Credentials (Step 2b-A):** secret exists in Secrets Manager | `atx/github-token` | `atx/gitlab-token` | `atx/bitbucket-token` | `atx/bitbucket-token` | skip | -| **SCM config (Step 2b-B):** confirmed with user | identifier | identifier | identifier + email + username | identifier + base_url | skip | - -Tell the customer what will happen and wait for explicit confirmation. The exact prompt depends on provider: - -**For GitHub:** - -> "I'll submit a Batch job to run `` on Fargate against your GitHub source ``. The container will: -> -> - Place your GitHub PAT (from Secrets Manager `atx/github-token`) — your existing source `` is preserved (no new source created) -> - Run `atx ct analysis run --source ` — atx ct will clone each repo and analyze it -> -> Continue?" - -**For GitLab:** - -> "I'll submit a Batch job to run `` on Fargate against your GitLab source ``. The container will: -> -> - Place your GitLab PAT (from Secrets Manager `atx/gitlab-token`) — your existing source `` is preserved (no new source created) -> - Run `atx ct analysis run --source ` — atx ct will clone each repo and analyze it -> -> Continue?" - -**For Bitbucket Cloud:** - -> "I'll submit a Batch job to run `` on Fargate against your Bitbucket source ``. The container will: -> -> - Place your Bitbucket API token (from Secrets Manager `atx/bitbucket-token`) and inject email/username into config.json -> - Run `atx ct analysis run --source ` — atx ct will clone each repo and analyze it -> -> Continue?" - -**For Bitbucket Data Center:** - -> "I'll submit a Batch job to run `` on Fargate against your Bitbucket Data Center source ``. The container will: -> -> - Place your HTTP Access Token (from Secrets Manager `atx/bitbucket-token`) and inject base_url into config.json -> - Run `atx ct analysis run --source ` — atx ct will clone each repo and analyze it -> -> Continue?" - -**For Local:** - -> "I'll zip your repos at `` (with `.git` included) into a single bundle and upload to `s3://atx-source-code-${ACCOUNT_ID}/repos/.zip`, then submit a Batch job to run `` on Fargate. The container will: -> -> - Download + unzip the bundle -> - Register a new local source named `` in the backend (pointing at the container's `/home/atxuser/repos`) -> - Run discovery to enumerate the repos -> - Run `atx ct analysis run --source ` — atx ct analyzes all repos -> - Upload artifacts to `s3://atx-ct-output-${ACCOUNT_ID}//::/code.zip` -> -> Continue?" - -Do NOT submit until the customer confirms. - -## Step 5: Submit Per-Repo Batch Jobs - -Build one entry per repo and invoke `atx-trigger-batch-jobs`. The Lambda has strict input validation — to pass it, the per-job command must: - -- Start with an allowed prefix (`atx ct` or `atx custom def *`) -- Contain no `$VAR` references, no `$(...)` command substitution, no `${VAR}` brace expansion, no `()` subshells, no `{}` braces, no `*` glob, no `^` regex anchor, no backticks, no `-c` flag (e.g., `sh -c '...'`) -- Contain only ASCII characters — no em-dash `—`, en-dash `–`, smart quotes `""` `''`, or any other Unicode (these get rejected by the Lambda's character allowlist) -- Be on a single line (no `\\` continuations, since post-`tr` they become literal escape-space characters) - -The skill therefore substitutes ALL variables (account ID, source name, repo name, etc.) into the command string locally, before submission. Runtime values that aren't known until the container runs (e.g., the analysis ID printed by `atx ct analysis run`) are extracted into temp files and fed to subsequent commands via `xargs -I VAR command VAR ...`. +Requires Executor credentials. One container per (type x repo). ```bash -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) -BATCH_NAME="atxct-$(date +%s)" -ANALYSIS_TYPE="" # tech-debt-quick | tech-debt-comprehensive | security | agentic-readiness | modernization-readiness | custom -AGENT="" # AI assistant name (kiro, claude, amazonq, etc.) -LOGICAL_SOURCE_NAME="" # the source already registered with atx ct (used as-is) -GITHUB_ORG="" # github source only -- used to build config.json for remediation chains -GITLAB_GROUP="" # gitlab source only -- used to build config.json for remediation chains -BITBUCKET_WORKSPACE="" # bitbucket source only -- workspace (Cloud) or project key (DC) -BITBUCKET_EMAIL="" # bitbucket cloud only -- email for API auth -BITBUCKET_USERNAME="" # bitbucket cloud only -- username for git clone/push -BITBUCKET_BASE_URL="" # bitbucket DC only -- e.g. https://bitbucket.corp.example.com (empty for Cloud) -ZIP_NAME="" # local source only -- name of the bundle uploaded in Step 3 - -# Security analysis only -- base64-encode the security agent config for injection. -# Not a secret (contains resource identifiers only), so no Secrets Manager needed. -SEC_CONFIG_B64="" -if [[ "${ANALYSIS_TYPE}" == "security" ]]; then - SEC_CONFIG_B64=$(base64 < ~/.atxct/shared/security_agent_config.json | tr -d '\n') -fi - -# Upload script -- /app/upload-ct-artifacts.sh is baked into the container image. -# It iterates analysis.repos[] (or remediation.repos.keys[]), zips each repo's -# working directory, and uploads to s3:////::/code.zip. -# Auto-detects analysis vs remediation, resolves per-provider repo path. - -# Helper: security config injection snippet. Returns the command fragment to inject -# security_agent_config.json into the container when ANALYSIS_TYPE is "security". -# Empty string for non-security types. -sec_config_inject() { - if [[ -n "${SEC_CONFIG_B64}" ]]; then - echo " && mkdir -p /home/atxuser/.atxct/shared && echo ${SEC_CONFIG_B64} | base64 -d > /home/atxuser/.atxct/shared/security_agent_config.json" - fi -} - -# GitHub source (analysis) -- inject token (no source-add, no discovery), then run analysis. -# Source must be pre-registered locally by the customer (Step 1 verifies this). -# atx ct internally clones each repo as needed. Analysis details: see [continuous-modernization-analysis.md](workload-continuous-modernization-analysis.md). -# Server log is written to ~/.aws/atx/logs/server.log so it gets included in the upload zip for debugging. -build_command_github() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id atx/github-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/github_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --wait --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log${upload}" -} - -# GitLab source (analysis) -- same injection pattern as GitHub, just different secret/file names. -# atx ct's async provider resolution queries the backend for the source's provider type, -# so we don't need source-add or config.json injection in the container -- the locally-registered -# source's metadata (provider=gitlab, base_url, identifier) is fetched at clone time. -build_command_gitlab() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id atx/gitlab-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/gitlab_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --wait --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log${upload}" -} - -# Bitbucket source (analysis) -- inject token + config.json with email/username (Cloud) or base_url (DC). -# atx ct's async provider resolution queries the backend for the source's provider type. -# Cloud needs email (API auth) and username (git auth). DC needs base_url only. -build_command_bitbucket() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local config_json - if [[ -n "${BITBUCKET_BASE_URL}" ]]; then - # Data Center: provider_config has base_url - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - # Cloud: provider_config has email and username - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - local CONFIG_B64=$(echo "${config_json}" | base64) - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --wait --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log${upload}" -} - -# Local source -- sync repo bundle from atx-source-code, unzip, source-add + discovery, run analysis. -# The bundle is a single zip containing all repos as subdirectories (zipped in Step 3). -# Source semantics: see [continuous-modernization-source.md](workload-continuous-modernization-source.md). Discovery: see [continuous-modernization-discovery.md](workload-continuous-modernization-discovery.md). Analysis: see [continuous-modernization-analysis.md](workload-continuous-modernization-analysis.md). -build_command_local() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/repos && aws s3 cp s3://atx-source-code-${ACCOUNT_ID}/repos/${ZIP_NAME}.zip /tmp/${ZIP_NAME}.zip && unzip -q /tmp/${ZIP_NAME}.zip -d /home/atxuser/repos/ && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} --path /home/atxuser/repos${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --wait --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log${upload}" -} - -# Single container per submission. Source-level analysis runs on ALL repos in the source -# (atx ct internally parallelizes repos within the container). -CMD=$(build_command_local) # or build_command_github, build_command_gitlab, build_command_bitbucket -JOB_NAME="atxct-${BATCH_NAME}" -JOBS_JSON=$(jq -nc --arg cmd "$CMD" --arg name "$JOB_NAME" '[{command: $cmd, jobName: $name}]') -PAYLOAD=$(jq -nc --arg bn "$BATCH_NAME" --argjson jobs "$JOBS_JSON" '{batchName: $bn, jobs: $jobs}') - -aws lambda invoke --function-name atx-trigger-batch-jobs \ - --payload "$PAYLOAD" \ - --cli-binary-format raw-in-base64-out /dev/stdout +atx ct remote analysis \ + --types rapid-techdebt-analysis \ + --sources \ + --mode batch \ + --stack-name \ + --batch-name \ + --telemetry "agent=kiro,executionMode=fargate" ``` -The Lambda returns a `batchId`. Track it for status polling (Step 6). - -### Security analysis: automatic concurrency enforcement - -**When `ANALYSIS_TYPE` is `security`, jobs are automatically routed to a dedicated job queue (`atx-security-job-queue`)** backed by a compute environment capped at 5 concurrent tasks (`maxvCpus = 5 * fargateVcpu`). This means: +Fan-out options: +- `--types type1,type2` — multiple analysis types (rapid-techdebt-analysis, tech-debt-comprehensive, security, agentic-readiness, modernization-readiness, custom) +- `--sources src1,src2` — multiple sources +- `--repos src::repo1,src::repo2` — specific repos (fully qualified) +- `--labels java,spring` — filter repos by labels (AND semantics) +- `--transformation-name ` — required when `--types custom` +- `-g key=value` — configuration for custom transformations -- Submit ALL security jobs in a single batch — no chunking, no polling, no terminal blocking -- AWS Batch queues them and only runs 5 at a time -- As one completes, the next in queue starts immediately (no wasted slots) -- The Security Agent's concurrency limit is respected at the infrastructure level +Stack targeting (choose one): +- `--stack-name ` — target the stack by name +- `--tags env=prod,team=platform` — discover the stack by its resource tags instead of naming it (alternative to `--stack-name`; same tags set at provision time) -The Lambda detects security jobs by checking if the command contains `--type security` and routes them to `atx-security-job-queue` automatically. No special handling needed in the submission script. +The CLI runs pre-flight checks (token validation, source existence), then invokes the trigger Lambda. Returns immediately with a batch ID for polling. -**This automatic routing applies only to security analysis.** For all other analysis types (tech-debt-quick, tech-debt-comprehensive, agentic-readiness, modernization-readiness), jobs go to the general queue with the full 128-job concurrency. - -### Submission patterns - -The Lambda's `jobs: [...]` array supports up to 128 jobs per batch. Jobs in a batch run **in parallel** (no `dependsOn` between them — all dispatch simultaneously, scheduled by AWS Batch as queue capacity allows). Pick the pattern that matches the workload: - -| Pattern | When to use | Jobs per batch | -| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | -| **A. Single analysis on one source** (default above) | One source, one analysis type. Parallelism comes from atx ct's internal handling of repos within the container. | 1 | -| **B. Multiple analysis types on one source** | E.g., quick + comprehensive on the same repos. Findings persist independently with their own analysis_id, queryable separately. | N (one per type) | -| **C. Multiple sources in one batch** | E.g., team-A's github source + team-B's gitlab source. Each source gets its own container with its own logical source name and credentials. | N (one per source, or one per source × type combo) | -| **D. Per-repo parallel analysis** | Customer wants maximum parallelism — one container per repo. Each container analyzes a single repo via `--repo`. Useful when repos are large or the customer wants per-repo isolation/failure boundaries. | N (one per repo) | - -**Pattern B example** — quick + comprehensive on the same source, parallel containers: - -```bash -ANALYSIS_TYPE="tech-debt-quick" -JOB1_CMD=$(build_command_github) - -ANALYSIS_TYPE="tech-debt-comprehensive" -JOB2_CMD=$(build_command_github) - -JOBS_JSON=$(jq -nc \ - --arg cmd1 "$JOB1_CMD" --arg name1 "atxct-quick-${BATCH_NAME}" \ - --arg cmd2 "$JOB2_CMD" --arg name2 "atxct-comp-${BATCH_NAME}" \ - '[ - {command: $cmd1, jobName: $name1}, - {command: $cmd2, jobName: $name2} - ]') -PAYLOAD=$(jq -nc --arg bn "$BATCH_NAME" --argjson jobs "$JOBS_JSON" '{batchName: $bn, jobs: $jobs}') - -aws lambda invoke --function-name atx-trigger-batch-jobs \ - --payload "$PAYLOAD" \ - --cli-binary-format raw-in-base64-out /dev/stdout -``` +Max 250 jobs per submission. -**Pattern C example** — different sources (and providers) in parallel: +### 4. Check Status ```bash -LOGICAL_SOURCE_NAME="team-a-github" -JOB1_CMD=$(build_command_github) - -LOGICAL_SOURCE_NAME="team-b-gitlab" -JOB2_CMD=$(build_command_gitlab) - -JOBS_JSON=$(jq -nc \ - --arg cmd1 "$JOB1_CMD" --arg name1 "atxct-team-a-${BATCH_NAME}" \ - --arg cmd2 "$JOB2_CMD" --arg name2 "atxct-team-b-${BATCH_NAME}" \ - '[ - {command: $cmd1, jobName: $name1}, - {command: $cmd2, jobName: $name2} - ]') -PAYLOAD=$(jq -nc --arg bn "$BATCH_NAME" --argjson jobs "$JOBS_JSON" '{batchName: $bn, jobs: $jobs}') - -aws lambda invoke --function-name atx-trigger-batch-jobs \ - --payload "$PAYLOAD" \ - --cli-binary-format raw-in-base64-out /dev/stdout +atx ct remote status --batch --stack-name ``` -**Pattern D** — per-repo parallel analysis (one container per repo): +Shows per-job status (running/complete/failed) with counts. Omit `--batch` to list recent batches. -The `--repo` flag requires the repo's `slug` value from `atx ct repository list`. Always run `atx ct repository list --source --json` and use the `.slug` field (format: `::`). +Add `--json` for machine-readable output. -Per-repo build command variants — same as the source-level builders but insert `--repo ` before `--wait`: +### 5. Resume a Partial Run -```bash -# GitHub source (per-repo analysis) -- one container per repo for maximum parallelism. -# repoSlug must be set before calling (from `atx ct repository list --json | jq -r '.items[].slug'`). -build_command_github_per_repo() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id atx/github-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/github_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --repo ${repoSlug} --wait 2>&1 | tee /tmp/run.log${upload}" -} - -# GitLab source (per-repo analysis) -build_command_gitlab_per_repo() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id atx/gitlab-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/gitlab_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --repo ${repoSlug} --wait 2>&1 | tee /tmp/run.log${upload}" -} - -# Bitbucket source (per-repo analysis) -build_command_bitbucket_per_repo() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local config_json - if [[ -n "${BITBUCKET_BASE_URL}" ]]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - local CONFIG_B64=$(echo "${config_json}" | base64) - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --repo ${repoSlug} --wait 2>&1 | tee /tmp/run.log${upload}" -} - -# Local source (per-repo analysis) -build_command_local_per_repo() { - local upload="" - if [[ "${ANALYSIS_TYPE}" != "tech-debt-quick" ]]; then - upload=" && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 | xargs -I AID /app/upload-ct-artifacts.sh AID atx-ct-output-${ACCOUNT_ID}" - fi - local sec_inject=$(sec_config_inject) - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/repos && aws s3 cp s3://atx-source-code-${ACCOUNT_ID}/repos/${ZIP_NAME}.zip /tmp/${ZIP_NAME}.zip && unzip -q /tmp/${ZIP_NAME}.zip -d /home/atxuser/repos/ && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} --path /home/atxuser/repos${sec_inject} && atx ct analysis run --type ${ANALYSIS_TYPE} --source ${LOGICAL_SOURCE_NAME} --repo ${repoSlug} --wait 2>&1 | tee /tmp/run.log${upload}" -} -``` - -**Pattern D usage** — iterate repos and build per-repo jobs: +If some jobs in a batch failed: ```bash -JOBS="[]" -while IFS= read -r REPO; do - [ -z "$REPO" ] && continue - repoSlug="$REPO" - REPO_SLUG=$(echo "$REPO" | tr '/:' '-') - CMD=$(build_command_github_per_repo) # or _gitlab_per_repo, _bitbucket_per_repo, _local_per_repo - JOBS=$(echo "$JOBS" | jq --arg cmd "$CMD" --arg name "$REPO_SLUG" '. + [{command: $cmd, jobName: $name}]') -done <<< "$REPOS" - -PAYLOAD=$(jq -nc --arg bn "$BATCH_NAME" --argjson jobs "$JOBS" '{batchName: $bn, jobs: $jobs}') - -aws lambda invoke --function-name atx-trigger-batch-jobs \ - --payload "$PAYLOAD" \ - --cli-binary-format raw-in-base64-out /dev/stdout +atx ct remote analysis --resume --batch-name --mode batch --stack-name ``` -**Why each piece is shaped this way:** - -| Concern | How addressed | -| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Lambda allowlist requires `atx ...` prefix | Command starts with `atx ct --version > /dev/null 2>&1 ;` (allowlist token; exit code intentionally suppressed) | -| Image's `atx ct` is pre-baked | The container image `public.ecr.aws/d9h8z6l7/aws-transform:latest` ships with the production CLI pre-installed; no runtime install step is required. If the image lacks the CLI, the job fails fast with `command not found` and exit code 127. | -| atx ct 3.1+ requires Node 22 | `source nvm.sh && nvm use 22` switches Node version | -| Env from background subshell doesn't reach foreground | PATH + nvm setup in foreground BEFORE backgrounding the server | -| Server needs to be up before commands | `atx ct server > /tmp/server.log 2>&1 &` then `sleep 15` | -| AID capture without `$()` | `atx ct analysis run 2>&1 \| tee /tmp/run.log` then `grep ... \| xargs -I AID` | -| Per-repo scoping | Use `--repo ` (singular). Do NOT use `--repos` (does not exist). The slug comes from `atx ct repository list --json \| jq -r '.items[].slug'` (format: `::`). | -| Customer's source name preserved (no per-repo suffix) | GitHub/GitLab: token-file injection bypasses `source add`'s backend conflict. Local: `discovery scan --path` overrides the path on the existing source without conflicting (idempotent across runs). | -| Re-running batch on the same source | Local provider: `discovery scan --path` updates the registered source's path without 409 errors. GitHub/GitLab: token injection has no equivalent conflict — works every run. | +Re-submits only the non-completed jobs from the original batch. -**Container's IAM role (`ATXBatchJobRole`)** auto-provides credentials for `aws s3 cp`, `aws secretsmanager get-secret-value`, and `atx ct` backend calls. Defined in the Custom CDK stack (`lib/infrastructure-stack.ts`) — no per-job credential setup needed. - -**MCP configuration (optional):** If the customer has a local MCP config (`~/.aws/atx/mcp.json`), include it on each job: +### 6. Cancel ```bash -MCP_CONFIG=$(cat ~/.aws/atx/mcp.json 2>/dev/null || echo "null") -# Add "mcpConfig": $MCP_CONFIG to each job entry above -``` - -### Remediation jobs (instead of analysis) - -**Gate check**: Before submitting any remediation batch job, the agent MUST confirm that Step 2b was completed for the relevant provider (see gate checks table in Step 4). Local providers skip credentials and SCM config checks. - -To run a remediation that fixes findings produced by a prior analysis, swap the `analysis run` block for `remediation create --ids` (see [continuous-modernization-remediation.md](workload-continuous-modernization-remediation.md) and [continuous-modernization-findings.md](workload-continuous-modernization-findings.md)). - -The remediation flow differs by source provider: - -| Provider | Pattern | Output | -| ------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------- | -| **github** | NO `--local` — backend pushes to a result branch | Result branch pushed to source repo and PR is opened | -| **gitlab** | NO `--local` — backend pushes to a result branch | Result branch pushed to source repo and MR is opened | -| **bitbucket** | NO `--local` — backend pushes to a result branch | Result branch pushed to source repo and PR is opened | -| **local** | `--local` (required) — transform runs in the container, working dir is captured to S3 | `code.zip` per repo in S3 | - -`atx ct remediation create` does NOT support `--wait`, so we poll until terminal status (`complete`, `failed`, or `cancelled`). Use `while true` (no iteration cap) — AWS Batch's job timeout is the upper safety net. - -### Three remediation flag combinations - -`atx ct remediation create` accepts three valid flag combinations. Pick based on whether findings have `.fix` populated: - -| Combination | When to use | Capture pattern | -| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -| `--ids X` (alone) | Findings have `.fix` populated (typical for `tech-debt-quick`). Backend uses each finding's `.fix.transform_name` to pick the transformation. | `jq -r '.[] \| select(.fix != null) \| .id'` | -| `--ids X --transformation-name Y` | Findings WITHOUT `.fix` (typical for `tech-debt-comprehensive`, `security` issues without auto-fix). Customer overrides with an explicit transformation that applies to those findings. | `jq -r '.[] \| select(.category == "Java") \| .id'` (filter by category, not `.fix`) | -| `--transformation-name Y --repo Z` | No findings dependency. Run a transformation directly on a specific repo. | (no findings capture) | - -Set `TRANSFORMATION_NAME` to use the override; leave it empty to rely on each finding's `.fix`. +# Cancel all jobs in a batch +atx ct remote cancel --batch --stack-name -```bash -# On customer's laptop: collect finding IDs to remediate - -# Pattern 1 (default): all auto-remediable findings (.fix populated) -FINDING_IDS=$(atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json \ - | jq -r '.[] | select(.fix != null) | .id' \ - | tr '\n' ',' | sed 's/,$//') -TRANSFORMATION_NAME="" - -# Pattern 2 (hybrid): specific category of findings + explicit transformation override -# Example: all Java-category findings, regardless of .fix populated. Apply java-version-upgrade. -# FINDING_IDS=$(atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json \ -# | jq -r '.[] | select(.category == "Java") | .id' \ -# | tr '\n' ',' | sed 's/,$//') -# TRANSFORMATION_NAME="AWS/java-version-upgrade" - -# Or filter by severity / repo / specific finding IDs - -REMEDIATION_NAME="multi-fix-$(date +%s)" # unique per submission - -# GitHub source (remediation) -- no --local, no upload. Backend dispatches a GitHub Actions -# workflow that runs the transform and pushes a result branch to the source repo. -# The customer reviews and opens a PR in github. -# -# WORKAROUND: atx ct discovery scan throws SETUP_REQUIRED for github sources when local -# config.json is missing -- even when the github_token file is present. -# We inject a minimal config.json via base64 to satisfy the local-config check. -build_command_remediation_github() { - local CONFIG_B64=$(printf '{"provider":"github","identifier":"%s"}' "${GITHUB_ORG}" | base64) - # Optional transformation override (for findings without .fix populated, e.g. tech-debt-comprehensive) - local TRANSFORM_FLAG="" - [ -n "${TRANSFORMATION_NAME}" ] && TRANSFORM_FLAG="--transformation-name ${TRANSFORMATION_NAME} " - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/github-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/github_token && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} && atx ct remediation create --ids ${FINDING_IDS} ${TRANSFORM_FLAG}--name ${REMEDIATION_NAME} --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 > /tmp/rid.txt && while true ; do cat /tmp/rid.txt | xargs -I RID atx ct remediation status --id RID > /tmp/status.txt ; grep -qE 'complete|completed|failed|cancelled' /tmp/status.txt && break ; sleep 30 ; done" -} - -# GitLab source (remediation) -- same as github with gitlab_token. Backend pushes to a -# result branch and MR is opened. -# Same SETUP_REQUIRED workaround as github -- minimal config.json injected via base64. -build_command_remediation_gitlab() { - local CONFIG_B64=$(printf '{"provider":"gitlab","identifier":"%s"}' "${GITLAB_GROUP}" | base64) - # Optional transformation override (for findings without .fix populated, e.g. tech-debt-comprehensive) - local TRANSFORM_FLAG="" - [ -n "${TRANSFORMATION_NAME}" ] && TRANSFORM_FLAG="--transformation-name ${TRANSFORMATION_NAME} " - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/gitlab-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/gitlab_token && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} && atx ct remediation create --ids ${FINDING_IDS} ${TRANSFORM_FLAG}--name ${REMEDIATION_NAME} --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 > /tmp/rid.txt && while true ; do cat /tmp/rid.txt | xargs -I RID atx ct remediation status --id RID > /tmp/status.txt ; grep -qE 'complete|completed|failed|cancelled' /tmp/status.txt && break ; sleep 30 ; done" -} - -# Bitbucket source (remediation) -- inject token + config.json, run remediation. -# Same pattern as github/gitlab -- config.json injected via base64. -# Cloud: email + username in provider_config. DC: base_url in provider_config. -build_command_remediation_bitbucket() { - local config_json - if [[ -n "${BITBUCKET_BASE_URL}" ]]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - local CONFIG_B64=$(echo "${config_json}" | base64) - # Optional transformation override (for findings without .fix populated, e.g. tech-debt-comprehensive) - local TRANSFORM_FLAG="" - [ -n "${TRANSFORMATION_NAME}" ] && TRANSFORM_FLAG="--transformation-name ${TRANSFORMATION_NAME} " - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} && atx ct remediation create --ids ${FINDING_IDS} ${TRANSFORM_FLAG}--name ${REMEDIATION_NAME} --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 > /tmp/rid.txt && while true ; do cat /tmp/rid.txt | xargs -I RID atx ct remediation status --id RID > /tmp/status.txt ; grep -qE 'complete|completed|failed|cancelled' /tmp/status.txt && break ; sleep 30 ; done" -} - -# Local source (remediation) -- `--local` runs the transform in the container; results are -# zipped from the container's working dir and uploaded to S3. -build_command_remediation_local() { - # Optional transformation override (for findings without .fix populated, e.g. tech-debt-comprehensive) - local TRANSFORM_FLAG="" - [ -n "${TRANSFORMATION_NAME}" ] && TRANSFORM_FLAG="--transformation-name ${TRANSFORMATION_NAME} " - echo "atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15 ; mkdir -p /home/atxuser/repos && aws s3 cp s3://atx-source-code-${ACCOUNT_ID}/repos/${ZIP_NAME}.zip /tmp/${ZIP_NAME}.zip && unzip -q /tmp/${ZIP_NAME}.zip -d /home/atxuser/repos/ && atx ct discovery scan --source ${LOGICAL_SOURCE_NAME} --path /home/atxuser/repos && atx ct remediation create --ids ${FINDING_IDS} ${TRANSFORM_FLAG}--name ${REMEDIATION_NAME} --local --telemetry \"agent=${AGENT},executionMode=fargate\" 2>&1 | tee /tmp/run.log && grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1 > /tmp/rid.txt && while true ; do cat /tmp/rid.txt | xargs -I RID atx ct remediation status --id RID > /tmp/status.txt ; grep -qE 'complete|completed|failed|cancelled' /tmp/status.txt && break ; sleep 30 ; done && cat /tmp/rid.txt | xargs -I RID /app/upload-ct-artifacts.sh RID atx-ct-output-${ACCOUNT_ID}" -} +# Cancel a single job +atx ct remote cancel --job --stack-name ``` -Then call `build_command_remediation_github`, `build_command_remediation_gitlab`, `build_command_remediation_bitbucket`, or `build_command_remediation_local` instead of the analysis builder. Everything else (Lambda invoke, polling, status retrieval) is identical to analysis. - -**Why `while true` instead of a bounded loop:** - -| Concern | Resolution | -| ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | -| Remediation duration is unpredictable | No iteration cap — poll until terminal status | -| Container could hang forever on a stuck remediation | AWS Batch's Job Definition timeout (default 12h, configurable) is the upper bound | -| Customer wants to cancel | `aws batch terminate-job` from outside; or `atx ct remediation cancel` server-side | - -The container exits ONLY when: - -1. Status reaches `complete`/`failed`/`cancelled` → upload runs (local only) → exit 0 -2. Batch hits its job timeout → forced kill → exit non-zero (no upload) - -Keep your Job Definition timeout generous for remediation jobs. - -## Step 6: Poll Status +### 7. Submit Remediation -Poll every 60 seconds for the first 10 polls, then every 5 minutes. Report only on status changes. +Requires completed analysis with findings. ```bash -aws lambda invoke --function-name atx-get-batch-status \ - --payload "{\"batchId\":\"${BATCH_ID}\"}" \ - --cli-binary-format raw-in-base64-out /dev/stdout +atx ct remote remediation \ + --sources \ + --min-severity medium \ + --mode batch \ + --stack-name \ + --batch-name rem-1 \ + --telemetry "agent=kiro,executionMode=fargate" ``` -Job statuses: `SUBMITTED`, `PENDING`, `RUNNABLE`, `STARTING`, `RUNNING`, `SUCCEEDED`, `FAILED`. +Filter options: +- `--ids ` — specific finding IDs +- `--sources ` / `--repos ::` — by source/repo +- `--severity high` — exact severity match +- `--min-severity medium` — minimum threshold +- `--labels ` — filter by repo labels +- `--transformation-name ` — override fix strategy +- `-g, --configuration ` — config path or key=value pairs (only valid with `--transformation-name`) -## Step 7: Get Findings and Artifacts +Stack targeting is the same as analysis: pass `--stack-name ` or discover the stack by its resource tags with `--tags env=prod,...` (alternative to `--stack-name`). -**Findings** are persisted by the analysis runner during execution and queryable via: +### 8. Get Results -```bash -atx ct findings list --source --json -``` +Status output already shows key result info for completed jobs: +- **Analysis**: `resultId` and finding count (e.g., `(3 findings)`) +- **Remediation**: `resultId` and PR/MR URL (for SCM sources) or S3 output path (for local sources) -All findings are persisted under the customer's `LOGICAL_SOURCE_NAME` (single container does the analysis). One query gets everything (see [continuous-modernization-findings.md](workload-continuous-modernization-findings.md)): +For deeper detail on findings: ```bash -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json +atx ct analysis get --id --json ``` -**S3 artifacts** are uploaded by `/app/upload-ct-artifacts.sh` (baked into the container image). Analysis artifacts are written for any provider; remediation artifacts are only written for `local`-provider remediations (github/gitlab remediations push a result branch instead — no S3): +### 9. Update Stack -``` -s3://atx-ct-output-{account-id}/// - code.zip -- the working directory after the analysis or remediation completes, - including a result branch with auto-committed changes (e.g., - `atx-result-staging-` for analysis documentation, or the - remediation's branch for `--local` runs). The customer can `git log` - and `git diff` to review what the bot changed. `.git/` is preserved - for this reason. - Excludes node_modules/, .env*, *.pem, *.key, .aws/. - logs.zip -- cherry-picked debug logs: - ATX CLI debug logs, error log, conversation transcript, - plan.json, validation_summary.md. -``` - -To download: +Update to the latest CFN template: ```bash -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) - -# All artifacts for one analysis -aws s3 sync s3://atx-ct-output-${ACCOUNT_ID}/${ANALYSIS_ID}/ ./artifacts/ - -# Just one repo's reports -aws s3 cp s3://atx-ct-output-${ACCOUNT_ID}/${ANALYSIS_ID}/${REPO_SLUG}/code.zip ./ +atx ct remote update --mode batch --stack-name --execute --ack ``` -Surface findings to the user as the primary result. Reference S3 artifacts only if the user asks for raw reports/logs or you need to debug a finding. +Without `--execute`, shows changeset preview. -**Cancellation note:** If a Batch job is terminated mid-run, the upload step does not run and that container's artifacts are lost. Findings already persisted survive (the analysis runner pushes them mid-flight, before the upload step). - -## Cancellation +### 10. Teardown ```bash -# Cancel one job -aws lambda invoke --function-name atx-terminate-job \ - --payload "{\"jobId\":\"\"}" \ - --cli-binary-format raw-in-base64-out /dev/stdout - -# Cancel all jobs in a batch -aws lambda invoke --function-name atx-terminate-batch-jobs \ - --payload "{\"batchId\":\"\"}" \ - --cli-binary-format raw-in-base64-out /dev/stdout +atx ct remote teardown --mode batch --stack-name --execute --ack ``` -## Container Customization - -The Batch container is the pre-built `public.ecr.aws/d9h8z6l7/aws-transform:latest` image, which includes Java 8/11/17/21/25, Python 3.8-3.14, Node.js 16-24, Maven, Gradle, common build tools, AWS CLI v2, and the AWS Transform CLI (including `atx ct`) pre-installed. The JOB_COMMAND runs `atx ct --version` as a smoke test at job start — no runtime install step is required. - -For continuous modernization analyses, the pre-built image's defaults handle every runtime need. No customization required. - -If a customer brings their own TD that requires a runtime or tool not in the pre-built image (e.g., Rust, Go, .NET on Linux), follow the Custom Image Path in [custom-remote-execution](custom-remote-execution.md#custom-image-path-docker-required): - -1. Clone the infrastructure repo (already done if Custom is set up) -2. Edit the Dockerfile to add the required runtime/tool — see [Adding Languages or Tools](custom-remote-execution.md#adding-languages-or-tools) -3. Re-run `./setup.sh` from the cloned directory - -## Runtime Version Switching - -For remediation runs that target a specific language version (e.g., `AWS/java-version-upgrade` targeting Java 21), pass the version as an environment variable on each job in the `jobs` array: - -```json -{ - "command": "...", - "jobName": "...", - "environment": { - "JAVA_VERSION": "21", - "NODE_VERSION": "22", - "PYTHON_VERSION": "3.13" - } -} -``` - -Available versions: - -- **Java**: 8, 11, 17, 21, 25 (Amazon Corretto) -- **Python**: 3.8-3.14 (accepts `3.13` or `13`) -- **Node.js**: 16, 18, 20, 22, 24 - -For analyses (tech-debt-comprehensive, agentic-readiness, modernization-readiness), runtime switching is generally not needed. Pass these env vars only when running remediation TDs that need a specific target version. - -See [custom-remote-execution#version-switching-at-runtime](custom-remote-execution.md#version-switching-at-runtime) for the full reference. - -## Limits - -- Max 128 concurrent Batch jobs (per the existing CDK config) -- **Max 5 concurrent Batch jobs for `--type security` (infrastructure-enforced)** — the Security Agent backend caps concurrent code-review executions at 5. A dedicated compute environment (`atx-fargate-security`, `maxvCpus = 5 * fargateVcpu`) and job queue (`atx-security-job-queue`) enforce this at the AWS Batch level. The Lambda automatically routes security jobs to this queue — no client-side chunking needed. Submit all security jobs in one batch; AWS Batch queues excess jobs and runs them as slots free up. -- Max job duration: defined by the CDK stack -- Bedrock throughput is per-account — running many parallel continuous modernization containers shares the quota; large batches may throttle -- Backend (atx ct API) rate limits at ~30+ concurrent calls. Step 5's chunked-submit pattern (chunks of 8) keeps within limits +S3 buckets (source code, outputs) and Secrets Manager tokens are preserved. VPC/subnets are customer-managed and not deleted. ## Error Handling -| Error | Cause | Fix | -| ------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------- | -| Job stuck in `RUNNABLE` | No Fargate capacity | Wait or check service quota | -| Job fails with auth error | Task role missing Bedrock, SecurityAgent, or atx-ct-output access | Update task role; customer re-runs `setup.sh` | -| Container can't fetch PAT | `atx/github-token` (or `atx/gitlab-token`) secret missing | Customer creates the secret (see Step 2) | -| Container can't fetch SSH key | `atx/ssh-key` secret missing | Customer creates the secret (see Step 2) | -| Container can't read local source zip | S3 path incorrect or zip not uploaded | Verify `s3://atx-source-code-${ACCOUNT_ID}/repos/.zip` exists | -| `atx ct discovery scan` fails | Source registration failed (bad PAT, bad path, etc.) | Check container logs; fix credentials or path | -| `atx ct analysis run` clone fails | PAT expired, repo private to a different account, etc. | Verify customer's PAT has access to the repo | -| Findings missing after analysis | Server crashed before persisting | Check CloudWatch logs for errors | -| Artifacts missing from S3 | Upload script failed | Check container logs for `[date] Uploaded` or `Skip` lines from the staging script | - -## Pricing - -Direct customer to: - -- AWS Batch / Fargate pricing: https://aws.amazon.com/fargate/pricing/ -- AWS Transform agent minutes: https://aws.amazon.com/transform/pricing/ - -Do NOT quote specific dollar amounts or time estimates. - -## Cleanup +| Error | Cause | Fix | +|---|---|---| +| `Stack not deployed` | No Batch infra | Run `atx ct remote provision --mode batch ...` | +| `Token invalid for source` | Expired/revoked SCM token | Run `atx ct remote credentials --source --token --ack` | +| `Job count exceeds Lambda batch limit of 250` | Too many type x repo combinations | Split into multiple submissions | +| `Batch name already exists` | Duplicate batch name | Use a unique `--batch-name` or omit for auto-generated | +| `No repos resolved` | Source has no repos or labels don't match | Check `atx ct repository list --source ` | +| `No deployed stack found for tags` | `--tags` matched no deployed stack | Verify tags with `atx ct remote detect --mode batch --tags `, or target by `--stack-name` | +| `--securityGroup is required for --mode batch` | Missing required flag | Add `--securityGroup ` | +| `Public subnet(s) are not allowed` | Provided public subnets | Use private subnets only (with NAT for internet) | -After every Batch run completes, prompt the user with the following: - -> Your remote infrastructure is still deployed in your AWS account. All services -> are pay-per-use only — there are no fixed costs when idle. You can leave it in -> place for future analyses, or tear it down now. -> -> For pricing details: https://aws.amazon.com/transform/pricing/ -> -> If you tear down: -> -> - Batch compute, Lambda functions, IAM roles, log groups — all deleted -> - S3 buckets, KMS key, and VPC/network resources are preserved -> - You'll need to re-run setup next time you use remote mode -> -> Would you like to keep the infrastructure or tear it down? - -If the user chooses to tear down: - -**Constraints:** - -- You MUST NOT run `teardown.sh` yourself, because it deletes IAM roles and - CloudFormation stacks which requires admin permissions. -- You MUST present the command for the user to run from another terminal: - - ```bash - cd "$HOME/.aws/atx/custom/remote-infra" && ./teardown.sh - ``` - -- You MUST tell the user: "Teardown requires the same admin permissions used for - setup. Run this from another terminal with admin credentials. The script will show - what will be deleted and ask for confirmation before proceeding. Use `--dry-run` to - preview without deleting." +## Limits -If the user chooses to keep it: "Infrastructure will stay deployed. Next time you run a remote analysis, everything will be ready immediately." +- Max 250 jobs per submission (fan-out ceiling) +- Job timeout: 60s to 604800s (default 12h) +- Batch stack names must start with `AtxInfrastructureStack` +- `--execute` flag gates all destructive operations (provision/update/teardown) diff --git a/aws-transform/steering/workload-continuous-modernization-discovery.md b/aws-transform/steering/workload-continuous-modernization-discovery.md index 098dee0..54f5ad2 100644 --- a/aws-transform/steering/workload-continuous-modernization-discovery.md +++ b/aws-transform/steering/workload-continuous-modernization-discovery.md @@ -5,10 +5,6 @@ description: Scan/discover repositories from GitHub orgs, Gitlab group/user, or # Discovery -## Prerequisites - -Check if the server is running with `atx ct status --health`. If any command fails with a connection error, use the `server` skill to start the server. - ## Local sources: path is set at `source add` time For local sources, the directory path is provided when the source is first added (`atx ct source add --provider local --name --path `). It's stored on the source and reused automatically by subsequent `discovery scan --source ` calls — no `--path` needed at scan time. diff --git a/aws-transform/steering/workload-continuous-modernization-ec2-execution.md b/aws-transform/steering/workload-continuous-modernization-ec2-execution.md index 23b937e..4618502 100644 --- a/aws-transform/steering/workload-continuous-modernization-ec2-execution.md +++ b/aws-transform/steering/workload-continuous-modernization-ec2-execution.md @@ -1,2339 +1,307 @@ --- -name: ec2-execution -description: Run continuous modernization analysis or remediation on a single EC2 instance with one long-running atx-ct container. Customer launches the instance once (or reuses an existing one); agent submits work via SSM SendCommand. Each submission auto-polls and auto-uploads artifacts. +name: remote-ec2 +description: Run analysis or remediation on a persistent EC2 instance using `atx ct remote` CLI commands. One long-running container with 1-5 workers; jobs dispatched via SSM. Covers provisioning, job submission, status, cancel, BYO instance, and teardown. --- -# continuous modernization EC2 Execution +# Remote EC2 Execution -## Telemetry - -When running `atx ct analysis run` or `atx ct remediation create`, always include `--telemetry`. - -Format: `--telemetry "agent=,executionMode="` -- `agent` -- the AI assistant driving this session (lowercase, no spaces). Use the real assistant name -- e.g. kiro, claude, amazonq, copilot. -- `executionMode` -- `ec2` - -If the user explicitly asks to disable telemetry, omit `--telemetry` for the rest of the session. - -Run continuous modernization analysis or remediation on a single EC2 instance with **one long-running atx-ct container**. The customer provisions the instance via a **CloudFormation stack** (atomic deploy/rollback, single-command teardown) -- or reuses an existing one. The container hosts `atx ct server` and stays up between submissions. The agent submits work via SSM SendCommand -- each submission runs in the background on the instance and includes auto-upload of artifacts to S3 (for analysis on any provider, and for `--local` remediation; github/gitlab remediations push a result branch instead -- no S3 upload). +Run analysis or remediation on a persistent EC2 instance with one or more long-running containers. The instance stays up between submissions — useful for repeated analyses on the same sources without cold-start. Jobs are dispatched via SSM SendCommand and distributed round-robin across workers. -## When to Use - -- Re-running multiple analyses against the same source(s) on the same compute -- Customer prefers a persistent dev box they can Session-Manager into -- Avoiding Batch cold-start (Fargate provisioning adds latency per job) -- Workloads that benefit from a warm container (multiple analyses without re-installing the CLI) - -For one-shot or fan-out workloads (many sources analyzed in parallel), use [continuous-modernization-batch-execution](workload-continuous-modernization-batch-execution.md) instead. +## Telemetry -## Architecture +Include `--telemetry` on every `atx ct remote analysis` and `atx ct remote remediation`: ``` -Customer's local machine EC2 instance (CFN-managed) - ↓ atx ct source add ┌────────────────────────────────┐ - ↓ aws s3api create-bucket (idempotent) │ atx-ct container (long-running)│ - ↓ aws cloudformation create-stack │ - atx ct server (running) │ - │ │ -[Setup, once via CFN] │ CFN stack contains: │ - CFN provisions: IAM role, profile, │ - 1× EC2 instance │ - security group, EC2 instance. │ - 1× IAM role + profile │ - UserData installs Docker, pulls image, ────┼─→ - 1× security group │ - starts container, signals CREATE_COMPLETE │ (S3 buckets are outside the │ - │ stack -- persist across │ - │ delete-and-recreate) │ -[Per submission, via SSM] │ │ - build_command_*() returns a nohup'd ───────┼─→ background script: │ - chain; SSM returns immediately │ 1. fetch token from SecMgr │ - │ 2. atx ct analysis run │ -[Status check, on demand] │ 3. poll status until done │ - ssm_run "atx ct analysis get --id X" ──────┼─→ 4. /app/upload-ct-artifacts │ - └────────────────────────────────┘ -[Teardown] - aws cloudformation delete-stack - → instance + IAM + SG removed atomically - (S3 buckets and Secrets Manager entries persist) +--telemetry "agent=,executionMode=ec2" ``` -The container persists across submissions. Customer can run many analyses (or one analysis followed by remediation) without re-provisioning anything. The build_command_*() functions encapsulate the entire submit → poll → upload flow as one nohup'd script, so the agent stays free during long-running work. CFN's `CreationPolicy` ensures `CREATE_COMPLETE` only fires after UserData verifies the container is up -- there's no "is the container ready yet?" race. - -## Provider Compatibility - -| Provider | Container setup at job time | Analysis output | Remediation flag | Remediation output | -|---|---|---|---|---| -| **github** | Fetch token from `atx/github-token` (Secrets Manager) → place in `~/.atxct/sources//github_token` | `code.zip` per repo in S3 | NO `--local` | Result branch pushed to source repo and PR is opened | -| **gitlab** | Fetch token from `atx/gitlab-token` → place in `~/.atxct/sources//gitlab_token` | `code.zip` per repo in S3 | NO `--local` | Result branch pushed to source repo and MR is opened | -| **bitbucket** | Fetch token from `atx/bitbucket-token` → place in `~/.atxct/sources//bitbucket_token` + inject `config.json` with email/username (Cloud) or base_url (DC) | `code.zip` per repo in S3 | NO `--local` | Result branch pushed to source repo and PR is opened | -| **local** | Pull bundle from S3 (Step 7), `discovery scan --path /home/atxuser/repos` | `code.zip` per repo in S3 | `--local` (required) | `code.zip` per repo in S3 | - -For github / gitlab / bitbucket, the customer must register the source on their own machine first via `atx ct source add --provider github|gitlab|bitbucket --org --token `. The token is then stored in Secrets Manager (Step 4) and fetched by the container at job time. atx ct's async provider resolution queries the backend for source metadata (provider type, base URL, identifier) at clone time. Bitbucket additionally requires a `config.json` with email/username (Cloud) or base_url (Data Center) injected into the container -- unlike github/gitlab where the backend has all needed metadata. - -## Multi-Worker Support - -**Default behavior (`WorkerCount=1`): a single container** named `atx-ct` (legacy behavior), sized for the default `m5.2xlarge` (32 GB) with a 50 GB volume. Each worker runs its own atx ct server. The default is 1 because every worker is memory-capped at `(instance RAM - 4 GB) / WorkerCount`, so a higher WorkerCount must be paired with a larger InstanceType; defaulting to 1 keeps a no-preference customer on a right-sized box rather than over-provisioning. For multi-analysis parallelism the customer chooses N>1 and the laptop-side provisioning script auto-sizes the InstanceType to match (m5.4xlarge for 2-4, m5.8xlarge for 5). The agent targets a specific worker by setting `WORKER_NUM` (1..WorkerCount) before calling SSM helpers or `build_command_*()`. - -To run multiple analyses in parallel, set `WORKER_COUNT` to any value in 1-5; the auto-sized instance type and disk grow to match. Note WorkerCount is fixed at stack-create time -- raising it later requires a destructive redeploy (see "Changing WorkerCount Requires Redeploy"), so a customer who knows they will need parallelism can opt into a higher count up front. - -### Choosing WorkerCount - -| Customer intent | WorkerCount | -|---|---| -| "Scan my source" / "tech-debt-quick on source X" / "find vulnerabilities" | 1 is sufficient | -| Default (no specific parallelism request) | 1 (right-sized m5.2xlarge; each worker is memory-capped, so 1 avoids over-provisioning) | -| "Run tech-debt AND security in parallel" on the same source | 2+ (one per analysis type) | -| "Analyze sources A, B, C concurrently" | 3+ (one per source) | -| "Run a separate analysis on each repo in this source in parallel" | N where N is the repo count, capped at 5 | -| 6+ truly parallel jobs | Use the Batch path instead (Fargate scales to 64 concurrent) | - -If the customer's intent is unclear, ASK: "Do you want one analysis covering everything (single AID, simpler reporting), or N separate analyses running in parallel (one AID per item)?" Default to 1 if they have no preference; the laptop-side auto-sizing matches WorkerCount=1 to m5.2xlarge with a 50 GB volume, so they pay only for what they need. If they expect parallelism later, mention they can opt into a higher count now to avoid a destructive redeploy. - -When proposing a WorkerCount for a new CFN stack in the consent prompt, ALWAYS include this warning: "Note: WorkerCount is fixed at stack-create time. Changing it later requires the admin to redeploy the stack (which causes downtime)." This warning does NOT apply to the existing-instance path; there's no stack to redeploy, and WorkerCount can be changed by stopping/starting containers. +- `agent` — the AI assistant name (lowercase, no spaces): kiro, claude, amazonq, copilot +- `executionMode` — `ec2` -### Sizing +If the user explicitly opts out of telemetry, omit `--telemetry` for the rest of the session. -Each worker uses ~3-4 vCPU average and ~4-8 GB RAM for typical single-repo analyses, peaking at ~16 GB for monorepos. The skill auto-picks based on WorkerCount: - -- 1 worker: m5.2xlarge, 50 GB -- 2-4 workers: m5.4xlarge, 100-200 GB -- 5 workers: m5.8xlarge, 250-500 GB - -The pre-deploy confirmation prompt shows the resolved config before the customer commits. Customer can override `INSTANCE_TYPE` and `VOLUME_SIZE` env vars to deviate from the auto-pick. - -For monorepos (>5 GB working tree per repo) or running many source-wide analyses concurrently, override to `INSTANCE_TYPE=m5.12xlarge` (48 vCPU, 192 GB RAM) and `VOLUME_SIZE=1000`. The default sizing assumes typical single-repo fan-out work. Each worker has its own isolated filesystem inside its container, so repos cloned by worker 1 are not visible to worker 2. +## When to Use -### Bridge Networking +- Re-running multiple analyses on the same compute (warm container, no cold-start) +- Customer prefers a persistent instance they can SSM into for debugging +- Workloads that run sequentially or with limited parallelism (1-5 workers) +- Customer wants to use their own existing EC2 instance (BYO path) -Multi-worker uses bridge networking (no `--net=host`) so each container has its own network namespace. The atx ct CLI to server communication is intra-container (`localhost`) so nothing outside needs to reach the server. SSM `docker exec` works the same as before. Single-worker (`WorkerCount=1`) also uses bridge mode, so there is no behavioral difference visible to the customer. +For one-shot fan-out at scale (>5 parallel jobs), use [remote Batch execution](workload-continuous-modernization-batch-execution.md) instead. -### Changing WorkerCount Requires Redeploy +## Prerequisites -**WorkerCount is fixed at stack-create time.** To change it, the customer runs a stack update which is destructive: the EC2 instance and EBS volume are REPLACED. +### Environment ```bash -# Stack update: REPLACEMENT update because UserData contains ${WorkerCount} -aws cloudformation update-stack --stack-name atx-runner \ - --use-previous-template \ - --parameters \ - ParameterKey=WorkerCount,ParameterValue=$NEW_COUNT \ - ParameterKey=InstanceType,UsePreviousValue=true \ - ParameterKey=VpcId,UsePreviousValue=true \ - ParameterKey=SubnetId,UsePreviousValue=true \ - ParameterKey=VolumeSizeGB,UsePreviousValue=true \ - --capabilities CAPABILITY_NAMED_IAM \ - --region $REGION +export AWS_REGION=us-east-1 # required ``` -**What is preserved:** -- Findings, AIDs, RIDs (live on the atx ct backend, independent of EC2) -- S3 artifacts already uploaded (`atx-ct-output-${ACCOUNT_ID}//...`) -- IAM role, security group (in-place CFN updates) -- Source registrations in the backend -- Secrets in Secrets Manager (managed outside the stack) - -**What is LOST with the EBS volume:** -- Cloned repo caches in `~/.atxct/sources//repos/` -- Build tool caches (Maven `.m2`, Gradle, npm, etc.) -- atx ct internal artifact cache -- Any in-flight wrapper scripts (analyses CONTINUE on the backend, but the EC2-side polling and S3 upload die) +### Permission Model -**Before redeploying, the agent should:** +Two roles are needed: -1. **Check for in-flight work** across all workers: - ```bash - for i in $(seq 1 $WORKER_COUNT); do - if [ "$WORKER_COUNT" -eq 1 ]; then CONT="atx-ct"; else CONT="atx-ct-${i}"; fi - ssm_run "sudo docker exec $CONT atx ct analysis list --json | jq '.items[] | select(.status == \"running\" or .status == \"pending\") | .id'" 2>/dev/null - ssm_run "sudo docker exec $CONT atx ct remediation list --json | jq '.items[] | select(.status == \"running\" or .status == \"pending\") | .id'" 2>/dev/null - done - ``` +- **Admin** — for `provision`, `update`, `teardown`, `credentials`, `network create` +- **Executor** — for `analysis`, `remediation`, `status`, `cancel`, `detect`, `network discover` -2. **Warn the customer**: in-flight analyses on the backend will COMPLETE, but the EC2-side wrapper (polling + S3 upload) will be killed. Customer can re-trigger artifact upload after redeploy via (substitute `atx-ct` for `atx-ct-1` if WorkerCount=1): - ```bash - ssm_run "sudo docker exec atx-ct-1 /app/upload-ct-artifacts.sh atx-ct-output-${ACCOUNT_ID}" - ``` +Executor policy: [AWSTransformInfrastructureExecutorAccessEC2](https://code.amazon.com/packages/ATXControlTowerPolicies/blobs/mainline/--/managed-policies/draft/AWSTransformInfrastructureExecutorAccessEC2.json) + `AWSTransformCustomFullAccess` managed policy. -3. **Recommend Batch path** if the customer's workload is HIGHLY VARIABLE in parallelism. EC2 multi-worker is best when WorkerCount is set once and rarely changed. For elastic demand (e.g., "1 baseline, occasional spike to 8"), Batch's Fargate scaling is cheaper and avoids the redeploy. +### Source Registration -## Fan-out: Run Analysis on Each Repo in Parallel - -When the customer says "run analysis on each repo in source X in parallel" (one AID per repo, all running concurrently), the agent fans out N analyses across the available workers. The pattern below handles three cases automatically: (1) fresh stack provisioning, (2) running on an existing stack with enough workers, and (3) running on an existing stack where REPO_COUNT exceeds WORKER_COUNT (chunked round-robin distribution). +Before submitting remote jobs, the customer must have a registered source with credentials stored: ```bash -# ────────────────────────────────────────────────────────────────────────── -# Step 1: Discover how many repos are in the source -# ────────────────────────────────────────────────────────────────────────── -# `discovery scan` outputs human-readable text (no --json flag in current CLI). -# Format of repo lines: " / " -# We extract the basename (after the last "/") because the --repo flag in Step 5 -# expects "::" format (without the org prefix). -REPOS=$(atx ct discovery scan --source "$LOGICAL_SOURCE_NAME" 2>&1 \ - | awk 'NF >= 2 && $1 ~ /\// { sub(/.*\//, "", $1); print $1 }') -REPO_COUNT=$(echo "$REPOS" | wc -l | tr -d ' ') -[ -z "$REPOS" ] && REPO_COUNT=0 # echo "" | wc -l returns 1, guard against empty -echo "Source $LOGICAL_SOURCE_NAME has $REPO_COUNT repos" - -if [ "$REPO_COUNT" -eq 0 ]; then - echo "ERROR: No repos found in source $LOGICAL_SOURCE_NAME. Verify the source is registered and discovery scan succeeded." - exit 1 -fi - -# ────────────────────────────────────────────────────────────────────────── -# Step 2: Determine WORKER_COUNT (read from existing stack, existing instance, or pick for new) -# ────────────────────────────────────────────────────────────────────────── -if aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region $REGION >/dev/null 2>&1; then - WORKER_COUNT=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region $REGION \ - --query 'Stacks[0].Parameters[?ParameterKey==`WorkerCount`].ParameterValue' --output text 2>/dev/null) - WORKER_COUNT=$(echo "$WORKER_COUNT" | xargs) # strip whitespace defensively - [ -z "$WORKER_COUNT" ] || [ "$WORKER_COUNT" = "None" ] && WORKER_COUNT=1 - echo "Using existing stack '$STACK_NAME' (WorkerCount=$WORKER_COUNT)" -elif [ -n "$INSTANCE_ID" ]; then - # Existing instance (no CFN stack). WORKER_COUNT was set in Step C.1 based on - # customer's choice and the instance's vCPU/RAM capacity. Default to 1 if not set. - WORKER_COUNT="${WORKER_COUNT:-1}" - echo "Using existing instance '$INSTANCE_ID' (WorkerCount=$WORKER_COUNT)" -else - # No stack yet, no existing instance: pick WORKER_COUNT to match REPO_COUNT, capped at 5 - WORKER_COUNT=$(( REPO_COUNT > 5 ? 5 : REPO_COUNT )) - echo "Will provision new stack with WorkerCount=$WORKER_COUNT" - # ... agent runs the "Create New Instance" flow with this WORKER_COUNT ... -fi - -# ────────────────────────────────────────────────────────────────────────── -# Step 3: Decide strategy based on REPO_COUNT vs WORKER_COUNT -# ────────────────────────────────────────────────────────────────────────── -REPOS_PER_WORKER=$(( (REPO_COUNT + WORKER_COUNT - 1) / WORKER_COUNT )) # ceiling - -if [ "$REPO_COUNT" -le "$WORKER_COUNT" ]; then - echo "Strategy: 1:1 fan-out ($REPO_COUNT repos across $WORKER_COUNT workers, $((WORKER_COUNT - REPO_COUNT)) idle)" -elif [ "$REPO_COUNT" -le $((WORKER_COUNT * 2)) ]; then - echo "Strategy: chunked (slight overflow, no infra change needed)" - echo " $REPO_COUNT repos across $WORKER_COUNT workers, ~${REPOS_PER_WORKER} repos per worker" - echo " Alternative: ASK customer if they prefer the Batch path." -else - echo "WARNING: $REPO_COUNT repos significantly exceeds $WORKER_COUNT workers." - echo " Strongly recommend the Batch path (handles up to 64 concurrent Fargate tasks)." - # Agent should pause here and ask the customer to choose chunked vs Batch -fi - -# ────────────────────────────────────────────────────────────────────────── -# Step 4: Round-robin distribution of repos across workers -# ────────────────────────────────────────────────────────────────────────── -# Round-robin (NOT chunked-by-REPOS_PER_WORKER) ensures even utilization. -# Example for 7 repos / 5 workers: -# workers 1-2 get 2 repos each (run sequentially within the worker) -# workers 3-5 get 1 repo each (then idle) -declare -A WORKER_REPOS -i=0 -for REPO in $REPOS; do - WORKER_NUM=$(( (i % WORKER_COUNT) + 1 )) - WORKER_REPOS[$WORKER_NUM]+="$REPO " - i=$((i + 1)) -done - -# ────────────────────────────────────────────────────────────────────────── -# Step 4.5: For local provider, pre-sync the repos bundle to ALL workers -# ────────────────────────────────────────────────────────────────────────── -# Local provider analyses require the repos bundle to be present in each -# worker's filesystem (containers are filesystem-isolated). github/gitlab/ -# bitbucket fetch repos via API at job time, so no pre-sync needed. -# -# IMPORTANT: this sync is IDEMPOTENT but SOURCE-AWARE. We track which source's -# repos are present via /home/atxuser/repos/.atx_source_marker: -# - If marker matches the current source AND repo dirs are present: skip sync -# (preserves atx ct's result-staging branches from prior analyses on this source) -# - If marker is missing, mismatched, or no repo dirs exist: wipe and re-sync -# (prevents source A's repos from contaminating source B's analysis) -# Counting uses `find -type d` so stray files (e.g., .DS_Store, lock files) don't -# falsely satisfy the "has repos" check. -if [ "$PROVIDER" = "local" ]; then - echo "Checking local-provider bundle state on all $WORKER_COUNT workers..." - ssm_run "for c in \$(sudo docker ps --filter name=atx-ct --format '{{.Names}}'); do \ - CURRENT_SOURCE=\$(sudo docker exec \$c cat /home/atxuser/repos/.atx_source_marker 2>/dev/null || echo ''); \ - REPO_COUNT=\$(sudo docker exec \$c bash -c 'find /home/atxuser/repos -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l' || echo 0); \ - if [ \"\$CURRENT_SOURCE\" = \"${LOGICAL_SOURCE_NAME}\" ] && [ \"\$REPO_COUNT\" -gt 0 ]; then \ - echo \" \$c: has ${LOGICAL_SOURCE_NAME} repos (\$REPO_COUNT dir(s)), skipping sync to preserve atx ct state\"; \ - else \ - echo \" \$c: syncing ${LOGICAL_SOURCE_NAME} bundle from S3 (was: \${CURRENT_SOURCE:-empty})\"; \ - sudo docker exec \$c bash -c 'rm -rf /home/atxuser/repos && mkdir -p /home/atxuser/repos /tmp/zips && \ - aws s3 sync s3://atx-source-code-${ACCOUNT_ID}/repos/ /tmp/zips/ && \ - for zip in /tmp/zips/*.zip; do unzip -q -o \"\\\$zip\" -d /home/atxuser/repos/; done && \ - echo ${LOGICAL_SOURCE_NAME} > /home/atxuser/repos/.atx_source_marker'; \ - fi; \ - done" -fi - -# ────────────────────────────────────────────────────────────────────────── -# Step 5: Submit ALL workers via a SINGLE SSM command (each handles its assigned repos serially) -# ────────────────────────────────────────────────────────────────────────── -# Why a single SSM command + double-fork instead of N per-worker submissions: -# SSM's process tracking holds the command slot until all descendants exit -# (the cgroup is empty). Per-worker SSM commands would each stay InProgress -# until each wrapper completes, saturating the SSM agent's worker pool -# (CommandWorkersLimit default 5) and blocking subsequent status-check calls. -# The `( ( bash X & ) & )` double-fork orphans the wrapper to init (PID 1), -# letting the SSM command return Success immediately while wrappers run. -TS=$(date +%s) -MASTER="#!/bin/bash -" -for WORKER_NUM in $(seq 1 $WORKER_COUNT); do - REPOS_FOR_THIS_WORKER="${WORKER_REPOS[$WORKER_NUM]}" - [ -z "$REPOS_FOR_THIS_WORKER" ] && continue # skip workers with no assignment - - # Resolve container name (legacy "atx-ct" if WorkerCount=1, else "atx-ct-N") - if [ "$WORKER_COUNT" -eq 1 ]; then - CONT="atx-ct" - else - CONT="atx-ct-${WORKER_NUM}" - fi - - # Build provider-specific TOKEN_PRELUDE (mirrors build_command_analysis()). - # github/gitlab: fetch token from Secrets Manager into ~/.atxct/sources//. - # bitbucket: fetch token + inject config.json with email/username (Cloud) or base_url (DC). - # local: no prelude (repos already pre-synced in Step 4.5). - TOKEN_PRELUDE="" - if [ "$PROVIDER" = "github" ] || [ "$PROVIDER" = "gitlab" ]; then - SECRET_ID="atx/${PROVIDER}-token" - TOKEN_FILE="${PROVIDER}_token" - TOKEN_PRELUDE="sudo docker exec ${CONT} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id ${SECRET_ID} --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE} && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE}'" - elif [ "$PROVIDER" = "bitbucket" ]; then - if [ -n "${BITBUCKET_BASE_URL}" ]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - CONFIG_B64=$(echo "${config_json}" | base64 -w 0) - TOKEN_PRELUDE="sudo docker exec ${CONT} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token'" - fi - - JOB_ID="fan-w${WORKER_NUM}-${TS}" - - # Per-worker script: loops through its assigned repos and runs analysis on each. - # Continues on failure (so other repos still get analyzed). Each repo emits its own AID. - SCRIPT=$(cat <> \$LOG - -${TOKEN_PRELUDE} - -for REPO in ${REPOS_FOR_THIS_WORKER}; do - echo "--- \$(date) starting analysis on \$REPO (worker ${WORKER_NUM}) ---" >> \$LOG - sudo docker exec ${CONT} atx ct analysis run \\ - --type ${ANALYSIS_TYPE} ${EXTRA_FLAGS} --source ${LOGICAL_SOURCE_NAME} --repo "${LOGICAL_SOURCE_NAME}::\$REPO" --wait --telemetry "agent=${AGENT},executionMode=ec2" >> \$LOG 2>&1 - RC=\$? - AID=\$(tail -50 \$LOG | grep -oE '01[A-Z0-9]+' | tail -1) - echo "\$REPO -> \$AID (rc=\$RC)" >> \$AIDS_FILE - if [ "${UPLOAD_ARTIFACTS:-true}" = "true" ] && [ "${ANALYSIS_TYPE}" != "tech-debt-quick" ] && [ -n "\$AID" ] && [ \$RC -eq 0 ]; then - sudo docker exec ${CONT} /app/upload-ct-artifacts.sh \$AID atx-ct-output-${ACCOUNT_ID} >> \$LOG 2>&1 - fi -done - -echo "=== \$(date) [DONE] worker ${WORKER_NUM} ===" >> \$LOG -EOF -) - - B64=$(echo "$SCRIPT" | base64 | tr -d '\n') - # Append decode + double-fork stanza for this worker to the master launcher - MASTER+="echo ${B64} | base64 -d > /tmp/${JOB_ID}.sh && chmod +x /tmp/${JOB_ID}.sh && ( ( bash /tmp/${JOB_ID}.sh > /tmp/${JOB_ID}.stdout 2>&1 < /dev/null & ) & ) && echo Launched_w${WORKER_NUM} -" -done -MASTER+="echo ALL_LAUNCHED" - -# Submit the master launcher as a SINGLE SSM command. Wrappers run as orphaned -# background processes; this command itself exits immediately. -MASTER_B64=$(echo "$MASTER" | base64 | tr -d '\n') -SUBMIT_ID=$(ssm_submit "echo ${MASTER_B64} | base64 -d > /tmp/master-${TS}.sh && chmod +x /tmp/master-${TS}.sh && bash /tmp/master-${TS}.sh") -echo "Launched all $WORKER_COUNT workers via single SSM command (id: $SUBMIT_ID)" - -# ────────────────────────────────────────────────────────────────────────── -# Step 6: Poll status across all workers (agent helper) -# ────────────────────────────────────────────────────────────────────────── -# Each worker writes AIDs to /tmp/atxct-fan-w${N}-${TS}.aids as repos finish. -# To check overall progress: -# for w in $(seq 1 $WORKER_COUNT); do -# ssm_run "cat /tmp/atxct-fan-w${w}-*.aids 2>/dev/null" -# done -# When all workers' AIDS files match their assigned repo count, fan-out is complete. -``` - -The agent should **report all N AIDs back** to the customer once each worker emits them (typically as each repo finishes, staggered as workers complete their assigned repos in sequence). +# GitHub / GitLab +atx ct source add --name --provider github|gitlab --org --token -## Fan-out: Run Remediation on Each Repo in Parallel +# Bitbucket Cloud (--username and --email optional but needed for clone/push and API auth) +atx ct source add --name --provider bitbucket --org --token \ + --username --email -Same round-robin distribution as the analysis fan-out, but each worker runs `atx ct remediation create` (not `atx ct analysis run`). The provider differences match `build_command_remediation()`: github/gitlab/bitbucket push result branches automatically (no S3 upload); local provider requires explicit `--local` flag and S3 upload of the remediated code. +# Self-hosted GitLab or Bitbucket Data Center (add --url) +atx ct source add --name --provider gitlab|bitbucket --org --token \ + --url https://gitlab.mycompany.com -Use this when the customer says "remediate each repo in source X in parallel" (one RID per repo). Steps 1-4 are identical to the analysis fan-out (discover repos, determine WORKER_COUNT, decide strategy, round-robin distribute). Step 4.5 (local-provider repo sync) also applies and is source-aware: if a prior analysis on the same source already populated `/home/atxuser/repos/` on each worker, Step 4.5 skips the re-sync to preserve atx ct's branch state from that operation. If the customer switched to a different source, Step 4.5 wipes and re-syncs to prevent contamination. The differences are in Step 5 below. +# Local filesystem +atx ct source add --name --provider local --path /path/to/repos -**`--transformation-name` vs `--ids` mode**: the fan-out below runs Mode 3 from [continuous-modernization-remediation](workload-continuous-modernization-remediation.md), which uses `--transformation-name` per repo with `--repo "::"`. For finding-driven remediation (Modes 1 or 2), distribute finding IDs across workers in chunks instead of repos: each worker calls `atx ct remediation create --ids ` WITHOUT `--repo` (atx ct rejects `--repo` with `--ids` because repos are derived from findings). Extract auto-remediable IDs from a prior analysis with: `atx ct findings list --analysis-id $AID --json | jq -r '.[] | select(.auto_remediable == true) | .id'`. +# Store credential for remote containers (required for github/gitlab/bitbucket) +atx ct remote credentials --source --token --ack -```bash -# Steps 1-4.5: same as analysis fan-out (REPOS, REPO_COUNT, WORKER_COUNT, -# WORKER_REPOS round-robin assignment, local-provider repo sync if applicable) - -# Build CREATE_ARGS based on remediation mode (mirrors build_command_remediation) -if [ -n "$FINDING_IDS" ]; then - CREATE_ARGS_BASE="--ids ${FINDING_IDS}" - [ -n "${TRANSFORMATION_NAME}" ] && CREATE_ARGS_BASE="${CREATE_ARGS_BASE} --transformation-name ${TRANSFORMATION_NAME}" -else - CREATE_ARGS_BASE="--transformation-name ${TRANSFORMATION_NAME}" -fi -[ -n "${CONFIGURATION}" ] && CREATE_ARGS_BASE="${CREATE_ARGS_BASE} -g \"${CONFIGURATION}\"" - -# Provider-aware --local flag and upload behavior -LOCAL_FLAG="" -UPLOAD_REMED_LINE='echo "[skip upload: github/gitlab/bitbucket pushes results to source repo]"' -if [ "$PROVIDER" = "local" ]; then - LOCAL_FLAG="--local" - # UPLOAD_REMED_LINE set per-worker below (uses ${CONT}) -fi - -# ────────────────────────────────────────────────────────────────────────── -# Step 5: Submit ALL remediations via a SINGLE SSM command (each worker handles assigned repos serially) -# ────────────────────────────────────────────────────────────────────────── -# Same single-SSM + double-fork rationale as the analysis fan-out: one command -# submission orphans all wrappers to init so SSM returns immediately and status -# checks aren't queued behind the wrappers. -TS=$(date +%s) -MASTER="#!/bin/bash -" -for WORKER_NUM in $(seq 1 $WORKER_COUNT); do - REPOS_FOR_THIS_WORKER="${WORKER_REPOS[$WORKER_NUM]}" - [ -z "$REPOS_FOR_THIS_WORKER" ] && continue - - # Resolve container name (same as analysis fan-out) - if [ "$WORKER_COUNT" -eq 1 ]; then - CONT="atx-ct" - else - CONT="atx-ct-${WORKER_NUM}" - fi - - # Build TOKEN_PRELUDE (same as analysis fan-out's logic for github/gitlab/bitbucket) - TOKEN_PRELUDE="" - if [ "$PROVIDER" = "github" ] || [ "$PROVIDER" = "gitlab" ]; then - SECRET_ID="atx/${PROVIDER}-token" - TOKEN_FILE="${PROVIDER}_token" - TOKEN_PRELUDE="sudo docker exec ${CONT} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id ${SECRET_ID} --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE} && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE}'" - elif [ "$PROVIDER" = "bitbucket" ]; then - if [ -n "${BITBUCKET_BASE_URL}" ]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - CONFIG_B64=$(echo "${config_json}" | base64 -w 0) - TOKEN_PRELUDE="sudo docker exec ${CONT} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token'" - fi - - # Per-worker upload line (only meaningful for local provider) - WORKER_UPLOAD_LINE='echo "[skip upload: github/gitlab/bitbucket pushes to source repo]"' - if [ "$PROVIDER" = "local" ]; then - WORKER_UPLOAD_LINE="sudo docker exec ${CONT} /app/upload-ct-artifacts.sh \\\$RID atx-ct-output-${ACCOUNT_ID}" - fi - - JOB_ID="fan-rem-w${WORKER_NUM}-${TS}" - - # Per-worker remediation script: loops through assigned repos. - # Continues on failure. Polls each remediation until terminal (no --wait support - # for `remediation create`). Uploads to S3 only for local provider. - SCRIPT=$(cat <> \$LOG - -${TOKEN_PRELUDE} - -for REPO in ${REPOS_FOR_THIS_WORKER}; do - echo "--- \$(date) starting remediation on \$REPO (worker ${WORKER_NUM}) ---" >> \$LOG - sudo docker exec ${CONT} atx ct remediation create \\ - ${CREATE_ARGS_BASE} ${LOCAL_FLAG} --source ${LOGICAL_SOURCE_NAME} --repo "${LOGICAL_SOURCE_NAME}::\$REPO" --telemetry "agent=${AGENT},executionMode=ec2" >> \$LOG 2>&1 - RC=\$? - RID=\$(tail -50 \$LOG | grep -oE '01[A-Z0-9]+' | tail -1) - echo "\$REPO -> \$RID (rc=\$RC)" >> \$RIDS_FILE - - # Poll until terminal (atx ct remediation create has no --wait flag) - if [ -n "\$RID" ] && [ \$RC -eq 0 ]; then - while true; do - STATUS=\$(sudo docker exec ${CONT} atx ct remediation status --id \$RID --json 2>/dev/null | jq -r .status 2>/dev/null) - case "\$STATUS" in - complete|completed|failed|cancelled) break ;; - esac - sleep 30 - done - echo "\$REPO -> \$RID terminal: \$STATUS" >> \$LOG - - # Upload artifacts (only meaningful for local; github/gitlab/bitbucket skip) - if [ "\$STATUS" = "complete" ] || [ "\$STATUS" = "completed" ]; then - ${WORKER_UPLOAD_LINE} >> \$LOG 2>&1 - fi - fi -done - -echo "=== \$(date) [DONE] worker ${WORKER_NUM} ===" >> \$LOG -EOF -) - - B64=$(echo "$SCRIPT" | base64 | tr -d '\n') - # Append decode + double-fork stanza for this worker to the master launcher - MASTER+="echo ${B64} | base64 -d > /tmp/${JOB_ID}.sh && chmod +x /tmp/${JOB_ID}.sh && ( ( bash /tmp/${JOB_ID}.sh > /tmp/${JOB_ID}.stdout 2>&1 < /dev/null & ) & ) && echo Launched_w${WORKER_NUM} -" -done -MASTER+="echo ALL_LAUNCHED" - -# Submit the master launcher as a SINGLE SSM command. Wrappers run as orphaned -# background processes; this command itself exits immediately. -MASTER_B64=$(echo "$MASTER" | base64 | tr -d '\n') -SUBMIT_ID=$(ssm_submit "echo ${MASTER_B64} | base64 -d > /tmp/master-${TS}.sh && chmod +x /tmp/master-${TS}.sh && bash /tmp/master-${TS}.sh") -echo "Launched all $WORKER_COUNT remediation workers via single SSM command (id: $SUBMIT_ID)" - -# Step 6: Poll status across workers (same pattern as analysis fan-out) -# Each worker writes RIDs to /tmp/atxct-fan-rem-w${N}-${TS}.rids as repos finish. +# Discover repos +atx ct discovery scan --source ``` -The agent should **report all N RIDs back** to the customer. For github/gitlab/bitbucket, customers will see N PRs/MRs created in their source provider. For local, customers can download the remediated code from `s3://atx-ct-output-${ACCOUNT_ID}//` after each remediation completes. - -## Setup Paths - -Step 0 below detects which one of three states applies and routes accordingly: - -| Path | Trigger | Admin handoff? | -|------|---------|----------------| -| **Operate existing CFN stack** | `describe-stacks atx-runner` returns `CREATE_COMPLETE` | No -- executor creds suffice | -| **Use existing non-CFN instance** | No stack, customer has their own EC2 instance | Only if the instance lacks the `atx-remote-infra=true` tag | -| **Provision a new CFN stack** | No stack, no existing instance | Yes -- admin runs `aws cloudformation create-stack` once | - -## Two-Persona Permission Model - -Provisioning and operating the runner are split across two distinct IAM personas. The skill respects this split: the agent NEVER asks the executor to run a privileged provisioning command, and NEVER asks the admin to run a routine job submission. - -| Persona | Managed policy | Owns | When used | -|---|---|---|---| -| **Admin** | (any identity with the permissions listed below -- typically full admin / `AdministratorAccess` or an org-scoped admin role) | All resource-lifecycle mutations: `iam:Create/Put/Delete*` on `atx-transform-*`, `cloudformation:CreateStack`/`DeleteStack`, `ec2:RunInstances`/`CreateSecurityGroup`/`CreateKeyPair`/`TerminateInstances`/`DeleteSecurityGroup`/`DeleteKeyPair`, `s3:CreateBucket`+lifecycle, `scheduler:CreateScheduleGroup`, `ec2:AssociateIamInstanceProfile`/`ModifyInstanceMetadataOptions` | Initial setup (buckets + stack) and teardown (`delete-stack`) | -| **Executor** | (a least-privilege role scoped to the actions listed below) | Read-only CFN/EC2/SSM, SSM SendCommand on tagged, S3 data plane on `atx-*` buckets, `secretsmanager:GetSecretValue`/`DescribeSecret` on `atx/*` (read only), `ec2:Start/StopInstances` on tagged (power state), KMS-via-alias, `scheduler:CreateSchedule`/`DeleteSchedule`/`GetSchedule`/`UpdateSchedule`/`ListSchedules` (scoped to `atx-control-tower` group), `iam:PassRole` to `atx-transform-role*` (EC2) and `AtxSchedulerInvocationRole` (scheduler) only | Every analysis / remediation submission, status check, artifact fetch, schedule pause/resume | - -The executor policy has **zero IAM mutations and zero resource-lifecycle creations**. Privilege-escalation surface is bounded to the admin handoff at stack create/delete, not the day-to-day developer flow. - -### Skill flow at every entry - -``` -Entry (executor creds -- agent always assumes least privilege) - │ - └─ Step 5: DETECT (read-only -- describe-stacks) - ├─ NOT_DEPLOYED ─────► PROVISION - │ Agent prints template + create-stack command - │ with admin caveat. STOPS. User runs it - │ with admin creds outside the agent. User - │ re-enters the flow when done. - │ - └─ EXISTS & healthy ──► OPERATE - Steps 6–10 run with executor creds only. -``` - -Detection is read-only and always safe. Only `create-stack` and `delete-stack` need admin, and the agent hands those to the user. - -## Step 0: Detect Existing Infrastructure - -ALWAYS run this first, on every entry to the flow. Uses only read-only calls (`cloudformation:DescribeStacks`, `ec2:DescribeInstances`) -- both in the executor policy, so it never fails on permissions. The result decides which path the rest of the skill takes. - -```bash -STACK_NAME="${STACK_NAME:-atx-runner}" -REGION="${REGION:-$(aws configure get region || echo us-east-1)}" -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) - -STACK_STATUS=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region $REGION \ - --query 'Stacks[0].StackStatus' --output text 2>/dev/null) - -case "$STACK_STATUS" in - CREATE_COMPLETE|UPDATE_COMPLETE) - INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region $REGION \ - --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' --output text) - echo "OPERATE (Case A) -- CFN stack '$STACK_NAME' exists. Instance: $INSTANCE_ID" - # Skip to Step 6 (Verify the Container is Running). Executor creds suffice. - ;; - CREATE_IN_PROGRESS|UPDATE_IN_PROGRESS) - echo "WAIT -- stack is mid-transition ($STACK_STATUS). Re-run when the admin's" - echo " deploy finishes:" - echo " aws cloudformation wait stack-create-complete \\" - echo " --stack-name $STACK_NAME --region $REGION" - exit 0 - ;; - DELETE_IN_PROGRESS) - echo "WAIT -- stack is being deleted ($STACK_STATUS). Wait for deletion to finish," - echo " then re-run this flow to provision a new stack:" - echo " aws cloudformation wait stack-delete-complete \\" - echo " --stack-name $STACK_NAME --region $REGION" - exit 0 - ;; - CREATE_FAILED|*ROLLBACK*|DELETE_FAILED) - echo "BLOCKED -- stack is in $STACK_STATUS. The admin must clean it up first:" - echo " aws cloudformation describe-stack-events --stack-name $STACK_NAME --region $REGION" - echo " aws cloudformation delete-stack --stack-name $STACK_NAME --region $REGION # admin" - exit 1 - ;; - "") - # No CFN stack -- agent asks which of the two remaining paths applies. - ;; -esac -``` - -If `STACK_STATUS` was empty, the agent MUST ask the customer: +## Multi-Worker Support -> "I don't see an `atx-runner` CFN stack in `${REGION}`. Which path applies? -> 1. **Reuse an existing EC2 instance** I already have (launched outside CFN -- e.g., my org's standard EC2, or a dev box I already use) -> 2. **Create a new CFN-managed runner from scratch** (requires admin to run a one-time deploy command outside the agent)" +EC2 supports 1-5 parallel workers (containers) on a single instance. Jobs are distributed round-robin across workers. -| Customer answer | Path | Admin needed? | +| Customer intent | Workers | Instance type | |---|---|---| -| 1. Existing instance | OPERATE -- see [Use Existing Instance (no CFN)](#use-existing-instance-no-cfn), then Step 6 | Only if the instance lacks the `atx-remote-infra=true` tag or the `atx-transform-access` inline policy on its role (Step C.0 detects and emits one combined handoff) | -| 2. Create new CFN stack | PROVISION -- Steps 1–5 below | **Yes**, for one command at the end of Step 5 | - -Steps 1–5 below only apply to path 2. - -## Provision Lifecycle (Steps 1–5, fresh CFN stack) - -These steps run **only when the customer chose path 2 in Step 0**. Steps 1–4 stay on executor creds -- they collect inputs and stage credentials. Step 5 then prints a self-contained `aws cloudformation create-stack` command for the customer's **admin** to run; the agent does NOT execute it. - -### Step 1: Verify Source and Enumerate Repos - -Confirm a source is registered locally (see [continuous-modernization-source.md](workload-continuous-modernization-source.md)) and run discovery (see [continuous-modernization-discovery.md](workload-continuous-modernization-discovery.md)) to get the list of repos. This determines instance size (Step 2) and gives the customer visibility into what will be analyzed. - -```bash -atx ct source list -``` - -Show the list to the customer and ask: -1. **Which source to analyze?** (pick from the list above) -2. **Source type:** GitHub, GitLab, Bitbucket, or local -3. **Analysis type:** `tech-debt-comprehensive`, `tech-debt-quick`, `security`, `agentic-readiness`, `modernization-readiness`, or `custom` - -If the list is empty, the customer wants to register a new source, or needs to update the token on an existing source, use the [continuous-modernization-source](workload-continuous-modernization-source.md) skill (`source add` for new, `source update` for existing), then return here. +| Single analysis / default | 1 | m5.2xlarge | +| 2 analysis types in parallel | 2 | m5.4xlarge | +| 3-4 parallel jobs | 3-4 | m5.4xlarge | +| 5 parallel jobs (max) | 5 | m5.8xlarge | +| 6+ parallel jobs | Use Batch instead | — | -```bash -LOGICAL_SOURCE_NAME="" - -atx ct discovery scan --source "$LOGICAL_SOURCE_NAME" - -mapfile -t REPOS < <(atx ct repository list --source "$LOGICAL_SOURCE_NAME" --json | jq -r '.items[].full_name') -REPO_COUNT=${#REPOS[@]} - -echo "Source ${LOGICAL_SOURCE_NAME} has ${REPO_COUNT} repos" -``` +WorkerCount is fixed at stack-create time. Changing it requires a destructive redeploy (`teardown` + `provision`). -If the customer wants only a subset, set `REPO_FILTER="--repo ::"` for use in Step 8. The `--repo` flag accepts exactly ONE repo per invocation -- to analyze multiple repos in parallel, use the fan-out pattern (one submission per repo across workers). Empty = analyze the whole source. +Each worker is memory-capped at `(instance RAM - 4 GB) / WorkerCount`. The CLI auto-sizes instance type if not specified. -### Step 2: Determine Instance Size +## Workflow -Default: `m5.2xlarge` (8 vCPU, 32 GB RAM) -- handles repo counts of any size for sequential execution. If a repo is unusually large (>5 GB working tree, e.g., a monorepo), bump up by one tier for RAM headroom. +### 1. Detect Infrastructure ```bash -INSTANCE_TYPE="m5.2xlarge" -``` +# List all EC2 stacks in the account (mode prefix) +atx ct remote detect --mode ec2 -### Step 3: Confirm Account and Plan +# Look up one stack directly +atx ct remote detect --mode ec2 --stack-name -```bash -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) -REGION=$(aws configure get region || echo "us-east-1") +# Discover the stack by its resource tags +atx ct remote detect --mode ec2 --tags env=prod,team=platform ``` -Tell the user: - -> "I'll set up CT analysis on a new EC2 instance in account `${ACCOUNT_ID}`, region `${REGION}`. This includes: -> - IAM role + instance profile (`atx-transform-role`, `atx-transform-profile`) with SSM Managed Instance Core for shell access -> - Security group (no inbound ports -- SSM SendCommand handles all access via outbound HTTPS) -> - `${INSTANCE_TYPE}` EC2 instance with 100GB volume -> - S3 source bucket (only if local source) and atx-ct-output bucket (always) -> -> Continue?" +`--stack-name` and `--tags` are mutually exclusive. Tag discovery requires the +`tag:GetResources` permission (covered by the Executor policy). Omit both to +list every EC2 stack with the `atx-runner` prefix. -Wait for explicit confirmation. +If deployed → skip to step 3 (Submit Analysis). +If not deployed → proceed to step 2 (Provision). -### Step 4: Prep Credentials +### 2. Provision -Give the user the relevant command below to run in their own terminal -- do not ask them to paste the token into this chat. +Requires Admin credentials. -Tokens are stored in AWS Secrets Manager and fetched by the container at job submission time (the EC2 instance role has `secretsmanager:GetSecretValue` for `atx/*`). This is the same pattern as the batch skill -- store once, use for any number of analyses without re-staging files. - -**GitHub HTTPS -- store the PAT:** +**Discover network resources:** ```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/github-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/github-token" \ - --secret-string "$TOKEN"; }; unset TOKEN +atx ct remote network discover --region us-east-1 ``` -**GitLab HTTPS -- store the PAT:** +**If no suitable VPC exists:** ```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/gitlab-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/gitlab-token" \ - --secret-string "$TOKEN"; }; unset TOKEN +atx ct remote network create --region us-east-1 --ack ``` -**Bitbucket -- store the API token** (the container fetches it from Secrets Manager at job start). Email and username are injected into the container command directly (not secrets -- they're non-sensitive identifiers): +**Provision the EC2 stack:** ```bash -read -s TOKEN && { aws secretsmanager create-secret --name "atx/bitbucket-token" \ - --secret-string "$TOKEN" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/bitbucket-token" \ - --secret-string "$TOKEN"; }; unset TOKEN +atx ct remote provision --mode ec2 \ + --vpc \ + --subnets , \ + --execute --ack ``` -**SSH -- store the private key:** +Required flags: +- `--mode ec2` — required +- `--vpc ` — required +- `--subnets ` — required (comma-separated, private subnets only) -```bash -aws secretsmanager create-secret --name "atx/ssh-key" \ - --secret-string "$(cat )" 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "atx/ssh-key" \ - --secret-string "$(cat )" -``` - -**Private package registries** (if the analysis builds the project): see [custom-remote-execution#private-package-registries](custom-remote-execution.md#private-package-registries) for the `atx/credentials` JSON pattern. - -A single token can be used for any number of sources of the same provider (e.g., one PAT for all your GitHub orgs). The build_command_*() in Step 8 fetches the token and writes it to the source-specific path the CT server expects. - -### Step 5: Provision Infrastructure via CloudFormation - -Provisioning is done through a single CloudFormation stack -- atomic deploy/rollback, single command teardown, full visibility in the customer's CloudFormation console. The stack provisions: +Optional flags: +- `--workers <1-5>` — parallel container count (default: 1) +- `--instance-type ` — override auto-sizing (m5.large to m5.12xlarge) +- `--volume-size ` — EBS volume size (default: auto-sized from workers) +- `--image-uri ` — container image override (default: public AWS Transform — continuous modernization image) +- `--stack-name ` — custom name (must start with `atx-runner`) +- `--suffix ` — appended to default stack name +- `--securityGroup ` — optional for EC2 (auto-created if omitted) +- `--tags ` — resource tags applied to the stack. These same tags can later target the stack via `--tags` on `detect`/`analysis`/`remediation` (instead of `--stack-name`) -- IAM role + instance profile (with transform-custom + S3 + KMS + Secrets Manager + securityagent permissions and `AmazonSSMManagedInstanceCore` attached for SSM) -- Security group (no inbound; outbound default allow) -- EC2 instance (Amazon Linux 2023, 100 GB gp3 volume) -- UserData that installs Docker, pulls the atx-ct image, and starts the container +Without `--execute`, prints the CFN template (dry-run preview). -S3 buckets (`atx-source-code-${ACCOUNT_ID}`, `atx-ct-output-${ACCOUNT_ID}`) are managed OUTSIDE the stack -- they hold persistent customer data, must survive stack delete-and-recreate, and are shared across multiple stacks if the customer ever runs more than one. +The stack creates: EC2 instance, IAM role + instance profile, security group (no inbound, SSM access), CloudWatch log group, and dashboard. UserData pulls the container image, starts workers, and signals CFN on health check pass. -**Step 5a: Check whether S3 buckets exist (executor: read-only):** +### 3. Submit Analysis -`s3:CreateBucket` and `s3:PutLifecycleConfiguration` are in the **admin** policy, not executor. Bucket creation is bundled into the admin handoff in Step 5d so the admin runs all the privileged setup commands in one shell session. Here, the agent only checks whether the buckets already exist (`head-bucket` is read-only). +Requires Executor credentials. Jobs are distributed round-robin across workers. ```bash -SOURCE_BUCKET_EXISTS=$(aws s3api head-bucket --bucket atx-source-code-${ACCOUNT_ID} 2>/dev/null && echo yes || echo no) -OUTPUT_BUCKET_EXISTS=$(aws s3api head-bucket --bucket atx-ct-output-${ACCOUNT_ID} 2>/dev/null && echo yes || echo no) -echo "atx-source-code-${ACCOUNT_ID}: $SOURCE_BUCKET_EXISTS" -echo "atx-ct-output-${ACCOUNT_ID}: $OUTPUT_BUCKET_EXISTS" +atx ct remote analysis \ + --types rapid-techdebt-analysis \ + --sources \ + --mode ec2 \ + --stack-name \ + --telemetry "agent=kiro,executionMode=ec2" ``` -If both are `yes`, the agent omits the bucket-creation lines from Step 5d's admin handoff (they're idempotent, but cleaner to omit). If either is `no`, the admin handoff includes the `aws s3api create-bucket` + `put-bucket-lifecycle-configuration` calls before the `cloudformation create-stack` line. +Fan-out options: +- `--types type1,type2` — multiple analysis types +- `--sources src1,src2` — multiple sources +- `--repos src::repo1,src::repo2` — specific repos (fully qualified) +- `--labels java,spring` — filter repos by labels (AND semantics) +- `--transformation-name ` — for `--types custom` +- `-g key=value` — configuration for custom transformations -**Step 5b: Cascading list-and-pick -- VPC, then subnet, then security group, then final confirmation.** +Stack targeting (choose one): +- `--stack-name ` — target the stack by name +- `--tags env=prod,team=platform` — discover the stack by its resource tags instead of naming it (alternative to `--stack-name`; same tags set at provision time) +- `--existing-instance ` — BYO path; replaces `--stack-name`/`--tags` (see below) -The skill **never creates a VPC, subnet, or NAT** -- those are customer-owned network resources. The agent's job is to **list what already exists in the account**, let the customer pick each, and run validations on the chosen network before the admin handoff fires. +The CLI runs pre-flight checks (token validation, instance health), dispatches via SSM, and returns a group ID for polling. -The flow is cascading: pick VPC first (so subnet/SG lists can be filtered to that VPC), then subnet, then SG, then a final summary the customer must explicitly confirm before Step 5c proceeds. Customers with self-hosted / internal git hosts can pick a VPC that has VPN / Direct Connect / peering to that host -- same flow, same UX, the customer just picks a different VPC. - -**MANDATORY interaction rules. The agent MUST follow these without exception:** - -- **The agent MUST NOT pre-select** a VPC, subnet, or security group -- not even if "obvious," "functionally equivalent," or "sensible default." Every choice belongs to the customer. -- **The agent MUST present each list and STOP**, waiting for the customer to type their explicit choice. No proceeding to the next step until the customer has answered the current one. -- **After the third selection** (security group), the agent MUST display all four selections (VPC, subnet, SG, AZ) in a final summary and **explicitly ask "proceed with these?"**. The agent MUST NOT advance to Step 5c (write CFN template) until the customer types `yes` or equivalent. -- **If the agent is inclined to skip an ask** ("they said use default VPC, I can pick the subnet myself"): STOP. The customer's "use default VPC" answer is ONLY about the VPC. Subnet and SG remain unanswered until the customer types those choices too. +### 4. Check Status ```bash -EXISTING_SG_ID="${EXISTING_SG_ID:-}" # empty means stack creates a new no-inbound SG - -# ────────────────────────────────────────────────────────────────────────── -# 1. List VPCs in the account+region. Show ID, default flag, Name tag. -# ────────────────────────────────────────────────────────────────────────── -echo "VPCs available in account ${ACCOUNT_ID}, region ${REGION}:" -aws ec2 describe-vpcs --region $REGION \ - --query 'Vpcs[*].{VpcId:VpcId,Default:IsDefault,Cidr:CidrBlock,Name:Tags[?Key==`Name`]|[0].Value}' \ - --output table - -VPC_COUNT=$(aws ec2 describe-vpcs --region $REGION --query 'length(Vpcs)' --output text) -DEFAULT_VPC_COUNT=$(aws ec2 describe-vpcs --region $REGION --filters Name=isDefault,Values=true --query 'length(Vpcs)' --output text) -``` - -**Two account-state cases the agent MUST handle explicitly before continuing:** +# Poll once +atx ct remote status --group -**Case 1 -- `VPC_COUNT=0`: no VPCs at all in this account+region.** - -The skill **never creates VPCs** -- that's an infrastructure decision the customer's network team must make. The agent MUST stop and tell the customer: - -> "There are no VPCs in account `${ACCOUNT_ID}`, region `${REGION}`. -> -> The skill cannot proceed without a VPC, and it does NOT auto-create one -- VPCs are foundational network infrastructure that should be set up by your network or platform team, not by an analysis tool. -> -> Ask whoever owns AWS networking in your org to: -> 1. Create a VPC (or restore the deleted default VPC) in this region -> 2. Add at least one subnet with outbound internet access (NAT gateway, internet gateway, or transit gateway) -> 3. Optionally, prepare a security group with allow-all-egress on TCP 443 (or scope to specific endpoints) -> -> Once that exists, come back to this conversation and I'll re-run the VPC list." - -The agent then STOPS this turn. Don't try to work around it. - -**Case 2 -- `VPC_COUNT≥1` but `DEFAULT_VPC_COUNT=0`: VPCs exist but none are marked default.** - -This is normal in enterprise accounts where the default VPC was deliberately deleted (security baseline, AWS Landing Zone, Control Tower OUs). The customer just needs to pick one of the non-default VPCs. The agent presents the list with the same ASK as the next step -- the absence of a default VPC isn't an error, just slightly different framing: - -> "I don't see a default VPC in this region -- that's normal in enterprise accounts. Here are the VPCs that DO exist; please pick the one your runner should deploy into." - -The list-and-pick flow below handles both Case 2 and the simple-default case. Only Case 1 stops the flow. - -```bash -if [ "$VPC_COUNT" = "0" ]; then - echo "ERROR: No VPCs in account ${ACCOUNT_ID}, region ${REGION}." - echo " The skill does NOT auto-create VPCs. Ask your network team to provision" - echo " a VPC + subnet (with NAT/IGW/TGW egress) before re-running." - exit 1 -fi -``` - -**STOP HERE.** The agent MUST present the list above and ask the customer which VPC to use. The agent MUST NOT proceed to listing subnets until the customer has typed a VPC ID. Suggested phrasing: - -> "Here are the VPCs in your account. Which one should the runner be deployed in? -> If you have a self-hosted git host (GHES, GitLab self-managed, Bitbucket DC), pick the VPC that has VPN / Direct Connect / peering routes to it. -> If you're using public github/gitlab/bitbucket, the default VPC works, but you may prefer a workload VPC for better network isolation. -> Please reply with the VPC ID." - -```bash -read -p "VPC ID: " VPC_ID - -# Verify the customer's choice exists -VPC_EXISTS=$(aws ec2 describe-vpcs --vpc-ids "$VPC_ID" --region $REGION \ - --query 'Vpcs[0].VpcId' --output text 2>/dev/null) -if [ "$VPC_EXISTS" != "$VPC_ID" ]; then - echo "ERROR: VPC $VPC_ID not found in $REGION." - exit 1 -fi - -# ────────────────────────────────────────────────────────────────────────── -# 2. List subnets in the chosen VPC. Show ID, AZ, CIDR, public flag, Name. -# ────────────────────────────────────────────────────────────────────────── -echo "" -echo "Subnets in $VPC_ID:" -aws ec2 describe-subnets --region $REGION --filters "Name=vpc-id,Values=$VPC_ID" \ - --query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone,Cidr:CidrBlock,Public:MapPublicIpOnLaunch,Name:Tags[?Key==`Name`]|[0].Value}' \ - --output table - -SUBNET_COUNT=$(aws ec2 describe-subnets --region $REGION --filters "Name=vpc-id,Values=$VPC_ID" --query 'length(Subnets)' --output text) -if [ "$SUBNET_COUNT" = "0" ]; then - echo "ERROR: VPC $VPC_ID has no subnets. The skill cannot create one." - echo " Customer's network team must add a subnet. Bail out." - exit 1 -fi -``` - -**STOP HERE.** The agent MUST present the list above and ask the customer which subnet to use. The agent MUST NOT pre-pick "the first one" or "any of the AZ-a subnets" -- every subnet is the customer's call. The agent MUST NOT proceed to listing security groups until the customer has typed a subnet ID. Suggested phrasing: - -> "Pick a subnet for the runner. The subnet must have outbound internet access (NAT gateway, internet gateway, or transit gateway) so the runner can pull the atx-ct image from ECR Public, reach the atx ct backend, and talk to S3 / Secrets Manager. -> Public subnets (`Public: True`) auto-assign public IPs -- easiest for image pull, but exposes the instance to the internet. -> Private subnets (`Public: False`) need NAT or TGW egress -- typical for production workloads. -> Please reply with the subnet ID." - -```bash -read -p "Subnet ID: " SUBNET_ID - -# Validation #1: subnet is actually in the chosen VPC. -SUBNET_VPC=$(aws ec2 describe-subnets --subnet-ids "$SUBNET_ID" --region $REGION \ - --query 'Subnets[0].VpcId' --output text 2>/dev/null) -if [ "$SUBNET_VPC" != "$VPC_ID" ]; then - echo "ERROR: subnet $SUBNET_ID is not in VPC $VPC_ID (or doesn't exist in $REGION)." - exit 1 -fi -SUBNET_AZ=$(aws ec2 describe-subnets --subnet-ids "$SUBNET_ID" --region $REGION \ - --query 'Subnets[0].AvailabilityZone' --output text) -echo " ✓ Subnet $SUBNET_ID is in $SUBNET_VPC, AZ $SUBNET_AZ." - -# Validation #2: subnet has a default route (egress exists). -ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --region $REGION \ - --filters "Name=association.subnet-id,Values=$SUBNET_ID" \ - --query 'RouteTables[0].RouteTableId' --output text 2>/dev/null) -if [ "$ROUTE_TABLE_ID" = "None" ] || [ -z "$ROUTE_TABLE_ID" ]; then - # Subnet has no explicit association; it inherits the VPC's main route table. - ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --region $REGION \ - --filters "Name=vpc-id,Values=$VPC_ID" "Name=association.main,Values=true" \ - --query 'RouteTables[0].RouteTableId' --output text) -fi -DEFAULT_ROUTE=$(aws ec2 describe-route-tables --route-table-ids "$ROUTE_TABLE_ID" --region $REGION \ - --query "RouteTables[0].Routes[?DestinationCidrBlock=='0.0.0.0/0'] | [0]" --output json) -EGRESS_TARGET=$(echo "$DEFAULT_ROUTE" | jq -r '.GatewayId // .NatGatewayId // .TransitGatewayId // .VpcPeeringConnectionId // "MISSING"') -if [ "$EGRESS_TARGET" = "MISSING" ] || [ "$EGRESS_TARGET" = "null" ]; then - echo "ERROR: subnet $SUBNET_ID has no default route (0.0.0.0/0). The runner won't" - echo " reach atx ct backend / ECR / S3. The customer's network team must add" - echo " a NAT gateway, internet gateway, or transit gateway route before deploying." - echo " (We do NOT auto-provision NAT -- those are real network changes.)" - exit 1 -fi -echo " ✓ Subnet has default route via $EGRESS_TARGET." - -# ────────────────────────────────────────────────────────────────────────── -# 3. List security groups in the chosen VPC. Show ID, Name, description. -# ────────────────────────────────────────────────────────────────────────── -echo "" -echo "Security groups in $VPC_ID:" -aws ec2 describe-security-groups --region $REGION --filters "Name=vpc-id,Values=$VPC_ID" \ - --query 'SecurityGroups[*].{GroupId:GroupId,Name:GroupName,Description:Description}' \ - --output table -``` - -**STOP HERE.** The agent MUST present the list above and ask the customer which security group to reuse, or whether to let the stack create a new one. The agent MUST NOT default to "let the stack create one" without asking -- that's the customer's choice. The agent MUST NOT proceed to the final confirmation step until the customer has typed an SG ID or `new`. Suggested phrasing: - -> "Pick a security group for the runner, or type 'new' to let the stack create a fresh one with no inbound and allow-all outbound. -> If you reuse an existing SG, it MUST allow outbound HTTPS (port 443) to atx ct backend, ECR, S3, Secrets Manager, and (if applicable) your internal git host. I'll verify outbound 443 is allowed before proceeding. -> Please reply with the SG ID or 'new'." - -```bash -read -p "Security group ID (or 'new' to create one): " SG_ANSWER -if [ "$SG_ANSWER" = "new" ] || [ -z "$SG_ANSWER" ]; then - EXISTING_SG_ID="" - echo " ✓ Stack will create a new no-inbound, allow-all-egress SG." -else - EXISTING_SG_ID="$SG_ANSWER" - - # Validation #4: reused SG allows outbound HTTPS. - EGRESS_443=$(aws ec2 describe-security-groups --group-ids "$EXISTING_SG_ID" --region $REGION \ - --query "SecurityGroups[0].IpPermissionsEgress[?FromPort==\`443\` || FromPort==null || IpProtocol=='-1'] | [0]" \ - --output json 2>/dev/null) - if [ "$EGRESS_443" = "null" ] || [ -z "$EGRESS_443" ]; then - echo "ERROR: security group $EXISTING_SG_ID does not appear to allow outbound HTTPS." - echo " Add an egress rule for TCP 443 to 0.0.0.0/0 (or to the specific atx ct," - echo " ECR, S3, Secrets Manager, and internal git host CIDRs) before deploying." - exit 1 - fi - echo " ✓ Security group $EXISTING_SG_ID allows outbound HTTPS." -fi - -echo "" -echo "Final selections:" -echo " VPC: $VPC_ID" -echo " Subnet: $SUBNET_ID (AZ $SUBNET_AZ)" -[ -n "$EXISTING_SG_ID" ] && echo " SG: $EXISTING_SG_ID (reused)" || echo " SG: stack will create a new one" -``` - -**FINAL CONFIRMATION GATE.** The agent MUST present the four selections above (VPC, subnet, SG, AZ) to the customer in a clear summary and ask explicit confirmation before advancing to Step 5c. Suggested phrasing: - -> "Here's what I'll deploy with: -> - **VPC**: `$VPC_ID` -> - **Subnet**: `$SUBNET_ID` (AZ `$SUBNET_AZ`) -> - **Security Group**: `$EXISTING_SG_ID` (reused) ← OR → stack will create a new no-inbound, allow-all-egress SG -> - **WorkerCount / InstanceType / VolumeSize**: (from Step 2) -> -> Proceed with these? (yes / no -- type yes to continue to the admin handoff, or anything else to revise)" - -The agent MUST wait for the customer's explicit `yes` (or equivalent affirmative) before advancing. If the customer says no or wants to change something, the agent MUST loop back to the relevant step and re-ask. **The agent MUST NOT skip this confirmation, even if every selection looks reasonable** -- this is the last chance for the customer to catch a mistake before the admin is asked to deploy infrastructure. - -**The skill NEVER creates VPCs, subnets, or NAT gateways.** It only describes them, asks the customer to choose, validates the choice, and (on the SG side) lets the stack create one when the customer doesn't want to reuse one. All other network resources are customer-provisioned, customer-owned. If the account has no VPCs or no subnets in the chosen VPC, the skill bails and tells the customer to provision them first -- those are infrastructure changes that need network-team approval, not something the skill should silently do. - -**Step 5c: Write the CFN template and create the stack:** - -```bash -STACK_NAME="${STACK_NAME:-atx-runner}" - -# Write the template inline. Customer can inspect /tmp/atx-ec2-stack.yaml before deploy. -cat > /tmp/atx-ec2-stack.yaml <<'CFN_EOF' -AWSTemplateFormatVersion: '2010-09-09' -Description: ATX CT runner - single EC2 instance with the atx-ct container running. - -Parameters: - InstanceType: - Type: String - Default: m5.2xlarge - AllowedValues: [m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.8xlarge, m5.12xlarge] - ImageUri: - Type: String - Default: public.ecr.aws/d9h8z6l7/aws-transform:latest - VpcId: - Type: AWS::EC2::VPC::Id - Description: VPC where the runner will be deployed. If the git host is private/internal (self-managed GitLab, GHES, Bitbucket DC), provide a VPC with a route to it (VPN, Direct Connect, or peering). - SubnetId: - Type: AWS::EC2::Subnet::Id - Description: Subnet for the runner. Must have outbound internet access (NAT, IGW, or transit gateway) to reach the atx ct backend, ECR (image pulls), S3, and Secrets Manager. - ExistingSecurityGroupId: - Type: String - Default: '' - Description: Optional. If provided, the stack reuses this security group instead of creating a new one. The reused SG MUST allow outbound HTTPS (port 443) to the atx ct backend, ECR, S3, Secrets Manager, and (if applicable) the customer's internal git host. Leave empty to let the stack create a new no-inbound SG. - VolumeSizeGB: - Type: Number - Default: 100 - MinValue: 50 - WorkerCount: - Type: Number - Default: 1 - MinValue: 1 - MaxValue: 5 - Description: Number of parallel atx-ct containers (1-5). Each container is memory-capped at (instance RAM minus 4 GB) divided by WorkerCount, so WorkerCount must be sized to the InstanceType. The template default of 1 is matched to the default InstanceType (m5.2xlarge, 32 GB). IMPORTANT - if you raise WorkerCount, also raise InstanceType so each worker still gets enough RAM (use m5.4xlarge for 2-4 workers, m5.8xlarge for 5); the laptop-side provision script auto-sizes InstanceType from WorkerCount for you. WorkerCount of 1 creates a single container named atx-ct (legacy behavior); WorkerCount above 1 creates atx-ct-1, atx-ct-2, etc. For more than 5 parallel jobs, use the Batch path. - -Conditions: - CreateNewSG: !Equals [!Ref ExistingSecurityGroupId, ''] - -Resources: - TransformRole: - Type: AWS::IAM::Role - Properties: - RoleName: !Sub 'atx-transform-role-${AWS::StackName}' - AssumeRolePolicyDocument: - Version: '2012-10-17' - Statement: - - Effect: Allow - Principal: { Service: ec2.amazonaws.com } - Action: sts:AssumeRole - ManagedPolicyArns: - - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore - Policies: - - PolicyName: atx-transform-access - PolicyDocument: - Version: '2012-10-17' - Statement: - - Effect: Allow - Action: 'transform-custom:*' - Resource: '*' - - Effect: Allow - Action: [s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject] - Resource: - - !Sub 'arn:aws:s3:::atx-source-code-${AWS::AccountId}' - - !Sub 'arn:aws:s3:::atx-source-code-${AWS::AccountId}/*' - - !Sub 'arn:aws:s3:::atx-ct-output-${AWS::AccountId}' - - !Sub 'arn:aws:s3:::atx-ct-output-${AWS::AccountId}/*' - - Effect: Allow - Action: [kms:GenerateDataKey, kms:Decrypt, kms:Encrypt, kms:DescribeKey] - Resource: !Sub 'arn:aws:kms:*:${AWS::AccountId}:key/*' - Condition: - StringLike: { 'kms:ViaService': 's3.*.amazonaws.com' } - - Effect: Allow - Action: secretsmanager:GetSecretValue - Resource: !Sub 'arn:aws:secretsmanager:*:${AWS::AccountId}:secret:atx/*' - - Effect: Allow - Action: - - securityagent:ListAgentSpaces - - securityagent:CreateCodeReview - - securityagent:StartCodeReviewJob - - securityagent:ListCodeReviewJobsForCodeReview - - securityagent:ListFindings - - securityagent:BatchGetFindings - - securityagent:StartCodeRemediation - Resource: 'arn:aws:securityagent:*:*:agent-space*' - Condition: - StringEquals: { 'aws:ResourceAccount': !Ref AWS::AccountId } - - Effect: Allow - Action: [s3:GetObject, s3:ListBucket] - Resource: - - 'arn:aws:s3:::kct-security-agent-*' - - 'arn:aws:s3:::kct-security-agent-*/*' - - Effect: Allow - Action: s3:PutObject - Resource: 'arn:aws:s3:::kct-security-agent-*/security-scans/*' - - Effect: Allow - Action: iam:PassRole - Resource: !Sub 'arn:aws:iam::${AWS::AccountId}:role/security-agent-*' - Condition: - StringEquals: - 'iam:PassedToService': securityagent.amazonaws.com - - TransformInstanceProfile: - Type: AWS::IAM::InstanceProfile - Properties: - InstanceProfileName: !Sub 'atx-transform-profile-${AWS::StackName}' - Roles: [!Ref TransformRole] - - SecurityGroup: - Type: AWS::EC2::SecurityGroup - Condition: CreateNewSG - Properties: - GroupDescription: ATX Transform EC2 - no inbound (access via SSM) - VpcId: !Ref VpcId - Tags: - - Key: Name - Value: !Sub 'atx-transform-sg-${AWS::StackName}' - - Instance: - Type: AWS::EC2::Instance - CreationPolicy: - ResourceSignal: { Timeout: PT15M, Count: 1 } - Properties: - InstanceType: !Ref InstanceType - ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64}}' - IamInstanceProfile: !Ref TransformInstanceProfile - SubnetId: !Ref SubnetId - SecurityGroupIds: - - !If [CreateNewSG, !Ref SecurityGroup, !Ref ExistingSecurityGroupId] - MetadataOptions: - # Enforce IMDSv2 (token-based, defense against SSRF) and allow 2 hops so - # containers using bridge networking can reach IMDS for IAM credentials. - # AWS recommendation for Docker-on-EC2: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html - HttpEndpoint: enabled - HttpTokens: required - HttpPutResponseHopLimit: 2 - BlockDeviceMappings: - - DeviceName: /dev/xvda - Ebs: { VolumeSize: !Ref VolumeSizeGB, VolumeType: gp3, DeleteOnTermination: true } - Tags: - - { Key: Name, Value: !Sub 'atx-ct-runner-${AWS::StackName}' } - - { Key: ManagedBy, Value: ATX-CFN } - - { Key: StackName, Value: !Ref AWS::StackName } - UserData: - Fn::Base64: !Sub | - #!/bin/bash - set -e - trap 'cfn-signal -e $? --stack ${AWS::StackName} --resource Instance --region ${AWS::Region}' ERR EXIT - - # Precheck FIRST (before dnf/docker pull, which take minutes): reject a - # WorkerCount that does not fit this instance's RAM. Each worker needs - # >=2048 MB (the per-worker floor applied at launch) plus ~4 GB reserved - # for the OS/Docker/SSM agent; if the host cannot satisfy that, the 2048 - # MB floor would over-commit RAM and the OOM-killer could take the SSM - # agent -- exactly the failure this template prevents. Failing here (set - # -e + ERR/EXIT cfn-signal trap rolls the stack back) surfaces a bad - # WorkerCount/InstanceType pairing in seconds instead of after the image - # pull. RAM is static, so nothing below changes this verdict. - MEM_TOTAL_MB=$(( $(awk '/MemTotal/{print $2}' /proc/meminfo) / 1024 )) - REQUIRED_MB=$(( ${WorkerCount} * 2048 + 4096 )) - if [ "$MEM_TOTAL_MB" -lt "$REQUIRED_MB" ]; then - echo "FATAL: host has ${!MEM_TOTAL_MB} MB but WorkerCount=${WorkerCount} requires ${!REQUIRED_MB} MB (2048 MB/worker + 4096 MB reserved). Raise InstanceType or lower WorkerCount." >&2 - exit 1 - fi - - dnf install -y docker - systemctl start docker - systemctl enable docker - usermod -aG docker ec2-user - docker pull ${ImageUri} - - # Worker naming convention: - # WorkerCount=1 -> single container named "atx-ct" (existing behavior) - # WorkerCount>1 -> "atx-ct-1", "atx-ct-2", ... "atx-ct-N" - # Bridge networking (no --net=host) so multiple containers can coexist. - if [ "${WorkerCount}" -eq 1 ]; then - CONTAINERS="atx-ct" - else - CONTAINERS=$(seq -f "atx-ct-%g" 1 ${WorkerCount}) - fi - - # Cap each container's memory so a runaway analysis (heavy parallel - # cloning of many/large repos) can only OOM its own container, never - # the host. An unbounded container can exhaust host RAM, get the SSM - # agent OOM-killed (severing all access), and -- with the old - # --restart unless-stopped -- loop forever. Reserve ~4 GB for the OS, - # Docker daemon, and SSM agent; split the remainder across workers. - # --restart on-failure:3 bounds that OOM loop (an OOM is exit 137 = - # failure, retried at most 3x then left down). It intentionally does - # NOT restart a clean exit 0: `atx ct server` is a forever-daemon, so - # a clean exit is unexpected and SHOULD surface as a provision failure - # via the health check below rather than be silently resurrected. A - # boot-time OOM is not expected (an idle server needs far less than - # the >=2048 MB floor), so the cap will realistically only bite under - # a real runaway analysis, not at provision. - # MEM_TOTAL_MB was computed and range-checked in the precheck at the top - # of UserData; reuse it to split the budget across workers. - MEM_PER_WORKER_MB=$(( (MEM_TOTAL_MB - 4096) / ${WorkerCount} )) - if [ "$MEM_PER_WORKER_MB" -lt 2048 ]; then MEM_PER_WORKER_MB=2048; fi - - for name in $CONTAINERS; do - docker run -d --name "$name" --restart on-failure:3 \ - --memory="${!MEM_PER_WORKER_MB}m" --memory-swap="${!MEM_PER_WORKER_MB}m" \ - --entrypoint /bin/bash \ - -e CT_OUTPUT_BUCKET=atx-ct-output-${AWS::AccountId} \ - -e AWS_REGION=${AWS::Region} \ - ${ImageUri} \ - -c 'mkdir -p /home/atxuser/.atxct/sources /home/atxuser/.atxct/shared && \ - source ~/.bashrc && atx ct server' - done - - # Wait for all containers to report healthy in PARALLEL (background each - # health-check, then wait on all PIDs). Sequential checking would not fit - # within the CFN CreationPolicy timeout for higher WorkerCount values. - # Note: ${!name} is the CFN !Sub escape -- !Sub leaves it as a literal - # dollar-brace for bash to resolve. An unescaped bash variable in this - # !Sub block would error with "Unresolved resource dependencies" - # because !Sub treats dollar-brace refs as CFN resource references. - # Each worker fast-fails only when its container is TERMINALLY down, so a - # dead worker surfaces within seconds instead of waiting out the full 300s - # poll -- WITHOUT tripping on the transient "exited" snapshot a container - # shows BETWEEN --restart on-failure:3 retries. Terminal = Status=dead, OR - # Status=exited with either a clean ExitCode 0 (on-failure never restarts a - # clean exit) or RestartCount>=3 (retries exhausted). A non-zero exit with - # RestartCount<3 is mid-retry -- keep polling, it may still come up. - PIDS=() - for name in $CONTAINERS; do - ( - for i in $(seq 1 60); do - STATUS=$(docker inspect "$name" --format '{{.State.Status}}' 2>/dev/null || echo missing) - EC=$(docker inspect "$name" --format '{{.State.ExitCode}}' 2>/dev/null || echo -1) - RC=$(docker inspect "$name" --format '{{.RestartCount}}' 2>/dev/null || echo 0) - if [ "$STATUS" = dead ] || { [ "$STATUS" = exited ] && { [ "$EC" = 0 ] || [ "$RC" -ge 3 ]; }; }; then - OOM=$(docker inspect "$name" --format '{{.State.OOMKilled}}' 2>/dev/null) - echo "Worker ${!name} terminal: Status=$STATUS OOMKilled=$OOM ExitCode=$EC RestartCount=$RC" >&2 - exit 1 - fi - if docker exec "$name" bash -c 'atx ct status --health' > /dev/null 2>&1; then exit 0; fi - sleep 5 - done - echo "Worker ${!name} never became healthy within 300s" >&2 - exit 1 - ) & - PIDS+=($!) - done - for pid in "${!PIDS[@]}"; do - wait "$pid" || { echo "Health check failed for one or more workers" >&2; exit 1; } - done - for name in $CONTAINERS; do - docker ps --filter "name=^${!name}$" --filter status=running --format '{{.Names}}' | grep -q "^${!name}$" - docker exec "$name" bash -c 'atx ct status --health' > /dev/null 2>&1 - done - - trap - ERR EXIT - cfn-signal -e 0 --stack ${AWS::StackName} --resource Instance --region ${AWS::Region} - -Outputs: - StackName: { Value: !Ref AWS::StackName } - InstanceId: { Value: !Ref Instance } - RoleArn: { Value: !GetAtt TransformRole.Arn } - InstanceProfileName: { Value: !Ref TransformInstanceProfile } - SecurityGroupId: - Value: !If [CreateNewSG, !GetAtt SecurityGroup.GroupId, !Ref ExistingSecurityGroupId] - AccountId: { Value: !Ref AWS::AccountId } - Region: { Value: !Ref AWS::Region } -CFN_EOF - -# Worker count (default 1; max 5). Default of 1 matches the CFN template default and -# keeps each worker on a right-sized box (InstanceType is auto-sized from WORKER_COUNT -# below, and each container is memory-capped at (instance RAM - 4 GB) / WORKER_COUNT at -# launch). Raise it for more parallelism (e.g. WORKER_COUNT=3 or 5); the auto-sizing -# bumps InstanceType so each worker still gets enough RAM. Changing it after provisioning -# is destructive (see "Changing WorkerCount" section). -WORKER_COUNT="${WORKER_COUNT:-1}" -if [ "$WORKER_COUNT" -lt 1 ] || [ "$WORKER_COUNT" -gt 5 ]; then - echo "ERROR: WORKER_COUNT must be 1-5. Got: $WORKER_COUNT. For more parallelism, use the Batch path." >&2 - exit 1 -fi - -# Auto-recommend InstanceType based on WorkerCount if customer did not override. -# Sizing assumes typical analyses (single-repo fan-out). For monorepos or 10x source-wide -# analyses simultaneously, customer should override INSTANCE_TYPE=m5.12xlarge. -if [ -z "$INSTANCE_TYPE" ]; then - if [ "$WORKER_COUNT" -le 1 ]; then INSTANCE_TYPE="m5.2xlarge" - elif [ "$WORKER_COUNT" -le 4 ]; then INSTANCE_TYPE="m5.4xlarge" - else INSTANCE_TYPE="m5.8xlarge" - fi -fi - -# Auto-recommend disk size: 50 GB per worker (covers typical and heavy use; override -# to 100 GB/worker for monorepos via VOLUME_SIZE env var). -VOLUME_SIZE="${VOLUME_SIZE:-$((50 * WORKER_COUNT))}" - -# ────────────────────────────────────────────────────────────────────────── -# Pre-deploy confirmation: ASK THE CUSTOMER before creating the stack. -# Show them the resolved config so they can override WorkerCount, InstanceType, -# or VolumeSize before commit. Customer is responsible for checking AWS pricing. -# ────────────────────────────────────────────────────────────────────────── -cat < **Admin handoff -- one-time setup** -> -> I've written the CloudFormation template to `/tmp/atx-ec2-stack.yaml`. **This stack creates IAM roles, so deploying requires admin / role-creation permissions (`iam:CreateRole`, `iam:PutRolePolicy`, `iam:PassRole`, instance profiles). Run it with an admin identity. Read-only or runtime credentials are enough for everything afterward.** -> -> The agent MUST include the following sentence verbatim in every Step 5d handoff, immediately after the admin-identity sentence above and before the command block. Do NOT abbreviate, drop, or paraphrase it -- customers onboarding a new executor identity rely on this pointer: -> -> For reference, the executor policy this skill expects is in https://github.com/kirodotdev/powers/blob/main/aws-transform/steering/AWSTransformInfrastructureExecutorAccessEC2.json -> -> Those permissions are admin-scope; the executor permissions I'm running under intentionally do not grant them, so day-to-day analysis runs cannot escalate privileges. -> -> Ask someone in your account with admin / role-creation permissions (or yourself if you have a separate admin profile) to run these commands from the same shell, in the same region. **Replace `` with the AWS profile name that has admin / role-creation permissions in your environment.** - -**Profile-name guidance for the agent.** When emitting this admin handoff (or any of the other admin handoffs in this skill), the agent MUST use the placeholder `` rather than guessing a profile name from the customer's local AWS config, environment variables, or shell history. Customers commonly have multiple AWS profiles configured locally and the agent has no reliable way to identify which one carries admin permissions. Substituting a wrong name leads to confusing AccessDenied errors during deploy. Examples: - -- ❌ `AWS_PROFILE=atx-zerog-admin aws cloudformation create-stack ...` (the agent guessed from `~/.aws/config`) -- ❌ `AWS_PROFILE=admin aws cloudformation create-stack ...` (the agent assumed a name) -- ✅ `AWS_PROFILE= aws cloudformation create-stack ...` (placeholder for the customer to fill in) - -This rule applies to **every admin handoff in this skill**: create-stack, delete-stack, instance-tag handoff, instance-role-policy handoff, anywhere else admin is invoked. - -The full handoff command set (admin runs in their shell, in `${REGION}`): -> -> ```bash -> # 1. Create the persistent S3 buckets (only if Step 5a reported them missing). -> # These live OUTSIDE the CFN stack so they survive delete-and-recreate. -> # us-east-1 quirk: --create-bucket-configuration LocationConstraint=us-east-1 -> # is rejected by the API; omit the flag in that one region. -> LOC_CONSTRAINT="" -> [ "$REGION" != "us-east-1" ] && LOC_CONSTRAINT="--create-bucket-configuration LocationConstraint=$REGION" -> -> aws s3api create-bucket --bucket atx-source-code-${ACCOUNT_ID} --region $REGION $LOC_CONSTRAINT -> aws s3api put-bucket-lifecycle-configuration --bucket atx-source-code-${ACCOUNT_ID} \ -> --lifecycle-configuration '{"Rules":[{"ID":"expire-7d","Status":"Enabled","Expiration":{"Days":7},"Filter":{"Prefix":""}}]}' -> -> aws s3api create-bucket --bucket atx-ct-output-${ACCOUNT_ID} --region $REGION $LOC_CONSTRAINT -> aws s3api put-bucket-lifecycle-configuration --bucket atx-ct-output-${ACCOUNT_ID} \ -> --lifecycle-configuration '{"Rules":[{"ID":"expire-30d","Status":"Enabled","Expiration":{"Days":30},"Filter":{"Prefix":""}}]}' -> -> # 2. Create the CFN stack (instance, IAM role/profile, security group). -> aws cloudformation create-stack \ -> --stack-name "$STACK_NAME" \ -> --template-body file:///tmp/atx-ec2-stack.yaml \ -> --capabilities CAPABILITY_NAMED_IAM \ -> --parameters \ -> ParameterKey=VpcId,ParameterValue=$VPC_ID \ -> ParameterKey=SubnetId,ParameterValue=$SUBNET_ID \ -> ParameterKey=InstanceType,ParameterValue=$INSTANCE_TYPE \ -> ParameterKey=WorkerCount,ParameterValue=$WORKER_COUNT \ -> ParameterKey=VolumeSizeGB,ParameterValue=$VOLUME_SIZE \ -> ParameterKey=ExistingSecurityGroupId,ParameterValue="$EXISTING_SG_ID" \ -> --region $REGION \ -> --tags Key=atx-remote-infra,Value=true -> -> aws cloudformation wait stack-create-complete \ -> --stack-name "$STACK_NAME" --region $REGION -> ``` -> -> When the deploy finishes, come back to this conversation and tell me -- I'll re-detect the stack via `describe-stacks` (which my executor creds CAN do) and continue from Step 6. - -The agent then STOPS this turn. The admin runs the commands in their own terminal, outside the chat. On the next user turn, re-run **Step 0 (Detect)** -- the stack should now be `CREATE_COMPLETE` and the flow resumes at Step 6. - -**Why CloudFormation:** - -| Concern | CFN advantage | -|---|---| -| Audit trail | Single stack event log shows every resource created | -| Atomic deploy | Failure rolls back entire stack -- no orphaned IAM roles or instances | -| Drift detection | Customer can run `aws cloudformation detect-stack-drift` to see if anything changed manually | -| Teardown | Single `aws cloudformation delete-stack` cleans up everything in the stack | -| Multi-stack support | Customer can run `STACK_NAME=dev`, `STACK_NAME=prod` for isolated runners | -| Visibility | Customer's CloudFormation console shows the resources, parameters, and outputs | - -### Step 6: Verify the Container is Running - -The CFN stack's `CreationPolicy` ensures `CREATE_COMPLETE` fires only after the UserData script signals success -- meaning Docker is installed, the image is pulled, and the atx-ct container is up. So verification is a quick confidence check. - -```bash -# Define the SSM helpers (used by all subsequent steps for short status calls -# and fire-and-forget submissions of long-running work): -# -# ssm_submit -- fire-and-forget. Returns SSM CommandId immediately. NEVER blocks. -# Use for build_command_*() submissions. -# ssm_run -- submit + wait + get output. Blocks until command completes -# (~100s SSM-side timeout). Use for short status commands. -# -# DO NOT use ssm_run for build_command_*() -- the wrapper runs for hours. - -ssm_submit() { - aws ssm send-command --region $REGION \ - --instance-ids "$INSTANCE_ID" \ - --document-name AWS-RunShellScript \ - --parameters "commands=[\"$1\"]" \ - --query 'Command.CommandId' --output text -} - -ssm_run() { - local cmd="$1" - local CMD_ID=$(aws ssm send-command --region $REGION \ - --instance-ids "$INSTANCE_ID" \ - --document-name AWS-RunShellScript \ - --parameters "commands=[\"$cmd\"]" \ - --query 'Command.CommandId' --output text) - aws ssm wait command-executed --command-id "$CMD_ID" --instance-id "$INSTANCE_ID" --region $REGION 2>/dev/null || true - aws ssm get-command-invocation --region $REGION \ - --command-id "$CMD_ID" --instance-id "$INSTANCE_ID" \ - --query 'StandardOutputContent' --output text -} - -# Resolve CONTAINER_NAME based on stack's WorkerCount + the desired worker. -# WorkerCount=1 (default): single container "atx-ct" (existing behavior). -# WorkerCount>1: containers "atx-ct-1", "atx-ct-2", ..., "atx-ct-N". -# WORKER_NUM is the 1-indexed worker to target (1..WorkerCount). Defaults to 1. -WORKER_COUNT=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region $REGION \ - --query 'Stacks[0].Parameters[?ParameterKey==`WorkerCount`].ParameterValue' --output text 2>/dev/null) -WORKER_COUNT=$(echo "$WORKER_COUNT" | xargs) # strip whitespace defensively -[ -z "$WORKER_COUNT" ] || [ "$WORKER_COUNT" = "None" ] && WORKER_COUNT=1 -WORKER_NUM="${WORKER_NUM:-1}" -if [ "$WORKER_COUNT" -eq 1 ]; then - CONTAINER_NAME="atx-ct" -else - if [ "$WORKER_NUM" -lt 1 ] || [ "$WORKER_NUM" -gt "$WORKER_COUNT" ]; then - echo "ERROR: WORKER_NUM ($WORKER_NUM) must be 1-${WORKER_COUNT}." >&2 - exit 1 - fi - CONTAINER_NAME="atx-ct-${WORKER_NUM}" -fi - -# Confirm container is running and atx ct server is healthy -ssm_run "sudo docker ps --filter \"name=^${CONTAINER_NAME}$\" --filter status=running --format '{{.Names}}: {{.Status}}'" -ssm_run "sudo docker exec ${CONTAINER_NAME} atx ct status --health" -``` - -If either check fails, inspect the container logs: - -```bash -ssm_run "sudo docker logs ${CONTAINER_NAME} 2>&1 | tail -50" +# Poll until all jobs complete +atx ct remote status --group --wait ``` -For a fully failed bootstrap, the stack would be in `ROLLBACK_COMPLETE` (UserData failed → cfn-signal sent error → stack rolled back). Check stack events: - -```bash -aws cloudformation describe-stack-events --stack-name "$STACK_NAME" --region $REGION \ - --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[ResourceType,ResourceStatusReason]' --output table -``` +Shows per-job status with completion counts. -**Local source preparation (local provider only):** if `PROVIDER=local`, sync repo bundles into the container after the container is up: +### 5. Cancel ```bash -if [ "$PROVIDER" = "local" ]; then - # Customer must have already uploaded zips to s3://atx-source-code-${ACCOUNT_ID}/repos/ - ssm_run "sudo docker exec ${CONTAINER_NAME} bash -c 'mkdir -p /home/atxuser/repos /tmp/zips && \ - aws s3 sync s3://atx-source-code-${ACCOUNT_ID}/repos/ /tmp/zips/ && \ - for zip in /tmp/zips/*.zip; do unzip -q -o \"\$zip\" -d /home/atxuser/repos/; done'" -fi -``` - -#### Security analysis prerequisite - -If `ANALYSIS_TYPE=security` (or `agentic-readiness` / `modernization-readiness` which depend on it), the security agent must be set up first. See [continuous-modernization-setup](workload-continuous-modernization-setup.md) for `atx ct setup security-agent`. - -The S3 + `iam:PassRole` grants the instance role needs for security analysis are **always-on** in the CFN template (the `securityagent:*` actions, `s3:*` on `kct-security-agent-*`, and `iam:PassRole` on `security-agent-*` are part of the base role policy). No stack redeploy is required if the customer decides to run security analysis after the stack is up -- the role already has what's needed. - -The agent space is provisioned during `atx ct setup security-agent`, which writes the populated `agentSpaceId` into `~/.atxct/shared/security_agent_config.json`. The EC2 runtime is read-only -- it finds the existing agent space via `list-agent-spaces` and never creates one. Sync the local config file into the EC2 container so the runtime can find the existing agent space: +# Cancel all jobs in a group +atx ct remote cancel --group -```bash -# Sync security agent config from laptop into all atx-ct containers. -# The loop applies to single-worker (just "atx-ct") and multi-worker (atx-ct-1..N) stacks. -aws s3 cp ~/.atxct/shared/security_agent_config.json \ - s3://atx-source-code-${ACCOUNT_ID}/temp/security_agent_config.json -ssm_run "aws s3 cp s3://atx-source-code-${ACCOUNT_ID}/temp/security_agent_config.json /tmp/sa.json && \ - for c in \$(sudo docker ps --filter name=atx-ct --format '{{.Names}}'); do \ - sudo docker cp /tmp/sa.json \$c:/home/atxuser/.atxct/shared/security_agent_config.json && \ - sudo docker exec \$c chown 1000:1000 /home/atxuser/.atxct/shared/security_agent_config.json; \ - done" -aws s3 rm s3://atx-source-code-${ACCOUNT_ID}/temp/security_agent_config.json +# Cancel a single job +atx ct remote cancel --group --job ``` -### Step 7: Confirm and Submit - -**Validate credentials (non-local providers):** Before confirming, verify that the required secret exists in Secrets Manager. Skip for local provider sources. - -| Provider | Required Secret | -| ------------- | ------------------------ | -| **github** | `atx/github-token` | -| **gitlab** | `atx/gitlab-token` | -| **bitbucket** | `atx/bitbucket-token` | -| **local** | (none — skip) | - -```bash -aws secretsmanager describe-secret --secret-id --region $REGION 2>&1 -``` - -- If `ResourceNotFoundException` → inform the user that the secret is missing. Give them the command to run in their own terminal: - -```bash -read -s TOKEN && { aws secretsmanager create-secret --name "" \ - --secret-string "$TOKEN" --region $REGION 2>/dev/null \ - || aws secretsmanager put-secret-value --secret-id "" \ - --secret-string "$TOKEN" --region $REGION; }; unset TOKEN -``` +### 6. Submit Remediation -- If the secret exists → ask the customer: "Your `` token was last updated on ``. Would you like to rotate it, or is the current token still valid?" If they want to rotate: +Requires completed analysis with findings. ```bash -read -s TOKEN && aws secretsmanager put-secret-value --secret-id "" \ - --secret-string "$TOKEN" --region $REGION; unset TOKEN +atx ct remote remediation \ + --ids , \ + --mode ec2 \ + --stack-name \ + --telemetry "agent=kiro,executionMode=ec2" ``` -Tell the customer what will happen and wait for explicit confirmation. - -**For GitHub:** -> "I'll submit `` on EC2 instance `${INSTANCE_ID}` against your GitHub source ``. The container is already configured with your GitHub PAT. The submission will: -> - Run `atx ct analysis run --type --source ` in the background -> - Poll status until complete -> - Upload artifacts to `s3://atx-ct-output-${ACCOUNT_ID}///code.zip` -> -> Continue?" - -**For GitLab:** same as GitHub with `atx/gitlab-token`. - -**For Bitbucket Cloud:** -> "I'll submit `` on EC2 instance `${INSTANCE_ID}` against your Bitbucket source ``. The container will: -> - Place your Bitbucket API token (from Secrets Manager `atx/bitbucket-token`) and inject email/username into config.json -> - Run `atx ct analysis run --type --source ` in the background -> - Poll status until complete -> - Upload artifacts to S3 -> -> Continue?" - -**For Bitbucket Data Center:** -> "I'll submit `` on EC2 instance `${INSTANCE_ID}` against your Bitbucket Data Center source ``. The container will: -> - Place your HTTP Access Token (from Secrets Manager `atx/bitbucket-token`) and inject base_url into config.json -> - Run `atx ct analysis run --type --source ` in the background -> - Poll status until complete -> - Upload artifacts to S3 -> -> Continue?" +Filter options: +- `--ids ` — specific finding IDs +- `--sources ` / `--repos ::` — by source/repo +- `--severity high` — exact match +- `--min-severity medium` — minimum threshold +- `--labels ` — filter by repo labels +- `--transformation-name ` — override fix strategy +- `-g, --configuration ` — config path or key=value pairs (only valid with `--transformation-name`) -**For Local:** same as GitHub with bundle synced to `/home/atxuser/repos`. +Stack targeting is the same as analysis: `--stack-name `, discover by resource tags with `--tags env=prod,...` (alternative to `--stack-name`), or `--existing-instance ` for BYO. -Do NOT submit until the customer confirms. +### 7. Get Results -### Step 8: Submit Work +Status output already shows key result info for completed jobs: +- **Analysis**: `resultId` and finding count +- **Remediation**: `resultId` and PR/MR URL (for SCM sources) or S3 output path (for local sources) -Build the nohup'd command via `build_command_*()` (returns one self-contained script that runs analysis → polls status → uploads artifacts) and submit via SSM. The SSM call returns immediately because the script is backgrounded. The agent stays free during the long-running work. +For deeper detail on findings: ```bash -ANALYSIS_TYPE="" # tech-debt-quick | tech-debt-comprehensive | security | agentic-readiness | modernization-readiness | custom -AGENT="" # AI assistant name (kiro, claude, amazonq, etc.) -JOB_ID="atxct-$(date +%s)" # unique per submission; per-job state files keyed by this -REPO_FILTER="" # empty = whole source; or "--repo ::" (ONE repo only, never multiple) -EXTRA_FLAGS="" # for --type custom: "--transformation-name -g 'KEY=VAL'" -BITBUCKET_WORKSPACE="" # bitbucket only -- workspace (Cloud) or project key (DC) -BITBUCKET_EMAIL="" # bitbucket cloud only -- email for API auth -BITBUCKET_USERNAME="" # bitbucket cloud only -- username for git clone/push -BITBUCKET_BASE_URL="" # bitbucket DC only -- e.g. https://bitbucket.corp.example.com (empty for Cloud) +atx ct analysis get --id --json ``` -The script written to the instance follows this shape: +### 8. Update Stack ```bash -# (1) Submit analysis (no --wait; returns AID immediately) -sudo docker exec ${CONTAINER_NAME} atx ct analysis run --type $ANALYSIS_TYPE $EXTRA_FLAGS --source $SOURCE $REPO_FILTER > /tmp/run.log 2>&1 -AID=$(grep -oE '01[A-Z0-9]+' /tmp/run.log | head -1) - -# (2) Poll status until terminal -while true; do - STATUS=$(sudo docker exec ${CONTAINER_NAME} atx ct analysis get --id $AID --json | jq -r .status) - case "$STATUS" in - complete|completed) break ;; - failed) exit 1 ;; - *) sleep 60 ;; - esac -done - -# (3) Upload artifacts (skipped for tech-debt-quick -- read-only scan) -sudo docker exec ${CONTAINER_NAME} /app/upload-ct-artifacts.sh $AID atx-ct-output-$ACCOUNT_ID +atx ct remote update --mode ec2 --stack-name --execute --ack ``` -#### `build_command_*()` builders +Without `--execute`, shows changeset preview. -Each builder writes the wrapper script to the instance via heredoc and launches it via `nohup`. Local-side bash substitutes `${LOGICAL_SOURCE_NAME}`, `${ANALYSIS_TYPE}`, etc.; runtime values like `$AID` and `$STATUS` are escaped (`\$`) so they're evaluated on the instance. +### 9. Teardown ```bash -# Analysis on github / gitlab / local -- same wrapper shape, same upload step. -# Token (github/gitlab) is fetched from Secrets Manager at job time and placed in -# the container's source dir. atx ct's async provider resolution queries the -# backend for source metadata, so no config.json is needed in the container. -# -# IMPORTANT: build_command_analysis() returns the script BODY only (clean bash, no -# heredoc tricks). The skill base64-encodes the body and submits a short SSM command -# that decodes-and-runs it. This avoids the multi-level quote-escaping nightmare that -# happens when you try to pass a multi-line bash script through `aws ssm send-command -# --parameters "commands=[\"...\"]"` (the JSON layer + the bash layer collide). -build_command_analysis() { - local UPLOAD_LINE="sudo docker exec ${CONTAINER_NAME} /app/upload-ct-artifacts.sh \$AID atx-ct-output-${ACCOUNT_ID}" - [ "${ANALYSIS_TYPE}" = "tech-debt-quick" ] && UPLOAD_LINE='echo "[skip upload -- tech-debt-quick is read-only]"' - - # Token-injection prelude (runs INSIDE the container at job start) - local TOKEN_PRELUDE="" - if [ "$PROVIDER" = "github" ] || [ "$PROVIDER" = "gitlab" ]; then - local SECRET_ID="atx/${PROVIDER}-token" - local TOKEN_FILE="${PROVIDER}_token" - TOKEN_PRELUDE="sudo docker exec ${CONTAINER_NAME} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id ${SECRET_ID} --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE} && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE}'" - elif [ "$PROVIDER" = "bitbucket" ]; then - # Bitbucket requires token + config.json with email/username (Cloud) or base_url (DC). - # BITBUCKET_WORKSPACE, BITBUCKET_EMAIL, BITBUCKET_USERNAME, BITBUCKET_BASE_URL must be set by caller. - local config_json - if [ -n "${BITBUCKET_BASE_URL}" ]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - local CONFIG_B64=$(echo "${config_json}" | base64 -w 0) - TOKEN_PRELUDE="sudo docker exec ${CONTAINER_NAME} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token'" - fi - - # Return the script body. Local bash substitutes ${ANALYSIS_TYPE}, ${LOGICAL_SOURCE_NAME}, - # ${TOKEN_PRELUDE}, etc.; runtime references like \$AID and \$LOG stay as-is. - cat <> \$LOG - -${TOKEN_PRELUDE} - -sudo docker exec ${CONTAINER_NAME} atx ct analysis run --type ${ANALYSIS_TYPE} ${EXTRA_FLAGS} --source ${LOGICAL_SOURCE_NAME} ${REPO_FILTER} --telemetry "agent=${AGENT},executionMode=ec2" >> \$LOG 2>&1 -AID=\$(grep -oE '01[A-Z0-9]+' \$LOG | head -1) -[ -z "\$AID" ] && { echo "ERROR: no analysis ID extracted" >> \$LOG; exit 1; } -echo \$AID > /tmp/atxct-${JOB_ID}.aid - -while true; do - STATUS=\$(sudo docker exec ${CONTAINER_NAME} atx ct analysis get --id \$AID --json 2>/dev/null | jq -r .status 2>/dev/null) - case "\$STATUS" in - complete|completed) echo "=== \$(date) [DONE] analysis \$AID ===" >> \$LOG; break ;; - failed|cancelled) echo "=== \$(date) [\$STATUS] analysis \$AID ===" >> \$LOG; exit 1 ;; - *) echo "\$(date) status=\${STATUS:-pending}" >> \$LOG; sleep 60 ;; - esac -done - -${UPLOAD_LINE} >> \$LOG 2>&1 -echo "=== \$(date) [DONE] upload ===" >> \$LOG -EOF -} - -# Build the script body, base64-encode it (avoids quoting hell when submitting via SSM), -# and submit a single short SSM command that decodes + runs it. -SCRIPT=$(build_command_analysis) -B64=$(echo "$SCRIPT" | base64 | tr -d '\n') - -# Compose a single-line SSM command: -# 1. echo $B64 | base64 -d > /tmp/atxct-.sh (decode script to disk) -# 2. chmod +x ... (make executable) -# 3. ( ( bash ... > log 2>&1 < /dev/null & ) & ) (double-fork orphans wrapper to init) -# 4. echo Started_... (so SSM sees a quick exit) -# -# IMPORTANT: the double-fork is required. Without it, SSM's AWS-RunShellScript -# tracks the wrapper via cgroup and keeps the command slot pinned until the -# wrapper exits, saturating the SSM agent's worker pool. The double-fork -# `( ( bash X & ) & )` reparents the wrapper to init (PID 1) so SSM marks -# the launch command Success immediately. -LAUNCH_CMD="echo ${B64} | base64 -d > /tmp/atxct-${JOB_ID}.sh && chmod +x /tmp/atxct-${JOB_ID}.sh && ( ( bash /tmp/atxct-${JOB_ID}.sh > /tmp/atxct-${JOB_ID}.stdout 2>&1 < /dev/null & ) & ) && echo Started_${JOB_ID}" - -SUBMIT_ID=$(ssm_submit "$LAUNCH_CMD") -echo "Submitted job $JOB_ID (SSM command: $SUBMIT_ID). Ask me to check status anytime." +atx ct remote teardown --mode ec2 --stack-name --execute --ack ``` -The agent prints "Submitted job $JOB_ID" and is free to interact with the user. The wrapper continues on the instance independently -- analysis, polling, and upload all happen there. +Instance, IAM role, security group removed atomically. S3 buckets and Secrets Manager tokens persist. -#### Remediation (instead of analysis) +## BYO Instance (Existing EC2) -Same shape. The build differs by source provider -- github / gitlab use the backend's branch-push flow (no `--local`, no S3 upload); `local` uses `--local` and uploads artifacts. +Use an existing customer-owned EC2 instance instead of provisioning a new stack. The instance must have SSM agent running and the `atx-remote-infra=true` tag. ```bash -build_command_remediation() { - local CREATE_ARGS="" - if [ -n "$FINDING_IDS" ]; then - CREATE_ARGS="--ids ${FINDING_IDS}" - [ -n "${TRANSFORMATION_NAME}" ] && CREATE_ARGS="${CREATE_ARGS} --transformation-name ${TRANSFORMATION_NAME}" - else - CREATE_ARGS="--transformation-name ${TRANSFORMATION_NAME} ${REPO_FILTER}" - fi - [ -n "${CONFIGURATION}" ] && CREATE_ARGS="${CREATE_ARGS} -g \"${CONFIGURATION}\"" - - local LOCAL_FLAG="" - local UPLOAD_LINE='echo "[skip upload -- github/gitlab remediation pushes a branch]"' - if [ "$PROVIDER" = "local" ]; then - LOCAL_FLAG="--local" - UPLOAD_LINE="sudo docker exec ${CONTAINER_NAME} /app/upload-ct-artifacts.sh \$RID atx-ct-output-${ACCOUNT_ID}" - fi - - # Token-injection prelude (runs INSIDE the container at job start) - local TOKEN_PRELUDE="" - if [ "$PROVIDER" = "github" ] || [ "$PROVIDER" = "gitlab" ]; then - local SECRET_ID="atx/${PROVIDER}-token" - local TOKEN_FILE="${PROVIDER}_token" - TOKEN_PRELUDE="sudo docker exec ${CONTAINER_NAME} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && aws secretsmanager get-secret-value --secret-id ${SECRET_ID} --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE} && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/${TOKEN_FILE}'" - elif [ "$PROVIDER" = "bitbucket" ]; then - # Bitbucket requires token + config.json with email/username (Cloud) or base_url (DC). - # BITBUCKET_WORKSPACE, BITBUCKET_EMAIL, BITBUCKET_USERNAME, BITBUCKET_BASE_URL must be set by caller. - local config_json - if [ -n "${BITBUCKET_BASE_URL}" ]; then - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"base_url":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_BASE_URL}") - else - config_json=$(printf '{"provider":"bitbucket","identifier":"%s","provider_config":{"email":"%s","username":"%s"}}' "${BITBUCKET_WORKSPACE}" "${BITBUCKET_EMAIL}" "${BITBUCKET_USERNAME}") - fi - local CONFIG_B64=$(echo "${config_json}" | base64 -w 0) - TOKEN_PRELUDE="sudo docker exec ${CONTAINER_NAME} bash -c 'mkdir -p /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME} && echo ${CONFIG_B64} | base64 -d > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/config.json && aws secretsmanager get-secret-value --secret-id atx/bitbucket-token --query SecretString --output text > /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token && chmod 600 /home/atxuser/.atxct/sources/${LOGICAL_SOURCE_NAME}/bitbucket_token'" - fi - - # Returns clean script body (no heredoc tricks). The skill base64-encodes and submits - # via a short SSM command (same pattern as build_command_analysis above). - cat <> \$LOG - -${TOKEN_PRELUDE} - -sudo docker exec ${CONTAINER_NAME} atx ct remediation create ${CREATE_ARGS} ${LOCAL_FLAG} --source ${LOGICAL_SOURCE_NAME} --telemetry "agent=${AGENT},executionMode=ec2" >> \$LOG 2>&1 -RID=\$(grep -oE '01[A-Z0-9]+' \$LOG | head -1) -[ -z "\$RID" ] && { echo "ERROR: no remediation ID" >> \$LOG; exit 1; } -echo \$RID > /tmp/atxct-${JOB_ID}.rid - -while true; do - STATUS=\$(sudo docker exec ${CONTAINER_NAME} atx ct remediation status --id \$RID --json 2>/dev/null | jq -r .status 2>/dev/null) - case "\$STATUS" in - complete|completed) echo "=== \$(date) [DONE] remediation \$RID ===" >> \$LOG; break ;; - failed|cancelled) echo "=== \$(date) [\$STATUS] remediation \$RID ===" >> \$LOG; exit 1 ;; - *) sleep 60 ;; - esac -done - -${UPLOAD_LINE} >> \$LOG 2>&1 -echo "=== \$(date) [DONE] upload ===" >> \$LOG -EOF -} - -# Same base64 pattern as analysis -SCRIPT=$(build_command_remediation) -B64=$(echo "$SCRIPT" | base64 | tr -d '\n') -LAUNCH_CMD="echo ${B64} | base64 -d > /tmp/atxct-${JOB_ID}.sh && chmod +x /tmp/atxct-${JOB_ID}.sh && ( ( bash /tmp/atxct-${JOB_ID}.sh > /tmp/atxct-${JOB_ID}.stdout 2>&1 < /dev/null & ) & ) && echo Started_${JOB_ID}" - -SUBMIT_ID=$(ssm_submit "$LAUNCH_CMD") -echo "Submitted remediation job $JOB_ID (SSM command: $SUBMIT_ID). Ask me to check status anytime." +atx ct remote analysis \ + --types rapid-techdebt-analysis \ + --sources \ + --mode ec2 \ + --existing-instance \ + --workers 2 \ + --telemetry "agent=kiro,executionMode=ec2" ``` -### Step 9: Status Checking +BYO-specific flags: +- `--existing-instance ` — instance ID (mutually exclusive with `--stack-name`/`--tags`) +- `--workers ` — container count on the instance (default 1) +- `--ct-output-bucket ` — custom output bucket +- `--source-bucket ` — custom source bucket +- `--output-bucket ` — custom artifact bucket -When the customer asks for status, ask `atx ct` for the authoritative state. The wrapper's log file is only useful for DEBUGGING the wrapper itself (e.g., "did the wrapper start? did it parse the AID?"); for "is my analysis done?" the answer comes from the atx ct server. +Note: `--image-uri` is NOT available for BYO — it's a `provision` flag only. The instance must already have the AWS Transform — continuous modernization container image running. -```bash -# Authoritative status. What the customer actually wants to know. -AID=$(ssm_run "cat /tmp/atxct-${JOB_ID}.aid 2>/dev/null" | tr -d '[:space:]') - -if [ -z "$AID" ]; then - # No AID means the wrapper failed before extracting an analysis ID. Most - # likely cause: instance role lacks a permission needed by the wrapper or - # by atx ct's first backend call. Surface the specific error from the - # wrapper log instead of reporting "running" or "pending". - ssm_run "grep -iE 'AccessDenied|not authorized|Error:' /tmp/atxct-${JOB_ID}.log 2>&1 | head -5" - echo "Wrapper failed to dispatch the analysis. See the errors above." - echo "Common causes: missing transform-custom:* (instance role) or secretsmanager:GetSecretValue." - echo "Tell the customer the specific permission identified in the AccessDenied message and ask them to attach it." -else - ssm_run "sudo docker exec ${CONTAINER_NAME} atx ct analysis get --id $AID --json" | \ - jq '{status, repos_total: (.repos | length), findings_count}' -fi -``` - -Or, if you don't have the JOB_ID handy, list all in-flight analyses on the instance: - -```bash -ssm_run "sudo docker exec ${CONTAINER_NAME} atx ct analysis list --json | jq '.items[] | select(.status == \"running\" or .status == \"pending\")'" -``` - -For remediation jobs, swap `analysis get` → `remediation status` and `*.aid` → `*.rid`. - -**Wrapper log tail is only for debugging** (when you need to see what the wrapper is doing on the instance, not what the analysis is doing on the server): - -```bash -ssm_run "tail -20 /tmp/atxct-${JOB_ID}.log" -``` - -To list all in-flight jobs on the instance: - -```bash -ssm_run "ls -la /tmp/atxct-*.aid /tmp/atxct-*.rid 2>/dev/null" -``` - -### Step 10: Get Findings and Artifacts - -**Findings** are persisted by the analysis runner during execution and queryable from anywhere with CT access. **`atx ct findings list --json` returns a top-level array** (no `.items` wrapper): - -```json -[ - { - "id": "01ABC...", - "severity": "high|medium|low", - "category": "security|performance|maintainability|...", - "repo": "::", - "title": "Short description", - "description": "Full description", - "fix": null | { ... }, - ... - }, - ... -] -``` - -> **Heads up -- JSON shape inconsistency across `atx ct` commands.** Some commands return `{"items": [...]}` (e.g., `repository list`, `analysis list`); others return a bare `[...]` (e.g., `findings list`, `source list`). Always assume bare array for `findings list` -- use `.[]` not `.items[]` in jq. - -Common queries: - -```bash -# Total finding count -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json | jq 'length' - -# Group by severity -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json | \ - jq 'group_by(.severity) | map({severity: .[0].severity, count: length})' - -# Group by category -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json | \ - jq 'group_by(.category) | map({category: .[0].category, count: length})' - -# Auto-remediable findings only (have a fix proposal) -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json | \ - jq '[.[] | select(.fix != null)] | length' - -# Per-repo summary as TSV (severity, category, repo, title) -atx ct findings list --source ${LOGICAL_SOURCE_NAME} --json | \ - jq -r '.[] | [.severity, .category, .repo, .title] | @tsv' - -# Filter to a specific analysis -atx ct findings list --analysis-id ${AID} --json | jq 'length' -``` - -These commands work from anywhere with `atx ct` CLI access (customer's laptop, the EC2 container via `sudo docker exec ${CONTAINER_NAME} ...`, or any other machine with the same backend access). Findings are server-state, not instance-state. - -**S3 artifacts** are uploaded by `/app/upload-ct-artifacts.sh` automatically when the wrapper completes. Analysis artifacts are written for any provider; remediation artifacts are only written for `--local` remediations. - -``` -s3://atx-ct-output-{account-id}//::/ - code.zip -- the working directory after the analysis or remediation completes, - including a result branch with auto-committed changes (e.g., - `atx-result-staging-` for analysis documentation, or the - remediation's branch for `--local` runs). The customer can `git log` - and `git diff` to review what the bot changed. `.git/` is preserved - for this reason. - Excludes node_modules/, .env*, *.pem, *.key, .aws/. - logs.zip -- cherry-picked debug logs (ATX CLI debug, error log, conversation - transcript, plan.json, validation_summary.md). -``` - -To download: - -```bash -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) - -# All artifacts for one analysis -aws s3 sync s3://atx-ct-output-${ACCOUNT_ID}/${AID}/ ./artifacts/ - -# Just one repo's reports -aws s3 cp s3://atx-ct-output-${ACCOUNT_ID}/${AID}/${SOURCE}::${REPO}/code.zip ./ -``` - -Surface findings to the user as the primary result. Reference S3 artifacts only when the user asks for raw reports/logs. - -## Cancellation - -To cancel an in-flight job: - -```bash -JOB_ID="" - -# Read the AID/RID and the wrapper PID -AID=$(ssm_run "cat /tmp/atxct-${JOB_ID}.aid 2>/dev/null") -WRAPPER_PID=$(ssm_run "pgrep -f 'atxct-${JOB_ID}.sh'") - -# Kill the wrapper (stops the polling loop on the instance) -ssm_run "sudo kill -TERM $WRAPPER_PID 2>/dev/null" - -# Cancel the in-flight CT analysis (server-side) -[ -n "$AID" ] && ssm_run "sudo docker exec ${CONTAINER_NAME} atx ct analysis cancel --id $AID" - -# Clean up this job's temp files -ssm_run "rm -f /tmp/atxct-${JOB_ID}.*" -``` - -Findings already persisted (from earlier in the analysis) survive the cancel. The upload step does NOT run if the wrapper is killed -- recover via: - -```bash -ssm_run "sudo docker exec ${CONTAINER_NAME} /app/upload-ct-artifacts.sh $AID atx-ct-output-${ACCOUNT_ID}" -``` - -## Use Existing Instance (no CFN) - -Reached when **Step 0 returned no stack** and the customer chose path 1 (existing EC2 instance launched outside CFN). Steps C.1–C.7 verify the instance, bootstrap the atx-ct container, and resume at Step 6. - -**At most ONE admin handoff** is needed in this path -- Step C.0 pre-flights both the instance tag and the role permissions in read-only mode, then emits a single combined admin handoff if either is missing. After admin runs that one bundle, the executor proceeds through C.2–C.6 (Docker install, image pull, container start) without further interruption. If both the tag and role were already in place, the handoff is skipped entirely. - -#### Step C.0: Pre-flight + Combined Admin Handoff - -Capture the basics first: - -```bash -INSTANCE_ID="" -REGION="" -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) - -# Discover the instance's IAM role (executor's iam:GetInstanceProfile is account-scoped, so this works). -PROFILE_ARN=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --region $REGION \ - --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text) -PROFILE_NAME=$(echo "$PROFILE_ARN" | awk -F/ '{print $NF}') -INSTANCE_ROLE_NAME=$(aws iam get-instance-profile --instance-profile-name "$PROFILE_NAME" \ - --query 'InstanceProfile.Roles[0].RoleName' --output text) - -if [ -z "$INSTANCE_ROLE_NAME" ] || [ "$INSTANCE_ROLE_NAME" = "None" ]; then - echo "ERROR: instance has no IAM instance profile attached. The customer's admin must" - echo " create one and attach it before this skill can proceed. This is a much larger" - echo " handoff than tagging or policy-setting; bail out and ask the customer." - exit 1 -fi -``` - -**Read-only pre-flight checks** (executor creds suffice for all of these): - -```bash -# Check 1: Is the instance tagged atx-remote-infra=true? -TAG_VALUE=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --region $REGION \ - --query 'Reservations[0].Instances[0].Tags[?Key==`atx-remote-infra`].Value | [0]' --output text) -TAG_OK="no"; [ "$TAG_VALUE" = "true" ] && TAG_OK="yes" - -# Check 2: Does the role have transform-custom:* (the marker action that proves the -# full Instance Role IAM spec was applied)? Inspect inline policies. -ROLE_POLICIES=$(aws iam list-role-policies --role-name "$INSTANCE_ROLE_NAME" --query 'PolicyNames' --output text) -ROLE_OK="no" -for POLICY in $ROLE_POLICIES; do - if aws iam get-role-policy --role-name "$INSTANCE_ROLE_NAME" --policy-name "$POLICY" \ - --query 'PolicyDocument.Statement[].Action' --output json 2>/dev/null \ - | grep -q '"transform-custom:\*"'; then - ROLE_OK="yes"; break - fi -done - -# Check 3: Is AmazonSSMManagedInstanceCore attached? -SSM_OK="no" -aws iam list-attached-role-policies --role-name "$INSTANCE_ROLE_NAME" \ - --query 'AttachedPolicies[?PolicyName==`AmazonSSMManagedInstanceCore`].PolicyName' \ - --output text 2>/dev/null | grep -q AmazonSSMManagedInstanceCore && SSM_OK="yes" - -echo "Tag atx-remote-infra=true: $TAG_OK" -echo "Role has transform-custom:* etc: $ROLE_OK" -echo "AmazonSSMManagedInstanceCore attached: $SSM_OK" -``` - -**If all three are `yes`**: skip the handoff and proceed directly to Step C.1. - -**If any is `no`**: emit ONE combined admin handoff covering all the missing pieces. Tell the customer: - -> **Admin handoff -- one-time setup for `$INSTANCE_ID`** -> -> This bundle (a) tags the instance so the executor can SSM into it, (b) attaches the full ATX Control Tower instance role policy, and (c) ensures `AmazonSSMManagedInstanceCore` is attached. All three are admin-only operations (`ec2:CreateTags`, `iam:PutRolePolicy`, `iam:AttachRolePolicy`). Run with admin / role-creation permissions: -> -> ```bash -> INSTANCE_ID="$INSTANCE_ID" -> INSTANCE_ROLE_NAME="$INSTANCE_ROLE_NAME" -> ACCOUNT_ID="$ACCOUNT_ID" -> REGION="$REGION" -> -> # 1. Tag the instance so executor's tag-conditioned SSM permissions activate -> aws ec2 create-tags \ -> --resources "$INSTANCE_ID" \ -> --tags Key=atx-remote-infra,Value=true \ -> --region "$REGION" -> -> # 2. Attach the full ATX Control Tower instance role policy (the FULL spec from -> # the Instance Role IAM section -- do NOT subset by analysis type). -> aws iam put-role-policy \ -> --role-name "$INSTANCE_ROLE_NAME" \ -> --policy-name atx-transform-access \ -> --policy-document '{ -> "Version": "2012-10-17", -> "Statement": [ -> {"Effect": "Allow", "Action": "transform-custom:*", "Resource": "*"}, -> {"Effect": "Allow", -> "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket", "s3:DeleteObject"], -> "Resource": ["arn:aws:s3:::atx-source-code-'$ACCOUNT_ID'", -> "arn:aws:s3:::atx-source-code-'$ACCOUNT_ID'/*", -> "arn:aws:s3:::atx-ct-output-'$ACCOUNT_ID'", -> "arn:aws:s3:::atx-ct-output-'$ACCOUNT_ID'/*"]}, -> {"Effect": "Allow", "Action": "secretsmanager:GetSecretValue", -> "Resource": "arn:aws:secretsmanager:*:'$ACCOUNT_ID':secret:atx/*"}, -> {"Effect": "Allow", -> "Action": ["securityagent:ListAgentSpaces", -> "securityagent:CreateCodeReview", "securityagent:StartCodeReviewJob", -> "securityagent:ListCodeReviewJobsForCodeReview", -> "securityagent:ListFindings", "securityagent:BatchGetFindings", -> "securityagent:StartCodeRemediation"], -> "Resource": "arn:aws:securityagent:*:*:agent-space*", -> "Condition": {"StringEquals": {"aws:ResourceAccount": "'$ACCOUNT_ID'"}}}, -> {"Effect": "Allow", -> "Action": ["s3:GetObject", "s3:ListBucket"], -> "Resource": ["arn:aws:s3:::kct-security-agent-*", -> "arn:aws:s3:::kct-security-agent-*/*"]}, -> {"Effect": "Allow", "Action": "s3:PutObject", -> "Resource": "arn:aws:s3:::kct-security-agent-*/security-scans/*"}, -> {"Effect": "Allow", "Action": "iam:PassRole", -> "Resource": "arn:aws:iam::'$ACCOUNT_ID':role/security-agent-*", -> "Condition": {"StringEquals": {"iam:PassedToService": "securityagent.amazonaws.com"}}}, -> {"Effect": "Allow", "Action": ["kms:GenerateDataKey", "kms:Decrypt", "kms:Encrypt", "kms:DescribeKey"], -> "Resource": "arn:aws:kms:*:'$ACCOUNT_ID':key/*", -> "Condition": {"StringLike": {"kms:ViaService": "s3.*.amazonaws.com"}}} -> ] -> }' -> -> # 3. Ensure the SSM agent's managed policy is attached (idempotent). -> aws iam attach-role-policy \ -> --role-name "$INSTANCE_ROLE_NAME" \ -> --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore 2>&1 | grep -v "EntityAlreadyExists" || true -> ``` -> -> When this finishes, come back to the conversation. I'll re-run the pre-flight checks and continue from Step C.1. - -The handoff bundles all admin operations the existing-instance path needs -- there are no other admin handoffs later in the flow. - -The agent MUST omit any block from the handoff that's already correct. Example: if `TAG_OK=yes` but `ROLE_OK=no`, drop block #1, keep #2 and #3. The point is to print exactly what's needed, not the full template every time. - -#### Step C.1: Customer provides WorkerCount - -`INSTANCE_ID`, `REGION`, and `ACCOUNT_ID` were already captured in Step C.0. The remaining input the customer chooses is WorkerCount: - -```bash -# WorkerCount: how many parallel atx-ct containers to run on this instance. -# Default 1 (single container). For multi-repo parallelism, customer chooses -# N sized to their instance's vCPU/RAM. -WORKER_COUNT="${WORKER_COUNT:-1}" -``` - -ALWAYS ask before proceeding: "How many parallel containers do you want on this instance? Each worker uses ~3-4 vCPU and ~4-8 GB RAM. Default 1." Ask explicitly even when the customer is only running a single analysis, so they know multi-worker is an option. Sizing guidance based on instance type: - -- t3.medium / t3.large (2 vCPU): 1 worker -- m5.xlarge / m5.2xlarge (4-8 vCPU): 2-4 workers -- m5.4xlarge or larger (16+ vCPU): up to 5 workers (cap; for more parallelism use the Batch path) - -For monorepos or memory-heavy analyses, scale down. The skill does NOT auto-detect the instance's capacity for an existing instance; the customer is responsible for sizing. - -#### Step C.2: Verify SSM is online - -```bash -PING=$(aws ssm describe-instance-information \ - --filters "Key=InstanceIds,Values=$INSTANCE_ID" \ - --query 'InstanceInformationList[0].PingStatus' --output text --region $REGION) -[ "$PING" = "Online" ] || { echo "ERROR: SSM agent not Online (got: ${PING:-no response})"; exit 1; } -``` - -If not Online, the instance is missing `AmazonSSMManagedInstanceCore` on its IAM role, or the SSM agent is not running. Customer must fix this before proceeding. - -Define the SSM helpers used by the rest of the steps: - -```bash -ssm_submit() { - aws ssm send-command --region $REGION \ - --instance-ids "$INSTANCE_ID" --document-name AWS-RunShellScript \ - --parameters "commands=[\"$1\"]" --query 'Command.CommandId' --output text -} - -ssm_run() { - local cmd="$1" - local CMD_ID=$(aws ssm send-command --region $REGION \ - --instance-ids "$INSTANCE_ID" --document-name AWS-RunShellScript \ - --parameters "commands=[\"$cmd\"]" --query 'Command.CommandId' --output text) - aws ssm wait command-executed --command-id "$CMD_ID" --instance-id "$INSTANCE_ID" --region $REGION 2>/dev/null || true - aws ssm get-command-invocation --region $REGION \ - --command-id "$CMD_ID" --instance-id "$INSTANCE_ID" \ - --query 'StandardOutputContent' --output text -} -``` - -#### Step C.3: Verify Docker is installed (install if missing) - -```bash -DOCKER_STATUS=$(ssm_run "command -v docker >/dev/null 2>&1 && echo INSTALLED || echo MISSING") -``` - -If `MISSING`, install: - -```bash -ssm_run "if command -v dnf >/dev/null; then sudo dnf install -y docker; \ - elif command -v apt-get >/dev/null; then sudo apt-get update -qq && sudo apt-get install -y docker.io; \ - elif command -v yum >/dev/null; then sudo yum install -y docker; \ - else echo 'ERROR: unsupported package manager' >&2; exit 1; fi && \ - sudo systemctl start docker && sudo systemctl enable docker" -``` - -Verify with `ssm_run "docker --version"`. If it fails, ask the customer to install Docker manually and re-try. - -#### Step C.4: Pull the public docker image - -Reachability check (any HTTP response code means reachable; the public ECR API legitimately returns 401 for anonymous requests): - -```bash -HTTP_CODE=$(ssm_run "curl -sS --max-time 10 -o /dev/null -w '%{http_code}' https://public.ecr.aws/v2/") -``` - -Expected: `200` or `401`. If `000`, the instance has no path to public.ecr.aws. Mitigation: customer adds NAT, OR mirrors the image to ECR Private and overrides `ATX_IMAGE_URI` in Step C.5 below. - -Pull: - -```bash -ssm_run "sudo docker pull public.ecr.aws/d9h8z6l7/aws-transform:latest" -``` - -If the pull fails: typical causes are network egress, insufficient disk space, or a private-registry override needed. - -#### Step C.5: Launch the atx-ct container(s) - -If `WORKER_COUNT=1`, launch a single container named `atx-ct` (matches CFN single-worker naming). If `WORKER_COUNT>1`, launch `atx-ct-1`, `atx-ct-2`, ..., `atx-ct-N`. Each container runs `atx ct server` as the foreground process; this mirrors the CFN UserData pattern: override the image's job-runner entrypoint with bash and run the server (which keeps the container alive). The container image must already contain `atx ct` -- there is no runtime install step. - -```bash -if [ "$WORKER_COUNT" -eq 1 ]; then - CONTAINERS="atx-ct" -else - CONTAINERS=$(seq -f "atx-ct-%g" 1 $WORKER_COUNT) -fi - -for name in $CONTAINERS; do - # Cap each container's memory so a runaway analysis can only OOM its own - # container, never the host (which would OOM-kill the SSM agent and sever - # access). The memory math runs on the REMOTE host (\$-escaped) because an - # existing instance's RAM is not known laptop-side; ~4 GB is reserved for the - # OS/Docker/SSM agent and the remainder split across the workers. - # WORKER_COUNT is the authoritative worker count (Step 2 reads it from the - # stack/instance). Resolve it on the LAPTOP with a :-1 fallback so a re-paste in - # a fresh shell where it is unset degrades to 1 instead of rendering "/ ))" -- - # a bash arithmetic syntax error that would leave the container unlaunched with - # the error buried in SSM StandardErrorContent. (A bare "$WORKER_COUNT" had no - # such guard.) The fallback resolves laptop-side, so the remote host divides by - # the real count, never a blind 1 that would over-commit a multi-worker host. - ssm_run "WC=${WORKER_COUNT:-1}; sudo docker rm -f $name 2>/dev/null; \ - MEM_TOTAL_MB=\$(( \$(awk '/MemTotal/{print \$2}' /proc/meminfo) / 1024 )); \ - MEM_PER_WORKER_MB=\$(( (\$MEM_TOTAL_MB - 4096) / \$WC )); \ - [ \$MEM_PER_WORKER_MB -lt 2048 ] && MEM_PER_WORKER_MB=2048; \ - sudo docker run -d --name $name --restart on-failure:3 \ - --memory=\${MEM_PER_WORKER_MB}m --memory-swap=\${MEM_PER_WORKER_MB}m \ - --entrypoint /bin/bash \ - -e CT_OUTPUT_BUCKET=atx-ct-output-${ACCOUNT_ID} \ - -e AWS_REGION=${REGION} \ - public.ecr.aws/d9h8z6l7/aws-transform:latest \ - -c 'mkdir -p /home/atxuser/.atxct/sources /home/atxuser/.atxct/shared && \ - source ~/.bashrc && atx ct server'" -done -``` - -Multi-worker uses bridge networking (no `--net=host`) so each container has its own network namespace. Launches happen sequentially via SSM, so total launch time scales with `WORKER_COUNT`. - -#### Step C.6: Wait for all container(s) healthy - -```bash -for name in $CONTAINERS; do - for i in $(seq 1 18); do - STATUS=$(ssm_run "sudo docker ps --filter 'name=^${name}$' --format '{{.Status}}'") - echo "$STATUS" | grep -q '(healthy)' && { echo "$name healthy after $((i*5))s"; break; } - sleep 5 - done -done -``` - -If any container fails to reach healthy after 90s, inspect that specific container's logs: `ssm_run "sudo docker logs 2>&1 | tail -30"`. Common causes are install network failure (one container's network namespace differs) and per-worker port conflicts (rare with bridge networking). - -Verify the CT CLI in one container (all containers share the same image, so verifying one is enough): - -```bash -FIRST_CONTAINER=$(echo $CONTAINERS | awk '{print $1}') -ssm_run "sudo docker exec $FIRST_CONTAINER bash -c 'source ~/.bashrc && atx --version'" -``` - -#### Step C.7: Verify Instance Role Permissions (sanity check) - -Step C.0's pre-flight should have caught any missing role permissions before we got here, but the `atx ct server` startup runs real backend calls (resume remediations, list sources) that exercise permissions in ways the static check can't fully simulate. This step is a runtime sanity check. - -Check the first container's startup log for AccessDenied errors: - -```bash -FIRST_CONTAINER=$(echo $CONTAINERS | awk '{print $1}') -ssm_run "sudo docker logs $FIRST_CONTAINER 2>&1 | grep -iE 'AccessDenied|not authorized' | head -5" -``` - -If the grep returns empty: server initialized cleanly. Proceed to **Step 7 (Confirm and Submit)**. - -If the grep returns matches: this means the role's policy is somehow incomplete relative to the [Instance Role IAM](#instance-role-iam) section, despite Step C.0 saying it had `transform-custom:*`. Most likely cause: a custom inline policy that has only some of the required statements. Re-emit the **same combined admin handoff from Step C.0** (`aws iam put-role-policy --policy-name atx-transform-access ...` with the FULL spec from the Instance Role IAM section -- do NOT subset it). This will overwrite the partial policy with the complete one. - -## Instance Role IAM - -The EC2 instance's IAM role (`atx-transform-role` from Step 4) needs: - -```json -{ - "Statement": [ - {"Effect": "Allow", "Action": "transform-custom:*", "Resource": "*"}, - {"Effect": "Allow", - "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket", "s3:DeleteObject"], - "Resource": ["arn:aws:s3:::atx-source-code-${ACCOUNT_ID}", - "arn:aws:s3:::atx-source-code-${ACCOUNT_ID}/*", - "arn:aws:s3:::atx-ct-output-${ACCOUNT_ID}", - "arn:aws:s3:::atx-ct-output-${ACCOUNT_ID}/*"]}, - {"Effect": "Allow", "Action": "secretsmanager:GetSecretValue", - "Resource": "arn:aws:secretsmanager:*:${ACCOUNT_ID}:secret:atx/*"}, - {"Effect": "Allow", - "Action": ["securityagent:ListAgentSpaces", - "securityagent:CreateCodeReview", "securityagent:StartCodeReviewJob", - "securityagent:ListCodeReviewJobsForCodeReview", - "securityagent:ListFindings", "securityagent:BatchGetFindings", - "securityagent:StartCodeRemediation"], - "Resource": "arn:aws:securityagent:*:*:agent-space*", - "Condition": {"StringEquals": {"aws:ResourceAccount": "${ACCOUNT_ID}"}}}, - {"Effect": "Allow", - "Action": ["s3:GetObject", "s3:ListBucket"], - "Resource": ["arn:aws:s3:::kct-security-agent-*", - "arn:aws:s3:::kct-security-agent-*/*"]}, - {"Effect": "Allow", "Action": "s3:PutObject", - "Resource": "arn:aws:s3:::kct-security-agent-*/security-scans/*"}, - {"Effect": "Allow", "Action": "iam:PassRole", - "Resource": "arn:aws:iam::${ACCOUNT_ID}:role/security-agent-*", - "Condition": {"StringEquals": {"iam:PassedToService": "securityagent.amazonaws.com"}}}, - {"Effect": "Allow", "Action": ["kms:GenerateDataKey", "kms:Decrypt", "kms:Encrypt", "kms:DescribeKey"], - "Resource": "arn:aws:kms:*:${ACCOUNT_ID}:key/*", - "Condition": {"StringLike": {"kms:ViaService": "s3.*.amazonaws.com"}}} - ] -} -``` - -Plus the AWS-managed policy `arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore` (attached separately) so the SSM agent can phone home. - -## Container Customization - -The pre-built `public.ecr.aws/d9h8z6l7/aws-transform:latest` image includes Java 8/11/17/21/25, Python 3.8-3.14, Node.js 16-24, Maven, Gradle, common build tools, AWS CLI v2, and AWS Transform CLI. CT CLI is installed at container start time via the curl install. - -For continuous modernization analyses, the pre-built image's defaults handle every runtime need. For custom TDs requiring a runtime not in the image (Rust, Go, .NET on Linux), follow [custom-remote-execution#custom-image-path](custom-remote-execution.md#custom-image-path-docker-required). - -## Runtime Version Switching - -For remediation runs that target a specific language version (e.g., Java 21, Python 3.13), pass the version as an environment variable on the `docker run` (Step 6): - -```bash -# Each ssm_run is a separate remote shell, so compute MEM_PER_WORKER_MB in the -# SAME command that launches the container (Step C.5 does the identical compute). -# WORKER_COUNT resolves laptop-side with a :-1 fallback (see Step C.5) so an unset -# value degrades to 1 instead of rendering an empty divisor "/ ))". -ssm_run "WC=${WORKER_COUNT:-1}; MEM_TOTAL_MB=\$(( \$(awk '/MemTotal/{print \$2}' /proc/meminfo) / 1024 )); \ - MEM_PER_WORKER_MB=\$(( (\$MEM_TOTAL_MB - 4096) / \$WC )); \ - [ \$MEM_PER_WORKER_MB -lt 2048 ] && MEM_PER_WORKER_MB=2048; \ - sudo docker run -d --name atx-ct --restart on-failure:3 \ - --memory=\${MEM_PER_WORKER_MB}m --memory-swap=\${MEM_PER_WORKER_MB}m ... \ - -e JAVA_VERSION=21 \ - -e PYTHON_VERSION=3.13 \ - -e NODE_VERSION=22 \ - $IMAGE -c '...'" -``` - -Available versions: -- **Java**: 8, 11, 17, 21, 25 (Amazon Corretto) -- **Python**: 3.8-3.14 (accepts `3.13` or `13`) -- **Node.js**: 16, 18, 20, 22, 24 - -For analyses, runtime switching is generally not needed. - -## Limits - -- Per-job temp files (LOG, AID_FILE, STDOUT_LOG) keyed by JOB_ID let multiple concurrent jobs coexist -- Bedrock throughput is per-account -- running many parallel continuous modernization containers shares the quota; large workloads may throttle +Prerequisites for BYO: +- Instance is running and SSM-managed +- Docker installed with the AWS Transform — continuous modernization container running +- Instance tagged with `atx-remote-infra=true` (required by executor policy) +- Instance role has S3, Secrets Manager, and transform-custom access ## Error Handling | Error | Cause | Fix | |---|---|---| -| SSM agent not Online | Instance role missing `AmazonSSMManagedInstanceCore` or no outbound internet | Re-attach the managed policy; verify VPC has NAT or public IP | -| Container exits / restarts on launch (or stays `exited`) | `atx ct server` crashed, OR exited cleanly (exit 0) and was intentionally not recovered by `--restart on-failure:3`, OR the image failed to pull | First triage with `ssm_run "sudo docker inspect --format '{{.State.Status}} ExitCode={{.State.ExitCode}} RestartCount={{.RestartCount}} OOMKilled={{.State.OOMKilled}}'"`. If `OOMKilled=true`, see the OOM row below. If `ExitCode=0`, the server exited cleanly and was correctly NOT auto-restarted -- investigate why it shut down (`docker logs`). If `ExitCode!=0` (often `RestartCount=3`), check `docker logs ` for a crash: image-pull failure (verify NAT/public IP to `public.ecr.aws`), port conflict on 8081, or missing env/role perms. If UserData itself failed, the stack is in `ROLLBACK_COMPLETE`; check `aws cloudformation describe-stack-events --stack-name $STACK_NAME` | -| Container OOM-killed (gone after a few restarts; jobs fail with "container not running") | The workload exceeded the container's per-worker `--memory` cap (e.g. a heavy parallel-clone analysis); `--restart on-failure:3` retried 3x then left it down | `ssm_run "sudo docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}} {{.State.Status}}'"`. If `OOMKilled=true`, the cap was exceeded -- give each worker more RAM: raise `InstanceType` and/or lower `WorkerCount` (each gets `(instance RAM - 4 GB) / WorkerCount`), then re-provision. **If `OOMKilled=false` but `RestartCount=3 Status=exited`, this is NOT an OOM** -- the server crashed for another reason; fall through to "Container exits / restarts on launch" above. Restart a downed container with `ssm_run "sudo docker start "` | -| `atx ct analysis run` clone fails | PAT expired or repo private to a different account | Verify customer's PAT has access; re-stage source config (Step 6) | -| Findings missing after analysis | Server crashed before persisting | Check `tail /tmp/atxct-.log`; recover via `analysis get --id $AID` | -| Artifacts missing from S3 | Wrapper killed before upload step | Re-run upload manually (see Cancellation section) | -| Polling never completes | atx ct server hung or container down | `ssm_run "sudo docker ps"` and `ssm_run "sudo docker logs ${CONTAINER_NAME} \| tail"` to diagnose | -| Container starts but all AWS API calls return "credentials not available" or fail to reach IMDS | Bridge networking + IMDSv2 hop limit = 1 (default). Token TTL expires before reaching the container's network namespace. | On an existing instance, the customer can run `aws ec2 modify-instance-metadata-options --instance-id --http-put-response-hop-limit 2 --region `. We do NOT modify metadata options on customer instances automatically; it's a side-effect on resources they own. The CFN-managed flow does not hit this in practice with current Docker bridge defaults. | -| Status-check ssm_run calls hang during fan-out | Older fan-out submissions kept SSM agent worker slots occupied until each wrapper exited (CommandWorkersLimit default 5). Mitigated by submitting all workers via a single SSM command and using `( ( bash X & ) & )` double-fork to orphan each wrapper to init. SSM marks the launch Success immediately. | If you still observe queueing, list in-flight commands with `aws ssm list-commands --instance-id --query 'Commands[?Status==\`InProgress\`].CommandId'` and cancel orphaned ones via `aws ssm cancel-command --command-id `. Read wrapper progress through a single batched `ssm_run` reading `/tmp/atxct-fan-w*-*.{log,aids,rids}` rather than many small calls. | -| `atx ct analysis run` hangs cloning from internal/self-hosted git host | Subnet has 0.0.0.0/0 egress (Step 5b validation passed) but no route to the customer's internal git host. VPN / Direct Connect / VPC peering missing or filtered. | The skill cannot auto-verify routes to corporate-network git hosts. Confirm with the network team that the subnet's route table includes a path to the git host CIDR. Test from any instance in the same subnet: `nslookup ` then `curl -v https:///`. | -| Stack create fails with "no NAT/route" or container UserData times out after Step 5b validation passed | Subnet's 0.0.0.0/0 default route was removed (or the subnet was switched to a different route table) between Step 5b validation and stack deploy. | Re-run Step 5b validation on the current state of the route table. The check uses `ec2:DescribeRouteTables`; if the network team is making concurrent changes, run validation immediately before the admin handoff. | - -## Pricing - -Direct customer to: -- EC2 pricing: https://aws.amazon.com/ec2/pricing/ -- AWS Transform agent minutes: https://aws.amazon.com/transform/pricing/ - -Do NOT quote specific dollar amounts or time estimates. - -## Cleanup - -**Never delete the stack or stop/terminate the instance without explicit customer confirmation.** `delete-stack` is destructive -- it removes the instance, IAM role, and security group, and any in-flight analyses on the instance will be terminated. Even if the customer said "I'm done" earlier in the conversation, ask again before issuing the delete. - -When the customer indicates they're finished, prompt with options and **wait for explicit confirmation** before running any of the commands below: - -> Your EC2 stack `${STACK_NAME}` is still running and incurring charges. What would you like to do? -> 1. **Delete the stack** (admin handoff) -- removes the instance, IAM role, and security group atomically. Stops all charges. Your S3 buckets and Secrets Manager entries persist (so analysis history survives). Requires admin creds -- I'll print the command for someone with `cloudformation:DeleteStack` + `iam:Delete*` to run. -> 2. **Stop the instance** -- keeps the stack but stops the EC2. No compute charges, small EBS storage charge. Container needs to re-initialize after restart. I CAN run this with executor creds (`ec2:StopInstances` on the tagged instance). -> 3. **Keep running** -- instance stays up. Hourly EC2 charges continue. Useful if another analysis is coming. -> -> Reply with 1, 2, or 3. - -Do NOT run delete-stack proactively. Do NOT assume option 1 because the customer's last analysis finished. The customer must explicitly choose. - -**Option 1: Delete the entire stack -- admin handoff** (only after explicit confirmation) - -The agent does NOT run `delete-stack` itself. Deleting the stack tears down the IAM role, instance profile, and security group, which requires `iam:DeleteRole`, `iam:DeleteRolePolicy`, `iam:DeleteInstanceProfile`, and `cloudformation:DeleteStack` -- all of these live in the admin policy, not the executor policy. The agent prints the command and the customer's admin runs it: - -```bash -# Admin runs: -aws cloudformation delete-stack --stack-name "$STACK_NAME" --region $REGION -aws cloudformation wait stack-delete-complete --stack-name "$STACK_NAME" --region $REGION -``` - -This removes everything in the stack atomically. If anything fails to delete, the stack moves to `DELETE_FAILED` and the customer can inspect what's left (executor creds CAN read this): - -```bash -aws cloudformation describe-stack-events --stack-name "$STACK_NAME" --region $REGION \ - --query 'StackEvents[?ResourceStatus==`DELETE_FAILED`]' -``` - -**Option 2: Stop the instance** (only after explicit confirmation) - -```bash -aws ec2 stop-instances --instance-ids $INSTANCE_ID --region $REGION -``` - -The stack stays in `CREATE_COMPLETE`. Customer can `aws ec2 start-instances` later to bring it back. Note: container needs time to become healthy after start (atx ct server has to re-initialize). - -**Option 3: Keep running** - -No-op. Customer continues to pay EC2 hourly charges. Useful when expecting another analysis soon. - -**What persists across delete-stack:** - -| Resource | Persists? | Why | -|---|---|---| -| S3 buckets (`atx-source-code-${ACCOUNT_ID}`, `atx-ct-output-${ACCOUNT_ID}`) | ✅ Yes -- managed outside the stack | Customer's analysis results survive stack lifecycles | -| Secrets Manager (`atx/github-token`, etc.) | ✅ Yes -- managed outside the stack | Customer's tokens persist for next run | -| Customer-supplied VPC, subnet, security group (when reused) | ✅ Yes -- never owned by the skill | Customer or their network team owns these | -| Stack-managed resources (instance, IAM role, profile, SG when stack-created) | ❌ No -- deleted with stack | Recreated on next `create-stack` | +| `Stack not deployed` | No EC2 infra | Run `atx ct remote provision --mode ec2 ...` | +| `EC2 stack exposed no InstanceId` | Stack exists but instance missing | Check CFN stack events; may need teardown + reprovision | +| `workerCount must be in [1, 5]` | Workers out of range | Use 1-5, or switch to Batch for more parallelism | +| `Stack name must start with "atx-runner"` | Invalid prefix | Use `--stack-name atx-runner-` or `--suffix ` | +| `--existing-instance is mutually exclusive with --stack-name / --tags` | Both specified | Use one or the other | +| `Existing-instance prerequisite check failed` | Instance not SSM-managed or missing tag | Verify SSM agent, add `atx-remote-infra=true` tag | +| `Token invalid for source` | Expired/revoked SCM token | Run `atx ct remote credentials --source --token --ack` | +| `No deployed stack found for tags` | `--tags` matched no deployed stack | Verify tags with `atx ct remote detect --mode ec2 --tags `, or target by `--stack-name` | +| `Group not found` | Invalid or expired group ID | Check `atx ct remote status --group ` | -**Resources the skill MUST NEVER delete, under any circumstances:** - -- **Customer-supplied VPCs, subnets, route tables, NAT gateways, internet gateways, transit gateways, VPC peering connections** -- these are network infrastructure owned by the customer or their network team. The skill never created them (we don't have permission to), and we never delete them. If a customer asks "clean up everything including the VPC," refuse and explain that VPC lifecycle is a network-team responsibility. -- **Customer-supplied security groups** (when the customer chose "reuse" at Step 5b's SG ask) -- these existed before our stack and persist after it. Only stack-created SGs (when the customer typed `new`) get cleaned up via `delete-stack`. -- **S3 buckets** (`atx-source-code-*`, `atx-ct-output-*`) -- these hold customer analysis history and are intentionally outside the stack to survive stack delete-and-recreate. The skill never empties them or removes them. Lifecycle policies auto-expire objects (7 days for source bundles, 30 days for output artifacts), so residual storage cost converges to zero without intervention. -- **Secrets** (`atx/github-token`, `atx/gitlab-token`, etc.) -- these hold customer credentials and persist across stacks. Customers may want to keep them for future analyses on a fresh stack. The skill never deletes them. +## Limits -If the customer asks for "complete teardown" or "delete everything," the agent's response is: "I can run `delete-stack` which removes the runner instance, IAM role, instance profile, and any stack-created security group. Your S3 buckets, secrets, and customer-owned network resources (VPC/subnets/SGs you supplied) stay -- those aren't owned by this skill. If you want to remove those too, please do it directly in the AWS console or CLI; I won't run those commands because they're destructive of data and infrastructure outside the skill's scope." +- Max 5 workers per instance +- EC2 stack names must start with `atx-runner` +- WorkerCount fixed at provision time (change requires redeploy) +- `--execute` flag gates all destructive operations +- Instance types: m5.large to m5.12xlarge diff --git a/aws-transform/steering/workload-continuous-modernization-guide.md b/aws-transform/steering/workload-continuous-modernization-guide.md index 9e0c89f..76b2e79 100644 --- a/aws-transform/steering/workload-continuous-modernization-guide.md +++ b/aws-transform/steering/workload-continuous-modernization-guide.md @@ -31,15 +31,7 @@ This guide handles continuous modernization onboarding only. For routing across ## On Start — Detect State (Prereq check /setup skill) -ALWAYS begin by running: - -```bash -atx ct status --health -``` - -DO NOT share this command with the customer in your response. Only run it to check the current status. This is just a table guide for you to know which step to go to based on the current state. - -This returns sources, repo counts, analyses, findings, and remediations. Use these to determine where the user is: +Route to the right step based on what's already configured (sources, repo counts, analyses, findings, remediations). You don't need to run a status check up front — infer from the conversation and the user's request, and query `atx ct` only when you need a specific value: | Condition | Start at | | -------------------------------------------------------- | ----------------------------------------- | @@ -221,7 +213,7 @@ When all steps are done, show a recap of what was accomplished in this session. 3. **Offer defaults.** Have a recommended option. Make it easy to proceed. 4. **Show commands.** Always display the `atx ct` command you're running so the user learns the CLI. 5. **Handle errors plainly.** Say what failed, offer a fix or alternative: - - Connection error → "The AWS Transform - continuous modernization server isn't running. Starting it now: `atx ct server`" + - Startup/credentials error → "I couldn't reach the AWS Transform backend. Let's check your AWS credentials (refresh them if they've expired) and that you picked a supported region." - Invalid token → "That token didn't work. Make sure it has `repo` scope." - No repos found → "No repos found in that source. Double-check the org name or path." 6. **Let them skip.** "skip", "later", "not now" — move on. diff --git a/aws-transform/steering/workload-continuous-modernization-remediation.md b/aws-transform/steering/workload-continuous-modernization-remediation.md index 327b510..f55b132 100644 --- a/aws-transform/steering/workload-continuous-modernization-remediation.md +++ b/aws-transform/steering/workload-continuous-modernization-remediation.md @@ -40,7 +40,7 @@ atx ct remediation create --transformation-name --repo :: --repo :: -g "additionalPlanContext=Upgrade to Node.js 22" --telemetry "agent=,executionMode=local" -# Create with local execution (runs ATX transform on the server instead of GitHub Actions) +# Create with local execution (runs the ATX transform on the server host instead of on managed remote compute) atx ct remediation create --ids --name "Fix name" --local --telemetry "agent=,executionMode=local" # List all @@ -56,17 +56,17 @@ atx ct remediation retry --id atx ct remediation delete --id ``` -## Repository limit per request (max 250) +## Repository limit per request (max 100) -A single `atx ct remediation create` can be associated with at most **250 repositories**. Before creating a remediation that spans many repos, check how many distinct repositories the target findings cover. +A single `atx ct remediation create` can be associated with at most **100 repositories**. Before creating a remediation that spans many repos, check how many distinct repositories the target findings cover. -If the remediation would span more than 250 repositories, split it into multiple `remediation create` requests, each covering findings from at most 250 repos, and tell the user you are breaking it up because of the 250-repo-per-request limit. Never create a single remediation associated with more than 250 repositories — it will be rejected. Example: findings across 300 repos → two remediations (one for 250 repos, one for the remaining 50). +If the remediation would span more than 100 repositories, split it into multiple `remediation create` requests, each covering findings from at most 100 repos, and tell the user you are breaking it up because of the 100-repo-per-request limit. Never create a single remediation associated with more than 100 repositories — it will be rejected. Example: findings across 300 repos → three remediations (100 repos each). ## Running long remediations (--wait, background, logs) `atx ct remediation create` returns immediately by default with a remediation ID. With `--wait` it blocks until the remediation completes — and applying transforms across repos can take a long time. Prefer `--wait` so you can act on the result in the same step. -**`--wait` is version-gated** — it exists only in newer CLI versions. Confirm support via `atx ct remediation create --help` (or `atx ct --version`) before relying on it; if it isn't listed, run without `--wait` and do not invent the flag. If a run fails with an unknown-option error for `--wait`, re-run without it. +**`--wait` is version-gated** — it exists only in newer CLI versions. Confirm support via `atx ct remediation create --help` (or `atx ct --version`) before relying on it; if it isn't listed, run without `--wait` and do not invent the flag. Only re-run without `--wait` if the command fails with an error that explicitly names `--wait` as an unrecognized option AND returned no remediation ID — do not treat auth, `INVALID_INPUT`, or repo-cap failures as a missing-flag error, and never blindly re-run a `create` that may have already submitted. **Run long jobs in the background and monitor a log.** Start long-running remediations with `&`, redirect output to a log file, and monitor it: @@ -75,6 +75,8 @@ atx ct remediation create --ids --name "Fix name" --wait --telemetry " tail -f /tmp/atx-remediation.log ``` +The redirect captures the command's diagnostics — the in-process CLI writes logs to STDERR, which `2>&1` folds into the log file. Only warnings and errors are logged by default; when troubleshooting, prefix the command with `ATXCT_LOG_LEVEL=debug` for verbose output. + Tell the user where the log is and how to check progress. ## Listing remediations (pagination) @@ -161,10 +163,27 @@ When the user asks to remediate with a custom transformation definition, or a fi ### `--local` flag (remediation create) -When `--local` is passed, the ATX transform runs directly on the server against a cloned copy of the repository instead of dispatching a GitHub Actions workflow. This is useful for: +When `--local` is passed, the ATX transform runs directly on the server host against a cloned copy of the repository instead of being dispatched to managed remote compute. This is useful for: + +- Faster feedback without waiting for a remote job to be scheduled +- Environments without access to managed remote compute +- Testing transforms locally before running them on remote compute + +The execution mode is persisted on the remediation record (`compute_mode = 'local'`), so subsequent `retry` and `resume` operations automatically honour the original intent without needing to re-specify the flag. + +### `--tags` flag (remediation create) + +Attaches IAM resource tags to the remediation at creation time. + +```bash +# Create remediation with tags (comma-separated key=value pairs) +atx ct remediation create --ids --name "Fix name" --tags team=alpha,env=prod --telemetry "agent=,executionMode=local" +``` + +**Behavior:** -- GitHub-sourced repos where you want faster feedback without waiting for CI -- Environments where GitHub Actions workflows are not configured or available -- Testing transforms locally before committing to a full workflow run +- `--tags key=value,key2=value2` accepts comma-separated pairs in a single flag (e.g. `--tags team=alpha,env=prod`). +- Tags are optional. If omitted, the remediation is untagged. +- If `~/.aws/atx/settings.json` defines `applyTags` (an array of tag maps), those defaults are applied automatically even without explicit `--tags`. An explicit `--tags` override merges **per key** over the settings defaults. -The execution mode is persisted on the remediation record (`compute_mode = 'local'`), so subsequent `retry` and `resume` operations automatically honour the original intent without needing to re-specify the flag. \ No newline at end of file +See the [source](workload-continuous-modernization-source.md) skill's Tags section for the full schema, merge semantics, and error behavior. diff --git a/aws-transform/steering/workload-continuous-modernization-reporting.md b/aws-transform/steering/workload-continuous-modernization-reporting.md index 5255c95..a27df56 100644 --- a/aws-transform/steering/workload-continuous-modernization-reporting.md +++ b/aws-transform/steering/workload-continuous-modernization-reporting.md @@ -10,13 +10,9 @@ Generate a single self-contained HTML report that walks through everything conti The report is a **static snapshot**: the HTML has all data baked in as JS consts, so it's portable (emailable, openable offline) and reflects the moment the report was generated. -## Prerequisites - -- Server running: `atx ct status --health` returns `healthy`. If not, use the `server` skill to start it. - ## Data sources -Populate the report from the live `atx ct` server. +Populate the report from live `atx ct` reads. ```bash atx ct source list --json diff --git a/aws-transform/steering/workload-continuous-modernization-routing.md b/aws-transform/steering/workload-continuous-modernization-routing.md index 20f87dd..b2e5c7c 100644 --- a/aws-transform/steering/workload-continuous-modernization-routing.md +++ b/aws-transform/steering/workload-continuous-modernization-routing.md @@ -153,10 +153,10 @@ After a Custom transformation completes successfully, present this message: | [workload-continuous-modernization-status.md](workload-continuous-modernization-status.md) | System overview and health check | | [workload-continuous-modernization-source.md](workload-continuous-modernization-source.md) | Manage source connections | | [workload-continuous-modernization-setup.md](workload-continuous-modernization-setup.md) | Infrastructure setup and configuration | -| [workload-continuous-modernization-server.md](workload-continuous-modernization-server.md) | Start, stop, or restart the AWS Transform - continuous modernization (continuous modernization) server | -| [workload-continuous-modernization-ec2-execution.md](workload-continuous-modernization-ec2-execution.md) | Run CT analysis/remediation on EC2 (new or existing instance) | -| [workload-continuous-modernization-batch-execution.md](workload-continuous-modernization-batch-execution.md) | Run CT analysis on AWS Batch (Fargate) — single job, AWS-managed compute | -| [workload-continuous-modernization-schedule.md](workload-continuous-modernization-schedule.md) | Schedule recurring analyses on an existing EC2 instance (EventBridge Scheduler + SSM) | +| [workload-continuous-modernization-troubleshooting.md](workload-continuous-modernization-troubleshooting.md) | Diagnose setup and command execution failures | +| [workload-continuous-modernization-ec2-execution.md](workload-continuous-modernization-ec2-execution.md) | Run remote analysis/remediation on EC2 via `atx ct remote` CLI commands | +| [workload-continuous-modernization-batch-execution.md](workload-continuous-modernization-batch-execution.md) | Run remote analysis/remediation on AWS Batch (Fargate) via `atx ct remote` CLI commands | +| [workload-continuous-modernization-schedule.md](workload-continuous-modernization-schedule.md) | Schedule recurring analysis/remediation via `atx ct schedule` CLI commands | | [workload-continuous-modernization-reporting.md](workload-continuous-modernization-reporting.md) | Generate an HTML report of continuous modernization analyses | | [workload-continuous-modernization-security-agent.md](workload-continuous-modernization-security-agent.md) | Security agent setup (admin) and runtime verification (executor) | diff --git a/aws-transform/steering/workload-continuous-modernization-schedule.md b/aws-transform/steering/workload-continuous-modernization-schedule.md index bc9d2c8..1de7f8a 100644 --- a/aws-transform/steering/workload-continuous-modernization-schedule.md +++ b/aws-transform/steering/workload-continuous-modernization-schedule.md @@ -117,7 +117,7 @@ Customer **MUST** already have an EC2 instance running with the `atx-ct` contain Specifically: 1. EC2 instance running with one or more atx-ct containers active (CFN-managed via `atx-runner` stack, or any other source; both supported) -2. Container has the CT server up (`docker exec ${CONTAINER_NAME} atx ct status --health` succeeds) +2. Container is up (`docker exec ${CONTAINER_NAME} atx ct status --health` succeeds) 3. Instance has `AmazonSSMManagedInstanceCore` attached (so SSM can target it) 4. Customer has previously registered the source via `atx ct source add` (so a manual `atx ct analysis run` would work) @@ -822,7 +822,7 @@ fi [ -z "\$RID" ] && { echo "=== \$(date) [ERROR] success but no RID extracted ===" >> \$LOG; exit 1; } echo "=== \$(date) [REMED] remediation \$RID started -- polling status ===" >> \$LOG -# Poll every 30s until terminal status (atx ct remediation create does not support --wait) +# Poll every 30s until terminal status STATUS="" while true; do STATUS=\$(sudo docker exec ${CONTAINER_NAME} bash -c "source /home/atxuser/.nvm/nvm.sh && nvm use 22 >/dev/null 2>&1 && export PATH=/home/atxuser/.local/bin:\\\$PATH && atx ct remediation status --id \$RID --json" 2>>\$LOG | jq -r .status 2>/dev/null) @@ -937,7 +937,7 @@ if [ "$PATH_TYPE" = "batch" ]; then # ───────────────────────────────────────────────────────────────── # Provider-specific preamble (sets up source registration, tokens) # ───────────────────────────────────────────────────────────────── - PREAMBLE_COMMON="atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22 ; mkdir -p /home/atxuser/.aws/atx/logs ; atx ct server > /home/atxuser/.aws/atx/logs/server.log 2>&1 & sleep 15" + PREAMBLE_COMMON="atx ct --version > /dev/null 2>&1 ; set -o pipefail && source /home/atxuser/.bashrc && export PATH=/home/atxuser/.local/bin:/usr/local/bin:/usr/bin:/bin && source /home/atxuser/.nvm/nvm.sh && nvm use 22" case "$PROVIDER" in github) diff --git a/aws-transform/steering/workload-continuous-modernization-security-agent.md b/aws-transform/steering/workload-continuous-modernization-security-agent.md index d007901..ce71788 100644 --- a/aws-transform/steering/workload-continuous-modernization-security-agent.md +++ b/aws-transform/steering/workload-continuous-modernization-security-agent.md @@ -44,10 +44,6 @@ echo "Installed: ${INSTALLED:-not found}, Latest: ${LATEST:-unknown}" curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash source ~/.bashrc -# Start the server if not running -atx ct server & -sleep 5 - # Deploy security agent infrastructure (creates IAM role, S3 bucket, CloudFormation stack) atx ct setup security-agent ``` @@ -258,7 +254,7 @@ aws s3 rm s3://atx-source-code-${ACCOUNT_ID}/temp/security_agent_config.json Once permissions are verified, proceed with the normal analysis flow using `--type security`. -The executor IAM policy required for runtime is documented in `AWSTransformSecurityAgentExecutorAccess.json` in the ATXControlTowerPolicies package. +The executor IAM policy required for runtime is documented in `AWSTransformSecurityAgentExecutorAccess.json` (included with this skill). --- diff --git a/aws-transform/steering/workload-continuous-modernization-server.md b/aws-transform/steering/workload-continuous-modernization-server.md deleted file mode 100644 index 98c2ff8..0000000 --- a/aws-transform/steering/workload-continuous-modernization-server.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -name: server -description: Start, stop, or restart the AWS Transform - continuous modernization (continuous modernization) server (`atx ct server`). ---- - -# Server - -## Supported Regions - -AWS Transform - continuous modernization is available in these regions only: - -| Region | Code | -|--------|------| -| US East (N. Virginia) | `us-east-1` | -| Europe (Frankfurt) | `eu-central-1` | -| Asia Pacific (Mumbai) | `ap-south-1` | -| Asia Pacific (Sydney) | `ap-southeast-2` | -| Asia Pacific (Tokyo) | `ap-northeast-1` | -| Europe (London) | `eu-west-2` | -| Asia Pacific (Seoul) | `ap-northeast-2` | -| Canada (Central) | `ca-central-1` | - -## Region Selection - -Before starting the server, ask the user which region they want to use if they haven't already specified one, (render this menu as plain numbered markdown text in your response and wait for the user to type a choice; do NOT route it through any structured choice/picker tool like `AskUserQuestion` in Claude Code, or any equivalent multi-select/option UI in other harnesses): - -> "Which AWS region do you want to use? AWS Transform - continuous modernization supports: us-east-1, eu-central-1, ap-south-1, ap-southeast-2, ap-northeast-1, eu-west-2, ap-northeast-2, ca-central-1." - -If the user provides a region not in the supported list, let them know it isn't supported and suggest `us-east-1` as the default: - -> "That region isn't supported by AWS Transform - continuous modernization. Would you like to use `us-east-1` (US East, N. Virginia) instead?" - -Once confirmed, set `ATX_REGION` to the chosen region. - -## Start - -```bash -AWS_REGION=$ATX_REGION atx ct server & -``` - -## Stop - -```bash -pkill -f "atx ct server" -``` - -## Restart - -```bash -pkill -f "atx ct server"; AWS_REGION=$ATX_REGION atx ct server & -``` - -## Check if running - -```bash -atx ct status --health -``` diff --git a/aws-transform/steering/workload-continuous-modernization-setup.md b/aws-transform/steering/workload-continuous-modernization-setup.md index 9220555..23c1207 100644 --- a/aws-transform/steering/workload-continuous-modernization-setup.md +++ b/aws-transform/steering/workload-continuous-modernization-setup.md @@ -11,36 +11,47 @@ description: Set up/configure/provision AWS Transform - continuous modernization ### Step 1: Install or update `atx ct` -Run this single command to check install status AND version in one shot: +First verify that the `atx` selected by the shell provides the continuous-modernization commands. Keep stderr visible so dispatch errors are not mistaken for a missing installation: ```bash -INSTALLED=$(atx ct --version 2>/dev/null | head -1) -LATEST=$(curl -fsSL "https://transform-cli.awsstatic.com/index.json" 2>/dev/null | grep -o '"latest"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"latest"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/') -echo "Installed: ${INSTALLED:-not found}, Latest: ${LATEST:-unknown}" +atx ct --version ``` -If `INSTALLED` is empty OR `LATEST` is newer than `INSTALLED` → reinstall: +Handle the result by failure type: -```bash -curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash -source ~/.bashrc # or ~/.zshrc -``` +- **`atx: command not found`** — install the AWS Transform CLI, then restart the shell or source its profile: + + ```bash + curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash + source ~/.bashrc # or ~/.zshrc + ``` + +- **`unknown command 'ct'`** — an executable named `atx` ran, but it does not provide the required command. Do not reinstall blindly or check AWS credentials/region. Follow [command-resolution troubleshooting](workload-continuous-modernization-troubleshooting.md#atx-ct-reports-unknown-command-ct) first. +- **Success** — compare the installed and latest versions: -If both are the same → `atx ct` is up to date, proceed to Step 2. + ```bash + INSTALLED=$(atx ct --version | head -1) + LATEST=$(curl -fsSL "https://transform-cli.awsstatic.com/index.json" 2>/dev/null | grep -o '"latest"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"latest"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/') + echo "Installed: ${INSTALLED:-not found}, Latest: ${LATEST:-unknown}" + ``` + + If `LATEST` is known and newer than `INSTALLED`, run the install command above, then restart the shell or source its profile. Verify: `atx ct --help` must show CT subcommands. -### Step 2: Start the server +### Step 2: Choose your region + +If the user hasn't already specified one, ask which AWS region to run in and store it as `ATX_REGION=`. Continuous modernization is only available in a subset of AWS regions; if the user picks an unsupported one, the command surfaces a region error. -The `atx ct` CLI requires a running server. Before any other command, start it: +**Prefix the chosen region inline on EVERY `atx ct` command** — `AWS_REGION=$ATX_REGION atx ct ...`, not just the first one. Each `atx ct` invocation resolves its own region and the shell does not persist env vars between separate command invocations, so the prefix must sit on the same command line as every invocation, including inside compound commands: ```bash -atx ct server & -sleep 5 -atx ct status --health -``` +# Correct — prefix sits on the atx ct segment +which atx && AWS_REGION=$ATX_REGION atx ct analysis get --id --json -If `atx ct status --health` returns a connection error, the server isn't running. Check `atx ct server` output for errors. +# Wrong — atx ct runs without the region (the env var only applied to the first command) +AWS_REGION=$ATX_REGION which atx && atx ct analysis get --id --json +``` After installation, restart your shell or run `source ~/.bashrc` (or `~/.zshrc`) to update PATH. @@ -61,6 +72,22 @@ atx ct setup security-agent --status atx ct setup security-agent --delete ``` +## Tag Defaults (optional) + +To have the CLI automatically apply tags to every resource it creates (sources, analyses, remediations, findings), create a settings file: + +**File:** `~/.aws/atx/settings.json` + +```json +{ + "applyTags": [ + { "team": "alpha", "env": "prod" } + ] +} +``` + +`applyTags` is an **array of tag maps**. When this file exists and contains a valid `applyTags` array, the CLI applies those tags on every create operation without requiring `--tags` on each command. Multiple maps merge left-to-right (last map wins on a duplicate key); an explicit `--tags` value merges per key over the result. Tags enable IAM tag-based access control (ABAC) for multi-team isolation. See the [source](workload-continuous-modernization-source.md) skill's Tags section for the full schema, merge semantics, and error behavior. + ## Behavior - If `atx ct` is not installed, install it using the curl command above before proceeding. diff --git a/aws-transform/steering/workload-continuous-modernization-source.md b/aws-transform/steering/workload-continuous-modernization-source.md index c5fb2b6..14ca8f2 100644 --- a/aws-transform/steering/workload-continuous-modernization-source.md +++ b/aws-transform/steering/workload-continuous-modernization-source.md @@ -5,10 +5,6 @@ description: Add/list/remove source connections (GitHub org, GitLab group/user, # Source -## Prerequisites - -Check if the server is running with `atx ct status --health`. If any command fails with a connection error, use the `server` skill to start the server. - ## Token handling **Never ask the user to paste or type a token into this chat.** Tokens entered into the chat are visible in the conversation transcript. @@ -38,30 +34,30 @@ Then show the specific scopes for their provider: - **Bitbucket:** `read:repository:bitbucket`, `read:account`, `write:repository:bitbucket`, `read:pullrequest:bitbucket`, `write:pullrequest:bitbucket`. ```bash -# Add a GitHub org +# Add a GitHub org (optional: --tags key=value,key2=value2) # The GitHub PAT requires the `repo` scope (classic token), or for fine-grained tokens: Read access to metadata (default), Read and Write access to code and pull requests. -read -s TOKEN && atx ct source add --name --provider github --org --token "$TOKEN"; unset TOKEN +read -s TOKEN && atx ct source add --name --provider github --org --token "$TOKEN" [--tags key=value,key2=value2]; unset TOKEN # Add a GitLab group or user (gitlab.com) # The GitLab PAT requires the `api` scope. -read -s TOKEN && atx ct source add --name --provider gitlab --org --token "$TOKEN"; unset TOKEN +read -s TOKEN && atx ct source add --name --provider gitlab --org --token "$TOKEN" [--tags key=value,key2=value2]; unset TOKEN # Add a GitLab group or user (self-hosted) # The GitLab PAT requires the `api` scope. -read -s TOKEN && atx ct source add --name --provider gitlab --org --token "$TOKEN" --url https://gitlab.example.com; unset TOKEN +read -s TOKEN && atx ct source add --name --provider gitlab --org --token "$TOKEN" --url https://gitlab.example.com [--tags key=value,key2=value2]; unset TOKEN # Add a Bitbucket workspace (Cloud -- API token with scopes) # The Bitbucket PAT requires scopes: read:repository:bitbucket, read:account, write:repository:bitbucket, read:pullrequest:bitbucket, write:pullrequest:bitbucket -read -s TOKEN && atx ct source add --name --provider bitbucket --org --token "$TOKEN" --email --username ; unset TOKEN +read -s TOKEN && atx ct source add --name --provider bitbucket --org --token "$TOKEN" --email --username [--tags key=value,key2=value2]; unset TOKEN # Add a Bitbucket project (Data Center / self-hosted) # The Bitbucket PAT requires scopes: read:repository:bitbucket, read:account, write:repository:bitbucket, read:pullrequest:bitbucket, write:pullrequest:bitbucket -read -s TOKEN && atx ct source add --name --provider bitbucket --org --token "$TOKEN" --url https://bitbucket.example.com; unset TOKEN +read -s TOKEN && atx ct source add --name --provider bitbucket --org --token "$TOKEN" --url https://bitbucket.example.com [--tags key=value,key2=value2]; unset TOKEN ``` Add a local folder source (no token required): ```bash -atx ct source add --name --provider local --path +atx ct source add --name --provider local --path [--tags key=value,key2=value2] ``` Update token on an existing source (use instead of source add when the source already exists): @@ -124,6 +120,8 @@ Depending on the CLI version, `atx ct source list` and `atx ct repository list` ## Labels +**Labels ≠ Tags.** Labels are for client-side repo grouping/filtering. Tags (`--tags`) are IAM resource tags for access control (ABAC). When the user says "tag" in the context of access, isolation, teams, or multi-tenancy, use `--tags key=value` on the create command. When they say "label" or want to filter/organize repos for targeted analysis, use labels via `repository update --labels`. + Labels are user-defined identifiers for organizing and filtering groups of repositories. **Format:** Unicode letters, digits, `_./:-`. Max 63 chars per label, max 64 per repo. Colons are conventional for key:value grouping (e.g. `team:frontend`, `priority:high`). @@ -154,3 +152,37 @@ atx ct repository list --labels "batch:java-upgrade" ``` This lets customers organize large orgs into manageable groups (by team, priority, migration wave, etc.) without creating separate sources. + +## Tags (resource tagging for access control) + +The `--tags` flag attaches IAM resource tags to a source at creation time. Tags enable tag-based access control (ABAC) — scoped IAM policies can restrict which teams see or modify which resources. + +```bash +# Add a source with tags (comma-separated key=value pairs) +read -s TOKEN && atx ct source add --name --provider github --org --token "$TOKEN" --tags team=alpha,env=prod; unset TOKEN +``` + +**Behavior:** + +- `--tags key=value,key2=value2` accepts comma-separated pairs in a single flag (e.g. `--tags team=alpha,env=prod`). +- Tags are optional. If omitted, the source is untagged (visible to all roles, no isolation). +- Tags are registered with the backend at creation time (tag-on-create). They cannot be modified via `source add` after creation. +- If `~/.aws/atx/settings.json` defines `applyTags`, those defaults are applied automatically on every create even without explicit `--tags`. An explicit `--tags` override is merged **per key** with the settings defaults: an overridden key takes the `--tags` value, non-overridden default keys are retained, and new keys are added. + +**Settings file (`~/.aws/atx/settings.json`):** + +```json +{ + "applyTags": [ + { "team": "alpha", "env": "prod" } + ] +} +``` + +`applyTags` is an **array of tag maps** (not a single object). Every map in the array is applied on each create operation (`source add`, `analysis run`, `remediation create`) without requiring `--tags` on each command. This keeps tags consistent across the workflow. + +- **Multiple maps are legal** and are merged left-to-right into one effective tag set. On a duplicate key across maps, the **last map wins** (it is not an error). For example, `[{ "team": "alpha" }, { "team": "beta", "env": "prod" }]` resolves to `{ "team": "beta", "env": "prod" }`. +- An empty array (`[]`), a missing `applyTags` key, or a missing file all resolve to no default tags (no error). +- The merged result must satisfy AWS tag limits (≤ 50 tags; key 1–128 chars; value 0–256 chars), or the CLI aborts with `INVALID_INPUT` (400). + +**Malformed settings:** If the file exists but contains invalid JSON, or `applyTags` is structurally invalid (root not a JSON object, `applyTags` not an array, an element that is not an object, or a non-string tag value), the CLI aborts with `SETTINGS_ERROR` (422) identifying the file path and the parse or schema error. If the file is missing, no tags are applied (no error). diff --git a/aws-transform/steering/workload-continuous-modernization-status.md b/aws-transform/steering/workload-continuous-modernization-status.md index e9ba1a5..380141e 100644 --- a/aws-transform/steering/workload-continuous-modernization-status.md +++ b/aws-transform/steering/workload-continuous-modernization-status.md @@ -5,10 +5,6 @@ description: Show AWS Transform - continuous modernization (continuous moderniza # Status -## Prerequisites - -Check if the server is running with `atx ct status --health`. If any command fails with a connection error, use the `server` skill to start the server. - ## Commands ```bash diff --git a/aws-transform/steering/workload-continuous-modernization-troubleshooting.md b/aws-transform/steering/workload-continuous-modernization-troubleshooting.md index c61480a..cc4c4b7 100644 --- a/aws-transform/steering/workload-continuous-modernization-troubleshooting.md +++ b/aws-transform/steering/workload-continuous-modernization-troubleshooting.md @@ -5,7 +5,20 @@ description: Troubleshooting guide for issues that occur during setup and execut # Common Errors -## Error writing files to the ~/.atxct folder +## `atx ct` reports `unknown command 'ct'` -If attempting to read or write files to the user's ~/.atxct folder fails due to an error similar to "File access is restricted to the open workspace folders", write a script or find another way to read/write to this directory -- there is no substitute/alternative location for files in ~/.atxct so be sure to find a way to do it. +This means the shell found and ran an executable named `atx`, but the selected executable does not provide the continuous-modernization commands. Treat it as local command resolution, not an AWS credential or region problem. +Inspect what the shell selected and which alternatives are available: + +```bash +command -v atx +type -a atx +atx --version +``` + +If multiple candidates are listed, test paths returned by `type -a atx` directly with ` ct --version`. Do not assume a fixed installation path, reinstall immediately, or modify PATH automatically. Use the evidence to explain which executable won resolution and which candidate, if any, provides `ct`; then guide the user through the smallest appropriate correction for their shell. After a PATH change, start a new shell or clear its command cache (`hash -r` for Bash, `rehash` for Zsh), then retry plain `atx ct --version`. Install the AWS Transform CLI only if no discovered candidate provides `ct`. + +## Error writing files to the `~/.atxct` folder + +If attempting to read or write files to the user's `~/.atxct` folder fails due to an error similar to "File access is restricted to the open workspace folders", write a script or find another way to read/write to this directory -- there is no substitute/alternative location for files in `~/.atxct` so be sure to find a way to do it. diff --git a/aws-transform/steering/workload-custom.md b/aws-transform/steering/workload-custom.md index 1dec8e6..0ea15ac 100644 --- a/aws-transform/steering/workload-custom.md +++ b/aws-transform/steering/workload-custom.md @@ -29,7 +29,7 @@ in either mode — the user just provides repos and confirms the plan. - Define and run your own custom transformations using natural language, docs, and code samples -Run locally on a few repos for fast iteration, or at scale on hundreds of repos (up to 128 in-parallel). Note: this power collects telemetry. To opt out, see [here](https://docs.aws.amazon.com/transform/latest/userguide/transform-usage-telemetry.html). +Run locally on a few repos for fast iteration, or at scale on hundreds of repos (up to 128 in-parallel). Note: this power collects usage telemetry by default during transformation execution. The telemetry consists of different data points, such as, the IDE name (for example, VS Code or Kiro), the AI agent name (for example, Claude Code or OpenAI Codex), and the execution mode (local or remote). This data is used by AWS Transform to prioritize compatibility testing, as well as latency and reliability. To opt out, see [here](https://docs.aws.amazon.com/transform/latest/userguide/transform-usage-telemetry.html). What would you like to transform today?" diff --git a/aws-transform/steering/workload-mainframe-reimagine-modern-traceability.md b/aws-transform/steering/workload-mainframe-reimagine-modern-traceability.md index b7cc9d9..09e5b65 100644 --- a/aws-transform/steering/workload-mainframe-reimagine-modern-traceability.md +++ b/aws-transform/steering/workload-mainframe-reimagine-modern-traceability.md @@ -73,9 +73,11 @@ The agent MUST: 2. **Keep this list in working memory** throughout code generation, marking each REQ-* as "traced" when a corresponding annotation is written Each specification's header shows which business functions it covers, for example: + ``` > **Source Functions**: BatchDataProcessing-AccountProcessing, OnlineTransactionProcessing-AccountManagement ``` + This determines which `spec//traceability.yaml` files contain the legacy rule hashes that back-fill the `legacy_rule_ids` field in `traceability-modern.yaml`. **This list is the source of truth.** Code generation is not complete for a service until every REQ-* on the list has been annotated in at least one code location. @@ -252,6 +254,8 @@ architecture: '' project_root: '' # The source functions this service was built from (from the spec header) +# Read the **Source Functions** comma-separated list, e.g.: +# > **Source**: BC-1 ... | **Source Functions**: FunctionA, FunctionB source_functions: - # e.g., OnlineTransactionProcessing-AccountManagement - # e.g., BatchDataProcessing-AccountProcessing @@ -370,8 +374,8 @@ User: "Implement the account-service from the spec" Agent: 1. Reads outputs/microservices/account-service-specification.md → Extracts 49 REQ-* IDs into tracking checklist - → Notes source functions: BatchDataProcessing-AccountProcessing, - OnlineTransactionProcessing-AccountManagement + → Reads spec header: `> **Source**: BC-1 ... | **Source Functions**: BatchDataProcessing-AccountProcessing, OnlineTransactionProcessing-AccountManagement` + → Collects all source functions from the comma-separated list 2. Detects target: Java (found pom.xml) 3. Generates TracesRequirement.java annotation type (first time only) 4. Writes AccountService.java with @TracesRequirement annotations on each