refactor: split oversized files into focused modules#133
Conversation
Break the 7,000-line worker/gcp.rs into a directory module so each concern can be read and reviewed in isolation: - gcp/mod.rs: controller struct, compute-operation tracking, and the #[controller] handler impl (kept whole for macro state generation) - gcp/support.rs: naming helpers, error classifiers, heartbeat emission, and GcsNotificationTracker - gcp/helpers.rs: the plain helper-method impl block - gcp/tests.rs: the controller test module Pure code motion: moved items are bumped to pub(super) where needed; external paths are unchanged via the existing worker::gcp re-exports.
Greptile SummaryThis PR splits the monolithic
Confidence Score: 5/5Pure code-motion refactor with no logic changes; the proc-macro input in mod.rs is byte-for-byte identical and all 478 tests pass. Every changed file is either a new module boundary file containing code moved verbatim from the original gcp.rs, or the slimmed-down mod.rs that retains the macro-processed handler block unchanged. The only non-motion deltas are the 14 pub(super) visibility bumps (all internal) and two rustfmt reflows from the dedent and a signature wrap — none affecting behavior. The re-export of GcsNotificationTracker correctly preserves the external API path. No files require special attention; the split is mechanical and the verification evidence (cargo check, nextest, byte-level diff) is thorough.
|
| Filename | Overview |
|---|---|
| crates/alien-infra/src/worker/gcp/mod.rs | Root module retaining the #[controller] struct and its handler impl block; declares helpers/support/tests sub-modules and re-exports GcsNotificationTracker via pub use support::GcsNotificationTracker. |
| crates/alien-infra/src/worker/gcp/helpers.rs | Extracted helper impl block with 14 methods bumped to pub(super); imports support items via use super::support::* and conditionally imports GcpWorkerState under test-utils feature. |
| crates/alien-infra/src/worker/gcp/support.rs | Free functions, consts, and GcsNotificationTracker type; all items are pub(super) except GcsNotificationTracker which is pub (re-exported from mod.rs) to preserve the external API path. |
| crates/alien-infra/src/worker/gcp/tests.rs | Test module moved out of the inline mod block with mandatory dedent; imports are adjusted to use super:: paths for support utilities, no logic changes. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["worker/mod.rs\n(crate re-exports)"]
B["gcp/mod.rs\n(#[controller] struct\n+ handler impl block\n3,985 lines)"]
C["gcp/helpers.rs\n(helper impl block\n14 pub(super) methods\n1,422 lines)"]
D["gcp/support.rs\n(free fns, consts, types\n248 lines)"]
E["gcp/tests.rs\n(#[cfg(test)]\n1,441 lines)"]
A --> B
B -->|"mod helpers;"| C
B -->|"mod support;\nuse support::*;\npub use support::GcsNotificationTracker"| D
B -->|"#[cfg(test)] mod tests;"| E
C -->|"use super::support::*"| D
E -->|"use super::{ get_cloudrun_service_name, ... }"| D
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A["worker/mod.rs\n(crate re-exports)"]
B["gcp/mod.rs\n(#[controller] struct\n+ handler impl block\n3,985 lines)"]
C["gcp/helpers.rs\n(helper impl block\n14 pub(super) methods\n1,422 lines)"]
D["gcp/support.rs\n(free fns, consts, types\n248 lines)"]
E["gcp/tests.rs\n(#[cfg(test)]\n1,441 lines)"]
A --> B
B -->|"mod helpers;"| C
B -->|"mod support;\nuse support::*;\npub use support::GcsNotificationTracker"| D
B -->|"#[cfg(test)] mod tests;"| E
C -->|"use super::support::*"| D
E -->|"use super::{ get_cloudrun_service_name, ... }"| D
Reviews (1): Last reviewed commit: "refactor: split gcp worker controller in..." | Re-trigger Greptile
Pure git mv of worker/azure.rs to worker/azure/mod.rs with no content changes, so rename detection and git log --follow stay intact ahead of the module carve-out.
Break the 6,500-line worker/azure.rs into a directory module so each concern can be read and reviewed in isolation: - azure/mod.rs: controller struct, environment-wake retry tracking, and the #[controller] handler impl (kept whole for macro state generation) - azure/support.rs: naming helpers, wait/delay helpers, error classifiers, heartbeat emission, and AzureStorageTriggerInfrastructure - azure/helpers.rs: the plain helper-method impl block - azure/tests.rs: the controller test module Pure code motion: moved items are bumped to pub(super) where needed; external paths are unchanged via the existing worker::azure re-exports.
Break the 5,000-line worker/aws.rs into a directory module so each concern can be read and reviewed in isolation: - aws/mod.rs: controller struct and the #[controller] handler impl (kept whole for macro state generation) - aws/support.rs: naming helpers, error classifiers, heartbeat emission, and the load balancer state types - aws/helpers.rs: the plain helper-method impl blocks - aws/tests.rs: the controller test module Pure code motion: moved items are bumped to pub(super) where needed; external paths are unchanged via the existing worker::aws re-exports.
Break the 5,700-line gcp/compute.rs into a directory module so each concern can be read and reviewed in isolation: - compute/mod.rs: module doc, ComputeServiceConfig, and re-exports - compute/api.rs: the ComputeApi trait (automock still generates MockComputeApi here, re-exported unchanged) - compute/client.rs: ComputeClient and its ComputeApi impl - compute/types/: the serde data structures, split along the existing section banners (operations, network, load_balancer, instance) - compute/tests.rs: the PSC serde test module Pure code motion: no items were renamed or reordered; external paths are unchanged via glob re-exports from compute/mod.rs. Note in gcp/AGENTS.md that compute/ is the directory-module variant of the cloudrun.rs pattern.
Break the 4,800-line lib.rs into concern modules so each area can be read and reviewed in isolation: - lib.rs: build_stack orchestration, build_resource, and build_target_to_file - push.rs: push_stack, push targets, OCI tarball selection, and image index assembly - cache.rs: artifact cache keys, source hashing, cargo metadata, and cached artifact lookup - base_images.rs: base image resolution, entrypoint/cmd handling, and pull/retry classification - tests.rs: the unit test module, still gated by #[cfg(test)] Pure code motion: moved items are bumped to pub(crate); the public API is unchanged via the root pub use of push_stack.
Break the 4,647-line generator.rs into a generator/ directory so each concern can be read and reviewed in isolation: - mod.rs: the options types, the generate_helm_chart and render_manager_fetch_values entrypoints, and shared JSON/YAML/name helpers - operator.rs: generate_operator_manifest and the operator document builders - values.rs: ChartAnalysis, values.yaml assembly, and cloud-identity mapping - schema.rs: values.schema.json - templates.rs: the Helm template (.tpl) bodies - examples.rs: the per-target values examples and README - tests.rs: the unit test module, still gated by #[cfg(test)] Pure code motion: moved helpers are bumped to pub(super); the public API is unchanged (lib.rs still re-exports generate_operator_manifest and the other entrypoints). The file drops off the hk.pkl max-lines exclude now that every module is under the 2000-line cap.
Break the 2,262-line generator.rs into a generator/ directory so each concern can be read and reviewed in isolation: - mod.rs: the template constants, the RegistrationMode / CloudFormationOptions / CloudFormationTarget types, the generate_cloudformation_template entrypoint, YAML serialization, stack validation, and the shared resource/output helpers - parameters.rs: the standard/network/compute parameter blocks, the supported-region and custom-domain rules, the standard conditions, and the parameter-default helpers - expressions.rs: the kubernetes/network/domains/management settings expressions, the JSON <-> CfExpression bridge, and the CfParameter / CfOutput constructors - tests.rs: the unit test module, still gated by #[cfg(test)] Pure code motion: moved helpers are bumped to pub(super); the public API is unchanged (lib.rs still re-exports generate_cloudformation_template and the other entrypoints). The file drops off the hk.pkl max-lines exclude now that every module is under the 2000-line cap.
The EC2 client lived in a single 3616-line file. Split it into an ec2/ module mirroring the existing client layout: api (the Ec2Api trait), client (Ec2Client and its impls plus the cpu-options unit tests), and types/ grouped by resource (common, network, security_group, instance, volume, launch_template). Pure code motion with import redistribution and module wiring; no behavior change. Remove the file from the max-lines exclude list now that every resulting file is under the 2000-line cap.
Break the 3,140-line gcp_compute_client_tests.rs into a directory test target so each area can be read and reviewed in isolation. The binary name (gcp_compute_client_tests) and all 17 tests are unchanged: - main.rs: crate doc, module wiring, and the framework smoke test - context.rs: the ComputeTestContext fixture and its cleanup/wait helpers - vpc.rs, load_balancing.rs, disks.rs, instances.rs, ssl_proxy.rs: the comprehensive lifecycle tests, one area per module - errors.rs: the not-found/error-mapping tests plus the operation-status test Pure code motion: no test logic changed. Fixture items used across modules are bumped from private to pub(crate), per-module imports are consolidated, the mid-function rcgen import is hoisted to the top of ssl_proxy.rs, and a mislabeled "Error Handling Tests for New APIs" banner that sat above the SSL lifecycle test is dropped. The file is removed from the hk.pkl max-lines exclude list now that every module is under the 2,000-line cap.
Break the 2,265-line aws_s3_client_tests.rs into a directory test target so each area can be read and reviewed in isolation. The binary name (aws_s3_client_tests) and all 37 tests are unchanged: - main.rs: the crate attribute and module wiring - context.rs: the S3TestContext fixture, its cleanup helpers, and the put_test_object helper - bucket.rs: bucket lifecycle, policy, lifecycle-config, public-access block, name validation, and notification-config tests - object.rs: object put/get/head, listing, delete, and empty-bucket tests - versioning.rs: bucket-versioning and versioned-object tests - location.rs: the get_bucket_location tests Pure code motion: no test logic changed. Fixture items used across modules are bumped from private to pub(crate) and per-module imports are consolidated. The file is removed from the hk.pkl max-lines exclude list now that every module is under the 2,000-line cap.
Break the 4,726-line distribution.rs into a distribution/ directory so each concern can be read and reviewed in isolation: - mod.rs: DistributionArtifactCleanup, setup_distribution and prepare_distribution, the flow-availability and stack-settings helpers, the cross-domain flow orchestrators (run_cloudformation_k8s, run_terraform_cloud/k8s, run_onprem_k8s), and stack import/finalize - cleanup.rs: post-error cleanup and retained-resource teardown - cloudformation.rs: run_cloudformation_aws and the CloudFormation request/output/stack-event helpers - terraform.rs: the Terraform apply/import flow, tfvars, outputs, and handoff-debug helpers - helm.rs: chart/values rendering, the Kubernetes helm target and kubeconfig materialization, and image-rewrite helpers - permissions.rs: the GCP and Azure management permission probes - env.rs: cloud env assembly (aws/gcp/azure/terraform) - exec.rs: the process runners and shell-quote helpers - tests.rs: the unit test module, still gated by #[cfg(test)] Pure code motion: moved helpers are bumped to pub(super); the public API is unchanged (setup_distribution and DistributionArtifactCleanup are still defined in mod.rs). The file drops off the hk.pkl max-lines exclude now that every module is under the 2,000-line cap.
Break the 2,700-line heartbeat.rs into a heartbeat/ directory so each family of provider heartbeat types can be read and reviewed on its own: - mod.rs: the ResourceHeartbeat envelope, ObservedInventoryBatch / ObservedResourceSample, shared health/lifecycle/backend/collection enums, event snapshots, raw snippets, and metric types - storage.rs: StorageHeartbeatData and the S3/Blob/GCS/Local variants - workload.rs: worker, container, and daemon heartbeat data plus the runtime unit and pod status types - cluster.rs: compute-cluster and kubernetes-cluster heartbeat data with capacity/drain/fleet and node status types - data_services.rs: queue, kv, postgres, and vault families - platform.rs: service-account, network, and remote-stack-management families - registry_build.rs: artifact-registry, build, and service-activation families - azure_platform.rs: Azure resource-group, storage-account, container apps environment, and service-bus-namespace families - tests.rs: the serde round-trip unit module, still gated by #[cfg(test)] Pure code motion: submodules are glob re-exported from mod.rs so the public API at the crate root is unchanged. Drop the file from the hk.pkl max-lines exclude list now that every module is under the cap.
The generator split dedented the tests module by one level to extract it into its own file, which unintentionally stripped four spaces from a line inside a raw-string YAML literal, placing deploymentLabelValue at the document root instead of under logCollector.scope. Helm 4 schema validation rejects the root-level key, failing log_collector_enabled_chart_lints_and_templates. Restore the four-space indentation so the literal is byte-identical to pre-split. Verified with helm 4.2.3 + kubeconform: alien-helm 15/15 tests pass.
Relocate post-split changes from main into their focused modules and extract test modules that crossed the 2,000-line limit after the merge.
Why
Several source files in this repo had grown past ~2,000 lines, which makes them slow to navigate, painful to review, and merge-conflict prone. This umbrella PR splits them into focused modules along seams that already exist in the code (section boundaries, impl-block boundaries, test modules). Every split is strictly behavior-preserving: pure code motion plus the minimal import/visibility wiring needed to compile — no logic, signature, or public-API changes.
Splits land incrementally, one commit per file. The list below tracks what has landed; a handful of pre-existing large files remain and are staged for their own follow-up splits (see Planned additions).
Included so far
1.
crates/alien-infra/src/worker/gcp.rs(7,070 lines) →worker/gcp/gcp/mod.rs#[controller]struct and its single#[controller]handler impl block, kept wholegcp/helpers.rspub(super)gcp/support.rspub(super))gcp/tests.rsProof: the entire retained
mod.rssegment — the#[controller]struct and the full handler impl block — is byte-for-byte identical (cmp) to the corresponding segment of the original file, so the proc-macro input tokens are unchanged. Nothing was madepub;GcsNotificationTrackerkeeps its external path via apub usere-export. Git records the split as a rename (55% similarity), sogit log --followstill tracks history. The--testswarning set is message-identical to a baseline captured on the parent commit, proving no imports were silently dropped.2.
crates/alien-infra/src/worker/azure.rs(6,488 lines) →worker/azure/azure/mod.rsazure/helpers.rsazure/support.rsazure/tests.rsProof: done as two commits — a pure
git mvtoazure/mod.rs, then the carve-out — touching only the four azure files. A--color-movedanalysis accounts for all 6,642 changed lines: 3,226 verbatim-moved line pairs, with the remaining ~190 plain lines beingmoddeclarations, import redistribution, and visibility bumps only.3.
crates/alien-infra/src/worker/aws.rs(5,003 lines) →worker/aws/aws/mod.rsaws/helpers.rsaws/support.rsaws/tests.rsProof: a single commit touching only the four aws files; rename detected at 69% similarity; byte-level comparison of each moved segment against the original confirms pure motion.
worker/mod.rsuntouched.4.
crates/alien-gcp-clients/src/gcp/compute.rs(5,682 lines) →gcp/compute/compute/mod.rscompute/api.rscompute/client.rscompute/tests.rscompute/types/mod.rscompute/types/operations.rscompute/types/network.rscompute/types/load_balancer.rscompute/types/instance.rsProof: byte-level comparison of each new file's body against its original segment confirms pure code motion. The only other change is a one-line pointer update in the sibling
AGENTS.md.5.
crates/alien-build/src/lib.rs(4,817 lines) → module fileslib.rsbase_images.rscache.rspush.rstests.rsProof:
--color-movedclassification shows 3,255 verbatim-moved lines; all 105 non-moved added and 53 removed lines are module wiring and import redistribution only. Only the five files undercrates/alien-build/src/are touched.6.
crates/alien-test/src/distribution.rs(4,726 lines) →distribution/distribution/mod.rsdistribution/tests.rsdistribution/terraform.rsdistribution/helm.rsdistribution/permissions.rsdistribution/cloudformation.rsdistribution/cleanup.rsdistribution/env.rsdistribution/exec.rsProof: independently verified pure code motion. Only these ten files plus the one-line
hk.pklexclude removal change; no lockfiles or generated artifacts. Moves classified with--color-moved; the residual plain lines aremoddeclarations,use super::*imports, andpub(super)visibility bumps.cargo test -p alien-test --libpasses.7.
crates/alien-helm/src/generator.rs(4,647 lines) →generator/generator/templates.rsgenerator/values.rsgenerator/tests.rsgenerator/operator.rsgenerator/schema.rsgenerator/mod.rsgenerator/examples.rsProof: verified pure code motion — the diff touches only the seven
generator/files plus the one-linehk.pklexclude removal.generate_operator_manifestkeeps its public path through apub usere-export frommod.rs, solib.rscompiles with zero changes.cargo test -p alien-helmpasses against the moved test module.8.
crates/alien-deploy-cli/src/commands/up.rs(3,991 lines) →up/up/pull.rsup/push.rsup/tests.rsup/resolve.rsup/mod.rsup/inputs.rsup/config.rsup/machines.rsProof: verified behavior-preserving pure code motion. The only non-move text changes are 73
pub(super)visibility bumps and six path rewrites fromsuper::operator::tocrate::commands::operator::(plus one hoisted import), which are equivalent references to the same items.cargo test -p alien-deploy-clipasses (the offlineup/tests.rsunit tests: config parsing, tfvars/values rendering, token handling); sibling commands (down,join,list,status) that import the re-exporteduppaths still compile.9.
crates/alien-aws-clients/src/aws/ec2.rs(3,616 lines) →ec2/ec2/client.rsec2/types/network.rsec2/types/security_group.rsec2/types/instance.rsec2/types/launch_template.rsec2/types/volume.rsec2/api.rsec2/types/common.rsec2/mod.rsec2/types/mod.rsProof: independently verified as a clean, behavior-preserving pure code-motion split into a ten-file
ec2/module.MockEc2Apikeeps resolving via the glob re-export, so the existingaws_ec2_client_testsintegration tests are unchanged.cargo check -p alien-aws-clients --features test-utils --all-targetsandcargo test -p alien-aws-clients --libpass; downstream importers of the ec2 types (network/aws.rs,worker/aws) still compile.10.
crates/alien-gcp-clients/tests/gcp_compute_client_tests.rs(3,140 lines) →gcp_compute_client_tests/gcp_compute_client_tests/context.rsgcp_compute_client_tests/load_balancing.rsgcp_compute_client_tests/instances.rsgcp_compute_client_tests/vpc.rsgcp_compute_client_tests/ssl_proxy.rsgcp_compute_client_tests/errors.rsgcp_compute_client_tests/disks.rsgcp_compute_client_tests/main.rsProof: verified pure code motion. Only the expected files change: the deleted original, the eight new module files, a one-line doc-pointer update in the sibling
AGENTS.md, and thehk.pklexclude removal — no lockfiles, generated artifacts, or stray files. The directory test target keeps the same binary name, so all tests are still discovered undergcp_compute_client_tests(cargo test -p alien-gcp-clients --test gcp_compute_client_tests -- --list).11.
crates/alien-terraform/src/generator.rs(2,903 lines) →generator/generator/variables.rsgenerator/mod.rsgenerator/providers.rsgenerator/registration.rsgenerator/readme.rsgenerator/tests.rsProof: verified pure code-motion split. The diff touches only those six
generator/files plus the one removedhk.pklexclude line — no logic changes.cargo test -p alien-terraformpasses against the movedcfg(test)module.12.
crates/alien-core/src/heartbeat.rs(2,729 lines) →heartbeat/heartbeat/data_services.rsheartbeat/workload.rsheartbeat/mod.rsheartbeat/tests.rsheartbeat/cluster.rsheartbeat/registry_build.rsheartbeat/platform.rsheartbeat/azure_platform.rsheartbeat/storage.rsProof: independently verified clean pure code motion into a directory module (
mod.rsplusstorage/workload/cluster/data_services/platform/registry_build/azure_platform/testssubmodules). The diff is confined to those ten files plus the one-linehk.pklexclude removal. Heartbeat types are re-exported at thealien_coreroot and consumed across many crates, so path stability is proven by a cleancargo check --workspace --all-targets;cargo test -p alien-corepasses.13.
crates/alien-aws-clients/tests/aws_s3_client_tests.rs(2,265 lines) →aws_s3_client_tests/aws_s3_client_tests/object.rsaws_s3_client_tests/bucket.rsaws_s3_client_tests/versioning.rsaws_s3_client_tests/location.rsaws_s3_client_tests/context.rsaws_s3_client_tests/main.rsProof: faithful pure code-motion split into a directory test target (
main.rspluscontext/bucket/object/versioning/location), mirroring the gcp compute-tests layout. Exactly nine files touched — noCargo.toml, lockfile, generated, or unrelated changes. A sorted line-multiset comparison of old vs. new confirms every test line is preserved; the binary name is unchanged so the integration test still compiles.14.
crates/alien-cloudformation/src/generator.rs(2,262 lines) →generator/generator/mod.rsgenerator/parameters.rsgenerator/expressions.rsgenerator/tests.rsProof: rigorously verified pure code-motion split. Reconstructing the base file and running both a sorted line-multiset diff and an order-preserving per-region diff shows
mod.rsequals the base's retained regions andparameters.rs/expressions.rsequal their carved-out regions exactly.cargo test -p alien-cloudformationpasses against the movedcfg(test)module.Planned additions
The remaining pre-existing files over 2,000 lines are excluded in
hk.pklas legacy, each pending its own follow-up split:crates/alien-deploy-cli/src/commands/join.rscrates/alien-infra/src/kubernetes_public_endpoint.rscrates/alien-infra/src/network/aws.rscrates/alien-bindings/src/provider.rscrates/alien-bindings/tests/storage.rscrates/alien-infra/src/container/kubernetes.rscrates/alien-infra/src/network/azure.rscrates/alien-infra/src/core/executor.rscrates/alien-commands/src/server/mod.rscrates/alien-cli/src/commands/release.rscrates/alien-preflights/src/mutations/compute_cluster.rsHow verified (on the current branch tip)
rustfmt --check --edition 2021, trailing-whitespace, end-of-file, merge-conflict, and themax-linesstep.cargo check --workspace --all-targetsclean (this is the real cross-crate path-stability proof — the heartbeat and ec2 splits re-export widely used types).cargo check -p alien-core --features openapi,cargo check -p alien-aws-clients --features test-utils --all-targets,cargo check -p alien-gcp-clients --features test-utils --all-targets.cfg(test)unit modules pass:cargo nextest run --libacrossalien-helm,alien-terraform,alien-cloudformation,alien-core,alien-test,alien-deploy-cli.How to review
Use move detection:
The
--color-moved-wsflag matters for thetests.rshunks because of the dedent out of the former inlinemod tests {}blocks. Nearly everything renders as moved code; the remaining lines are import redistribution,moddeclarations, and the visibility bumps noted above. Each split is its own commit (azure is two:git mvthen carve-out), so commits can also be reviewed one at a time.Regression guard
To keep files from growing back past the threshold, this PR adds a
max-linesstep tohk.pklthat runs in thepre-commit,check, andfixhooks.*.rs,*.ts,*.tsx,*.js,*.jsx.wc -l {{files}}piped through awk (portable BSD awk, no GNU extensions); any file over the limit fails the hook with a message naming the file and its line count.client-sdks/**(Speakeasy codegen) andpackages/sdk/src/worker-runtime/generated/**(protoc-gen-ts_proto).Grandfathered files
The three worker controller modules stay over the limit by design: the
#[controller]macro requires the annotated struct and its single annotated impl block to live in one module, so after the splits above these cannot shrink further.crates/alien-infra/src/worker/gcp/mod.rscrates/alien-infra/src/worker/aws/mod.rscrates/alien-infra/src/worker/azure/mod.rsThe legacy files still over the limit are listed under Planned additions above; each is excluded in
hk.pkluntil its own split lands.Verified:
hk check --step max-lines --allpasses across the repo; a probe file over the limit fails with a clear message; every excluded path is skipped by the step.