feat: introduce two LoKR variants - KronA and CDKA

[gesen2egee]

This PR introduces two Kronecker-product-based network adapter variants designed for parameter-efficient fine-tuning: KronA and CDKA.

KronA is inspired by the WACV-2025 paper "DiffuseKronA".
CDKA is an improved successor to KronA, operating with a default R=1 and adapted for LoKR structures.

These adapters achieve significant parameter reduction while enhancing anti-overfitting capabilities, making them highly effective for personalizing and fine-tuning diffusion models on small datasets.

Both modules share a unified isomorphic codebase architecture. By adjusting parameters and default configurations, they can easily convert into each other:

• KronA: Supports custom Rank configurations as described in the original paper.
• CDKA: Operating with a default rank constraint of $R=1$, adapted for LoKR structures, and supports the custom cdka_alpha scaling mechanism.

References

• KronA (DiffuseKronA):

  • Title: DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models (WACV-2025)
  • Project Website: https://diffusekrona.github.io/
  • arXiv Paper: https://arxiv.org/abs/2402.17412

• CDKA:

  • Title: Diving into Kronecker Adapters: Component Design Matters
  • arXiv Paper: https://arxiv.org/abs/2602.01267
  • Repository: https://github.com/rainstonee/CDKA

Extra Features Supported:

• DoRA: Introduced weight decomposition support (weight_decompose=True) to decouple weight updates into magnitude and direction.

  • arXiv Paper: https://arxiv.org/abs/2402.09353

• ALLoRA: Dynamically schedules layer-wise learning rates based on adapter weight norms. It automates learning rate balancing (like LoRA+) and prevents gradient collapse across deep layers.

  • arXiv Paper: https://arxiv.org/abs/2410.09692

Recommended Configurations

You can configure the network in your training configuration (e.g., .toml files) as follows.
And use any method that supports general LoKR to infer the trained model.

[network]
network_module = "networks.cdka"
network_args = [
    'exclude_patterns=["^(?=.*attn)(?!.*adaln).*$"]',
    "allora=True"
]
network_train_unet_only = true

A slightly higher learning rate than usual (e.g., 3e-4) can be used.