Compute exact matrix powers in matrix_inverse_pth_root for any p - #1735
Open
TheSaiEaranti wants to merge 1 commit into
Open
Compute exact matrix powers in matrix_inverse_pth_root for any p#1735TheSaiEaranti wants to merge 1 commit into
TheSaiEaranti wants to merge 1 commit into
Conversation
mat_power selected an unrolled matrix power via round(log2(p)), which is
exact only for p in {1, 2, 4, 8}. For any other p the coupled Newton
iteration converged to the wrong fixed point while still reporting a tiny
error indicator: p=3 returned results with 18.6% max relative error against
the eigendecomposition reference, p=5 10.7%, p=6 7.4%, p=7 3.0%, all with
internal error ~1e-7. The docstring contract is matrix^(-1/p) for any
positive integer p, and p=6 arises in practice as 2*ndim when
preconditioning rank-3 tensors.
Replace the switch with exponentiation by squaring in a lax.while_loop,
exact for any positive p including traced values. All p in 1..8 now match
the eigendecomposition reference to ~1e-7. Add a parameterized regression
test over p in 1..8; the four previously-silent p values fail on the old
implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
optax.matrix_inverse_pth_root's docstring promisesmatrix^(-1/p)for any positive integerp, but the internalmat_powerhelper selects an unrolled matrix power viaround(log2(p)), which is exact only forp ∈ {1, 2, 4, 8}. For every otherpthe coupled Newton iteration converges to the wrong fixed point while still reporting a tiny error indicator:The failure is independent of any reference implementation: for
p=3,inv(result³)differs from the input matrix by 12% relative error.p=6arises in practice as2 * ndimwhen preconditioning rank-3 tensors Shampoo-style. Nothing surfaces the restriction — no error, no docstring note — and the existing test only exercisesp=4.Fix
Replace the rounded-power switch with exponentiation by squaring in a
lax.while_loop— exact for any positive integerp, including traced values. This mirrors what the upstreamgoogle-research/scalable_shampooimplementation did when it replaced this same construction with an exactmat_power. Allpin 1..8 now match the eigendecomposition reference to ~1e-7, on and offjit.One trade-off worth flagging: the loop does ~2 matmuls per bit of
p(e.g. 6 forp=8versus 3 unrolled before). If that matters for Shampoo-scale workloads, a concrete-intfast path that unrolls exactly at trace time is easy to layer on — happy to add it here if preferred.Tests
test_matrix_inverse_pth_root_arbitrary_p, parameterized overp ∈ 1..8, asserts both the returned error indicator and the max relative error against theeigh-basedM^(-1/p). On the previous implementation exactly the four silent values (3, 5, 6, 7) fail and the four correct ones pass. Fulllinear_algebra_test.py: 31 passed, 240 subtests.