[MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths#29709
Open
JonathanC-ARM wants to merge 12 commits into
Open
[MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths#29709JonathanC-ARM wants to merge 12 commits into
JonathanC-ARM wants to merge 12 commits into
Conversation
Avoid holding a reference into lhs_ptrs_cache_by_pad across the lookup/update sequence so the cache remains keyed by the current pad buffer identity. Source-commit: 923422f Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Native fp16 accumulation paths need a slightly wider tolerance, and WebGPU needs a separate bound for this coverage. Source-commit: 2483990 Source-commit: 08f4c6a8c51dc018b4440c63d279bf5995d3386a Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
The Resize(13) MLFloat16 test should not run against EPs that do not provide this kernel. Source-commit: 2483990 Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Allow the fp16 FusedConvWithSum transformer test to tolerate native fp16 numerical drift. Source-commit: 2483990 Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
These internal-domain NHWC fp16 pool tests target ORT CPU/MLAS coverage and should not be offered to unrelated registered EPs. Source-commit: a504a0c Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
- Add validation for null HalfGemm data parameters and reject invalid backend-native packed-B parameter combinations before the zero-K fast path. - Make HalfGemm packed-B size calculation return a valid zero size for degenerate N/K shapes. - Restore the KleidiAI header include needed by debug logging in mlasi_kleidiai.h. - Update HalfGemm PackB tests to avoid exception-only gtest macros in no-exception builds, and add sentinel checks for early-return paths. Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Set MLAS_CONV_PARAMETERS::ChannelsLast in the HalfConv prepare path so Prepare fully initializes the convolution parameter layout state. Add a focused MLAS test that verifies Prepare overwrites stale layout flag values for both NCHW and NHWC HalfConv configurations. Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com>
Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com>
Register CPU fp16 MatMul/Gemm kernels and route fp16 Conv through the native half-conv/prepack paths using the MLAS APIs from the previous branch. This carries the current branch session_state.cc relaxation as WIP; the maintainer-requested sentinel modeling for backend-owned prepack buffers still needs to replace it before review. Source-commit: 923422f Source-commit: a504a0c Source-commit: 152ef31 Source-commit: b81f7ab9f2e35b85d9e4f825c31aa4ddaad7e875 Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
Introduce an explicit kernel-owned packed-weights marker so fp16 MatMul can report successful backend-owned prepacking without relaxing the existing shared prepack invariant for broken kernels. Update fp16 MatMul, Gemm, Conv, Moe, and MLAS test call sites for the branch-2 half-GEMM API shape where backend selector config is carried through MLAS_HALF_GEMM_DATA_PARAMS. Clarify HalfConv working-buffer sizing as bytes, zero-initialize MLAS_CONV_PARAMETERS, and add/adjust tests for the fp16 MatMul kernel-owned prepack path, broken prepack invariant, half-GEMM selector handling, and HalfConv byte-size reporting. Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR builds on fp16-split/02-mlas-half-api-squashed by connecting its FP16 MLAS APIs and KleidiAI backends to the CPU Execution Provider.
Motivation and Context
This is pr 3 of 5 as per #28786 splitting suggestion
The preceding PR introduces the MLAS primitives for FP16 HalfGemm, backend-native B packing, and HalfConv, but standard ONNX CPU operators do not yet consume them.
This PR connects those primitives to CPU
MatMul,Gemm, andConv, making native FP16 execution and KleidiAI acceleration available to the operator layer while retaining portable fallback behavior for unsupported hardware, shapes, attributes, or backend configurations.The prepacking changes avoid repeatedly packing constant weights while preventing backend-specific or bias-dependent layouts from being incorrectly reused through the shared prepack cache.
This PR is limited to CPU kernel registration and execution. The follow-up InsertCastTransformer PR introduces the opt-in policy and shape heuristics that decide when graph optimization should preserve FP16 CPU nodes. Apple ARM64 build enablement is also handled separately.