-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support MLX in WhisperAX #200
base: mlx-support
Are you sure you want to change the base?
Conversation
…lity (#192) * Make additional initializers, functions, members public, for WKPro * Allows use of default internal functions & member accesses which have increased protections when imported * Initializers were Xcode generated: right click class name -> refactor -> generate memberwise initializers * memberwise initializer defaults to internal, mark as public. * Formatting --------- Co-authored-by: ZachNagengast <[email protected]>
… models (#193) * Add initial mlpackage loading (if .mlmodelc not present) -- Does not modify model loading in OS WK. This is a hook to modify load path URLs. * Always load audio encoder last * Adjust timings to account for decoder<>encoder order swap * Add helper for mlpackage detection --------- Co-authored-by: ZachNagengast <[email protected]>
* Fix start time logic for file loading and resampling * Add test file
As far as I can tell, these stored properties are not meant to be changed. Therefore, change them to be immutable. This change also makes these static properties concurrency-safe.
* Add VoiceActivityDetector base class Add base class to allow different VAD implementations * fix spaces
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, added some comments, let me know what you think
======= | ||
>>>>>>> main |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
git conflict markers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah good catch 👍
Package.resolved
Outdated
"location" : "https://github.com/ml-explore/mlx-swift", | ||
"location" : "https://github.com/davidkoski/mlx-swift.git", | ||
"state" : { | ||
"revision" : "597aaa5f465b4b9a17c8646b751053f84e37925b", | ||
"version" : "0.16.0" | ||
"revision" : "3314bc684f0ccab1793be54acddaea16c0501d3c" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curious, why this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, now I can see why, nvm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has been merged into mlx-swift and tagged 0.16.2
@@ -171,6 +137,74 @@ public struct ModelComputeOptions { | |||
} | |||
} | |||
|
|||
public struct ModelInfo: Identifiable, Hashable { | |||
public let id = UUID() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curious, why this property public let id = UUID()
is needed? can it be uniquely identified by name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed to make it identifiable, although I was using this for the picker at one point and may be vestigial, will check.
let keyCache = try? MLX.stacked(keyCacheResult).asMLMultiArray() | ||
let valueCache = try? MLX.stacked(valueCacheResult).asMLMultiArray() | ||
let decodingCache = DecodingCache( | ||
keyCache: keyCache, | ||
valueCache: valueCache, | ||
alignmentWeights: nil | ||
) | ||
|
||
let logits = try? result.logits?.asMLMultiArray() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems like you've changed it to try ?
in couple of places, is it intended?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, we want this to pass the throw up through the call stack. Reviewing build warnings as well.
that fork was merged and is now the |
Awesome, thanks for the update @davidkoski! |
@ZachNagengast - This looks awesome! Apologies - rookie question... It looks like currently the MLX repo only has the base and tiny models. How hard is it for mere mortals to "build" some of the larger models for testing this out with larger models? |
Add support for the WhisperAX example app, as well as various refactors and cleanup.
Note: this is using an unreleased version of MLX, pending merge of ml-explore/mlx-swift#130
There are still some memory issues to address, will look more into this soon.
CoreML:
MLX:
Important note, first time running this will likely throw an error about PrepareMetalShaders, which requires Trust & Enable on this popup when selecting the error.