Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

run inference on NNAPI or GPU #12

Open
MichaelMcCulloch opened this issue Mar 1, 2023 · 8 comments
Open

run inference on NNAPI or GPU #12

MichaelMcCulloch opened this issue Mar 1, 2023 · 8 comments
Assignees
@MichaelMcCulloch
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@MichaelMcCulloch
Copy link
Owner

No description provided.

@MichaelMcCulloch
Copy link
Owner Author

Logcat when the interpreter is loaded thus:

try {
        
        Vocab vocab = ExtractVocab.extractVocab(getAssets().open("filters_vocab_gen.bin"));
        HashMap<String, String> phraseMappings = new HashMap<>();
        
        
        _dictionary = new Dictionary(vocab, phraseMappings);
        MappedByteBuffer model = loadWhisperModel(getAssets());
        
        Interpreter.Options options = new Interpreter.Options();
        options.setUseNNAPI(true);
        NnApiDelegate.Options nnApiOptions = new NnApiDelegate.Options();
        NnApiDelegate nnApiDelegate = new NnApiDelegate(nnApiOptions);
        
        
        options.addDelegate(nnApiDelegate);
        
        
        _whisperInterpreter = new Interpreter(model, options);
        
        
        } catch (IOException e) {
        e.printStackTrace();
        }
2023-03-01 15:19:27.213 20666-20666 NetworkSecurityConfig   com.mjm.WhisperVoiceKeyboard         D  No Network Security Config specified, using platform default
2023-03-01 15:19:27.213 20666-20666 NetworkSecurityConfig   com.mjm.WhisperVoiceKeyboard         D  No Network Security Config specified, using platform default
2023-03-01 15:19:27.231 20666-20691 libEGL                  com.mjm.WhisperVoiceKeyboard         D  loaded /vendor/lib64/egl/libEGL_emulation.so
2023-03-01 15:19:27.232 20666-20691 libEGL                  com.mjm.WhisperVoiceKeyboard         D  loaded /vendor/lib64/egl/libGLESv1_CM_emulation.so
2023-03-01 15:19:27.233 20666-20691 libEGL                  com.mjm.WhisperVoiceKeyboard         D  loaded /vendor/lib64/egl/libGLESv2_emulation.so
2023-03-01 15:19:27.261 20666-20666 erVoiceKeyboar          com.mjm.WhisperVoiceKeyboard         W  Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed)
2023-03-01 15:19:27.261 20666-20666 erVoiceKeyboar          com.mjm.WhisperVoiceKeyboard         W  Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed)
2023-03-01 15:19:27.305 20666-20689 HostConnection          com.mjm.WhisperVoiceKeyboard         D  HostConnection::get() New Host Connection established 0x78fae4917250, tid 20689
2023-03-01 15:19:27.307 20666-20689 HostConnection          com.mjm.WhisperVoiceKeyboard         D  HostComposition ext ANDROID_EMU_CHECKSUM_HELPER_v1 ANDROID_EMU_native_sync_v2 ANDROID_EMU_native_sync_v3 ANDROID_EMU_native_sync_v4 ANDROID_EMU_dma_v1 ANDROID_EMU_direct_mem ANDROID_EMU_host_composition_v1 ANDROID_EMU_host_composition_v2 ANDROID_EMU_vulkan ANDROID_EMU_deferred_vulkan_commands ANDROID_EMU_vulkan_null_optional_strings ANDROID_EMU_vulkan_create_resources_with_requirements ANDROID_EMU_YUV_Cache ANDROID_EMU_vulkan_ignored_handles ANDROID_EMU_has_shared_slots_host_memory_allocator ANDROID_EMU_vulkan_free_memory_sync ANDROID_EMU_vulkan_shader_float16_int8 ANDROID_EMU_vulkan_async_queue_submit ANDROID_EMU_sync_buffer_data ANDROID_EMU_read_color_buffer_dma GL_OES_EGL_image_external_essl3 GL_OES_vertex_array_object GL_KHR_texture_compression_astc_ldr ANDROID_EMU_host_side_tracing ANDROID_EMU_gles_max_version_3_0 
2023-03-01 15:19:27.308 20666-20689 OpenGLRenderer          com.mjm.WhisperVoiceKeyboard         W  Failed to choose config with EGL_SWAP_BEHAVIOR_PRESERVED, retrying without...
2023-03-01 15:19:27.322 20666-20689 EGL_emulation           com.mjm.WhisperVoiceKeyboard         D  eglCreateContext: 0x78fad493a110: maj 3 min 0 rcv 3
2023-03-01 15:19:27.351 20666-20689 EGL_emulation           com.mjm.WhisperVoiceKeyboard         D  eglMakeCurrent: 0x78fad493a110: ver 3 0 (tinfo 0x78fa7495dde0) (first time)
2023-03-01 15:19:27.366 20666-20689 Gralloc4                com.mjm.WhisperVoiceKeyboard         I  mapper 4.x is not supported
2023-03-01 15:19:27.367 20666-20689 HostConnection          com.mjm.WhisperVoiceKeyboard         D  createUnique: call
2023-03-01 15:19:27.367 20666-20689 HostConnection          com.mjm.WhisperVoiceKeyboard         D  HostConnection::get() New Host Connection established 0x78fae4918c90, tid 20689
2023-03-01 15:19:27.367 20666-20689 goldfish-address-space  com.mjm.WhisperVoiceKeyboard         D  allocate: Ask for block of size 0x100
2023-03-01 15:19:27.367 20666-20689 goldfish-address-space  com.mjm.WhisperVoiceKeyboard         D  allocate: ioctl allocate returned offset 0x3e3ffe000 size 0x2000
2023-03-01 15:19:27.397 20666-20689 HostConnection          com.mjm.WhisperVoiceKeyboard         D  HostComposition ext ANDROID_EMU_CHECKSUM_HELPER_v1 ANDROID_EMU_native_sync_v2 ANDROID_EMU_native_sync_v3 ANDROID_EMU_native_sync_v4 ANDROID_EMU_dma_v1 ANDROID_EMU_direct_mem ANDROID_EMU_host_composition_v1 ANDROID_EMU_host_composition_v2 ANDROID_EMU_vulkan ANDROID_EMU_deferred_vulkan_commands ANDROID_EMU_vulkan_null_optional_strings ANDROID_EMU_vulkan_create_resources_with_requirements ANDROID_EMU_YUV_Cache ANDROID_EMU_vulkan_ignored_handles ANDROID_EMU_has_shared_slots_host_memory_allocator ANDROID_EMU_vulkan_free_memory_sync ANDROID_EMU_vulkan_shader_float16_int8 ANDROID_EMU_vulkan_async_queue_submit ANDROID_EMU_sync_buffer_data ANDROID_EMU_read_color_buffer_dma GL_OES_EGL_image_external_essl3 GL_OES_vertex_array_object GL_KHR_texture_compression_astc_ldr ANDROID_EMU_host_side_tracing ANDROID_EMU_gles_max_version_3_0 
2023-03-01 15:19:29.109 20666-20682 System                  com.mjm.WhisperVoiceKeyboard         W  A resource failed to call close. 
2023-03-01 15:19:33.910 20666-20666 Compatibil...geReporter com.mjm.WhisperVoiceKeyboard         D  Compat change id reported: 147798919; UID 10130; state: ENABLED
2023-03-01 15:19:42.337 20666-20666 System.out              com.mjm.WhisperVoiceKeyboard         I  Succeeded in Loading Vocab! 50257 (50257) Words.
2023-03-01 15:19:42.340 20666-20666 InterpreterApi          com.mjm.WhisperVoiceKeyboard         I  Loaded native library: tensorflowlite_jni
2023-03-01 15:19:42.340 20666-20666 InterpreterApi          com.mjm.WhisperVoiceKeyboard         I  Didn't load native library: tensorflowlite_jni_gms_client
2023-03-01 15:19:42.342 20666-20666 tflite                  com.mjm.WhisperVoiceKeyboard         I  Initialized TensorFlow Lite runtime.
2023-03-01 15:19:42.345 20666-20666 tflite                  com.mjm.WhisperVoiceKeyboard         I  Created TensorFlow Lite delegate for NNAPI.
2023-03-01 15:19:42.348 20666-20666 AndroidRuntime          com.mjm.WhisperVoiceKeyboard         D  Shutting down VM
2023-03-01 15:19:42.349 20666-20666 AndroidRuntime          com.mjm.WhisperVoiceKeyboard         E  FATAL EXCEPTION: main
                                                                                                    Process: com.mjm.WhisperVoiceKeyboard, PID: 20666
                                                                                                    java.lang.RuntimeException: Unable to create service com.mjm.whisperVoiceRecognition.VoiceKeyboardInputMethodService: java.lang.IllegalArgumentException: Internal error: Error applying delegate: 
                                                                                                    	at android.app.ActivityThread.handleCreateService(ActivityThread.java:4204)
                                                                                                    	at android.app.ActivityThread.access$1500(ActivityThread.java:237)
                                                                                                    	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1932)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                    	at android.os.Looper.loop(Looper.java:223)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:7656)
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
                                                                                                    Caused by: java.lang.IllegalArgumentException: Internal error: Error applying delegate: 
                                                                                                    	at org.tensorflow.lite.NativeInterpreterWrapper.createInterpreter(Native Method)
                                                                                                    	at org.tensorflow.lite.NativeInterpreterWrapper.init(NativeInterpreterWrapper.java:106)
                                                                                                    	at org.tensorflow.lite.NativeInterpreterWrapper.<init>(NativeInterpreterWrapper.java:73)
                                                                                                    	at org.tensorflow.lite.NativeInterpreterWrapperExperimental.<init>(NativeInterpreterWrapperExperimental.java:36)
                                                                                                    	at org.tensorflow.lite.Interpreter.<init>(Interpreter.java:227)
                                                                                                    	at com.mjm.whisperVoiceRecognition.VoiceKeyboardInputMethodService.onCreate(VoiceKeyboardInputMethodService.java:76)
                                                                                                    	at android.app.ActivityThread.handleCreateService(ActivityThread.java:4192)
                                                                                                    	at android.app.ActivityThread.access$1500(ActivityThread.java:237) 
                                                                                                    	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1932) 
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:106) 
                                                                                                    	at android.os.Looper.loop(Looper.java:223) 
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:7656) 
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method) 
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) 
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) 
2023-03-01 15:19:42.350 20666-20666 Process                 com.mjm.WhisperVoiceKeyboard         I  Sending signal. PID: 20666 SIG: 9

@nyadla-sys
Copy link

It appears that the NNDelegate code has crashed, and my suspicion is that it is using the xnn pack delegate by default. To use the NNDelegate, certain flags may need to be enabled during the building of the tflite library.

If you wish to run a model other than the English one, you will need to load the "filters_vocab_multilingual.bin" file from this GitHub repository: https://github.com/usefulsensors/openai-whisper/blob/main/models/filters_vocab_multilingual.bin. Please try running the model with this file loaded.

@nyadla-sys
Copy link

to just confirm if nnapi delegate is working or not with tflite model.
please use the below tflite model and run the test again
https://github.com/tensorflow/tflite-micro/blob/main/third_party/xtensa/examples/pytorch_to_tflite/mobilenet_v2_quantized_1x3x224x224.tflite

@MichaelMcCulloch
Copy link
Owner Author

MichaelMcCulloch commented Mar 1, 2023 •
edited
Loading

to just confirm if nnapi delegate is working or not with tflite model. please use the below tflite model and run the test again https://github.com/tensorflow/tflite-micro/blob/main/third_party/xtensa/examples/pytorch_to_tflite/mobilenet_v2_quantized_1x3x224x224.tflite

This tflite model appears to work.

It appears that the NNDelegate code has crashed, and my suspicion is that it is using the xnn pack delegate by default. To use the NNDelegate, certain flags may need to be enabled during the building of the tflite library.

If you wish to run a model other than the English one, you will need to load the "filters_vocab_multilingual.bin" file from this GitHub repository: https://github.com/usefulsensors/openai-whisper/blob/main/models/filters_vocab_multilingual.bin. Please try running the model with this file loaded.

Bookmarked, thank you.

@nyadla-sys
Copy link

to just confirm if nnapi delegate is working or not with tflite model. please use the below tflite model and run the test again https://github.com/tensorflow/tflite-micro/blob/main/third_party/xtensa/examples/pytorch_to_tflite/mobilenet_v2_quantized_1x3x224x224.tflite

This tflite model appears to work.
Looks like Whisper model is not compatible for NNAPI delegate. If I learn more about it I will post here..

@MichaelMcCulloch
Copy link
Owner Author

Are you able to tell me what factors contribute to it being compatible?

@nyadla-sys
Copy link

The whisper hybrid tflite model may not be compatible due to its mixed precision format, where weights are in int8 and activations are in float32. It's possible that generating a full float model could resolve this issue. Additionally, other causes of incompatibility could be related to unsupported ops, meaning that certain operations included in the model are not supported by the NNDelegate's available ops

@MichaelMcCulloch MichaelMcCulloch added enhancement New feature or request help wanted Extra attention is needed labels Mar 3, 2023
@MichaelMcCulloch MichaelMcCulloch self-assigned this Mar 3, 2023
@MichaelMcCulloch MichaelMcCulloch mentioned this issue Mar 11, 2023
Closed
@stopthinking102
Copy link

Any success with running it on gpu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MichaelMcCulloch MichaelMcCulloch

Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MichaelMcCulloch @stopthinking102 @nyadla-sys