v0.3.1

See https://github.com/quic/ai-hub-apps/releases/v0.3.1 for changelog.

Signed-off-by: QAIHM Team <[email protected]>

README.md

@@ -47,6 +47,7 @@ __NOTE: These apps will run without NPU acceleration on non-Snapdragon® chipset
 | [Image Classification](apps/android/ImageClassification) | Java | TensorFlow Lite |
 | [Semantic Segmentation](apps/android/SemanticSegmentation) |  Java | TensorFlow Lite | OpenCV, Live Camera Feed |
 | [Super Resolution](apps/android/SuperResolution) | Java | TensorFlow Lite |
+| [WhisperKit (Speech to Text)](https://github.com/argmaxinc/WhisperKitAndroid) | Various | TensorFlow Lite |
 
 ## _Windows_ App Directory
 

apps/android/WhisperKit/README.md

@@ -0,0 +1,9 @@
+[![Qualcomm® AI Hub Apps](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/quic-logo.jpg)](https://aihub.qualcomm.com)
+
+# WhisperKit, by ArgMax
+
+Our SDK collaborator, [ArgMax](https://www.takeargmax.com/), presents [WhisperKit](https://www.takeargmax.com/blog/whisperkit), a state-of-the-art on-device speech-to-text solution.
+
+WhisperKit supports Qualcomm Devices on the Android Operating System.
+
+### [WhisperKit for Android Repository](https://github.com/argmaxinc/WhisperKitAndroid)

apps/windows/cpp/ChatApp/ChatApp.cpp

@@ -6,7 +6,6 @@
 #include "PromptHandler.hpp"
 #include <fstream>
 #include <iostream>
-#include <regex>
 
 using namespace App;
 
@@ -48,42 +47,14 @@ void GenieCallBack(const char* response_back, const GenieDialog_SentenceCode_t s
     }
 }
 
-//
-// LoadModelConfig - Loads model config file
-//  - Loads config file in memory
-//  - Replaces place-holders with user provided values
-//
-std::string LoadModelConfig(const std::string& model_config_path,
-                            const std::string& models_path,
-                            const std::string& htp_model_config_path,
-                            const std::string& tokenizer_path)
-{
-    std::string config;
-    // Read config file into memory
-    std::getline(std::ifstream(model_config_path), config, '\0');
-
-    // Replace place-holders in config file with user provided paths
-    config = std::regex_replace(config, std::regex("<models_path>"), models_path);
-    config = std::regex_replace(config, std::regex("<htp_backend_ext_path>"), htp_model_config_path);
-    config = std::regex_replace(config, std::regex("<tokenizer_path>"), tokenizer_path);
-    return config;
-}
-
 } // namespace
 
-ChatApp::ChatApp(const std::string& model_config_path,
-                 const std::string& models_path,
-                 const std::string& htp_config_path,
-                 const std::string& tokenizer_path)
+ChatApp::ChatApp(const std::string& config)
 {
-
-    // Load model config in-memory
-    std::string config = LoadModelConfig(model_config_path, models_path, htp_config_path, tokenizer_path);
-
     // Create Genie config
     if (GENIE_STATUS_SUCCESS != GenieDialogConfig_createFromJson(config.c_str(), &m_config_handle))
     {
-        throw std::runtime_error("Failed to create the Genie Dialog config. Please check config file.");
+        throw std::runtime_error("Failed to create the Genie Dialog config. Please check config.");
     }
 
     // Create Genie dialog handle

apps/windows/cpp/ChatApp/ChatApp.hpp

@@ -24,21 +24,15 @@ class ChatApp
   public:
     /**
      * ChatApp: Initializes ChatApp
-     *    - Loads Model config with provided model path, htp config and tokenizer
+     *    - Uses provided Genie configuration string
      *    - Creates handle for Genie
      *
-     * @param model_config_path: local path to model Genie config file
-     * @param models_path: local path to directory that contains model context binaries (e.g., downloaded from AI Hub)
-     * @param htp_config_path: local path to backend htp configuration
-     * @param tokenizer_path: local path to tokenizer to use
+     * @param config: JSON string containing Genie configuration
      *
-     * @thows on failure to create handle for Genie config, dialog
+     * @throws on failure to create handle for Genie config, dialog
      *
      */
-    ChatApp(const std::string& model_config_path,
-            const std::string& models_path,
-            const std::string& htp_config_path,
-            const std::string& tokenizer_path);
+    ChatApp(const std::string& config);
     ChatApp() = delete;
     ChatApp(const ChatApp&) = delete;
     ChatApp(ChatApp&&) = delete;
@@ -49,7 +43,7 @@ class ChatApp
     /**
      * ChatWithUser: Starts Chat with user using previously loaded config
      *
-     * @param user_name: User name to  use during chat
+     * @param user_name: User name to use during chat
      *
      * @throws on failure to query model response during chat
      *

apps/windows/cpp/ChatApp/ChatApp.vcxproj.user

@@ -1,11 +1,11 @@
-<?xml version="1.0" encoding="utf-8"?>
-<Project ToolsVersion="Current" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM64'">
-    <LocalDebuggerCommandArguments>--model_config $(ProjectPath)\..\assets\configs\genie\llama2_7b.json --tokenizer $(ProjectPath)\..\assets\configs\tokenizer\llama2_7b.json --htp_config $(ProjectPath)/../assets/configs/htp_backend_ext/v73.json --models "{directory containing context binary models downloaded from AI Hub}"</LocalDebuggerCommandArguments>
-    <DebuggerFlavor>WindowsLocalDebugger</DebuggerFlavor>
-  </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'">
-    <LocalDebuggerCommandArguments>--model_config $(ProjectPath)\..\assets\configs\genie\llama2_7b.json --tokenizer $(ProjectPath)\..\assets\configs\tokenizer\llama2_7b.json --htp_config $(ProjectPath)/../assets/configs/htp_backend_ext/v73.json --models "{directory containing context binary models downloaded from AI Hub}"</LocalDebuggerCommandArguments>
-    <DebuggerFlavor>WindowsLocalDebugger</DebuggerFlavor>
-  </PropertyGroup>
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="Current" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|ARM64'">
+    <LocalDebuggerCommandArguments>--genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\</LocalDebuggerCommandArguments>
+    <DebuggerFlavor>WindowsLocalDebugger</DebuggerFlavor>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'">
+    <LocalDebuggerCommandArguments>--genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\</LocalDebuggerCommandArguments>
+    <DebuggerFlavor>WindowsLocalDebugger</DebuggerFlavor>
+  </PropertyGroup>
 </Project>

apps/windows/cpp/ChatApp/Main.cpp

@@ -12,23 +12,17 @@
 namespace
 {
 
-constexpr const std::string_view c_option_model_config_path = "--model_config";
-constexpr const std::string_view c_option_models_path = "--models";
-constexpr const std::string_view c_option_htp_config_path = "--htp_config";
-constexpr const std::string_view c_option_tokenizer_path = "--tokenizer";
+constexpr const std::string_view c_option_genie_config = "--genie-config";
+constexpr const std::string_view c_option_base_dir = "--base-dir";
 constexpr const std::string_view c_option_help = "--help";
 constexpr const std::string_view c_option_help_short = "-h";
 
 void PrintHelp()
 {
     std::cout << "\n:::::::: Chat with " << App::c_bot_name << " options ::::::::\n\n";
-    std::cout << c_option_model_config_path << " <Local file path>: [Required] Path to local genie config for model.\n";
-    std::cout << c_option_models_path
-              << " <Local directory path>: [Required] Path to local models downloaded from AI Hub.\n";
-    std::cout << c_option_htp_config_path << " <Local file path>: [Required] Path to local HTP backend extensions.\n"
-              << "\tAvailable htp configurations are located here <project_dir>/assets/configs/htp_backend_ext/\n";
-    std::cout << c_option_tokenizer_path << " <Local file path>: [Required] Path to local tokenizer json.\n"
-              << "\tAvailable tokenizers are located here <project_dir>/assets/configs/tokenizer/\n";
+    std::cout << c_option_genie_config << " <Local file path>: [Required] Path to local Genie config for model.\n";
+    std::cout << c_option_base_dir
+              << " <Local directory path>: [Required] Base directory to set as the working directory.\n";
     std::cout << "\nDuring chat with " << App::c_bot_name << ", please type " << App::c_exit_prompt
               << " as a prompt to terminate chat.\n ";
 }
@@ -40,17 +34,16 @@ void PrintWelcomeMessage()
     std::cout << "At any time during chat, please type `" << App::c_exit_prompt
               << "` to terminate the conversation.\n\n";
     std::cout << "Let's begin with an introduction,\n";
-    std::cout << "I'm `" << App::c_bot_name << "`! what's your name ? ";
+    std::cout << "I'm `" << App::c_bot_name << "`! What's your name? ";
 }
 
 } // namespace
 
 int main(int argc, char* argv[])
 {
-    std::string model_config_path;
-    std::string models_path;
-    std::string htp_config_path;
-    std::string tokenizer_path;
+    std::string genie_config_path;
+    std::string base_dir;
+    std::string config;
     bool invalid_arguments = false;
 
     // Check if argument file path is accessible
@@ -67,25 +60,31 @@ int main(int argc, char* argv[])
     // Arg parser
     for (int i = 1; i < argc; ++i)
     {
-        if (c_option_model_config_path == argv[i])
+        if (c_option_genie_config == argv[i])
         {
-            model_config_path = argv[++i];
-            check_arg_path(c_option_model_config_path, model_config_path);
+            if (i + 1 < argc)
+            {
+                genie_config_path = argv[++i];
+                check_arg_path(c_option_genie_config, genie_config_path);
+            }
+            else
+            {
+                std::cout << "\nMissing value for " << c_option_genie_config << " option.\n";
+                invalid_arguments = true;
+            }
         }
-        else if (c_option_models_path == argv[i])
+        else if (c_option_base_dir == argv[i])
         {
-            models_path = argv[++i];
-            check_arg_path(c_option_models_path, models_path);
-        }
-        else if (c_option_htp_config_path == argv[i])
-        {
-            htp_config_path = argv[++i];
-            check_arg_path(c_option_htp_config_path, htp_config_path);
-        }
-        else if (c_option_tokenizer_path == argv[i])
-        {
-            tokenizer_path = argv[++i];
-            check_arg_path(c_option_tokenizer_path, tokenizer_path);
+            if (i + 1 < argc)
+            {
+                base_dir = argv[++i];
+                check_arg_path(c_option_base_dir, base_dir);
+            }
+            else
+            {
+                std::cout << "\nMissing value for " << c_option_base_dir << " option.\n";
+                invalid_arguments = true;
+            }
         }
         else if (c_option_help == argv[i] || c_option_help_short == argv[i])
         {
@@ -99,18 +98,29 @@ int main(int argc, char* argv[])
         }
     }
 
-    // If invalid arguments, print help and exit.
-    if (invalid_arguments)
+    // If invalid arguments or required arguments are missing, print help and exit.
+    if (invalid_arguments || genie_config_path.empty() || base_dir.empty())
     {
         PrintHelp();
         return 1;
     }
 
     try
     {
+        // Load genie_config_path into std::string config before changing directory
+        std::ifstream config_file(genie_config_path);
+        if (!config_file)
+        {
+            throw std::runtime_error("Failed to open Genie config file: " + genie_config_path);
+        }
+
+        config.assign((std::istreambuf_iterator<char>(config_file)), std::istreambuf_iterator<char>());
+
+        std::filesystem::current_path(base_dir);
+
         std::string user_name;
 
-        App::ChatApp app(model_config_path, models_path, htp_config_path, tokenizer_path);
+        App::ChatApp app(config);
 
         // Get user name to chat with
         PrintWelcomeMessage();

apps/windows/cpp/ChatApp/README.md

@@ -21,12 +21,17 @@ The app demonstrates how to use the Genie APIs from [QNN SDK](https://qpm.qualco
 
 ## Build App
 
-### Downloading models from AI Hub
+### Compile to QNN Binary via AI Hub and Generate Genie Bundle
 
-1. Please follow the instructions from [Llama2 from AI Hub Models](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models/llama_v2_7b_chat_quantized/gen_ondevice_llama) to generate models and assets required by ChatApp
-2. Copy assets to target device (e.g., Snapdragon® based Windows)
+1. Copy `ChatApp` to target device (e.g., Snapdragon® based Windows)
+
+2. Please follow [this
+tutorial](https://github.com/qcom-ai-hub/ai-hub-apps-internal/tree/main/tutorials/llm_on_genie)
+to generate `genie_bundle` required by ChatApp
+
+3. Copy bundle assets from step 2 to `ChatApp\genie_bundle`. You should see
+`ChatApp\genie_bundle\*.bin` QNN binary files.
 
-NOTE: This process takes a long time e.g. ~2 hours to export models, ~2 hours to build QNN binaries to run on-device.
 
 ### Setup QNN SDK
 
@@ -58,49 +63,51 @@ Make sure `QNN_SDK_ROOT` is set globally pointing to QNN SDK before you build th
 
 1. Open `ChatApp.sln`
 2. Build project in Visual Studio
-   - It takes around 2 mins to build on X Elite.
 
 ## Running App
 
 ### Providing local paths
 
-Make sure to provide paths to local config file and models using `\\` or `/` as a path separator and not `\`
-
-
-#### Correct file path examples
+### Running via Visual Studio
 
-```powershell
-1. C:\\Path\\To\\Model\\Config\\llama2_7b.json
-2. C:/Path/To/Model/Config/llama2_7b.json
-```
+Click on the green play button to build and run.
 
-#### Incorrect file path example
+Visual studio project is configured with the following command arguments:
 
 ```powershell
-1. C:\Path\To\Model\Config\llama2_7b.json
+.\ARM64\Debug\ChatApp.exe --genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\
 ```
 
 ### Running app via CLI
 
 ```powershell
 cd {Project directory}
-.\ARM64\Debug\ChatApp.exe --model_config .\\assets\\configs\\genie\\llama2_7b.json --htp_config .\\assets\\configs\\htp_backend_ext\\v73.json --tokenizer .\\assets\\configs\\tokenizer\\llama2_7b.json --models {Directory path with models and htp-extension-config downloaded from AI Hub}
+.\ARM64\Debug\ChatApp.exe --genie-config .\\genie_bundle\\genie_config.json --base-dir .\\genie_bundle\\
 ```
 
-You can additionally run `--help` to get more information about all available options:
+Run `--help` to learn more:
 
 ```powershell
 .\ARM64\Debug\ChatApp.exe --help
 ```
 
-### Running via Visual Studio
+Make sure to provide paths to local config file and models using `\\` or `/` as a path separator and not `\`
 
-Visual studio project is configured with the following command arguments:
+
+#### Correct file path examples
+
+```powershell
+1. C:\\Path\\To\\Model\\Config\\llama2_7b.json
+2. C:/Path/To/Model/Config/llama2_7b.json
+```
+
+#### Incorrect file path example
 
 ```powershell
---model_config .\\assets\\configs\\genie\\llama2_7b.json --htp_config .\\assets\\configs\\htp_backend_ext\\v73.json --tokenizer .\\assets\\configs\\tokenizer\\llama2_7b.json --models {Directory path with models and htp-extension-config downloaded from AI Hub}
+1. C:\Path\To\Model\Config\llama2_7b.json
 ```
 
+
 ### Sample Output
 
 ![sample_output](assets/images/sample_output.png)

apps/windows/cpp/ObjectDetection/README.md

@@ -14,8 +14,8 @@ The app demonstrates how to use the [QNN execution provider](https://onnxruntime
 ### Tools and SDK
 
 - Visual Studio 22
+  - Make sure **Desktop development with C++ tools** are selected during installation or installed separately later
   - Download any variant of [Visual Studio here](https://visualstudio.microsoft.com/vs/)
-  - Make sure Desktop development with C++ tools are selected during installation or installed separately later
 - QNN SDK: [Qualcomm AI Engine Direct](https://qpm.qualcomm.com/#/main/tools/details/qualcomm_ai_engine_direct)
   - Download and install the latest Qualcomm AI Engine Direct SDK
   - Make libraries from `<QNN_SDK>/libs/<target_platform>` accessible to app target binary