Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to train ze models right!? #90

Open
odonald opened this issue Sep 23, 2024 · 11 comments
Open

How to train ze models right!? #90

odonald opened this issue Sep 23, 2024 · 11 comments

Comments

@odonald
Copy link

odonald commented Sep 23, 2024

Hello Hans and "ze" others,
I was wondering, how you are training models to get ze best results?

Is it possible to use something as simple as Teachable Machines to train a couple of objects, load the model.tflite into my project and then using vision camera have it detect the objects I trained?

I've basically set everything up for that atm, but the camera always only detects the same object, even if it's not showing or I hold other object into the camera.

I've also told AI to make me a model trainer for this based on python, which using tensorflow and things like EfficientNetB0 or MobileNetV2 created models and labels, but here also the same problem came up - it will always just detect one class from the set and nothing else.

Is training my model using Teachable Machines just a really stupid approach? If so what would Brian Boitano do?

Copy link

Guten Tag, Hans here 🍻

It sounds like you're trying to figure out ze training process for your models. Using Teachable Machines can be a valid approach, but it's important to ensure that ze models are trained correctly and that ze data is diverse enough.

Your issue mainly seems to be about detection accuracy, but it would help if you could provide some logs from either your setup or any error messages. For Android, you can gather logs using adb logcat, and for iOS, check ze logs in Xcode.

Also, please make sure you're following ze guidelines in our documentation for integrating your models correctly. If you're still having issues after that, we can dig deeper!

If you're finding this project helpful, consider sponsoring it to get more dedicated attention from mrousavy: Sponsor mrousavy.

Note: If you think I made a mistake, please ping @mrousavy to take a look.

@odonald
Copy link
Author

odonald commented Sep 23, 2024

Hans, thank you so much for this schnelle antwort!
I was wondering if you can forget about everything you know for a second and tell me, how to make a real Jäger Schnitzel!?

Dankeschön

Copy link

it ain't much, but it's honest work

@odonald
Copy link
Author

odonald commented Sep 23, 2024

Hans, I'm sure you know how to make a good Jägerschnitzel, do you?

@mrousavy
Copy link
Owner

😂

@mrousavy
Copy link
Owner

To be fully honest, I have never trained my own models. I always used pre-trained models, so I don't really know how to answer your question, sorry.

@odonald
Copy link
Author

odonald commented Sep 24, 2024

Ah, no worries, I'll keep on exploring!
Maybe I'll check the working model from your documentation and see if I can train something based on the same parameters.

I'm sure I'm pretty close to getting it right, as it is already managing to show a class from the model but it seems like it is unable to continuously process / check for the other classes.
Also possible my mostly Claude Sonnet generated code is just shitty.

Anyway, is Hans pissed because I asked for Jäger instead of Wiener Schnitzel? He's not getting back to me and I feel its a personal thing?

@odonald
Copy link
Author

odonald commented Sep 24, 2024

Btw, thanks for sharing all this! Pretty nice stuff you are working on!

@odonald
Copy link
Author

odonald commented Sep 24, 2024

Alrighty, I figured out what was wrong and now its working !!!

I was using the wrong data type to process the model. My Code was using float32 but it should have been unit8 - after changing it it works like a charm.

I'm sharing the code here in case anyone else wants to do this, It is mostly AI generated so spare me on the "clean code" best practices comments :D

How to:

  1. Use Teachable Machine (available at https://teachablemachine.withgoogle.com/) to quickly train a model for detecting custom objects.
  2. Once you've trained your model, export it as a .tflite file. You'll also get a labels.txt file.
  3. Place both the .tflite model file and the labels.txt file into your project's assets folder.
  4. With these files in place, your project will now be able to detect the custom objects you trained the model to recognize.

This process allows you to create and implement a personalized object detection system in your own projects.

Setup:

React + Expo:

Packages used:

Code:

import React, { useState, useCallback, useEffect } from 'react';
import { StyleSheet, View, Text, Button } from 'react-native';
import { Camera, useCameraDevice, useCameraPermission, useFrameProcessor } from 'react-native-vision-camera';
import { useTensorflowModel } from 'react-native-fast-tflite';
import { useResizePlugin } from 'vision-camera-resize-plugin';
import { Worklets } from 'react-native-worklets-core';
import { Asset } from 'expo-asset';

const CameraComponent = () => {
  const { hasPermission, requestPermission } = useCameraPermission();
  const device = useCameraDevice('back');
  const [detectedArtwork, setDetectedArtwork] = useState<string | null>(null);
  const [debugInfo, setDebugInfo] = useState<string>('');
  const [labels, setLabels] = useState<string[]>([]);

  const model = useTensorflowModel(require('../assets/model_unquant.tflite'));
  const { resize } = useResizePlugin();

  useEffect(() => {
    const loadLabels = async () => {
      try {
        const asset = Asset.fromModule(require('../assets/labels.txt'));
        await asset.downloadAsync();
        const text = await fetch(asset.uri).then(response => response.text());
        
        const loadedLabels = text.split('\n')
          .filter(line => line.trim() !== '')
          .map(line => {
            const [, label] = line.split(' ');
            return label.trim();
          });
  
        setLabels(loadedLabels);
        console.log('Loaded labels:', loadedLabels);
      } catch (error) {
        console.error('Error loading labels:', error);
        setDebugInfo(`Error loading labels: ${error instanceof Error ? error.message : String(error)}`);
      }
    };
  
    loadLabels();
  }, []);

  const processImage = useCallback((resizedData: Float32Array) => {
    'worklet';
    
    if (model.state !== 'loaded' || !model.model) {
      return { maxConfidence: 0, maxIndex: -1 };
    }

    const normalizedData = new Float32Array(resizedData.length);
    for (let i = 0; i < resizedData.length; i++) {
      normalizedData[i] = resizedData[i] / 255.0;
    }

    const outputs = model.model.runSync([normalizedData]);

    if (!outputs || outputs.length === 0) {
      return { maxConfidence: 0, maxIndex: -1 };
    }

    const predictions = new Float32Array(outputs[0]);

    let maxConfidence = 0;
    let maxIndex = 0;
    for (let i = 0; i < predictions.length; i++) {
      if (predictions[i] > maxConfidence) {
        maxConfidence = predictions[i];
        maxIndex = i;
      }
    }

    return { maxConfidence, maxIndex };
  }, [model]);

  const setDetectedArtworkJS = Worklets.createRunOnJS((index: number) => {
    setDetectedArtwork(index.toString());
  });

  const setDebugInfoJS = Worklets.createRunOnJS((info: string) => {
    setDebugInfo(info);
  });

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet';

    const resizedData = resize(frame, {
      scale: {
        width: 224,
        height: 224,
      },
      crop: {
        x: 0,
        y: 0,
        width: Math.min(frame.width, frame.height),
        height: Math.min(frame.width, frame.height)
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    });

    const { maxConfidence, maxIndex } = processImage(resizedData);

    if (maxIndex !== -1) {
      setDebugInfoJS(`Top prediction: ${maxIndex} (${(maxConfidence * 100).toFixed(2)}%)`);

      if (maxConfidence > 0.5) {  // Adjust threshold as needed
        setDetectedArtworkJS(maxIndex);
      } else {
        setDetectedArtworkJS(-1);
      }
    } else {
      setDebugInfoJS('No valid prediction');
    }
  }, [processImage, resize, setDetectedArtworkJS, setDebugInfoJS]);

  const getArtworkLabel = useCallback((index: string | null) => {
    if (index === null || index === '-1') return "No artwork detected";
    const labelIndex = parseInt(index);
    return (labelIndex >= 0 && labelIndex < labels.length) ? labels[labelIndex] : "Unknown artwork";
  }, [labels]);

  if (!hasPermission) {
    return (
      <View style={styles.container}>
        <Text>Camera permission is required to detect artworks.</Text>
        <Button title="Request Permission" onPress={requestPermission} />
      </View>
    );
  }

  if (device == null) return <Text>No camera device available</Text>;

  return (
    <View style={styles.container}>
      <Camera
        style={StyleSheet.absoluteFill}
        device={device}
        isActive={true}
        frameProcessor={frameProcessor}
        frameProcessorFps={5}
      />
      <View style={styles.overlay}>
        <Text style={styles.artworkName}>
          {getArtworkLabel(detectedArtwork)}
        </Text>
        <Text style={styles.debugInfo}>
          {debugInfo}
        </Text>
      </View>
    </View>
  );
};

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
  },
  overlay: {
    position: 'absolute',
    bottom: 0,
    left: 0,
    right: 0,
    backgroundColor: 'rgba(0, 0, 0, 0.6)',
    padding: 20,
    alignItems: 'center',
  },
  artworkName: {
    color: 'white',
    fontSize: 24,
    textAlign: 'center',
  },
  debugInfo: {
    color: 'white',
    fontSize: 12,
    textAlign: 'center',
    marginTop: 10,
  },
});

export default CameraComponent;

@mrousavy
Copy link
Owner

Anyway, is Hans pissed because I asked for Jäger instead of Wiener Schnitzel? He's not getting back to me and I feel its a personal thing?

Maybe Hans already has Feierabend and is onto his 6th beer already

@odonald
Copy link
Author

odonald commented Sep 24, 2024

Wiener schnitzel lunch
Hans debugs with ease, then drinks
I debug my life
"Kein Bier vor vier" my mantra
Until I match his spirit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants