Similar mAP when splitting data into train, val and test

[n-patricia]

Question

Hi,
I use yolov5x with this setting (train: 70%, val: 10%, test: 20%)

train: images/train  # train images (relative to 'path') 128 images
val: images/val  # val images (relative to 'path') 128 images
test: images/test # test images (optional)

I get similar mAP starts from epoch 18. The columns here are precision, recall, mAP0.5, mAP0.95.

However when I don't divide the training data, the metrics keep increasing until 300 epochs.

train: images/  # train images (relative to 'path') 128 images
val: images/  # val images (relative to 'path') 128 images
test: # test images (optional)

I use hyp-scratch-low for both experiments. The reason I do this is bacause I want to try my custom dataset like GlobalWheat2020 after this. Do I miss anything?

Additional

No response

[github-actions]

👋 Hello @n-patricia, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@n-patricia hi there,

Thank you for sharing your observations and detailed results! It's great to see your interest in optimizing your training process with YOLOv5.

From your description, it seems like the issue might be related to the dataset split and the size of your dataset. Here are a few points to consider:

1. Dataset Size and Split: With only 128 images in your dataset, splitting it into train, val, and test sets might be causing each subset to be too small to effectively train and validate the model. Typically, we recommend having at least 1500 images per class for training to achieve good results. If possible, try to increase the size of your dataset.

2. Training with Default Settings: It's always a good idea to start with default settings to establish a performance baseline. Ensure that your dataset is well-labeled and representative of the deployment environment. You can refer to our Tips for Best Training Results for more detailed guidance.

3. Validation and Test Sets: The purpose of the validation set is to tune the model during training, while the test set is used to evaluate the final model performance. If you combine all images into the training set, the model might overfit, leading to artificially high metrics. Ensure that your validation set is representative and sufficiently large to provide meaningful feedback during training.

4. Hyperparameters and Overfitting: Using the hyp.scratch-low.yaml is a good start. However, if you notice overfitting or plateauing metrics, you might want to experiment with different hyperparameters or use techniques like data augmentation to improve generalization.

Here is an example of how you might structure your dataset configuration for a larger dataset:

train: images/train  # train images (relative to 'path')
val: images/val  # val images (relative to 'path')
test: images/test # test images (optional)

And for training, you can use:

python train.py --data custom.yaml --weights yolov5x.pt --epochs 300

If you continue to face issues, please ensure you are using the latest version of YOLOv5 and provide additional details such as training logs, loss curves, and any other relevant metrics. This will help us diagnose the problem more effectively.

Feel free to reach out if you have any more questions or need further assistance. Happy training! 🚀

[n-patricia]

@glenn-jocher thank you for your response. I understand your point. The problem from this data is imbalanced dataset. I tried to select 40 classes, with range from 13k to 200 samples per class. I just randomized the images for train and val set, because it is difficult to select balance classes for each set. Do you think it is better to let the algorithm select the samples by itself because of the imbalanced problem?

[glenn-jocher]

Hi @n-patricia,

Thank you for your detailed follow-up! Imbalanced datasets can indeed pose challenges during training, but there are several strategies you can employ to mitigate these issues and improve your model's performance.

Strategies for Handling Imbalanced Datasets

1. Class Weights: One effective approach is to use class weights to give more importance to underrepresented classes. YOLOv5 supports this through the --weights argument. You can manually set class weights in your training script or use a method to calculate them based on the frequency of each class.

2. Data Augmentation: Augmenting your dataset can help balance the classes by artificially increasing the number of samples for underrepresented classes. Techniques like rotation, flipping, and color jittering can be beneficial.

3. Oversampling and Undersampling: You can oversample the minority classes or undersample the majority classes to balance the dataset. This can be done manually or using libraries like imbalanced-learn.

4. Focal Loss: Focal Loss is designed to address class imbalance by down-weighting easy examples and focusing more on hard, misclassified examples. While YOLOv5 does not natively support Focal Loss, you can modify the loss function in the codebase if you're comfortable with custom implementations.

Example: Using Class Weights

Here's an example of how you might set class weights in your training script:

# Example of setting class weights
class_weights = [1.0, 2.0, 0.5, ...]  # Adjust these values based on your class distribution

Automated Sample Selection

Regarding your question about letting the algorithm select the samples, YOLOv5 does not currently have an automated sample selection feature based on class balance. However, you can implement a custom data loader that ensures balanced batches during training.

Next Steps

1. Verify Latest Version: Ensure you are using the latest version of YOLOv5 to benefit from the latest features and bug fixes.
2. Experiment with Class Weights: Try setting class weights to see if it improves your model's performance.
3. Augment Your Data: Use data augmentation techniques to increase the diversity and balance of your dataset.

Feel free to share your results or any further questions you might have. The YOLO community and Ultralytics team are here to help you achieve the best possible results with your model. Good luck! 🚀