Video2Text dataset idea💡💡💡

[Danik843]

Greetings, gentlemen!
To compile the dataset, OpenAI used a special neural network to describe video clips in detail. I propose to use the following repository for this purpose:

Video-LLaVA - [https://github.com/PKU-YuanGroup/Video-LLaVA]

I have already tried it on one video clip. Well, I want to say that it was not perfect and the repository needs to be improved in terms of attention length, because I wrote a query and the neural network just ignored most of it.

To describe it, I used the following video clip - [https://www.pexels.com/video/tangier-drone-20214791/]

Here's my request to the neural network:

---

In this task, you are to generate a detailed description of a stock video clip. The description should capture the essence of the video, including its setting, key objects, and any characters. It must also articulate the style and mood of the clip, the colors used, and any significant symbols or themes. Crucially, your description should unfold along a time axis, noting changes in scene, action, or atmosphere as they occur from the beginning to the end of the clip:

Overview Description: Start with a broad overview of the video clip, describing the central theme, setting, and the main subject. This should include the type of scene (e.g., urban, natural, interior), the primary action or activity, and the overall mood or atmosphere.

Detailed Scene Elements: Break down the scene into its key visual elements. Describe each significant object, character, or feature in the scene with detailed adjectives. Mention colors, shapes, sizes, and any notable textures or patterns.

Time Axis Description: Include a detailed description of the time axis, highlighting any changes that occur throughout the clip. Describe how the scene evolves, including changes in lighting (e.g., from dawn to daylight), movements of objects or characters, and any dynamic elements like weather conditions, camera movements, or shifts in focus.

Special Effects and Style: If the video includes special effects like slow motion, time-lapse, or augmented reality elements, describe these in detail. Mention any stylistic themes, such as a vintage look, a futuristic aesthetic, or a documentary style, and how these are achieved through color grading, filters, or editing techniques.

Atmosphere and Emotions: Discuss the emotional tone or atmosphere of the clip. Describe how the combination of visual elements, movements, and effects contribute to the mood. Mention if the clip is meant to evoke feelings of joy, tension, serenity, etc.

Technical Specifications: Conclude with any technical specifications that are relevant to the description, such as the aspect ratio, resolution, or any specific requirements for text overlays or branding elements.

Here's an example of a description based on these guidelines:

"A stock video clip depicting a bustling city street scene at dusk. The video opens with a wide shot of the city skyline against a deep blue sky, gradually transitioning into the busy streets below. Neon signs flicker in bright shades of neon blue and hot pink, illuminating the faces of a variety of pedestrians walking along the crowded sidewalks. Midway through the clip, the camera zooms in to capture the smallest details of everyday urban life: street vendors selling exotic foods, couples sharing intimate moments, and street performers mesmerizing viewers. As the clip progresses, the natural light dims, demonstrating the transition from dusk to night. The clip utilizes a handheld cinematic style of filming, creating a sense of immediacy and immersion. Special effects include slow motion footage of raindrops hitting the sidewalk, which enhances the atmospheric mood. The overall effect is one of dynamic energy and urban beauty, evoking a sense of wanderlust and the vibrant pulse of city life. Specifications: 16:9 aspect ratio, 4K resolution, there is space for text overlays at the beginning and end."

---

Answer of Video-LLaVA:

The video clip showcases a picturesque view of a small village nestled in a valley, surrounded by lush greenery and a clear blue sky. The camera pans around the village, capturing the quaint houses, narrow streets, and the vibrant colors of the village's architecture. The camera then focuses on a group of people walking around the village, with some carrying backpacks and handbags, suggesting that they might be tourists or locals going about their daily routines. The camera captures their movements, highlighting the village's charm and the peaceful atmosphere. The camera then pans around the village, showcasing the village's layout and the beautiful scenery surrounding it. The camera captures the village's serene beauty, with the sound of birds chirping and the rustling of leaves in the background. The video concludes with a panoramic view of the village, showcasing its beauty and the harmony between the village and its surroundings.

---

My initial experimentation with Video-LLaVA on a specific video clip offered promising insights, albeit with some areas needing improvement. Notably, the neural network's response to complex queries was found to be somewhat lacking, particularly in terms of adequately capturing the full breadth of the content as specified in our detailed request. This indicates a potential area of enhancement in the neural network's attention mechanism to ensure comprehensive coverage of the input query.

For your reference, our request aimed at generating a thorough description across several dimensions: from a broad overview encompassing the setting, main subjects, and mood, to intricate details regarding scene elements, temporal changes, stylistic effects, and emotional ambiance. Despite these detailed instructions, the neural network's output did not fully align with our expectations, as it provided a generic description lacking in the requested specificity and depth.

In conclusion, I am confident that Video-LLaVA can become an invaluable asset in our dataset preparation efforts.

[kabachuha]

Hi, I have already made a VideoLLaVA/LLMs gradio UI for captioning long form video https://github.com/kabachuha/video2scenario, check it out

[Danik843]

I think a more detailed promt should be made, for the dataset. I made a promt based on the rewritten ChatGPT promt for DALL-E, but with the addition of a description of changes over time.

[LinB203]

Please note that Video-LLaVA currently has weak control over text, which means it may not be very controlled by your instructions. But it is sufficient for captioning. However, it currently only supports 8 frames, and for long videos (especially 30s-60s), it may lose information. Additionally, its output may exceed 77 words, which is beyond the scope of CLIP.