How to Annotate Videos Data on Datature Nexus

We are excited to announce that Nexus now supports video annotation on our platform. This means that we now allow videos to be uploaded or synced through our connection manager, the S3 bucket, and then annotated using Intellibrush, our flagship annotation tool. Videos are a very common data source in a lot of industries, for example, streaming data from live cameras. The frames of these videos can be used as training data for your computer vision model. This can be super useful since a single video can be a great source for supplying a lot of input images to your model. The data annotation stage can be relatively tedious for videos specifically, given the large number of frames per second that need to be annotated. Keeping these factors in mind, we designed a video annotation pipeline that is quite similar to our image annotation process and allows you to seamlessly upload, annotate and train your model on any video data you’d like.

Introduction

We already have an article: Introducing Video Compatibility to Support Broader Use Cases in Computer Vision that details important factors you should consider when uploading, annotating and training your video data. The article briefly touches upon the process of uploading and annotation of videos as well, but we would like to use this article as a step-by-step guide so that each step of the process is clear.

Uploading Videos on Datature Nexus

Within your project dashboard, navigate to the Assets tab in the left pane. You will see a rectangular box in the centre that allows you to upload assets from your device or drag and drop them into the designated space. As you can see, this box currently supports videos that are in the .MP4 format. While you can easily convert videos not in the MP4 format, native support on Nexus for other popular video file formats such as AVI and MOV.

You could also upload videos directly through Nexus’ Connection Manager from an external cloud storage like your Amazon S3 bucket. We provide this external data syncing service in case your files are too big and require external storage space, as well as accommodate your preexisting data infrastructure. When syncing video assets from an external storage such as Amazon S3, any MP4 files are supported, but with the following two restrictions: the major brand must be mp42 and the pixel format is yuv420p. These restrictions ensure that the videos can play in all supported browsers in the annotator. Typical MP4 files should be able to meet these requirements.

Unlike native uploading, externally-synced video assets will not be modified in any way. This means that they will retain their original dimensions, quality, and audio, if any.

Once your video is uploaded, you can proceed to annotation by simply clicking on the video or navigating to the Annotator tab in the left pane and then selecting the video you’d like to annotate.

Annotating Your Videos on Datature Nexus

Navigation

The bottom pane of the video annotation page has two buttons: one that allows you to navigate between video assets and one that allows you to navigate between frames of the same video. Selecting the first view lets you view all the videos that have been uploaded as part of the project, and you can switch back and forth between these videos. 

The second view allows you to navigate between frames of the video you are currently annotating. Since video annotation is done on a frame-by-frame basis, this view shows to you which frame of the video you’re currently on. To go back and forth between different frames, you could use the arrow keys next to the frame number, the hot keys ‘Q’ and ‘E’ to move left and right respectively, or use the frame window to scrub back and forth by dragging your mouse along. 

‍

Creating Annotations

If you have used our image annotation tools before, you will notice that our video annotation page looks extremely similar, except for some navigation and visibility features that provide you with more clarity on what’s happening with the annotation. Feel free to check out our article on Intellibrush, our flagship annotation tool, in case you’re unfamiliar with it. To annotate an object, simply select the appropriate shape tool (Rectangle, Polygon, Freedraw) and drag it over the object. Right click over the shape and select the tag that you would like to label the object with. You could also add new tags and manage existing tags in the right pane by clicking on the pen-shaped icon in the Tabs section.

Checking Your Annotations

The play button near the frame number allows you to play through your annotated video frame-by-frame. As you play through your video, you will also be able to see a coloured timeline of the frames each annotated object tag appears in. This is a good sense check of whether or not you’re annotating your video correctly as well as keeping track of the stretch of frames your object appears in and the tags you’ve annotated so far. If your video is very long and there’s a lot of frames, the timeline might get a little dense to look at. In such cases, you might want to zoom into specific frames or sets of frames to get a better sense of what’s happening. You can do so by increasing the playback or zoom settings, as we shall now walk you through.

Playback and Display Settings

You can adjust the playback settings for your video by clicking the button on the top right of the navigation panel. A pop-up window should open up, allowing you to adjust the playback speed, volume and zoom settings. The playback speed allows you to adjust the rate at which the frames are displayed. For videos with very similar frames, you could increase the playback speed, while for videos that have a lot of movement and constantly changing objects, it’s better to reduce the playback speed so you can ensure you haven’t missed out on labelling anything. 

The zoom setting allows you to adjust the range of frames that are displayed on the timeline at once, allowing you to look at more or less frames at once, regardless of whether the video is playing or not. This allows you to check your annotations at different levels. You could also change the display settings of your video (annotation opacity, brightness, contrast and saturation) by clicking the Settings icon on top of the navigation panel. 

Video Tutorial

Next Steps

After you’ve annotated the videos in your project, you can begin defining your workflow and training your model just like you would with a project with images. We hope to create a video annotation process that is as seamless and easy as our image pipeline despite the additional complexity of video output. In the future, we plan to introduce more features that help you annotate multiple frames and ease the pain of having to annotate every single frame on the video. 

Once your model has finished training, you can analyse its performance in our Aggregation Statistics tab to know whether it has been effective in learning from the input data. Given that video data might be heavily concentrated and similar, it is important to maintain data variety to make the model more robust. The aggregation statistics can help you better understand how you can improve your data quality and optimise the performance of your model.

Our Developer’s Roadmap

We are excited to expand Nexus’ capabilities to include end-to-end video annotation and processing. Video annotation is typically a tedious process, and our goal is to make it as easy and user-friendly as possible. Introducing video compatibility is just our first step: we plan to introduce additional tools in the future to further improve the user annotation process. These include features like advanced polygon interpolation and AI assisted object labelling across multiple frames for annotation. We hope that these tools can remove additional complexities and make video annotation as seamless as image annotation.