Activity Detection

A Jupyter Notebook tool developed to facilitate the testing of ML models in human-object interaction.

GitHub

The Activity Detection Toolkit is a project undertaken by SIT and NVIDIA, aimed at advancing the field of human-object interaction (HOI) through the development of a Jupyter Notebook tool. This tool is designed to assist in the testing and experimentation of machine learning models dedicated to activity detection, leveraging synthetic datasets.

Image

Watch the demo here

Demo Video

Key Components

Data Exploration

  • Interact with video data from the Toyota Smarthome project.
  • UI components for video selection and playback.

Inference

  • Perform inference using pretrained HOI models.
  • Output video with detected activity captions.

Training

  • UI elements for dataset selection, model initialization, and training configuration.
  • Visual training progress indicators and integration of trained models for subsequent inference.

Testing

  • Evaluate models with visual progress indicators and detailed results for model performance assessment.

Pipeline Configuration

  • Toggle between different pipelines (e.g., TSU, NVIDIA STEP) with UI elements and modular .py files.

Feature Extraction

  • Script for validating and updating train/test splits based on available video data.

Objectives and Collaboration

This tool represents a significant step forward in the domain of activity detection. By streamlining the process of testing different inputs, models, and configurations in the HOI ML pipeline. The toolkit is designed with clarity and usability in mind, ensuring that it can serve as a valuable resource for ML practitioners and researchers alike.