Articles producció científicaEnginyeria Informàtica i Matemàtiques

StreamSense: Policy-driven Semantic Video Search in Streaming Systems

  • Identification data

    Identifier:  imarina:9441490
    Authors:  Finol G; Gabriel A; García-López P; Gracia-Tinedo R; Liu L; Docea R; Kirchner M; Bodenstedt S
    Abstract:
    Streaming systems are an increasingly appealing substrate for managing video data via the stream abstraction. However, if we consider a large stream collection, it can be hard for data scientists to discover and locate relevant videos, let alone specific video fragments. In this paper, we propose StreamSense: a policy-driven, semantic video search solution for streaming systems. StreamSense allows users to deploy AI models that generate embeddings from video frames via policies. Our system uses such embeddings for building a two-level index in a vector DB that efficiently handles inter/intra video queries. StreamSense abstracts users from vector DB interactions so they can perform semantic search using images as input and visualize the results. We built our prototype on top of a tiered streaming storage system (Pravega) and validated it on a health-related use case. We show that StreamSense allows data scientists to search for video fragments in real surgery datasets in < 30ms. StreamSense also reduces data ingestion related to AI training data loading in +80% compared to simple bulk loading video streams.
  • Others:

    Link to the original source: https://dl.acm.org/doi/10.1145/3700824.3701097
    Funding program action: Research and innovation programme
    APA: Finol G; Gabriel A; García-López P; Gracia-Tinedo R; Liu L; Docea R; Kirchner M; Bodenstedt S (2024). StreamSense: Policy-driven Semantic Video Search in Streaming Systems.
    Paper original source: Middleware Industrial Track 2024 - Proceedings Of The Middleware Industrial Track, Part Of: Middleware 2024. 29-35
    Article's DOI: 10.1145/3700824.3701097
    Funding program: Horizon Europe
    Journal publication year: 2024
    Entity: Universitat Rovira i Virgili
    Paper version: info:eu-repo/semantics/acceptedVersion
    Record's date: 2025-02-18
    URV's Author/s: García López, Pedro Antonio / GRACIA TINEDO, RAÚL
    Project code 2: 101092646
    Department: Enginyeria Informàtica i Matemàtiques
    Licence document URL: https://repositori.urv.cat/ca/proteccio-de-dades/
    Acronym: NEARDATA
    Publication Type: Proceedings Paper
    Author, as appears in the article.: Finol G; Gabriel A; García-López P; Gracia-Tinedo R; Liu L; Docea R; Kirchner M; Bodenstedt S
    Project code: 101092644
    licence for use: https://creativecommons.org/licenses/by/3.0/es/
    Author's mail: pedro.garcia@urv.cat
    Acronym 2: CLOUDSKIN
  • Keywords:

    Data streams
    Semantic search
    Vector embeddings
    Video analytics
  • Documents:

  • Cerca a google

    Search to google scholar