Navigation auf zora.uzh.ch

Search ZORA

ZORA (Zurich Open Repository and Archive)

Maintenance: On Wednesday, February 12th 2025, maintenance work will take place on the MariaDB servers from 9:45pm to approx. 10:45pm. During this period, ZORA and JDB will be unavailable. Thank you for your understanding.

SignCLIP: Connecting Text and Sign Language by Contrastive Learning

Jiang, Zifan; Sant Muniesa, Gerard; Moryossef, Amit; Müller, Mathias; Sennrich, Rico; Ebling, Sarah (2024). SignCLIP: Connecting Text and Sign Language by Contrastive Learning. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, 12 November 2024 - 16 November 2024. Association for Computational Linguistics, 9171-9193.

Abstract

We present SignCLIP, which re-purposes CLIP (Contrastive Language-Image Pretraining) to project spoken language text and sign language videos, two classes of natural languages of distinct modalities, into the same space. SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs, without directly optimizing for a specific task or sign language which is often of limited size.We pretrain SignCLIP on Spreadthesign, a prominent sign language dictionary consisting of ~500 thousand video clips in up to 44 sign languages, and evaluate it with various downstream datasets. SignCLIP discerns in-domain signing with notable text-to-video/video-to-text retrieval accuracy. It also performs competitively for out-of-domain downstream tasks such as isolated sign language recognition upon essential few-shot prompting or fine-tuning.We analyze the latent space formed by the spoken language text and sign language poses, which provides additional linguistic insights. Our code and models are openly available.

Additional indexing

Item Type:Conference or Workshop Item (Paper), original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
06 Faculty of Arts > Zurich Center for Linguistics
Dewey Decimal Classification:410 Linguistics
000 Computer science, knowledge & systems
Language:English
Event End Date:16 November 2024
Deposited On:05 Jan 2025 17:02
Last Modified:30 Jan 2025 09:36
Publisher:Association for Computational Linguistics
OA Status:Green
Official URL:https://aclanthology.org/2024.emnlp-main.518
Download PDF  'SignCLIP: Connecting Text and Sign Language by Contrastive Learning'.
Preview
  • Content: Published Version
  • Language: English
Download PDF  'SignCLIP: Connecting Text and Sign Language by Contrastive Learning'.
Preview
  • Content: Other
  • Language: English
  • Description: screen-reader-ready abstract

Metadata Export

Statistics

Downloads

5 downloads since deposited on 05 Jan 2025
5 downloads since 12 months
Detailed statistics

Authors, Affiliations, Collaborations

Similar Publications