Manual transcription still beats AI: A comparative study on transcription services

From Hashes to Ashes – A Comparability of Transcription Providers. Credit: CISPA

A analysis crew from the Empirical Research Help (ERS) at CISPA Helmholtz Heart for Data Safety has carried out a scientific comparability of the most well-liked transcription companies. The comparability concerned 11 suppliers of handbook in addition to AI-based transcriptions.

It exhibits that, good high quality however, the latter nonetheless have issues with speaker attribution and that there are discrepancies between recording and transcription that distort which means. Whisper AI from OpenAI delivered the very best outcomes among the many AI suppliers.

Interviews are a well-liked technique for amassing scientific data. There’s a primary distinction between quantitative and qualitative interviews. Whereas the previous is designed to acquire statistically usable data from numerous contributors with the assistance of standardized questionnaires, the latter is aimed toward acquiring interview information that permit for interpretation by the researchers.

A particular kind is the guided interview, in which there’s a ready record of questions, which may, nevertheless, be deviated from through the interview. “In cybersecurity research, these interviews are utilized when exploring the patterns of action and interpretation of actors who operate through digital means,” explains sociologist Dr. Rafael Mrowczynski from CISPA’s Empirical Research Help (ERS) crew. The ERS crew advises the Heart’s researchers on methodological points.

Changing an audio file into textual content

Transcription is an important step in qualitative information evaluation. “The standard procedure is to convert the audio recordings of the interviews into text. It is important for the quality of the data that the transcriptions are adequate,” Mrowczynski explains. Relying on the scientific area, there are completely different requirements for transcription.

“In cybersecurity research, we usually work with transcripts that precisely reproduce the content of the conversation,” says Mrowczynski. An sufficient transcript, due to this fact, solely comprises the related spoken phrases. The researchers can get hold of the transcript in two methods: Both it’s created by the analysis crew itself, or the duty is outsourced to third-party suppliers.

Among the many third-party suppliers, moreover handbook transcription, there has not too long ago been actual hype about automated, AI-based transcription. That is because of the exponential leaps in growth and high quality that AI purposes have skilled in lots of areas during the last two years.

The researchers from CISPA’s ERS crew wished to know which supplier available on the market achieves the very best outcomes and the way automated, AI-based transcription performs as compared with handbook transcription. The objective was to have the ability to present the researchers at CISPA and the cybersecurity group with a advice for working with qualitative interviews.

The strategy of the ERS crew

For his or her analysis venture, Mrowczynski and his colleagues Dr. Maria Hellenthal, Dr. Rudolf Siegel, and Dr. Michael Schilling created a take a look at dataset. This consisted of particular person interviews lasting about ten minutes and group discussions with CISPA researchers in German and English. The content material centered on the analysis area of cybersecurity.

“It was important that technical terms from the community were included so that the precision of the transcription could be assessed,” Mrowczynski explains. A few of the interviews have been moreover enhanced with background noise to be able to mirror actual settings in on a regular basis analysis higher.

The information have been despatched to eleven suppliers in December 2022. Amongst these have been the transcription companies Amberscript, GoTranscript, QualTranscribe, Rev, and Scribbl, in addition to the AI-based transcription suppliers Amazon Transcribe, AssemblyAI,, Google Cloud, Microsoft Azure, and Whisper by OpenAI.

For the evaluation of the obtained transcripts, Mrowczynski and his colleagues created a reference transcript that served as the idea for the comparative evaluation. The evaluation itself then centered on two central standards. First, the researchers assessed the phrase error charge, which signifies by what number of phrases a transcript differs from the reference transcript. Second, the qualitative deviation from the reference transcript was coded manually.

Guide transcription companies beat AI

Of their paper, Mrowczynski and his colleagues conclude that, normally, “most of the manual transcription services achieve a commendable level of performance, while AI-based services often show meaning-distorting discrepancies between recording and transcription.”

The distortion of which means will be clearly seen in technical phrases; Mrowczynski explains, “In the transcript, for example, the term ‘hashes’ became ‘ashes.” That’s how we got here up with the title of the paper.”

OpenAI’s Whisper achieved the very best outcomes among the many AI-based suppliers. Most suppliers dealt with English higher than German. Three suppliers didn’t provide transcription for German in any respect. Background noise typically had a unfavorable impact on the consequence. The AI-based suppliers notably had issues with speaker assignments.

As well as, the transcripts created by an AI needed to be reformatted earlier than it was potential to additional course of them in software program for qualitative information evaluation. Nonetheless, the researchers level out that their evaluation displays the cutting-edge as of December 2022 and that present developments couldn’t be taken into consideration.

The analysis was presented on the 2023 CCS ACM Convention on Laptop and Communications Safety.

Extra data:
Rudolf Siegel et al, Poster: From Hashes to Ashes – A Comparability of Transcription Providers, Proceedings of the 2023 ACM SIGSAC Convention on Laptop and Communications Safety (2023). DOI: 10.1145/3576915.3624380

Supplied by
CISPA Helmholtz Heart for Data Safety

Guide transcription nonetheless beats AI: A comparative examine on transcription companies (2024, April 5)
retrieved 5 April 2024

This doc is topic to copyright. Other than any truthful dealing for the aim of personal examine or analysis, no
half could also be reproduced with out the written permission. The content material is offered for data functions solely.

Click Here To Join Our Telegram Channel

Source link

When you’ve got any issues or complaints relating to this text, please tell us and the article shall be eliminated quickly. 

Raise A Concern

Show More

Related Articles

Back to top button