Real-Time Voice Translation for Customer Service

A customer in the United States calls your support line in English. Your Hong Kong agent knows how to solve the problem but cannot explain the answer with enough confidence in Englis

The agent can transfer the call to a bilingual colleague, add an interpreter, or ask the customer to continue by email. Each option adds delay when the customer wants an answer.

DAL built a prototype to start another approach.

The customer speaks English. The agent hears the message in Cantonese and responds in Cantonese. The customer then hears the answer in English, usually within one to two seconds.

Both people remain on the same call. They do not need to install software, press translation buttons, or wait for another person to join.

Two pipelines run simultaneously inside a single call session — one for each direction of the conversation. Taking an English-speaking customer and a Mandarin-speaking agent as an example:

Customer to Agent

1. Amazon Transcribe converts the customer’s English speech to text in real time
2. Amazon Translate converts the text to Mandarin, applying a custom glossary
3. Amazon Polly speaks the Mandarin translation to the agent

Agent to Customer

1. Amazon Transcribe converts the agent’s Mandarin speech to text
2. Amazon Translate converts to English, applying the same glossary
3. Amazon Polly speaks the English translation back to the customer

Audio streams over a WebSocket the entire time. The system translates each clause as it stabilises — the other side hears the first part of your sentence while you are still finishing it.

What is real-time two-way voice translation?

Real-time two-way voice translation converts each speaker’s live audio into text, translates it into the other person’s language, and generates translated speech during the same call.

The system relays the conversation between two people. It does not answer for the agent or behave like a customer-service chatbot.

DAL’s prototype uses three AWS services:

1. Amazon Transcribe converts live speech into text.
2. Amazon Translate converts the text into the listener’s language.
3. Amazon Polly generates the translated voice.

The system runs this process in both directions at the same time.

How the AWS voice translation system works

Consider an English-speaking customer and a Mandarin-speaking service agent.

Customer to agent

The customer speaks in English. Amazon Transcribe creates a live transcript, Amazon Translate converts it into Mandarin, and Amazon Polly speaks the translation to the agent.

Agent to customer

The agent responds in Mandarin. The system transcribes the response, translates it into English, and generates English speech for the customer.

Audio streams between the browser and the backend through a WebSocket connection. The system starts processing each clause once it becomes stable instead of waiting for the full sentence.

In our prototype, processing took around 150 milliseconds after a phrase stabilised. The customer experienced a total delay closer to one or two seconds because the system also needed to detect the end of the phrase.

These figures describe prototype testing. Network conditions, audio quality, language choice, and deployment region can affect performance.

The two problems that determine whether people use it

A clean demonstration can translate a sentence in a quiet room. A real customer-service call includes product terms, accents, background noise, corrections, and interruptions.

Two areas have the greatest effect on the experience.

1. Industry terminology

Generic translation engines often struggle with the terms that carry the most business value.

A manufacturer may need to discuss tooling fixtures, material grades, part numbers, or tolerances. A financial-services customer may refer to a product name or transaction type. One poor translation can change the meaning of the request.

DAL adds the company’s terminology to both the recognition and translation stages.

The recognition layer expects the relevant product names, acronyms, and technical terms. The translation layer then returns the company’s approved wording.

A practical glossary may include:

• Product and service names
• Model numbers and part codes
• Technical specifications
• Delivery and warranty terms
• Industry abbreviations

A production deployment should test each critical term before the system handles live calls.

2. Interruptions

People interrupt each other during normal conversations. They correct details, confirm numbers, and respond before the other person finishes.

A translation system that keeps playing audio while the listener starts talking creates two competing voices.

DAL’s prototype includes a barge-in function. When one person starts speaking, the system stops the translated audio playing to them and removes any audio still waiting in the queue.

The deployment can use low, medium, or high interruption sensitivity. A quiet office may use a more responsive setting. A contact centre, shop floor, or warehouse may need a lower setting to reduce false triggers from background noise.

Why use a three-stage AWS pipeline?

A single speech-to-speech model may appear simpler. Some models can also manage interruptions inside one session.

DAL tested that approach. For this product, the staged pipeline gave us more control over:

• Language selection
• Approved terminology
• Playback and interruption behaviour
• Call transcripts and metadata

The system also needs to relay the agent’s words without generating its own customer-service answer.

A staged architecture makes it easier to separate what the agent said from what the technology translated. That distinction matters for customer trust, quality review, and regulated industries.

One translation core, multiple customer-service channels

DAL demonstrates the product in a browser because participants can join through a link without installing software.

The translation core works with streamed audio, so the same architecture can connect to:

• A customer-service phone number
• An existing PBX
• A SIP connection
• A cloud contact-centre platform
• A web-based support session

Each channel requires its own connector, authentication, and call-control setup. The business does not need to rebuild the recognition, translation, glossary, and interruption logic for each channel.

This allows a company to test one use case first and add other channels later.

Where real-time voice translation fits in Hong Kong

Regional customer support

A Hong Kong contact centre can support overseas customers without transferring every cross-language enquiry to a bilingual team.

Manufacturing and supplier coordination

Service teams can discuss components, delivery issues, and production requirements with overseas suppliers while preserving approved technical language.

E-commerce

Agents can handle order enquiries, returns, delivery problems, and complaints across several languages from one service operation.

Hospitality and travel

Reservation and customer-care teams can support overseas guests before arrival, during their stay, or after a booking issue.

Financial and professional services

Teams can use translation for routine enquiries and appointment coordination. Legal instructions, regulated disclosures, and high-value transactions may still require a qualified interpreter or bilingual specialist.

From translated calls to customer intelligence

The immediate benefit is communication. The longer-term value comes from the information produced during the call.

With appropriate consent and data controls, the business can send transcripts and call metadata into systems such as HubSpot and Amplitude.

Teams can then examine:

• Which language groups have the highest transfer rate?
• Which issues take the longest to resolve?
• Which terms cause repeated confusion?
• Which customer journeys generate multilingual support calls?
• Does translated support improve first-call resolution?

This follows DAL’s Signal → Decide → Act → Learn methodology.

The call produces a signal. The business identifies the source of friction, improves the service journey, and measures whether the next customer receives a better result.

Companies should define consent, transcript retention, encryption, and access control before storing or analysing translated calls.

The limits

Real-time voice translation is not live dubbing. The listener hears each translated clause after a short pause.

Synthesised voices may also differ from the customer’s preferred regional accent or speaking style. Noise, echo, and unstable connections can reduce recognition quality.

High-risk conversations still need human oversight. Medical decisions, legal instructions, emergency situations, and regulated financial transactions may require a qualified interpreter.

A pilot should test the system with real accents, terminology, devices, and call conditions before production use.

How to run a focused pilot

Start with one repetitive, lower-risk call type such as booking confirmation, delivery support, or supplier coordination.

Choose one language pair that already causes transfers or long handling times. Build a glossary using the terms agents use during those calls, then test the system with different speakers and background conditions.

Measure:

• First-call resolution
• Transfer rate
• Average handling time
• Translation accuracy
• Customer satisfaction

The pilot should also define when the agent must bring in a bilingual colleague or interpreter.

Keep the customer on the call

A customer will not judge your cloud architecture. They will remember the delay, the repeated questions, and whether anyone solved the problem.

Real-time two-way voice translation gives service teams another way to keep the conversation moving. The customer uses the language they know. The agent uses the language in which they can provide the best answer.

The technology stays between them without taking control away from either person.

Want to assess whether this architecture fits your customer-service operation?

Bring DAL one target language pair, one call scenario, and a sample of the terms your agents use. We will help you identify where the system fits, where human support remains necessary, and which results the pilot should measure.

Frequently Asked Questions

Does the customer need to install an application?

A browser-based session can work through a web link. Phone and contact-centre deployments can make the translation invisible to the caller but require backend integration.

Is the translation simultaneous?

The system translates stable clauses rather than providing word-for-word dubbing. In DAL’s prototype, the listener usually heard the translation within one to two seconds.

Can it connect to an existing PBX or contact centre?

Yes, provided the platform can send and receive audio through a suitable connector. The channel integration changes, but the translation core can remain the same.

DAL (Data & AI Lab) is the Customer Intelligence practice of Master Concept, helping Hong Kong and Asia-Pacific enterprises build data-driven customer experience infrastructure. We integrate and deploy Intercom, HubSpot, Amplitude, and the broader best-in-class stack.

📩 Want to assess how this architecture fits your customer service operation?