MBUX Virtual Assistant
At CES 2024, Mercedes-Benz is showcasing a raft of developments that define its vision for the hyper-personalized user experience of the future – in-car and beyond. Headlining those is the new integrated MBUX Virtual Assistant. It uses advanced software and generative AI to create an even more natural and intuitive relationship with the car, with proactive support that makes the customer’s life more convenient. This game-changing development takes the ‘Hey Mercedes’ voice assistant into a whole new visual dimension with Unity’s high-resolution game-engine graphics. Running on the Mercedes-Benz Operating System (MB.OS) developed in-house, its rollout starts with the vehicles on the forthcoming MMA platform (Mercedes-Benz Modular Architecture). The Concept CLA Class, which celebrates its North American premiere at CES 2024, is based on this platform and likewise provides a preview of MB.OS.
“Through advances in artificial intelligence, tomorrow’s Mercedes-Benz will know its driver like never before. It will enhance and complement our customers’ lives – in their cars and in other areas too. We have already made strong progress on this journey. The latest proof points on show at CES provide an exciting glimpse of what lies ahead.”
Markus Schäfer, Member of the Board of Management at Mercedes-Benz Group AG, Chief Technology Officer
MBUX Virtual Assistant makes life easier with natural interaction and proactive intelligence
The class-defining MBUX Virtual Assistant running on MB.OS is the most human-like interface with a Mercedes-Benz yet. Combining the intelligent systems of MBUX into a single entity, it presents a new face to the customer with natural and empathetic interaction. The MBUX Virtual Assistant is a further development of the system first showcased in the VISION EQXX. It uses generative AI and proactive intelligence to make life as easy, convenient and comfortable as possible. For instance, it can offer helpful suggestions based on learned behavior and situational context.
“The Mercedes-Benz user experience of tomorrow will be hyper-personalized. With generative AI, our MBUX Virtual Assistant brings more trust and empathy to the relationship between car and driver. Thanks to our MB.OS chip-to-cloud architecture, our future vehicles will provide customers with exactly what they need when they need it.”
Magnus Östberg, Chief Software Officer, Mercedes-Benz AG
Four characteristics ensure the assistant is always in tune with customer needs and preferences
Four ‘personality traits’ underpin the MBUX Virtual Assistant – Natural, Predictive, Personal and Empathetic – blended to create a fully rounded whole. The underlying principle of each personality trait is to offer seamless interaction that leverages the best of voice and graphic interfaces. The customer can choose to speak with the assistant with or without using the keyword ‘Hey Mercedes’. Being predictive, the assistant can also offer proactive suggestions and routines based on situational context, learned behavior and using generative AI. At the same time, a more emotional neural voice that can express different speaking styles generates a sense of empathy. To ensure it is always a very personal assistant, the driver can tailor it to their own specific preferences via a range of options in a central hub.
Visual communication made more intuitive and emotional through powerful 3D graphics
The holistic approach pairs more natural dialogue with visual feedback through advanced 3D graphics from the Unity game engine. In the new MBUX Virtual Assistant, a ‘living’ star avatar uses animations to express different moods and states of being. Emotions range from calm to excited and even sensitive. Further animations indicate whether the assistant is speaking, listening, thinking, making a suggestion or providing a warning. Movement, brightness, intensity and color interact seamlessly to communicate intuitively with the driver.
MBUX Surround Navigation marries assistance with route guidance for a seamless experience
Integrated visual communication also enters a new dimension with MBUX Surround Navigation. Facilitated by the domain integration of MB.OS, this advanced feature pairs a real-world view of the car’s surroundings with its reactions to it in real time. The driver benefits from significantly enhanced situational awareness by being able to ‘see what the car sees’. The seamless incorporation of information into the driver display using 3D graphics from the Unity game engine provides a highly intuitive overview. It offers everything the driver needs to know at a single glance. For instance, it shows the type of traffic that is ahead by indicating another car, van, truck or even a cyclist. It also shows pedestrians close to the roadside and other potential hazards and superimposes route guidance into a realistic representation of the surroundings – particularly helpful in busy urban environments. As a result, the customer will be able to clearly see where exactly their next turn will take them.
More productivity, entertainment and convenience with new 3rd-party apps and content1
Over-the-air (OTA) updates ensure customers have access to the latest content and features tailored to their wishes and regional preferences. The new E-Class, for example, is already delivering these benefits – its infotainment domain is a precursor of MB.OS technology. Starting Q1 2024, productivity in the E-Class will be further elevated through the addition of the new Meetings app, where customers can participate in TEAMS meetings. This, paired with existing offerings with Zoom and WebEx, covers all market leader for video conferencing. The extensive update also includes the addition of further apps for everyday convenience. These include Booking.com, getAbstract, Just Eat Takeaway.com and more.2 In entertainment, there are plans to introduce cloud-based retro gaming from Antstream Arcade and further immersive experiences.
Video streaming offering to be enhanced with Sony Pictures Entertainment’s RIDEVU3
In an exciting development, the video streaming offering will be enhanced by an integration of Sony Pictures Entertainment’s (SPE) cutting-edge RIDEVU service. RIDEVU promises to elevate the new vehicle entertainment experience for Mercedes-Benz customers, granting them access to a diverse array of video entertainment directly within their vehicles. RIDEVU provides users with advanced controls and features designed for the car. It includes a curated library of films available for streaming, enriching the Mercedes-Benz entertainment bundle.
Hyper-personalization through digital art with MBUX Collectibles and more
MBUX Collectibles is Mercedes-Benz’s first in-car app to showcase digital art and collectibles based on NFTs. It enables the creation of a private art gallery by connecting the customer’s personal NFT wallet to the user interface. Users can also browse a digital art exhibition curated by Mercedes-Benz. On display at CES are NFT collections from Mercedes-Benz NXT within the app. The collection Mercedes-Benz NXT Icons Era 1 ‘Luxury’ is the first in a series of seven exploring the rich Mercedes-Benz design heritage. At CES, there is also a first sneak peak of Era 2 ‘Technology’, which is yet to be launched. And as an homage to last year’s CES, visitors can enjoy a first glimpse of the Mercedes-Benz NXT Superdackel NFT. A limited edition of 1000 Superdackel NFTs by Mercedes-Benz NXT will drop soon.
Concept CLA Class showcases MB.OS as the foundation for the hyper-personalized Mercedes-Benz
The first vehicles to run on MB.OS will be the family based on the forthcoming MMA platform. The Concept CLA Class, which celebrates its North American premiere at CES 2024, offers a close-to-production insight into the future range of four vehicles. To stress the central role played by MB.OS in shaping the customer experience, the interior of the show car offers a striking visualization of it. The complex multi-faceted architecture is represented by one of its many hardware components, the high-performance water-cooled chip from collaboration partner NVIDIA. The advanced three-display MBUX Superscreen hints at the role of the Mercedes-Benz of the future as a hyper-personalized part of the customer’s digital life. MB.OS provides the foundation for achieving this. It comprises four domains – Infotainment, Automated Driving, Body & Comfort and Driving & Charging – and the connectivity module linked to the Mercedes-Benz Intelligent Cloud. The chip-to-cloud architecture ensures deep integration into the vehicle and full control of all sensors and actuators. Furthermore, by decoupling the software and hardware innovation cycles, the company can ensure its vehicles are constantly up to date.
MBUX SOUND DRIVE with will.i.am
Mercedes-AMG has announced the launch of its latest state-of-the-art technology, MBUX SOUND DRIVE – an industry-first partnership with creative artist, innovator and entrepreneur will.i.am, that delivers a ground-breaking interactive music experience. The technology uses software that allows music to react to the way the car is being driven, creating a harmonious relationship between motion and melody. This turns the car into a virtual musical instrument, transforming an already exceptional Mercedes-AMG driving experience into an emotional, musical journey.
Utilizing a suite of sensors and advanced software, MBUX SOUND DRIVE interprets various driving dynamics – such as recuperation, acceleration, steering, and braking – and converts them into musical expressions, allowing the driver to ‘compose’ in real-time.
“At Mercedes-Benz, we want digital innovation to bring a smile to our customers’ faces through exciting new features that make a real difference to every journey. With MBUX SOUND DRIVE, AMG drivers are in control of the way their music sounds. It makes in-car entertainment much more immersive.”
Markus Schäfer, Member of the Board of Management of Mercedes-Benz Group AG, Chief Technology Officer
“Imagine a world where your car can become an instrument creating musical journeys. MBUX SOUND DRIVE enhances driving and sound technology, letting motorists reshape music just by driving. I’m excited to see how composers, producers, and songwriters will harness this to create new works and reimagine classics for motorist to drive to. MBUX SOUND DRIVE is not only a new frontier for music creation, but also opens up fresh, exciting and interactive listening experiences for drivers.”
will.i.am, Creative Artist, Innovator and Entrepreneur
From the studio to the streets – a unique sound experience on the road
The creation of MBUX SOUND DRIVE takes the long and successful brand partnership between Mercedes‑AMG and multi-platinum rapper will.i.am into exciting new territory. As the project progressed, will.i.am fostered the idea of using his skills as a record producer to develop an interactive music experience for the car. The technology connects the car hardware with the music software through precise ‘in-car signals’ that enable the music to react to the driving characteristics in real-time.
Platform power – digital driving music for everyone
Providing an immersive sound experience is just one part of the story. The long-term goal with MBUX SOUND DRIVE is to turn the technology into an open music platform. As a result, artists from around the world will be able to create tracks, representing any music genre, in a new immersive audio format using MBUX SOUND DRIVE studio software.
MBUX SOUND DRIVE rollout starts globally from mid-2024 and will be available on Mercedes-AMG and Mercedes-Benz vehicles equipped with the second-generation MBUX system.1 Existing Mercedes customers can access the MBUX Software Update via an over-the-air (OTA) update using the Update Wizard on their MBUX display or the Mercedes me app after receiving a notification that it is available in the U.S.
Dynamic demos at CES – immersive music on the road
To fully demonstrate the capabilities of MBUX SOUND DRIVE, Mercedes-AMG is offering exclusive test drives at CES 2024. Visitors can head outside to the MBUX SOUND DRIVE stand at the Diamond Lot and book an experience in a Mercedes-AMG EQE Sedan or EQE SUV.
Talking tech – an extensive guide to the musical masterpiece
Mercedes-Benz is also offering media and visitors the opportunity to take part in an entertaining and informative discussion on this exciting technology. The topic will be presented by Mercedes-Benz CTO Markus Schäfer together with will.i.am, who will explain the creative vision, the science, and future plans. The presentation will take place on January 9 at 2:30 PM PST at the Mercedes-Benz booth (Las Vegas Convention Center, West Hall, Booth 4941).