BESTBUY_VirtualTryOn

TEAM
Sunny Lee - UX Lead
Devansh Garg - Product Management
Justin Morgan - iOS Lead
Andrew Bradnan - iOS Development
Harrison Shapley - Android Lead
Alexander Biemann - Android Development
Suresh Peddisetti - React Native
Mike Gourdin - Quality Assurance Lead
Aaron Wedin - Quality Assurance
Sudharsan Uthayasuriya - 3D Modeling Lead
Vairam Vignesh - 3D Modeling
Jonathan Bergeron - UX Research
Georgia Carlsson - UX Content
Dave Kover - UX Standards
Vidhya Subramanian - Accessibility

DURATION
6 months

CLIENT
Best Buy - Seattle Technology Development Center

TOOLS
Figma, UserTesting.com, MIRO, 3DStudioMax, VRay, Blender, Reality Composer, Photoshop, Illustrator, After Effects, Lottie, PowerBI Analytics, OpinionLab, and JIRA.

PLATFORMS
iOS: ARKit, SceneKit, and RealityKit
Android: ARCore and Sceneform

OVERVIEW

In late 2022, following the pandemic, Best Buy retail stores faced substantial losses due to theft, leading the company to implement protective enclosures for high-value products to reduce shrink. This measure restricted customers’ ability to try on smartwatches and glasses in-store, negatively impacting sales and increasing return rates as shoppers were unable to accurately assess sizing before purchase.

To mitigate the risks associated with high-value items on shelves while still enabling customers to virtually try on wearables, we were tasked with developing augmented reality (AR) solutions to provide a comprehensive alternative to in-store fittings.

TARGET DEMOGRAPHIC

Through consultation with business intelligence expert Elyssa Mason, we determined that the success of smartwatches and glasses hinges on features, cost, and compatibility with users’ lifestyles. These devices appeal to a diverse audience, necessitating functionality across varying skin tones, eye colors, and hand sizes. To meet company diversity, equity, and inclusion (DEI) standards, we partnered with Applause and Best Buy’s accessibility teams to refine detection capabilities and user interfaces.

WEARABLES

Smartwatches appeal to a wide audience because they seamlessly integrate into daily life, offering a blend of practicality, health benefits, and style. Whether it’s staying connected, tracking fitness, or simplifying tasks, smartwatches provide a convenient and efficient way to enhance everyday routines.

Companies like Gunnar introduced blue light glasses to reduce eye strain, while Meta and Bose made AR audio glasses popular among cyclists. The growing demand for smart wearables meant that there was an opportunity for Best Buy to help customers find the right fit with ease.

Ray-Ban Meta glasses with open ear audio

Gunnar Blue Light Reduction Amber Lenses

DESIGN CONSIDERATIONS

I conducted a competitive analysis to better understand the market landscape and customer expectations. Surprisingly, I found few players in the space. Notably, Warby Parker stood out as one of the only major competitors equipped to handle legal challenges, such as those posed by the Illinois Biometric Information Privacy Act (BIPA). This insight highlighted the unique positioning of Warby Parker in addressing both market demands and regulatory complexities.

I also assessed the Wanna Watch app which showcased robust performance in virtual watch try-on experiences. By comparison, our experiments with Apple’s ML model revealed significant limitations. Specifically, the model was restricted to open-hand gestures, despite users naturally preferring a closed-fist position for such interactions. These findings underscored critical areas for improvement and provided valuable direction for refining our strategy moving forward.

After evaluating key players in the watch try-on space, we spent several weeks attempting to develop our own hand-tracking ML model. However, upon further investigation of Wanna Watch, we discovered they’d partnered with SNAP Inc. Recognizing the potential of leveraging SNAP’s proven technology, we reached out to explore the possibility of a licensing agreement.

PERMISSIONS

Following the Target class action lawsuit on April 26, 2024, which accused the company of violating the Illinois Biometric Information Privacy Act (BIPA), our legal team prioritized mitigating similar risks for our organization. To address this, I worked with Erin Anderson-Kipp (content) to design and develop a biometric permissions message. The goal was to ensure compliance with privacy laws, improve transparency, and clearly communicate to users how their biometric data is collected, stored, and used. This initiative aimed to build trust with users and reassure them that their data was not being collected.

FEATURE PRIORITIZATION

At this point, the team could discuss and work together to outline our strategy. Drawing on findings from the competitive analysis, we carefully assessed the potential influence of each feature. We also weighed whether users would likely expect specific capabilities in later versions, ensuring our development plan aligned with both customer needs and industry trends. This approach enabled us to prioritize features strategically and design a product that would connect with our audience.

After consulting with our Data Scientist, Rad Cyrus, and UX Research Director, Jonathan Bergeron, we got the confirmation we needed that our approach could significantly reduce theft and alleviate stress for in-store employees.

MODELING & LOADING

We kicked off the project by identifying our most popular glasses SKU. After downloading an existing 3D model from our server, I collaborated with our lead engineers and our on-staff modelers to identify the optimal points of contact for the glasses and watch categories. Once the pivot points were finalized, we created documentation and contacted vendors to ensure their 3D teams understood the updates.

The bridge of the nose was chosen as the pivot point for all glasses models and scaling was based on inter-pupillary distances (IPD), ensuring a custom fit for every user.

Due to the complexities of wrist tracking technology, extensive experimentation was required to determine whether this position could function as the pivot point. While it was ideal for scaling, achieving a stable positional lock without an open hand was uncertain. SNAP’s ML model enabled us to successfully secure a stable lock with a closed fist.

The models posed additional challenges when it came to loading. While working with Luxottica, we found their 30MB 3D models were too large for most app users—causing delays, especially in low-bandwidth environments. Since our models are typically under 5MB, we optimized five files for the launch and trained their team in mesh optimization. I also developed a loading system to display progress, preventing users from staring at a streaming video of themselves without a 3D model superimposed.

Preview of loading glasses

DYNAMIC MESSAGES

While the SNAP ML model was still being tested, I collaborated with UX Content Manager Georgia Carlson to design two messaging options. If the model proved ineffective, we would then use the prompt “Keep open hand in view.”

The message would surface when no hand or wrist was detected. Once detected, the message would disappear, and a 3D watch would be overlaid on the user’s wrist.

With glasses, a dynamic message wasn’t necessary. Internal Applause tests showed that users naturally understood that a hidden 3D glasses model meant that their face wasn’t detected. Customer feedback later confirmed this. We kept the design simple without extra alerts so we could better utilize resources that at the time were already being dedicated to producing a loading model.

UI NORTH STAR

We initially designed a bottom toolbar with a menu drawer for browsing related SKUs. Users could navigate between watch series by swiping the drawer or using on-screen gestures. The feature also included dimension tools to ensure accurate sizing.

For glasses, we adopted a similar user interface approach, differing primarily in the presentation of variations within the drawer and the display of dimensions to users.

MVP IOS

For the MVP we chose to reduce the features to the essential requirements, displaying product information and enabling SKU variation browsing through a carousel.

MVP ANDROID

For our Android developers, repurposing existing components from previous projects provided a quicker way to achieve functionality similar to the North Star design.

GLASSES DEPLOYED

The Glasses Try-On feature developed in partnership with Meta and Luxottica was deployed on October 17, 2023, enabling real-time AR visualization to assist users in selecting optimally fitting eyewear styles. This innovation enhanced decision-making precision while providing a personalized, interactive interface.

WATCHES LAUNCH

The Watch Try-On feature was launched on January 12, 2024, exclusively on Android. A functional iOS prototype was also developed and submitted to Apple for review but was not approved for release. This process underscored the political, technical, and platform-specific complexities of implementing advanced ML features across different ecosystems.

LESSONS LEARNED

A key mistake was inadequate communication with our Apple representative. Leadership had hoped that presenting a functional prototype would generate enthusiasm for the technology and its potential to connect users with their products. Obtaining Apple’s blessing from the beginning should have be a key requirement for moving forward. Without Apple’s approval, leadership chose not to advance beyond the MVP unless circumstances changed, leading to a halt in production.
Despite this, engagement with wearables and the virtual try-on feature remains strong, with placement rates significantly exceeding those for in-space product visualization.