I’m a human-computer interaction designer and researcher promoting digital inclusion in my work.
Education:
MA/MSc Global Innovation Design (RCA & ICL) - Distinction
BA Internet and New Media (Sun Yat-sen University) - First Class Honours
Experience:
I worked as an interaction designer for NetEase Game before I started my master’s.
Technological innovations have made life easier for many. However, others with capability limitations (either permanent, temporary, or situational) have disabilities thrust upon them by inconsiderate and inadequate interaction design.
The overarching theme of my practice and research has been designing inclusive interaction languages to enable the broadest possible audience to benefit from novel digital technologies.
I have a keen interest in exploring how different modalities and interaction metaphors can define and shape the inclusivity of digital technologies. By Integrating inclusivity as a fundamental guideline, we’ll be able to ensure accessible and human-centred experiences, revolutionizing the power and reach of emerging technologies at a crucial point of near maturity.
Please do get in touch if you fancy a chat.
Text entry and editing have always been cumbersome for visually impaired users. Current solutions for eyes-free text entry demand high cognitive and motor abilities. Mainstream touchscreen-based solutions usually involve both hands, making them difficult to use for visually impaired users with a cane in one hand.
Moception is a single-handed eyes-free text entry and editing method using a hybrid approach of speech and mid-air gesture input. Initiated by intuitive gestures, speech-to-text is used to input text content and correct the unwanted text using an ‘audio patching’ technique. Mid-air gestures detected by a wrist-mounted wearable are used to navigate through text contents, locate the insertion point, and perform discrete commands.
Moception applies principles of inclusive design and extreme-character methodologies. By addressing the needs of extreme users (visually impaired users) in extreme situations (with a cane in one hand), Moception provides a solution that could also be applied in many other contexts to benefit user groups beyond visually impaired users (e.g., elderly users or everyone with one hand occupied).
Special thanks to:
Wei Lin, staff of Tree of Life Disability Innovation Centre
Wrist-worn Wearable:
Moception presents a wrist-worn wearable for gesture recognition and audio/haptic feedback. Mid-air gestures will be detected by an embedded accelerometer. Audio feedback will be given through bond conduct headphones. Haptic feedback will be given through an embedded vibration module. With two stretchable junctions, the wristband enables eye-free interaction.
Mid-air Gestures:
Wrist-rotation:
Even with our eyes closed, we can still tell the positions of different body parts confidently. Taking advantage of proprioception, Moception maps the spatial layout of a line of text to the spherical space under our palm, which improves visually impaired user’s awareness and perception of the text layout. By rotating their wrist (as if when touching a football), users can control the insertion point and navigate through a line of text.
Gestures for discrete commands:
Low-key and intuitive gestures like swipe, draw a line were defined by 10 visually impaired participants in gesture study, following the participatory design paradigm proposed by Wobbrock et al. Each gesture was mapped to a basic command of the text entry system.
Moception aims to provide a one-handed eyes-free text entry and editing method that is intuitive, effortless, and efficient to use. Three user studies were conducted to understand and address the challenges in a holistic way.
User study 1 combined semi-structured interviews with 4 visually impaired participants and an online survey among 21 participants to define the challenges. Difficulties in orientation when locating the insertion point and wrong text correction were identified to be the two challenges Moception would tackle.
User study 2 was a participatory workshop where four prototypes making use of different modalities including head, arm, finger, and wrist, as well as the baseline method, were tested with 4 users and graded by them after their sessions. Wrist rotation for text review and audio-patch for text correction were the options that enabled the best user performance and provided the best user experience.
User study 3 was a gesture elicitation study where 140 gestures for 9 commands were defined by 10 visually impaired participants. The final gesture set was generated based on the participatory design paradigm (Wobbrock et al., 2009).
The final working prototype was built in Unity with a Leap Motion Controller for gesture recognition and tested with 4 visually impaired participants. The average completion time of task “Enter-review-edit” was cut down by 53.2% compared to the current speech-based input method on an iPhone. Moception was also perceived to be more effortless and intuitive to use.
Reimagination of inclusive human interactions in smart kitchens
Novina is a mid-air gesture interface that integrates all the interfaces on our kitchen appliances and a redesigned cupboard area that has a movable inner layer also controlled by the gesture interface. With simple and intuitive gestures like air tap, drag and move up on a near-body mid-air interface, users can interact with all of their kitchen appliances and move the inner layer of the cupboard to the desired position when they need to get a plate.
Gesture interactions in the Novina system were designed based on behavioral insights on modalities and interaction metaphors acquired from experiments. Direct manipulation by hand, World-in-hand spatial navigation are the two metaphors that proved to be intuitive and inclusive in a kitchen setting, which can significantly cut down cognitive loads and interaction burden.
The interactive prototype of Novina was tested with a Leap Motion gesture controller. The ‘exclusivity’ of Novina and the current kitchen solution was calculated with the Exclusion Calculator developed by Cambridge University. Based on the results, Novina is more friendly to users with different levels of motor, sensory and cognitive capabilities.
Cepi is a platform enabling victims of domestic violence to collect admissible audio evidence safely using speech modalities embedded in conversations. Users can start capturing audio evidence or start an emergency call by saying trigger words defined by themselves.
To help victims to make more informed decisions, the algorithms of Cepi help identifying abusive languages related to insult, threat and manipulation, and emotional cues indicating violence. Users can also see what other victims having audios with similar toxicity did - how many of their peers reported their cases, how many percent of them got a protective order etc.
Several disguise mechanisms were designed to ensure the victims’ safety when the abusers constantly check their phones.
Beyond its current scope, Cepi has the potential to be applied in many other contexts including toxicity estimation in court systems and evidencing in the contexts of sexual harassment and school bullying.
