The starting points for this project were designerly explorations of bodily engagements and the assumption that experiencing one’s body, movement and/or shared bodily actions could be enhanced, enriched, altered, re-invented, and/or challenged by interactive designs utilizing Machine Learning in order to become (more) accessible to more people. Our task was to address and potentially go beyond existing practices related to the sub-theme (i.e. Combat and focused force - e.g. contact sports) and question what is the core of the practice. This includes thinking not only in obstacles to people, but also in people’s resources and their lived life. The project has its focus on fencing and is based on insights that suggest that:
To come up with a concept we had to consider a few different aspects. Among them were bodily joy, applying oneself physically; the social side of things (i.e. not just centered on the individual), and the sensorially rich experiences.
En-garde is an AI application that uses machine learning to make fencing accessible to more people by giving them the opportunity to practice fencing in the comfort of their homes. All you need to do that is a computer and a mobile phone camera.
The app challenges the main notion that you need to be physically present to be able to fence by implementing a digital platform for a multi-layer experience.
<p-under-img>En-garde gives users the chance to try out and practice basic bodily movements and exercises (e.g., stance, lunge, footwork) while offering suggestions on how to improve one’s posture and movements. These could be found in the form of tutorials, drills, and a variety of pre-planned programs to choose from. Furthermore, the app’s gamification factor consisting of various challenges scaled up to several difficulty levels increases participation and proves that you do not need to be on-premise and interact with others in order to be a good fencer.<p-under-img>
This project was built using the Double Diamond design framework. As a starting point we spent some time to ponder about different practices, their virtues (e.g. gentle force, bodily control, self-regulation, culture/ninja-association, ...), actions (2-person ‘fight’, group meditation, sword demonstration, rituals, ...) and accessibility (physical access, sensory issues, social interplay, feeling part of a team, …).
After pragmatically deciding on the practice to work with (i.e. fencing) we did a desktop and field research and conducted interviews with fencing professionals and students. The purpose of the interviews was to gain a deeper understanding of the fencing lifestyle, explore what do people think of fencing, what skills are practiced in fencing, what are the main challenges and how does the fencing club address accessibility issues. During the field observation we were interested in the visual representations, what are people doing when they are not fencing and the verbal and non-verbal communication.
<p-under-img>In addition to that, we were able to participate in a beginners class and experience the sport ourselves. This helped us identify design opportunities and define the target group. Following that, we explored Machine Learning as a design material and looked into how and what can the machine be trained to recognize.<p-under-img>
Later in the process we did a workshop where we taught participants some of the fencing movements we had learnt using fencing equipment and observed them trying them out to identify similarities in their performance and struggles. This was combined with a brainstorming exercise to collect some data and jumpstart our creative ideation process and a questionnaire to reflect on their fencing experience.
Using the insights from our field research we were able to identify a design opportunity, formulate a HMW and develop our design vision:
To prototype and test the concept we used a combination of Lobe and Runway ML to bring the concept to life.
The model we created acted both as an implementation prototype and a means for material exploration with machine learning and programming in java. It helped us widen our vision on what is possible in the field of Fencing and Machine Learning. Furthermore, it taught us what approach to take when defining fencing with mathematics and what kind of visual feedback to use to ensure a more pleasant experience.
<p-under-img>To do that we considered nuanced feedback for the user. We started with a simple white stick figure. However this was later changed because it told very little about whether or not you are doing the movement right. In the next iteration we added an orange and green color to indicate when the posture is (almost) correct. Nevertheless, this made the users frustrated as they were looking to achieve the perfect posture. To avoid that, we implemented a color gradient where wrong angles made the limbs turn into more of a red shade and better posture resulted in a greener color of the limbs. Furthermore, we thought about adding semantic feedback in addition to the color of the angles. The performance of the model made it difficult to test with users and was, therefore, used mainly for internal insights.<p-under-img>
We used Lobe to analyze the users’ posture and offer suggestions for improvement based a on a trained data model. To train the model we started by creating a list of the most common mistakes associated with each position - the stance and the lunge. While working on the first model we noticed that most of the estimations were incorrect due to the the huge similarity in most pictures. However, this helped us define new labels and parameters for the next version of the model. The second model was divided into 2 smaller ones to avoid confusion, using the updated labels.
To validate our concept we wanted to challenge some of our assumptions related to our models and overall concept design. We selected 8 assumptions to test using a questionnaire and a set of exercises to understand how do users perceive and interpret the various design and interaction components. The test resulted in 5 main takeaways that were later used to revise our concept. The adjustments of the concept included the inclusion of semantic feedback in the drills and the challenges. This was based on the analysis of the test where it became clear to us that abstract feedback was not entirely clear to the participants. Most of the users were able to guess correctly but we came to the conclusion it would be more clear to represent it with numbers and letters too.
<p-under-img>During the testing phase it also became apparent to us that it is important to show what is expected of the participants (especially in the challenges). A good example of a clear introduction to an activity are the introductions to the Mario Party mini games where one can see the controls and how it would play out in real time. The drills, on the other hand, would be better explained by having the ghost figure show the participants what to do, since this feature is more about repeating a certain movement in an exact manner.<p-under-img>
<p-under-img>We noticed that the communication of the basics during the tutorial are crucial for participants to understand the basics of fencing based off a video. During the workshop we realized that the best course of action was to first explain what is the final goal and then go into details about what comes first and next. We also noticed that it was easier to teach participants by doing the movements alongside them using both visual and auditory feeback. As for the application, this would be in the form of voiceover guide as opposed to just a video as we had previously planned.<p-under-img>
Aside from the semantic feedback for counting, we decided to use abstract feedback for supporting the user in improving their lunge (similar to in Wii Fit). A circle is used to signify where to hold a certain body part. Through testing we learned that people connect an area to where their limbs need to be.
