Despite difficulties such as the postponement of the Games and the decision to allow no spectators, the HSR team continued to pursue a human-centered approach with the omotenashi robot. This article shares their key learnings from that experience.
As a worldwide partner, Toyota supported the Olympic and Paralympic Games Tokyo 2020 not only with vehicles, but also with a variety of robots. This three-part report highlights the tireless efforts of project members behind the scenes of the Games. Part 2 focuses on the HSR project team.
In 2012, the team began developing the Human Support Robot (HSR) as a robot assistant for household tasks. At a height of approximately 100cm, the HSR stands around the eye level of a wheelchair user. Its highly maneuverable, compact cylindrical body sits atop an omnidirectional base and features a folding arm which can move vertically and carry loads of up to 1.2kg. This enables the HSR to pick up objects off the floor, retrieve items from shelves, and perform other household tasks.
In 2015, Toyota teamed up with several research institutions in Japan and overseas to establish the HSR Developers’ Community, lending HSR units for use in research. This network has since grown to 46 facilities in 14 countries (as of January 2022), which collaboratively develop the technology by sharing software, expertise, and other R&D results. The team is rapidly working towards commercial release with continued improvement based on feedback from care workers and users with disabilities.
Tandem development driven by user perspectives
The HSR was initially chosen to provide mobility and spectator support for visitors using wheelchair-accessible seats at the National Stadium during the Tokyo 2020 Games.
Specifically, HSRs were to assist wheelchair users by 1. Guiding them from entrances to spectator seats to reduce the stress of moving through crowded areas, and 2. Distributing bottled water to ensure carefree enjoyment of the Games at the height of summer.
Two additional services by the HSR were available for all visitors including: 3. Mobility-related support involving garbage collection when leaving the venue, ensuring visitors did not need to carry unnecessary items home, and 4. Providing a “photo-taking service” to record once-in-a-lifetime memories.
“The HSR was originally developed to assist with tasks in the home, so we needed to improve various aspects to make sure it could move freely in the unique environment of a sports stadium,” reflects Masahiro Takahashi of R-Frontier Division.
Masahiro Takahashi (operations leader)
Unlike indoor living spaces, moving around a vast stadium required that the HSR know its exact location.
To make that possible, we first modeled a map of the venue. We equipped the HSR with front and rear laser sensors, known as LiDAR, which allow it to measure distances to target objects and obtain information about its surroundings. However, in walkways filled with people, these lasers may be unable to reach a wall, preventing the HSR from accurately measuring its position using LiDAR alone. As such, we also installed two upward-facing cameras that can discern the robot’s location by matching footage to pre-recorded ceiling patterns.
Combining these four different-purpose devices enables the HSR to safely navigate a crowded concourse. The rear-facing LiDAR can also monitor the wheelchair user the robot may be guiding.
These features weren’t included until we began developing the HSR for Tokyo 2020. Instead of full automation, for the Games, we went with a combination of autonomous functions and remote control by human operators watching via a monitor. This approach allows the human operators to chat with customers while serving them.
To make the HSR easier for operators to control, we also changed the front camera to a wide-angle lens, expanding the field of view from 65 to 165 degrees.
We also worked to reduce noise. During trials at Toyota Stadium in 2018, we found that loud cheering created so much noise that the operator could not hear what was said, despite the HSR being right beside the wheelchair user. We changed the microphone to ensure operators could clearly hear the user’s voice interacting with the HSR.
For all these specifications, we worked with operators to fine-tune the sensory aspects of the technology through repeated trial and error.
Alongside these technical updates, the team also worked closely with remote operators to determine what kind of support was actually needed by visitors in wheelchairs. One of the key figures who provided many insights was Remi Murase of Toyota Loops, a wheelchair user herself.
Remi Murase (operator)
The original plan was to distribute water as a welcome drink upon arrival. However, from the perspective of a wheelchair user, receiving this before getting to their seat creates extra baggage that makes it harder to move around. In that respect, I suggested that the development team minimize the burden by handing out water once people had gotten to their seats.
Similarly, there was a suggestion that handing out water bottles in plastic bags would be convenient, but this presented another potential burden. Someone with upper body paralysis, for example, would have difficulty removing the bottle from the bag.
One of my colleagues at Toyota Loops* has an impairment of hand function, so I asked the person to try out the HSR to figure out what heights and positions were easiest to receive the water. We continued to fine-tune the robot in this way.
*Toyota Loops is a special subsidiary of Toyota, established with the desire to expand the employment opportunities and social networks of people with disabilities.Development was progressing, driven by the perspectives of both engineers and wheelchair users, who offered the unique knowledge of firsthand experience. It was at this point that the Games were officially postponed due to the spread of COVID-19, forcing the team to consider changing or abandoning their implementation plans.
Following discussions with the Tokyo Organising Committee of the Olympic and Paralympic Games, it was decided that the HSR would only be used at the Tokyo 2020 Paralympic Games. At the same time, the area to be serviced by the robots was expanded from a section of 1F stands with wheelchair-accessible seating to all of the floor’s outer corridors and entrance gates. However, since the event would be held without spectators, the HSR would no longer guide spectators to their seats, instead distributing water, collecting trash, and taking photographs for event staff.
With an expanded service area, development was restarted with greater focus on remote control by operators.
In pursuit of human-centered services befitting Toyota
Prior to postponement, the service “involved little real-time conversation between visitors and operators, who would simply repeat the same set of phrases,” says Murase. “I felt that I wanted to offer a more personal service.” Following the postponement, the scope for operator participation grew, and the plan was revised to include more natural conversation with users. The team set about to instill these robots with human warmth.
One aspect was the movement of the HSR’s head. “The operator’s face is displayed on the HSR’s monitor, so we tried tilting the angle of the HSR’s head to match the user’s eye level, ensuring they could speak face-to-face with the operator on screen,” explains Murase. “These movements came about through discussions between operators during testing.” They have since been added to the manual as basic operations.
Consciously including other human gestures, such as nodding during conversation or turning the body and using arm movements in responses when asked for directions, completely transformed the appearance of the HSR’s motions.
During the actual Tokyo 2020 Paralympic Games, operators controlled the units remotely from Toyota Loops (in Toyota City), making the robots at the stadium behave as though each had its own personality.
As the HSRs roamed the spectator areas, operators chatted remotely with event staff while taking photos and avoided collisions by calling out to alert passers-by who may not have noticed the moving robot’s quiet hum.
Murase also made it a daily habit to always greet staff at the stores her HSR passed during operations.
“I thought it would be fun to have a robot greet you every time you saw it. As I kept doing so, store staff would notice it was me inside the robot and strike up conversations more often. We naturally got to know each other, and more people began to greet me and say hello,” says Murase, who gradually came to be recognized by people involved in the Games.
One memorable part of the job for Murase was the interaction she had with foreign media. “Despite the robot being an intermediary, I wanted people to feel a human presence. As a joke, I moved the HSR in front of a fan at the edge of the seating area and pretended to cool off, when someone from the foreign media called out, ‘You enjoying the breeze?' in English. I was able to convey a joke even through the robot, and to a person from overseas, no less. That episode gave me a great deal of confidence as an operator,” she said. Images of such interactions were shared around the world.
Upon witnessing such exchanges, Takemitsu Mori from R-Frontier Division realized that “an operator can still convey their spirit of hospitality, even through a robot.”
Takemitsu Mori (engineering leader)
Ultimately, I believe interpersonal communication is about feeling comfort and warmth by engaging with others.
Commemorative photos taken by our robots during the Tokyo 2020 Paralympic Games numbered in the hundreds. By having people at the core of our service, I believe we spurred more communication, which helped create lasting memories for so many visitors.
This was only possible with the input of the operators at Toyota Loops. This project arose from a strong desire to create and deliver a service that brings happiness to users. In the tense atmosphere of stadiums with no spectators, I’m pleased that we were able to offer a service that created memories and moments of comfort for many of the people involved in the event.
We saw the potential of using our robots (which combine remote control and autonomous functions) to provide distinctly Toyota-style services that involve reaching out to those in need, being there for them, and lending a helping hand.
It’s not only about simply replacing a person’s job with a robot, but finding ways for them to support and work in symbiosis with humans. This human-centered approach has sparked a great deal of communication.
On the final day of competition, the 23 deployed HSRs were stationed evenly around the stadium’s first floor perimeter to bid a final farewell to Games officials.
“While some of the HSRs were in autonomous mode without operators, those with operators received smiles and waves from everyone on the other side of the screen.”
These scenes reaffirmed that, albeit remotely, Games staff and operators were connected in spirit through the HSR.
Unbound by location – the hidden potential of robots
The experience of Tokyo 2020 provided the HSR team with tremendous insights. For Murase, the joy of her first customer service experience added to the sense of accomplishment of being part of the Games.
“The Paralympics were like a dream come true. Above all, it was my first experience of customer service. I’d never dreamed that one day I would be serving customers. Given the career options for people with disabilities, I had long given up on anything other than a desk job in an office, so it was a surprise to learn that robots could open up new paths.”
This project also offered hope to those with limited job options.
Murase’s experience uncovered further prospects for developing the HSR, according to project leader Takahiro Toda.
Takahiro Toda (project leader)
We still don’t know whether combining remote controls and autonomous functions in a robot is the ideal approach for a service. However, Tokyo 2020 has reaffirmed that for people who require care, have disabilities, or perhaps are caring for family or children, robots can offer value that is not bound by the concept of “location.”
Simple, repetitive tasks can be left to robots while humans engage in enjoyable work. I think achieving this may be a big step in reforming the way we work and creating new jobs. With this vision in mind, I want to re-examine HSR development in terms of what is to be done by operators and what should be automated.
Just as the internet has made teleworking easier, robots will bring the next change in the way we work. As we continue to develop this technology, we want to be conscious of the fact that it not only assists in people’s daily lives but also has the potential to create jobs.
Robots like the HSR need not be bound by location. Looking beyond lifestyle assistance to the further possibility of creating jobs, the HSR team is taking the next leap forward.
Toyota is the Official Worldwide Sponsor of the Olympic and Paralympic Games