Toyota's violin-playing robot first drew attention at Expo 2010 Shanghai. Though its stage has changed, its music continues to this day. This year, a technical issue reunited it with two former developers for the first time in over a decade.
With bow poised, the robot presses the strings with its left hand. As its right arm slowly begins to move, the violin’s sound fills the room.
Located in Nagoya, the Toyota Commemorative Museum of Industry and Technology traces the Toyota Group’s history of monozukuri, from looms to automobiles. In one corner of the museum, a robot performs for visitors.
Meet Pearl, the violin-playing robot.
Toyota’s robot violinist made its debut at the Japan Pavilion during the Shanghai World Expo in 2010, performing the Chinese folk song “Mo Li Hua” (Jasmine Flower). Today, Pearl has a repertoire of four pieces, including “Canon” and “Pomp and Circumstance,” and performs two of them five times a day.
In late June, two of the robot’s former developers, Hiroki Izu and Makoto Kobayashi, paid a visit to the Toyota Commemorative Museum of Industry and Technology.
At the time, both were part of Toyota’s Partner Robot development division. Today, Izu is with the Advanced Mobility System Development Division and Kobayashi with the New Business Planning Division, each working on new technologies and business initiatives.
Reunited with the violin-playing robot after more than a decade, the pair first looked on with nostalgia. Before long, however, they were examining it with an engineer’s eye.
Unbeknownst to them, people behind the scenes had been working for years to keep the robot running.
Building robots for people
Since 2000, Toyota has drawn on technologies and expertise developed at its manufacturing genba to create Partner Robots that work alongside people and contribute to society.
When Izu joined Toyota in 2004, his background in robotics led to an assignment on the Partner Robot development team.
Hiroki Izu
Almost immediately, I was tasked with putting the finishing touches on the instrument-playing robots for Expo 2005 Aichi. The expo was held the year after I joined Toyota, so I was thrown into it almost right away.
At Expo 2005 Aichi, a band of eight robots performed at the Toyota Group Pavilion, playing trumpets and other wind instruments. The sight of humanoid robots playing musical instruments drew widespread attention, and the performance became one of the pavilion’s most popular attractions.
While the expo was still in full swing, the team was already taking on its next challenge: developing a robot that could play the violin.
For a robot, playing the violin requires even more complex control than playing a wind instrument. While the left hand presses the strings at precise positions, the right hand controls the speed, pressure, and angle of the bow. Both arms must work in coordination to produce a beautiful sound.
The first prototype was relatively simple: a mechanical bow was drawn across a violin fixed to a stand. Yet even producing a proper sound proved difficult. Searching for clues, Izu and other members of the development team went so far as to attend meetings of the Acoustical Society of Japan.
Kobayashi, who joined Toyota two years after Izu, arrived in the midst of this trial-and-error process.
Makoto Kobayashi
There was this strange “screech, screech” coming from the development room. I had just started at Toyota, and I remember thinking, “How is this ever going to work?” (laughs).
Despite the difficulties, development gradually moved forward.
A standard violin was modified with a metal plate so it could be held by the robot, and a vibrato mechanism was incorporated into the left hand. To produce a performance with dynamics and subtle variations in tempo, Toyota invited a professional violinist into the lab and recorded the player’s arm movements using motion capture. The captured movements were then incorporated into the robot’s motion.
Hiroki Izu
We tried programming the motions from scratch, but they always felt flat. They lacked a human touch.
Thanks to these efforts, the violin-playing robot took the stage at Expo 2010 Shanghai. Its coordinated arm movements, complete with vibrato, delighted many visitors.
Away from the spotlight, however, there were also some nail-biting moments for the team.
"Your flight is booked. Shanghai tomorrow."
One day, one of the performing robots slipped and fell over, leaving it unable to go on stage. Although the team had a backup, there was no guarantee the same thing wouldn’t happen again. Tensions mounted among the team as they wondered whether the robots would make it through the rest of the Expo.
That evening in Japan, Kobayashi received a call from his supervisor, asking him to come immediately to the Hirose Plant, the Partner Robot development hub.
“I’ve booked your flight. I need you in Shanghai tomorrow.”
Kobayashi had been in charge of the robot’s electrical systems. Once in Shanghai, he worked with colleagues in Japan by phone, mapping out the situation on a whiteboard as they searched for the cause. In the end, they carried out maintenance on the entire machine.
By day the robots performed, and at night they were serviced, one by one. Sometimes the work continued well past midnight.
Makoto Kobayashi
In the end, I was cooped up in Shanghai for about two weeks. A senior colleague and I said, “Let’s at least spend one day sightseeing,” so we went out into the city. But I must have been exhausted—one beer was enough to get me tipsy. I remember heading back to the hotel without seeing much of anything and just resting.
Their efforts paid off, and Toyota’s robots continued to perform at the Expo without any major problems.
As his troubleshooting work was winding down, Kobayashi watched from the wings as the robot finished its performance to loud applause and cheers. The audience was full of smiling faces.
“Seeing that made all the hard work worthwhile. I was grateful that I could be part of something that brought the audience such joy.”