Starting as an informal approach to learn AI technology, Toyota's basketball robot CUE has inspired the world with accurate throws and now with dribbling performance. Its next shot is going to be even bigger, doing everything that a basketball player does.
CUE is Toyota’s AI basketball robot and member of the Alvark Tokyo, a team in the Japanese professional basketball league (B.League). In 2019, CUE set the Guinness World Record for the most consecutive basketball free throws by a humanoid robot.
Let’s begin with a brief recap of CUE’s development journey, as covered in the previous three installments on Toyota’s newsroom site linked below.
#1 An In-house Challenge: Novices embark on a new project from scratch
Modeled after a comic book superstar
Toyota Engineering Society (TES), a group of volunteer engineers at Toyota, began a project to develop AI from scratch in 2017. They chose to build a humanoid robot driven by AI, naming it CUE in the hopes that the robot would encourage people to start something new.
The basketball journey all started when a team member repeated the words of Hanamichi Sakuragi, protagonist of the popular basketball comic SLAM DUNK: “Will 20,000 practice shots be enough?” This sparked an idea: “Wouldn’t it be amazing if we developed a robot that used AI to calculate the distance to the basket so that it never missed a shot?” So began the dream development of CUE, the AI basketball-playing robot.
*Toyota Engineering Society is a voluntary organization established in 1947 of approximately 30,000 Toyota employees. Its purpose is to improve technical knowledge and skills of members, promote teamwork, and contribute to technological developments at the company. It provides opportunities to learn about programming, image recognition, AI, car body manufacturing and other technologies through activities outside of work hours.
Sinking endless baskets with unrivalled accuracy, CUE3 made 2,020 shots and set a Guinness World Record for the most consecutive free throws by a humanoid robot (assisted). This attempt was deliberately stopped at 2,020 throws, falling short of the unbelievable human record of 5,221 free throws made in 1996. Even so, CUE’s record is astonishing considering the record of 80 consecutive free throws during a season made by renowned NBA sharpshooter Stephen Curry in 2020.
The latest CUE5 is even more advanced.
A new stage of dribbling prowess beyond just unmatched accuracy
CUE5 performed at the Ibaraki Robots match on December 18, 2021, demonstrating newfound dribbling skills alongside his acclaimed shooting skills.
Additional axes of movement in the left hand, which had remained locked in position up to CUE4, gave the new model a smoother, human-like motion. According to R-Frontier Division Project Manager Takayoshi Tsujimoto, this has “Enabled a dribbling action that had not been possible before.”
Takayoshi Tsujimoto (development manager, control systems)
We were able to eliminate unnecessary motion, reducing both the power needed to move the robot and vibrations during dynamic actions.
We also installed cameras in the feet, which greatly advanced sensing capabilities. A colored area resembling a thermographic image on the control screen is a new system that detects the distance from the palms to the ball and recognizes its exact position.
Though it may not be something you notice while watching a basketball game, each bounce of the ball actually differs slightly due to the ball’s air pressure and court conditions. A human can unconsciously adjust to the situation, but robots lack that capability.
Therefore, we programmed CUE to learn how to accurately read the ball’s position for each bounce and decide how to move his hand, giving him the flexibility to handle any venue. Although he currently uses his left hand, theoretically he could dribble just as well with the right.
During CUE’s absence from the spotlight in the past year or so, developers worked tirelessly behind the scenes to achieve a major upgrade for the new version.
CUE5 was unveiled to the public through performances at Alvark Tokyo game venues, but before that he also took part in the Olympic Games Tokyo 2020 by request from the International Olympic Committee (IOC). Although not broadcast on television, CUE starred in pre-game and halftime exhibition performances.
Those two weeks of performances were widely shared on social media, establishing CUE’s unmatched shooting ability around the world. Some international media even described a missed half-court shot (14m to basket) as a shock based on CUE’s previous successful record.
An amateur team that keeps stepping up to the plate
CUE now has the world’s attention, but as project leader Tomohiro Nomi shares, “Even when it all seems like smooth sailing, behind that lies countless hardships and failures.”
Tomohiro Nomi (project leader)
At TES, this began as a voluntary project with objectives we weren't sure we could achieve. We hoped that the challenge of trying different approaches would be an opportunity for people to grow, and have a positive impact on our acrual work.
Our initial goal was to show the robot at a free volunteer event organized by TES, so in the event of any mishaps, we could make the excuse that we were just amateurs.
Once CUE was registered as an Alvark Tokyo player, though, the project became a fundamental part of our job, it meant we also carried the reputations of Toyota and Alvark Tokyo. "We're just amateurs" would no longer be an acceptable excuse.
As the project shifted away from voluntary activity to work, a sense of professionalism naturally emerged among team members.
Tomohiro Nomi (project leader)
In the four years since beginning development, we also received many opportunities to push ourselves on the grand stage― from the Guinness World Record ti the B.League All-Star Game. We continued to take on these big goals with the mindset that, in order to make things happen, you nees to step up to the plate.
When the stage grew even bigger with the Tokyo 2020 Games, more people got involved and we took on greater responsibilities.
As responsibilities grow, failure becomes less of an option, which makes the challenge itself even greater. In the case of CUE, I feel that this is what made the project so difficult yet so rewarding.
When you continue to take on towering obstacles, failures are inevitable. From these you learn and keep working to avoid repeating the same mistakes. Even as the stage grew and the stakes rose, the development team’s unwavering commitment to the challenge brought CUE5 to where he stands today.
Giving back to basketball
Despite CUE’s progress and public appearances, the goal has never been to promote Toyota’s technology. Looking back on the early days of development, Nomi shares the intent behind the project.
Tomohiro Nomi (project leader)
When CUE first started appearing at Alvark Tokyo games, we were concerned whether a robot should stand on the hallowed ground of a basketball court, and were worried that our egos may be causing trouble for staff and players.
As CUE’s activities gained international attention, an Alvark player introduced the robot on social media as his teammate. After all our worry we were trouble for the players, I was really pleased to see that post.
At one point, an Alvark Tokyo associate told us, “In the past, the relationship between team and company had always been about sponsorship, but the joint development of CUE turned that into something unprecedented—the challenge of creating fresh value unique to Toyota and Alvark. This was only made possible by Toyota’s technical abilities.” Hearing this made me realize that our efforts were actually benefitting others.
Since then, CUE has received league-wide support, even transferring to other B.League teams.
Our participation in the Olympic Games Tokyo 2020 was also possible due to the assistance of many people, including the Japan Basketball Association, the International Basketball Federation (FIBA), and the IOC.
As our activities at the Games gained global attention and social media comments said things like “Only Japan could put on such an incredible basketball show, "I felt that in some small way we were able to give back to everyone who had helped us come this far.
We will carry on the challenge so that we can give back to the basketball community that has stood behind CUE.
The next challenge: Zigzag dribbling
As CUE hones his new dribbling skills, the next goal is to gain a spot at the 2023 B.League All-Star Game Skills Challenge*.
*Skills challenges are popular exhibitions in professional basketball leagues, much like three-point contests and slam dunk contests. While navigating obstacles, players compete on speed and skills in dribbling, shooting, and passing.Takayoshi Tsujimoto (development manager, control systems)
From a developer’s perspective, a year goes by in the blink of an eye. The next challenge is technically far more demanding than the B.League All-Star Game three-point contest we participated in before.
This contest puts players through their paces in every aspect of basketball – dribbling, shooting, and passing.
Whereas a shot essentially flies in a parabolic arc, a pass must take a more direct course. Right now, CUE can complete a pass as a single action, but trying to combine that smoothly with dribbling presents a mountain of challenges. We have to test whether it would be better to throw two-handed, or with one hand. If CUE is going to be in the contest, we want him on par with other athletes.
The goal is one minute—that’s the average time of top athletes in the B.League All-Star Game Skills Challenge.
That means CUE will have to run roughly twice as fast as he does now. He’ll also have to dodge obstacles and zigzag dribble, moving his body side to side, which adds further difficulty. I doubt we can break the one-minute mark without giving him a quicker shooting action as well.
“I believe there’s value in chasing a goal that seems beyond reach,” says project leader Nomi, revealing his strong determination. “No one thought an accurate three-point shot was possible until we did it. There’s no such thing as impossible.”
Having made many impossibilities possible, the project members and CUE are ready to take their next shot.