A Tokyo medical group warned Thursday that the Olympics should be held without spectators because of virus risks, while a doctors' association urged outright cancellation, warning of potential "disaster".
Several parts of Japan are under virus states of emergency over a surge in infections, and the public is largely opposed to holding the Games this summer.
Medical groups have warned the massive event could introduce new variants to Japan and strain medical resources as the country battles a fourth virus wave.
The Tokyo Medical Association, with over 20,000 members, has not called for an outright cancellation.
But its chairman Haruo Ozaki said Thursday that "holding the Games without spectators is the bare minimum given the current situation".
"This is the Olympics in a time of emergency," he said at a press conference.
Organisers have already barred overseas fans from the Tokyo Games, an unprecedented decision for the Olympics.
A decision on domestic spectators is expected in late June.
The virus states of emergency in several regions, including Tokyo, are expected to be extended this week until June 20.
Ozaki urged the government to take stricter measures against the virus immediately, warning "this is the last chance" to subdue infections before the Games.
Tokyo's current virus restrictions allow venues to have up to 5,000 fans or 50 percent capacity, whichever is fewer.
Olympic organisers say extensive rulebooks with virus countermeasures will keep participants and the Japanese public safe.
The measures include daily testing for athletes and a ban on using public transport.
But Naoto Ueyama, chair of the Japan Doctors Union, said bringing tens of thousands of athletes, staff, media and officials from around the world posed a real "danger", particularly of spreading variants.
"Initially the Tokyo Olympics was dubbed a Games to showcase reconstruction from the 2011 earthquake and tsunami disaster, but the Tokyo Olympics could generate another disaster," he said.
"A new variant might arise and may be called the Tokyo Olympic strain, and the criticism that the Games were a great act of folly by humankind may stalk us for the next 100 years to come," he added.
"Some people are acting almost like dictators and asking people to make sacrifices in order for the Olympic Games to go ahead."
His union represents 130 doctors across Japan, and along with another association representing around 6,000 doctors in Tokyo has explicitly backed an Olympics cancellation.
Olympic organisers have insisted they will not create additional pressure on Japan's healthcare system, with the International Olympic Committee offering to bring in medical staff to ensure the Games do not prove a burden.
Calls from organisers for volunteer nurses caused an uproar, but a request for volunteer doctors actually ended up attracting nearly double the required specialists.
Around 80 percent of people in Japan want the Games either cancelled or postponed for another year, polls show.
But Japan's government and Olympic organisers say the show will go on, and the first athletes are expected to begin arriving for training camps from next week.