(CNN) — More service members than initially believed were involved in misconduct in Colombia ahead of last weekend’s Summit of the Americas, the Pentagon said Monday.

The U.S. military had said five U.S. troops who were working with the Secret Service were under investigation for missing curfew and alleged misconduct at the Colombian hotel where Secret Service agents are said to have brought back prostitutes.

“We believe there may be more than five,” Pentagon spokesman George Little said Monday in reference to U.S. military personnel involved.

Little could not say how many more military personnel might be involved. But he indicated the personnel might come from more than one branch of the military. Pentagon officials had originally thought only Army personnel were involved.

The personnel were expected to return to the United States on Monday.

President Barack Obama called Sunday for a “thorough” and “rigorous” investigation into the allegations from the summit in Cartagena.

Eleven Secret Service agents and officers are being investigated over preliminary findings that they allegedly brought back several prostitutes to a hotel in Cartagena, U.S. government sources familiar with the investigation have told CNN.

Although Little would not specify the alleged misconduct, U.S. officials have said they are looking into allegations the personnel were involved in heavy drinking and engaging prostitutes. The military is specifically looking into whether the personnel had prostitutes in their hotel rooms and violated their military curfew.

Little said the personnel were not directly involved in presidential security and did not have any contact with Obama. He defined their role as in “support” of the Secret Service.

Because presidential security was the overall mission, however, it is not clear whether the Pentagon will ever publicly describe what the military personnel were doing as part of their work in Colombia or what branches of the military they belonged to.

An officer who was already in Colombia gathered initial facts, according to Little. A more senior officer was leaving for Colombia on Monday to continue the investigation.

The original five personnel were not identified by name. But on Saturday, U.S. Southern Command announced the investigation and said the personnel were restricted to their hotel rooms when not performing their duties.

The alleged misconduct occurred before Obama arrived in Cartagena.

“We’re representing the people of the United States, and when we travel to another country, I expect us to observe the highest standards,” Obama said Sunday. “Obviously, what’s been reported doesn’t match up with those standards.”

Still, he cautioned, “I’ll wait until the full investigation is completed until I pass final judgment.”

Rep. Peter King, chairman of the House Homeland Security Committee, was briefed on the matter and said Saturday that the government personnel brought prostitutes back to their rooms Wednesday night, and “one of the women did not leave the room in the morning.”

A hotel manager tried to get in the room, and eventually the woman emerged and said “they owed her money,” according to King. Similarly, U.S. government sources said there was a dispute between at least one Secret Service member and a woman brought back to his hotel over a request to be paid.

At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.

“The Secret Service saw that report, and they immediately began an investigation,” King said.

Secret Service spokesman Edwin Donovon said the agents were relieved of duty Thursday, before the president’s arrival in Colombia, and replaced after “allegations of misconduct.”

The agency’s assistant director, Paul Morrissey, noted his agency’s “zero tolerance policy on personal misconduct.”

“This incident is not reflective of the behavior of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner,” Morrissey said Saturday. “We regret any distraction from the Summit of the Americas this situation has caused.”