KABUL, Afghanistan — U.S. Defense Secretary Leon Panetta warned Thursday that the United States was “reaching the limits of our patience” with Pakistan over militants that attack U.S. forces in neighboring Afghanistan from havens within its borders.

“It’s extremely important that Pakistan take action to prevent this kind of safe haven,” he said during an unannounced visit to Kabul, and that militants cannot use the country as a “safety net” from which to attack U.S. soldiers.

“We have made that very, very clear time and time again, and we will continue to do that,” he said.

Panetta also stressed that troops on the Afghan border have every right to defend themselves against the al Qaeda-linked Haqqani network and other militant groups when they launch attacks from Pakistan.

“Anybody who attacks U.S. soldiers is our enemy. We are not going to take it,” he said.

“We have every responsibility to defend ourselves, and we are going to make very clear that we are prepared to take them on — and we have to put pressure on Pakistan to take them on as well.”

Afghan Defense Minister Gen. Abdul Rahim Wardak, speaking alongside Panetta at a news conference in Kabul, said havens in Pakistan were the “most important” issue confronting Afghanistan as it seeks to secure a stable future.

Panetta’s strongly worded remarks come three days after a CIA drone strike in Pakistan’s tribal region killed Abu Yahya al-Libi, the No. 2 man in al Qaeda and a longtime public face of the terrorist network, and at a time of strained Pakistani-U.S. relations.

Monday’s strike was the third such deadly attack in as many days and the 21st suspected U.S. drone strike in Pakistan this year. At least six missiles were fired at a militant compound near the town of Mir Ali in the North Waziristan region, near the Afghan border.

Pakistani lawmakers have called for an immediate end to the drone strikes, which have been denounced for killing civilians.

But speaking on a stop in India on Wednesday, Panetta said Washington had made clear to Pakistan that it would defend itself against those who sought to attack the United States.

“And we have done just that. We have gone after their leadership, and we have done it effectively … targeting al Qaeda leadership and terrorists.

“And very frankly, the terrorists who threaten the United States threaten Pakistan as well. This is not just about protecting the United States; it’s also about protecting Pakistan. And we have made very clear that we are going to continue to defend ourselves.”

President Barack Obama’s administration recently defended its use of unmanned drones to target suspected terrorists overseas in a rare public statement, with John Brennan, the president’s top counterterrorism adviser, saying the strikes are conducted “in full accordance with the law.”

The program uses unmanned aerial vehicles, often equipped with Hellfire missiles, to target suspected terrorists in remote locations overseas, with many such strikes occurring in Yemen and Pakistan, despite internal opposition to the practice within the latter country.

Brennan said the United States “respects national sovereignty and international law” and is guided by the laws of war in ordering those attacks.

The Pakistani border area is widely believed to be the operating base for the Haqqani network and other militant groups that have attacked international troops in neighboring Afghanistan.

Panetta is not the first high-ranking U.S. official to point the finger at Pakistan over a failure to take on extremists within its borders.

Addressing Congress last September, just before he retired as chairman of the Joint Chiefs of Staff, Adm. Mike Mullen said the Haqqani network acted as a “veritable arm” of Pakistan’s Inter-Service Intelligence agency.

He also told CNN that elements in Pakistan’s spy service were “very active” with the Haqqani network in launching attacks on U.S. forces in Afghanistan, and that the U.S. and Pakistani governments and military must work together to tackle the problem.

Mullen’s blunt remarks to U.S. senators were widely interpreted as an expression of the growing impatience of U.S. officials with Pakistan’s unwillingness to stop the attacks, and the belief that elements in the Pakistani government are actively supporting the insurgents.

Relations between Washington and Islamabad have been particularly tense since NATO airstrikes killed two dozen Pakistani soldiers in November.

Following the fatal airstrikes, the Pakistani government shut down the two NATO supply routes in the country, asked the United States to vacate an air base on its territory and boycotted a conference about the future of Afghanistan. NATO insists the incident was an accident.

Pakistan was already angry that in May of last year, it was kept out of the loop as U.S. intelligence tracked al Qaeda founder Osama bin Laden to a Pakistani town, flew undetected into the country and killed him.

Panetta did not visit Islamabad, the Pakistani capital, during his nine-day Asia trip, which included stops in Vietnam and Singapore, as well as New Delhi.