AI in the Weeds

Pennsylvania ranks fourth in the nation for apple production and acreage, making apples one of the state’s most economically important specialty crops and a cornerstone of many rural communities. Protecting those orchards from threats that reduce yields—including weeds—is critical not only for growers’ livelihoods, but also for the industry’s long-term sustainability.  

Weed control in apple orchards has long been a stubborn challenge. Weeds compete with trees for water, nutrients, and sunlight, reducing fruit quality and yield. Traditionally, growers rely on blanket herbicide sprays applied across entire rows, combined with occasional hand pulling. That approach is time-consuming, chemically intensive, and increasingly a challenge as weeds develop resistance to commonly used herbicides. It also raises concerns about environmental impacts, worker safety, and chemical drift onto tree trunks or fruit.

A team of Penn State researchers believes precision weed management can change that equation. This work focuses on using artificial intelligence and machine vision to detect weeds accurately and treat only what needs to be treated—no more, no less.
 

From the side

At the heart of the effort are Long He, associate professor of agricultural and biological engineering, and Lawrence Arthur, a doctoral candidate in the same department. 

“Precision agriculture has been very efficient in open fields, but orchards present a very different situation. If we can apply precision agriculture in orchards, we can make production much more efficient,” He said.  

Most existing precision weed systems rely on top-down cameras, such as drones or overhead sensors. In apple orchards, low-hanging branches and dense canopies block that view. “It’s very challenging to have a top-view camera in orchards because the branches obstruct the camera,” He continued.  

The solution was a side-view camera system positioned to look into the tree row. But that introduced a new set of problems. Weeds appear and disappear from view as a vehicle moves, lighting conditions change constantly, and tree trunks, irrigation lines, and shadows clutter the scene. “You have the robot moving through the field, and there are so many frames moving through the system,” He said. “It’s very difficult to keep track of weeds consistently.” 

Discover the Benefits of ASME Membership

Arthur led the effort to overcome those challenges by enhancing an existing deep-learning vision model. The team integrated an attention mechanism that allows the model to focus on relevant features—the weeds—while suppressing irrelevant background information. “The attention mechanism helped us detect very small weeds and avoid spraying the trunk or tree tissue,” Arthur said.  

Weed tracking was equally important. Without tracking, the same weed might be detected repeatedly across video frames, leading to double counting or repeated spraying. “We needed a tracking mechanism that assigns identities to weeds and preserves those identities across frames,” He said. “That way, the system picks the right coordinates and avoids sending duplicate information to the sprayer.” 

 

Targeted effort

The research was painstaking. One of the most time-consuming steps was building the training dataset. Each orchard image contained as many as 25 individual weeds, all of which had to be manually labeled across different seasons, lighting conditions, and growth stages.

The team focused initially on four of the most prevalent orchard weeds: dandelion, lambsquarters, horseweed, and horsenettle. Arthur noted that while the current system recognizes those species, it can be expanded as more images are collected. 

After months of development, the researchers experienced a breakthrough moment. “The happiest moment was when we first saw the trained weeds being detected and tracked consistently,” He said. “The side-view camera combined with the attention mechanism allowed us to detect even the smallest weeds and maintain their identities.” 

You Might Also Enjoy: Robot Introduces Technology to Terroir

The potential benefits extend beyond technical achievement. For growers, precision spot spraying could dramatically reduce herbicide use while improving weed control. “Right now, growers mostly do blanket sprays,” Arthur said. “Even after multiple sprays, some weeds still come out, and then they either pull them by hand or leave them. With spot spraying, we can target weeds as they emerge and keep them from getting big.” 

Reducing chemical inputs also provides environmental and consumer benefits. Less herbicide means less risk of runoff, leaching, and drift, and lower exposure for workers and reduces the chance of herbicide resistance.  

“We want to apply the right amount of herbicide, so the weed dies without unnecessary excess,” He said.  

The team is now moving from vision to action—integrating the detection system with a precision spraying mechanism. They are developing a swivel-controlled nozzle that can reach deep into tree rows, guided by the weed coordinates identified by the AI system. Initial testing is taking place in a lab-based orchard mock-up, with field trials planned in Penn State’s research orchards. 

Looking ahead, the researchers see applications well beyond apples. The same vision system could be adapted for peaches, other tree fruits, vegetables, and even mechanical weed control. Additional data to be collected, such as soil moisture, could further refine how much herbicide is applied and when. 

“This is really about solving a practical problem for growers,” Arthur said. “If this can help farmers reduce inputs, protect the environment, and maintain productivity, that’s a win for the whole community.” 

Annemarie Mannion is a technology writer in Chicago.