Deep Learning
Computer Vision in Agriculture
September 7, 2021
15 min read
Deep Learning in the Field: Modern Computer Vision for Agriculture
In today’s fast-paced world of city living and stressful work-life imbalances, especially on the (hopefully) tail-end of a year of pandemic quarantine measures, many young workers are yearning to get closer to nature and family. In the face of re-emerging commutes and the push-and-pull of back-to-the-office versus hybrid or fully remote working, many young robots would rather ditch the status quo and return to the countryside to scratch a living from the land like their ancestors before them. And they’ll bring lasers, too.
Of course, we’re not talking about the weary office drones being herded back to the office after a year of blissfully working at home, but of robots armed with deep learning computer vision systems and precision actuators for a new breed of farming automation. This new breed of automated agriculture promises to decrease inputs and the side effects of modern agriculture while helping farmers deal with everything from labor shortages to climate change.
Deep learning isn’t just for placing ads or identifying cats anymore. Instead, a slew of young startups has started to incorporate the advances in computer vision made possible through larger and larger neural networks to real working robots in the fields.
For most of these nascent businesses, the initial product offering is similar: get rid of those pesky weeds (and those pesky human weeders). The weapon of choice may differ substantially between each robot, ranging from lasers to herbicide applicators, mechanical disruption or even fire, but the computer vision advances that make it possible are all of a lineage inheriting from modern machine learning and deep convolutional networks. Removing weeds may make a natural minimum viable product to showcase intelligent automation in agriculture, but it’s not the only game in town.
Other projects are working on everything from harvesting delicate fruits to fully automated luxury greenhouses.
Interested in a deep learning solution?
Learn more about Exxact workstations built to accelerate AI development.
A Fitting Solution to the Problem of Tractor Vision: Convolutional Neural Networks
The key component of most of the agro-robotic systems described in this article is computer vision. Built on top of the deep learning frameworks and convolutional neural networks we’ve grown to know over the past decade, these models can handle classification, localization, and both semantic and instance segmentation.
Old hands of deep learning can skip this next part, which will outline some of the characteristics of convolutional neural networks that make modern computer vision so effective.
Convolutional neural networks (CNNs) are built out of multiple convolutional layers, and convolution is an operation where a sliding window of weights is multiplied with an input matrix, the sum of the product of this multiplication at each point is the output of the convolution.
Convolutional layers apply weights in a sliding window across an input matrix such as an image. Diagram in the public domain, source.
A key characteristic of convolution, and what makes it an ideal component of vision systems, is the translation invariance (or equivariance more precisely) of the position of the inputs. Due to the sliding window (called a kernel in neural networks), it doesn’t matter if a tomato appears on the left center or the right bottom corner, the result of convolution at that position will be the same. Rotational invariance is another story, and something that convolutional models in general don’t possess. But this is unlikely to be a major issue for tractors taking a top-down view of crops and weeds, thanks to plenty of training data and the radial symmetry of many plants viewed from above.
Another appealing aspect of convolutional neural networks as vision systems is the close analogy to animal vision systems. The successive application of convolutional kernels in a deep convolutional network is reminiscent of the bundles of neurons in the retina (and further along the visual system) that activate together in response to stimuli following a specific pattern. These are called receptive fields, and they closely resemble the edges, points, circles, and more abstracted features learned by convolutional kernels given a simple training objective.
Convolution has the valuable characteristic of being robust to objects appearing in different parts of an image; a tomato in the top left of an image is recognized the same as a tomato in the bottom right corner. Diagram in the public domain, source.
Who's Working on the Agriculture Automation Problem?
There are quite a few startups simultaneously pursuing the idea of enhanced agricultural automation augmented by deep learning. For many of them, the first product is something along the lines of an autonomous weeding tractor (or semi-autonomous tractor attachment) with a convolutional neural network vision system. Another popular approach is applying vision-guided automation to indoor gardening at a massive scale, at the extreme end, this looks like a robot as large as an entire greenhouse.
We'll mostly focus on the case for robotic weeding and only make a brief mention of other applications being developed at the end.
Semi-Autonomous: The Pull-Behinds
There’s a midpoint between manually operating farm equipment with a human pilot at the helm, and arming 5 tons of agricultural industrial equipment with autonomous brains and 150-Watt lasers, and a few companies have chosen to take this route initially. What this looks like in the field is an attachment for a conventional tractor, equipped with camera sensors, actuators, and analysis software, that still relies on a human driver and supervisor.
This is the approach of Swiss company Ecorobotix’s Ara mounted sprayer. Ara uses computer vision to guide the application of herbicides, fungicides, or insecticides to plants targeted by the onboard cameras, and the company claims this can reduce pesticide inputs by 95%. This is also the approach taken by Blue River Technology, based in California when they developed their “See & Spray” technology. The company claim for “See & Spray” is an average of a 77% reduction in herbicide usage.
One of the major incentives for targeted spraying of weeds, rather than spraying the whole field evenly (broadcast spraying) is that weeds and other plants can develop resistance to various pesticides over time. That leads to an increased tolerance for herbicide sprays like glyphosate (aka “RoundUp”) thanks to the selective pressure of being heavily exposed season after season.
This is a little like antibiotic resistance, which occurs rapidly after a new antibiotic enters use. Resistance to penicillin, the first modern antibiotic was noticed at least as early as 1940, concurrently with its development and before widespread use in human patients. In fact, glyphosate resistance was developed by directed evolution experiments long before the famous “RoundUp Ready” transgenic crops were developed for the market in the 1990s, so weeds developing resistance in response to increasing selective pressure is not surprising.
The primary advantage of targeted versus broadcast spraying is a decrease in this selective pressure on the field as a whole, complemented by a decrease in cost associated with lower inputs. It is only a partial solution, however, and there are additional advantages to doing away with a chemical weeding solution entirely, if possible. Normally this might mean aggressive mechanical tilling or manual weeding requiring additional (and slow) human labor, but the startups in the next section are working on a cadre of robotic alternatives.
Fully Autonomous Weeding Robots
Autonomous weeding robots, Source
The solutions described in this section take weeding automation one step further, from targeted to precision weeding. These autonomous agricultural robots range in size from a few hundred pounds to nearly 5 tons or more!
Each solution differs in the details, most noticeably in the choice of weeding implement, which ranges from lasers, mechanical disruption, herbicide, or even electric current. There are also many similarities: most of these robots use a deep-learning computer vision solution to target weeds and leave crop plants alone, bringing a weeding implement to bear on their target with physical actuators or optical targeting.
Carbon robotics is an imposing example of a company tackling the autonomous weeding problem. That’s not least of all because of the imposing weeding platform they’ve developed: nearly five tons of laser-wielding heavy machinery, it uses a combination of cameras, GPS, and lidar sensors to operate in large fields. The makers claim that their robot reduces the need for herbicides and labor, reduces costs, and doesn’t disrupt the soil. On top of those advantages, the robotic method is certified for use on fully organic farms.
Carbon Robotics’ Deep Learning, Laser-Wielding, Farmbot
|
The Small Robot Company is a nascent company developing agrotech in Great Britain, and their robots offer a lightweight solution to field monitoring and weed management. They’ve split the functionality of their robotics out into three separate entities: Tom, which is a mobile sensor platform that handles field monitoring; Dick, a robotic weeding robot armed with a powerful electric shock; and Wilma, the suite of machine learning and analysis tools that orchestrates the endeavor. A third robot, Harry, is also in the works. Much like other robotic weeders in this article, the weeding robot Dick uses deep convolutional networks to target weeds in the midst of desirable crops. Unlike the others, it applies an electric current to weeds to burn out the core of unwanted plants.
You’ll remember Ecorobotix from the description of their “Ara” mounted sprayer in the pull-behind section. They’ve also developed a standalone robot, Avo, for fully autonomous operation. They cite the same 95% reduction in herbicide use as the Ara, and a cost savings of 50%. In addition to the vision system, the Avo uses GPS, lidar, and ultrasound for navigation and obstacle detection. Compared to larger tractors, Avo weighs in at 750 kg and promises reduced soil compaction during operation.
Some of the companies that developed automated farm services offer robotic tractors as a service instead of selling or leasing capital equipment directly. Businesses taking this approach include Farmwise, which is trialing services with a massive automated weeding tractor in parts of California.
Robotic Greenhouses, Automated Mules, and Picker Robots
Another category of "agtech" currently experiencing a boom in interest is advanced greenhouse automation. Using a combination of sensory modalities including vision, soil moisture, temperature, and humidity, these greenhouses can record vast troves of data and alert their operators when action is needed. Companies pursuing this automated greenhouse approach to agtech include Iron Ox and iUNU. Of those two, iUNU seems to take a more ad-hoc approach to add automation and analytics to existing greenhouse systems, while Iron Ox’s website features a sleek medical-white machine that wouldn’t look out of place in a hospital or an animated film.
Another application area is automation for carting produce and plants from one area to another. Harvest Automation offers the HV-100, a medium-sized robot that is to the plant nursery what a warehouse robot is to a distribution center. Burro.ai offers a workhorse self-driving cart that works alongside human laborers to bring hand-picked produce from the field to nearby processing centers. Think of the eponymous Burro robot as a heavy-duty wheelbarrow that sports full self-driving capabilities, except unlike certain automobiles offering the feature this machine can and does operate without a human at the wheel.
Taking up the burden of transporting picked produce autonomously is a useful feature, but what about the actual harvest? Tevel and Abundant Robotics are two early-stage companies working on robots for picking fruits. They’re both focusing on harvesting in orchards at first, but their approach couldn’t look more different. Tevel is developing a swarm of flying drones to zip through the rows of an orchard grabbing fruit with a forward-facing gripper, while Abundant has built something that looks a little like a giant mechanical caterpillar, vacuuming up apples as it trundles through the rows. By all appearances, it is a very hungry caterpillar. The one thing they undoubtedly have in common, though, is the use of deep convolutional neural networks to power the vision systems that allow them to zero in on their targets.
Automated robots picking products, Source
Why Computer Vision Agriculture Innovations?
The plethora of early-stage startups might lead one to believe that the idea of using autonomous robots for weeding and other labor-intensive (yet delicate) tasks in agriculture is a new idea. In fact, it’s been more of a slow creep (followed by a mad rush) and there have been very similar projects building autonomous robots for the farm over a decade ago. Over 14 years ago in 2007, the “Hortibot” garnered mainstream, albeit not particularly reputable, news coverage for its inventors from Aarhus University. The Hortibot was an automated tractor, mainly for targeted spraying of herbicide for weed control.
Practically, its objective was a close cousin to the deep learning-powered weeding robots described in this article. While it seems to be the case that nothing much remains of the Hortibot project (and the old website is a strange melange of defunct Danish spam), it was at least a realistic enough prospect to be covered again in a 2012 New Scientist article on farming robots, mostly academic projects unlikely to scale to working agriculture.
Robotics projects like these are always on an ebb and flow as brittle implementations bump up against messy reality, and one might be tempted to conclude that this batch of eager farm startups will be no different than agro-robotics projects of the past. However, as Kevin Dunlap writes for Venture Beat, venture capitalists invested 60% more money into agricultural technology startups in 2020 than in the previous year, and nearly 20 times as much as in 2010. Time will tell if these investments are as smart as the robots they are investing in.
Increasing farm automation set a trend of a slow and steady march over the past two decades, and the SARS-CoV-2 pandemic accelerated the pre-existing development of ag-tech as it did for so many other trends. Labor and supply chain disruptions associated with the virus might encourage farmers to reconsider technological solutions that might have previously been too uncertain, too risky, or otherwise just not worth it.
Have any questions?
Contact Exxact Today
Deep Learning
Computer Vision in Agriculture
September 7, 202115 min read
Deep Learning in the Field: Modern Computer Vision for Agriculture
In today’s fast-paced world of city living and stressful work-life imbalances, especially on the (hopefully) tail-end of a year of pandemic quarantine measures, many young workers are yearning to get closer to nature and family. In the face of re-emerging commutes and the push-and-pull of back-to-the-office versus hybrid or fully remote working, many young robots would rather ditch the status quo and return to the countryside to scratch a living from the land like their ancestors before them. And they’ll bring lasers, too.
Of course, we’re not talking about the weary office drones being herded back to the office after a year of blissfully working at home, but of robots armed with deep learning computer vision systems and precision actuators for a new breed of farming automation. This new breed of automated agriculture promises to decrease inputs and the side effects of modern agriculture while helping farmers deal with everything from labor shortages to climate change.
Deep learning isn’t just for placing ads or identifying cats anymore. Instead, a slew of young startups has started to incorporate the advances in computer vision made possible through larger and larger neural networks to real working robots in the fields.
For most of these nascent businesses, the initial product offering is similar: get rid of those pesky weeds (and those pesky human weeders). The weapon of choice may differ substantially between each robot, ranging from lasers to herbicide applicators, mechanical disruption or even fire, but the computer vision advances that make it possible are all of a lineage inheriting from modern machine learning and deep convolutional networks. Removing weeds may make a natural minimum viable product to showcase intelligent automation in agriculture, but it’s not the only game in town.
Other projects are working on everything from harvesting delicate fruits to fully automated luxury greenhouses.
Interested in a deep learning solution?
Learn more about Exxact workstations built to accelerate AI development.
A Fitting Solution to the Problem of Tractor Vision: Convolutional Neural Networks
The key component of most of the agro-robotic systems described in this article is computer vision. Built on top of the deep learning frameworks and convolutional neural networks we’ve grown to know over the past decade, these models can handle classification, localization, and both semantic and instance segmentation.
Old hands of deep learning can skip this next part, which will outline some of the characteristics of convolutional neural networks that make modern computer vision so effective.
Convolutional neural networks (CNNs) are built out of multiple convolutional layers, and convolution is an operation where a sliding window of weights is multiplied with an input matrix, the sum of the product of this multiplication at each point is the output of the convolution.
Convolutional layers apply weights in a sliding window across an input matrix such as an image. Diagram in the public domain, source.
A key characteristic of convolution, and what makes it an ideal component of vision systems, is the translation invariance (or equivariance more precisely) of the position of the inputs. Due to the sliding window (called a kernel in neural networks), it doesn’t matter if a tomato appears on the left center or the right bottom corner, the result of convolution at that position will be the same. Rotational invariance is another story, and something that convolutional models in general don’t possess. But this is unlikely to be a major issue for tractors taking a top-down view of crops and weeds, thanks to plenty of training data and the radial symmetry of many plants viewed from above.
Another appealing aspect of convolutional neural networks as vision systems is the close analogy to animal vision systems. The successive application of convolutional kernels in a deep convolutional network is reminiscent of the bundles of neurons in the retina (and further along the visual system) that activate together in response to stimuli following a specific pattern. These are called receptive fields, and they closely resemble the edges, points, circles, and more abstracted features learned by convolutional kernels given a simple training objective.
Convolution has the valuable characteristic of being robust to objects appearing in different parts of an image; a tomato in the top left of an image is recognized the same as a tomato in the bottom right corner. Diagram in the public domain, source.
Who's Working on the Agriculture Automation Problem?
There are quite a few startups simultaneously pursuing the idea of enhanced agricultural automation augmented by deep learning. For many of them, the first product is something along the lines of an autonomous weeding tractor (or semi-autonomous tractor attachment) with a convolutional neural network vision system. Another popular approach is applying vision-guided automation to indoor gardening at a massive scale, at the extreme end, this looks like a robot as large as an entire greenhouse.
We'll mostly focus on the case for robotic weeding and only make a brief mention of other applications being developed at the end.
Semi-Autonomous: The Pull-Behinds
There’s a midpoint between manually operating farm equipment with a human pilot at the helm, and arming 5 tons of agricultural industrial equipment with autonomous brains and 150-Watt lasers, and a few companies have chosen to take this route initially. What this looks like in the field is an attachment for a conventional tractor, equipped with camera sensors, actuators, and analysis software, that still relies on a human driver and supervisor.
This is the approach of Swiss company Ecorobotix’s Ara mounted sprayer. Ara uses computer vision to guide the application of herbicides, fungicides, or insecticides to plants targeted by the onboard cameras, and the company claims this can reduce pesticide inputs by 95%. This is also the approach taken by Blue River Technology, based in California when they developed their “See & Spray” technology. The company claim for “See & Spray” is an average of a 77% reduction in herbicide usage.
One of the major incentives for targeted spraying of weeds, rather than spraying the whole field evenly (broadcast spraying) is that weeds and other plants can develop resistance to various pesticides over time. That leads to an increased tolerance for herbicide sprays like glyphosate (aka “RoundUp”) thanks to the selective pressure of being heavily exposed season after season.
This is a little like antibiotic resistance, which occurs rapidly after a new antibiotic enters use. Resistance to penicillin, the first modern antibiotic was noticed at least as early as 1940, concurrently with its development and before widespread use in human patients. In fact, glyphosate resistance was developed by directed evolution experiments long before the famous “RoundUp Ready” transgenic crops were developed for the market in the 1990s, so weeds developing resistance in response to increasing selective pressure is not surprising.
The primary advantage of targeted versus broadcast spraying is a decrease in this selective pressure on the field as a whole, complemented by a decrease in cost associated with lower inputs. It is only a partial solution, however, and there are additional advantages to doing away with a chemical weeding solution entirely, if possible. Normally this might mean aggressive mechanical tilling or manual weeding requiring additional (and slow) human labor, but the startups in the next section are working on a cadre of robotic alternatives.
Fully Autonomous Weeding Robots
Autonomous weeding robots, Source
The solutions described in this section take weeding automation one step further, from targeted to precision weeding. These autonomous agricultural robots range in size from a few hundred pounds to nearly 5 tons or more!
Each solution differs in the details, most noticeably in the choice of weeding implement, which ranges from lasers, mechanical disruption, herbicide, or even electric current. There are also many similarities: most of these robots use a deep-learning computer vision solution to target weeds and leave crop plants alone, bringing a weeding implement to bear on their target with physical actuators or optical targeting.
Carbon robotics is an imposing example of a company tackling the autonomous weeding problem. That’s not least of all because of the imposing weeding platform they’ve developed: nearly five tons of laser-wielding heavy machinery, it uses a combination of cameras, GPS, and lidar sensors to operate in large fields. The makers claim that their robot reduces the need for herbicides and labor, reduces costs, and doesn’t disrupt the soil. On top of those advantages, the robotic method is certified for use on fully organic farms.
Carbon Robotics’ Deep Learning, Laser-Wielding, Farmbot
|
The Small Robot Company is a nascent company developing agrotech in Great Britain, and their robots offer a lightweight solution to field monitoring and weed management. They’ve split the functionality of their robotics out into three separate entities: Tom, which is a mobile sensor platform that handles field monitoring; Dick, a robotic weeding robot armed with a powerful electric shock; and Wilma, the suite of machine learning and analysis tools that orchestrates the endeavor. A third robot, Harry, is also in the works. Much like other robotic weeders in this article, the weeding robot Dick uses deep convolutional networks to target weeds in the midst of desirable crops. Unlike the others, it applies an electric current to weeds to burn out the core of unwanted plants.
You’ll remember Ecorobotix from the description of their “Ara” mounted sprayer in the pull-behind section. They’ve also developed a standalone robot, Avo, for fully autonomous operation. They cite the same 95% reduction in herbicide use as the Ara, and a cost savings of 50%. In addition to the vision system, the Avo uses GPS, lidar, and ultrasound for navigation and obstacle detection. Compared to larger tractors, Avo weighs in at 750 kg and promises reduced soil compaction during operation.
Some of the companies that developed automated farm services offer robotic tractors as a service instead of selling or leasing capital equipment directly. Businesses taking this approach include Farmwise, which is trialing services with a massive automated weeding tractor in parts of California.
Robotic Greenhouses, Automated Mules, and Picker Robots
Another category of "agtech" currently experiencing a boom in interest is advanced greenhouse automation. Using a combination of sensory modalities including vision, soil moisture, temperature, and humidity, these greenhouses can record vast troves of data and alert their operators when action is needed. Companies pursuing this automated greenhouse approach to agtech include Iron Ox and iUNU. Of those two, iUNU seems to take a more ad-hoc approach to add automation and analytics to existing greenhouse systems, while Iron Ox’s website features a sleek medical-white machine that wouldn’t look out of place in a hospital or an animated film.
Another application area is automation for carting produce and plants from one area to another. Harvest Automation offers the HV-100, a medium-sized robot that is to the plant nursery what a warehouse robot is to a distribution center. Burro.ai offers a workhorse self-driving cart that works alongside human laborers to bring hand-picked produce from the field to nearby processing centers. Think of the eponymous Burro robot as a heavy-duty wheelbarrow that sports full self-driving capabilities, except unlike certain automobiles offering the feature this machine can and does operate without a human at the wheel.
Taking up the burden of transporting picked produce autonomously is a useful feature, but what about the actual harvest? Tevel and Abundant Robotics are two early-stage companies working on robots for picking fruits. They’re both focusing on harvesting in orchards at first, but their approach couldn’t look more different. Tevel is developing a swarm of flying drones to zip through the rows of an orchard grabbing fruit with a forward-facing gripper, while Abundant has built something that looks a little like a giant mechanical caterpillar, vacuuming up apples as it trundles through the rows. By all appearances, it is a very hungry caterpillar. The one thing they undoubtedly have in common, though, is the use of deep convolutional neural networks to power the vision systems that allow them to zero in on their targets.
Automated robots picking products, Source
Why Computer Vision Agriculture Innovations?
The plethora of early-stage startups might lead one to believe that the idea of using autonomous robots for weeding and other labor-intensive (yet delicate) tasks in agriculture is a new idea. In fact, it’s been more of a slow creep (followed by a mad rush) and there have been very similar projects building autonomous robots for the farm over a decade ago. Over 14 years ago in 2007, the “Hortibot” garnered mainstream, albeit not particularly reputable, news coverage for its inventors from Aarhus University. The Hortibot was an automated tractor, mainly for targeted spraying of herbicide for weed control.
Practically, its objective was a close cousin to the deep learning-powered weeding robots described in this article. While it seems to be the case that nothing much remains of the Hortibot project (and the old website is a strange melange of defunct Danish spam), it was at least a realistic enough prospect to be covered again in a 2012 New Scientist article on farming robots, mostly academic projects unlikely to scale to working agriculture.
Robotics projects like these are always on an ebb and flow as brittle implementations bump up against messy reality, and one might be tempted to conclude that this batch of eager farm startups will be no different than agro-robotics projects of the past. However, as Kevin Dunlap writes for Venture Beat, venture capitalists invested 60% more money into agricultural technology startups in 2020 than in the previous year, and nearly 20 times as much as in 2010. Time will tell if these investments are as smart as the robots they are investing in.
Increasing farm automation set a trend of a slow and steady march over the past two decades, and the SARS-CoV-2 pandemic accelerated the pre-existing development of ag-tech as it did for so many other trends. Labor and supply chain disruptions associated with the virus might encourage farmers to reconsider technological solutions that might have previously been too uncertain, too risky, or otherwise just not worth it.
Have any questions?
Contact Exxact Today
