And on that farm, he had a robot

BONUS CONTENT

• Design, integration, and field evaluation of a robotic apple harvester (Wiley)
• Robotics for Sugarcane Cultivation: Analysis of Billet Quality using Computer Vision (IEEE Xplore)
   

TRANSCRIPT

[MUSIC: Excerpt from “John Henry” by Ragtime Texas, 1927, public domain]

HAUTALA: We just heard a little bit from the folk blues standard “The Ballad of John Henry,” a classic tale of man vs. machine. If you  aren’t familiar with the legend – or haven’t seen the Disney cartoon version – John Henry was a steel driver, a railroad worker who earned his living swinging a hammer.

[MUSIC: “John Henry,” continues softly under narration.]

HAUTALA: In the story, John Henry goes head to head with a steam drill, a monstrous piece of machinery that threatens to put him out of a job. John Henry vows that “before that steam drill shall beat me down, I’ll die with a hammer in my hand.”

[MUSIC: Fade in “… I’ll die with a hammer in my hand …”]

HAUTALA: In a modern light, the story serves as something of a cautionary tale. According to the legend, John Henry makes good on his vow. He beats the machine. Then, exhausted from overwork, he dies, with a hammer in his hand.

That story is about 150 years old. It just goes to show, the anxiety people have about machines taking over jobs once done by humans is nothing new. This season on “Engineering Out Loud,” we’re taking a deep look at both the promise and the peril of robotics and artificial intelligence. In this episode, we’ll turn an eye toward robots working in fields where you might not expect to see them. Like, in actual fields, on farms and in orchards. That’s coming up.

[MUSIC: “The Ether Bunny,” by Eyes Closed Audio, used with permission of a Creative Commons Attribution License.]

HAUTALA: From the College of Engineering at Oregon State University, this is “Engineering Out Loud.”

HAUTALA: Hello again. I’m Keith Hautala. When we think about robots taking on human jobs, most of us probably think of industrial jobs, like car manufacturing. But one area where robotics is poised to make a huge difference in how work gets done is in agriculture. Today you’ll hear from our resident ag-robot expert, Joe Davidson.

DAVIDSON: My name is Joe Davidson. I’m an assistant professor in the School of Mechanical, Industrial, and Manufacturing Engineering and a member of the Collaborative Robotics and Intelligent Systems Institute. And this is my second year at OSU.

HAUTALA: Joe was already an engineer when he started working with robots at Washington State University.

DAVIDSON: I got interested in robotic systems when I was a project manager at an engineering company called CH2M HILL and I was working on my master’s at night. So I would go to work during the day and then head over to class at WSU in the evening. And my professor, as I was finishing my master’s, my professor said, “Hey, I got this new grant that’s for robotic apple harvesting. Um, are you interested in doing it?” At the time, I had no interest in doing a Ph.D., but the stars aligned, and I was going to make a career change anyway at that time. So I said, “Hey, what the heck?”

HAUTALA: Mechanization in agriculture is nothing new for major crops like corn, soy, and wheat. But when it comes to so-called specialty crops like fresh fruits and vegetables, most of the harvesting is still done by hand. Here’s Joe again, with a little background.

DAVIDSON: We look back over the 20th century, commodities like large row crops have been heavily mechanized all the way going back to the tractor in the 1930s. So the percentage of the U.S. workforce that works on farms today is less than 2%, where it used to be a third back in around the time of World War II. And so whereas commodities have seen a lot of mechanization, the production of specialty crops is still heavily dependent on people. And by specialty crops, I mean fresh market fruits and vegetables. And that’s different than processed market fruit, where there’s a little bit higher tolerance for bruising and external defects — maybe oranges that will go into orange juice or apples that’ll go into apple juice. So we’re talking about the fruits and vegetables that you would see on your grocery shelf.

HAUTALA: While harvesting fruits and vegetables might be a hard job, it doesn’t seem at first glance to be all that complicated. I mean, how difficult could it be make an apple-picking machine? Why don’t we have them already? According to Joe, it’s not as easy as you might think.

DAVIDSON: It’s physically demanding work. But generally, yes, it’s considered low skilled. It’s a low skilled job. And so, it sounds like a simple task, but there’s a very specific way that people try and pick an apple.

[MUSIC: “Glen Canyon” by Dan Lebowitz, part of the YouTube Audio Library. Licensed under a Creative Commons license]

You want to grab it in a certain way, and you want to put pressure against the stem, and pull it and twist it in a certain way to make sure that you minimize bruising and retain the stem for storage. And so, you know, I, as a person, if I knew how to pick an apple, I could look down the middle of a row, and I could close my eyes, and I could reach into a tree, and I could feel around and say, “OK, that’s a branch, that’s a leaf. Oh, there is an apple. Let me kind of move my hand around the apple using the sense of touch and say, OK, there’s the stem, let me put pressure against it and pull it.” And I wouldn’t even have to look at the tree in order to do that, as long as I was reaching into an area where there are fruit. And so trying to replicate that with the robot is just a really difficult thing to do.

HAUTALA: And it’s not that people haven’t tried. In fact, researchers have been working on this problem for decades.

DAVIDSON: Some of the first papers on robotic apple harvesting were from the 1980s in Europe. And so it’s not necessarily a new concept. So, you know, people had been working on it for 40 years, but it’s proved really difficult to do and to implement at the commercial scale. And that’s because the agricultural environment is so highly unstructured. It’s not like an automotive factory, where I can make sure that I have a very controlled environment in the work cell, and the parts can arrive at the exact same place the exact same time, over and over again. It’s a highly repeatable process. So now we’re talking about an outdoor environment with a biological system. So, every tree is different. You know, the lighting conditions change from day to day. We have shadows, we have sun glare, we have moisture and dust. And then we have just the physical act of trying to interact with an environment where every situation is different. It’s just proved very difficult for robotics.

HAUTALA: I asked Joe to sort of talk us through what an apple-picking robot might look like, and how it would go about doing its job.

DAVIDSON: Two of the startups that I know that are working on robotic apple harvesting, one of them is an Israeli company called F.F. Robotics. And they have a system that’s got multiple kind of arms on each side, and these arms are linear arms, so they just reach out towards the tree. So you go down a row, it’s got multiple arms working down both sides of the row. And they have vision sensors or cameras that identify and tell these arms where to go.

[MUSIC: “Glen Canyon” by Dan Lebowitz, part of the YouTube Audio Library. Licensed under a Creative Commons license]

Then it’s got this kind of a, you know, like a claw-looking mechanism — like the one from the arcade that always takes your money and takes your kids’ money. And so, and they never get the teddy bear, right? And so it has one of these claw-looking mechanisms on the end of these arms that grabs the fruit.

HAUTALA: Hopefully, the robot is better at grabbing apples than your kids are at grabbing teddy bears.

DAVIDSON: And then there’s another company called Abundant Robotics, and they’re from California and they have — they don’t have multiple arms, but they have a really fast system, and on the end of the system is a giant vacuum. And it literally just goes — and you know, they have a camera that’s telling it, localizing the fruit in the tree. And this vacuum goes and just vacuums the apples off the tree. And I think probably one of the really interesting things they must have figured out is, OK, now that I’ve got this apple that’s moving at high speed that’s been backing off the tree, how do I decelerate it and store it in this container without bruising it. And so I’m sure that was a challenge that took quite a bit of effort to work through.

HAUTALA: And that’s just one of the problems designers of apple-picking robots need to work out. It’s simply not enough for the robot to just roll up on a tree, identify an apple that is ready to be picked, and yank it from the branch. It has to be able to do it without damaging the fruit or harming the tree itself. It’s something people can learn how to do pretty easily. Actually doing it as a job, though …

DAVIDSON: You grab it with what’s called a power grasp, where you put your palm against the fruit and you envelop it with your fingers, and you usually take your index finger and you apply pressure against the abscission joint, where the stem connects to the tree, right? And you apply it, you do a pulling and twisting motion while applying pressure against the stem. And so that’s the general technique they use. But you know, a lot of times they’re doing two hands, right? And they’re picking maybe one apple every two seconds. And they’re on top of a ladder when they’re doing this. And they’ve got a 50-pound bucket on their backs filled with apples when they’re doing this. And so, it’s a lot of work.

HAUTALA: It does sound like a lot of work.

DAVIDSON: And then, at the same time, you have some varieties like a Honeycrisp, a high-dollar variety, and they stem clip those after they pick. Because they don’t want that stem exposed and puncturing other fruit in the storage bin. And so there’s this extra task, where: OK, I pick the fruit with one hand, then I come in with my stem clippers and cut the stem below the top plane of the fruit. And then I put it in the storage container. So if you’re going to do that with a robot … I mean, how do you, you know, it’s hard enough just getting the apple off the tree. What are you going to do with these high-dollar varieties where you need to stem clip as well? So, just one other additional challenge that goes into the process.

HAUTALA: So, before anything else, Joe the engineer had to learn how to be an apple-picker.

DAVIDSON: Our idea, our thought was, OK, let’s study how people pick fruit and let’s try and embody that in the robot. And so one of the first things we did is, we went out to an orchard and met with an orchard manager and, you know, got an hour on the job training to be a fruit picker. And you know, obviously we didn’t have the skill set that a professional picker does and we weren’t up on ladders, you know, doing it 12 hours a day. But we would instrument the hand — we developed a glove that had force sensors on the fingertips and then inertial measurement units on the back. And so what we would do is we would try and replicate these apple picking patterns and just kind of understand the forces that are involved.

HAUTALA: It turns out there is a fair amount of physics and math involved in picking an apple. Blessed with a sense of touch, we humans can do it intuitively, without performing any explicit computations. But to teach a machine to do the same thing, you first need to gather a lot of data in order to answer some specific questions.

DAVIDSON: How much do you have to squeeze in order to cause bruising? And during the picking process, how far do you actually have to displace the fruit before it separates from the tree? And so then, we tried to use that data when we developed robotic methods and built our robot to go and pick apples. So, I know that if I approach a fruit this way, and I twist it 45 degrees, I only have to pull it away maybe 5 cm, from the tree and I know it’s going to separate, and then I can go and store it. And so we tried to use that information when we developed our robotic system.

HAUTALA: At this point, it might start to seem like designing a robotic system might actually be more work than just going out there and picking the apples by hand. So why don’t we just keep doing it the old-fashioned way? One problem is there just aren’t enough people to pick all of the apples that need picking during a fairly short harvest season.

DAVIDSON: You could do a quick Google search, and — just look at California from the last couple of summers — there’s no shortage of articles talking about: Hey, this is a real problem. We don’t have enough people to do the work. And there is uncertainty. You know, a lot of the workforce is a seasonal workforce. And there’s uncertainty about immigration policy. And, so there’s a lot of questions about how, how viable is it to rely on the current workforce in the future.

HAUTALA: So in the short term, at least, robots aren’t likely to be replacing farm workers, so much as taking up the slack where there just aren’t enough humans to get the job done.

DAVIDSON: If we’re honest about the technology, I think, in the near term, it’s just going to augment any shortages that currently exist. And so maybe a robot picks 10% of the apples in Washington, and it’s going to be a long time, I think, before it’s picking 100% of the fruit. The technology is not just there yet. And so, certainly, you know, with the potential to have a shortage, and then you have what’s a very seasonal activity that’s highly weather dependent. And then you have a workforce that transitions from farm to farm, or state to state. And so there’s always this risk of not having the workers that you need at the right time. And so I think initially, the robots will help fill that gap. And then as the technology matures over the next, you know, 10 to 20 years, then we’ll start to see the kind of the widespread penetration where robots are picking most of the fruit rather than just helping fill a shortage.

HAUTALA: So farm jobs, as we know them today, appear to be safe — for now. But in another decade or two, will we see farm workers going head to head with their robot rivals? That brings us back to John Henry. History seems to show that whenever it comes down to a battle royale between machines and human workers, the humans lose out, every time. So, what is to become of farm labor in the 21st century?

DAVIDSON: I’m sensitive to the fact that the growers and the farm managers, you know, are really interested in this technology, but there are still people that depend on this work to take care of their families. And so there are a couple of really interesting initiatives, and one example is at the University of California, Davis, they have this initiative called the smart farm.

[MUSIC: “See you soon” by Otis McDonald, part of the YouTube Audio Library. Licensed under a Creative Commons license]

And so they recognize that technology is going     to displace ag workers in the future. And so to compensate that for that, what they’re doing is they’re reaching out — to trade unions, local community colleges, technical schools — and trying to help come up with programs and processes to retrain workers to become technicians. Right? And so now instead of being bent over in a strawberry patch all day, or going up and down a ladder — and let’s be honest, this is very, very difficult, physically strenuous work — that now, instead of doing that, maybe we can may have you overseeing a drone that’s collecting data, or being a technician to make sure that the robotic picker is operating the way it needs to, and try and take it from a low skilled, physically demanding work to a highly skilled technician-type job.

HAUTALA: And, of course, in the years and decades to come, advanced technology will find many other jobs to do on smart farms, creating need for more technicians, operators, and other skilled jobs. Joe’s interest in agricultural robotics goes beyond apple-picking as well.

DAVIDSON: We’re still working on harvesting, and we are getting ready to start a new project with Cindy Grimm, another one of my colleagues from the robotics group. We’re working on pruning. So harvesting is the most labor intensive activity, and pruning is No. 2. And so we’re trying to study robotic pruning, right? How do you create a model of a tree? How do you decide which branches to cut? And so I think in a lot of ways, pruning is probably more complex than harvesting, because now you’re not just thinking, “OK, how do I get this apple off the tree or this piece of fruit off the tree?” It’s “OK, which branch looks dead? Which one do I want to cut? How do I want this tree to look next year? What’s the density of fruiting sites across this linear meter of branch that I want to keep?” So there’s a lot more of these decision variables that come into the process.

HAUTALA: Speaking of lots of decision variables, the next big thing in farm automation is something called precision agriculture. It’s a data-intensive way of farming that relies on gathering information about individual plants and applying inputs specifically where they are needed. Joe is working to develop a new project in this area as well.

DAVIDSON: Right now growers make all their management decisions — you know, how much fertilizer am I gonna put down, herbicides, pesticides, all of that — at the orchard, the large-block level. And what we want to try and do is make these decisions at the individual tree level because, even in an orchard, there’s variability in soils and there’s just variability because of that. And for lots of different reasons, there’s variability in the nutrition status of an individual plant. So we want to do is say, OK, can we develop systems that can identify what’s the amount of nitrogen this tree needs? And can we record that? Can we map that and make a big database of the entire — every tree in the orchard and each of the — this tree’s specific status, its nutrition.

And nitrogen is where we’re starting. And then can we come back at the right time of the year and do variable rate application of the product. So can we come back, access our map when it’s time to put down nitrogen, say, OK, we knew this tree needed this much. And so now we have a system that targets the right decision for that tree. And so, you know, we would have the, the optimal yields and can we get our yields up because we have these targeted applications for the plant.

HAUTALA: For robots, the future on the farm looks bright indeed. This episode was produced by me, Keith Hautala, with recording and audio editing assistance from Molly Aton and production assistance by Rachel Robertson. Our intro music, as always, is “The Ether Bunny” by Eyes Closed Audio. You can find them on SoundCloud, and we used their song with permission of a Creative Commons attribution license. Other music and sound effects in this episode were also used with the appropriate license. For more episodes, bonus content, and links to the licenses, visit engineeringoutloud.oregonstate.edu. Subscribe to this podcast by searching for “Engineering Out Loud” on Spotify or your favorite podcast app.

Thanks for joining us this season. We hope you’ve enjoyed talking about robots and what they might do for us in the future. Uh, one more thing. Jimmy Fallon: Watch out! Jon the Robot is coming for your job next.

JON THE ROBOT: In my family. I’m the black sheep. My father is IBM’s Watson. He beat Ken Jennings at Jeopardy. My sister is Google’s AlphaGo. She beat Lee Sedol, a world champion Go player, and I am a comedian. My audience has beaten my self confidence into the ground.

For more Oregon State University Engineering Out Loud podcasts, visit: https://engineering.oregonstate.edu/outloud

Original post https://alertarticles.info