VIMS scientists want to hear your fish tales… or at least see your photos

Researchers at William & Mary’s Virginia Institute of Marine Science are casting a wide net in search of fish photos. Their immediate goal is to use angler snapshots to train software to identify different fish species. Their ultimate goal is to put that artificial intelligence into a “RecFish” cell-phone app, giving anglers a multi-use field guide and scientists a collaborative tool for better management of recreational species.

“We’re really excited about building out our app,” says RecFish founder Lisa Kellogg, a senior research scientist at VIMS. “Once finished, all you’ll need to do is open the app and point your cell-phone camera at a fish and it will instantly tell you the species, size, and approximate weight, and let you record that info to your logbook with the touch of a button.” The app will also instantly provide information on whether the fish is legal to keep based on the date, location, and local fishing regulations.

But before that can happen, the app-development team must first train machine-learning software to recognize different fish species, a process that requires lots and lots of photos. “Right now,” says Kellogg, “we’re working on training models to identify fish and that requires at least 5,000 photos per species. The more photos we have, the more accurate the models will be.”

To help collect these images, the team has developed a website at recfish.org that makes it easy for anglers to upload their photos. Kellogg says, “We want people to look through their photos and upload anything that has a fish in it, so that we can use them as part of the model training.” Their initial focus is on fishes of the Chesapeake Bay, but their long-term plans include expanding coverage to the U.S. East Coast and beyond.

The team is particularly interested in photos of less common species and species that people rarely photograph. “There are 270 resident species in Chesapeake Bay,” says VIMS fisheries professor and Recfish team member Dr. Eric Hilton, “from common fishing targets such as striped bass, croakers, and flounder to more unfamiliar species such as stargazers and sea robins.”

“In reviewing the photos we’ve collected so far,” says Kellogg, “we’ve found that everyone takes photos of that huge striped bass they caught but far fewer take photos of their toadfish.” Particularly surprising has been the difficulty in getting photos of even some of the Bay’s most commonly caught species.

“Spot are the most commonly caught species in Maryland and Virginia but we still don’t have enough photos to model the species,” says Kellogg. “We’re hoping to get the word out that we need photos of all of the fish you catch, not just the ones you’re most proud of catching.” Despite the low number of photos they are currently working with, initial modeling is going well, with some species models already more than 90% accurate.

Kellogg and her project team appreciate that participating anglers may not want to share the whereabouts of their secret fishing spot. “RecFish will let you choose whether to share your location data with scientists and resource managers,” she says. “You’ll be able to share your exact location, which has the greatest value for scientists, but if you’re uncomfortable with that, you can submit your data just by tributary segment–upper, middle, or lower York River, for instance–or with no geo-reference whatsoever.”

“Even if some anglers never contribute their catch data,” she adds, “the app will contribute to sustainable fisheries management by helping them identify their catch and providing up-to-date fishing regulations.”

Hilton notes that the app will also aid with basic fisheries science. “Anglers collect more data than any fisheries biologist ever could,” he says, “because they’re covering such a large area, including places that aren’t sampled scientifically.” Shared photos will also help document new or unusual occurrences of species within the Bay. This includes introduced species such as snakeheads and blue catfish, as well as more southerly species like the long-spined porgy that are entering the Chesapeake naturally due to its warming waters.

In addition to Kellogg and Hilton, the app-development team includes VIMS researchers Sarah Muffelman and Jennifer Dreyer; Dharmesh Trivedi, Harshil Shah, and their staff at the machine and deep learning firm DXFactor; and Rob Quartel, Innovation Advisor to VIMS Dean & Director Dr. John Wells.

App development was jumpstarted by support from the Dean & Director’s Innovation Fund at VIMS, established in 2018 by the Joan and Morgan Massey Foundation and the Nunnally Charitable Trust. Kellogg was one of two VIMS researchers to receive the fund’s inaugural award in 2019, and she received it again in 2020 for further app development.

“The goal of the Innovation fund is to provide support for research and education activities that foster innovation and economic productivity at VIMS,” says Wells. “What Lisa and her team are doing with their Recfish app is a great example of the kinds of projects we are looking to foster.”

“The VIMS Innovation Fund encouraged us to think outside the box and come up with this concept in the first place,” says Kellogg. The National Fish and Wildlife Foundation has provided additional funding for app development, and a wide range of management agencies and NGOs have expressed support. Kellogg expects the app to be available to the public by the end of 2021.

The mismatch that spawned the RecFish app
Kellogg says her initial idea for the RecFish app wasn’t based on fish, as you might expect, but on oysters and birds.

“I’ve spent most of my career studying oysters,” says Kellogg, “but when we started looking at oyster reefs as a foraging ground for finfish, we found mismatches with what our Trawl Survey folks were seeing in their broader data on the diet of some species. What we realized is there’s a lot of oyster reef and seagrass areas up in Bay tributaries that don’t really get sampled even by a state-of-the-art survey like ChesMMAP.” That’s because oyster reefs tend to damage trawl nets, and trawl nets tend to damage grass beds. Those concerns limit trawl surveys to waters deeper than about 9 feet, excluding large areas of the Bay’s generally shallow tributaries.

This discrepancy led Kellogg to wonder what fishery managers and ecologists might be missing. “There’s a big push towards ecosystem-based fisheries management,” she says, “but by not sampling oyster reefs or seagrass beds, we’re missing some of the most important connections between fishes and their environment, including predator-prey interactions.”

Knowing that anglers often specifically target these structured habitats, Kellogg started thinking about how she could collaborate to leverage their efforts. “There’s this huge untapped potential of recreational anglers,” she says, “so we began trying to figure out how we might be able to turn them into community scientists, how we might have them collect and share data.” NOAA estimates anglers take about 8 million saltwater fishing trips each year, over half of them along the Atlantic coast.

That’s when Kellogg’s mind turned from fish to birds. “I had in mind the success the Cornell Lab of Ornithology has had with eBird and Merlin,” she says, “apps that allow birders to collect data on the birds they see. They’ve been so successful they now have millions of birders submitting over 100 million bird sightings every year.”

To create a similar app for anglers, Kellogg and her team realized they would have to make some changes. “We recognized there are some pretty big differences,” she says. “Some of these will be advantages and others will be challenges.” One advantage is that anglers directly handle and frequently measure the length and weight of their catch, providing the opportunity to collect more data than from a bird sighting. On the other hand, many birders keep a life list, while there has been no similar tradition among anglers. Anglers may also be more hesitant to share the exact location of their catch.

To address these concerns and encourage the app’s use, the RecFish team has already incorporated a set of core reporting functions: species, length¬ and weight, date and time, legality, and a choice of data granularity in terms of catch location. They now want to hear from anglers about other looked-for features, with a form on the Recfish website for user feedback.

“We want anglers to tell us what else they’d like to see in a recreational fishing app,” says Kellogg. “Do they want to know whether the fish is edible, or about health hazards, like a high mercury level? Do they want more information about the fish–it’s diet, habitat, and range? Do they want to know the maximum size, the state record for that species? Would they like the app to show what you were catching last year on that day? Do they want to this to be a photo-sharing app where you can brag about your catch or compete with friends? The sky’s the limit. We’d love to hear everyone’s suggestions.”

###