Translating whale, with the help of AI

Canadian marine biologist Shane Gero was off the coast of Dominica in the Caribbean when two whales who were about 3 years old, nicknamed Drop and Double Bend, were playing and chatting. He was eavesdropping on them, with a microphone, and he wanted to know more about the conversation.

“We landed on the moon long before we really spent any time paying attention to the deep, dark ocean,” said Gero. “And here’s a family just on the other side of the surface talking to one another. And even after decades of listening, we’ve only started to scratch the surface about what they might be saying to each other.”

Gero is now working on Project CETI — an ambitious effort that stands for Cetacean Translation Initiative. The project uses advanced technology like artificial intelligence and robotics to better understand whale communication.

Gero talked with Shannon Henry Kleiber for “To The Best Of Our Knowledge” about the latest breakthroughs in understanding what whales are saying.

This transcript has been edited for clarity and length.

Shannon Henry Kleiber: I love how you have such curiosity about what these whales are saying to each other. What did it sound like?

Shane Gero: The first time I heard sperm whale codas, it sounded, to be honest, kind of simple. It’s a series of clicks and pauses, sort of like Morse code.

It’s amazing because we’ve learned so much about their social lives in Dominica, about who the whales are as individuals and what they do in their family and how they babysit one another, and how these different families spend time with each other. This rich, complicated life that’s happening.

And so it must be mediated in some way by the sounds they make to one another. But exactly what that is unknown. So we record huge amounts of these codas, these patterns of clicks, as the whales talk to one another when they’re diving, when they’re socializing as a family or when calves are playing with each other, like Drop and Double Bend were on that day.

SHK: I want to know what they’re saying, too, and whether they’re saying a command like, “Come here,” or something that has emotion. Why do you think people are so fascinated with what whales are saying?

SG: I think we humans have been thinking about what animals are saying for centuries. Every kid at some point between 6 and 12 — it doesn’t matter what country you’re from — gets struck by how amazing the world is. Maybe it’s rocks. Maybe it’s bugs, maybe it’s birds, maybe it’s whales. It’s a realization that there’s this entirely complex, amazing, dynamic thing going on in parallel to our own small world inside our own brain.

SHK: You’ve been following these clans of whales in Dominica for decades. Do you get to know them like part of your family?

SG: Oh, absolutely. My kids have bath toy sperm whales that all have names of real sperm whales in Dominica. We do call them my “whale family” as opposed to my human family here in Ottawa.

Back in Canada, we found a population, a community of sperm whales that were easier to study. Traditionally, these were nomadic, open ocean animals. In Dominica, we’ve identified 35 different families now, and there’s probably another 16 or 17 other sperm whale families that spend a lot of time in the waters off Dominica.

So we see them for a number of days in a row, and then they will leave and another family will move in. And the residing rate is high, meaning we see this family this week, this month, next month, next year, and year on year on year. That means we can really drill in and ask questions about who the animals are as individuals. When were they born? When were they first a mom? These really kind of delicate parts of their family life. How does the babysitting work? And Family A versus Family B versus Family C? Because unsurprisingly, every family is different.

These are big-brained mammals. The community’s probably less than 600 animals. And that’s divided into different cultural groups. So sperm whale families behave differently. They have different diets, different habitat use, different movement patterns, and they label those differences by differences in their vocal dialects. Different repertoires of these coda patterns, and some of those codas are symbolic of the set of families they’re from.

SHK: So really like a dialect.

SG: Yeah. In the same way that at least in certain instances, I spend a lot of time saying I am Canadian because that becomes important, say, at a border or during the Olympics or whatever it may be. Sperm whales appear to take a lot of time broadcasting their identity, and in the Caribbean, they make a coda that’s not recorded anywhere else in the world.

And it’s called the one plus one plus three. And it sounds like click, click, click click click:

SHK: And what does that mean?

SG: That’s a great question in terms of the exact information that’s encoded in that. The way we think it functions is as a symbolic marker of belonging to a clan, to all the families that share the same dialect, that share the same behavior, the same habitat, use the same movement pattern, the same diet, the same social behavior, and probably a myriad of different cultural variations to how to live a sperm whale life.

Culture, whether it’s humans or whales, is really just a set of solutions on how to survive in the world that you live in, right? Behavior is what you do, but culture is how you’ve learned to do it. Humans are all humans, but some eat with chopsticks, some eat with forks, some eat fish on Friday, some never eat beef. We still all eat and move and love and care for our babies and mourn and have funerals. But how do we do all of those different things? You learn as you grow up as a baby, and that seems to be true in sperm whales, too.

SHK: Have you had any notable surprises in your work? Something that has just really taken you aback or been most exciting?

SG: The whales find ways to surprise me all the time, but I think of the intangibles that we can’t talk about in science yet. When we first started doing research, we’d get behind the whale and the whale would lift its tail in that sort of quintessential whale watch moment. And we take a photo because we identify them by the trailing edge of their tail. Then when they dive, they leave this calm circle of water called the fluke print. We would go and collect skin and feces, and drop our hydrophone in the water.

This one whale named Can Opener realized that pattern. She saw something novel in her environment and that we did the same thing over and over again. And then, at least as an outsider, it’s hard not to see this as future planning, which is something very hard to show in an animal. Can Opener would fake her dive. She would lift her tail out, dive and then wait for us just under the surface. When we would pull into the fluke print and drop the hydrophone, she would blow all the bubbles out of her blowhole and come to the surface. And importantly, she didn’t echolocate on the boat. She didn’t use her sound to see the hull that was below the surface. She would roll her eye out of the water and follow the people — the biologists who at this point are giddy to have this sperm whale pay attention to them and look at them run up and down the boat.

SHK: It sounds like she was playing a game with you.

SG: Oh, yeah. We would play hide-the-hydrophone because she wouldn’t go about her business. So we’d have to wait longer and let her actually start a dive before bringing the hydrophone in. Or she would run her jaw along the length of the hydrophone, sort of inspecting it because they can’t touch things with their pectoral flippers.

So that, to me, breaks through what we can show quantitatively in the science of what these animals are and who they are. But it sort of propels us to be like, well, how do we devise science to understand that to be true? How do we show that using the process of science? And that’s really tricky.

SHK: One of the things I think is so interesting about this time is that technology is helping you do this. It’s kind of ironic, yet beautiful that technology can help us understand nature better. You’re using artificial intelligence and robotics. Is that helping — in this moment — in understanding whales better?

SG: Totally. In so many different ways. A big part of learning a new language is hearing it a lot. And so the challenge of recording large amounts of audio has been met through novel ways in Project CETI, by creating new technology to do the recording, to record continuously, to record on the backs of the whales, to record autonomously through things like gliders, which are small autonomous submarines that can patrol off the island of Dominica, and can listen for whales when humans can’t be at sea. So that’s one way where technology is bringing us a huge step forward.

There are a number of problems in the way we studied sperm whale quotas that were limiting. For example, if you wanted to learn English and you put one microphone and only one microphone in a dentist’s office, you would think the word “root canal” is critically important to English-speaking society, when really it’s just a huge bias in the context in which you’re recording. Breaking out of that and being able to record across context and continuously helps us learn a broader swath of language than we would otherwise.

But then you face this massive problem of, well, now we have hundreds of hours of audio, so what do we do? And that’s where things like machine learning and neural network processing has accelerated a lot of things, but used to take eight to 12 minutes of human time per minute of recording to just extract the sperm whale codas out of the recordings. And now that can be largely automated by teaching neural networks to detect, annotate, classify each of the calls. And that allows us to have a much bigger data set to answer these questions about what animals are saying. That’s the first step.

But what is the variability? It’s not just changing in rhythm from baba. They change the tempo when maintaining the rhythm.

So you can have bap “bap bap bap bap” and you can have “baba baba baba“:

SHK: So it might be the same words but a different meaning.

SG: Before we knew that variability existed, we would treat those as one thing. Now that we know that variability exists, we can ask that question: Is it an emotional state?

Like I tend to speak higher and faster when I’m excited or scared. So then you need to take all of that variability that you’ve discovered and look at where and when they’re doing those things. Is it before diving? Is it when they’re babysitting? And then who are they doing it with? Is it only with elders? Do they talk slower when they’re talking to calves as a lot of adults do? So then you have the what and the who and the where and the when. Then you can start asking the why, what kinds of information might be they exchanging, why are they talking to each other? Why make sound now as opposed to any other time? And that’s where AI or neural networks are really excellent at drawing patterns between different types of data streams.

SHK: Are you putting whale sounds that AI has created down into the water to communicate with the whales?

SG: We’re not doing that yet. From my perspective, Project CETI is a large-scale listening project. We’re trying to understand what whales say to one another when whales are being whales.

And there may come a point when we need to ground our ideas of what these things mean. Answering that why question about the meaning of these sounds is where we may need to play sounds back to sperm whales. The project is geared at learning from the animals rather than centering it around people, and what we might want to say to them.

SHK: We’re trying to understand more by listening.

SG: How much good can we do in the world by all just listening to someone so different from us for just a little while, figuring out what’s important to them, and then instead of asking, what are we going to do about it, we ask, “What can we do in changing our behavior in order to make the world better for them?” And then there’s the second question later of, “Do they even want to have a conversation with us?”

And I think doing that centers humans in a way that is not a part of my ethos on studying animals. There’s a great quote from Cormac McCarthy, who, of course, was a long time friend of Roger Payne — the famous marine biologist who died in 2023 and is credited with introducing humans to whale song — where in one of his writings, he wrote, “If God comes back and asks you if you’ve figured it all out, and before you can answer, he asks, did you think about asking the whales? And in looking around you said, where have all the whales gone?” And I think well, what lessons can we learn by listening to them first before we worry about whether or not they’re even interested in hearing about what we have to say?

Because a big takeaway about being a sperm whale is to be a good neighbor. And fundamentally, humanity has not been a particularly good neighbor to any of the citizens of the ocean. So I would probably understand if they were pretty hesitant.