Historically, yes. Very few cultures are known for having a large populations of swimmers in the past, even the average Japanese civillian before the 20th century probably never learned to swim despite living on a chain of islands.