Swiss voters have rejected a proposal to cap the country's population at 10 million.