We need to have a public discussion about the balance between security and privacy. What can the government learn with telephone metadata? It is relatively simple to sketch a profile of every individual with some simple metadata.
Inference is one of the emerging fields of technology in business and presumably national intelligence. Did you list your job title on a professional social networking sight? Is your number public? If so, it would be trivial for the NSA to add a job title to phone numbers it is collecting.
With a collection of metadata the power comes in matching patterns and in linking it with other data like publicly listed phone numbers or addresses. All of that can be amassed without court order and can simply be bought on the data market. Should you be concerned? Yes. Let's discuss the next steps in a transparent and thoughtful way.