So far the biggest problem with NSA's collection of meta-data from various American phone companies is that it is spying on Americans. NSA, Obama, and Congress argue that collecting just the phone numbers, locations, times, etc., in not an infringement of Fourth Amendment protection against searches and seizures. However, it is collecting information about Americans that can be used for intelligence purposes, and the fact that it is stored by NSA means that it is already treated as intelligence data. This data can be mined for many types of information by NSA, some legitimate and permitted under the Fourth Amendment, and some not. It's sort of like saying that the government has the right to set up microphones and cameras in your house to record your every move, but it doesn't have the right to look at it, unless it gets a court order. Maybe NSA is being law abiding, and maybe they are not. Maybe they are being law abiding now, but won't be in ten years, but they will still have the data to mine for inappropriate information.
If the US faced a clear and present danger to its survival, then this program might be justifiable, but I don't think that it does. The terrorism threats we face are low-level and usually amateurish. If you weigh the threat against the loss of civil liberties, I think that loss of civil liberties far outweighs the threat. The threat does not justify spying on Americans, even if this spying is just recorded and not looked at. This is exactly the kind of thing that Hitler would have used against the Jews. In today's world, he probably could have found out where Ann Frank was hiding within hours by collecting and analyzing the meta-data of the electronic footprint of the family hiding her.