Given the recent problems on CodePlex, I have had to change the TFS server that one of my projects is hosted on. I found this to be a problem because I wanted to use the same directory for the source on my local drive. Even after removing the source control bindings and removing the TFS server settings, Visual Studio still complained that the directory is configured for the old server address.
TFS stores workspace directory mapping information in a VersionControl.config file located in C:\Documents and Settings\[YOUR PROFILE]\Local Settings\Application Data\Microsoft\Team Foundation\1.0\Cache\. The old server isn't removed from this configuration when you remove the server from TFS in Visual Studio. This bit, you have to do yourself.