It is always your choice not to have this test. But, it is important to understand the possible consequences that may be involved with this decision.
In some cases your doctor can still provide you with medicine and therapy that can minimize your symptoms. However, without a clear diagnosis, the best therapy becomes more difficult to prescribe. Furthermore, with certain patients, medication and therapy is NOT enough to keep your condition from potentially getting worse.
The final decision to have or not to have this test rests with you, but all concerns relating to this decision should be discussed with your doctor at length. You and your doctor can then come up with a treatment method that is comfortable for you.