Why are you buying it? If you are buying it to store your wealth, then now is as good a time as any since gold is roughly as valuable as it has always been. The price of gold has been driven down by fractional reserve paper gold flooding the market, so actual physical gold is really dirt cheap at these prices.
If you are buying to cash them in for dollars in the near future then I guess you need to worry about these things, but it is still a good time to buy. The only thing the Fed could do to crash the price of gold is to jack interest rates up, and that isn't going to happen. They would bankrupt the federal government if they did. They would sooner destroy the dollar than bankrupt the federal government.
They can sell more paper to try and keep the price as low as possible, but at a certain point people will catch on and demand their gold, so they won't want to trigger a run on the paper gold funds. There is very little they can do to bring down the price, they can only fight to keep it from rising too much.
Being an election year though, you could see the Fed printing money to give Obama a better looking economy, you could have a new war or two, you could have more and bigger Euro bailouts. All of these would shoot up the price of gold. I would say no matter the reason it is a good time to buy if you are able.