When in neutral if I apply the gas the idle comes back down VERY slowly

The first think to look at would be the Idle Air Control Valve, IACV. I have included a diagram of its location. On all models it is in the rear of the rear center of the engine compartment, except for the SHO. I included it too, just in case.
You can try cleaning it or you can replace it. Check the connections and any vacuum lines it may have.
Before you clean or replace the IACV, go to get the trouble codes pulled. They will help to diagnose the issue. It is a good way to look at the whole intake/fuel management system as a whole.
Most Advance or Auto Zone parts stores will scan the codes for free.
If you need any further help, I will watch this post for your reply and get back to you ASAP.

That sounds like more of a fuel pressure issue. This can be related to a fuel pumps' check valve which keeps fuel in the system over extended periods of time. Even though the car has a priming function, where it starts the fuel pump in the ON position to put the fuel pressure in operating range to start the car, if the fuel has drained back into the tank the priming function will not be enough to start the car and extended periods of cranking will result.
Since you have the problem with the hanging idle and the hard starting, another possibility is the fuel pressure regulator. It is vacuum operated and can cause running and starting problems if it is not functioning correctly.
Here is a test procedure for checking the fuel system in general;

Basic Fuel System Component Check

Inspect the gas tank cap for damage or corrosion. Replace the cap if its gasket has a broken or damaged sealing imprint.

Look over the fuel supply and return lines. Check for any cracks in the lines. Make sure the lines' connections to the injection system and the in-line fuel filter are tight.

Check the gas tank and filler neck for cracks, punctures or other damage. You need to raise the car on jack stands for this. C
heck the connection to the tank and filler neck, looking for a loose clamp or deteriorating rubber.

Make sure the mounting brackets and straps securely attach the tank to the car.

Inspect all the metal lines and rubber hoses leading away from the tank. Replace or repair any loose connections, crimped lines or deteriorated hoses.

Relieve the fuel pressure before inspecting the system internally.

The Taurus has an inertia switch reset button you can push to relieve the pressure immediately. This switch is in the trunk behind the trim panels on the right side (1996 to 1999 models) or driver's side (later models).

Fuel Pump Check

Make sure the fuel pump works. Remove the gas cap and listen near the filler opening while someone turns the ignition on. A whirring sound for a couple of seconds means the fuel pump is operating.

Check the fuel pump's fuse if you didn't hear the sound (this fuse is Number 10 in the fuse box). Replace the fuse if it is blown. If it blows again, check for a short in the fuel pump circuit. If the original fuse is good, test the relay operation by applying battery voltage to Circuit 238 in the fuse box (1998 or later models only).

Test the fuel pump with the ignition switch again. If you hear the whirring sound now, the fuel pump relay or its control circuit is bad. If you still get no sound, the problem can be the relay, control circuit or the fuel pump itself.

Fuel Pressure

BE SURE TO RELIEVE THE FUEL PRESSURE VIA PRESSURE RELEASE PROCEDURE OR FUEL
Relieve the fuel pressure before inspecting the system internally.

The fuel pressure regulator is located on the fuel rail. Connect a fuel pressure gauge to fuel pressure test port (you need to remove its cap). The gauge must have Schrader valve adapter. Turn the ignition on and wait for the fuel pump to pressurize the lines.

Observe the gauge's reading, once it has stabilized. For a 1996 to 2000 Taurus, the pressure should be 26 to 45 pounds per square inch. On newer models, it should be 45 to 56.

Start the engine and let it idle. The fuel pressure should now be 26 to 45 psi for a 2000 or older Taurus and 50 to 56 for newer models.

Remove the vacuum hose from the fuel pressure regulator. Test the fuel pressure regulator if the fuel pressure doesn't rise several psi with the hose removed. Connect the vacuum line back immediately.

Pinch the fuel return line shut if the pressure reading was low and observe the gauge again. Replace the pressure regulator if the pressure sharply rises now. If it doesn't rise, check for a restriction in the supply line. If there is no restriction, you have a defective fuel pump.

Check for a blockage in the fuel return line if the pressure was too high. Turn off the engine, disconnect the line and blow into it. Replace the regulator if there is no blockage.

Use a vacuum gauge on the regulator's vacuum hose to check for vacuum with the engine started. Replace the regulator if you get a good amount of vacuum. If there is no vacuum, check for a clogged vacuum port or hose.

Turn off the ignition and check the fuel pressure gauge after1 mintue and 5 minutes. The fuel pressure should be about 25 PSI after one minute and if the pressure drops more than 5 to 7 psi, you could have a defective fuel pump, leaking fuel lines or injectors or fuel pressure bleeding to the return line.

The last step is to check the check valve that keeps the fuel in the line and pressurized for starting the car after it sits without running.

Before you do any of this, I may have already mentioned it, go to an Advance or Auto Zone parts store and have them pull your trouble codes. They will do it for free and it will help to eliminate some of the possibilities in diagnosing the problem.

Let me know if you have any more questions or need any more help.