Engine Knocking at hot idle

I own a 1984 AMC Eagle station wagon, with a 258 cid engine, (4.2L), approximately 103,400 miles on it. I have made some improvements to the original engine, such as I installed a Throttle Body Fuel Injection system, which uses a GM computer for control. I then installed a GM HEI distributor, to improve the spark. (The fuel injection was hard starting and flooded with the older Duraspark ignition, so I replaced it with the aftermarket HEI, which solved the flooding at start up.) The engine ran good, for over a year, then a few months ago, it started running rough when it was idling, and even started knocking, but only when idling. At higher rpms, and running down the road, it didn’t sound at all. I once again replaced all the spark plugs, thinking maybe one had become fouled, although they had been new when I installed the HEI distributor, new plugs and new wires. The plugs looked okay, only slight tanning. I thought that maybe the rocker arms had started knocking, so I replaced all the rocker arms, balls and bridges, tightening them up to specs. While replacing the rocker arms, I thought I might as well replace the intake manifold, as I wanted to increase my torque and horsepower a little. The car is heavy, being a 4 wheel drive, so the 6 cylinder engine is in need of a boost. While I had the intake manifold off, I realized that the TBI that I was sent was a small bore, made for a 2.5 Liter engine and not the 4.3 Liter TBI that I was supposed to get. So I then bought another TBI for a 4.3 Liter. (My engine is a 4.2 Liter). That helped improve the power and torque, along with the Offenhauser intake manifold. But my knocking problem still exists. Here is how it happens. When I start the car cold, it runs and doesn’t knock, except for some mild tappet noise, typical of a 6 cylinder. But when it heats up, such as after a run for ten minutes down the highway at 55 to 65 miles per hour, then the knocking is very loud, BUT only when the engine rpm comes down to idling. More so, when the car is in gear, and the idle is slower. A local mechanic says it might be the camshaft thrusting forward and hitting the timing case. I think it might be a weak hydraulic lifter, which pumps up okay while cold, but loses pressure when hot, and causes the lifter to rattle. My problem is, that this car is sort of a “keeper” car, I don’t want to trade it in. My option therefore, is to replace the lifter, if in fact it is causing the knocking, and if I did that, I would end up replacing all the lifters, and also, probably the camshaft, ... Well, old part, new parts. I would probably be better off replacing the whole engine. Sad, because it runs good at higher rpms. What else could be causing the knocking, when the oil is hot, and the rough car shaking idle?