Overheating only uphill either trans fluid or oil in coolant reservoir?

So I recently bought this car not even a month ago ( obviously I got fecked over) but it has been running great for the most part no overheating that I noticed, until I went out of town and had to drive on straight incline. It was struggling to go up hill and I heard bubbling and pulled over, coolant shot out everywhere and was bubbling out the coolant resevoir. I look in thereafter it cools down and there is brown sludge problem is I don't know if it's oil or trans fluid. I bought a new thermostat to put in but I haven't drove it and won't until I figure out whether it's the radiator or the head gasket? Help! I can't afford a mechanic, but have a friend to replace a head gasket IF needed.