I think parts of it are deeply controversial, but Guns, Germs, and Steel argues this was because Europe had higher population densities for longer + more domesticated livestocks providing a more potent breading ground for deadly diseases. I also think that disease being a factor hardly removes culpability from the conquers, there are plenty of quotes of some of them saying things about how the plagues were a gift from god and similarly terrible things. I also am not an export, but I believe there was some transfer in the other direction, particularly syphilis.

If we're just speculating though, I wonder if the fact that one group was traveling by boat could have insulated the disease transfer a bit. Most really bad diseases would run their course by the time a sailing ship made it back across the ocean and certainly people knew to quarantine ships with sick people on them in Europe. For a disease the ship crews were resistant to reach the Americas they just had to visit a village, where to go the other way it had to survive an in built month plus quarantine which is plenty of time for most diseases to show up