This is what's up.
Back a few thousand years ago, there was a lot more energy present in the atmosphere of the Earth.
This energy supported the life of giant mammals- mammoths, sloths, and humans, among others.
In those days, there were giant people, separately people like us, and very small people. These different people all had their own areas of the world that belonged to themselves, and other kinds of people generally wenre't allowed to enter.
All these different types of people had their own roads, bridges, infrastructure, etc etc
In different places there were certain groups or classes of normal people who were in charge of interacting with these other types of people.
Stories of people meeting trolls and having to solve riddles to cross bridges and things like that, are memories of people encourntering non-human infrastructure.
In these cases, only the classes and groups of normal people who were designated to interact with the other people and knew the 'secret passwords' were allowed to use the infrastructure of other kinds of people.
When non-insider people encountered these things, it sounded like they were being given nonsense riddles.
This division of the Earth's surface was the cause of the historically low human population relative to what we see today.
For unknown reasons, the energy present in the atmosphere became dramatically reduced in a short period of time.
This led to the death of all the large animals that were reliant on that atmospheric energy- including the giant people.
At that time, it appeared to them to be some sort of disease or sickness. Very likely the decline in energy available to their bodies led to increased susceptibility to disease, but this was not the underlying cause.
Due to the isolation of the different types of people in the world from each other, it was not immediately obvious to normal people that the giants were dying