Arrival in Westeros
Around 12,000 years before Aegon's Landing, during the Dawn Age, the First Men came to Westeros from Essos by crossing the land bridge called the Arm of Dorne. The First Men crossed the Arm under the leadership of the First King, according to legends from the north, or Garth Greenhand, the High King of the First Men, according to some legends from the Reach.
The First Men supposedly found the Seastone Chair upon the shores of Old Wyk when they first came to the Iron Islands. It was the First Men who named those who sing the song of earth "children".
War with the Children of the Forest
The men came with bronze swords and great leathern shields, riding horses. As the men settled in the new land, carving out holdfasts and farms, they chopped down the carved weirwoods that were sacred to the children of the forest's gods and burned them. This provoked a war between the children and the First Men. Though the children had powerful magic, and, according to the old songs, used dark magic to shatter the Arm of Dorne into an island chain, the First Men were larger, stronger, and more technologically advanced.
Whenever the First Men warred upon the children they cut down the trees as they believed (quite correctly) that the greenseers could see through the eyes of the weirwoods.
Moat Cailin was raised roughly 10,000 years ago, by the First Men. It appears as if at some point the children were able to take possession of at least Moat Cailin's Children's Tower. According to myth, the children attempted to use Moat Cailin to hold back the flood of invading First Men by calling upon their nameless gods from the Children's Tower to send down the "hammer of the waters" to break the lands of Westeros in two by shattering the Neck and completely separating the north from the south, in the same manner they shattered the Arm of Dorne centuries earlier. However, the children failed in the attempt and only succeeded in flooding the Neck, creating bogs and swamps.
However, the cataclysm proved the strength of their power and may have proved instrumental in bringing the First Men to agree to the peace terms that ended hostilities between the two races.
The Pact
The wars went in the First Men's favour until the two sides reached a peace agreement, called the Pact, on the Isle of Faces. The First Men gave dominion of the deep woods to the children and promised not to put any weirwood trees to the axe anywhere in the realm. In return, they received claim to the rest of the Westeros.
Four Thousand Years of Friendship
The Pact began 4,000 years of friendship and peace between the two peoples. The years that followed the forging of the Pact is known as the Age of Heroes. In time, the First Men eventually set aside their religion to worship the children's secret gods of the wood. Like the children they would carve faces into the weirwoods. The children of the forest taught them to use ravens to communicate over long distances, but in those days the birds would speak the words, and the greenseers of the children could change their skins and speak through the birds.
When the Long Night came to pass and the Others began to invade from the far north of Westeros the First Men and the children joined forces. A legendary hero, called the last hero, is said to have led the coalition against the Others. The Others were driven back and the Night's Watch was created to keep them at bay. The First Men are also the people responsible for building the Wall, which still stands to this day.
Invasion of the Andals
Roughly around 6,000 years before Aegon's Landing, the Andals crossed the narrow sea and began their invasion of Westeros, ending the Pact. For several hundred years the First Men and the Andals warred, fighting for control of the continent. Eventually, the Andals conquered or married into the kingdoms in the southern half of the continent, while the First Men Kings of Winter stopped all Andal incursions through the Neck at Moat Cailin.