The set up for Season 6 included the mystery surrounding Jacob's apparent rival, the Man in Black.
Non-canon references aside, we never learn this man's name.
In Season 6, MIB was the haphazard Island tour guide marching groups all around the Island in circles waiting for the final showdown. It is unclear whether Jacob kept MIB from leaving the Island, or whether the immortality of the Smoke Monster form kept him trapped by the Light cave cork.
The Man in Black lived with the original Others, the other survivors from his real mother's shipwreck. By living with these survivors, he believed that the Island wasn't his real home. He learned to question things by living with the survivors. Over time, he developed knowledge on how to leave the Island by creating the Frozen Donkey Wheel, to harness the power of the Light. He was correct, since the FDW did transport people from the Island to Tunisia. Crazy Mother thwarted his plans and killed all the ship survivors in his village. He, in turn, killed Crazy Mother with a dagger in a rage. Jacob took revenge by casting him into the Light Cave, which transformed him into the Smoke Monster.
The brothers spent the following centuries in conflict, drawing people to the island to test their nature. MIB grew weary of the humans Jacob brought to the Island. (It must have been just as Crazy Mother did, to find a replacement for her island prison). One would have thought that after centuries of humans being brought to the Island, MIB would have found a clear path of escape. The 815ers as a group were not that different or brilliant. They never solved the mysteries of the Island. They never answered Charlie's basic question, "what is this place?"
Though their mother had prevented them from hurting one another, the Man in Black eventually killed Jacob indirectly by assuming the form of John Locke, the Others lost new leader, and convincing Ben to kill Jacob with a dagger, just as MIB had done to Crazy Mother. The Greek tragedy mythology had come full circle. The Island could have been this family's sideways world waiting room to the final afterlife.
But, apparently, the killing of Jacob did not allow MIB to leave the Island. He then tried to kill Jacob's Candidates for Island protector role. MIB finally tried to use Desmond's unique power to destroy the Light Cave, thinking it would set himself free himself and destroy the Island. However, the disruption of the stone "cork" in the center of the Light Cave only rendered MIB mortal. A state of being that MIB himself did not realize until he was physically injured. Before the mortal MIB could leave the Island, he died falling off a cliff as a result of a confrontation with Jack, the new Island guardian, and Kate.
It is unclear whether the destruction of Jacob and MIB were necessary in order for the remaining survivors to leave the Island, or to "move on."
Who was MIB? Jacob's twin brother.
What was MIB? An immortal representation of Jacob's dead brother.
When did MIB arrive on the Island? More than 2000 years ago.
Where did MIB come from? Jacob's brother was born from shipwrecked Roman mother, Claudia, who was killed on the Island by Crazy Mother. MIB, as the personification of Jacob's brother, was created when Jacob killed his brother by throwing him into the Light cave.
Why was MIB important to the LOST story?
The last question is really the dead end alley in the previous Jacob post. For if the Jacob-MIB story line did not exist, there were alternative "conflicts" that the 815 survivors would have had to deal with in order for their characters to develop and grow together.