I believe the keys to understanding LOST are all contained in one episode, "Ab Aeterno," which means "from eternity" in Latin. LOST may be frosted with the mission statement that it is a character driven show about basic human behavior, good, evil, temptation, redemption, life and death. Those are eternal struggles of mankind. The underlying hidden cake, the layers and layers of mystery, is the nature and purpose of the island.
From this episode, we can glean seven golden island rules:
Rule #1: No one comes to the island without Jacob’s permission.
Rule #2: MIB is trapped on the island because Jacob won’t let him leave. MIB cannot leave if Jacob (or his successor) is alive.
Rule #3: MIB cannot leave the island even by killing Jacob because someone will take his place.
Rule #4: Jacob can grant eternal life (with his touch to Richard), but he cannot give resurrect dead people or absolve people of their sins.
Rule #5: MIB believes that it is in all human nature to be bad, to sin, to corrupt or destroy, and Jacob brings people to the island to “prove him wrong.” All prior people brought to the island to prove MIB wrong are dead.
Rule #6: The island is a gate or cap that stops evil, malevolence from spreading (to somewhere?)
Rule #7: Jacob does not intervene or interfere with people brought to the island (even though MIB can), so he appoints Richard to be his intermediary, to act on his behalf.
MIB calls this place Hell, and tells Richard that Jacob is the devil. In order to escape Hell, one must kill the devil. Richard's attempt to kill Jacob was MIB's first attempt to kill Jacob. We can infer that MIB cannot directly "kill" Jacob. He needs to act through some one else.
Jacob denies that he is the devil or this place is Hell. He nearly drowns Richard in order to prove that Richard is "still alive." Richard's fear of going to hell for eternity for his crime of murder is absolved by Jacob's touch, the grant of eternal life on the island (which is a similar "deal" he made with Dogen.)
Jacob and MIB give Richard conflicting information about themselves and the nature of the island. There is no critical reference point to judge the credibility of either Jacob or MIB.
The relationship between Jacob and MIB is telling: they act like co-workers at a postal sorting station. Jacob is the supervisor and MIB is his subordinate. MIB has grown tired of his job, and wants to leave, but Jacob won’t let him quit or go home because there is still work to be done. They have the mannerisms and inside jokes of long time colleagues. It is clear that they are not human beings. They are supernatural beings who have been on the island for an extremely long time (from the beginning of time?) The enhanced version of the episode used the word "form" to describe Richard's visions. A manifestation, a version of something or someone, a ghost or spirit. MIB freely admits that he is the smoke monster. MIB states the vision of Isabella was taken by Jacob (the Devil). He also states that Jacob is not the smoke monster who killed the Black Rock crew, but that he was. (If MIB is correct, then there appears to be two different shape shifting entities on the island. If MIB is conning Richard by deception, then it would appear that MIB can form multiple manifestations at the same time, i.e. Isabella and the smoke monster.)
It may not make any great difference in the end. Since the dramatic opening plane crash, we have led to believe that our main characters are "survivors," human beings who are thrown into a chaotic survival story, with dangers and mysteries to solve in order to get back to their true lives. We have been so caught up on what is real, believable or true, that eyes have been glazed over by the possibility that everything we have seen is not real.
The Black Rock was lost at sea in 1845.
But in Ab Aeterno, it is setting sail for the New World in 1867. Or is it, Jack Sparrow?
In 1867, the Black Rock could have been a ghost ship, a means to ferry "lost souls" to the island by Jacob.
All ships/craft come to island are lost/ghost ships. Everything is an illusion to prove a point between Jacob and MIB about human nature.
Souls that arrive on the island are meant to believe that they are still "alive." Richard sees other captive slaves being "killed" by the first mate after the ship has been impossibly tossed more than 20 stories through a carved stone statue to rest in the middle of the jungle. Jacob makes Richard "believe" that he is alive by nearly drowning him in the surf. The same is true with the Flight 815 survivors, they see a scene of carnage on the beach, and a man getting "killed" by being sucked into a crashed, but running airplane engine. Horror, fear and chaos are motivators to create strong illusions.
The episode was also cast in clues about the nature of the show.
Isabella died of TB, but Richard's quest for a cure was a mere illusion. There was no cure for TB in 1867. The medicine was a hoax and the doctor was conning Richard for his money. The evil art of the con, misdirection and manipulation are key themes to LOST.
In the enhanced subtitles, it is stated Richard never had chance to bury Isabella. His burial of her cross necklace was symbolic for her burial. One could infer that Richard never had his trial for murder, that he could have been killed by angry mob chasing him from the doctor's residence to his home. Also, the scene in the prison is suspect for reality. A priest will not absolve a sin of murder at the same time that he is committing his own immorality in selling a prisoner into slavery. (Besides, the priest would not be the one to release a prisoner set for execution). This sets into Richard's mind the false salvation as "living" as a slave.
The men in the Black Rock cargo hold miraculously "survive" a wooden ship being raised and thrown from a height of more than 20 stories onto its keel (where they were chained to the rafters). They "awaken" with the thought of themselves being alive.
Ship officer Jonas Whitfield elects to kill the "surviving prisoners," fearing that if they lived, they would turn on their captors. He was about to kill Richard when the Smoke Monster (MIB) appears and attacks the ship, killing the remaining crew but leaving Richard in chains. We are told that five days pass, when Richard sees an apparition of his dead wife, Isabella. We are told that this is not Isabella. Hearing the return of the smoke monster, Richard tells her to run, helplessly listening to her screams as she was "seemingly attacked" upon going above deck by the smoke monster. On the sixth day, Richard is "awakened" by the smoke monster who has taken the "form" of MIB. Richard has had no water or food for six (6) days. In normal circumstances, a human being could not reasonably survive 6 days without food or water in tropical conditions and if the person was not top physical condition.
The focus on the butterfly traveling through the cargo hold is another important clue. The butterfly is symbolic of souls in a journey to the afterlife. What we are seeing in the first meeting of MIB and Richard is not the last survivor of the Black Rock being rescued, but the next person to be tested by the supernatural beings in the experiment about the corrupt ability of human nature.
When MIB comes to Richard and gives him water and says he is a a friend, this is a lie. He has started to weave his arc of corruption of Richard's heart and mind. He confirms Richard's assertion that they are in Hell to instill the "fear" that he can use to "control" Richard's actions. When Richard asks about Isabella, MIB says that "he has her," implying that she was captured by the Devil (Jacob). MIB explains that Isabella was not running from the black smoke (him) but from the Devil. He adds that he saw the Devil take Isabella but couldn't stop him, which we can assume is a lie in order to gain Richard's trust. Richard begs to be freed so he can save her and MIB says he will help, as he also wants to be free, so he has a "deal" to make with Richard. He will free Richard, if Richard will murder the Devil. Richard is confused because he was sent to Hell because of murder, and another murder would not set him free; but MIB waves that off as mere word play. MIB plays to Richard's known weakness to generate the bargain: freedom in exchange for murder. Thereby, MIB proves the point that human nature is weak and corruptible.
Richard sets off to murder Jacob to gain his freedom from Hell. But Jacob disarms him in a fight, and convinces Richard that he is not the Devil, the person he saw was not Isabella, that he was not dead, and the island is not Hell. Now, Jacob weaves his own illusions upon Richard's scarred psyche:
Jacob says that he has brought everyone, including the Black Rock to the island. Jacob uses a bottle of wine as a metaphor for the island. The wine inside the bottle is evil, malevolence; the bottle (universe or realm of existence) is containing it because otherwise "it would spread" to some unknown place. Jacob states that the cork represents the island, holding the darkness where it belongs. All the people Jacob has brought to the island in the past "are all now dead." Nothing Jacob says to Richard confirms that the island is an actual place on planet Earth. It is used to manipulate Richard's mind into buying into the premise that the island is an important place.
Jacob says he wants people to know the difference between right and wrong without being told what to do. He is betting on the inherit "goodness" of humanity to will out over "evil" tendencies or base instincts of human nature. Jacob says that MIB believes the opposite: everyone can be corrupted because it is in their nature to be bad.
And here is the never ending experiment that Jacob and MIB have been conducting from the beginning of time: what is the true nature of humanity? Jacob's theory about the true nature of humanity has never come to pass because everyone that has been brought to the island has failed, by either corrupting, destroying, killing . . . which angers MIB, but which Jacob casts off as "progress." When they talk about "it only ends once," it means Jacob only needs to find one incorruptible soul to prove humanity has a chance of redemption.
Richard is given a second "deal," this one from Jacob. He will stand by his principles and will not interfere with human behavior and events on the island, but he says Richard can be his "representative" who can step in on his behalf. When Richard says that in return he wants his wife back; Jacob admits he cannot do this. Richard then asks to be absolved of his sins, so that he will not go to Hell. Jacob says he cannot do that either. It shows that Jacob is not a god over humans. He is not part of their rituals of death. He has no power to forgive mortal sins. Richard then asks to be granted eternal life, and to never die. Jacob says that he can do this, and touches Richard on the shoulder, sealing Richard's fate as island advisor forever.
It is stated that this was the first attempt by MIB to "kill" Jacob in order to leave the confines of the island. After Richard returns, MIB is aware that he failed. Richard gives him a white stone from Jacob. Later, when Jacob meets MIB he asks about the gift of the stone, MIB tersely tells Jacob not to gloat. MIB admits that he did send Richard to kill Jacob because he wants to leave the island. MIB asks Jacob to let him go. Jacob tells MIB that, "as long as he is alive, that won't happen." MIB responds that is why he will kill Jacob. Jacob responds that someone will replace him, to which MIB says that he'll "just kill them too."
One of the definitions of the word "kill" is to put an end to or cause the failure or defeat of something, such as stopping the experiment, switching off a computer or a process. I believe that the terminology of "killing" someone in the island realm as not physically killing human beings, but ending their mind-soul involvement in the Jacob-MIB human behavior study.
The term "mind-soul" represents two similar possibilities that could be used as the foundation of the story mythology. Jacob may be bringing the "minds" of people to the island, re-creating their forms and mental processes, and allow their avatars to draw on their memories to interact with others to determine their true natures. In the alternative, Jacob could be hijacking "souls" of departed humans to test his theory of human nature in a purgatory type setting. In either situation, we are not dealing with real human beings in a real human world situation. It is similar to the super-intelligent alien beings experimenting with human behavior in the Star Trek pilot, where mind manipulation made illusions reality.
If Jacob and MIB are alien scientists drawn to human research, it would make sense to use the minds or souls of the mentally ill, or criminals, the "lost souls" in society, that would not be easily missed in their own time lines, to be made captives in the island experimental realm.
And what would be the ultimate "pay-off" of this long human experiment? Something many people have dreamed about for a long time: first contact with intelligent life outside our planet, and answers to humanity's big questions.