There is a massive black hole in logic and story telling.
Prior to any known character coming to the island, Crazy Mother was alone as the guardian. When a Roman ship wrecks, she steals two newly born children, Jacob and his brother. Jacob's brother yearns to leave the island after he finds a survivor camp. He uses the knowledge of the Romans to try to harness the power of the island to get back "home," which would be Rome. Jacob's brother embraced technology in order to find a way to get to his dream or goal.
Crazy Mother and Jacob did not embrace technology. They lived in a primitive camp, making due on a self-sufficiency level including weaving their own materials. Crazy Mother destroyed the camp and its technology in order to stop Jacob's brother from leaving the island. At this point in time, Jacob and his brother stopped aging. She decreed that neither could harm the other. But when Jacob's brother killed her, Jacob's rage set off events that eventually killed his brother.
When Jacob's brother's body floated into the light cave, we see the smoke monster fly out. We would later find Jacob's brother's body down stream. Some speculated that the brother's death created a smoke monster, but most believe that smoke monsters existed prior to this event as Crazy Mother's actions in wiping out the Roman camp was the work of a smoke monster. Temple graphics from an apparent earlier age depicts the smoke monster seated in the ancient Egyptian underworld. The smoke monster then took the form of MIB in order to pester and engage Jacob in conversations and games. Both Jacob and MIB were immortal beings, and the "rules" did not allow each other to harm the other.
MIB said that he was growing tired of Jacob bringing humans to the island, since they eventually turn greedy and corrupt and die. It seems that Jacob and MIB were playing a human pawn game of philosophy or Senet.
So if Jacob was playing a game, and he was the only person who could allow people on the island, then why did he allow technically advanced people (like the ones Crazy Mother hated and killed) to build massive research facilities. MIB was the only person aware that there could be a technology answer to allow it to "leave" the island. Apparently, smoke monster(s) are imprisoned on the island even though they are thought to be "security systems." If MIB absorbed all the memories, emotions and logic of Jacob's brother, then MIB's purpose probably changed to match his: find a way to leave the island.
So MIB would have wanted technology advanced humans to come to the island, so it could manipulate them to find the answer that Jacob's brother was so close in finding. The question remains why would Jacob allow that? Jacob was stuck on the island for as long as he was the guardian. Perhaps that was the bargain: humans came to the island as candidates to replace Jacob while at the same time puzzle pieces for MIB to find a way to leave the island barrier.
Some believe that Jacob was himself a smoke monster. When he visited Jack at the LA hospital, the lobby smoke detector went off for no apparent reason. So if Jacob as a smoke monster could leave the island, why could not MIB?
This leads to two alternatives. First, Jacob was not a smoke monster so he could leave the island. Second, Jacob was a smoke monster and the off-island scenes we saw were illusions. That would mean that Jacob learned about mental manipulation technologies from the Dharma researchers and used it to create vivid holographic (Star Trek NG) worlds so real that humans could not tell any difference.
So it is possible that both Jacob and MIB wanted new technologies present on the island to meet their own goals. If Jacob truly wanted to release his obligations as eternal guardian, he would need to create such an environment where some human would want to become "a hero" in order to save his friends. The only way to become such a hero would be to accept the island guardianship, thereby releasing Jacob from his duties.
But that does not explain how resetting the light cave cork made MIB mortal or made an immortal being like Jacob dead. One could argue that MIB and Jacob were not real beings but holographic horror projections that seemed so real that the humans accepted them as real.
Which gets us back to the opening question. If Dharma was brought to the island for its knowledge and research, why were there other less tech people (the Others) still on the island? Perhaps the conflict between two groups was part of Jacob's plan to find his "hero." It also could be that MIB kept certain human "pawns" to play the game with Jacob. Or, another alternative would be that the Others were actually Dharma members who splintered off from the group because they did not like the research paths established by Horus. The latter conflict would have made a good side story on why the Others were so hostile to outsiders and what were the island's true powers.
For if LOST was solely about the origin story of Dharma and the island, it would have been just as interesting as the actual story lines.
