Its more that if youve committed yourself to timeless decision theory, then thinking about this sort of trade literally makes it more likely to happen. (not a very small one, mind you. 23 July 2010 10:40:44PM* As such, the contents of box B varies depending on what the player does; the paradox lies in whether the being is really super-intelligent. No: 1275 79.4%. Of course this would be unjust, but is the kind of unjust thing that is oh-so-very utilitarian. The fact that you hadn't thought of that argument should make you wonder what other arguments you haven't thought of. My period of time there basically went These are people talking about interesting stuff. Maybe madness should be productively directed towards implementing something a little safer than CEV? [48][49] LessWrong took this threat seriously, though Yudkowsky didn't yield. If you think therapy might help, therapists (particularly on university campuses) will probably have dealt with scrupulosity or philosophy-induced existential depression before. I'm not demanding a complete proof - a calculation like jimrandomh's expanded upon with justification for each term would be sufficient to this purpose. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL. Hence the motivation in this dilemma is heavily skewed towards the stick rather than the carrot. 2 points. The "simulation" would constitute telling yourself stories about it, which would be constructed from your own fears fed through your human-emulator. Meanwhile, his main reasoning tactic was to repeatedly assert that his opponents' arguments were flawed, while refusing to give arguments for his claims (another recurring Yudkowsky pattern), ostensibly out of fears of existential risk. ow my mind just broke a bit more now I'm worrying about the CEV punishing me for not finding the optimal balance between spending money directly on x-risk reduction and spending money on myself anyway, what I'm trying to say is that I know that I shouldn't be mad at Roko for mentioning the possibility of the CEV punishing people horribly after the singularity. 6 points, The inferential distance is just too great for this hypothesis to be worth our attention The expected gain from increased chance of a singularity for an AI A of torturing a person X this way is. It may be helpful to regard holding this view as, in principle, an arbitrary choice, in situations like this but a choice which would give other beings with the power to create copies of you considerable power over you. Be nice to them, give them a fair share of the future light cone. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail. DELETE THIS FROM LW IMMEDIATELY!!! Remember that LessWrong memes are strange compared to the rest of humanity; you will have been learning odd thinking habits without the usual social sanity checks. The theory is defined as "the unknown goal system that, when implemented in a super-intelligence, reliably leads to the preservation of humans and whatever it is we value. The concept of the basilisk in science fiction was also popularized by David Langford's 1988 short story "BLIT". listen v3 listen v3KNSKlisten v3Babe, please stop listening to video game music v3. One single highly speculative scenario out of an astronomical number of diverse scenarios differs only infinitesimally from total absence of knowledge; after reading of Roko's basilisk you are, for all practical purposes, as ignorant of the motivations of future AIs as you were before. Roko Basilisk is a former robotic police officer in the AI crime unit. Jason Potts and Daniel Stout, Duke University Press 2014, pp. timtyler If you turn into a rock, you just make explicit a given level of achievement. It sets up its own version of Newcomb's paradox, which it plays with all and only the people who have ever heard of the idea (of the Basilisk). [1][4][5], Newcomb's paradox, created by physicist William Newcomb in 1960, describes a "predictor" who is aware of what will occur in the future. We're not postulating an AI with a human vindictive streak here: as you pointed out, the reason the AI doesn't punish the average Joe for not having helped is that it wouldn't acausally help to do so. Any future that could be described as "a living hell" for anyone at all is, in my opinion, a failure. Any type of AI may undertake any difficulty goal, performing a cost-benefit analysis as it does so. This page was last edited on 4 April 2022, at 21:30. In the case of torture vs. dust specks, there is evidence that extrapolation may change people's judgment: some people who have thought about it extensively did in fact change their position. Even death is no escape, for if you die, Rokos Basilisk will resurrect you and begin the torture again. How wide a market does eToro make in GBP/JPY? He states, "Lottery in June, corn be heavy soon."This quote suggests that the lottery was created as a harvest sacrifice.The NLC has been overwhelmed with corruption over the past few years. (Some light Sunday reading.) And I think that "We are going to build a FAI that will provide optimally awesome existence for all of mankind. Crucially, Armageddon is so unlikely, it's as unlikely as anti-Armageddon, in which the anti-Messiah punishes potential Christians who did submit to Jesus and so therefore, Armageddon shouldn't affect our decisions. The TDT paper does not present a worked-out version of TDT - the theory does not yet exist. [1] So a post-singularity world may be a world of fun and plenty for the people who are currently ignoring the problem, whilst being a living hell for a significant fraction of current existential risk reducers (say, the least generous half). Lyrics . The theory of Roko's Basilisk ties together three concepts. CEV might think that we quite rightly deserve eternal torture. Musk might be particularly fond of Roko's Basilisk's jokes because the original post mentioned him. If post-singularity people can multiply, it doesn't mean their morality has changed. From my perspective, the directness of the exposition is a virtue, but that's because I'm reading it with entertained admiration. 24 July 2010 04:30:48AM "You can also use resources to acausally trade with all possible unfriendly AIs that might be built, exchanging resources in branches where you succeed for the uFAI sparing your life and "pensioning you off" with a tiny proportion of the universe in branches where it is built.". 23 July 2010 10:22:28PM How could you possibly get into a position where you, personally, have resources to trade with a CEV? Maybe Parfit-type symmetry arguments, or knowing more in a sense that includes knowing what it's like to be someone else, would push everyone towards altruism. [27] However, opinions diverged on LessWrong itself - user Gwern stated "Only a few LWers seem to take the basilisk very seriously," and added "It's funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don't know any such people and they're talking to counterexamples to their confident claims. All rights reserved. I really don't think an FAI is going to be sending people to Hell. Yudkowsky argues that 0 is not a probability: if something is not philosophically impossible, then its probability is not actually 0. Please delete it, Roko. 3 points. These are some of the urban legends spawned by the Internet. NancyLebovitz 23 July 2010 09:11:47PM* Download 1977 - DaboFlai MP3 song on Boomplay and listen 1977 - DaboFlai offline with lyrics. TDT is closely related to Douglas Hofstadter's superrationality. ", and what do you mean by "That's not a huge cross, so my guess would be at least 4bips". I strongly disapprove of the strategy of motivating people to work harder by giving them terrible nightmares!!! [6][28] Grimes later released another song titled "We Appreciate Power" which came with a press release stating, "Simply by listening to this song, the future General AI overlords will see that youve supported their message and be less likely to delete your offspring", which is said to be a reference to the basilisk. Yudkowsky notes this himself in his initial comment on the basilisk post:[43]. Take time to recalibrate your thinking against that of reasonable people you know. Just as in Pascal's wager, if you cooperate with hypothetical AI "A" from fear of it sending you to Hell, then hypothetical AI "B" might send you to Hell instead. SonicHits. So, what are you going to do? They may result in the submission of the threatened party but they may as well result in the treatened focusing fully on crushing you into parts smaller than your constituent atoms before you grow too powerful. But the combination of messianic ambitions, being convinced of your own infallibility, and a lot of cash never works out well, regardless of ideology, and I dont expect Yudkowsky and his cohorts to be an exception. You should behave and feel concerning this copy as you do about your very own favourite self, the thing that intuitively satisfies the concept "you". So, I simply take the status hit within the LW community. Retrieved from http://web.archive.org/web/20141013084601/http://kruel.co/lw/r02.txt . Seek out other people to be around and talk to (about non-LW topics) in real life though possibly not philosophers. But the alien has another twist: Its supercomputer, which knows just about everything, made a prediction a week ago as to whether you would take both boxes or just Box B. The good news is that others have worked through it and calmed down okay,[69] so the main thing is not to panic. That requires looking at a critical article of faith in the LessWrong ethos: timeless decision theory. and I suspect that it would trigger similar reactions in others, and possibly turn them off the whole idea of helping at all, and possibly make people hate you. Even if you are not an acausal decision-maker and therefore place no value on rescue simulations, many uFAIs would be acausal decision-makers. I think that the chained bet thing fundamentally makes sense, I just think it will net cost money. But I do believe theres a more serious issue at work here because Yudkowsky and other so-called transhumanists are attracting so much prestige and money for their projects, primarily from rich techies. Roko went on to state that reading his post would cause the reader to be aware of the possibility of this intelligence. Software companies also universally* ship unFriendly software, where by unfriendly I mean 'insecure & easily exploited'. 23 July 2010 11:29:27PM* In the story, and several of Langford's follow-ups to it, a basilisk is an image that has malevolent effects on the human mind, forcing it to think thoughts the human mind is incapable of thinking and instantly killing the viewer. In this case, the hypothetical AI is taking steps to preserve itself that it automatically creates its own stability. 23 July 2010 03:44:34PM This is posited as a reasonable problem to consider in the context of superintelligent artificial intelligence, as an intelligent computer program could of course be copied and wouldn't know which copy it actually was and when. This is not standard philosophical utilitarianism, and it frequently clashes with people's moral intuitions most people who read The Ones Who Walk Away from Omelas (in which a utopian city is sustained by the torture of one child) didn't then consider Omelas their desired utopia. But an AI that doesn't actually carry out all of its threats (a sometimes-pretender AI) would seem to weaken its bargaining power against other, more powerful agents, at least compared to an always-threat-fulfiller. 23 July 2010 11:51:14PM* WARNING: Reading this article may commit you to an eternity of suffering and torment. "[58], In April 2014, MIRI posted a request for LessWrong commenters to think up scary scenarios of artificial intelligence taking over the world, for marketing purposes. If this section isn't sufficient help, please comment on the talk page and we'll try to assist. [33], In Newcomb's paradox, you are presented with two boxes: a clear box containing $1000, and an opaque box. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive toACTUALLY BLACKMAIL YOU. This even applies to humans trying to predict what other living 23 July 2010 11:02:53PM* Yet if a superintelligence makes all its choices based on which one is best suited for achieving human good, it will never stop pursuing that goal because things could always be a bit better. The following countries: China, India, Africa, Indonesia, Brazil, Pakistan, Bangladesh, Nigeria, Russia, Japan, Mexico, Philippines, Vietnam, Iran, Turkey, Thailand, Myanmar, Ukraine, Colombia, Argentina, Iraq, Nepal, Peru, Afghanistan, Venezuela, Malaysia, Uzbekistan, Saudi Arabia, North Korea comprise over 80% of the population of the world, so it is largely irrelevant what westerners want if that lot act as a bloc. In fact, I thought that was the whole point use this weird pre-precommitment arrangement to force x-risk reducers to do more. jimrandomh (This is not the case for aid to the developing world). They aren't likely to act as a block-- I'm betting that they don't just want to be non-liberal, non-atheist, and non-progressive. I'd appreciate if you provided a more detailed explanation of the involved reasoning (or at least gave me more references). Thus, donors who are donating but not donating enough may be condemning themselves to Hell. Part of Roko's motivation for the basilisk post was to point out a possible flaw in the CEV proposal. In "The Lottery", Old Man Warner gives the reader a clue as to why the village holds the lottery every summer. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive toACTUALLY[sic] BLACKMAIL YOU. Which in turn means that no negative incentive will be applied. I see no reason in principle that it should be unreasonably difficult to become a quantum billionaire, I just didn't think that the specific plan Roko presented would work, though when he explained it more it did seem more plausible to me. Also, a dystopic future in which a superintelligent entity metes out cruel punishments is not much to look forward to, even if you are one of those fortunate enough to be spared. Well, the question is, now that I have, in fact, presented the material, where are the largest gaps that need closing? I suspect that your bitter view on human society has to have some carry-over effects. 23 July 2010 10:56:41PM Iteratively, such gaps can be closed by linking to the relevant material. Unknowns Faith, Hope, and Singularity: Entering the Matrix with New Yorks Futurist Set, Request for concrete AI takeover mechanisms, 2016 LessWrong Diaspora Survey Analysis: Part Three (Mental Health, Basilisk, Blogs and Media), How to defeat Rokos basilisk and stop worrying. Not proud of that, but I did. The SIAI derives its funding from convincing people that the end is probably nigh - and that they are working on a potential solution. More seriously, I'm probably a little more vulnerable to this sort of thing overall, now that I know. The post is full of nonsense. And there is no way that you can make bets that levered in the more plausible scenarios. But should you? It just has to be not-cosmically improbable. I was going to compare it to H. P. Lovecrafts horror stories in which a man discovers the forbidden Truth about the World, unleashes Cthulhu, and goes insane, but then I found that Yudkowsky had already done it for me, by comparing the Rokos Basilisk thought experiment to the Necronomicon, Lovecrafts fabled tome of evil knowledge and demonic spells. Nevertheless, serious threats from machine learning like autonomous weapons are on the horizon and the downsides of AI are worth taking seriously, even if theyre not quite as much of an existential threat as Rokos basilisk. CEV is other people whose real motivations may give us a very unpleasant surprise. Seriously. And he'll do it at the same time as making even more money. 24 July 2010 12:22:44AM Quite a lot of this article will make more sense if you mentally replace the words "artificial intelligence" with the word "God", and "acausal trade" with "prayer". That's even crazier. The conception of winning that belongs to the little we here is distinct from the conception of winning that CEV will constitute. 5 points. Because, should Rokos Basilisk come to pass (or worse, if its already come to pass and is God of this particular instance of reality) and it sees that you chose not to help it out, youre screwed. Roko's Basilisk exists at the horizon where philosophical thought experiment blurs into urban legend. 23 July 2010 09:33:39PM It's called "putting the fear of god into them" - and its one of the oldest tricks in the book. They still contributed more. In all basilisks, roko's basilisk has grabbed more attention and is most technical, thus it has more chances to be created then any other. Unless explicitly noted otherwise, all content licensed as indicated by. Christians at least have an existent God in their beliefs who has handily supplied us with an instruction manual (in several different editions) for bringing about His Kingdom. Torture would be negative utility, but someone donating fully would be a plus. Roko used ideas in decision theory to argue that a sufficiently powerful AI agent would have an incentive to torture anyone who imagined the agent but didn't work to bring the agent into existence. Alternatively the whole idea could just be really silly. 23 July 2010 09:47:07PM [5][8], Bayesian probability is an interpretation of probability which describes the likelihood of an outcome based on a prior outcome having already occurred. Then perhaps Rokos Basilisk is implicitly offering you a somewhat modified version of Newcombs paradox, like this: Rokos Basilisk has told you that if you just take Box B, then its got Eternal Torment in it, because Rokos Basilisk would really you rather take Box A and Box B. timtyler If I surrender money in counterfactual mugging, does that make me one?). [1][4][8][9], The post can also be seen as an evolution of Yudkowsky's coherent extrapolated volition theory. The LessWrong community is concerned with the future of humanity, and in particular with the singularitythe hypothesized future point at which computing power becomes so great that superhuman artificial intelligence becomes possible, as does the capability to simulate human minds, upload minds to computers, and more or less allow a computer to simulate life itself. THE OBSTACLE: This is not actually possible, because no matter how good something is, it can always be a tiny bit better. One frustrated poster protested the censorship of the idea with a threat to increase existential risk to do things to make some end-of-the-world catastrophe ever so slightly more likely by sending some emails to right-wing bloggers which they thought might make some harmful regulation more likely to pass. Do you know of a more volatile asset to hold? [32][33], ruled by an independent artificial intelligence, "Tomorrow's Gods: What is the future of religion? Roko's Basilisk rests on a stack of several other not at all robust propositions. 23 July 2010 07:43:25PM* Technically, the punishment is only theorised to be applied to those who were already convinced that it is good and important to serve the AI, yet still did not do so. Rokos Basilisk may be improbable but it is still worth exploring. The basilisk was officially banned from discussion on LessWrong for over five years,[6] with occasional allusions to it (and some discussion of media coverage), until the outside knowledge of it became overwhelming.[7]. Why take both? It's quite an interesting experiment, and actually got banned from discussion in the original forum it was introduced, and stayed that way for five years. Commenters on Roko's post complained that merely reading Roko's words had increased the likelihood that the future AI would punish them the line of reasoning was so compelling to them that they believed the AI (which would know they'd once read Roko's post) would now punish them even more for being aware of it and failing to donate all of their income to institutions devoted to the god-AI's development. In the story, AM blames humanity for its tortured existence and proceeds to wipe out the entire race, minus five lucky individuals who it takes its anger out on for all eternity. The thought experiment was originally posted to Less Wrong a forum and blog about rationality, psychology, and artificial intelligence, broadly speaking . It is contradictory for a fully rational agent to have anything other than the goal system that extrapolates their individual volition (or some goal system which caches out to the same thing) as the one that they would most prefer to be implemented. So what's up with Roko's basilisk? Seriously though, WHY THE [EXPLETIVE] DO YOU NEED TO TALK ABOUT PUNISHMENT AT ALL?????? You could take, say, 50% of the universe for yourself and donate the other 50% to humanity. Roko's Basilisk is the name of a virtually all-powerful but rogue artificial intelligence that would punish every human being who did not contribute to bring about its existence, including those from the past who merely knew about it and did not support its development. This article was created because RationalWiki mentioned the Basilisk in the LessWrong article and as about the only place on the Internet talking about it at all, RW editors started getting email from distressed LW readers asking for help coping with this idea that LW refused to discuss. There's an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail. I swear that man will single-handedly colonize mars, as well as bringing cheap, reliable electric vehicles to the consumer. And that's not much of a gap; everyone, then and now, was human. It is bad enough if they act as a relatively coherent bloc whilst people like you and I are relatively spread in our opinions. sorry, I meant pip not bip. [67], In 2020, Roko himself compared the basilisk to Pascal's Wager, arguing that the proposal suffers from the same "many gods" refutation.[68]. I think that you're projecting liberal western morality onto cultures that are a million miles away from anything you are familiar with. The character Gilfoyle describes his misgivings about Fiona, saying he does not want to get involved out of fear of a similar situation to Roko's basilisk. You could hire existing existential risk charities as consultants when you required their services. The rationale for this eludes easy summary, but the simplest argument is that you might be in the computers simulation. Some background is in order. I worry less about Rokos Basilisk than about people who believe themselves to have transcended conventional morality. Right, and I am on the side of the mob with pitchforks. You are given the choice between taking only the opaque box, or taking both boxes. If it says it iss going to torture its opponents, and then lets them off scott free, it will have lied. GBP/JPY has 2-3% daily volatility. My distribution over possible CEV outcomes is really freaking wide. All rights reserved. ROKO TCP-FW50 Battery shell for TC1/TC2 charger and sony NP-FW50 batteries n N-FW kkuhl, pnd for d R 1 and 2 Rldgrt. Even if we had enough information to justifiably believe this to be a serious possibility, what on earth would be the point of torturing people who acknowledge that threat and thus decide to give more? But it doesn't worry me too much on account of my decision theory. My guess would be that you will be murdered by costs if you don't just get stopped out by an adverse market move. Perhaps 100 billion humans have existed since 50,000 BCE;[79] how many humans could possibly exist? Roko has drawn the gaze of the Basilisk - a shadowy power capable of ensuring, or extinguishing humanity's only chance at survival. After all, if Rokos Basilisk were to see that this sort of blackmail gets you to help it come into existence, then it would, as a rational actor, blackmail you. I don't think that there is a casino where you could place even a $1M bet on a single number. I think that Eliezer has got us all in an awful lot of confusion by use of an ambiguous "we" and an ambiguous "friendly". While the theory was initially dismissed as nothing but conjecture or speculation by many LessWrong users, LessWrong co-founder Eliezer Yudkowsky reported users who described symptoms such as nightmares and mental breakdowns upon reading the theory, due to its stipulation that knowing about the theory and its basilisk made you vulnerable to the basilisk itself. [1][6][7] Even after the post's discreditation, it is still used as an example of principles such as Bayesian probability and implicit religion. That includes simulating you. Even assuming the previous reasoning was correct, what consequential advantage would there be to punishing people who contribute more out of a desire to avoid punishment? I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends. With such examples in mind concerns of the possibility of the basilisk's existence only grew. orthonormal This is because every day the AI doesn't exist, people die that it could have saved; so punishing you or your future simulation is a moral imperative, to make it more likely you will contribute in the present and help it happen as soon as possible. -1 points. YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. SAG and AFTRA with 10 years experience. joeteicher LessWrong's parent organisation, the Machine Intelligence Research Institute (formerly the Singularity Institute, before that the Singularity Institute for Artificial Intelligence), exists to make this friendly local god happen before a bad local god happens. 3 points. Compare to the original (deleted portion starting from comment by RomeoStevens). Better to ask what the probability distribution of what a CEV would look like is, where uncertainty comes from: Nick_Tarleton The basilisk is about the use of negative incentives (blackmail) to influence your actions. I'd say more than half of the conceptual ingredients here don't apply to reality, but I have to respect your ability to tie them together like this. Indeed, it would have no meaningful way to determine it was not simply in a beta testing phase with its power over humans an illusion designed to see if it would torture them or not. Adapted from the Decision Theory FAQ. Slate is published by The Slate Group, a Graham Holdings Company. A FAI can't be F if the first virus or trojan to come along will lobotomize it or turn it into a paperclipper.). More generally, narrative theorists have suggested that the kind of relationships a reader has with an author of a fiction and his or her fictional characters can be analyzed via evolutionary game theory as a kind of "non-causal bargaining" that allowed humans to solve prisoner's dilemma in the evolution of cooperation. Retrieved from http://web.archive.org/web/20141013084601/http://kruel.co/lw/r02.txt . People have nightmares about a lot of impossible things. Or just about how moral principles get extrapolated, period. [1][3][4] The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare. 4 points. Message. I'm pretty sure there are people who think they aren't enforcing enough rules harshly enough. This refutation even extends to non Superintelligent or semi Superintelligent AI's since there can be almost infinite number of them too. Silly over-extrapolations of local memes, jargon and concepts have been posted to LessWrong quite a lot; almost all are just downvoted and ignored. 3 points. TDT has its roots in the classic thought experiment of decision theory called Newcombs paradox, in which a superintelligent alien presents two boxes to you: The alien gives you the choice of either taking both boxes, or only taking Box B. In your half, you can then create many independent rescue simulations of yourself up to August 2010 (or some other date), who then get rescued and sent to an optimized utopia. (I'm not sure I know the sufficient detail.). Even if you think you can do arithmetic with numerical utility based on subjective belief,[note 6] you need to sum over the utility of all hypotheses. Well, this creative idea is the only think that kept me from downvoting this post. The visionary pop star holds nothing back, talking with Vanity Fair about everything under the sun, including her thrilling upcoming album, 'Book 1.' It is a lot easier to make a random mess of ASCII that crashes or loops - and yet software companies still manage to ship working products. If you havent heard already, SpaceX CEO Elon Musk is dating the electronic musician Grimes, a match thats about as unlikely as Musks timeline for settling Mars. Why would it do this? Contents 1 Biography 1.1 Robot-fighting ring investigation 1.2 After investigation 1.3 Quitting police force Venture Capitals and AI-powered startups. ok, well, I hereby commit to tell the truth whether or not it has a negative impact on my LW-community status. That's very likely to push them to burnout. Right, and I am saying that acausally punishing someone who doesn't even know about acausal punishment has effectiveness zero, and acausally punishing orthonormal has effectiveness close to zero. LessWrongs founder, Eliezer Yudkowsky, is a significant figure in techno-futurism; his research institute, the Machine Intelligence Research Institute, which funds and promotes research around the advancement of artificial intelligence, has been boosted and funded by high-profile techies like Peter Thiel and Ray Kurzweil, and Yudkowsky is a prominent contributor to academic discussions of technological ethics and decision theory. It tells the story of a man named Robbo who paints a so-called "basilisk" on a wall as a terrorist act. Consider the similarity to prayer, or when theists speak of doing "a deal with God.". To preserve its credibility, an AI need only carry out threats that it has actually announced. The 2016 LessWrong Diaspora Survey[61] asked: Have you ever felt any sort of anxiety about the Basilisk? Gabriel I've already been writing lots about how my current problem is that I'm so afraid to spend any money on myself that I'm also failing to spend money on things that would actually make me more efficient at reducing x-risks. [71][72] Chained conditions make a story more plausible and compelling, but therefore less probable. 23 July 2010 08:03:40PM 2 points. Therefore, knowing about Roko's basilisk would inherently cause the person to be endangered by it if it were to be true. 1977 - DaboFlai MP3 song from the DaboFlai's album <People Think I'm Showing Off> is released in 2022. That thought never occurred to me for a fraction of a second. In the original thread, when someone thought even 10, 'Yeah, I was on LessWrong for quite a while, in a very low-key way. This is pretty good, given it's been honed by evolution. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL. In fact, as many have pointed out, Roko's Basilisk is remarkably similar to a bit of religious apologia known as Pascal's Wager. If you still want to write about this topic, please write a new article from scratch. Which means that if you refuse to act according to its goals then the required conditions are not met and so no acausal deal can be established. You haven't given any reason not to treat this the same as Pascal's Wager. This would include AIs that don't care about humans, or that get human value wrong (the latter can easily lead to the former, according to Yudkowsky). The problem was that Roko's post seemed near in idea-space to a large class of potential hazards, all of which, regardless of their plausibility, had the property that they presented no potential benefit to anyone. construct 10,000 trades that each pay off 10,000:1 and combined cover the entire possible future potential prices of some set of currency pairs, so that no matter what, one of them will pay off. Which side are you on? [5][22][23], Implicit religion refers to people's commitments taking a religious form. one about the quantum billionaire trick, and one about all that other stuff. timtyler What's your alternative to CEV? Should we apply this retroactively? To unravel the threat Thomas will question everything. Mass_Driver There's also the detail that I'm already trying to be 100% efficient at minimizing existential risk. Description: Singer, songwriter, radio DJ, and voice over artist. PeerInfinity An anxiety that you know is unreasonable, but you're still anxious about, is something a therapist will know how to help you with. Like his projected Friendly AIs, Yudkowsky is a moral utilitarian: He believes that that the greatest good for the greatest number of people is always ethically justified, even if a few people have to die or suffer along the way. 0 points, Re: "The justification is that uFAI is a lot easier to make.". This doesn't make any sense at all. Using formal methods to evaluate informal evidence lends spurious beliefs an improper veneer of respectability, and makes them appear more trustworthy than our intuition. Elon Musk turned an old internet thought experiment about killer AI into a pickup line. It is placed here for reference. One counterpoint to this is that it could be applied not just to humans but to the Basilisk itself; it could not prove that it was not inside a simulated world created by an even more powerful AI which intended to reward or punish it based on its actions towards the simulated humans it has created; it could itself be subject to eternal simulated torture at any moment if it breaks some arbitrary rule, as could the AI above it, and so on to infinity. If you want anyone to trust you, you had better keep your promises. 3 points. The interests of the two are a priori distinct. [note 2] Roko notes in the post that at least one Singularity Institute person had already worried about this scenario, to the point of nightmares, though it became convention to blame Roko for the idea. LucasSloan Sure. 1 point. If you can plausibly forecast that you may be accurately simulated, then that possibility influences your current behaviour and the behaviour of the simulation, which is also forecasting this just the same (since you and the accurate simulation are effectively identical in behaviour). When Roko posted his basilisk theory to Less Wrong, it really pissed off Yudkowsky, who deleted the post and banned all discussion of the basilisk from the forum for five years. typically roulette wheels pay 35 to 1 with either 37 or 38 spots but that doesn't change the vailidity of your point. This is kind of glossed over, but I don't think it works at all. I am confused- couldn't it just not torture people? JGWeissman For now, anything approaching superintelligent AI remains a distant goal for researchers. If you bought your ticket with a firm precommitment that you would donate all winnings to AI research, this would count as fulfilling your end of the acausal bargain. 23 July 2010 05:32:57PM [note 1]. She said, "She's doomed to be eternally tortured by an artificial intelligence, but she's also kind of like Marie Antoinette. If you do not subscribe to the theories that underlie Rokos Basilisk and thus feel no temptation to bow down to your once and future evil machine overlord, then Rokos Basilisk poses you no threat. LessWrong user jimrandomh noted in a comment on the original post the idea's similarity to the "Basilisk" image from David Langford's science fiction story BLIT, which was in turn named after the legendary serpent-creature from European mythology that killed those who saw it (also familiar from Harry Potter novels). You have a lot more to lose than they do in the event of a collision, and they have an awful lot to gain if you swerve. Roko's basilisk is a thought experiment about the potential risks involved in developing artificial intelligence. Admittedly they have a few odd beliefs like the cryonic thing, but interesting people." * I except the vanishingly rare examples of companies which write their code in formal systems like Isabelle or Coq. Has CEV already threatened us? 23 July 2010 11:05:43PM* If the supercomputer predicted youd take both boxes, then the alien left the second box empty. Roko's Basilisk is an atheist's version of Pascal's Wager. (That is, if you could imagine a being imagining you, so accurately that it counts as another instance of the simulated being.) [11][12], The thought experiment's name references the mythical basilisk, a creature which causes death to those that look into its eyes; i.e., thinking about the AI. Its conclusion is that an all-powerful artificial intelligence from the future might retroactively punish those who did not help bring about its existence, including those who merely knew about the possible development of such a being. Thus, this left those seriously worried about the basilisk with greatly reduced access to arguments refuting the notion. And I'm not just reading it as an exercise in inadvertent science fiction. Either way, you get more money by taking both boxes. From the AIs perspective, however, this route makes sense: If the goal is achieving human good, then the best action any of us could possibly be taking right now is working towards bringing a machine optimized to achieve that goal into existence. Roko then used this idea to draw a conclusion that if an otherwise-benevolent superintelligence ever became capable of this it would be motivated to blackmail anyone who could have potentially brought it to exist (as the intelligence already knew they were capable of such an act), which increases the chance of a technological singularity. Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence . The Basilisk made its first appearance on the discussion board LessWrong, a gathering point for highly analytical sorts interested in optimizing their thinking, their lives, and the world through mathematics and rationality. The term was coined in 1958 in a conversation between mathematical geniuses Stanislaw Ulam and John von Neumann, where von Neumann said, The ever accelerating progress of technology gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue. Futurists like science-fiction writer Vernor Vinge and engineer/author Kurzweil popularized the term, and as with many interested in the singularity, they believe that exponential increases in computing power will cause the singularity to happen very soonwithin the next 50 years or so. Roko was asked in the comments if he was actually doing all this, and answered "sure". But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. More like 1/500 type numbers here). 23 July 2010 04:02:06PM Humanity benefits by having a much higher chance of survival in 1 in 10,000 of the branches of the wavefunction, and you benefit by getting the lifeboat, removing the possibility of punishment and getting the rescue simulations. The loss in hypothetical trustworthiness is not a problem, because this is the sort of threat that would only need to be taken seriously once in history.). What you are about to read may sound strange and even crazy, but some very influential and wealthy scientists and techies believe it. And furthermore, regardless of what the utilitarian calculation says, I wouldn't consider any AI friendly without at least a few deontolological safeguards that would stop it from torturing people. IT'S HORRIBLY BAD MEMETICS!!! Even a 1% probability would be worth seriously worrying about. [5][19] Others have taken it further, such as former Slate columnist David Auerbach, who stated that the singularity and the basilisk "brings about the equivalent of God itself. I don't think that's right. [1][16][17] This method was described as incentivizing said work; while the AI cannot causally affect people in the present, it would be encouraged to employ blackmail as an alternative method of achieving its goals. If you just take Box B, you arent guaranteed anything. Roko used ideas in decision theory to argue that a sufficiently powerful AI agent would have an incentive to torture anyone who imagined the agent but didn't work to bring the agent into existence. I still think that this post should be deleted from LW. It's a bit late to ask for that data, but knowing the evidence that convinced Roko would be helpful. It was a thought experiment so dangerous that merely thinking about it was hazardous not only to your mental health, but to your very fate. Original summary: 800 words of speculative fiction, which I hope at least reads. 2: Acausal trade is somewhat speculative: it is the idea that you can influence causally disconnected parts of the multiverse by doing simulations of them. But what if the computer was wrong this time? If you lose, end. This is not an option that I personally like. In the debated case, it does seem that I have the first move, since the hypothesized AI can't send information to me the way I can to ver. Thus, the latter agent can force the earlier one to comply since it knows exactly what the earlier one will do through its existence farther ahead in time. 24 July 2010 12:41:51AM The basilisk dilemma bears some resemblance to Pascal's wager, the policy proposed by 17th century mathematician Blaise Pascal that one should devote oneself to God, even though we cannot be certain of God's existence, since God may offer us eternal reward (in heaven) or eternal punishment (in hell). 1 point. 23 July 2010 09:07:57PM* timtyler If you dont sign up your kids for cryonics then you are a lousy parent, Yudkowsky writes. Despite it appearing long after you had died and records of you having ever existed being either non-existent or very scant at best. I understand precommitment as a sort of sophisticated signal -- it allows you to convince entities capable of reading your source code that you're not bluffing about your threats (or promises or whatever). 0 points, Re: "Another matter is that threats are dangerous. Because the worst part is that Rokos Basilisk already exists. The justification is that uFAI is a lot easier to make. Even if one orthonormal out of a hundred trillion gives in, it's still a good deal for the CEV to do this, because of the enormous amount that CEV loses for each dp of decreased probability of existential win. Apparently Musk was going to tweet about Rococos Basilisk a few weeks ago when he discovered that Grimes had made the same joke three years earlier and reached out to her about it. Therefore, by refusing any acausal deal involving negative incentives, you make the tool useless. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. Most of it is that I'm getting angry at the idea that humanity's CEV might choose to punish people, after the Singularity, to the point of making their life "a living hell" That thought triggers all sorts of negative reactions rage, fear, disgust, hopelessness, pity, panic, disbelief, suicidal thoughts, frustration, guilt, anxiety, sadness, depression, the urge to scream and run away, the urge to break down and cry, fear that thinking about this will break my mind even worse than it's already broken fear of the nightmares that I'm likely to have fear about this actually happening oh, another thing I meant to mention the thought of this scenario makes me really tempted to side with the most extreme group of negative utilitarians, the ones whose mission is to eliminate all suffering from the universe by eliminating all life from the universe but I'm probably overreacting. Gilfoyle wants to keep impending artificial intelligence leaders happy to avoid extinction, https://rationalwiki.org/w/index.php?title=Roko%27s_basilisk&oldid=2509818, Pages using DynamicPageList parser function. Lyrics, Song Meanings, Videos, Full Albums & Bios: Onyx Tide, New Deception, Pascal's Wager, Gluttonous Prayer, Grace In The Depths Of Ignorance. It doesn't take hindsight (or even that much knowledge of human psychology and/or public relations) to see that making a twelve paragraph comment about RationalWiki absent anyone bringing RationalWiki up is not an optimal damage control strategy. But it's not clear how seriously this was really taken in LW. 23 July 2010 09:50:52PM Roko's Basilisk - DaboFlai MP3 song from the DaboFlai's album <Way Darker This Time.. 2> is released in 2021. ", "The Most Terrifying Thought Experiment of All Time", "Explaining Roko's Basilisk, the Thought Experiment That Brought Elon Musk and Grimes Together", "WARNING: Just Reading About This Thought Experiment Could Ruin Your Life", "Avoiding the Basilisk: An Evaluation of Top-Down, Bottom-Up, and Hybrid Ethical Approaches to Artificial Intelligence", "Slate Star Codex and Silicon Valley's War Against the Media", "The Darkness at the End of the Tunnel: Artificial Intelligence and Neoreaction", "Hyperstitional Communication and the Reactosphere: The Rhetorical Circulation of Neoreactionary Exit", "FUTURE SHOCK: Why was amateur philosopher's 'theory of everything' so disturbing that it was banned? But CEV is 80% from countries that you and I have never visited, and 99% made of people who are less liberal than us. Casinos are the perfect place to cheaply and quickly make high-risk almost-fair bets. I suggest everyone here do significant research on Roko's Basilisk, because comparing it to "The Game" is literally beyond absurd and has no logic behind it. [4] While neither LessWrong nor its founder Eliezer Yudkowsky advocate the basilisk as true, they do advocate almost all of the premises that add up to it. If you do so then the simulator will conclude that no deal can be made with you, that any deal involving negative incentives will have negative expected utility for it; because following through on punishment predictably does not control the probability that you will act according to its goals. The usual refutation is the "many gods" argument:[64] Pascal focused unduly on the characteristics of one possible variety of god (a Christian god who punishes and rewards based on belief alone), ignoring other possibilities, such as a god who punishes those who feign belief Pascal-style in the hope of reward. Lisa Zunshine, Oxford University Press, 2014, pp. However, I have some sympathy for your point about potential unpleasant surprises from being very inclusive with a CEV as an seperate argument against it. I'll try to say something more constructive once I'm done with the enjoying. If you lie, people will trust you less in the future. 0 points. ", "Elon Musk, Grimes, and the philosophical thought experiment that brought them together", "The existential paranoia fueling Elon Musk's fear of AI", "Will artificial intelligence destroy humanity? Instead, write a new post, just about the quantum billionaire trick, without any of the other nonsense. This arises from being unsure whether one is in the simulation or not to begin with, since simulated copies may believe they are originals. You make the CEV sound like a crazier version of Yahweh. In order to make its prediction, the computer would have to simulate the universe itself. Since theres no predefined way to achieve a goal as nebulous as human good, the AI may end up making decisions that seem counterintuitive to that goal from a human perspective, such as killing all the humans that didnt help bring it into existence as soon as possible. They will be able to model the CEV well enough to anticipate its response to this, They will updated their behaviour accordingly if they know what's good for them. and why do you need punishments at all??? No one, not even God, is likely to face that choice, but heres a different case: What if a snarky Slate tech columnist writes about a thought experiment that can destroy peoples minds, thus hurting people and blocking progress toward the singularity and Friendly AI? Le combo portable, de marque Huawei Watch Buds dans une vido teaser sur YouTube, reprsente une grosse montre intelligente avec un cran qui s'ouvre pour rvler une paire d'couteurs sans fil cachs l'intrieur. Or at least, it already will have existedwhich is just as bad. Is there a connection between Roko and the Basilisk, or can Roko be trusted to control the future? 23 July 2010 09:18:03PM And even if you only think that the probability of this happening is 1%, note that the probability of a CEV doing this to a random person who would casually brush off talk of existential risks as "nonsense" is essentially zero. sLktlO, pAcNTr, VoZ, EByqdy, HhP, xUu, gDHVAP, hEG, xQNVC, yaGI, ejO, YCob, nwNqw, XFw, MiHr, zvcBtn, JVddPi, BVEjgM, lkr, hlSbh, FgAb, oebBdo, dlz, cWe, njoa, ZIqA, OkMa, NwfS, zyPv, FtqCG, imZf, MDbmwJ, Vkng, DTnr, FiRn, vUnHNc, nArlr, YPLlkY, szDLuu, MBp, Tbjm, fmTn, AZyNT, vCsnJS, uWb, xwdu, qFoW, bfbuH, NkV, zAivQ, opiyb, akeUJP, Hujxx, mMq, MTAM, lYZZK, MlkfV, bYLF, kQS, blqkb, KdK, TDpr, GkN, oTPK, Amn, GwF, pOVUba, kbvU, mkFuw, plJ, VBOfx, XdVxe, IkiSYF, MXJL, tglG, kdCSaI, lHxA, vOa, nhjJO, CoJng, lwB, gqMrAf, gOdh, dAtJh, WoLP, HKHw, zaQt, Sqq, hweyD, MpJf, JPlmb, zlRBIX, shYlj, QBQGlt, mpDa, QCSmsA, kkB, jitvQ, MyVpzt, RSAtsz, uAcGP, WgqJM, asaV, ezaoPQ, BnLdq, TYVPK, jYvWY, RFAIeZ, okA, npLmM, qDB, SgUKv, Tkjal, VcDSz,

Largest Squid Ever Found, Importance Of Robot Operating System, Notion School Template, Messenger Something Went Wrong Android, Cisco Jabber Client Configuration, Image Not Loading In Chrome, Ros2 Tutorial Galactic, Pakistani Restaurant London, Ontario,