Come to think of it, why is MIRI so ineffective? A number of their researchers (past and present) have Ph.D.s; it’s not like they don’t know how to do research, or like they don’t know what typical rates of academic productivity are.
I guess it’s possible that their goals just aren’t well-defined enough – they’re like grad students without an adviser, or with an adviser who just says “your project is to think about how to control an unimaginably intelligent being that can change itself at will” and then leave them to it.
(It’s also possible that it’s just a pleasantly easy and low-pressure alternative to other available positions, which seems quite possible, although kind of sad – if people say they’re trying to save humanity, you’d hope they’d at least try really hard, you know?)
I spent some time at MIRI, and the researchers seemed to me to be extremely driven and passionate about their work, and acutely aware of how important it is (or how important they believe it to be). “Pleasantly easy and low-pressure” would be just about the last descriptors I would give of the environment - everyone was extremely nice, of course, but one definitely got a palpable sense of “holy shit we’re all doomed unless we do something” from the people working there (I mean, to be fair they all do like math, so they also took pleasure in the research itself - that seemed fairly secondary to me, though). I guess I’d say it’s closer to the first option you gave, although that makes them sound kind of lost and aimless, which isn’t quite right either. There was definitely a lot of the kind of groping, speculative philosophizing you might expect from people who were in the early stages of grappling with a difficult research problem, but there was also some of the nitty-gritty, “let’s grind out some math” type stuff that typifies actual research progress once you’ve figured out a promising direction. I wasn’t able to get a sense of what the exact “thinking/grinding” split was, but it didn’t seem out of line with what I would expect from e.g. a tenured math professor trying to solve a famously difficult math problem. Honestly, it seemed to me that they were doing exactly what they say they’re doing: trying to confront a really challenging problem, and trying not to shy away from it by shifting to some other problem that’s related but easier to publish papers about.
(seriously, we complain all the time about how science has terrible incentives that force people to churn out irrelevant papers and exaggerate the significance of their results to get published and all that, and as soon we have an organization that escapes that mess we want to drag them back in? MIRI should absolutely not be immune to criticism, but criticizing them by comparing their raw paper output to that of academia seems like the wrong approach to take)
(of course, we really do need some way of judging MIRI’s effectiveness, and the last thing I would want to do is give them a free pass to always just say “it’s a challenging problem!” and exempt themselves from any criticism over lack of progress. It’s tricky, though, because…well, it is a challenging problem they’re working on, and it’s exactly the kind of problem that I would expect to involve frequent periods where no tangible results were being produced. I don’t really have any good solution to this - right now I fall on the “give MIRI the benefit of the doubt” side of things, but I can see why other people wouldn’t. I guess I would just suggest that if you’re going to evaluate MIRI on it’s effectiveness and research progress, you do so in the same frame of mind you might use to evaluate a mathematician working on, say, P=NP, or some other math problem where a solution is generally viewed as being very far off)
This is really interesting, thanks. I see / am involved in lots of conversations about MIRI on tumblr but I rarely get a view from the inside.
Re: the last two paragraphs, I think an important aspect here is that it may be valuable to prioritize “looking good” in the short term so that they can attract more talent in the longer term. (By “longer term” I mean something like “more than a year or two,” I’m not talking decades.) I don’t mean they should look “less weird” so much as have more appealing technical content in addition to the weirdness. A talented mathematician who cares about MIRI’s goals isn’t going to be evaluating a MIRI position solely on the basis of whether they can and want to work toward those goals; they’re also going to be evaluating it on the basis of whether they’ll have interesting colleagues, whether their technical skills will actually be useable on a day-to-day basis, whether the team has some weird set of biases/preferences that would prevent real progress, etc. These people are more likely to be attracted to a research group that has produced interesting reports that let them say “yeah, it would be cool to work on that.” In that regard, MIRI is kind of a black box right now.
That isn’t to say that you can’t get diminishing, or even negative, returns, when you add more people to a project. (The Mythical Man-Month and all that.) But I think that’s mostly a worry with more down-to-earth engineering projects. With big questions like P=NP, having more minds working in parallel is generally better because to a first approximation every is just trying random things by themselves (whereas in a software development team you need close coordination, with everyone “on the same page”). Once enough people are working on the problem at once, some ideas bear fruit, and this results in a literature that other people can build on. In this analogy, MIRI is like a small research team working on P=NP when no one else cares about it – it’s not surprising that they haven’t gotten anywhere. Attracting more minds to the problem and building a literature of (small) useful results should be the goal – but this requires some short-term effort to make it clear to people that the problem can provide opportunities for interesting, non-wasted work.
At the risk of seeing patterns in clouds, I am starting to wonder whether there is a broader tendency for LW people to assume that their work will be received positively because it would be important if done well, even if they don’t provide evidence that it is being done well (in MIRI’s case, “that something can be done at all”). The other example I’m thinking of is Zvi Mowshowitz’s comments on the failure of MetaMed. He notes that people seemed to care more about whether their reports were glossy and professional than whether they contained useful information, and interprets this as people preferring “the symbolic representation of a thing” to “the thing itself.”
I’m not familiar with the details, but on the basis of Zvi’s post there seems to be an obvious alternative explanation: people wanted to believe that MetaMed was finding useful information, but could not directly evaluate that for themselves – after all, they were looking to MetaMed for expertise they did not have – so they used the best proxies they had, which were how professional MetaMed appeared. I’ve had a fair amount of experience with the alternative medicine world, and one thing I’ve learned is that (because the medical literature is so complicated and equivocal) any quack can claim their treatments are “based on science” – with citations – in a way that is not obviously suspect to people who are neither scientists nor doctors. These people end up evaluating medical advice on the basis of things like “how sketchy are these people? do they seem like quacks?”, rather than on a direct evaluation of their scientific legitimacy, which they don’t have the expertise to do.
The shared quality I’m proposing here is that MIRI and MetaMed seemed to have assumed that because they knew (or believed) they were doing good work, others would also realize this, unless they were people who “cared too much about signaling.” But signaling isn’t valueless – people do need some way to distinguish you from people who merely say they’re doing good work but aren’t.
So the thing that jumps out to me about @thepenforests’s comments are that it sounds like MIRI is working on this exactly the way everyone always says not to work on a hard problem.
If you want to prove the Riemann Hypothesis, you don’t sit down and think really hard about the Riemann Hypothesis. If that were gonna work, it would have worked already (and then it wouldn’t be the Riemann Hypothesis, at least not reputationally). You work on problems on the same area, and keep an eye out for ways they can relate back to RH, but you work on problems that you have some idea how to solve.
This is not the same as “work on easy problems to get papers out because you need a paper trail.” The point is that if you don’t know how to work on a problem then you don’t know how to work on it! And if you decide to just work on a hard problem that you don’t have a plausible path to solving, you wind up just flailing around a bit and not making any progress becaueys you don’t have any good ideas what to do and can’t always even tell if you are making progress.
Or in other words, you look exactly like how thepenforests just described MIRI. So yeah, that sounds like the problem of having smart people who don’t have the right kind of organization/approach to plausibly get stuff done.
(Doesn’t Andrew Wiles show that sometime it is a good idea to sit down and think very hard?)
But in general my impression of MIRI is that they are doing exactly what you are suggesting. If you read their technical agenda, they tried to identify a bunch of problems which are related to their vision of how friendly AI should work, but are small and well-defined enough that they should actually be possible to solve. And if you look at the list of publications, the results they announce also seem to have that flavour (e.g. the paper about definability of truth is not directly a solution to anything, but it came about because they poked around with “systems that reason about themselves”).
I mean, Wiles wanted to just sit down and think hard about FLT, but his advisor wouldn’t let him and made him learn about and work on modular forms and shit. Which is, of course, how he proved FLT. It’s not like the guy didn’t publish stuff before the modularity result. (The Coates-Wiles homomorphism was pretty important to my thesis work).
As for MIRI: they have one result. Which means that either they’re really bad at working on the intermediate problems, or they picked intermediate problems that are too hard/that they don’t have good traction for working on. I really don’t know which it is.
