Maybe-Mathematical Musings — I just read your post about how GPT-4 does a lot...

1.5M ratings
277k ratings

See, that’s what the app is perfect for.

Sounds perfect Wahhhh, I don’t wanna
nostalgebraist

marcusseldon asked:

I just read your post about how GPT-4 does a lot better on many trick questions posed to it. Do you suspect OpenAI is specifically finetuning/prompting/training (not sure right terminology on this) GPT on these trick questions? It seems like often when GPT flubbing a trick question goes viral, GPT magically starts getting the question right a week later. If this is what's going on, it makes GPT seem less impressive tbh.

nostalgebraist answered:

(Here’s the post in question.)

I doubt it, since ChatGPT 3.5 still gets the questions wrong – I checked. And I would assume it’s much cheaper and easier to re-tune that model than to re-tune GPT-4.

Also, we now have some degree of ability to check for these changes, because OpenAI’s API provides “snapshot” versions of these models from a particular date. As of this writing:

  • For ChatGPT 3.5, you can either use the latest model, or a snapshot from March 1.
  • For GPT-4, you can either use the latest model, or a snapshot from March 14.

Both of these dates are later than that twitter thread (Feb 22), but not by a huge amount.

I briefly tried comparing the snapshots to the latest models, on these questions, for GPT 3.5 and 4.

I didn’t observe any differences between snapshot and latest.

But I did notice that GPT-4, in the API, often got the same questions wrong!

I initially chalked this up to some difference between the API and the chat.openai.com experience (e.g. temperature, system message). However, after re-trying the questions on chat.openai.com, I found that GPT-4 sometimes fails there as well.

On the car crash and river crossing questions, it sometimes gives the right answer, but it also sometimes behaves like 3.5 and recites the solution to the more common version of problem.

I also some some odd cases like this, which aren’t easy to classify as success or failure:

image

This is not a failure of attentiveness, I think, though it is a failure to be helpful / to play along.

(Though there is a possible version of the situation in which the model’s response is correct: in real life, the resolution to an apparent paradox often involves the discovery of a flan in one of your assumptions, rather than devising an ingenious way to wriggle out of the paradox “as written.” From the model’s perspective, this might well be the case for the user here.)

It has a much higher success rate on the Monty Hall and feathers/bricks problems. On these, after trying a number of times, I was not able to elicit a failure.

Intriguingly, if I set temperature=0 in the API, the resulting (deterministic) answer is…

  • Wrong, for the questions it sometimes gets wrong at nonzero temperature (car crash and river crossing)
  • Correct, for the questions it always gets right at nonzero temperature (Monty Hall and feathers/bricks)

For example, in the car crash question, the temperature=0 answer is the single sentence

The surgeon is the boy’s biological mother.

We might interpret the screenshot above as the model first sampling a wrong answer very close to this, then noticing the problem, and attempting to work its way out of the hole.

I guess this underscores the necessity of running the same question multiple times and computing a success rate, if we want to seriously investigate the model’s performance.

(Even a single success shows that it’s possible for the model to succeed, which is something. But even then, it’s worth verifying that one didn’t just get lucky.

“It got the right answer on my first try!” sounds very different from “it gets the right answer 25% of the time.” Yet, for ~25% of questions for which the latter is true, someone who only tries once would report the former experience.)

jadagul

It seems like GPT4 does much better on these things, possibly better than a well-read but lazy and inattentive human, but still worse than a human who is paying attention. Which maybe is what I’d expected originally, so nice to see that maybe that’s true?

One question obviously is how much you can get better at this just by scaling up what we’re currently doing.

And then, as you’ve pointed out, how much we can scale up what we’re currently doing; we might be running out of text. The AI people I’ve talked to think they can solve that by using AI to generate text and then training the next generation of AI on that generated text, And I gotta say that I just don’t believe that can work for this.

ai gpt

See more posts like this on Tumblr

#ai #gpt

More you might like

My model of writing is something like: we have ideas, which are vague and amorphous and imprecise. And then we have to translate them into language, which has two benefits:

  1. This can commuicate those ideas to other people
  2. Language both allows and requires you to refine your ideas and make them more precise. This also helps expose contradictions and catch errors.

GPT doesn’t really work like this!

With GPT-2, this always made it really obvious to me when I was looking at GPT text. The words and sentences made sense locally, but there was no idea underpinning them so they didn’t connect to each other usefully. The newer GPTs are way less obvious, because they do seem like they can hold some sort of train of thought later.

But it still feels like they don’t “think”, and I was trying to probe that intuition. (And articulate it to make it more precise: step 2 above!) The first draft of the thought was something like “It just makes words, and doesn’t know things about the world”. But that’s obviously not exactly true. If I ask it “what color is the sky?”, I get this:

The color of the sky can vary depending on the time of day and the weather conditions. During the day, the sky can appear blue, while at sunset or sunrise, it can display shades of orange, pink, or red. At night, the sky can appear black, but if the moon is out, it can have a bluish or grayish hue.

And that represents some amount of information about the world! The model encodes a bunch of information about the world, which it learned by consuming text. But it does still seem like something is missing.

And on consideration, my new thought is: GPT4 has ideas (whatever information/data/priors are embedded in the model). And it has point 1 above, the ability to communicate those ideas to people. But it doesn’t have point 2. It doesn’t look at the words it writes and use them to refine the model.

And like this bare fact is uncontroversial, right? The training weights of the model don’t update based on the conversations it has. So if the weights in the model are the things GPT “believes”, then it is not capable of changing its mind.

(Of course the character it’s simulating in any given conversation can change their mind. But that’s a local update within the conversation. It’s not drawing on all the information and beliefs GPT encodes, and it doesn’t cause those beliefs to change. I can argue with ChatGPT. I can win an argument with ChatGPT. But that won’t change what the next person gets out of it at all.)

So GPT can produce text like I can, but it doesn’t use text like I do. It’s a way to output precached thoughts; it’s not a tool for thinking. (I almost want to say that it did all its thinking during training, and then doesn’t think while you’re talking to it.)

The LessWrong crowd was always worried about the dangers of an AI that could recursively self-improve. That seems to be exactly what GPT can’t do.

ai gpt

While I’m getting myself in trouble over gender:

I matched with someone on OKCupid yesterday. By which I mean she liked my profile and messaged me to tell me how much she liked it, and I agreed to the match and responded.

And her first followup message was to quote my list of movies and books I like and “gently ask me my fave women/POC creators”.

And like there are various ways I could answer that question. (I just enjoyed Gideon the Ninth. I subscribe to KB Spangler’s patreon at like ten dollars a month. I love Unpretty’s fanfic here on tumblr. Ursula LeGuin is amazing. Etc.)

But my real answer is, I think that’s kind of a poisonous way of engaging with media; I mostly don’t care about the demographics of the author, and I mostly don’t think you should either. I find the question, and the tone in which it was asked (“"gently”“), so offputting that my interest is now at zero.

from a place of privilege gender okcupid dating

I really liked this Sarah Constantin article on why current AI approaches fundamentally aren’t going to lead to AGI.

Her discussion of ontological robustness reminded me of the point I was making the other day, about how GPT on a deep level can’t learn.

And for the agent to be able to conceive of [the possibility that its fundamental model might be wrong] and count it as something to avoid, for it to be able to say I don’t want my loss function to be spoofed by something that kills me, it needs to have a “goal” or a “value” that goes beyond its loss function.

AKA, not a static terminal goal. A way to distinguish the “true real-world goal” from one’s mere current specification of it, and update the specification as necessary.

AKA, cross-ontological transfer of goals.

GPT-like systems just…can’t do this. They have their training weights, and that’s what they have; nothing ever updates.

And of course there are ways to retrain them and fine-tune them, but that’s awkward, and manual, and not feeding off of any of the AI capabilities that GPT has. There’s no iterative improvement; and there’s no obvious or even really suggested path for how to get there.

Sarah says no one is even really working on this, and I believe her. I haven’t encountered anyone genuinely working on this; and it doesn’t seem terribly commercially useful, while it does seem super hard. And this is part of why I’m just not terribly worried about the AI apocalypse.

ai gpt ml

More generally, Yudkowsky has long been

  1. Correct that a powerful optimization agent that we don’t understand the inner workings of would be extremely scary, and
  2. Wrong in his belief that creating such a thing would be easy.

If you grant that a super-powerful uncontrolled optimization process is about to be launched, you should in fact worry quite a lot. But it’s not.

In particular, the successes of GPT and the Transformer should reduce the credibility of Yudkowskian arguments, not raise it. Yudkowsky was arguing from a premise that any recognizable AI would necessarily have a certain collection of capabilities. GPT either does, or probably will soon, satisfy the conditions of a “recognizable AI”, but it doesn’t have those capabilities.

And the Yudkowskian conclusion is basically, therefore it must have those capabilities somewhere.

ai ml gpt yudkowsky

bulbous-oar

We don’t know that no possible large neural network implements an optimizer. If one does, we could stumble into it via gradient descent, just as we’ve stumbled into large neural networks that have world models.

Sure, we don’t know that no possible neural network implements an optimizer. In fact, I’ll go farther: it seems like if it’s possible to write one, which it probably is, we should probably be able to encode one in a large neural network. It’s just a very flexible framework for a model.

(Of course that flexibility is also the weakness: if it could be anything it has no strong pressure to be a specific thing.)

But I can say something stronger and also dumber than that, right? If an optimizer exists, then it’s possible that if I have my computer spit out 10TB of random bits and then interpret that as the source code to a C program, it will compile to that optimizer. It’s just…unlikely.

Large neural networks like GPT are not attempting to create optimizing agents. The training process is (outer-)optimizing for something, and the thing it’s optimizing for is very much not an (inner-)optimizer.

So this argument is something like, maybe writing an extremely powerful optimizer is so easy that we’ll do it by accident. And then not only will we create one, but it will be so powerful that it can out-optimize us, even if we’re trying to stop it, even though that’s if anything counterproductive to what we were training the model to do. That’s not impossible but it seems extremely unlikely.

(In contrast, I don’t think we “stumbled into” world models exactly; they are something we were more or less explicitly trying to get for the past couple years. This is part of Sarah’s point: people are trying to get world models, but they aren’t trying to get ontological stability or train inner optimizing loops basically at all.)

ai ml gpt
tanadrin
tanadrin

very high net immigration rates (the per capita ones) simply are not possible for large countries. fun number trivia: to have a net immigration rate as high as luxembourg’s, the US would have to import all of Italy or France every 10 years. to get a net immigration rate as high as Qatar’s, the US would have to import all of the Netherlands every year.

this is a good thing, though! it means big countries can absorb comparatively more migrants with less short-term disruption–the US has tons of small towns and cities people can move to, form immigrant communities in, and start living their lives.

you could turn the question around and ask, ok, but what would the US look like if it suddenly had a huge amount of immigration from poorer countries? wouldn’t it be poorer itself? well yes, on average, it would. because moving to a new country doesn’t instantly give you a salary that’s average for that new country. but the people already living in the US wouldn’t become poorer–they would probably become richer, because immigration is a net benefit to the economy. and the people moving would become richer over time. so the actual amount of wealth available to everyone would increase, even if on paper the US per capita gdp went down. obviously the wealth of new immigrants would go up faster if there was a UBI available to them, and in that respect, a UBI could be an important equalizing tool between old citizens and new citizens, a valuable source of capital to start new small businesses or to invest in education.

jadagul

So the question (in the context of the original post) is: in an open borders situation, what actually imposes the limit?  Especially in an open borders + UBI situation.

Like, the US is really far away from open borders right now.  We could (and probably should) double or triple immigration while still being really far away from open borders.  And that’s because so goddamn many people want to immigrate.  Gallup says 150 million people would move to the US if they could.  (And that’s without the addition of UBI as hypothesized in the original post!)

And obviously that would be fucking unmanageable over a few years.  So like one response here is to say yes, we’d never allow a hundred million immigrants in a five-year period.  And that is true.  But that’s denying the hypothetical of “open borders”.

So if we did move to an actual open borders situation, how many people would actually move here?  It probably wouldn’t be 150m; talk is cheap.  But would it be 50m?  20m?  Honestly, 20m over five years sounds manageable.  (There are currently about 750k green cards per year, so that would be quintupling the flow.)  But 50m definitely isn’t.  Would the inflow be self-limiting?  What would create the limit?

politics us politics immigration
freebroccoli
discoursedrome

keep forgetting that we do italics like this and not like *this* here

jadagul

Markdown editor sucks. (I keep trying to use it, but it sucks. It doesn't even fully implement Markdown!)

(On the Old Post Editor, the markdown editor randomly deletes all your line breaks on posting sometimes.)

But I've always been confused why Markdown allows *this* to be italics. That's obviously bold! _this_ is italics.

(In my Markdown code I use _this_ and **this** so everything is unambiguous.)

freebroccoli

I've always known _this_ to be underlined, and /this/ to be italics.

jadagul

Underlining is just italics for handwriting, right?  That’s a big part of why I’m used to that convention.

See this in the app Show more