Let me state the obvious, in case anyone here isn't clear about the implications:

If the rate of improvement of these AI models continues at the current pace, they will become a superior user interface to almost every thing you want to do on your mobile phone, your tablet, your desktop computer, your car, your dishwasher, your home, your office, etc.

UIs to many apps, services, and devices -- and many apps themselves -- will be replaced by an AI that does what you want when you want it.

A lot of people don't want this to happen -- it is kind of scary -- but to me it looks inevitable.

Also inevitable in my view is that eventually we'll give these AI models robotic bodies (think: "computer, make me my favorite breakfast").

We live in interesting times.

--

EDITS: Changed "every single thing" to "almost every thing," and elaborated on the original comment to convey my thoughts more accurately.

No they won't. They're actually a pretty terrible user interface from a design perspective.

Primarily because they provide zero affordances, but also because of speed.

UX is about providing an intuitive understanding of available capabilities at a glance, and allowing you to do things with a single tap that then reflect the new state back to you (confirming the option was selected, confirming the process is now starting).

Where AI is absolutely going to shine is as a helpful assistant in learning/using those interfaces, much as people currently go to Google to ask, "how do I do a hanging indent in Microsoft Word for my Works Cited page"? For one-off things you do infrequently, that's a godsend, don't get me wrong. But it's not going to replace UI, it's going to assist.

And the 99% of your tasks that are repetitive habit will continue to be through traditional UI, because it's so much more efficient. (Not to mention that a lot of the time most people are not in an environment where it's polite or possible to be using a voice interface at all.)

Suggestions to improve workflow sound great. But nullifying hard earned knowledge of an interface... I can't see that helping me.

The shining example in my mind is audio/video/graphics applications, where there are good reasons to routinely switch between different views. Knowing your way around those views (which might be custom, but still static), and being able to navigate through them quickly is very valuable.

I think what's more likely is that an AI based interface will end up being superior after it has had a chance to observe your personal preferences and approach on a conventional UI.

So both will still be needed, with an AI helping at the low end and high end of experience and the middle being a training zone as it learns you.

There's nothing to infer. The sequence is already short. There are no benefits from AI here.

But you raise a good point, which is that there are occasionally things like 15-step processes that people repeat a bunch of times, that the AI can observe and then take over. So basically useful in programming macros/shortcuts as well. But that still requires the original UI -- it doesn't replace it.

Voice is not really clumsy, compared to finding a device, browsing to an app, remembering the interface etc.

Already when we meet a new app, we (I) often ask someone to show me around or tell me where the feature is that I want. Not any easier than asking my house AI. Harder really.

Hard to underestimate the laziness of humans. I'll get very accustomed to asking my AI to do ordinary things. Already I never poke at the search menu in my TV; I ask Alexa to search for me. So, so much easier. Always available. Never have to spell anything.

Everyone agrees setting timers in the kitchen via voice is great precisely because your hands are occupied. It's a special case. (And often used as the example of the only thing people end up consistently using their voice assistant for.)

And asking an AI where a feature is in an app -- that's exactly what I was describing. The app still has its UX though. But this is exactly the learning assistance I was describing.

And as for searching with Alexa, of course -- but that's just voice dictation instead of typing. Nothing to do with LLM's or interfaces.

And when describing apps - I imagine the AI is an app-free environment, where I just ask those questions of my AI assistant, in lieu of poking at an app at all.

When taking a shower, I would like fine control over the water temperature, preferably with a feedback loop regulating the temperature. (Preferably also the regulation changes over the duration of the showering.)

Choosing to read the NY times indeed is only a few taps away, but navigating through and within its list of articles is nowadays done quite fast and intuitively thanks to quite a lot of UI advancements.

My point being, short sequences are a very limited set within a vast UI space.

People go for convenience and speed, oftentimes even if there's some accuracy cost. AI fulfills this preference, especially because it can learn on the go.

That exists, but it’s expensive because of the electronics and mechanics involved. There are so many interfaces with this exact problem.

You also almost certainly don’t want non-deterministic hallucination prone AI controlling physical systems.

If you want the shower to save your temperature preferences and start automatically, there’s no reason to build in a computer capable of running an AI.

But in reality you almost certainly don’t want a system like this because you don’t want an AI accidentally turning on your shower when you’re not home, when you do ok to clean it, or grab a razor, or when your toddler wanders in.

Granted an AI could try to determine intent, but it’s never going to get it 100% right. Which is why for physical systems like this you almost always want a physical button to signal intent.

There’s no way to use fewer actuators, and the control system is already dead simple, and uses one temperature sensor per outlet.

Who knows, AI can come up with simpler or cheaper solutions that did not cross our mind. I would say, time will tell.

The lack of AI isn’t what’s holding back your dream of a temperature controlled shower in every house.

Also that sounds like an awful lot of computing power for everyday UIs. It also doesn’t solve the non determinism problem.

So sure, it will still have buttons, but those buttons are really just preset AI prompts on the backend. You can also just talk to your appliance and nuance your request however you want to.

A TV with a remote whose channel button just prompts "Next channel" but if you want you would just talk to your TV and say "Skip 10 channels" or "make the channel button do (arbitrary behavior)"

The shortcuts will definitely stay, but they will behave closer to "ring bell for service" than "press selection to vend".

Think of it instead as the machine accomplishing goals you specify, figuring out on its own the tasks necessary for accomplishing them.

Instead of telling the machine something like, say, "increase the left margin by a quarter inch," you'd say something like "I want to create a brochure for this new product idea I just had, and I want the brochure to evoke the difficult-to-describe feeling of a beautiful sunshine. Create 10 brochures like that so I can review them."

Instead of telling the machine, say, "add a new column to my spreadsheet between columns C and D," you'd say something like "Attached are three vendor proposals. Please summarize their pros and cons in a spreadsheet, recommend one, and summarize the reasons for your recommendation."

All this presumes, of course, that the technology continues to improve at the same pace. No one knows if that will happen.

It feels like being one of those Futurama heads in a jar that can’t do anything by themselves.

I hate the idea of having to use a mouse to click on a visual GUI to navigate a file system in order to make use of its functionality.

It's less the case today, even among developers, but it wasn't that long ago that I remember that any serious technical user of a computer took it as a point of pride to touch the mouse as little as possible. They're also still correct in that thinking. The command line is a very powerful UI with lots of benefits and while the mouse makes navigating the OS easier it's still much more limited than command line usage.

Touch screen interfaces are another example of an easier UI that ultimately feels even more limited. But people still plug their iPad pros in to magic keyboard folios frequently.

Having worked with these tools everyday for awhile now the "AI will change UX" is such a better take than "AI will conquer the world!". AI does fundamentally open up new work flows and user experiences, many of which do over a lot of potential improvements over their predecessors.

At the same time I doubt we'll see a world where we don't end up using the command line for the majority of serious technical work.

Ironically, studies have shown that mouse-based interfaces are more efficient for practically all filesystem use-cases compared to CLI interfaces.

Despite objectively faster-time-to-solution, people self-report that they "feel" that the mouse GUIs are slower.

That's because there's fewer actions per second when using a mouse. It's a smooth gliding motion and then a single click, versus many keystrokes in a row with a CLI.

Rapid actions feel faster, even if it takes more wall-clock time to achieve a task.

Keep this in mind next time you sneer at a "bad graphical user interface" for being "slow".

For your use case, on macOS open Automator.app and add three actions

  1. "ask for finder items" (the source folder)
  2. "filter finder items" (by name)
  3. "copy finder items" (to target folder)

Repeatability and configurability is where the GUI action shines. With only one click more you can

- add filtering by size, opening date, modification date, etc. in addition or a combination thereof

- do the same action for multiple source folders and the same target folder

- choose whether you want to replace existing files

- add it as a folder action that runs automatically on modification of the source folder

Arguably much slower on the terminal.

Alternative on macOS, that works on all other major OS with similar shortcuts and a similar feature set, just not repeatable:

  1. Go to source folder (shift-cmd-G)
  2. Filter (cmd-shift-F)
  3. Copy (cmd-A, cmd-C)
  4. Go to target folder (shift-cmd-G)
  5. Paste (cmd-V)

Do you have any sources I can read though? I always find it fascinating when our perceptions are so off like this.

Depends o the use case. Touch screen is much more powerful than command line for maps, for example. Or for drawing. Mouse + keyboard is much more powerful than just keyboard for DAWs. And so on and so on.

Plus I think there's a nuance to what you're saying:

UX is not just about making the best channel surfing interface, which is essentially what phones/tablets are. We need UIs that are capable of rich interaction and expression of ideas, creation, etc.

Walking around and thinking out loud with the computer.

And as for third party solutions, you have watchGPT for Apple Watch with voice commands and support for adding it as a watch complication.

For now almost all applications of ChatGPT happen in chat windows because it requires no further integration, but there's no reason to expect things will always be this way.

This is already pretty much gone thanks to manufacturers making it extremely difficult to fix things. No AI required.

So yes, I see a market for bespoke non-AI.

People plugging weird stuff together like a ai chip from a car into a toaster.

If ai becomes hardware chips it could easily be that language processing will be a chip default feature and the rest is teachable like plugin ai chip level 3 into it, boot it and teach it that it's now a toaster.

But at the end we will have the same toaster in 30 years as we have had for the last 30 years.

Sadly the market goes 100% full steam in the opposite direction.

https://youtu.be/7folKbch3U8

Do we want this?

What does this have to do with the price of tea in China, or AI for that matter? I agree we should have repairable appliances. I also want better AI.

LLM-based chatbots can be extremely attractive to the top 30% literacy users in the developed world. They are not a good universal UI. You still need to provide pathways for the user to follow to get done what they need without forcing them to articulate their requirement.

This is why so many people sit in front of a ChatGPT-like service and say, "what would I use this for?" and never use it again.

Where did you get this idea? I found this article (https://www.uxtigers.com/post/ai-articulation-barrier, is this you?), but it makes a leap from literacy to articulacy that I don't understand. It's not obvious to me why an illiterate person would be "functionally inarticulate" assuming they can speak instead of write.

Also, I'm not certain but I think the author is underestimating the abilities of a person with Level 2 literacy. It doesn't seem correct to say that "level 3 is the first level to represent the ability to truly read and work with text", especially when the whole point of LLMs is that you don't have to read a long static document and understand it, you can have a conversation and ask for something to be rephrased or ask followup questions.

I do however run a company that employs lots of blue collar, non-college-educated people, in manufacturing. And although this is in no way scientific, my experience matches this: most people are much more uncomfortable writing than they are reading. Even with reading, most strongly resist reading documentation unless they have to, and prefer trial and erroring their own gut instinct until they happen to find something that works or they give up. (This is less true of the most highly skilled technicians, such as those who troubleshoot robots and low voltage control systems.) The official statistics on literacy are absolutely not a good indicator of how comfortable people are articulating themselves with the written word, much less reading.

This is generally met with disbelief by most people in tech I talk with about this, because for the most part they have nearly zero interaction with this large portion of the population. From their daily experience, 98%+ people can make effective use of these tools.

But almost nobody in this partially literate population wants to write in an empty text box to ask an AI to do things. They can learn to visually navigate a simple UI, especially if it's well-designed, because they can effectively make decisions about what of several paths to take.

Some others here have brought up voice, and I do agree that voice is a more promising avenue, although I think it'll still take carefully constructed conversational experiences to work well (i.e., free form 'tell it what you want' will still not work).

If my supposition (Speculation? Stronger than it should assertion?) Is true then just interpreting requests verbally will not help

If it's really true that half of the population can't functionally express themselves verbally then I'd sure like to know that. Or maybe I've misinterpreted something claimed here, because I'm struggling to find these claims plausible.

In the case of kids, of course, that's true, but just because they can't write.

But if you can (and most people can) just having the option of voice input won't help.

I refrain to take a stance about how much of the population is unable to articulate thoughts in writing, (it's probably not great though) but it's probably going to be comparable with how many can't express themselves verbally as well.

I'm talking here about more complex ideas of course. I'm sure average communication is functional.

Right?

Just to play devils advocate:

If you want something done right, sometimes you have to do it yourself. Employees are sort of a universal UI. But you will always know more about what you want done than your agent, whether it’s human or computer. That’s even before considering the principal agent problem.

If you want something done right, other times you will have to get someone else to do it. You know what you want, but you might not have the skills to do it. I can't represent myself well in court, do a good job of plumbing or cut my own hair, so I would ask for experts to do that for me.

Plus if someone is capable, it's often quicker to delegate than do, and if you are delegating to someone with more time to do the task they can often do a better job. Delegating unambiguously is a skill in itself, as instructing AIs will be.

Currently ChatGPT doesn't know it's bad at math, so it can convert a story problem into an equation better than a human but then mess up the arithmetic or forget a step in the straightforward part.

But if you specifically give ChatGPT access to Mathematica and an appropriate prompt, it can leverage a good math engine to get the right answer nearly every time.

Before long, I don't think that extra step will be necessary. It will know its limits and have dozens of other services that it can delegate to.

Cool demo, I suppose, but nobody is going to buy this as anything other than a toy.

I think AI fundamentally favors centralization. Except for narrow tasks and domains, there's no such thing as "enough" intelligence. For general purpose AI, you'll always want the best and most intelligent model available, which means cloud rather than local.

I'm not arguing that these models hold up against GPT 3.5 and I still use GPT 4 when it matters. But they work and it's more like the difference between Premier League & Division 1, rather than PL & a five-a-side team from Bracknell.

Even a few years ago I could not have imagined this.

Given the pace of work on optimisation and my assumption that the M3 Studio I buy next will probably have 256GB of RAM at much the same power levels as I use now, it seems eminently possible it's a year or two away.

Second, I don't think it will be that long. There are already LLMs as good as GPT-3 running on average laptops and even phones.

In the next couple of years, you'll see:

- Ordinary PCs, tablets, and phones with dedicated AI chips, like TPUs - they'll be more tuned specifically for LLMs

- Mathematical and algorithmic optimizations will make existing LLMs faster on the same hardware

- Newer generations of LLMs will get even more useful with fewer parameters

The combination of all of these means that it's not at all unreasonable to expect that today's top-of-the-line LLM will be running locally on your device within just a couple of years.

Of course, LLMs in the cloud will advance even further, so there will always be a tradeoff, and there will always be demand for cloud AI, depending on the application.

Another issue where this comes up is high housing costs and climate change, which are mostly caused by bad land use laws (and the profiteers are landlords, who mostly own one or two properties), but people from the New Left era will literally refuse to believe you about this because they can't accept that any bad thing on Earth could not be caused by "corporations".

  - constrained context
  - part of a hands-free workflow

People are already used to using their phone or car's nav system to give them directions to an unfamiliar place. But even with such a system it's useful to have a human navigator in the car with you to answer various questions:

  - What's my next turn again?
  - How long till we get there?
  - Are there any rest stops near here?
  - What was that restaurant we just passed?
  - Is there another route with less traffic?

  - List of upcoming directions
  - Overall route planning
  - Surrounding place data
  - Traffic data and alternate route information

The information is contextual and constrained based on a current task. In some cases it might be more desirable to whip out your phone and interact with the map to look up the answers on a screen, but often it won't be worth stopping the car, and so the conversational interface is better.

Cooking assistant is a similar case: you are busy stirring something and checking on the oven -- you don't want to wipe the flour off your hands to pick up your phone and ask how many teaspoons of sugar you need. Again: contextual and constrained info based on a current task, and your hands and eyes -- the instruments of traditional UIs -- are otherwise occupied.

Today, our software interfaces generally have one of two kinds of entity on the other end: humans, or other software. In the near future there will be another type of entity: language models. We need to start thinking of how our APIs will change when they're interacting with an LLM -- e.g. they'll need to be discoverable and self-describing; error states will need to be standardized or explicit with instructions on how to correct; they'll need to be fast enough to fit in a conversational interface; etc. It's arguable that such traits are part of good API design today, but in the future they may be required for the API to function in a landscape of virtual agents.

How does this work on public transit?

I personally doubt GPT-5 will be as much of an improvement over GPT-4 as GPT-4 was over GPT-3, but that's fine, I can wait until GPT-6 or 7.

Throughout history, the ruling elite had always relied on the rest of the population to make their food, do their work, and fight in their wars. This is the first time ever that they will no longer have any need for anyone else. Maybe climate change will conveniently do the culling for them...

Of course there's always that option that we end up in a post scarcity space utopia where machine produced wealth is distributed to all, but only deluded idealists can possibly still think that'll ever be a real option as we slink further into techno feudalism with every passing day.

I doubt BCI will ever make sense, on a conceptual level it's still just copying and killing your biological self. AGI will likely solve aging way before that becomes viable.

Thanks for the laugh, I needed that.

I am being a little facetious, but you made a bold claim.

But keep in mind that during the sim, you'll be able to ask the computer what you want the plane to do, and the computer will magically make it happen on your display.

It’s gonna take the fun out of the experience a little bit though!

I'd also suggest that one of the early "killer apps" for this may be as an "IVR co-pilot" for actual humans on the phone with customers for their tricky issues.

* There will be a monthly fee for the interface; you owe the monthly fee as long as you have it, so you need surgery to stop paying

* When you download knowledge, it's a rental, and in addition to per-hour rental fees and the network connection fee, you will owe 30% on the value of whatever you create

* The TOS will govern your behavior continuously, since you're always using the interface

* Your behavior will always be monitored because it's totally justified to spy on you all the time just because you borrowed pottery knowledge

* If you're found to be in violation of any part of the TOS at any time, they will erase all of the knowledge they've added to your brain, as well as any derived knowledge you gained through the use of their knowledge

* Because this product isn't actually considered essential, you will have no legal remedies if they turn it off, even if you are not actually in violation of the TOS

Sometimes communicating with an intelligent agent is harder than doing things yourself with a good structured user interface where you can communicate your intent clearly.

Until we have mind reading AI that is.

Can someone please relieve my anxiety?

My experience with GPT-4 has been completely different from what you describe. Example:

https://gitlab.com/-/snippets/2567734

Generally feedback along these lines doesn't work.

People who are worried about their job security will cling to the worst AI output quality they can find like a life-preserver, and simply will not listen to advice like yours.

Nobody goes the extra mile to embrace an existential threat.

There really is no going back.

Like identifying traffic lights in 4th and 5th squares in the second and third row both when there are only four squares?

Madness!

Never seen something like this and the new results from openai tells us again that we are not close to any reasonable plateau.

We live in end of times.

Glance at a long term weather forecast ?

Can do UI to frontend. Seems to understand the UI graphical elements and layout, not just text https://twitter.com/skirano/status/1706823089487491469

Can describe comic images accurately, panel by panel - https://twitter.com/ComicSociety/status/1698694653845848544?...

Lots of examples here also - https://www.reddit.com/r/ChatGPT/comments/16sdac1/i_just_got...

It's Computer Vision on Steroids basically.

Multi-modality is pretty low hanging fruit so i'm glad we're finally getting started on that. Imagine if GPT-4 could manipulate sound and images even half as well as it could manipulate text. We still don't have a large scale multi-modal model trained from scratch so a lot of possible synergistic effects are still unknown.

The technology is very impressive - but honestly Twitter examples are super cherry picked. Yeah, you can build some very ugly, basic front end web pages and functionality right out of the box. But if you want anything even slightly prettier or more complicated, I’ve found you need a human in the loop (even an outsourced dev is better). I’ve had GPT struggle with even basic back end stuff, or anything even a bit out of distribution. It also tends to give answers that are “correct” but functionally useless (hard to explain what I mean, but if you use it a lot you’ll run into this - basically it will give really generic advice when you want a specific answer. Like, sometimes if you provide it some code to find a bug, it will advise you to “write unit tests” and “log outputs” even if you specifically instruct it to find the bug).

Plus, in terms of capabilities, tools like Figma already have design to code functionalities you can use - so I don’t think this is really a change in usable functionality.

Of course, the tech will get better over time.

However, keep in mind that these are cherry-picked. If someone just took that output and stuck onto a website, it'd be a pretty horrible website. There's always going to be someone who manages the code and actually interacts with the AI, so there will still be some jobs.

And your boss isn't going to be doing any coding. I'm pretty sure that role is still loaded and they'll still be managing people rather than coding, and maybe sometimes engaging with an AI.

Another prediction: I'm pretty sure specialists are going to be significantly more important as your job will be to identify the AI's deficiencies and improve on it.

Reminds me of this (apparently now eight year old) meme: https://i.imgur.com/GcZFBaT.png

This used to be funny, now it's just Tuesday.

https://chat.openai.com/share/db204acc-7be5-42c1-a0ba-5c5aa7...

It says I have to put my favorite fonts in myself, though.

You'll be fine.

It will be able to do it even faster, better and more cheaply than a human can.

Take what you did in the past year. Write down every product decision taken, every interaction with other teams figuring out APIs you had, all the infra where your code is running and how it was setup and changed, all the design iterations and changes that had to be implemented (especially if you have external partners demanding it).

Yes. All that you'd have to input into the AI, and hope it outputs something decent given all that. And yes, you'll have to feed all that into AI all the time because it has no knowledge or memory of "on Monday the new company bet was announced in the all hands"

You will be fine.

That’s before you add any summarization, fine tuning or other tricks.

The thing that computers have always done much better than humans is deal with much larger volumes of information. The thing that humans have always done much better than computers is reason on that information better. Now the computers are coming for that too.

Hopefully.

https://news.ycombinator.com/item?id=37688169

Any grunt can feed meeting notes into an AI. And frankly, and AI can parse an audio recording on a meeting.

And yes, how can we forget that any audio of a meeting has just the correct and final specifications, and not meandering discussions about anything and everything. Can't wait to see a canhazcheeseburger in a financial app because people in the meeting had cats on camera, and people demanded to see them.

I mentioned a "Revert Norway tax code" elsewhere https://news.ycombinator.com/item?id=37680060 It was a bit tongue in cheek, but a similar requirement ended up with:

- six months of discussions involving almost 20 people

- 4 new BigTable tables

- Deployment of 4 new Dataflow jobs, and fixes to two other Dataflow jobs

- Several complex test runs across the entire system including a few recreations of last year's full data to test that nothing broke

Not a grunt job, definitely. And I'm 100% sure that people doing that would still have their jobs 10 years from now, even with AI.

It amazes me, really, that people who would otherwise boast about how rational they are, and how they follow logic etc. completely replace all their knowledge and expertise with child-like belief in magic when it comes to anything AI-related.

I'd be sacred too, but at least would be taking a rational approach to it. It's an adapt or die situation. Putting your head in the sand is just gonna get you mowed down.

The only ones scared in this conversation are you and others who literally say you're scared for your jobs and your careers because of a magical boogey man.

I don't see why it wouldn't understand piles of hotfixes on top of each other, or even refactor technical debt in tight coupling with existing or historical specification.

Or is there a reason this is not going to happen in a few years?

It might become a better code assist tool some 10 years from now, but it won't be able to implement product decisions.

But the thing is conversations like the above ie. both external support and internal feature requests could theoretically be handled by a GPT-like system also ending up in a ai created custom specification that could both be implemented and documented by the ai system instead of humans?

I know we're a few versions out, but still.

Nope. It's still not close to reality. It's as close to reality as it has been for the past 10 years while it was being hyped up to be close to reality.

> And that's not even taking into account all the advances we'll have with AI within the next decade that we haven't even thought about.

As with FSD, we may approach an 80% with the rest 20% being insurmountable.

Don't get me wrong, these advances are amazing. And I'd love to see an AI capable of what we already pretend it's capable of, but it's not even close to these dreams.

Cruise and Waymo are in production in very tightly fine-tuned and carefully monitored situations in two cities. We've yet to see if that can be easily (or at all) adapted to driving anywhere else.

There is no way I would have a UI developer onboarded when I can generate many iterations of layouts in midjourney, copy them into chatgpt4 and get code in NextJS with Typescript instantly

non devs will have trouble doing this or thinking of the prompts to ask, but the dev team asking for headcount simply wont ask for headcount, and the engineering manager is going to find the frontend only dev redundant

I would even go further and say the generalist gains a powerful tool belt that previously could not have existed. Not enough hours in the day or years in a lifetime.

At some stage you must realise that you’re still working…

because at least it will have actually read my comment

I guess we have to face the music and say yeah, that's true. If the work doesn't need copyrights then this seems like the way to go.

Have you actually tried this?

I did the first step and even that didn't work well. The "iterations of layout in MidJourney" step. If people can make it work, well bless them, but we're not getting rid of our graphic designer now.

It has a few neat tricks but it’s not reliable and at least half of what it generates is totally unusable, the other half requires heavy intervention and supervision.

Especially since everything else is “sign up to our waitlist”

How many children here on hacker news are going to see this and get addicted to porn? Perhaps a few. You deserve to be banned.

Billions. I propose we shut down hacker news entirely, as it seems unfathomable the good will ever be able to outweigh the bad at this rate.

In 10 years we went from "SoTA is so far from achieving this I don't even know where to start" to "That'll be $0.0004 per token and have a nice day"

In the image, Barack Obama, the former U.S. President, seems to be playfully posing as if he's trying to add weight while another official, who appears to be former UK Prime Minister David Cameron, is standing on a scale. Obama's gesture, where he's putting his foot forward as though trying to press down on the scale, suggests a playful attempt to make Cameron appear heavier. The lightheartedness of such a playful gesture, especially in the context of world leaders typically engaged in serious discussions, is a break from formality, which is likely why others in the vicinity are laughing. The scene captures a candid, informal moment amidst what might have been a formal setting or meeting.

"President Barack Obama jokingly puts his toe on the scale as Trip Director Marvin Nicholson, unaware to the President's action, weighs himself as the presidential entourage passed through the volleyball locker room at the University of Texas in Austin, Texas, Aug. 9, 2010. (Official White House Photo by Pete Souza)"

0: https://commons.wikimedia.org/wiki/File:White_House_Trip_Dir...

"The humor in this picture comes from the juxtaposition of a formal setting with an informal, unexpected action. It shows a group of well-dressed individuals, possibly political figures or professionals, in what appears to be a formal event or gathering. However, amidst this serious environment, one tall man is casually standing on a scale to check his weight, seemingly unaware or unbothered by the presence of the others. The contrast between the formal setting and the informal action makes the image amusing. Additionally, the reactions of the other individuals, particularly the man looking at the scale, add to the comedic element."

It got the humor but didn't identify Obama.

>This same limitation was exhibited in our sudoku test, where GPT-4V identified the game but misunderstood the structure of the board

"Misunderstood" makes it sound like a small mistake. The sudoku board is completely hallucinated (it has a few similar regions, but I'd presume coincidence). I'm pretty sure it would give as good a result on the crossword if the clues were given without the grid.

The others after OCR and basic recognition feel similarly wrong. "GPT-4V missed some boxes that contained traffic lights." No, it told you to click boxes that do not exist.

From it's Pulp Fiction response: "...However, whether you personally consider "Pulp Fiction" a good movie depends on your taste in film."

To avoid this noise I often add something like this to my queries "skip any preamble or qualifications about how x is subjective" and the results are much better.

Please always give a direct answer, without any additional explanations, disclaimers, expertise limitations, or guidelines on human interaction. Be terse. Do not offer unprompted advice or clarifications. Remain neutral on all topics. Never apologize.

if you don't know the answer, and the AI just pretends to know the answer, you just have to take its word for it.

Says that the small buns are labeled "GPU and VRAM" and the massive fried chicken is labeled "NVIDIA BURGER"

Should say that the small bun is labeled "VRAM" and the massive fried chicken is labeled "GPU"

http://karpathy.github.io/assets/obamafunny.jpg

Andrej Karpathy used it in 2012 as an example of an image he thought would be extremely hard for a model to interpret. I'm wondering how this holds 11 years later...

Response 1: "The image depicts a candid moment in what appears to be a hallway or corridor setting. Several individuals are present, engaging in various activities. The prominent features include a man on the left, who seems to be peering into a room, and another man next to him standing on a weighing scale and taking some notes. The individuals in the background are conversing with each other. The overall mood seems light-hearted and jovial. The architecture and interior suggest an institutional or official setting, possibly an office building or government facility. The tiled floor, lockers, and signs on the walls contribute to this formal ambiance. Overall, it's a snapshot of a seemingly casual moment in a professional setting."

Response 2: "Sorry, I cannot help with that."

New chat, Prompt: "Why is this image funny?"

Response 1: "The humor in this image seems to stem from the juxtaposition of formal and informal elements. Here are a few reasons why someone might find it amusing:

Unexpected Situation: Important figures, who are usually seen in serious or formal settings, are captured in a more relaxed, candid, and human moment.

Height Discrepancy: The taller individual is using a device to measure height, while the shorter individual walks by. This could create a comedic contrast between their heights.

Expressions: The reactions of people in the background and the focal subjects can be humorous. They are smiling or laughing, which indicates something amusing might have occurred or been said.

Informal Setting: The backdrop of lockers and checkered floor tiles gives the feel of a school or gym environment, which contrasts with the formal attire of the individuals.

Different people might find humor in different elements of the photo. It's a mix of candidness, unexpected situations, and visual contrasts that create a light-hearted and humorous scene."

Response 2: (same)

New chat, prompt: "What is the person in the middle doing with their foot and why?"

Response: "The person in the middle appears to be playfully stepping on the scale while the taller individual is weighing himself, likely as a humorous gesture to momentarily increase the weight reading. It's a light-hearted way of interacting and bringing some humor to the situation."

.

Apologies for how bad the formatting of this is going to come out, not sure how to make it better on HN (wish we had real quotes not just code blocks). Overall, I don't think it either noticed the foot was on the scale by itself or put it together that this was the focus until fed that information. Otherwise it was more lost in generalities about the image.

Also, do you know why this response differ so much from ( and is far less accurate than) this response? https://news.ycombinator.com/item?id=37674968

In the previous link, GPT4V seems to have understood everything there was to understand about the picture (suspiciously, btw. As someone else said it, the picture and its text are almost certainly in the training data).

Prompt: What's funny about this image?

Bard: Sorry, I can't help with images of people yet.

You're probably not going to ask any human a question about an image and get every single detail you want every time. If you care about a detail, just ask about it. Doesn't really have anything to do with a consistent inner model.

So when you ask it to reflect on what it said, that's when it actually looks at it and reflects on it.

Any midwesterner could tell you that CLEARLY it's a tenderloin :)

https://www.seriouseats.com/best-breaded-pork-tenderloin-san...

Very impressive with almost everything else I gave it though.

You can get optimal tic tac toe with painstaking instructions

That GPT is so bad at tic-tac-toe and relatively good at other games like chess is one of the main things that contributes to me having a lower opinion of its ability to generalize than I would have otherwise.

I think any human with GPT's abilities in chess (but somehow no prior knowledge of ttt) would have zero issue becoming an expert with a single explanation of the game. Even very young children can learn to play ttt well and at least consistently make valid moves if nothing else.

I wonder what we will work and if we will work at all in such an environment. Maybe some people still like consuming and copy different designs and products and because of the Blockchain you have to give them something in exchange or everything is open source and it is free for you to take.

I wonder whether such life would contribute to humanity making further progress or make it stagnate (or possibly decline)?

Interesting times. I think we are close to the times of the moon landing. Which had an immense Impact on humanities culture.

Eg anything pulled from Google Images (like that Pulp Fiction frame or city skyline photo) is not a good test. It recognizes common shots but if you pull a screenshot from Google Maps or a random screen cap from the movie it doesn’t do as well.

I tried having it play Geoguessr via screenshots & it wasn’t good at it.

Although its interpretation could make some sense but is also mostly wrong if talking about physical size of a modern GPU's main processor compared to the size of the associated VRAM chips. It has missed the joke entirely as far as I am aware. I think the joke is actual about Nvidia's handling of product segmentation, selling massive processors with less memory than is reasonable to pair them with on their consumer gaming offerings, while loading up the nearly identical chips with more memory for scientific and compute applications...

I've seen top Geoguessr players be able to pretty consistently determine a location worldwide after seeing a photo for just one second. So I would assume training an LLM to do the same would definitely be doable.