I'd like to see a visual/language model/AI that learns to play minecraft as an actual inhabitant of the game. i.e. processing visual input, recognising objects, working out whats going on, learning how to move around. Learning how to make food and avoid monsters. It would be an 'Embodied AI' within the world of Minecraft.
The language part would allow us to talk to this being. You could ask it things like:
"Do you prefer to make a house, or dig a cave?"
"What are your hopes for the future?"
"Is there a recent achievement you are particularly proud of?"
You could ask it those things but it will tell you that it doesn't have feelings, preferences or hopes. You'd also need to give a reason to do anything. Eventually you get back to the point where you're "writing a bot" but with behaviour closer to how you imagine it should be.
>I'd like to see a visual/language model/AI that learns to play minecraft as an actual inhabitant of the game. i.e. processing visual input, recognising objects, working out whats going on, learning how to move around. Learning how to make food and avoid monsters. It would be an 'Embodied AI' within the world of Minecraft.
There is already an AI VTuber, Neurosama, trained to "play" games, including Minecraft (also OSU! and Among Us.)
I'd like to see a visual/language model/AI that learns to play minecraft as an actual inhabitant of the game. i.e. processing visual input, recognising objects, working out whats going on, learning how to move around. Learning how to make food and avoid monsters. It would be an 'Embodied AI' within the world of Minecraft.
The language part would allow us to talk to this being. You could ask it things like:
"Do you prefer to make a house, or dig a cave?"
"What are your hopes for the future?"
"Is there a recent achievement you are particularly proud of?"
etc