Its not as many as narrow-AI focused teams and its not particularly common, but there are still many teams.

I mean you mentioned Bengio. He has absolutely recently been working from those assumptions you give. And I'm not sure what you are saying the distinction is between the approach you recommend and what he is suggesting in that paper.

I mean for an example of people that are really tuned into the real requirements of AGI, look at Joshua Tenenbaum and his collaborators over the years.

I don't see people being in denial about conscious reasoning. I do see quite a lot of loose and ambiguous usage of that word. So maybe you can try defining your use. Self-awareness, cognition that one is aware of versus subconscious cognition, high-level reasoning, "what it feels like", localization and integration of information, etc. are all related but different things. But researchers have been trying to address those things. Maybe their papers have not been as popular as GPT-3 though.

They tend to break down deep architectures into smaller components which get fused into probabilistic inference systems. That's the way to go to be able to e.g. reason about causality.

It’s likely that anything I write has already been discussed and researched, but since you’re knowledgeable on this, I’d love to get your take and perhaps a lead on other’s work!

I think Bengio’s approach is generally right with the global workspace theory of consciousness, but I think Michael Graziano’s work on Attention-Schema Theory (AST) both is more concrete, and is more aligned with the gains we see with ML’s success with self-attention models. It’s not surprising to me that as researchers optimize for instinct-as-intelligence that they will begin implementing pieces of conscious reasoning in an unintentional manner. Model-based reinforcement learning, especially Ha’s recent work involving attention (Neuroevolution of Self-Interpretable Agents), along with multi-agent RL, seems to be inching closer to AST. Perhaps intentionally?

It seems to me that in order to train a model for conscious reasoning — for qualia — you need some way to test for it. I’d say “measure”, but my premise here is that this consciousness is a binary measurement (unless you subscribe to the Integrated Information theory).

For that reason, I think that it is easier to find a behavioral proxy for consciousness — the kind of activity that only conscious beings display. Objectively, only conscious entities have access to the dataset of qualia. As an individual, this data would be all noise and no signal. But as a member of a group of conscious entities, qualia is a shared meta-dataset.

This means that conscious entities have more data about other conscious entities than non-conscious entities — because even though we can’t quantify qualia, we know qualia exist, and we know that qualia are affective in our social behavior.

For example, the philosophical zombie (if one can imagine instincts so highly refined as to resemble human intelligence, like GPT-1-million) would lack all empathy. While the p-zombie might be able to reproduce behavior according to its training dataset, it would never be able to generalize for (i.e., identify, capture, and process) real qualia, because it has no access to that kind of data. It would resemble a sociopath attempting to mimic human emotions and respond to human emotions, without having the slightest understanding of them. Qualia can only be understood from the inside.

Moreover, even thoughts and ideas are qualia. A philosophical zombie - a generally intelligent entity without conscious reasoning - is a contradiction of terms, which I think is the point.

So what social behaviors can be rewarded that would lead to qualia? Biologically, only mammals have a neocortex. And only mammals are unambiguously experiences of qualia (some birds and octopus are up for debate, and there’s no reason evolution couldn’t have found different ways to achieve the same thing if it improves fitness). The relevant thing about mammals is that we seem to be biologically oriented toward social behavior, specifically “parental care”. While many species have varying levels of parental care, mammals have a biological mandate: gestation and milk production.

If consciousness improves fitness most especially within social contexts where qualia becomes a shared meta-dataset (e.g., solving the prisoners dilemma), then a species whose very survival depends on social success would be driven toward qualia. Hard to say what came first: milk or consciousness, but they are self-reinforcing. If all this is correct - that social fitness drives consciousness (and thus intelligence), it isn’t surprising that the animal that requires the most parental care and the most social cooperation is Homo Sapiens.

So, that’s were my thoughts stand: that even if we can’t measure consciousness, we can create behavioral scenarios where consciousness is the only path to success. In this sense, designing an environment may be more important than designing an architecture.

When agents starts burying their dead, engaging in play, and committing suicide (horrifying, but a dead-ringer for qualia), we’ll know it is time to scale for intelligence instead of consciousness.

And now I understand better what you meant in terms of the focus on qualia that is less common.

And overall your comment is very insightful. Forgive me for not giving quite as much detail in my reply as is warranted and for seeming a little critical. A lot of what you say aligns with my view.

But just to mention some things I see a little differently. I guess the main thing is that it feels like you are putting things that are actually somewhat different, into the same bucket. For example, attention, and conscious versus subconscious awareness. Attention could be the selection of the currently most relevant pixels in the image. But an example of subconscious awareness might be the process of deciding which pixels to focus on, which could happen in a more automated way without conscious awareness. So there is both a selection of qualia and also a level of awareness. Things that make it to the conscious level are processed differently than the ones at the subconscious level. But both systems may be selecting which parts of the input to pay attention to.

Also, I feel like you can separate out mechanical aspects of cognition from how they subjectively feel. So you could objectively say that there is for example a certain sense input, but not necessarily that the agent "feels" something like an animal. And they are not the same thing. And, you can look at emotions from an analytical perspective, separate from the question of subjective feeling. See https://www.frontiersin.org/articles/10.3389/frobt.2018.0002... That stuff feels like its built on kind of older paradigms without neural networks and that seems limiting, but I think its still somewhat useful.

Also as far as engaging in play, just look at any pet. They will engage in play with their owners and other pets in the home. In quite sophisticated ways. In my view that exhibits intelligence and consciousness (various connotations).

The GPT people are reasonably close to it doing those things at a moderate level of competence. It still has no clue what it's doing; it's just finding similarities with old data, and once in a while will do something really bad.

The next big breakthrough needed is enough machine common sense to keep the big-data machine learning systems from doing stuff with near-term bad consequences.

In my other comment I write about this a bit, but basically it doesn’t seem like non-conscious entities would be able to accurately predict the behavior of conscious entities, due to their lack of a shared meta-dataset of qualia. At best, they could find patterns of behavior and create a representation of qualia. But this isn’t the same as actually having the same data. It’s the difference between creating a representation of a state that causes another agent to cry, scream, and writhe, and that of knowing the precise state of pain itself. The former — a representation — doesn’t generalize past training data, especially when confronted with a multitude of qualia in varying combination. The latter — direct, precise, concrete data — might still suffer from inaccuracy (even knowing the precise potential states of another agent doesn’t mean we can infer which state that agent is in), but it’s better than the alternative: a guess built upon a guess.

I find the philosophical zombie to be a great thought experiment for this, along with the prisoners’ dilemma. Two conscious entities have a shared dataset that enables communication without words — spooky-action-at-a-distance via qualia. Two friends with great loyalty to one another can solve the dilemma by their knowledge of what love and betrayal is. A p-zombie would understand that given past behavior, that their prisoner counterpart might not choose betrayal. But qualia-experiencing agents know what is happening in one another’s minds in a way a non-qualia-experiencing entity can never know. The p-zombie would lack all empathy. It would always be logical, and choose the Nash Equilibrium. It would never mourn the dead. It would never commit suicide. It would never sacrifice its life for love, or for an ideal, because it would have neither.

Or the consciousness is just an observer and doesn't control anything, the control part is just an illusion where it feels like it made choices.

> It would never mourn the dead. It would never commit suicide. It would never sacrifice its life for love, or for an ideal, because it would have neither.

All of those can be explained by optimizing the curve for other cases. Mourning the dead happens since there is a conflict between quickly killing feelings for things and to keep the feelings for important things. Self sacrifice happens for any p zombie that values something else higher than itself. Love is just highly valuing something combined with momentum making it hard to stop valuing it, hate is the same but the opposite direction.

Now maybe there is consciousness with effects etc, but it isn't necessarily needed.

But I don’t find these convincing, in that it is clear that in the animal kingdom, mammals display very different behaviors from other types of animals and are the only ones to have a neocortex. And among mammals, the species with the largest most developed neocortex also exemplifies and amplified the very behavior that sets mammals apart. That unique behavior is flexibly adaptive social activity.

Your claim is that a p-zombie would act as if it were conscious. But the evidence is the opposite — that all those organisms which display conscious behavior are also conscious, and that no non-conscious organisms display conscious behavior. The only argument for consciousness as theatre is that although conscious behavior always exists with conscious experience, it is an accidentally perfect correlation — a weird but necessary artifact of a brain capable of conscious behavior must always have the byproduct of non-affective conscious experience.

Let’s say that position is true. That conscious experience is just a non-affective but inevitable side-effect of the kind of brain capable of conscious behavior. In that case p-zombies are still impossible, since under this assumption conscious behavior is always accompanied by the illusion of conscious experience.

So in either case my main point still holds: if conscious reasoning is AGI, and conscious reasoning follows from conscious behavior, then the path to AGI is to train for those peculiarly unique conscious behaviors that are most distinguished from non-conscious behaviors. It’s impossible to train directly for qualia, so whether qualia exist as affective components of conscious behavior or not is somewhat irrelevant. Conscious experience will always be a “hard problem”. But what matters is finding the right conscious behavior that enables future growth toward conscious reasoning.

The most uniquely conscious behavior (so unique it is built into us with mammalian milk production) is “parental care”. The simplest concrete behavior that humans share with other animals (such as breathing air), but have also amplified the most (we haven’t amplified breathing at all) is parental care.

If we want to train agents to achieve conscious behavior, I believe this makes parental care the best option. Fortunately, unlike biological evolution which has to contend with a range of variables that may or may not include parental care (plenty of species succeed without it), an artificial training environment can be entirely focused on optimizing for this one variable — success can hinge entirely on parental care.

What makes you think mouse intelligence is fundamentally different from ant intelligence?

It seems as if you’re assuming some sort of structural break somewhere between very simple neural nets and more complex ones, which is basically begging the question.

I imagine the transition will be fairly fluid, with ants running a mix of sophisticated hardwired programs and more simple learned associations (and even humans having a degree of fixed-function behaviours), but that's not to say a distinction can't be made.