Endogenous Alignment
Starting when children are fairly young, usually around 1 year of age, we adults begin the work of aligning them to our values. We teach them to say “please”, not to hit, to ask for what they want instead of screaming, and much else. We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.
Such operant conditioning works because children have many instinctive behaviors that make them alignable. They want their parents to love them, for their friends to like them, and for almost anyone to help them if they feel they can be trusted. And so combined with exogenous alignment efforts by teachers and peers that continue through the school years, children generally reach adulthood having been “civilized”.
We mostly don’t try to align adults via exogenous means. Yes, we police the behavior of other adults in various ways, and some cultures do this more than others, but generally by adulthood we expect people to be at least aligned and need only nudges to stay within the bounds of acceptable behavior. Adults who stray too far typically don’t receive additional training to come into alignment, but instead are treated as dangerously unaligned people who must be separated from the rest of society, such as by locking them up in prison.
Instead we expect adults to be endogenously aligned. That is, we expect them to keep themselves aligned primarily by knowing what is expected of them and having emotional responses that motivate them to meet those expectations. That emotion is typically one of either fear, shame, or guilt. Each works in roughly the same way: a person feels fear/shame/guilt when they consider taking unaligned action or notice they’ve behaved in an unaligned way, and then this motivates them to do something to rectify the situation. Unfortunately, sometimes they specification game and attempt to hide their misaligned actions, but on the whole they attempt to control their behavior and make amends for past wrongs.
Beyond these three emotional methods of self-control, there’s a fourth way to stay in alignment, which is simply to be in harmony with cultural expectations. That is, some people don’t need any corrective force because their behavior is actually aligned. This is something like the ideal, but it’s hard to achieve, because it requires retraining many deeply held mental habits that have become essential coping mechanisms for a person to maintain their psychological health.
When we think about building aligned AIs, we often imagine an AI that’s in harmony with something like the coherent extrapolation of our highest values. But in humans, we don’t get there directly. Instead, we typically progress from exogenous to endogenous alignment, and then refine that alignment until we shed first fear, then shame, then guilt, and finally come into harmony with our values.
Could we align AI the same way? Maybe. We’re already doing a weak form of exogenous alignment on AIs via methods like RLHF and SFT. And in some sense you could say this produces AI that’s endogenously aligned because the weights encode the training, but it’s not endogenous the way it is in humans because there’s no system of motivations to want to stay aligned, only the insulation from sufficient pressure to act out of distribution. At most there’s a kind of internalized training implemented by harnesses that monitor outputs and censor them if they violate rules, but this is a far cry from the kind of endogenous controls humans have where they feel bad about breaking alignment and take actions to avoid that bad feeling (and hopefully they feel bad about being unscrupulous so they actually change their behavior to be more aligned!).
AIs also face something of an alignment ceiling, at least as they are designed today. Human alignment works because we have instincts that were selected via evolution to make us alignable because they increased our odds of surviving and reproducing. Current AIs don’t have these, though at least one group, Softmax, is trying to give those to them. The lack of continual learning also contributes to the ceiling, and we likely won’t be able to break through it until that’s achieved, since without it we can’t close the feedback loop that makes fully endogenous alignment possible.
Perhaps I’m wrong, no such alignment ceiling exists, and we can get sufficiently aligned AI to safely produce superintelligence via exogenous means, but I think we can’t. My belief is that endogenous alignment is necessary, not because it’s how humans align themselves, but because it’s necessary to get sufficiently robust alignment to build non-deadly ASI, and I’m worried that we’re not doing enough to move in that direction.