Homeostatic reinforcement learning (HRL)
Introduced to the wiki by Petzschner et al. (2021), and important here mainly because it is the first rival the predictive-coding thread has had.
The move
Classical reinforcement learning has an agent maximizing long-run expected reward, where “reward” is exogenous — supplied by the experimenter, or by nature, but not derived. HRL (Keramati & Gutkin, 2014) derives it. Define drive as the distance between the organism’s current internal state and its desired internal state; then define reward as drive reduction. Actions predicted to bring the internal state closer to the desired one are, by construction, rewarding.
Everything else is ordinary RL. Estimated values, reward prediction errors computed by comparing expected to experienced reward, value updating, action selection. The homeostatic content is entirely in the redefinition of the reward term. An organism that maximizes long-run reward under this definition is thereby maintaining homeostasis.
Why it matters to this wiki: regulation might not need an interocept
The wiki’s working assumption, inherited from Craig through Seth to Barrett, is that regulating the body requires representing it — the brain infers the hidden state and acts on the estimate. HRL does not need that step, and Petzschner et al. draw the consequence explicitly.
Reinforcement learning “could also be performed in the absence of an explicit representation of internal states,” via direct connections from state-signalling sensors to reward circuitry. The candidate wiring is concrete: orexin/hypocretin neurons projecting from lateral hypothalamus to VTA (Burdakov 2019); receptors for ghrelin, leptin and insulin in the VTA, letting metabolic state act directly on reward computation (Palmiter 2007); expected drive-reduction effects signalled through the opioid system. On this route, “internal states can be signaled directly and do not need to be inferred, raising the possibility that this type of control could be performed in the absence of an interocept.”
That is a live architectural claim, not a quibble. It says a large fraction of body regulation may proceed with no percept of the body anywhere in the causal chain — and it locates the cases that do need inference (complex, goal-directed, emotional) as the minority extension rather than the paradigm.
The wiki has met this shape of question before from the other side. does-somatic-feedback-guide-decisions asks whether the somatic signal must be felt to bias choice, and body-loop-and-as-if-body-loop records Damasio’s willingness to bypass the actual body with a simulation. HRL is the same question posed by control theory rather than neuropsychology: how much of the body’s influence on behaviour survives if you delete the representation of the body?
Equivalent to active inference — and not
The paper’s most useful single result is a two-part verdict:
At the computational level, they are the same. “The drive in HRL can be formally re-expressed as surprise in IAI” (Hulme et al. 2019). Both frameworks then explain the same repertoire beyond reflex: reflexes adjusted to a learned context, responses issued in anticipation of predicted changes, and Pavlovian, habitual and goal-directed homeostatic control. So the free-energy account’s distinctive claim is not, at this level, distinctive — an independent tradition reaches it in different notation.
At the implementational level, they differ testably.
| interoceptive active-inference | HRL | |
|---|---|---|
| representation | predictions and prediction errors explicitly encoded in neuronal populations | not required; states can drive value directly |
| anatomy | visceromotor hierarchy (subgenual, ACC/MCC, insula, OFC) → hypothalamus, PAG, parabrachial | hypothalamic/midbrain reward circuitry (orexin LH→VTA, VTA metabolic receptors, opioid system) |
| multiple controllers | integrated as layers of one hierarchy | separate controllers competing or collaborating |
| does regulation need a percept of the body? | yes | not necessarily |
The paper does not adjudicate. Its position is that the computational level has proved unable to arbitrate and the implementational differences are “the basis for future research programs” — which is the correct conclusion and also an admission of how much of this remains schematic.
Status here
Held at one remove: Keramati & Gutkin (2014), Keramati et al. (2017, cocaine addiction as an HRL disorder) and Hulme et al. (2019) are all cited by Petzschner et al. and none are in raw/. What the wiki can say first-hand is that a serious non-Friston formalization of body regulation exists, that it is computationally equivalent to the one the wiki has been treating as canonical, and that it makes the representation of bodily state optional in a way the free-energy account cannot.