Gfyuki

I was trying to understand asymptotic normality of the posterior better, and came across a confusing point. So let's say we have a likelihood, $L(theta | X) = Pi_{i=1}^n p(X_i | theta)$, so the log-likelihood is $J(theta) = log L = Sigma_{i=1}^n log(p(X_i | theta))$.

J is itself a sum of random variables, so the log-likelihood J will be asymptotically normal, by the central limit theorem.

But we can also show the likelihood is asymptotically normal through a Taylor expansion. Let $hat{theta}$ be the mle. So we have

$J(theta) = J(hat{theta}) + nabla J cdot (theta-hat{theta}) + frac{1}{2}(theta-hat{theta})H(theta-hat{theta})$. Since $hat{theta}$ is the mle, we know $nabla J = 0$, and $I(theta)=-H$ so this reduces to

(1) $J(theta) = log(L) = J(hat{theta}) - frac{1}{2}(theta-hat{theta})I(theta)(theta-hat{theta})$

Now exponentiating (1), we get

$e^J = L = ke^{-frac{1}{2}(theta-hat{theta})I(theta)(theta-hat{theta})}$, which is also asymptotically normal, with L ~ $N(hat{theta},I(theta)^{-1})$.

Am I making a mistake here...?

I think you just have to be precise about what you mean by "asymptotically normal." For example, when people say that "a sum of random variables is asymptotically normal by the central limit theorem," they usually really mean a precise statement about convergence in distribution, e.g.,

Central Limit Theorem (Lindeberg–Lévy version).
Suppose $(X_n)_{n=1}^infty$ is a sequence of i.i.d. random variables with mean $mu$ and variance $sigma^2 < infty$. Let $S_n = n^{-1}(X_1 + cdots + X_n)$ (the $n$th sample mean). Then
$$
sqrt{n} (S_n - mu) Rightarrow N(0, sigma^2)
$$
as $n to infty$ (here $Rightarrow$ denotes convergence in distribution).

This doesn't say that $S_n Rightarrow N(mu, sigma^2/n)$ as $n to infty$, which is formally impossible because the expression on the right-hand side involves $n$, but it is often informally stated as $S_n approx N(mu, sigma^2/n)$ for large $n$ (the symbol $approx$ should be read "is approximately distributed as").

In your case, you have a sequence $(L_n)_{n=1}^infty$ of log-likelihoods that, after appropriate standardization, become a sequence $(S_n)_{n=1}^infty$ that satisfies
$$
sqrt{n}(S_n - theta) Rightarrow N(0, sigma^2)
$$
as $n to infty$ (for some $theta$ and $sigma^2$). Now you can recall the delta method:

Delta Method.
Suppose $(S_n)_{n=1}^infty$ is a sequence of random variables satisfying
$$
sqrt{n} (S_n - theta) Rightarrow N(0, sigma^2)
$$
as $n to infty$ for some constants $theta$ and $sigma^2$.
Let $g : mathbb{R} to mathbb{R}$ be a function such that $g^prime(theta)$ exists and is nonzero.
Then
$$
sqrt{n}(g(S_n) - g(theta)) Rightarrow N(0, sigma^2 left(g^prime(theta)right)^2)
$$
as $n to infty$.

The hand-wavey interpretastion of this is that if
$$
S_n approx N(theta, sigma^2 / n)
$$
for large $n$, then
$$
g(S_n) approx N(g(theta), sigma^2left(g^prime(theta)right)^2/n)
$$
for large $n$ (provided that $g^prime(theta)$ exists and is nonzero).

In particular, it shouldn't be surprising that the sequences $(S_n)_{n=1}^infty$ and $(exp(S_n))_{n=1}^infty$ are simultaneously "asymptotically normal."