Hfrxdjt

Suppose that we perform density estimation in m-dimensional space: we estimate the value $p(a)$ for some point $a$ given observations $x_1, dots, x_n $.

It is known that if region $A subset mathbbR^m$ is "small" enough to consider density being constant on points from $A$ then we can make the following estimate:
$$ p(a) approx frack / nA $$
where $k$ is the number of observations that lie in $A$ and $|A|$ is Lebesgue measure of $A$.

Let parameter $h$ be small enough to consider density as constant inside hypercube centered at $a$ with side length equal to $h$. The volume of this hypercube is equal to $h^m$ and point $x$ lies inside this hypercube iff $K(fracx-ah) = 1$ where
$$K(u) =cases
1textfracu^k - a^khcr
0text, otherwise
$$
It's easy to see that the number of observations inside this hypercube equals to
$$k = sum_i = 1^n K(fracx-ah)$$
and so the estimation described above gets the following form:
$$p(a) approx frac1n h^m sum_i = 1^n K(fracx-ah) $$

We can interpret $K$ as "weight" given to particular observations and one of the drawbacks of hypercubic approach is that all observations lying inside hypercube have equal weights despite having different distances from $a$. Yet another drawback is that the resulting estimate is not continuous. That's what i understand to be the main reason of using non-hypercubic kernels such as gaussian kernel which give more weight to points close to $a$ and yields continuous estimate.

But i have troubles with interpreting the usage of such kernels. The sum $sum_i = 1^n K(fracx-ah)$ is no longer equal to $k$ so we can't justify the usage of these kernels by formula $p(a) approx frack / nA $. Finally here are my questions: how do we justify the usage of smooth kernels? how can one interpret this usage?

Thank you for any ideas.

Histograms and methods based on binning have a number of well-known problems. Different anchor points etc. can introduce artificial patterns that make interpretation unreliable. Smooth kernels don't use a grid and thus smooth out the noise.

This also has the advantage that it makes it easier to get a single overall picture of the data because it takes into account neighboring points and smooths the data into areas where no data is observed.

Smooth kernels can also be justified by their favorable statistical properties. Popular methods like fastKDE use the fact that one can find "an empirical kernel that is optimal in the sense that the integrated, squared difference between the resulting KDE and the true PDF is minimized."

If we're estimating a continious distribution's density, perhaps we should introduce an integral in here right? A kernel estimate should be such that $int_-infty^inftyK(x)dx = 1$. Therefore, it should be relatively easy to see that an estimate for $f(x)$ called $hatf(x)$ should have the following:

$int_-infty^inftyhatf(x)dx = frac1nsum_j=1^nfrac1hK(fracx-ah) $
$= frac1nsum_j=1^n1 = 1$. Naturally since, the kernal and the estimate for the pdf are greater than 1, then our hat function is also a probability density function.

Now for a bit more detail: $hatf(x)$ is usually derived from a definition of the derivative of the emperical CDF. So instead of justifying it via the way you would a parzen window, you instead just justify it from what it means to be a pdf and what you want a good estimate for that pdf to be.

edit: With regards to knn and your estimator. I think it's also important to realize that the for any fixed point the nearest neighhor estiamte is the kernel estimate. However, it is different estimate for each point. The kernel still remains an estimate because each individual estimate is a density so overall the kernel is a linear combination of densities. Furthermore the coefficients for the k estimates will sum up to 1.