Hfrxdjt

I have an X matrix, a y variable, and another variable ORTHO_VAR. I need to predict the y variable using X, however, the predictions from that model need to be orthogonal to ORTHO_VAR while being as correlated with y as possible.

I would prefer that the predictions are generated with a non-parametric method such as xgboost.XGBRegressor but I could use a linear method if absolutely necessary.

This code:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
IND_VARNM = 'indep_var'
TARGET = 'target'
CORRECTED_VARNM = 'indep_var_fixed'

# Create regression dataset with two correlated targets
X, y = make_regression(n_features=20, random_state=245, n_targets=2)
indep_vars = ['var'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df['yhat'] = xgb.predict(df[indep_vars])

# Correlation should be low or preferably zero
pred_corr_w_ortho = df.corr().abs()['yhat']['ortho_var']
assert pred_corr_w_ortho < 0.01, pred_corr_w_ortho

Returns this:

---------------------------------------------------------------------------
AssertionError 
 1 pred_corr_w_ortho = df.corr().abs()['yhat']['ortho_var']
----> 2 assert pred_corr_w_ortho < 0.05, pred_corr_w_ortho

AssertionError: 0.5895885756753665

...and I would like something that maintains as much predictive accuracy as possible while remaining orthogonal to ORTHO_VAR

This requirement can be satisfied by adding sufficient noise to predictions $haty$ to decorrelate them from orthogonal values $v$. Ideally, if $haty$ is already decorrelated from $v$, no noise would be added to $haty$, thus $haty$ would be maximally correlated with $y$.

Mathematically, we want to create $haty'=haty+epsilon$ from $epsilon sim mathcalN(0, sigma_epsilon)$, to satisfy $$r_haty'v = fracsigma_haty'vsigma_haty'sigma_v < delta$$ for arbitrary threshold $delta$. Now, lets expand this inequality to find a lower-bound for std of noise $epsilon$, i.e. $sigma_epsilon$.
$$beginalign*
sigma_haty'^2&=sigma_haty^2 + sigma_epsilon^2,\
sigma_haty'v&=Bbb Eleft[(haty+epsilon - mu_haty - overbracemu_epsilon^=0)(v-mu_v)right]\
&=Bbb Eleft[(haty - mu_haty)(v-mu_v)right]+overbraceBbb Eleft[epsilon(v-mu_v)right]^=0&\
&=sigma_hatyv,\
r_haty'v &= fracsigma_haty'vsigma_haty'sigma_v =fracsigma_hatyvsigma_v sqrtsigma_haty^2+sigma_epsilon^2 < delta\
&Rightarrow sigma_hatysqrtleft(fracr_hatyvdeltaright)^2 - 1 < sigma_epsilon
endalign*$$

Since all the variables in the left side of inequality can be calculated, we can sample noises from $mathcalN(0, sigma_epsilon)$ and add them to $haty$ to satisfy the original inequality.

Here is a code that does the exact same thing:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
IND_VARNM = 'indep_var'
TARGET = 'target'
CORRECTED_VARNM = 'indep_var_fixed'

seed = 245
# Create regression dataset with two correlated targets
X, y = make_regression(n_samples=10000, n_features=20, random_state=seed, n_targets=2)
indep_vars = ['var'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df['yhat'] = xgb.predict(df[indep_vars])

delta = 0.01

# std of noise required to be added to y_hat to bring the correlation
# of y_hat with ORTHO_VAR below delta
std_y_hat = np.std(df['yhat'], ddof=1)
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
std_noise_lower_bound = std_y_hat * np.sqrt((corr_y_hat_ortho_var / delta)**2 - 1.0)
std_noise = max(0, std_noise_lower_bound) + 1
print('delta: ', delta)
print('std_y_hat: ', std_y_hat)
print('corr_y_hat_target: ', corr_y_hat_target)
print('corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
print('std_noise_lower_bound: ', std_noise_lower_bound)
print('std_noise: ', std_noise)

# add noise
np.random.seed(seed)
noises = np.random.normal(0, std_noise, len(df['yhat']))
noises -= np.mean(noises) # remove slight deviations from zero mean
print('noise_samples: mean:', np.mean(noises), ', std: ', np.std(noises))
df['yhat'] = df['yhat'] + noises

# measure new correlation
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
print('new corr_y_hat_target: ', corr_y_hat_target)
print('new corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
# Correlation should be low or preferably zero
assert corr_y_hat_ortho_var < delta, corr_y_hat_ortho_var
assert -delta < corr_y_hat_ortho_var, corr_y_hat_ortho_var

which outputs:

delta: 0.01
std_y_hat: 69.48568725585938
corr_y_hat_target: 0.8207672834857673
corr_y_hat_ortho_var: 0.7663936356880843
std_noise_lower_bound: 5324.885500165032
std_noise: 5325.885500165032
noise_samples: mean: 1.1059455573558807e-13 , std: 5373.914830034988
new corr_y_hat_target: -0.004125016071865934
new corr_y_hat_ortho_var: -0.000541131379457552

You can experiment with other deltas. By comparing std_y_hat with std_noise_lower_bound, you can see that a huge noise must be added to $haty$ to decorrelate it from $v$ bellow $0.01$, which dramatically decolerates $haty$ from $y$ too.

Note: Assertion might fail for too small thresholds $delta$ due to insufficient sample count.