Hfrxdjt

Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace.

In their paper, they explain the loss function and the training procedure of the model, but they don't emphasize much on the architecture of the model.

Does somebody know what the NN behind looks like?

Calling StarSpace a neural model would be misleading I think. You could certainly think of the it as a neural network with a single layer and a linear activation function, but I think don't think that would be very illuminating. They didn't discuss the architecture much in that paper for a reason- there isn't really any in terms of layers of neurons, activation functions, latent variables, or anything else except the constraints on the number of dimensions.

In fact, the most helpful way to think about StarSpace is that, at its core like many (maybe most) popular embedding techniques across natural language, graphs, etc it's a low rank matrix factorization. What the sampling procedures are doing is using the data in some way to produce a positive definite gram matrix. It doesn't appear that way initially because that is done solely through sampling- if you were to find the expectations of each input/target pair, however, you'd find that the optimization objective is maximizing the expected value of vector similarity over a joint distribution on pairs of items minus a marginal distribution (this is due to the negative samples). Essentially, the goal is to maximize the difference in similarity between items that are frequently sampled and the similarity between items sampled independently from the marginal distribution.

If this sounds familiar to SGNS implicitly factorizing a shifted PPMI matrix or GlOVe explicitly factorizing a relaxed variant of the same, good. The specifics are different and StarSpace has significantly more flexibility in the sampling distribution it works with, but the principle is the same. "Neural Word Embedding as Implicit Matrix Factorization" and "Improving Distributional Similarity with Lessons Learned from Word Embeddings" are fantastic papers by Levy, 2014 and 2015 if I recall correctly, that discuss the connections between neural embeddings and explicit matrix factorization techniques like PPMI-SVD and glove and the principles that make them successful.

Similarly, "Network Embedding as Matrix Factorization: Unifying
DeepWalk, LINE, PTE, and node2vec" is a great discussion of the connections between neural network embeddings and the same implicit objectives as the neural word embeddings.

In short- it doesn't sound like there's much going on in StarSpace as far as architecture because there isn't. It's quite literally adjusting the placement of points in the embedding space to make associated items more similar to each other than to unrelated items.