Hfrxdjt

Sci-Fi book where patients in a coma ward all live in a subconscious world linked together

Seeking colloquialism for “just because”

Generate an RGB colour grid

Storing hydrofluoric acid before the invention of plastics

Output the ŋarâþ crîþ alphabet song without using (m)any letters

Align equal signs while including text over equalities

Why did the rest of the Eastern Bloc not invade Yugoslavia?

How to deal with a team lead who never gives me credit?

Why was the term "discrete" used in discrete logarithm?

2001: A Space Odyssey's use of the song "Daisy Bell" (Bicycle Built for Two); life imitates art or vice-versa?

Resolving to minmaj7

Short Story with Cinderella as a Voo-doo Witch

What's the meaning of 間時肆拾貳 at a car parking sign

Single word antonym of "flightless"

Can a USB port passively 'listen only'?

51k Euros annually for a family of 4 in Berlin: Is it enough?

How to find all the available tools in macOS terminal?

Why is my conclusion inconsistent with the van't Hoff equation?

What causes the vertical darker bands in my photo?

Apollo command module space walk?

The logistics of corpse disposal

How do I stop a creek from eroding my steep embankment?

Coloring maths inside a tcolorbox

Dating a Former Employee

How to train an existing word2vec gensim model on new words?

0

$begingroup$

According to gensim docs, you can take an existing word2vec model and further train it on new words.

The training is streamed, meaning sentences can be a generator,
reading input data from disk on-the-fly, without loading the entire
corpus into RAM.

It also means you can continue training the model later:

model = Word2Vec.load("word2vec.model")
model.train([["hello", "world"]], total_examples=1, epochs=1)

Source: docs

But when I actually try it, it doesn't seem to learn the new terms.

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

After adding new terms in the 2nd code block, the model still only has the same number of terms as before.

How can I make this work?

python word2vec gensim

share|improve this question

asked 14 mins ago

tim_xyztim_xyz

717

$endgroup$

add a comment | 

0

$begingroup$

According to gensim docs, you can take an existing word2vec model and further train it on new words.

The training is streamed, meaning sentences can be a generator,
reading input data from disk on-the-fly, without loading the entire
corpus into RAM.

It also means you can continue training the model later:

model = Word2Vec.load("word2vec.model")
model.train([["hello", "world"]], total_examples=1, epochs=1)

Source: docs

But when I actually try it, it doesn't seem to learn the new terms.

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

After adding new terms in the 2nd code block, the model still only has the same number of terms as before.

How can I make this work?

python word2vec gensim

share|improve this question

asked 14 mins ago

tim_xyztim_xyz

717

$endgroup$

add a comment | 

$begingroup$

According to gensim docs, you can take an existing word2vec model and further train it on new words.

The training is streamed, meaning sentences can be a generator,
reading input data from disk on-the-fly, without loading the entire
corpus into RAM.

It also means you can continue training the model later:

model = Word2Vec.load("word2vec.model")
model.train([["hello", "world"]], total_examples=1, epochs=1)

Source: docs

But when I actually try it, it doesn't seem to learn the new terms.

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

After adding new terms in the 2nd code block, the model still only has the same number of terms as before.

How can I make this work?

python word2vec gensim

share|improve this question

asked 14 mins ago

tim_xyztim_xyz

717

$endgroup$

from gensim.models import Word2Vec

# initial a model
model = Word2Vec([["cat", "say", "meow"], ["dog", "say", "woof"]], min_count=1)

# count terms in model
print( len(model.wv.vocab) )

=> 5

# train existing model on new terms
model.train([['potoatoes', 'and', 'farmers']], total_examples=model.corpus_count, epochs=model.epochs)

# count terms in model
print( len(model.wv.vocab) )

=> 5

share|improve this question

asked 14 mins ago

tim_xyztim_xyz

717

share|improve this question

asked 14 mins ago

tim_xyztim_xyz

717

asked 14 mins ago

tim_xyztim_xyz

717

asked 14 mins ago

tim_xyztim_xyz

717