Hfrxdjt

What is the term when two people sing in harmony, but they aren't singing the same notes?

How will losing mobility of one hand affect my career as a programmer?

Can a malicious addon access internet history and such in chrome/firefox?

Left multiplication is homeomorphism of topological groups

Perfect riffle shuffles

How to interpret the phrase "t’en a fait voir à toi"?

Teaching indefinite integrals that require special-casing

How to prevent YouTube from showing already watched videos?

Identify a stage play about a VR experience in which participants are encouraged to simulate performing horrific activities

word describing multiple paths to the same abstract outcome

Java - What do constructor type arguments mean when placed *before* the type?

The One-Electron Universe postulate is true - what simple change can I make to change the whole universe?

Latex for-and in equation

How to deal with or prevent idle in the test team?

What does the "3am" section means in manpages?

Bob has never been a M before

Is it okay / does it make sense for another player to join a running game of Munchkin?

Would it be legal for a US State to ban exports of a natural resource?

Calculating the number of days between 2 dates in Excel

Have I saved too much for retirement so far?

Fast sudoku solver

Freedom of speech and where it applies

How can I successfully establish a nationwide combat training program for a large country?

I2C signal and power over long range (10meter cable)

Improving pandas speed for deriving data from other datasets

0

$begingroup$

I don't want to filter the set I'm starting with or perform mathematical operations on it. Instead, I want to count the number of entries which appear in another set based on the observations here:

Imagine I have a set like this:

year family count
1 A 5
2 B 7

Which continues for some 1000 entries.

And then I also have a separate data set which lists every single element, pretend here it's some kind of firm:

firm_id family year_started year_ended
1234 A 1234 1942
4567 B 1836 2011
...

And that continues for almost half a million entries. How do I most efficiently count the number of entries in the second data set given matches by family and years between year_started and year_ended.

Right now, I'm using an apply function:

def get_count(year, family, a=attributes):
 a = a[a['family'].str.startswith(naics_prefix)]
 a = a[a['year_started'].apply(lambda d: d.year) <= year]
 a = a[a['year_ended'].apply(lambda d: d.year) >= year] # can be 9999-12-31, so must be python date not pandas dt
 return len(a)

Invoked by

counts.progress_apply(lambda r: get_count(r['ending_year'], r['family']),
 axis=1)

Which understandably takes forever.

pandas

share|improve this question

asked 1 hour ago

ifly6ifly6

1062

$endgroup$

add a comment | 

0

$begingroup$

I don't want to filter the set I'm starting with or perform mathematical operations on it. Instead, I want to count the number of entries which appear in another set based on the observations here:

Imagine I have a set like this:

year family count
1 A 5
2 B 7

Which continues for some 1000 entries.

And then I also have a separate data set which lists every single element, pretend here it's some kind of firm:

firm_id family year_started year_ended
1234 A 1234 1942
4567 B 1836 2011
...

And that continues for almost half a million entries. How do I most efficiently count the number of entries in the second data set given matches by family and years between year_started and year_ended.

Right now, I'm using an apply function:

def get_count(year, family, a=attributes):
 a = a[a['family'].str.startswith(naics_prefix)]
 a = a[a['year_started'].apply(lambda d: d.year) <= year]
 a = a[a['year_ended'].apply(lambda d: d.year) >= year] # can be 9999-12-31, so must be python date not pandas dt
 return len(a)

Invoked by

counts.progress_apply(lambda r: get_count(r['ending_year'], r['family']),
 axis=1)

Which understandably takes forever.

pandas

share|improve this question

asked 1 hour ago

ifly6ifly6

1062

$endgroup$

add a comment | 

$begingroup$

I don't want to filter the set I'm starting with or perform mathematical operations on it. Instead, I want to count the number of entries which appear in another set based on the observations here:

Imagine I have a set like this:

year family count
1 A 5
2 B 7

Which continues for some 1000 entries.

And then I also have a separate data set which lists every single element, pretend here it's some kind of firm:

firm_id family year_started year_ended
1234 A 1234 1942
4567 B 1836 2011
...

And that continues for almost half a million entries. How do I most efficiently count the number of entries in the second data set given matches by family and years between year_started and year_ended.

Right now, I'm using an apply function:

def get_count(year, family, a=attributes):
 a = a[a['family'].str.startswith(naics_prefix)]
 a = a[a['year_started'].apply(lambda d: d.year) <= year]
 a = a[a['year_ended'].apply(lambda d: d.year) >= year] # can be 9999-12-31, so must be python date not pandas dt
 return len(a)

Invoked by

counts.progress_apply(lambda r: get_count(r['ending_year'], r['family']),
 axis=1)

Which understandably takes forever.

pandas

share|improve this question

asked 1 hour ago

ifly6ifly6

1062

$endgroup$

year family count
1 A 5
2 B 7

firm_id family year_started year_ended
1234 A 1234 1942
4567 B 1836 2011
...

def get_count(year, family, a=attributes):
 a = a[a['family'].str.startswith(naics_prefix)]
 a = a[a['year_started'].apply(lambda d: d.year) <= year]
 a = a[a['year_ended'].apply(lambda d: d.year) >= year] # can be 9999-12-31, so must be python date not pandas dt
 return len(a)

counts.progress_apply(lambda r: get_count(r['ending_year'], r['family']),
 axis=1)

share|improve this question

asked 1 hour ago

ifly6ifly6

1062

share|improve this question

asked 1 hour ago

ifly6ifly6

1062

asked 1 hour ago

ifly6ifly6

1062

asked 1 hour ago

ifly6ifly6

1062