Hfrxdjt

What features enable the Su-25 Frogfoot to operate with such a wide variety of fuels?

Does an advisor owe his/her student anything? Will an advisor keep a PhD student only out of pity?

How do you respond to a colleague from another team when they're wrongly expecting that you'll help them?

Why is so much work done on numerical verification of the Riemann Hypothesis?

Terse Method to Swap Lowest for Highest?

What are the balance implications behind making invisible things auto-hide?

How do I delete all blank lines in a buffer?

This is why we puzzle

Can I still be respawned if I die by falling off the map?

Why does the Sun have different day lengths, but not the gas giants?

Is this toilet slogan correct usage of the English language?

Why is the "ls" command showing permissions of files in a FAT32 partition?

Does Doodling or Improvising on the Piano Have Any Benefits?

Can a stoichiometric mixture of oxygen and methane exist as a liquid at standard pressure and some (low) temperature?

Strong empirical falsification of quantum mechanics based on vacuum energy density?

Does IPv6 have similar concept of network mask?

PTIJ: Haman's bad computer

How can mimic phobia be cured?

Quasinilpotent , non-compact operators

Probability that THHT occurs in a sequence of 10 coin tosses

What should you do if you miss a job interview (deliberately)?

Why is this estimator biased?

creating a ":KeepCursor" command

Redundant comparison & "if" before assignment

Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

Preprocessing in Data mining?Big Data and a fwf files reading with filtersTools to perform SQL analytics on 350TB of csv dataHow do I infer user activities from historical location data and other relevant data?Tools for ML on csv files and jsonsTransformation from Datawarehouse into Big Data structurePreprocessing to-be-predicted data in ML with R - “learn” and “apply” featuresWhat are recommended waystools for processing large data from Excel Files?Plot RDD data using a pyspark dataframe from csv fileLoading own train data and labels in dataloader using pytorch?

-1

$begingroup$

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2) Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers.
Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!

bigdata preprocessing

share|improve this question

edited 27 mins ago

Gabriel

asked Jan 11 at 22:16

GabrielGabriel

1065

$endgroup$

add a comment | 

-1

$begingroup$

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2) Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers.
Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!

bigdata preprocessing

share|improve this question

edited 27 mins ago

Gabriel

asked Jan 11 at 22:16

GabrielGabriel

1065

$endgroup$

add a comment | 

$begingroup$

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2) Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers.
Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!

bigdata preprocessing

share|improve this question

edited 27 mins ago

Gabriel

asked Jan 11 at 22:16

GabrielGabriel

1065

$endgroup$

share|improve this question

edited 27 mins ago

Gabriel

asked Jan 11 at 22:16

GabrielGabriel

1065

share|improve this question

edited 27 mins ago

Gabriel

asked Jan 11 at 22:16

GabrielGabriel

1065

asked Jan 11 at 22:16

GabrielGabriel

1065

asked Jan 11 at 22:16

GabrielGabriel

1065

1 Answer
1

active

oldest

votes

0

$begingroup$

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

• Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.


• Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

• Consistency management, as for errors or duplicates.


• Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks

share|improve this answer

answered 28 mins ago

GabrielGabriel

1065

$endgroup$

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f43873%2floading-data-into-dw-direct-insert-from-preprocessing-vs-preprocess-and-then-lo%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

$begingroup$

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

• Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.


• Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

• Consistency management, as for errors or duplicates.


• Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks

share|improve this answer

answered 28 mins ago

GabrielGabriel

1065

$endgroup$

add a comment | 

0

$begingroup$

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

• Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.


• Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

• Consistency management, as for errors or duplicates.


• Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks

share|improve this answer

answered 28 mins ago

GabrielGabriel

1065

$endgroup$

add a comment | 

$begingroup$

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

• Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.


• Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

• Consistency management, as for errors or duplicates.


• Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks

share|improve this answer

answered 28 mins ago

GabrielGabriel

1065

$endgroup$

share|improve this answer

answered 28 mins ago

GabrielGabriel

1065

answered 28 mins ago

GabrielGabriel

1065

answered 28 mins ago

GabrielGabriel

1065