Comparison between addition and multiplication function in deep neural network?2019 Community Moderator ElectionHOW TO: Deep Neural Network weight initializationDepth of the first pooling layer outcome in tensorflow documentationAdding hand-crafted features to a convolutional neural network (CNN) in TensorFlowAmount of multiplications in a neural network modelDo Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels?How to input & pre-process images for a Deep Convolutional Neural Network?Error in Neural NetworkHow can I combine images for Matlab deep learning?Numpy Python deep learning frameworkWhat is the motivation for row-wise convolution and folding in Kalchbrenner et al. (2014)?
Describing a person. What needs to be mentioned?
Class Action - which options I have?
How does it work when somebody invests in my business?
Is there a problem with hiding "forgot password" until it's needed?
Was Spock the First Vulcan in Starfleet?
Term for the "extreme-extension" version of a straw man fallacy?
Crossing the line between justified force and brutality
Why escape if the_content isnt?
Efficient way to transport a Stargate
How to pronounce the slash sign
Large drywall patch supports
Method to test if a number is a perfect power?
How to write papers efficiently when English isn't my first language?
Why didn't Theresa May consult with Parliament before negotiating a deal with the EU?
Implement the Thanos sorting algorithm
What is the difference between "behavior" and "behaviour"?
Sitecore Powershell - Adding new version for each item
Is expanding the research of a group into machine learning as a PhD student risky?
Energy of the particles in the particle accelerator
Can the discrete variable be a negative number?
Failed to fetch jessie backports repository
What is the best translation for "slot" in the context of multiplayer video games?
Pole-zeros of a real-valued causal FIR system
How to be diplomatic in refusing to write code that breaches the privacy of our users
Comparison between addition and multiplication function in deep neural network?
2019 Community Moderator ElectionHOW TO: Deep Neural Network weight initializationDepth of the first pooling layer outcome in tensorflow documentationAdding hand-crafted features to a convolutional neural network (CNN) in TensorFlowAmount of multiplications in a neural network modelDo Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels?How to input & pre-process images for a Deep Convolutional Neural Network?Error in Neural NetworkHow can I combine images for Matlab deep learning?Numpy Python deep learning frameworkWhat is the motivation for row-wise convolution and folding in Kalchbrenner et al. (2014)?
$begingroup$
I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.
The second image represents the pyramid pooling module which includes several scale images.
I am looking forward to the addition and multiplication function's attribute in a deep neural network.
The question is:
Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?
neural-network deep-learning convolution math image-preprocessing
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
$endgroup$
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 1 more comment
$begingroup$
I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.
The second image represents the pyramid pooling module which includes several scale images.
I am looking forward to the addition and multiplication function's attribute in a deep neural network.
The question is:
Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?
neural-network deep-learning convolution math image-preprocessing
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
$endgroup$
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
1
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36
|
show 1 more comment
$begingroup$
I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.
The second image represents the pyramid pooling module which includes several scale images.
I am looking forward to the addition and multiplication function's attribute in a deep neural network.
The question is:
Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?
neural-network deep-learning convolution math image-preprocessing
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
$endgroup$
I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.
The second image represents the pyramid pooling module which includes several scale images.
I am looking forward to the addition and multiplication function's attribute in a deep neural network.
The question is:
Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?
neural-network deep-learning convolution math image-preprocessing
neural-network deep-learning convolution math image-preprocessing
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
edited Feb 26 at 7:38
amir Maleki
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
asked Feb 16 at 9:24
amir Malekiamir Maleki
112
112
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
1
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36
|
show 1 more comment
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
1
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
1
1
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36
|
show 1 more comment
1 Answer
1
active
oldest
votes
$begingroup$
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
$endgroup$
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45678%2fcomparison-between-addition-and-multiplication-function-in-deep-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
$endgroup$
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
add a comment |
$begingroup$
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
$endgroup$
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
add a comment |
$begingroup$
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
$endgroup$
The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
answered Feb 19 at 7:39
Andreas LookAndreas Look
431110
431110
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
add a comment |
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
$begingroup$
dear @andreas-look, I read the link but I do not understand why you consider concatenation and addition equal. except for this, your answer is acceptable, and the link, which you sent, work for me. Could you explain or edit that part of the answer which I can accept it here.
$endgroup$
– amir Maleki
Feb 26 at 7:43
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45678%2fcomparison-between-addition-and-multiplication-function-in-deep-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Please ask only one question per post. Also, i'ts unclear what you mean by "the most important features of addition vs multiplication", and it's not clear how you are using addition or multiplication, so I don't think any of these questions are answerable in their current form. If you can edit your question to address this feedback, I encourage you to do so.
$endgroup$
– D.W.
Feb 16 at 21:46
$begingroup$
I don't quite understand this, but typically you're using dense layers for non-linear transformations. If all it does is sum combinations of inputs, it's a linear transformation. That almost surely defeats the purpose of what you're using it for.
$endgroup$
– Sean Owen♦
Feb 17 at 0:56
$begingroup$
dear @SeanOwen, I explained that in a specific part of the network there are two tensors which have to unify in order to feed into the next layer, in this case, there are several choices. one of these choices is basic mathematics operation such as addition, multiplication. we performed several experiments with each of these operations. I change the question and make it narrow. could u look at it again?
$endgroup$
– amir Maleki
Feb 17 at 8:14
$begingroup$
Dear @D.W., I change the question, I used basic addition which is provided by Tensorflow, tf.math.add which returns the a+b element-wise (each of a and b is equal tensors) and for multiplication I also used the tf.math.multiply function.
$endgroup$
– amir Maleki
Feb 17 at 8:26
1
$begingroup$
What do you mean 'unify'? there is no general answer to this. Which operation you use depends on what you are trying to do, and, practically, which one works better. If you mean to add things, you add them.
$endgroup$
– Sean Owen♦
Feb 17 at 15:36