Watching something be written to a file live with tailRedirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients

Why does Arabsat 6A need a Falcon Heavy to launch

Emailing HOD to enhance faculty application

How can saying a song's name be a copyright violation?

Is it inappropriate for a student to attend their mentor's dissertation defense?

What's the difference between 'rename' and 'mv'?

Is it possible to create light that imparts a greater proportion of its energy as momentum rather than heat?

Will google still index a page if I use a $_SESSION variable?

Is it unprofessional to ask if a job posting on GlassDoor is real?

Did converts (ger tzedek) in ancient Israel own land?

RG-213 Cable with electric strained wire as metallic shield of Coaxial cable

How could indestructible materials be used in power generation?

SSH "lag" in LAN on some machines, mixed distros

How to take photos in burst mode, without vibration?

Etiquette around loan refinance - decision is going to cost first broker a lot of money

Twin primes whose sum is a cube

Anagram holiday

Did Shadowfax go to Valinor?

What is the word for reserving something for yourself before others do?

Is it legal for company to use my work email to pretend I still work there?

UK: Is there precedent for the governments e-petition site changing the direction of a government decision?

Is the Joker left-handed?

Is there a hemisphere-neutral way of specifying a season?

A reference to a well-known characterization of scattered compact spaces

Neighboring nodes in the network



Watching something be written to a file live with tail


Redirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








21















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 9





    How about python myprog.py | tee output.txt instead?

    – n8te
    yesterday






  • 3





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    22 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    6 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    5 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    5 hours ago

















21















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 9





    How about python myprog.py | tee output.txt instead?

    – n8te
    yesterday






  • 3





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    22 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    6 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    5 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    5 hours ago













21












21








21


1






I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question
















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?







linux command-line redirection stdout






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 5 hours ago









Peter Cordes

2,4751621




2,4751621










asked yesterday









interstarinterstar

415413




415413







  • 9





    How about python myprog.py | tee output.txt instead?

    – n8te
    yesterday






  • 3





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    22 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    6 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    5 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    5 hours ago












  • 9





    How about python myprog.py | tee output.txt instead?

    – n8te
    yesterday






  • 3





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    22 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    6 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    5 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    5 hours ago







9




9





How about python myprog.py | tee output.txt instead?

– n8te
yesterday





How about python myprog.py | tee output.txt instead?

– n8te
yesterday




3




3





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
22 hours ago





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
22 hours ago




1




1





stdbuf can be used to alter the buffering status of file descriptors.

– studog
6 hours ago





stdbuf can be used to alter the buffering status of file descriptors.

– studog
6 hours ago













Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
5 hours ago






Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
5 hours ago














Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
5 hours ago





Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
5 hours ago










5 Answers
5






active

oldest

votes


















29














You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



import sys
...
print('Some message')
sys.stdout.flush()





share|improve this answer










New contributor




Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.















  • 4





    If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

    – mckenzm
    20 hours ago






  • 2





    You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

    – Dan
    12 hours ago







  • 4





    It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

    – glglgl
    11 hours ago


















18














Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



From man tee:




tee(1) - Linux man page



Name tee - read from standard input and write to standard output and files



Synopsis



tee [OPTION]... [FILE]...


Description



Copy standard input to each FILE, and also to standard output.



-a, --append 
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit


If a FILE is -, copy again to standard output.




So in your case you'd run:



python myprog.py | tee output.txt


EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






share|improve this answer

























  • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

    – Baldrickk
    16 hours ago






  • 7





    That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

    – eckes
    15 hours ago






  • 3





    This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

    – Barmar
    6 hours ago


















7














Run python with the unbuffered flag:



python -u myprog.py > output.txt


Output will then print in real time.






share|improve this answer








New contributor




BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



























    3














    Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



    This is a redirect to a regular file.



    C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



    You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



    On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



    Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




    Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






    share|improve this answer






























      0














      Another option (if you don't care about the contents, just the progress) is pv:



      NAME
      pv - monitor the progress of data through a pipe

      SYNOPSIS
      pv [OPTION] [FILE]...
      pv [-h|-V]


      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






      share|improve this answer


















      • 1





        There's no pipeline, just file redirection. And it won't solve the buffering problem.

        – Barmar
        6 hours ago











      Your Answer








      StackExchange.ready(function()
      var channelOptions =
      tags: "".split(" "),
      id: "3"
      ;
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function()
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled)
      StackExchange.using("snippets", function()
      createEditor();
      );

      else
      createEditor();

      );

      function createEditor()
      StackExchange.prepareEditor(
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader:
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      ,
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      );



      );













      draft saved

      draft discarded


















      StackExchange.ready(
      function ()
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

      );

      Post as a guest















      Required, but never shown

























      5 Answers
      5






      active

      oldest

      votes








      5 Answers
      5






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      29














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        20 hours ago






      • 2





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        12 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        11 hours ago















      29














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        20 hours ago






      • 2





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        12 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        11 hours ago













      29












      29








      29







      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.










      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()






      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this answer



      share|improve this answer








      edited 20 hours ago









      user2313067

      2,1001911




      2,1001911






      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      answered yesterday









      DaveyDavey

      32625




      32625




      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.







      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        20 hours ago






      • 2





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        12 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        11 hours ago












      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        20 hours ago






      • 2





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        12 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        11 hours ago







      4




      4





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      20 hours ago





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      20 hours ago




      2




      2





      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      12 hours ago






      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      12 hours ago





      4




      4





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      11 hours ago





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      11 hours ago













      18














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer

























      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        16 hours ago






      • 7





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        15 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        6 hours ago















      18














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer

























      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        16 hours ago






      • 7





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        15 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        6 hours ago













      18












      18








      18







      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer















      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.







      share|improve this answer














      share|improve this answer



      share|improve this answer








      edited 3 hours ago

























      answered yesterday









      n8ten8te

      5,33972235




      5,33972235












      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        16 hours ago






      • 7





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        15 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        6 hours ago

















      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        16 hours ago






      • 7





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        15 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        6 hours ago
















      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      16 hours ago





      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      16 hours ago




      7




      7





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      15 hours ago





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      15 hours ago




      3




      3





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      6 hours ago





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      6 hours ago











      7














      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.






      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.
























        7














        Run python with the unbuffered flag:



        python -u myprog.py > output.txt


        Output will then print in real time.






        share|improve this answer








        New contributor




        BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.






















          7












          7








          7







          Run python with the unbuffered flag:



          python -u myprog.py > output.txt


          Output will then print in real time.






          share|improve this answer








          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.










          Run python with the unbuffered flag:



          python -u myprog.py > output.txt


          Output will then print in real time.







          share|improve this answer








          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          share|improve this answer



          share|improve this answer






          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          answered 8 hours ago









          BHCBHC

          711




          711




          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





          New contributor





          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.






          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





















              3














              Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



              This is a redirect to a regular file.



              C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



              You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



              On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



              Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




              Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






              share|improve this answer



























                3














                Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                This is a redirect to a regular file.



                C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






                share|improve this answer

























                  3












                  3








                  3







                  Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                  This is a redirect to a regular file.



                  C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                  You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                  On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                  Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                  Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






                  share|improve this answer













                  Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                  This is a redirect to a regular file.



                  C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                  You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                  On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                  Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                  Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered 5 hours ago









                  Peter CordesPeter Cordes

                  2,4751621




                  2,4751621





















                      0














                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer


















                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        6 hours ago















                      0














                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer


















                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        6 hours ago













                      0












                      0








                      0







                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer













                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.







                      share|improve this answer












                      share|improve this answer



                      share|improve this answer










                      answered 9 hours ago









                      rrauenzarrauenza

                      1313




                      1313







                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        6 hours ago












                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        6 hours ago







                      1




                      1





                      There's no pipeline, just file redirection. And it won't solve the buffering problem.

                      – Barmar
                      6 hours ago





                      There's no pipeline, just file redirection. And it won't solve the buffering problem.

                      – Barmar
                      6 hours ago

















                      draft saved

                      draft discarded
















































                      Thanks for contributing an answer to Super User!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid


                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.

                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function ()
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      Popular posts from this blog

                      Францішак Багушэвіч Змест Сям'я | Біяграфія | Творчасць | Мова Багушэвіча | Ацэнкі дзейнасці | Цікавыя факты | Спадчына | Выбраная бібліяграфія | Ушанаванне памяці | У філатэліі | Зноскі | Літаратура | Спасылкі | НавігацыяЛяхоўскі У. Рупіўся дзеля Бога і людзей: Жыццёвы шлях Лявона Вітан-Дубейкаўскага // Вольскі і Памідораў з песняй пра немца Адвакат, паэт, народны заступнік Ашмянскі веснікВ Минске появится площадь Богушевича и улица Сырокомли, Белорусская деловая газета, 19 июля 2001 г.Айцец беларускай нацыянальнай ідэі паўстаў у бронзе Сяргей Аляксандравіч Адашкевіч (1918, Мінск). 80-я гады. Бюст «Францішак Багушэвіч».Яўген Мікалаевіч Ціхановіч. «Партрэт Францішка Багушэвіча»Мікола Мікалаевіч Купава. «Партрэт зачынальніка новай беларускай літаратуры Францішка Багушэвіча»Уладзімір Іванавіч Мелехаў. На помніку «Змагарам за родную мову» Барэльеф «Францішак Багушэвіч»Памяць пра Багушэвіча на Віленшчыне Страчаная сталіца. Беларускія шыльды на вуліцах Вільні«Krynica». Ideologia i przywódcy białoruskiego katolicyzmuФранцішак БагушэвічТворы на knihi.comТворы Францішка Багушэвіча на bellib.byСодаль Уладзімір. Францішак Багушэвіч на Лідчыне;Луцкевіч Антон. Жыцьцё і творчасьць Фр. Багушэвіча ў успамінах ягоных сучасьнікаў // Запісы Беларускага Навуковага таварыства. Вільня, 1938. Сшытак 1. С. 16-34.Большая российская1188761710000 0000 5537 633Xn9209310021619551927869394п

                      Беларусь Змест Назва Гісторыя Геаграфія Сімволіка Дзяржаўны лад Палітычныя партыі Міжнароднае становішча і знешняя палітыка Адміністрацыйны падзел Насельніцтва Эканоміка Культура і грамадства Сацыяльная сфера Узброеныя сілы Заўвагі Літаратура Спасылкі НавігацыяHGЯOiТоп-2011 г. (па версіі ej.by)Топ-2013 г. (па версіі ej.by)Топ-2016 г. (па версіі ej.by)Топ-2017 г. (па версіі ej.by)Нацыянальны статыстычны камітэт Рэспублікі БеларусьШчыльнасць насельніцтва па краінахhttp://naviny.by/rubrics/society/2011/09/16/ic_articles_116_175144/А. Калечыц, У. Ксяндзоў. Спробы засялення краю неандэртальскім чалавекам.І ў Менску былі мамантыА. Калечыц, У. Ксяндзоў. Старажытны каменны век (палеаліт). Першапачатковае засяленне тэрыторыіГ. Штыхаў. Балты і славяне ў VI—VIII стст.М. Клімаў. Полацкае княства ў IX—XI стст.Г. Штыхаў, В. Ляўко. Палітычная гісторыя Полацкай зямліГ. Штыхаў. Дзяржаўны лад у землях-княствахГ. Штыхаў. Дзяржаўны лад у землях-княствахБеларускія землі ў складзе Вялікага Княства ЛітоўскагаЛюблінская унія 1569 г."The Early Stages of Independence"Zapomniane prawdy25 гадоў таму было аб'яўлена, што Язэп Пілсудскі — беларус (фота)Наша вадаДакументы ЧАЭС: Забруджванне тэрыторыі Беларусі « ЧАЭС Зона адчужэнняСведения о политических партиях, зарегистрированных в Республике Беларусь // Министерство юстиции Республики БеларусьСтатыстычны бюлетэнь „Полаўзроставая структура насельніцтва Рэспублікі Беларусь на 1 студзеня 2012 года і сярэднегадовая колькасць насельніцтва за 2011 год“Индекс человеческого развития Беларуси — не было бы нижеБеларусь занимает первое место в СНГ по индексу развития с учетом гендерного факцёраНацыянальны статыстычны камітэт Рэспублікі БеларусьКанстытуцыя РБ. Артыкул 17Трансфармацыйныя задачы БеларусіВыйсце з крызісу — далейшае рэфармаванне Беларускі рубель — сусветны лідар па дэвальвацыяхПра змену коштаў у кастрычніку 2011 г.Бядней за беларусаў у СНД толькі таджыкіСярэдні заробак у верасні дасягнуў 2,26 мільёна рублёўЭканомікаГаласуем за ТОП-100 беларускай прозыСучасныя беларускія мастакіАрхитектура Беларуси BELARUS.BYА. Каханоўскі. Культура Беларусі ўсярэдзіне XVII—XVIII ст.Анталогія беларускай народнай песні, гуказапісы спеваўБеларускія Музычныя IнструментыБеларускі рок, які мы страцілі. Топ-10 гуртоў«Мясцовы час» — нязгаслая легенда беларускай рок-музыкіСЯРГЕЙ БУДКІН. МЫ НЯ ЗНАЕМ СВАЁЙ МУЗЫКІМ. А. Каладзінскі. НАРОДНЫ ТЭАТРМагнацкія культурныя цэнтрыПублічная дыскусія «Беларуская новая пьеса: без беларускай мовы ці беларуская?»Беларускія драматургі па-ранейшаму лепш ставяцца за мяжой, чым на радзіме«Працэс незалежнага кіно пайшоў, і дзяржаву турбуе яго непадкантрольнасць»Беларускія філосафы ў пошуках прасторыВсе идём в библиотекуАрхіваванаАб Нацыянальнай праграме даследавання і выкарыстання касмічнай прасторы ў мірных мэтах на 2008—2012 гадыУ космас — разам.У суседнім з Барысаўскім раёне пабудуюць Камандна-вымяральны пунктСвяты і абрады беларусаў«Мірныя бульбашы з малой краіны» — 5 непраўдзівых стэрэатыпаў пра БеларусьМ. Раманюк. Беларускае народнае адзеннеУ Беларусі скарачаецца колькасць злачынстваўЛукашэнка незадаволены мінскімі ўладамі Крадзяжы складаюць у Мінску каля 70% злачынстваў Узровень злачыннасці ў Мінскай вобласці — адзін з самых высокіх у краіне Генпракуратура аналізуе стан са злачыннасцю ў Беларусі па каэфіцыенце злачыннасці У Беларусі стабілізавалася крымінагеннае становішча, лічыць генпракурорЗамежнікі сталі здзяйсняць у Беларусі больш злачынстваўМУС Беларусі турбуе рост рэцыдыўнай злачыннасціЯ з ЖЭСа. Дазволіце вас абкрасці! Рэйтынг усіх службаў і падраздзяленняў ГУУС Мінгарвыканкама вырасАб КДБ РБГісторыя Аператыўна-аналітычнага цэнтра РБГісторыя ДКФРТаможняagentura.ruБеларусьBelarus.by — Афіцыйны сайт Рэспублікі БеларусьСайт урада БеларусіRadzima.org — Збор архітэктурных помнікаў, гісторыя Беларусі«Глобус Беларуси»Гербы и флаги БеларусиАсаблівасці каменнага веку на БеларусіА. Калечыц, У. Ксяндзоў. Старажытны каменны век (палеаліт). Першапачатковае засяленне тэрыторыіУ. Ксяндзоў. Сярэдні каменны век (мезаліт). Засяленне краю плямёнамі паляўнічых, рыбакоў і збіральнікаўА. Калечыц, М. Чарняўскі. Плямёны на тэрыторыі Беларусі ў новым каменным веку (неаліце)А. Калечыц, У. Ксяндзоў, М. Чарняўскі. Гаспадарчыя заняткі ў каменным векуЭ. Зайкоўскі. Духоўная культура ў каменным векуАсаблівасці бронзавага веку на БеларусіФарміраванне супольнасцей ранняга перыяду бронзавага векуФотографии БеларусиРоля беларускіх зямель ва ўтварэнні і ўмацаванні ВКЛВ. Фадзеева. З гісторыі развіцця беларускай народнай вышыўкіDMOZGran catalanaБольшая российскаяBritannica (анлайн)Швейцарскі гістарычны15325917611952699xDA123282154079143-90000 0001 2171 2080n9112870100577502ge128882171858027501086026362074122714179пппппп

                      ValueError: Expected n_neighbors <= n_samples, but n_samples = 1, n_neighbors = 6 (SMOTE) The 2019 Stack Overflow Developer Survey Results Are InCan SMOTE be applied over sequence of words (sentences)?ValueError when doing validation with random forestsSMOTE and multi class oversamplingLogic behind SMOTE-NC?ValueError: Error when checking target: expected dense_1 to have shape (7,) but got array with shape (1,)SmoteBoost: Should SMOTE be ran individually for each iteration/tree in the boosting?solving multi-class imbalance classification using smote and OSSUsing SMOTE for Synthetic Data generation to improve performance on unbalanced dataproblem of entry format for a simple model in KerasSVM SMOTE fit_resample() function runs forever with no result