python 3.x - Bash run task via ssh and get stdout, stderr live output to file in background

Question

I am running a python 3 script via ssh, and I want to see the stdout and stderr in a file on the remote server. Also, I would like to see the file updated live while the script is running and that the script will run in the background so the ssh connection will not wait for the script to finish. While looking at other questions, I managed to answer most of my requests. Here is what I come up with: ssh user@machine_ip "(python3 my_script.py 2>&1 | tee output.log) &"

The problem is that the ssh is waiting for the script to finish.

Answer 1

So combining the answer from this question with some hackidy hack nonsense...

Really all you need to do is this:

(( python3 my_script.py 0<&- 2>&1 | tee -a ${OUTFILE} | nc -kl ${PORT} &) &)

Explanation:

1. Run your python script: python3 my_script.py
2. Detach stdin (so it can run independently): ... 0<&-
3. Redirect stderr to stdout so we can pipe them along together: ... 2>&1
4. Append output to some file but also keep the output going to stdout so we can pipe it someplace else: ... | tee -a ${OUTFILE} |
5. Pipe stdout to a netcat listening port so this machine can essentially "serve" the stdout from your python script: ... | nc -kl ${PORT}
6. Create a "double nested background subshell" this is explained in the link above but this will allow you to orphan "blah" so that it'll run even if your ssh connection ends (( ... blah ... &) &)

To view the stdout/stderr of my_script.py you now have several options. If you are still logged into that remote machine you can:

1. tail -f ${OUTFILE} # Same "OUTFILE" used in explanation component 4
2. nc localhost $PORT # Same "PORT" used in explanation component 5

If you are no longer logged in and you are now on a different machine (but on the same network) you can:

1. nc ${remote_machine} $PORT # Same "PORT" used in explanation component 5

Where ${remote_machine} is the hostname or IP address of the machine you ssh'ed into to run your command