How To Get Subprocess Stdout While Running Git Command?

I have a program written in python and used git command in it.. For some reason I don't want to use git-python or others instead of subprocess. But I'm currently stuck in getting g

Solution 1:

When not writing to a terminal, git clone doesn't have any output to either stdout or stderr, except on error.

When writing to a terminal, of course, it has lots of output—but that output is progress bars that are continually overwritten. Usually, you don't want that—it's going to be a big mess of control characters and repeated lines.

But if you do want it, there are two options.

---

First, you can use a PTY (Pseudo-TTY). You can create a PTY with os.openpty, then hand the PTY off explicitly to the child process. Or you can use os.forkpty, which handles forking and automatically hooking up the PTY so all you have to do is call one of the os.exec functions. Or you can use the pty module. (It's not entirely clear which is more portable; openpty and forkpty claim that pty is more portable, and conceptually it's designed that way… but it's also only really tested on Linux.)

Note that git wants the PTY as its stderr, not its stdout.

---

Alternatively, most git commands have a --progress flag that causes them to write progress to stderr even if it's not a terminal. At least as of the version documented here, this includes clone, but of course you should check the man for your local version. So, that may be all you need. (Also see the --verbose flag.)

However, this may not be as nice. For me, when I provide a PTY with no attached termcaps, I get each line followed by a \r without \n to overwrite it; when I use the --progress option, git detects the termcaps of whatever terminal my script happens to be running it, which means I end up getting ANSI color codes as well as \rs.

---

Of course either way I'm getting hundreds of useless lines that I have no interest in, but I assume that's what you wanted? (Or maybe you want to use universal_newlines='\r' to translate the '\r' to '\n'? That's slightly cheating, because this is self-overwriting Unix terminal output, and you're pretending it's classic-Mac output… but it works.)

Solution 2:

I had a similar issue, but I also wanted real-time progress because the git clone was taking a long time (like 5 minutes), since it was a large git repo. And I wanted to provide real-time feedback to the user.

Here is a Python 3.7 working example:

# This will print stdout/stderr as it comes indefrun_shell_command_with_realtime_output(shell_command_string, working_dir='.'):
    # print the command to be executed, just for funprint("run_shell_command >", shell_command_string)

    # run it, this will NOT block
    sub_process = subprocess.Popen(shell_command_string,
                                   shell=True,
                                   cwd=working_dir, universal_newlines=True,
                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # print the stdout/stderr as it comes inwhileTrue:
        # The readline() will block until...# it reads and returns a string that ends in a '\n',# or until the process has ended which will result in '' string
        output = sub_process.stdout.readline()
        if output:
            print(output.strip())
        elif sub_process.poll() isnotNone:
            break# get the return code
    return_code = sub_process.wait()

    # Was there an error?if return_code != 0:
        print("FYI, the subprocess had an error, you might want to do something special...")

    # return the sub_process, in case the caller wants to check exit/return codesreturn sub_process

It turns out git clone does not seem to write to stdout/stderr like we normally are used to. Instead it uses a pager, which is how it updates the same line, like when it is cloning, and the % is incrementing on the same line.

So you need to call it like this.

# prepare to clone the git repo
git_clone_url_with_credentials = "https://<username>:<password>@bitbucket.org/myreponame.git"
git_clone_with_progress_cmd = "git --no-pager clone --progress {}".format(git_clone_url_with_credentials)
helpers_misc.run_shell_command_with_progress(git_clone_with_progress_cmd)

And that should do it...