NEVER Use PIPE with Python popen

I do use Python at work time-to-time, mostly for build tools and pipelnie stuff. This is the problem I ran into a week ago, and thought I should let other people know too. (I assume a lot of game programmers use Python? Well, not in game code, though)

This is basically a summary version of this article, which I got my answer from.

• If you use redirect stdout of a children process through PIPE, it will hang when the size of output is bigger than 64KB. (And it does not give you any meaningful error message. You might see some deadlock deep into some file handling functinos in ntdll or so)
• My advice is NEVER use PIPE with popen. Even if your current subprocess never outputs more than 64K, other programmers can change this shallow assumption sometime later. (This was my case)
• Instead, always use a tempfile. (see the code example below)

Code examples:

• Not-so-good code:

    p = Popen(cmd, stdout=PIPE)
    ...
    out_lines = p.stdout.readlines()
  

• Better code I ended up using (but was told a bad advice):

    temp_file = tempfile.TemporaryFile()
    d = temp_file.fileno()
    p = Popen(cmd, stdout = d)
  
    ...
  
    out_lines = temp_file.readlines()
  

• Probably the right solution: An anonymous commenter said this is the right solution on Feb 25,2013. I haven't tried it but it sounds like the right solution. So moving it into the main post. (see how bad python programmer I am..)

    p = Popen(cmd,)
  
    ...
  
    stdoutdata, stderrdata = p.communicate(input)
  

p.s.1 You don't have to close temp_file. It'll be closed when garbage collector collects this. But still donig so would be a good practice.

p.s.2 This was done in Python 2.5.1. The reason why I'm saying this is because I just heard Python 3.0 is not backward-compatible.