Generator combined with with/using statement - Python vs C#

Posted on Fri 30 December 2011 in Coding

The other day, I tried to return a generator expression from within a with statement in Python, like so:

#!/usr/bin/python

def readlines(fn):
    with open(fn) as fd: # <-- the with statement
        return (line for line in fd) # <-- the generator expression

if __name__ == "__main__":
    for line in readlines("/proc/partitions"):
        print line.strip()

Running the script above results in a nasty error:

$ ./genwith.py
Traceback (most recent call last):
  File "./genwith.py", line 8, in
    for line in readlines("/proc/partitions"):
  File "./genwith.py", line 5, in
    return (line for line in fd)
ValueError: I/O operation on closed file

I wanted to see how the same combination of mechanisms works in C# (which has using instead of with):

using System;
using System.IO;
using System.Collections.Generic;

class Program {
    private static IEnumerable ReadLines(string fn) {
        using (var reader = new StreamReader(fn)) { // <-- the using statement
            string line;
            while ((line = reader.ReadLine()) != null)
                yield return line; // <-- makes ReadLines a generator method
        }
    }
    public static void Main(string[] args) {
        foreach (var line in ReadLines("/proc/partitions"))
            Console.WriteLine(line);
    }
}

And this actually works very well:

$ dmcs genuse.cs
$ ./genuse.exe
major minor  #blocks  name

   8        0   20971520 sda...

But to be fair, the C# code doesn’t use a generator expression - it uses the yield statement which converts the entire ReadLines method into a generator. So it’s not an adequate comparison at all! Let’s go back to Python and use yield there as well:

#!/usr/bin/python

def readlines(fn):
    with open(fn) as fd: # <-- the with statement
        for line in fd:
            yield line # <-- makes readlines a generator function

if __name__ == "__main__":
    for line in readlines("/proc/partitions"):
        print line.strip()

…and now the result is much better:

$ ./genwith.py
major minor  #blocks  name

   8        0   20971520 sda...

Lesson learned: Use generator expressions with caution and use yield when a context manager (which is what the with statement uses) is involved.

c# generator Python