strange warnings from tempfile.mkstemped.__del__ on HP

[Guido van Rossum]

Lysator's snake-farm, which does regular builds of CVS Python
checkouts on a variety of uncommon platforms, has started reporting
two warnings that I don't understand. (Never mind the gettext.py
warnings; they're shallow; someone should fix them.)

The problem is the two exceptions ignored in __del__ methods. If I
look at the code of the new tempfile.py module and its
test_tempfile.py unittests, I see that there's a class mkstemped
defined in test_tempfile.py, which has a __del__ method that closes
the file descriptor. The only way I can see this failing with an
AttributeError exception is if the instance never makes it through its
__init__ call. But in that case I would have expect a failure
reported; the only instantiation of mkstemped() is inside a try/except
where the exceptclause calls self.failOnException() which causes the
unit tests to fail. But the unittest doesn't report any failures?!

I don't see this happening on Linux, so it's hard to go beyond
speculation.

--Guido van Rossum (home page: http://www.python.org/~guido/)

------- Forwarded Message

Date: Tue, 13 Aug 2002 23:04:56 -0400
From: ***@lysator.liu.se
To: snake-farm-***@lists.lysator.liu.se
Subject: [farm-report] Build python-HP_UX-B.11.00-9000_829-taylor was successfu
l.

Build test succeeded. Any warnings are appended below.
- --
/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/gettext.py:142: DeprecationWarnin
g: hex/oct constants > sys.maxint will return positive values in Python 2.4 and
up
LE_MAGIC = 0x950412de
/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/gettext.py:143: DeprecationWarnin
g: hex/oct constants > sys.maxint will return positive values in Python 2.4 and
up
BE_MAGIC = 0xde120495
/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/gettext.py:149: DeprecationWarnin
g: hex/oct constants > sys.maxint will return positive values in Python 2.4 and
up
MASK = 0xffffffff
Exception exceptions.AttributeError: "mkstemped instance has no attribute 'fd'"
in <bound method mkstemped.__del__ of <test.test_tempfile.mkstemped instance a
t 0x40705ee0>> ignored
Exception exceptions.AttributeError: "mkstemped instance has no attribute 'fd'"
in <bound method mkstemped.__del__ of <test.test_tempfile.mkstemped instance a
t 0x40afcdc8>> ignored

Stop.

_______________________________________________
Snake-farm-report mailing list
Snake-farm-***@lists.lysator.liu.se
http://lists.lysator.liu.se/mailman/listinfo/snake-farm-report

------- End of Forwarded Message

[Guido van Rossum]

Post by Guido van Rossum
The problem is the two exceptions ignored in __del__ methods. If I
look at the code of the new tempfile.py module and its
test_tempfile.py unittests, I see that there's a class mkstemped
defined in test_tempfile.py, which has a __del__ method that closes
the file descriptor. The only way I can see this failing with an
AttributeError exception is if the instance never makes it through its
__init__ call. But in that case I would have expect a failure
reported; the only instantiation of mkstemped() is inside a try/except
where the exceptclause calls self.failOnException() which causes the
unit tests to fail. But the unittest doesn't report any failures?!

Mmm, it seems the test script doesn't show the test output. Maybe one
of the tests is failing, but "make test" doesn't fail as a result? Or
only the first test run is failing? "make test" ignores the result of
the first test run (the tests are run twice, once without .pyc files
in place, once with).

--Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim Peters]

[Guido]

Post by Guido van Rossum
Lysator's snake-farm, which does regular builds of CVS Python
checkouts on a variety of uncommon platforms, has started reporting
two warnings that I don't understand. (Never mind the gettext.py
warnings; they're shallow; someone should fix them.)

I submitted a patch for that to SF and assigned it to Barry (I have no idea
how to test gettext.py).

Post by Guido van Rossum
The problem is the two exceptions ignored in __del__ methods. If I
look at the code of the new tempfile.py module and its
test_tempfile.py unittests, I see that there's a class mkstemped
defined in test_tempfile.py, which has a __del__ method that closes
the file descriptor. The only way I can see this failing with an
AttributeError exception is if the instance never makes it through its
__init__ call.

I agree, and, indeed, that's what would happen if it did fail during the
call to mkstemped.__init__(). So the call to tempfile._mkstemp_inner()
fails in two test cases (there were two distinct instances of the "no
attribute 'fd'" message), but we don't know which ones.

Post by Guido van Rossum
...
But in that case I would have expect a failure reported; the only
instantiation of mkstemped() is inside a try/except where the
exceptclause calls self.failOnException() which causes the
unit tests to fail. But the unittest doesn't report any failures?!

Well, I didn't see *any* test output in the report, neither successes nor
failures, just Python-produced exceptions and warnings. Maybe the script
only captures stderr? A failing unittest run *under* regrtest.py doesn't
normally print anything to stderr. It would have printed this to stdout,
though:

"""
...
test_tempfile
test test_tempfile failed -- errors occurred; run in verbose mode for
details
...
1 test failed:
test_tempfile
"""

So even if we had that, it wouldn't have helped. stdout from a regrtest -v
run is what we need, or from running test_tempfile.py directly (w/o
regrtest).

[Kalle Svensson]

[Tim Peters]

Post by Tim Peters
So even if we had that, it wouldn't have helped. stdout from a
regrtest -v run is what we need, or from running test_tempfile.py
directly (w/o regrtest).

Here you go.

: ***@taylor [python-HP_UX-B.11.00-9000_829-taylor]$ ; ./python ../python/dist/src/Lib/test/test_tempfile.py
There are no surprising symbols in the tempfile module ... ok
_once initializes its argument ... ok
_once calls the callback just once ... ok
_once does not modify anything but its argument ... ok
_RandomNameSequence returns a six-character string ... ok
_RandomNameSequence returns no duplicate strings (stochastic) ... ok
_RandomNameSequence supports the iterator protocol ... ok
_candidate_tempdir_list returns a nonempty list of strings ... ok
_candidate_tempdir_list contains the expected directories ... ok
_get_candidate_names returns a _RandomNameSequence object ... ok
_get_candidate_names always returns the same object ... ok
_mkstemp_inner can create files ... ok
_mkstemp_inner can create many files (stochastic) ... FAIL
Exception exceptions.AttributeError: "mkstemped instance has no attribute 'fd'" in <bound method mkstemped.__del__ of <__main__.mkstemped instance at 0x400e6fa8>> ignored
_mkstemp_inner can create files in a user-selected directory ... ok
_mkstemp_inner creates files with the proper mode ... ok
_mkstemp_inner file handles are not inherited by child processes ... ok
_mkstemp_inner can create files in text mode ... ok
gettempprefix returns a nonempty prefix string ... ok
gettempprefix returns a usable prefix string ... ok
gettempdir returns a directory which exists ... ok
gettempdir returns a directory writable by the user ... ok
gettempdir always returns the same object ... ok
mkstemp can create files ... ok
mkstemp can create directories in a user-selected directory ... ok
mkdtemp can create directories ... ok
mkdtemp can create many directories (stochastic) ... ok
mkdtemp can create directories in a user-selected directory ... ok
mkdtemp creates directories with the proper mode ... ok
mktemp can choose usable file names ... ok
mktemp can choose many usable file names (stochastic) ... ok
mktemp issues a warning when used ... ok
NamedTemporaryFile can create files ... ok
NamedTemporaryFile creates files with names ... ok
A NamedTemporaryFile is deleted when closed ... ok
A NamedTemporaryFile can be closed many times without error ... ok
TemporaryFile can create files ... ok
TemporaryFile creates files with no names (on this system) ... ok
A TemporaryFile can be closed many times without error ... ok

======================================================================
FAIL: _mkstemp_inner can create many files (stochastic)
- ----------------------------------------------------------------------
Traceback (most recent call last):
File "../python/dist/src/Lib/test/test_tempfile.py", line 295, in test_basic_many
File "../python/dist/src/Lib/test/test_tempfile.py", line 278, in do_create
File "../python/dist/src/Lib/test/test_tempfile.py", line 33, in failOnException
File "/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/unittest.py", line 260, in fail
AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24] Too many open files: '/tmp/aaU3irrA'

- ----------------------------------------------------------------------
Ran 38 tests in 43.182s

FAILED (failures=1)
Traceback (most recent call last):
File "../python/dist/src/Lib/test/test_tempfile.py", line 719, in ?
test_main()
File "../python/dist/src/Lib/test/test_tempfile.py", line 716, in test_main
test_support.run_suite(suite)
File "/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/test/test_support.py", line 188, in run_suite
raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
File "../python/dist/src/Lib/test/test_tempfile.py", line 295, in test_basic_many
File "../python/dist/src/Lib/test/test_tempfile.py", line 278, in do_create
File "../python/dist/src/Lib/test/test_tempfile.py", line 33, in failOnException
File "/mp/slaskdisk/tmp/sfarmer/python/dist/src/Lib/unittest.py", line 260, in fail
AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24] Too many open files: '/tmp/aaU3irrA'

Hmm, I wonder how many that is, and how to change it. I'll look
around.

Peace,
Kalle
- --
Kalle Svensson, http://www.juckapan.org/~kalle/
Student, root and saint in the Church of Emacs.

[Kalle Svensson]

[me, on the HP-UX snake farm build]

Post by Kalle Svensson
AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24]
Too many open files: '/tmp/aaU3irrA'
Hmm, I wonder how many that is, and how to change it. I'll look
around.

I've raised maxfiles from 200 to 2048, and the test now runs without
error.

Peace,
Kalle
- --
Kalle Svensson, http://www.juckapan.org/~kalle/
Student, root and saint in the Church of Emacs.

[Guido van Rossum]

Post by Kalle Svensson
[me, on the HP-UX snake farm build]

Post by Kalle Svensson
AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24]
Too many open files: '/tmp/aaU3irrA'
Hmm, I wonder how many that is, and how to change it. I'll look
around.

I've raised maxfiles from 200 to 2048, and the test now runs without
error.

Thanks! Maybe the test was a little too eager though -- perhaps it
could be happy with creating 100 instead of 1000 files.

--Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim Peters]

[Guido]

Post by Guido van Rossum
Thanks! Maybe the test was a little too eager though -- perhaps it
could be happy with creating 100 instead of 1000 files.

Just noting that this change has been made in current CVS, so there
shouldn't be a need to boost the HP default anymore.

[Guido van Rossum]

Post by Tim Peters

Post by Guido van Rossum
Thanks! Maybe the test was a little too eager though -- perhaps it
could be happy with creating 100 instead of 1000 files.

Just noting that this change has been made in current CVS, so there
shouldn't be a need to boost the HP default anymore.

I'd even suggest to roll back the change to the HP system, just so
that we have a realistic test environment. Having a (relatively) low
limit on open files may uncover real bugs in code that appears to work
on Linux.

--Guido van Rossum (home page: http://www.python.org/~guido/)

[Kalle Svensson]

[Guido van Rossum]

Post by Guido van Rossum
I'd even suggest to roll back the change to the HP system, just so
that we have a realistic test environment. Having a (relatively)
low limit on open files may uncover real bugs in code that appears
to work on Linux.

Done.

Peace,
Kalle
- --
Kalle Svensson, http://www.juckapan.org/~kalle/
Student, root and saint in the Church of Emacs.

[Aahz]

Post by Guido van Rossum

[Kalle, on the HP-UX snake farm build]

Post by Kalle Svensson
AssertionError: _mkstemp_inner raised exceptions.OSError: [Errno 24]
Too many open files: '/tmp/aaU3irrA'
Hmm, I wonder how many that is, and how to change it. I'll look
around.

I've raised maxfiles from 200 to 2048, and the test now runs without
error.

Thanks! Maybe the test was a little too eager though -- perhaps it
could be happy with creating 100 instead of 1000 files.

Hrm. Aren't there OSes with a default limit of 63 open files per
process? I'm pretty sure there are some with 127 (signed char).

[Guido van Rossum]

Post by Aahz
Hrm. Aren't there OSes with a default limit of 63 open files per
process? I'm pretty sure there are some with 127 (signed char).

We'll deal with those as we encounter them. All the mainstream
platforms go much beyond that (just as we have left the 640 KB limit
behind us :-).

--Guido van Rossum (home page: http://www.python.org/~guido/)